vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #478
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker2 # Disk image does not exist, creating the virtualisation disk image...13worker2: QEMU running (pid 52)14worker2 # Formatting '/build/vm-state-worker2/tmp.CBxVjC6QaD', fmt=raw size=107374182415worker1 # Disk image does not exist, creating the virtualisation disk image...16worker2 # mke2fs 1.47.4 (6-Mar-2025)17worker1 # Formatting '/build/vm-state-worker1/tmp.MUJb9Z6q3C', fmt=raw size=107374182418worker2 # Discarding device blocks: 0/262144 done19worker1 # mke2fs 1.47.4 (6-Mar-2025)20worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes21worker1 # Discarding device blocks: 0/262144 done22worker2 # Filesystem UUID: 06ffab16-8a9b-444b-94ea-b2e9bbfde4c323worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes24worker2 # Superblock backups stored on blocks:25worker1 # Filesystem UUID: 37c4a15f-dec7-4e6f-91fb-363765585e5c26worker2 # 32768, 98304, 163840, 22937627worker1 # Superblock backups stored on blocks:28worker2 # 29worker1 # 32768, 98304, 163840, 22937630worker2 # Allocating group tables: 0/8 done31worker1 # 32worker2 # Writing inode tables: 0/8 done33worker1 # Allocating group tables: 0/8 done34lb # Disk image does not exist, creating the virtualisation disk image...35worker1: QEMU running (pid 51)36worker1 # Writing inode tables: 0/8 done37lb # Formatting '/build/vm-state-lb/tmp.Xgi6EKBF5v', fmt=raw size=107374182438worker2 # Creating journal (8192 blocks): done39lb: QEMU running (pid 50)40worker1 # Creating journal (8192 blocks): done41lb # mke2fs 1.47.4 (6-Mar-2025)42worker1 # Writing superblocks and filesystem accounting information: 0/8 done43lb # Discarding device blocks: 0/262144 done44lb # Creating filesystem with 262144 4k blocks and 65536 inodes45worker1 # 46worker2 # Writing superblocks and filesystem accounting information: 0/8 done47lb # Filesystem UUID: 5e72b276-332e-476a-86e3-5a1d4b809f5748client # Disk image does not exist, creating the virtualisation disk image...49client: QEMU running (pid 49)50lb # Superblock backups stored on blocks:51client # Formatting '/build/vm-state-client/tmp.LxDTfb54Fi', fmt=raw size=107374182452worker2 # 53client # mke2fs 1.47.4 (6-Mar-2025)54(finished: start all VMs, in 0.23 seconds)55worker1 # Virtualisation disk image created.56client # Discarding device blocks: 0/262144 done57client: waiting for unit network-addresses-eth1.service58worker2 # Virtualisation disk image created.59lb # 32768, 98304, 163840, 22937660client # Creating filesystem with 262144 4k blocks and 65536 inodes61lb # 62lb # Allocating group tables: 0/8 done63lb # Writing inode tables: 0/8 done64lb # Creating journal (8192 blocks): done65lb # Writing superblocks and filesystem accounting information: 0/8 done66lb # 67client: waiting for the VM to finish booting68client # Filesystem UUID: 885cf7d2-9d3e-4c9d-96a6-a936e429955a69lb # Virtualisation disk image created.70client # Superblock backups stored on blocks:71client # 32768, 98304, 163840, 22937672client # 73client # Allocating group tables: 0/8 done74client # Writing inode tables: 0/8 done75client # Creating journal (8192 blocks): done76client # Writing superblocks and filesystem accounting information: 0/8 done77client # 78client # Virtualisation disk image created.79lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83lb # 84lb # 85client # 86client # 87lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0088client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0089lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...90lb # 91lb # 92lb # 93lb # 94worker2 # 95client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...96worker2 # 97client # 98client # 99client # 100client # 101worker1 # 102lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00103worker1 # 104worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00105client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00106worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00107worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...108worker2 # 109worker2 # 110worker2 # 111worker2 # 112lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...113lb # 114lb # 115worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...116worker1 # 117worker1 # 118worker1 # 119worker1 # 120client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...121client # 122client # 123worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00124worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00125worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...126worker2 # 127worker2 # 128worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129worker1 # 130worker1 # 131client # Booting from ROM...132lb # Booting from ROM...133worker1 # Booting from ROM...134worker2 # Booting from ROM...135client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test/init regInfo=/nix/store/v58y4g30w74cah80479jy4r0f6qilcdy-closure-info/registration console=ttyS0,115200n8 console=tty0137client # [ 0.000000] BIOS-provided physical RAM map:138client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved141client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable142worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026143client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved144lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026145client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved146client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved147client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved148client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved149worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/l2f4v9842hgzvs7iqw4n2zjhld4v8zg2-nixos-system-worker1-test/init regInfo=/nix/store/0hkcj9bpnyb9a7hh8nvrfypvwikqd0cr-closure-info/registration console=ttyS0,115200n8 console=tty0150worker1 # [ 0.000000] BIOS-provided physical RAM map:151lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/iyv34j70v67w8qs6r16rdz87av56zg0m-nixos-system-lb-test/init regInfo=/nix/store/flbrhhbd231srwr8j05wisqrx32kc1ys-closure-info/registration console=ttyS0,115200n8 console=tty0152client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved153lb # [ 0.000000] BIOS-provided physical RAM map:154client # [ 0.000000] NX (Execute Disable) protection: active155worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable156client # [ 0.000000] APIC: Static calls initialized157lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable158client # [ 0.000000] SMBIOS 2.8 present.159worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved160worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026161lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved162worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved163client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014164lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved165worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable166client # [ 0.000000] DMI: Memory slots populated: 1/1167client # [ 0.000000] Hypervisor detected: KVM168lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable169worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved170client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000171lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved172worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved173client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00174lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved175worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved176client # [ 0.000000] kvm-clock: using sched offset of 455929131 cycles177worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/1aib8bbfkjwbph7k3yays1wshrsq1fp8-nixos-system-worker2-test/init regInfo=/nix/store/8rmndx753jcg4jq0nykzhx9jj6j93vy5-closure-info/registration console=ttyS0,115200n8 console=tty0178worker2 # [ 0.000000] BIOS-provided physical RAM map:179lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved180worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved181client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns182worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable183lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved184worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved185client # [ 0.000005] tsc: Detected 2400.012 MHz processor186worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved187lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved188worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved189client # [ 0.000808] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000190worker1 # [ 0.000000] NX (Execute Disable) protection: active191worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved192lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved193client # [ 0.000835] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs194worker1 # [ 0.000000] APIC: Static calls initialized195lb # [ 0.000000] NX (Execute Disable) protection: active196worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable197worker1 # [ 0.000000] SMBIOS 2.8 present.198lb # [ 0.000000] APIC: Static calls initialized199client # [ 0.000837] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT200lb # [ 0.000000] SMBIOS 2.8 present.201worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved202client # [ 0.002732] found SMP MP-table at [mem 0x000f5450-0x000f545f]203worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014204client # [ 0.002743] Using GB pages for direct mapping205worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved206worker1 # [ 0.000000] DMI: Memory slots populated: 1/1207lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014208client # [ 0.002861] RAMDISK: [mem 0x5e366000-0x5ffcffff]209worker1 # [ 0.000000] Hypervisor detected: KVM210worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved211lb # [ 0.000000] DMI: Memory slots populated: 1/1212worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000213client # [ 0.002870] ACPI: Early table checksum verification disabled214lb # [ 0.000000] Hypervisor detected: KVM215worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved216worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00217lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000218client # [ 0.002872] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )219worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved220lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00221worker1 # [ 0.000000] kvm-clock: using sched offset of 470982905 cycles222client # [ 0.002876] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)223lb # [ 0.000000] kvm-clock: using sched offset of 483284434 cycles224worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved225worker2 # [ 0.000000] NX (Execute Disable) protection: active226worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns227client # [ 0.002880] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)228worker2 # [ 0.000000] APIC: Static calls initialized229worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor230worker2 # [ 0.000000] SMBIOS 2.8 present.231worker1 # [ 0.000808] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000232lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns233client # [ 0.002886] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)234lb # [ 0.000005] tsc: Detected 2400.012 MHz processor235client # [ 0.002888] ACPI: FACS 0x000000005FFE0000 000040236lb # [ 0.000804] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000237worker1 # [ 0.000835] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs238worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014239client # [ 0.002890] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)240worker2 # [ 0.000000] DMI: Memory slots populated: 1/1241lb # [ 0.000831] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs242worker1 # [ 0.000837] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT243worker2 # [ 0.000000] Hypervisor detected: KVM244lb # [ 0.000834] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT245worker1 # [ 0.002735] found SMP MP-table at [mem 0x000f5450-0x000f545f]246client # [ 0.002891] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)247worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000248worker1 # [ 0.002746] Using GB pages for direct mapping249worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00250lb # [ 0.002728] found SMP MP-table at [mem 0x000f5450-0x000f545f]251client # [ 0.002893] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)252worker1 # [ 0.002837] RAMDISK: [mem 0x5e367000-0x5ffcffff]253lb # [ 0.002739] Using GB pages for direct mapping254worker2 # [ 0.000000] kvm-clock: using sched offset of 475801351 cycles255lb # [ 0.002815] RAMDISK: [mem 0x5e367000-0x5ffcffff]256worker1 # [ 0.002845] ACPI: Early table checksum verification disabled257client # [ 0.002895] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)258worker1 # [ 0.002847] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )259lb # [ 0.002824] ACPI: Early table checksum verification disabled260worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns261client # [ 0.002896] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]262lb # [ 0.002827] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )263worker2 # [ 0.000004] tsc: Detected 2400.012 MHz processor264worker1 # [ 0.002851] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)265client # [ 0.002897] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]266worker2 # [ 0.000804] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000267lb # [ 0.002831] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)268worker1 # [ 0.002855] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)269client # [ 0.002897] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]270worker2 # [ 0.000831] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs271lb # [ 0.002835] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)272worker1 # [ 0.002862] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)273client # [ 0.002898] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]274worker2 # [ 0.000833] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT275worker1 # [ 0.002864] ACPI: FACS 0x000000005FFE0000 000040276lb # [ 0.002842] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)277client # [ 0.002898] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]278worker2 # [ 0.002740] found SMP MP-table at [mem 0x000f5450-0x000f545f]279lb # [ 0.002844] ACPI: FACS 0x000000005FFE0000 000040280worker2 # [ 0.002751] Using GB pages for direct mapping281worker1 # [ 0.002865] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)282client # [ 0.002899] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]283worker2 # [ 0.002854] RAMDISK: [mem 0x5e367000-0x5ffcffff]284lb # [ 0.002846] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)285client # [ 0.002899] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]286worker2 # [ 0.002863] ACPI: Early table checksum verification disabled287worker1 # [ 0.002867] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)288client # [ 0.003124] No NUMA configuration found289lb # [ 0.002847] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)290worker2 # [ 0.002866] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )291client # [ 0.003125] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]292worker1 # [ 0.002868] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)293lb # [ 0.002849] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)294worker2 # [ 0.002870] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)295client # [ 0.003128] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]296worker1 # [ 0.002870] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)297client # [ 0.005425] Zone ranges:298lb # [ 0.002850] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)299worker2 # [ 0.002873] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)300worker1 # [ 0.002871] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]301client # [ 0.005426] DMA [mem 0x0000000000001000-0x0000000000ffffff]302lb # [ 0.002852] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]303client # [ 0.005428] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]304worker1 # [ 0.002872] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]305worker2 # [ 0.002880] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)306client # [ 0.005429] Normal empty307worker2 # [ 0.002882] ACPI: FACS 0x000000005FFE0000 000040308client # [ 0.005430] Device empty309lb # [ 0.002853] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]310worker1 # [ 0.002873] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]311client # [ 0.005430] Movable zone start for each node312client # [ 0.005431] Early memory node ranges313lb # [ 0.002853] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]314worker1 # [ 0.002873] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]315worker2 # [ 0.002884] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)316client # [ 0.005432] node 0: [mem 0x0000000000001000-0x000000000009efff]317lb # [ 0.002854] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]318worker1 # [ 0.002874] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]319worker2 # [ 0.002885] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)320client # [ 0.005433] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]321lb # [ 0.002854] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]322worker1 # [ 0.002874] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]323worker2 # [ 0.002887] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)324client # [ 0.005434] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]325lb # [ 0.002855] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]326worker1 # [ 0.002875] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]327client # [ 0.005454] On node 0, zone DMA: 1 pages in unavailable ranges328worker1 # [ 0.003100] No NUMA configuration found329worker2 # [ 0.002888] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)330lb # [ 0.002855] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]331client # [ 0.005728] On node 0, zone DMA: 97 pages in unavailable ranges332lb # [ 0.003093] No NUMA configuration found333worker1 # [ 0.003101] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]334worker2 # [ 0.002889] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]335client # [ 0.032836] On node 0, zone DMA32: 40 pages in unavailable ranges336lb # [ 0.003094] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]337worker1 # [ 0.003104] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]338worker2 # [ 0.002891] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]339worker1 # [ 0.005425] Zone ranges:340client # [ 0.033290] ACPI: PM-Timer IO Port: 0x608341lb # [ 0.003096] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]342lb # [ 0.005516] Zone ranges:343client # [ 0.033301] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])344worker1 # [ 0.005426] DMA [mem 0x0000000000001000-0x0000000000ffffff]345worker2 # [ 0.002891] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]346client # [ 0.033328] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23347lb # [ 0.005517] DMA [mem 0x0000000000001000-0x0000000000ffffff]348worker1 # [ 0.005428] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]349worker2 # [ 0.002892] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]350worker1 # [ 0.005429] Normal empty351worker1 # [ 0.005430] Device empty352lb # [ 0.005519] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]353client # [ 0.033331] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)354worker2 # [ 0.002892] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]355lb # [ 0.005520] Normal empty356worker1 # [ 0.005430] Movable zone start for each node357lb # [ 0.005521] Device empty358client # [ 0.033333] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)359worker1 # [ 0.005431] Early memory node ranges360worker2 # [ 0.002893] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]361lb # [ 0.005521] Movable zone start for each node362lb # [ 0.005522] Early memory node ranges363client # [ 0.033334] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)364worker1 # [ 0.005431] node 0: [mem 0x0000000000001000-0x000000000009efff]365worker2 # [ 0.002893] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]366lb # [ 0.005522] node 0: [mem 0x0000000000001000-0x000000000009efff]367worker1 # [ 0.005432] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]368worker2 # [ 0.003127] No NUMA configuration found369client # [ 0.033335] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)370lb # [ 0.005524] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]371worker2 # [ 0.003128] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]372worker1 # [ 0.005434] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]373client # [ 0.033336] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)374worker2 # [ 0.003130] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]375worker1 # [ 0.005453] On node 0, zone DMA: 1 pages in unavailable ranges376client # [ 0.033338] ACPI: Using ACPI (MADT) for SMP configuration information377lb # [ 0.005525] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]378worker2 # [ 0.005513] Zone ranges:379client # [ 0.033339] ACPI: HPET id: 0x8086a201 base: 0xfed00000380lb # [ 0.005546] On node 0, zone DMA: 1 pages in unavailable ranges381worker1 # [ 0.006039] On node 0, zone DMA: 97 pages in unavailable ranges382worker2 # [ 0.005514] DMA [mem 0x0000000000001000-0x0000000000ffffff]383client # [ 0.033343] TSC deadline timer available384lb # [ 0.005824] On node 0, zone DMA: 97 pages in unavailable ranges385worker1 # [ 0.033110] On node 0, zone DMA32: 40 pages in unavailable ranges386client # [ 0.033347] CPU topo: Max. logical packages: 1387worker2 # [ 0.005515] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]388worker1 # [ 0.033569] ACPI: PM-Timer IO Port: 0x608389worker2 # [ 0.005516] Normal empty390client # [ 0.033348] CPU topo: Max. logical dies: 1391lb # [ 0.034147] On node 0, zone DMA32: 40 pages in unavailable ranges392worker2 # [ 0.005517] Device empty393worker1 # [ 0.033581] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])394lb # [ 0.034607] ACPI: PM-Timer IO Port: 0x608395client # [ 0.033348] CPU topo: Max. dies per package: 1396worker2 # [ 0.005518] Movable zone start for each node397worker2 # [ 0.005519] Early memory node ranges398lb # [ 0.034619] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])399worker1 # [ 0.033607] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23400client # [ 0.033352] CPU topo: Max. threads per core: 1401client # [ 0.033352] CPU topo: Num. cores per package: 1402worker2 # [ 0.005519] node 0: [mem 0x0000000000001000-0x000000000009efff]403lb # [ 0.034647] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23404worker1 # [ 0.033610] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)405client # [ 0.033353] CPU topo: Num. threads per package: 1406lb # [ 0.034649] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)407worker2 # [ 0.005520] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]408worker1 # [ 0.033612] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)409client # [ 0.033353] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs410worker2 # [ 0.005521] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]411lb # [ 0.034651] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)412worker1 # [ 0.033613] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)413client # [ 0.033371] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()414worker2 # [ 0.005541] On node 0, zone DMA: 1 pages in unavailable ranges415lb # [ 0.034652] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)416worker1 # [ 0.033614] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)417client # [ 0.033402] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]418worker2 # [ 0.005818] On node 0, zone DMA: 97 pages in unavailable ranges419lb # [ 0.034654] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)420worker1 # [ 0.033614] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)421client # [ 0.033403] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]422worker2 # [ 0.033934] On node 0, zone DMA32: 40 pages in unavailable ranges423lb # [ 0.034654] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)424worker1 # [ 0.033617] ACPI: Using ACPI (MADT) for SMP configuration information425worker2 # [ 0.034401] ACPI: PM-Timer IO Port: 0x608426client # [ 0.033405] [mem 0x60000000-0xafffffff] available for PCI devices427worker1 # [ 0.033618] ACPI: HPET id: 0x8086a201 base: 0xfed00000428lb # [ 0.034657] ACPI: Using ACPI (MADT) for SMP configuration information429worker2 # [ 0.034414] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])430client # [ 0.033406] Booting paravirtualized kernel on KVM431worker1 # [ 0.033621] TSC deadline timer available432lb # [ 0.034658] ACPI: HPET id: 0x8086a201 base: 0xfed00000433worker2 # [ 0.034443] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23434worker1 # [ 0.033625] CPU topo: Max. logical packages: 1435lb # [ 0.034661] TSC deadline timer available436worker1 # [ 0.033626] CPU topo: Max. logical dies: 1437client # [ 0.033409] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns438worker2 # [ 0.034446] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)439lb # [ 0.034666] CPU topo: Max. logical packages: 1440worker1 # [ 0.033626] CPU topo: Max. dies per package: 1441lb # [ 0.034666] CPU topo: Max. logical dies: 1442client # [ 0.037848] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1443worker2 # [ 0.034448] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)444worker1 # [ 0.033630] CPU topo: Max. threads per core: 1445lb # [ 0.034667] CPU topo: Max. dies per package: 1446worker1 # [ 0.033630] CPU topo: Num. cores per package: 1447client # [ 0.040030] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152448worker2 # [ 0.034449] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)449lb # [ 0.034670] CPU topo: Max. threads per core: 1450worker1 # [ 0.033631] CPU topo: Num. threads per package: 1451lb # [ 0.034671] CPU topo: Num. cores per package: 1452client # [ 0.040074] kvm-guest: PV spinlocks disabled, single CPU453worker2 # [ 0.034450] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)454worker1 # [ 0.033631] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs455lb # [ 0.034671] CPU topo: Num. threads per package: 1456worker2 # [ 0.034451] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)457lb # [ 0.034672] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs458worker1 # [ 0.033649] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()459worker2 # [ 0.034453] ACPI: Using ACPI (MADT) for SMP configuration information460lb # [ 0.034690] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()461worker1 # [ 0.033679] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]462worker2 # [ 0.034454] ACPI: HPET id: 0x8086a201 base: 0xfed00000463worker2 # [ 0.034458] TSC deadline timer available464lb # [ 0.034722] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]465worker1 # [ 0.033681] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]466worker2 # [ 0.034462] CPU topo: Max. logical packages: 1467client # [ 0.040076] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test/init regInfo=/nix/store/v58y4g30w74cah80479jy4r0f6qilcdy-closure-info/registration console=ttyS0,115200n8 console=tty0468worker1 # [ 0.033682] [mem 0x60000000-0xafffffff] available for PCI devices469worker2 # [ 0.034463] CPU topo: Max. logical dies: 1470lb # [ 0.034724] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]471worker2 # [ 0.034463] CPU topo: Max. dies per package: 1472worker1 # [ 0.033683] Booting paravirtualized kernel on KVM473lb # [ 0.034725] [mem 0x60000000-0xafffffff] available for PCI devices474worker2 # [ 0.034466] CPU topo: Max. threads per core: 1475lb # [ 0.034727] Booting paravirtualized kernel on KVM476client # [ 0.040165] Unknown kernel command line parameters "regInfo=/nix/store/v58y4g30w74cah80479jy4r0f6qilcdy-closure-info/registration", will be passed to user space.477worker2 # [ 0.034467] CPU topo: Num. cores per package: 1478worker1 # [ 0.033686] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns479client # [ 0.040178] random: crng init done480worker2 # [ 0.034467] CPU topo: Num. threads per package: 1481lb # [ 0.034730] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns482client # [ 0.040179] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes483worker1 # [ 0.038138] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1484worker2 # [ 0.034468] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs485lb # [ 0.039184] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1486client # [ 0.042332] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)487worker1 # [ 0.040335] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152488worker2 # [ 0.034486] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()489lb # [ 0.041795] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152490worker1 # [ 0.040379] kvm-guest: PV spinlocks disabled, single CPU491client # [ 0.043680] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)492worker2 # [ 0.034519] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]493lb # [ 0.041845] kvm-guest: PV spinlocks disabled, single CPU494client # [ 0.043709] Fallback order for Node 0: 0495worker2 # [ 0.034521] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]496client # [ 0.043712] Built 1 zonelists, mobility grouping on. Total pages: 393078497client # [ 0.043713] Policy zone: DMA32498worker2 # [ 0.034523] [mem 0x60000000-0xafffffff] available for PCI devices499client # [ 0.046508] mem auto-init: stack:all(zero), heap alloc:on, heap free:off500worker2 # [ 0.034524] Booting paravirtualized kernel on KVM501client # [ 0.049592] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1502worker1 # [ 0.040380] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/l2f4v9842hgzvs7iqw4n2zjhld4v8zg2-nixos-system-worker1-test/init regInfo=/nix/store/0hkcj9bpnyb9a7hh8nvrfypvwikqd0cr-closure-info/registration console=ttyS0,115200n8 console=tty0503worker2 # [ 0.034527] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns504client # [ 0.052957] allocated 3145728 bytes of page_ext505lb # [ 0.041846] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/iyv34j70v67w8qs6r16rdz87av56zg0m-nixos-system-lb-test/init regInfo=/nix/store/flbrhhbd231srwr8j05wisqrx32kc1ys-closure-info/registration console=ttyS0,115200n8 console=tty0506client # [ 0.062908] ftrace: allocating 48731 entries in 192 pages507worker2 # [ 0.038985] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1508client # [ 0.062910] ftrace: allocated 192 pages with 2 groups509worker1 # [ 0.040471] Unknown kernel command line parameters "regInfo=/nix/store/0hkcj9bpnyb9a7hh8nvrfypvwikqd0cr-closure-info/registration", will be passed to user space.510client # [ 0.063753] Dynamic Preempt: lazy511worker1 # [ 0.040483] random: crng init done512worker2 # [ 0.041256] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152513lb # [ 0.041950] Unknown kernel command line parameters "regInfo=/nix/store/flbrhhbd231srwr8j05wisqrx32kc1ys-closure-info/registration", will be passed to user space.514worker2 # [ 0.041304] kvm-guest: PV spinlocks disabled, single CPU515client # [ 0.063908] rcu: Preemptible hierarchical RCU implementation.516lb # [ 0.041963] random: crng init done517worker1 # [ 0.040484] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes518client # [ 0.063908] rcu: RCU event tracing is enabled.519lb # [ 0.041964] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes520worker1 # [ 0.042643] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)521client # [ 0.063909] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.522lb # [ 0.044202] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)523client # [ 0.063911] Trampoline variant of Tasks RCU enabled.524worker1 # [ 0.043723] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)525client # [ 0.063911] Rude variant of Tasks RCU enabled.526worker1 # [ 0.043754] Fallback order for Node 0: 0527lb # [ 0.045310] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)528client # [ 0.063912] Tracing variant of Tasks RCU enabled.529lb # [ 0.045341] Fallback order for Node 0: 0530worker1 # [ 0.043757] Built 1 zonelists, mobility grouping on. Total pages: 393078531worker2 # [ 0.041305] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/1aib8bbfkjwbph7k3yays1wshrsq1fp8-nixos-system-worker2-test/init regInfo=/nix/store/8rmndx753jcg4jq0nykzhx9jj6j93vy5-closure-info/registration console=ttyS0,115200n8 console=tty0532worker1 # [ 0.043758] Policy zone: DMA32533client # [ 0.063912] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.534lb # [ 0.045344] Built 1 zonelists, mobility grouping on. Total pages: 393078535lb # [ 0.045345] Policy zone: DMA32536worker1 # [ 0.046268] mem auto-init: stack:all(zero), heap alloc:on, heap free:off537client # [ 0.063913] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1538lb # [ 0.047958] mem auto-init: stack:all(zero), heap alloc:on, heap free:off539worker2 # [ 0.041402] Unknown kernel command line parameters "regInfo=/nix/store/8rmndx753jcg4jq0nykzhx9jj6j93vy5-closure-info/registration", will be passed to user space.540worker1 # [ 0.049560] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1541client # [ 0.063939] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.542worker2 # [ 0.041414] random: crng init done543worker1 # [ 0.052921] allocated 3145728 bytes of page_ext544lb # [ 0.051252] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1545lb # [ 0.055103] allocated 3145728 bytes of page_ext546worker1 # [ 0.062821] ftrace: allocating 48731 entries in 192 pages547worker2 # [ 0.041415] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes548client # [ 0.063941] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.549lb # [ 0.065457] ftrace: allocating 48731 entries in 192 pages550worker1 # [ 0.062823] ftrace: allocated 192 pages with 2 groups551worker1 # [ 0.063691] Dynamic Preempt: lazy552worker2 # [ 0.043643] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)553lb # [ 0.065459] ftrace: allocated 192 pages with 2 groups554client # [ 0.063942] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.555lb # [ 0.066389] Dynamic Preempt: lazy556worker1 # [ 0.063887] rcu: Preemptible hierarchical RCU implementation.557worker2 # [ 0.045022] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)558client # [ 0.068643] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16559worker1 # [ 0.063888] rcu: RCU event tracing is enabled.560lb # [ 0.066524] rcu: Preemptible hierarchical RCU implementation.561worker2 # [ 0.045054] Fallback order for Node 0: 0562lb # [ 0.066525] rcu: RCU event tracing is enabled.563worker1 # [ 0.063889] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.564client # [ 0.068929] rcu: srcu_init: Setting srcu_struct sizes based on contention.565worker2 # [ 0.045057] Built 1 zonelists, mobility grouping on. Total pages: 393078566lb # [ 0.066525] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.567worker1 # [ 0.063890] Trampoline variant of Tasks RCU enabled.568worker2 # [ 0.045058] Policy zone: DMA32569client # [ 0.068936] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns570worker1 # [ 0.063890] Rude variant of Tasks RCU enabled.571lb # [ 0.066527] Trampoline variant of Tasks RCU enabled.572worker2 # [ 0.047712] mem auto-init: stack:all(zero), heap alloc:on, heap free:off573lb # [ 0.066528] Rude variant of Tasks RCU enabled.574worker1 # [ 0.063891] Tracing variant of Tasks RCU enabled.575worker2 # [ 0.051017] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1576client # [ 0.069036] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)577lb # [ 0.066528] Tracing variant of Tasks RCU enabled.578worker1 # [ 0.063891] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.579client # [ 0.072550] Console: colour VGA+ 80x25580worker2 # [ 0.054503] allocated 3145728 bytes of page_ext581lb # [ 0.066529] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.582client # [ 0.072553] printk: legacy console [tty0] enabled583worker1 # [ 0.063892] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1584worker2 # [ 0.064660] ftrace: allocating 48731 entries in 192 pages585client # [ 0.101961] printk: legacy console [ttyS0] enabled586worker2 # [ 0.064662] ftrace: allocated 192 pages with 2 groups587lb # [ 0.066529] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1588worker2 # [ 0.065538] Dynamic Preempt: lazy589worker1 # [ 0.063914] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.590client # [ 0.198016] ACPI: Core revision 20250807591lb # [ 0.066545] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.592worker2 # [ 0.065679] rcu: Preemptible hierarchical RCU implementation.593worker1 # [ 0.063915] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.594worker2 # [ 0.065680] rcu: RCU event tracing is enabled.595lb # [ 0.066546] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.596client # [ 0.198918] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns597worker2 # [ 0.065680] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.598worker1 # [ 0.063916] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.599worker2 # [ 0.065682] Trampoline variant of Tasks RCU enabled.600client # [ 0.200425] APIC: Switch to symmetric I/O mode setup601lb # [ 0.066547] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.602worker1 # [ 0.068299] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16603worker2 # [ 0.065683] Rude variant of Tasks RCU enabled.604client # [ 0.201352] x2apic enabled605lb # [ 0.071066] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16606worker2 # [ 0.065683] Tracing variant of Tasks RCU enabled.607worker1 # [ 0.068581] rcu: srcu_init: Setting srcu_struct sizes based on contention.608lb # [ 0.071343] rcu: srcu_init: Setting srcu_struct sizes based on contention.609client # [ 0.202090] APIC: Switched APIC routing to: physical x2apic610worker2 # [ 0.065684] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.611worker1 # [ 0.068588] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns612worker2 # [ 0.065684] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1613lb # [ 0.071350] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns614worker1 # [ 0.068683] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)615worker2 # [ 0.065735] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.616client # [ 0.203883] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1617worker1 # [ 0.072207] Console: colour VGA+ 80x25618lb # [ 0.071456] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)619worker1 # [ 0.072209] printk: legacy console [tty0] enabled620lb # [ 0.075020] Console: colour VGA+ 80x25621worker2 # [ 0.065737] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.622worker1 # [ 0.101574] printk: legacy console [ttyS0] enabled623lb # [ 0.075023] printk: legacy console [tty0] enabled624client # [ 0.204836] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns625lb # [ 0.104481] printk: legacy console [ttyS0] enabled626worker1 # [ 0.197261] ACPI: Core revision 20250807627worker2 # [ 0.065738] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.628lb # [ 0.200007] ACPI: Core revision 20250807629worker2 # [ 0.070184] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16630client # [ 0.206440] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)631worker1 # [ 0.198188] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns632worker2 # [ 0.070461] rcu: srcu_init: Setting srcu_struct sizes based on contention.633client # [ 0.207525] x86/cpu: User Mode Instruction Prevention (UMIP) activated634lb # [ 0.200898] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns635worker1 # [ 0.199701] APIC: Switch to symmetric I/O mode setup636worker2 # [ 0.070469] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns637client # [ 0.208579] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127638lb # [ 0.202432] APIC: Switch to symmetric I/O mode setup639worker1 # [ 0.200678] x2apic enabled640lb # [ 0.203391] x2apic enabled641worker2 # [ 0.070578] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)642client # [ 0.209439] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0643worker1 # [ 0.201453] APIC: Switched APIC routing to: physical x2apic644worker2 # [ 0.074401] Console: colour VGA+ 80x25645lb # [ 0.204127] APIC: Switched APIC routing to: physical x2apic646worker2 # [ 0.074404] printk: legacy console [tty0] enabled647worker2 # [ 0.104093] printk: legacy console [ttyS0] enabled648client # [ 0.210442] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto649worker1 # [ 0.203247] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1650worker2 # [ 0.199785] ACPI: Core revision 20250807651lb # [ 0.205916] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1652client # [ 0.211438] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl653worker1 # [ 0.204243] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns654worker2 # [ 0.200679] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns655client # [ 0.212438] Transient Scheduler Attacks: Vulnerable: No microcode656lb # [ 0.206883] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns657worker2 # [ 0.202182] APIC: Switch to symmetric I/O mode setup658client # [ 0.213437] Spectre V2 : Mitigation: Enhanced / Automatic IBRS659worker1 # [ 0.205902] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)660worker2 # [ 0.203171] x2apic enabled661lb # [ 0.208505] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)662client # [ 0.214438] Speculative Return Stack Overflow: Mitigation: Safe RET663worker1 # [ 0.206986] x86/cpu: User Mode Instruction Prevention (UMIP) activated664worker2 # [ 0.203942] APIC: Switched APIC routing to: physical x2apic665lb # [ 0.209592] x86/cpu: User Mode Instruction Prevention (UMIP) activated666worker1 # [ 0.208018] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127667lb # [ 0.210634] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127668worker1 # [ 0.209627] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0669worker2 # [ 0.205751] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1670lb # [ 0.211502] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0671client # [ 0.215437] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization672worker2 # [ 0.206744] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns673worker1 # [ 0.210676] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto674lb # [ 0.212506] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto675client # [ 0.217445] Spectre V2 : Enabling IBPB for BPF676worker2 # [ 0.208392] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)677worker1 # [ 0.211899] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl678lb # [ 0.214502] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl679client # [ 0.218439] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier680worker1 # [ 0.212900] Transient Scheduler Attacks: Vulnerable: No microcode681worker2 # [ 0.209478] x86/cpu: User Mode Instruction Prevention (UMIP) activated682lb # [ 0.215503] Transient Scheduler Attacks: Vulnerable: No microcode683client # [ 0.219438] active return thunk: srso_alias_return_thunk684worker1 # [ 0.214603] Spectre V2 : Mitigation: Enhanced / Automatic IBRS685worker2 # [ 0.210535] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127686lb # [ 0.216501] Spectre V2 : Mitigation: Enhanced / Automatic IBRS687client # [ 0.220459] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'688worker1 # [ 0.214899] Speculative Return Stack Overflow: Mitigation: Safe RET689lb # [ 0.217502] Speculative Return Stack Overflow: Mitigation: Safe RET690worker2 # [ 0.212057] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0691client # [ 0.221438] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'692client # [ 0.222438] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'693worker2 # [ 0.212394] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto694client # [ 0.223438] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'695worker2 # [ 0.214389] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl696client # [ 0.224437] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'697worker1 # [ 0.215899] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization698lb # [ 0.218501] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization699worker2 # [ 0.215389] Transient Scheduler Attacks: Vulnerable: No microcode700worker1 # [ 0.217906] Spectre V2 : Enabling IBPB for BPF701client # [ 0.225437] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'702lb # [ 0.220509] Spectre V2 : Enabling IBPB for BPF703worker2 # [ 0.217081] Spectre V2 : Mitigation: Enhanced / Automatic IBRS704worker1 # [ 0.218900] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier705client # [ 0.226438] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'706lb # [ 0.221208] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier707worker2 # [ 0.217389] Speculative Return Stack Overflow: Mitigation: Safe RET708worker1 # [ 0.219899] active return thunk: srso_alias_return_thunk709client # [ 0.227438] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256710lb # [ 0.222503] active return thunk: srso_alias_return_thunk711client # [ 0.229153] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64712worker1 # [ 0.220920] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'713lb # [ 0.223523] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'714client # [ 0.230171] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512715worker1 # [ 0.221899] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'716lb # [ 0.224502] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'717client # [ 0.231148] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024718worker1 # [ 0.223649] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'719worker2 # [ 0.218389] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization720lb # [ 0.225501] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'721client # [ 0.231438] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8722worker2 # [ 0.220397] Spectre V2 : Enabling IBPB for BPF723worker1 # [ 0.224622] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'724lb # [ 0.226502] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'725worker1 # [ 0.225626] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'726lb # [ 0.227502] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'727worker2 # [ 0.221390] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier728client # [ 0.232438] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.729worker1 # [ 0.225899] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'730worker2 # [ 0.222390] active return thunk: srso_alias_return_thunk731lb # [ 0.228501] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'732worker1 # [ 0.226899] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'733worker2 # [ 0.223410] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'734lb # [ 0.229501] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'735worker1 # [ 0.228899] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256736worker2 # [ 0.224389] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'737lb # [ 0.230502] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256738worker1 # [ 0.229838] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64739lb # [ 0.231501] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64740worker2 # [ 0.226118] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'741worker1 # [ 0.230620] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512742lb # [ 0.232501] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512743worker2 # [ 0.226389] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'744worker1 # [ 0.231615] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024745lb # [ 0.233501] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024746worker2 # [ 0.227389] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'747worker1 # [ 0.232631] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8748lb # [ 0.234501] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8749worker2 # [ 0.228389] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'750worker2 # [ 0.229389] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'751worker1 # [ 0.233621] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.752lb # [ 0.235501] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.753worker2 # [ 0.231293] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256754worker2 # [ 0.232142] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64755worker2 # [ 0.233128] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512756worker2 # [ 0.234126] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024757worker2 # [ 0.235125] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8758worker2 # [ 0.236126] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759client # [ 0.267361] Freeing SMP alternatives memory: 44K760client # [ 0.268077] pid_max: default: 32768 minimum: 301761client # [ 0.268530] LSM: initializing lsm=capability,landlock,yama,bpf,ima762client # [ 0.269548] landlock: Up and running.763client # [ 0.270919] Yama: becoming mindful.764client # [ 0.271658] LSM support for eBPF active765client # [ 0.272555] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)766client # [ 0.273476] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)767worker1 # [ 0.267592] Freeing SMP alternatives memory: 44K768worker1 # [ 0.267902] pid_max: default: 32768 minimum: 301769lb # [ 0.270935] Freeing SMP alternatives memory: 44K770client # [ 0.276314] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)771worker1 # [ 0.268999] LSM: initializing lsm=capability,landlock,yama,bpf,ima772lb # [ 0.271506] pid_max: default: 32768 minimum: 301773worker1 # [ 0.269993] landlock: Up and running.774lb # [ 0.272586] LSM: initializing lsm=capability,landlock,yama,bpf,ima775worker1 # [ 0.270900] Yama: becoming mindful.776client # [ 0.276970] Performance Events: Fam17h+ core perfctr, AMD PMU driver.777lb # [ 0.273613] landlock: Up and running.778client # [ 0.277443] ... version: 2779worker1 # [ 0.271683] LSM support for eBPF active780lb # [ 0.274229] Yama: becoming mindful.781client # [ 0.278138] ... bit width: 48782lb # [ 0.274723] LSM support for eBPF active783client # [ 0.278440] ... generic counters: 6784worker1 # [ 0.272494] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)785client # [ 0.279147] ... generic bitmap: 000000000000003f786worker1 # [ 0.273806] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)787lb # [ 0.275645] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)788worker2 # [ 0.270740] Freeing SMP alternatives memory: 44K789client # [ 0.279486] ... fixed-purpose counters: 0790worker2 # [ 0.271392] pid_max: default: 32768 minimum: 301791client # [ 0.280170] ... fixed-purpose bitmap: 0000000000000000792lb # [ 0.276542] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)793worker2 # [ 0.272419] LSM: initializing lsm=capability,landlock,yama,bpf,ima794client # [ 0.280441] ... value mask: 0000ffffffffffff795worker1 # [ 0.276064] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)796worker2 # [ 0.273457] landlock: Up and running.797client # [ 0.281323] ... max period: 00007fffffffffff798worker2 # [ 0.274061] Yama: becoming mindful.799lb # [ 0.279454] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)800client # [ 0.282124] ... global_ctrl mask: 000000000000003f801worker1 # [ 0.277470] Performance Events: Fam17h+ core perfctr, AMD PMU driver.802worker2 # [ 0.274602] LSM support for eBPF active803client # [ 0.282556] signal: max sigframe size: 3376804worker1 # [ 0.277908] ... version: 2805lb # [ 0.280038] Performance Events: Fam17h+ core perfctr, AMD PMU driver.806worker1 # [ 0.278589] ... bit width: 48807client # [ 0.283301] rcu: Hierarchical SRCU implementation.808worker2 # [ 0.275517] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)809lb # [ 0.280506] ... version: 2810worker1 # [ 0.278902] ... generic counters: 6811client # [ 0.283983] rcu: Max phase no-delay instances is 400.812lb # [ 0.281174] ... bit width: 48813worker2 # [ 0.276440] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)814worker1 # [ 0.279598] ... generic bitmap: 000000000000003f815lb # [ 0.281504] ... generic counters: 6816worker1 # [ 0.279901] ... fixed-purpose counters: 0817lb # [ 0.282207] ... generic bitmap: 000000000000003f818worker1 # [ 0.280595] ... fixed-purpose bitmap: 0000000000000000819lb # [ 0.282550] ... fixed-purpose counters: 0820worker1 # [ 0.280901] ... value mask: 0000ffffffffffff821worker2 # [ 0.279203] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)822lb # [ 0.283235] ... fixed-purpose bitmap: 0000000000000000823worker1 # [ 0.281775] ... max period: 00007fffffffffff824lb # [ 0.283511] ... value mask: 0000ffffffffffff825worker2 # [ 0.279943] Performance Events: Fam17h+ core perfctr, AMD PMU driver.826worker1 # [ 0.282578] ... global_ctrl mask: 000000000000003f827worker2 # [ 0.280394] ... version: 2828client # [ 0.289045] smp: Bringing up secondary CPUs ...829lb # [ 0.284434] ... max period: 00007fffffffffff830worker1 # [ 0.283009] signal: max sigframe size: 3376831worker2 # [ 0.281083] ... bit width: 48832client # [ 0.289454] smp: Brought up 1 node, 1 CPU833lb # [ 0.285196] ... global_ctrl mask: 000000000000003f834worker1 # [ 0.283774] rcu: Hierarchical SRCU implementation.835worker2 # [ 0.281392] ... generic counters: 6836client # [ 0.290105] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)837lb # [ 0.285618] signal: max sigframe size: 3376838worker1 # [ 0.284447] rcu: Max phase no-delay instances is 400.839worker2 # [ 0.282077] ... generic bitmap: 000000000000003f840lb # [ 0.286376] rcu: Hierarchical SRCU implementation.841worker2 # [ 0.282403] ... fixed-purpose counters: 0842lb # [ 0.287082] rcu: Max phase no-delay instances is 400.843worker2 # [ 0.283102] ... fixed-purpose bitmap: 0000000000000000844client # [ 0.290618] Memory: 1454564K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)845client # [ 0.291674] devtmpfs: initialized846worker2 # [ 0.283391] ... value mask: 0000ffffffffffff847worker2 # [ 0.284291] ... max period: 00007fffffffffff848client # [ 0.292517] x86/mm: Memory block size: 128MB849worker2 # [ 0.285101] ... global_ctrl mask: 000000000000003f850worker1 # [ 0.289411] smp: Bringing up secondary CPUs ...851worker2 # [ 0.285505] signal: max sigframe size: 3376852worker1 # [ 0.289915] smp: Brought up 1 node, 1 CPU853client # [ 0.294322] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)854worker2 # [ 0.286305] rcu: Hierarchical SRCU implementation.855lb # [ 0.292096] smp: Bringing up secondary CPUs ...856worker1 # [ 0.290574] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)857worker2 # [ 0.286964] rcu: Max phase no-delay instances is 400.858client # [ 0.295311] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).859lb # [ 0.292518] smp: Brought up 1 node, 1 CPU860client # [ 0.296516] pinctrl core: initialized pinctrl subsystem861lb # [ 0.293155] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)862worker1 # [ 0.291104] Memory: 1454564K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)863client # [ 0.297604] PM: RTC time: 13:53:36, date: 2026-09-21864worker1 # [ 0.292136] devtmpfs: initialized865lb # [ 0.293677] Memory: 1454556K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)866worker1 # [ 0.292940] x86/mm: Memory block size: 128MB867lb # [ 0.294766] devtmpfs: initialized868lb # [ 0.295541] x86/mm: Memory block size: 128MB869worker2 # [ 0.291992] smp: Bringing up secondary CPUs ...870client # [ 0.301187] NET: Registered PF_NETLINK/PF_ROUTE protocol family871worker1 # [ 0.294663] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)872worker2 # [ 0.292405] smp: Brought up 1 node, 1 CPU873worker2 # [ 0.293058] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)874worker1 # [ 0.295781] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).875client # [ 0.301962] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations876lb # [ 0.297346] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)877worker1 # [ 0.296995] pinctrl core: initialized pinctrl subsystem878client # [ 0.302463] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations879lb # [ 0.298374] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).880worker2 # [ 0.293558] Memory: 1454572K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)881worker1 # [ 0.298069] PM: RTC time: 13:53:36, date: 2026-09-21882lb # [ 0.299597] pinctrl core: initialized pinctrl subsystem883worker2 # [ 0.294680] devtmpfs: initialized884client # [ 0.303722] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations885lb # [ 0.300765] PM: RTC time: 13:53:36, date: 2026-09-21886worker2 # [ 0.295499] x86/mm: Memory block size: 128MB887client # [ 0.304450] audit: initializing netlink subsys (disabled)888client # [ 0.305442] audit: type=2000 audit(1789998817.363:1): state=initialized audit_enabled=0 res=1889worker2 # [ 0.297201] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)890worker1 # [ 0.301813] NET: Registered PF_NETLINK/PF_ROUTE protocol family891client # [ 0.306572] thermal_sys: Registered thermal governor 'fair_share'892worker2 # [ 0.298305] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).893client # [ 0.306574] thermal_sys: Registered thermal governor 'bang_bang'894worker1 # [ 0.303147] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations895lb # [ 0.304749] NET: Registered PF_NETLINK/PF_ROUTE protocol family896worker2 # [ 0.299478] pinctrl core: initialized pinctrl subsystem897client # [ 0.307441] thermal_sys: Registered thermal governor 'step_wise'898worker1 # [ 0.303928] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations899client # [ 0.308343] thermal_sys: Registered thermal governor 'user_space'900worker2 # [ 0.300620] PM: RTC time: 13:53:36, date: 2026-09-21901lb # [ 0.306022] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations902client # [ 0.309129] thermal_sys: Registered thermal governor 'power_allocator'903worker1 # [ 0.305181] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations904lb # [ 0.306529] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations905client # [ 0.309462] cpuidle: using governor menu906worker1 # [ 0.305911] audit: initializing netlink subsys (disabled)907lb # [ 0.307796] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations908client # [ 0.311658] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5909worker1 # [ 0.307029] thermal_sys: Registered thermal governor 'fair_share'910lb # [ 0.308514] audit: initializing netlink subsys (disabled)911worker2 # [ 0.304290] NET: Registered PF_NETLINK/PF_ROUTE protocol family912worker1 # [ 0.307031] thermal_sys: Registered thermal governor 'bang_bang'913client # [ 0.312720] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]914worker1 # [ 0.307902] thermal_sys: Registered thermal governor 'step_wise'915lb # [ 0.309506] audit: type=2000 audit(1789998817.373:1): state=initialized audit_enabled=0 res=1916worker2 # [ 0.305658] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations917client # [ 0.313443] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry918worker1 # [ 0.308834] thermal_sys: Registered thermal governor 'user_space'919lb # [ 0.310657] thermal_sys: Registered thermal governor 'fair_share'920client # [ 0.314452] PCI: Using configuration type 1 for base access921worker2 # [ 0.306420] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations922worker1 # [ 0.309602] audit: type=2000 audit(1789998817.373:1): state=initialized audit_enabled=0 res=1923lb # [ 0.310660] thermal_sys: Registered thermal governor 'bang_bang'924lb # [ 0.311505] thermal_sys: Registered thermal governor 'step_wise'925worker2 # [ 0.307683] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations926client # [ 0.315559] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.927worker1 # [ 0.310910] thermal_sys: Registered thermal governor 'power_allocator'928worker1 # [ 0.310937] cpuidle: using governor menu929worker2 # [ 0.308402] audit: initializing netlink subsys (disabled)930lb # [ 0.312439] thermal_sys: Registered thermal governor 'user_space'931lb # [ 0.313228] thermal_sys: Registered thermal governor 'power_allocator'932worker2 # [ 0.309620] thermal_sys: Registered thermal governor 'fair_share'933worker1 # [ 0.313109] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5934lb # [ 0.313518] cpuidle: using governor menu935worker2 # [ 0.309623] thermal_sys: Registered thermal governor 'bang_bang'936worker1 # [ 0.314139] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]937worker2 # [ 0.310393] thermal_sys: Registered thermal governor 'step_wise'938lb # [ 0.315744] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5939worker1 # [ 0.314905] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry940worker2 # [ 0.311323] thermal_sys: Registered thermal governor 'user_space'941worker1 # [ 0.315912] PCI: Using configuration type 1 for base access942lb # [ 0.316748] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]943worker2 # [ 0.312094] audit: type=2000 audit(1789998817.380:1): state=initialized audit_enabled=0 res=1944lb # [ 0.317507] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry945client # [ 0.322458] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages946worker2 # [ 0.313394] thermal_sys: Registered thermal governor 'power_allocator'947worker1 # [ 0.317004] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.948lb # [ 0.318516] PCI: Using configuration type 1 for base access949worker2 # [ 0.313408] cpuidle: using governor menu950client # [ 0.323441] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page951lb # [ 0.319575] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.952worker2 # [ 0.315605] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5953worker2 # [ 0.316635] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]954worker2 # [ 0.317395] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry955worker1 # [ 0.322180] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages956client # [ 0.328442] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages957worker2 # [ 0.318404] PCI: Using configuration type 1 for base access958worker1 # [ 0.322903] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page959client # [ 0.329441] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page960worker2 # [ 0.319488] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.961lb # [ 0.326521] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages962lb # [ 0.327505] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page963worker1 # [ 0.327903] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages964worker1 # [ 0.328901] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page965worker2 # [ 0.326687] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages966lb # [ 0.332506] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages967worker2 # [ 0.327394] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page968lb # [ 0.333505] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page969client # [ 0.340708] ACPI: Added _OSI(Module Device)970client # [ 0.341442] ACPI: Added _OSI(Processor Device)971client # [ 0.342223] ACPI: Added _OSI(Processor Aggregator Device)972worker2 # [ 0.332393] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages973worker2 # [ 0.333392] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page974worker1 # [ 0.339230] ACPI: Added _OSI(Module Device)975worker1 # [ 0.339904] ACPI: Added _OSI(Processor Device)976worker1 # [ 0.340647] ACPI: Added _OSI(Processor Aggregator Device)977client # [ 0.348075] ACPI: 1 ACPI AML tables successfully acquired and loaded978lb # [ 0.344813] ACPI: Added _OSI(Module Device)979lb # [ 0.345506] ACPI: Added _OSI(Processor Device)980lb # [ 0.346285] ACPI: Added _OSI(Processor Aggregator Device)981client # [ 0.351876] ACPI: Interpreter enabled982worker2 # [ 0.342011] ACPI: Added _OSI(Module Device)983client # [ 0.352456] ACPI: PM: (supports S0 S3 S4 S5)984worker2 # [ 0.342393] ACPI: Added _OSI(Processor Device)985client # [ 0.353150] ACPI: Using IOAPIC for interrupt routing986worker2 # [ 0.343138] ACPI: Added _OSI(Processor Aggregator Device)987worker1 # [ 0.349123] ACPI: 1 ACPI AML tables successfully acquired and loaded988client # [ 0.355514] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug989lb # [ 0.352282] ACPI: 1 ACPI AML tables successfully acquired and loaded990client # [ 0.356441] PCI: Using E820 reservations for host bridge windows991worker1 # [ 0.353307] ACPI: Interpreter enabled992worker1 # [ 0.353920] ACPI: PM: (supports S0 S3 S4 S5)993worker1 # [ 0.354629] ACPI: Using IOAPIC for interrupt routing994client # [ 0.359572] ACPI: Enabled 2 GPEs in block 00 to 3F995worker2 # [ 0.349953] ACPI: 1 ACPI AML tables successfully acquired and loaded996lb # [ 0.356264] ACPI: Interpreter enabled997lb # [ 0.356520] ACPI: PM: (supports S0 S3 S4 S5)998worker1 # [ 0.354955] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug999lb # [ 0.357263] ACPI: Using IOAPIC for interrupt routing1000worker1 # [ 0.355903] PCI: Using E820 reservations for host bridge windows1001lb # [ 0.357577] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1002worker2 # [ 0.353363] ACPI: Interpreter enabled1003worker2 # [ 0.353887] ACPI: PM: (supports S0 S3 S4 S5)1004worker1 # [ 0.359042] ACPI: Enabled 2 GPEs in block 00 to 3F1005worker2 # [ 0.354393] ACPI: Using IOAPIC for interrupt routing1006worker2 # [ 0.355245] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1007client # [ 0.366985] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1008lb # [ 0.362505] PCI: Using E820 reservations for host bridge windows1009client # [ 0.367447] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1010lb # [ 0.363626] ACPI: Enabled 2 GPEs in block 00 to 3F1011client # [ 0.368528] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1012client # [ 0.369562] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1013worker2 # [ 0.360392] PCI: Using E820 reservations for host bridge windows1014client # [ 0.370803] PCI host bridge to bus 0000:001015worker2 # [ 0.361491] ACPI: Enabled 2 GPEs in block 00 to 3F1016client # [ 0.371445] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1017worker1 # [ 0.367630] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1018client # [ 0.372441] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1019worker1 # [ 0.367908] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1020client # [ 0.373441] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1021lb # [ 0.371468] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1022client # [ 0.374441] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1023worker1 # [ 0.369710] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1024client # [ 0.375441] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1025worker1 # [ 0.370881] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1026lb # [ 0.372251] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1027worker1 # [ 0.372095] PCI host bridge to bus 0000:001028client # [ 0.376440] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1029lb # [ 0.373596] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1030client # [ 0.377441] pci_bus 0000:00: root bus resource [bus 00-ff]1031worker1 # [ 0.372761] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1032lb # [ 0.374636] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1033worker2 # [ 0.370139] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1034worker1 # [ 0.373719] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1035client # [ 0.378408] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1036lb # [ 0.375881] PCI host bridge to bus 0000:001037worker1 # [ 0.374727] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1038lb # [ 0.376509] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1039worker2 # [ 0.371169] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1040client # [ 0.379886] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1041worker1 # [ 0.375802] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1042lb # [ 0.377505] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1043worker2 # [ 0.372470] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1044worker1 # [ 0.376781] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1045lb # [ 0.378505] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1046worker2 # [ 0.373522] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1047worker1 # [ 0.377819] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1048lb # [ 0.379504] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1049worker2 # [ 0.374782] PCI host bridge to bus 0000:001050worker1 # [ 0.378864] pci_bus 0000:00: root bus resource [bus 00-ff]1051lb # [ 0.380504] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1052worker2 # [ 0.375396] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1053worker1 # [ 0.379754] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1054lb # [ 0.381504] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1055worker2 # [ 0.376392] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1056lb # [ 0.382505] pci_bus 0000:00: root bus resource [bus 00-ff]1057worker2 # [ 0.377391] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1058worker1 # [ 0.381337] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1059lb # [ 0.383482] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1060worker2 # [ 0.378392] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1061client # [ 0.382530] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1062worker2 # [ 0.379391] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1063lb # [ 0.384952] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1064client # [ 0.383453] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1065worker2 # [ 0.380393] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1066client # [ 0.384397] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1067worker2 # [ 0.381392] pci_bus 0000:00: root bus resource [bus 00-ff]1068client # [ 0.385687] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1069worker2 # [ 0.382403] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1070client # [ 0.387155] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1071worker2 # [ 0.383840] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1072worker1 # [ 0.383991] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1073worker1 # [ 0.384914] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1074client # [ 0.390476] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1075lb # [ 0.387590] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1076worker1 # [ 0.385912] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1077client # [ 0.391322] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1078lb # [ 0.388587] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1079client # [ 0.392147] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1080worker1 # [ 0.387133] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1081client # [ 0.392446] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1082lb # [ 0.389526] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1083worker1 # [ 0.388608] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1084lb # [ 0.390883] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1085client # [ 0.394728] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1086worker2 # [ 0.386476] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1087lb # [ 0.392252] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1088worker2 # [ 0.387404] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1089worker1 # [ 0.391936] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1090worker2 # [ 0.388371] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1091worker1 # [ 0.392821] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1092client # [ 0.397474] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1093client # [ 0.398363] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1094worker1 # [ 0.393652] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1095lb # [ 0.395531] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1096worker2 # [ 0.389753] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1097client # [ 0.399175] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1098lb # [ 0.396369] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1099worker1 # [ 0.393908] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1100lb # [ 0.397267] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1101worker2 # [ 0.391105] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1102client # [ 0.400016] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1103worker1 # [ 0.395461] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1104lb # [ 0.398350] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1105worker2 # [ 0.394424] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1106lb # [ 0.400067] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1107worker1 # [ 0.397942] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1108worker2 # [ 0.395302] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1109client # [ 0.403476] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1110worker1 # [ 0.398821] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1111worker2 # [ 0.396147] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1112client # [ 0.404293] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1113worker1 # [ 0.399682] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1114lb # [ 0.402512] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1115client # [ 0.405185] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1116worker2 # [ 0.396398] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1117lb # [ 0.403331] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1118worker1 # [ 0.401277] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1119lb # [ 0.404248] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1120client # [ 0.406021] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1121worker2 # [ 0.397974] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1122lb # [ 0.405068] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1123client # [ 0.407474] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1124worker2 # [ 0.400366] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1125worker1 # [ 0.404859] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1126client # [ 0.408356] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1127worker2 # [ 0.401062] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1128worker1 # [ 0.405542] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1129worker2 # [ 0.401414] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1130worker1 # [ 0.405924] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1131client # [ 0.409224] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1132lb # [ 0.408534] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1133lb # [ 0.409386] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1134worker2 # [ 0.403022] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1135worker1 # [ 0.407452] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1136client # [ 0.411000] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1137lb # [ 0.410262] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1138lb # [ 0.411075] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1139client # [ 0.413335] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1140worker1 # [ 0.409934] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1141worker2 # [ 0.405424] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1142client # [ 0.414116] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1143worker1 # [ 0.410788] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1144worker2 # [ 0.406274] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1145client # [ 0.414462] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1146lb # [ 0.413471] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1147worker2 # [ 0.407146] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1148worker1 # [ 0.411644] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1149lb # [ 0.414125] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1150client # [ 0.416003] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1151worker2 # [ 0.407944] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1152worker1 # [ 0.412446] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1153lb # [ 0.414526] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1154worker2 # [ 0.409419] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1155lb # [ 0.416074] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1156worker1 # [ 0.414638] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1157worker2 # [ 0.410262] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1158client # [ 0.419476] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1159worker1 # [ 0.414909] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1160client # [ 0.420338] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1161worker1 # [ 0.415857] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1162lb # [ 0.418290] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1163worker2 # [ 0.411124] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1164client # [ 0.421207] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1165lb # [ 0.419144] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1166worker1 # [ 0.417364] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1167worker2 # [ 0.411958] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1168lb # [ 0.419526] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1169client # [ 0.422067] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1170lb # [ 0.421112] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1171worker2 # [ 0.413423] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1172client # [ 0.423478] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1173worker2 # [ 0.414299] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1174worker1 # [ 0.419932] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1175client # [ 0.424344] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1176worker2 # [ 0.415147] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1177worker1 # [ 0.420783] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1178worker1 # [ 0.421653] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1179client # [ 0.425186] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1180worker2 # [ 0.415955] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1181lb # [ 0.424528] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1182client # [ 0.425448] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1183lb # [ 0.425343] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1184worker1 # [ 0.422555] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1185lb # [ 0.426219] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1186client # [ 0.426961] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1187worker1 # [ 0.423935] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1188lb # [ 0.427086] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1189worker2 # [ 0.418401] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1190worker1 # [ 0.424778] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1191client # [ 0.428453] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1192worker2 # [ 0.420040] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1193client # [ 0.429427] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1194worker2 # [ 0.420414] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1195lb # [ 0.428538] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1196worker1 # [ 0.425656] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1197lb # [ 0.429365] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1198worker1 # [ 0.425909] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1199client # [ 0.430849] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1200worker2 # [ 0.422003] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1201lb # [ 0.430244] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1202worker1 # [ 0.427417] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1203lb # [ 0.430512] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1204worker2 # [ 0.423400] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1205worker2 # [ 0.424230] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1206client # [ 0.433448] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1207worker1 # [ 0.428914] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1208lb # [ 0.432071] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1209worker2 # [ 0.425153] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1210client # [ 0.435076] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1211worker1 # [ 0.429850] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1212client # [ 0.435462] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1213worker2 # [ 0.426177] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1214lb # [ 0.433524] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1215worker1 # [ 0.431286] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1216lb # [ 0.434482] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1217client # [ 0.436997] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1218worker2 # [ 0.427982] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1219lb # [ 0.435913] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1220worker2 # [ 0.429404] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1221client # [ 0.439230] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1222worker1 # [ 0.433712] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1223client # [ 0.440082] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1224worker2 # [ 0.430414] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1225worker1 # [ 0.434571] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1226client # [ 0.440462] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1227worker1 # [ 0.434924] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1228worker2 # [ 0.431957] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1229lb # [ 0.437992] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1230lb # [ 0.439151] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1231worker1 # [ 0.436448] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1232client # [ 0.441998] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1233lb # [ 0.439527] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1234worker2 # [ 0.434111] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1235client # [ 0.444319] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1236worker1 # [ 0.438617] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1237lb # [ 0.441056] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1238worker2 # [ 0.434399] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1239worker1 # [ 0.438908] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1240client # [ 0.445300] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1241worker2 # [ 0.435414] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1242worker1 # [ 0.439863] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1243client # [ 0.447156] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1244lb # [ 0.443160] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1245worker2 # [ 0.436965] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1246lb # [ 0.443512] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1247lb # [ 0.444483] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1248worker1 # [ 0.441405] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1249worker2 # [ 0.438423] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1250worker1 # [ 0.443716] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1251worker2 # [ 0.439336] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1252lb # [ 0.445946] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1253worker2 # [ 0.440160] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1254worker1 # [ 0.444754] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1255lb # [ 0.448314] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1256worker1 # [ 0.446646] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1257worker2 # [ 0.441816] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1258lb # [ 0.449372] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1259worker2 # [ 0.443364] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1260lb # [ 0.451241] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1261worker2 # [ 0.444256] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1262worker2 # [ 0.446026] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1263client # [ 0.447637] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1264client # [ 0.450216] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1265client # [ 0.450860] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1266client # [ 0.452158] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1267client # [ 0.452711] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1268worker1 # [ 0.447093] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1269client # [ 0.453723] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1270worker1 # [ 0.449694] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1271lb # [ 0.451705] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1272client # [ 0.454703] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1273worker1 # [ 0.451047] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1274lb # [ 0.454274] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1275worker1 # [ 0.452585] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1276worker2 # [ 0.446589] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1277client # [ 0.456468] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1278lb # [ 0.454939] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1279client # [ 0.457447] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1280worker1 # [ 0.453173] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1281worker2 # [ 0.449074] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1282lb # [ 0.456199] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1283client # [ 0.458703] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1284worker1 # [ 0.454189] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1285worker2 # [ 0.449618] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1286lb # [ 0.456737] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1287worker1 # [ 0.455175] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1288client # [ 0.460245] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1289worker2 # [ 0.451019] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1290lb # [ 0.457792] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1291worker2 # [ 0.451662] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1292worker1 # [ 0.456930] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1293lb # [ 0.458776] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1294worker2 # [ 0.452700] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1295worker1 # [ 0.457793] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1296client # [ 0.464064] ACPI: PCI: Interrupt link LNKA configured for IRQ 101297lb # [ 0.460533] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1298client # [ 0.464547] ACPI: PCI: Interrupt link LNKB configured for IRQ 101299worker2 # [ 0.453859] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1300worker1 # [ 0.458961] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1301lb # [ 0.461380] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1302client # [ 0.465538] ACPI: PCI: Interrupt link LNKC configured for IRQ 111303worker1 # [ 0.460726] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1304worker2 # [ 0.455420] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1305client # [ 0.466571] ACPI: PCI: Interrupt link LNKD configured for IRQ 111306lb # [ 0.462630] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1307worker2 # [ 0.456303] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1308client # [ 0.467552] ACPI: PCI: Interrupt link LNKE configured for IRQ 101309client # [ 0.468536] ACPI: PCI: Interrupt link LNKF configured for IRQ 101310lb # [ 0.464300] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1311worker2 # [ 0.457442] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1312client # [ 0.469547] ACPI: PCI: Interrupt link LNKG configured for IRQ 111313worker1 # [ 0.464552] ACPI: PCI: Interrupt link LNKA configured for IRQ 101314worker2 # [ 0.459073] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1315client # [ 0.470540] ACPI: PCI: Interrupt link LNKH configured for IRQ 111316worker1 # [ 0.465014] ACPI: PCI: Interrupt link LNKB configured for IRQ 101317client # [ 0.471477] ACPI: PCI: Interrupt link GSIA configured for IRQ 161318worker1 # [ 0.466001] ACPI: PCI: Interrupt link LNKC configured for IRQ 111319lb # [ 0.468074] ACPI: PCI: Interrupt link LNKA configured for IRQ 101320client # [ 0.472453] ACPI: PCI: Interrupt link GSIB configured for IRQ 171321worker1 # [ 0.467001] ACPI: PCI: Interrupt link LNKD configured for IRQ 111322lb # [ 0.468609] ACPI: PCI: Interrupt link LNKB configured for IRQ 101323client # [ 0.473395] ACPI: PCI: Interrupt link GSIC configured for IRQ 181324worker1 # [ 0.467999] ACPI: PCI: Interrupt link LNKE configured for IRQ 101325client # [ 0.474173] ACPI: PCI: Interrupt link GSID configured for IRQ 191326lb # [ 0.469612] ACPI: PCI: Interrupt link LNKC configured for IRQ 111327worker2 # [ 0.462407] ACPI: PCI: Interrupt link LNKA configured for IRQ 101328worker1 # [ 0.469018] ACPI: PCI: Interrupt link LNKF configured for IRQ 101329worker2 # [ 0.463415] ACPI: PCI: Interrupt link LNKB configured for IRQ 101330client # [ 0.474452] ACPI: PCI: Interrupt link GSIE configured for IRQ 201331lb # [ 0.470910] ACPI: PCI: Interrupt link LNKD configured for IRQ 111332worker1 # [ 0.470016] ACPI: PCI: Interrupt link LNKG configured for IRQ 111333client # [ 0.475454] ACPI: PCI: Interrupt link GSIF configured for IRQ 211334worker2 # [ 0.464409] ACPI: PCI: Interrupt link LNKC configured for IRQ 111335lb # [ 0.471603] ACPI: PCI: Interrupt link LNKE configured for IRQ 101336worker1 # [ 0.471032] ACPI: PCI: Interrupt link LNKH configured for IRQ 111337client # [ 0.476409] ACPI: PCI: Interrupt link GSIG configured for IRQ 221338worker2 # [ 0.465419] ACPI: PCI: Interrupt link LNKD configured for IRQ 111339lb # [ 0.472599] ACPI: PCI: Interrupt link LNKF configured for IRQ 101340worker1 # [ 0.471938] ACPI: PCI: Interrupt link GSIA configured for IRQ 161341client # [ 0.477163] ACPI: PCI: Interrupt link GSIH configured for IRQ 231342worker2 # [ 0.466413] ACPI: PCI: Interrupt link LNKE configured for IRQ 101343lb # [ 0.473604] ACPI: PCI: Interrupt link LNKG configured for IRQ 111344worker1 # [ 0.472850] ACPI: PCI: Interrupt link GSIB configured for IRQ 171345lb # [ 0.474601] ACPI: PCI: Interrupt link LNKH configured for IRQ 111346worker2 # [ 0.467489] ACPI: PCI: Interrupt link LNKF configured for IRQ 101347client # [ 0.478533] iommu: Default domain type: Translated1348worker1 # [ 0.473619] ACPI: PCI: Interrupt link GSIC configured for IRQ 181349lb # [ 0.475541] ACPI: PCI: Interrupt link GSIA configured for IRQ 161350worker2 # [ 0.468493] ACPI: PCI: Interrupt link LNKG configured for IRQ 111351worker1 # [ 0.473916] ACPI: PCI: Interrupt link GSID configured for IRQ 191352client # [ 0.479296] iommu: DMA domain TLB invalidation policy: lazy mode1353lb # [ 0.476445] ACPI: PCI: Interrupt link GSIB configured for IRQ 171354worker2 # [ 0.469493] ACPI: PCI: Interrupt link LNKH configured for IRQ 111355worker1 # [ 0.474824] ACPI: PCI: Interrupt link GSIE configured for IRQ 201356client # [ 0.480388] ACPI: bus type USB registered1357lb # [ 0.477231] ACPI: PCI: Interrupt link GSIC configured for IRQ 181358worker2 # [ 0.470429] ACPI: PCI: Interrupt link GSIA configured for IRQ 161359client # [ 0.480954] usbcore: registered new interface driver usbfs1360worker1 # [ 0.475608] ACPI: PCI: Interrupt link GSIF configured for IRQ 211361lb # [ 0.477519] ACPI: PCI: Interrupt link GSID configured for IRQ 191362client # [ 0.481454] usbcore: registered new interface driver hub1363worker2 # [ 0.471364] ACPI: PCI: Interrupt link GSIB configured for IRQ 171364worker1 # [ 0.476589] ACPI: PCI: Interrupt link GSIG configured for IRQ 221365lb # [ 0.478454] ACPI: PCI: Interrupt link GSIE configured for IRQ 201366client # [ 0.482308] usbcore: registered new device driver usb1367worker2 # [ 0.472116] ACPI: PCI: Interrupt link GSIC configured for IRQ 181368worker1 # [ 0.476917] ACPI: PCI: Interrupt link GSIH configured for IRQ 231369lb # [ 0.479207] ACPI: PCI: Interrupt link GSIF configured for IRQ 211370worker2 # [ 0.472407] ACPI: PCI: Interrupt link GSID configured for IRQ 191371client # [ 0.483955] NetLabel: Initializing1372lb # [ 0.479517] ACPI: PCI: Interrupt link GSIG configured for IRQ 221373worker2 # [ 0.473353] ACPI: PCI: Interrupt link GSIE configured for IRQ 201374worker1 # [ 0.478955] iommu: Default domain type: Translated1375client # [ 0.484441] NetLabel: domain hash size = 1281376lb # [ 0.480520] ACPI: PCI: Interrupt link GSIH configured for IRQ 231377worker2 # [ 0.474085] ACPI: PCI: Interrupt link GSIF configured for IRQ 211378worker1 # [ 0.479745] iommu: DMA domain TLB invalidation policy: lazy mode1379client # [ 0.485162] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1380worker2 # [ 0.474403] ACPI: PCI: Interrupt link GSIG configured for IRQ 221381worker1 # [ 0.480901] ACPI: bus type USB registered1382client # [ 0.485518] NetLabel: unlabeled traffic allowed by default1383lb # [ 0.482529] iommu: Default domain type: Translated1384worker2 # [ 0.475407] ACPI: PCI: Interrupt link GSIH configured for IRQ 231385client # [ 0.486453] PCI: Using ACPI for IRQ routing1386worker1 # [ 0.481647] usbcore: registered new interface driver usbfs1387lb # [ 0.483304] iommu: DMA domain TLB invalidation policy: lazy mode1388worker1 # [ 0.481932] usbcore: registered new interface driver hub1389lb # [ 0.484543] ACPI: bus type USB registered1390worker2 # [ 0.477455] iommu: Default domain type: Translated1391worker1 # [ 0.482826] usbcore: registered new device driver usb1392lb # [ 0.485275] usbcore: registered new interface driver usbfs1393worker2 # [ 0.478221] iommu: DMA domain TLB invalidation policy: lazy mode1394lb # [ 0.485523] usbcore: registered new interface driver hub1395worker2 # [ 0.479396] ACPI: bus type USB registered1396worker1 # [ 0.485146] NetLabel: Initializing1397lb # [ 0.486422] usbcore: registered new device driver usb1398worker2 # [ 0.480143] usbcore: registered new interface driver usbfs1399worker1 # [ 0.485804] NetLabel: domain hash size = 1281400worker2 # [ 0.480404] usbcore: registered new interface driver hub1401worker1 # [ 0.486486] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1402lb # [ 0.488088] NetLabel: Initializing1403worker2 # [ 0.481292] usbcore: registered new device driver usb1404worker1 # [ 0.486941] NetLabel: unlabeled traffic allowed by default1405lb # [ 0.488506] NetLabel: domain hash size = 1281406worker1 # [ 0.487862] PCI: Using ACPI for IRQ routing1407lb # [ 0.489281] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1408worker2 # [ 0.482962] NetLabel: Initializing1409worker2 # [ 0.483393] NetLabel: domain hash size = 1281410lb # [ 0.489550] NetLabel: unlabeled traffic allowed by default1411lb # [ 0.490516] PCI: Using ACPI for IRQ routing1412worker2 # [ 0.484130] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413worker2 # [ 0.485263] NetLabel: unlabeled traffic allowed by default1414worker2 # [ 0.486409] PCI: Using ACPI for IRQ routing1415client # [ 0.531326] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416client # [ 0.531437] pci 0000:00:01.0: vgaarb: bridge control possible1417client # [ 0.531437] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418client # [ 0.531445] vgaarb: loaded1419client # [ 0.532186] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01420client # [ 0.532441] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1421worker1 # [ 0.531823] pci 0000:00:01.0: vgaarb: setting as boot VGA device1422worker1 # [ 0.531898] pci 0000:00:01.0: vgaarb: bridge control possible1423client # [ 0.535526] clocksource: Switched to clocksource kvm-clock1424lb # [ 0.534402] pci 0000:00:01.0: vgaarb: setting as boot VGA device1425worker1 # [ 0.531898] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1426lb # [ 0.534500] pci 0000:00:01.0: vgaarb: bridge control possible1427worker1 # [ 0.531907] vgaarb: loaded1428worker1 # [ 0.532643] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01429lb # [ 0.534500] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1430lb # [ 0.534510] vgaarb: loaded1431worker1 # [ 0.532902] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1432worker2 # [ 0.531142] pci 0000:00:01.0: vgaarb: setting as boot VGA device1433client # [ 0.539324] VFS: Disk quotas dquot_6.6.01434lb # [ 0.535259] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01435worker2 # [ 0.531388] pci 0000:00:01.0: vgaarb: bridge control possible1436lb # [ 0.535505] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1437client # [ 0.540021] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1438client # [ 0.541326] pnp: PnP ACPI init1439worker2 # [ 0.531388] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1440worker2 # [ 0.531396] vgaarb: loaded1441client # [ 0.542131] ACPI: IRQ 4 override to edge(!), high(!)1442worker1 # [ 0.535995] clocksource: Switched to clocksource kvm-clock1443worker2 # [ 0.532165] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01444client # [ 0.543152] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1445worker2 # [ 0.532393] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1446lb # [ 0.538596] clocksource: Switched to clocksource kvm-clock1447client # [ 0.544706] pnp: PnP ACPI: found 6 devices1448worker1 # [ 0.539417] VFS: Disk quotas dquot_6.6.01449worker1 # [ 0.540148] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1450lb # [ 0.542179] VFS: Disk quotas dquot_6.6.01451worker1 # [ 0.541565] pnp: PnP ACPI init1452worker2 # [ 0.536526] clocksource: Switched to clocksource kvm-clock1453lb # [ 0.542905] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1454worker1 # [ 0.542409] ACPI: IRQ 4 override to edge(!), high(!)1455lb # [ 0.544242] pnp: PnP ACPI init1456worker1 # [ 0.543406] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1457lb # [ 0.545043] ACPI: IRQ 4 override to edge(!), high(!)1458worker1 # [ 0.544981] pnp: PnP ACPI: found 6 devices1459lb # [ 0.546046] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1460worker2 # [ 0.540047] VFS: Disk quotas dquot_6.6.01461client # [ 0.552332] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1462lb # [ 0.547591] pnp: PnP ACPI: found 6 devices1463worker2 # [ 0.540767] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1464client # [ 0.553786] clocksource: Switched to clocksource acpi_pm1465worker2 # [ 0.542127] pnp: PnP ACPI init1466client # [ 0.554763] NET: Registered PF_INET protocol family1467worker2 # [ 0.542971] ACPI: IRQ 4 override to edge(!), high(!)1468client # [ 0.555989] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1469worker2 # [ 0.543973] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1470worker2 # [ 0.545524] pnp: PnP ACPI: found 6 devices1471worker1 # [ 0.552596] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1472worker1 # [ 0.554123] clocksource: Switched to clocksource acpi_pm1473worker1 # [ 0.555146] NET: Registered PF_INET protocol family1474lb # [ 0.555518] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1475lb # [ 0.556974] clocksource: Switched to clocksource acpi_pm1476worker1 # [ 0.556371] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1477lb # [ 0.557956] NET: Registered PF_INET protocol family1478lb # [ 0.559184] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1479worker2 # [ 0.553261] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1480worker2 # [ 0.554748] clocksource: Switched to clocksource acpi_pm1481worker2 # [ 0.555770] NET: Registered PF_INET protocol family1482worker2 # [ 0.557031] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1483client # [ 0.572642] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1484client # [ 0.574104] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1485client # [ 0.575490] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1486client # [ 0.576897] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1487client # [ 0.578204] TCP: Hash tables configured (established 16384 bind 16384)1488client # [ 0.579358] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1489worker1 # [ 0.572781] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1490client # [ 0.580676] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1491worker1 # [ 0.574363] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1492client # [ 0.581789] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1493lb # [ 0.575906] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1494worker1 # [ 0.575679] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1495client # [ 0.582974] NET: Registered PF_UNIX/PF_LOCAL protocol family1496lb # [ 0.577408] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1497client # [ 0.583974] NET: Registered PF_XDP protocol family1498worker1 # [ 0.577019] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1499lb # [ 0.578685] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1500client # [ 0.584823] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1501worker1 # [ 0.578261] TCP: Hash tables configured (established 16384 bind 16384)1502client # [ 0.585824] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1503lb # [ 0.580021] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1504worker1 # [ 0.579393] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1505worker2 # [ 0.573816] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1506client # [ 0.586818] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1507lb # [ 0.581248] TCP: Hash tables configured (established 16384 bind 16384)1508worker1 # [ 0.580603] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1509worker2 # [ 0.575342] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1510client # [ 0.587888] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1511lb # [ 0.582306] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1512worker1 # [ 0.581652] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1513client # [ 0.588967] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1514worker2 # [ 0.576622] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1515lb # [ 0.583559] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1516worker1 # [ 0.582778] NET: Registered PF_UNIX/PF_LOCAL protocol family1517worker1 # [ 0.583694] NET: Registered PF_XDP protocol family1518client # [ 0.590015] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1519worker2 # [ 0.577885] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1520lb # [ 0.584589] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1521worker1 # [ 0.584478] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1522lb # [ 0.585732] NET: Registered PF_UNIX/PF_LOCAL protocol family1523worker2 # [ 0.579103] TCP: Hash tables configured (established 16384 bind 16384)1524client # [ 0.591682] ACPI: \_SB_.GSIA: Enabled at IRQ 161525worker1 # [ 0.585458] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1526lb # [ 0.586656] NET: Registered PF_XDP protocol family1527worker2 # [ 0.580189] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1528lb # [ 0.587487] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1529worker1 # [ 0.586449] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1530worker2 # [ 0.581450] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1531lb # [ 0.588446] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1532worker1 # [ 0.587535] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1533client # [ 0.594727] ACPI: \_SB_.GSIB: Enabled at IRQ 171534worker2 # [ 0.582490] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1535lb # [ 0.589417] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1536worker1 # [ 0.588612] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1537worker2 # [ 0.583644] NET: Registered PF_UNIX/PF_LOCAL protocol family1538lb # [ 0.590478] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1539client # [ 0.596655] ACPI: \_SB_.GSIC: Enabled at IRQ 181540worker1 # [ 0.589684] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1541worker2 # [ 0.584570] NET: Registered PF_XDP protocol family1542lb # [ 0.591543] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1543worker2 # [ 0.585359] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1544worker1 # [ 0.591377] ACPI: \_SB_.GSIA: Enabled at IRQ 161545client # [ 0.598535] ACPI: \_SB_.GSID: Enabled at IRQ 191546lb # [ 0.592632] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1547worker2 # [ 0.586395] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1548worker2 # [ 0.587377] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1549lb # [ 0.594284] ACPI: \_SB_.GSIA: Enabled at IRQ 161550client # [ 0.600159] PCI: CLS 0 bytes, default 641551worker1 # [ 0.593967] ACPI: \_SB_.GSIB: Enabled at IRQ 171552worker2 # [ 0.588458] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1553client # [ 0.601071] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1554worker2 # [ 0.589545] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1555lb # [ 0.596740] ACPI: \_SB_.GSIB: Enabled at IRQ 171556worker1 # [ 0.596064] ACPI: \_SB_.GSIC: Enabled at IRQ 181557client # [ 0.602905] Trying to unpack rootfs image as initramfs...1558worker2 # [ 0.590745] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1559lb # [ 0.598759] ACPI: \_SB_.GSIC: Enabled at IRQ 181560worker1 # [ 0.597981] ACPI: \_SB_.GSID: Enabled at IRQ 191561worker2 # [ 0.592422] ACPI: \_SB_.GSIA: Enabled at IRQ 161562worker1 # [ 0.599629] PCI: CLS 0 bytes, default 641563lb # [ 0.600700] ACPI: \_SB_.GSID: Enabled at IRQ 191564worker2 # [ 0.594744] ACPI: \_SB_.GSIB: Enabled at IRQ 171565lb # [ 0.602375] PCI: CLS 0 bytes, default 641566worker1 # [ 0.600507] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1567worker1 # [ 0.602235] Trying to unpack rootfs image as initramfs...1568lb # [ 0.603288] Trying to unpack rootfs image as initramfs...1569worker2 # [ 0.596903] ACPI: \_SB_.GSIC: Enabled at IRQ 181570worker2 # [ 0.599034] ACPI: \_SB_.GSID: Enabled at IRQ 191571lb # [ 0.605322] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1572worker2 # [ 0.600819] PCI: CLS 0 bytes, default 641573worker2 # [ 0.601794] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1574worker2 # [ 0.603635] Trying to unpack rootfs image as initramfs...1575client # [ 0.645663] Initialise system trusted keyrings1576client # [ 0.648577] workingset: timestamp_bits=40 max_order=19 bucket_order=01577lb # [ 0.646694] Initialise system trusted keyrings1578worker1 # [ 0.647450] Initialise system trusted keyrings1579lb # [ 0.649228] workingset: timestamp_bits=40 max_order=19 bucket_order=01580worker1 # [ 0.650384] workingset: timestamp_bits=40 max_order=19 bucket_order=01581worker2 # [ 0.647426] Initialise system trusted keyrings1582worker2 # [ 0.650991] workingset: timestamp_bits=40 max_order=19 bucket_order=01583client # [ 0.670611] Key type asymmetric registered1584client # [ 0.671342] Asymmetric key parser 'x509' registered1585client # [ 0.672263] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1586client # [ 0.677704] io scheduler mq-deadline registered1587lb # [ 0.672308] Key type asymmetric registered1588worker1 # [ 0.671849] Key type asymmetric registered1589lb # [ 0.673049] Asymmetric key parser 'x509' registered1590lb # [ 0.673938] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1591client # [ 0.680544] io scheduler kyber registered1592worker1 # [ 0.674373] Asymmetric key parser 'x509' registered1593client # [ 0.681839] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1594worker1 # [ 0.675255] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1595lb # [ 0.677296] io scheduler mq-deadline registered1596client # [ 0.683120] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1597lb # [ 0.678084] io scheduler kyber registered1598worker2 # [ 0.672215] Key type asymmetric registered1599worker1 # [ 0.679895] io scheduler mq-deadline registered1600worker1 # [ 0.680700] io scheduler kyber registered1601worker2 # [ 0.674975] Asymmetric key parser 'x509' registered1602client # [ 0.688284] Linux agpgart interface v0.1031603lb # [ 0.682360] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1604worker2 # [ 0.675856] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1605client # [ 0.689046] ACPI: bus type drm_connector registered1606lb # [ 0.683619] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1607worker1 # [ 0.683493] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1608worker2 # [ 0.679106] io scheduler mq-deadline registered1609worker1 # [ 0.684761] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1610worker2 # [ 0.679881] io scheduler kyber registered1611client # [ 0.691990] usbcore: registered new interface driver usbserial_generic1612client # [ 0.693065] usbserial: USB Serial support registered for generic1613lb # [ 0.689006] Linux agpgart interface v0.1031614worker2 # [ 0.682542] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1615lb # [ 0.689790] ACPI: bus type drm_connector registered1616worker2 # [ 0.683784] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1617worker1 # [ 0.690161] Linux agpgart interface v0.1031618client # [ 0.696538] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1619worker1 # [ 0.690931] ACPI: bus type drm_connector registered1620client # [ 0.696538] Please enable it if your BIOS has the CPPC option.1621client # [ 0.698766] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1622lb # [ 0.695297] usbcore: registered new interface driver usbserial_generic1623worker2 # [ 0.689352] Linux agpgart interface v0.1031624lb # [ 0.696443] usbserial: USB Serial support registered for generic1625worker2 # [ 0.690131] ACPI: bus type drm_connector registered1626client # [ 0.702714] drop_monitor: Initializing network drop monitor service1627worker1 # [ 0.696542] usbcore: registered new interface driver usbserial_generic1628lb # [ 0.697427] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1629client # [ 0.703886] NET: Registered PF_INET6 protocol family1630worker1 # [ 0.697589] usbserial: USB Serial support registered for generic1631lb # [ 0.697427] Please enable it if your BIOS has the CPPC option.1632lb # [ 0.699606] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1633worker2 # [ 0.693401] usbcore: registered new interface driver usbserial_generic1634worker1 # [ 0.698550] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1635worker1 # [ 0.698550] Please enable it if your BIOS has the CPPC option.1636worker2 # [ 0.694409] usbserial: USB Serial support registered for generic1637client # [ 0.706964] Segment Routing with IPv61638worker1 # [ 0.700659] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1639client # [ 0.709552] In-situ OAM (IOAM) with IPv61640lb # [ 0.703519] drop_monitor: Initializing network drop monitor service1641lb # [ 0.704713] NET: Registered PF_INET6 protocol family1642client # [ 0.710577] IPI shorthand broadcast: enabled1643worker2 # [ 0.697967] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1644worker2 # [ 0.697967] Please enable it if your BIOS has the CPPC option.1645worker1 # [ 0.705481] drop_monitor: Initializing network drop monitor service1646worker2 # [ 0.700207] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1647worker1 # [ 0.706645] NET: Registered PF_INET6 protocol family1648lb # [ 0.708634] Segment Routing with IPv61649lb # [ 0.709361] In-situ OAM (IOAM) with IPv61650worker1 # [ 0.709810] Segment Routing with IPv61651worker2 # [ 0.704073] drop_monitor: Initializing network drop monitor service1652worker2 # [ 0.705279] NET: Registered PF_INET6 protocol family1653lb # [ 0.712461] IPI shorthand broadcast: enabled1654worker1 # [ 0.712354] In-situ OAM (IOAM) with IPv61655client # [ 0.719036] sched_clock: Marking stable (586013958, 132500922)->(796526488, -78011608)1656worker1 # [ 0.713409] IPI shorthand broadcast: enabled1657worker2 # [ 0.708031] Segment Routing with IPv61658worker2 # [ 0.708711] In-situ OAM (IOAM) with IPv61659worker2 # [ 0.711221] IPI shorthand broadcast: enabled1660client # [ 0.724720] registered taskstats version 11661client # [ 0.725734] Loading compiled-in X.509 certificates1662lb # [ 0.720809] sched_clock: Marking stable (587016223, 133150924)->(797258413, -77091266)1663worker1 # [ 0.721468] sched_clock: Marking stable (588015699, 133310063)->(797213143, -75887381)1664lb # [ 0.724252] registered taskstats version 11665lb # [ 0.725255] Loading compiled-in X.509 certificates1666worker2 # [ 0.719502] sched_clock: Marking stable (585014474, 133919040)->(796562977, -77629463)1667worker1 # [ 0.726419] registered taskstats version 11668worker1 # [ 0.727398] Loading compiled-in X.509 certificates1669worker2 # [ 0.725151] registered taskstats version 11670worker2 # [ 0.726175] Loading compiled-in X.509 certificates1671client # [ 0.744531] Demotion targets for Node 0: null1672client # [ 0.745429] Key type .fscrypt registered1673client # [ 0.746131] Key type fscrypt-provisioning registered1674client # [ 0.747110] ima: No TPM chip found, activating TPM-bypass!1675lb # [ 0.744178] Demotion targets for Node 0: null1676worker1 # [ 0.743810] Demotion targets for Node 0: null1677lb # [ 0.745061] Key type .fscrypt registered1678client # [ 0.751530] ima: Allocated hash algorithm: sha11679lb # [ 0.745771] Key type fscrypt-provisioning registered1680client # [ 0.752296] ima: No architecture policies found1681lb # [ 0.746696] ima: No TPM chip found, activating TPM-bypass!1682worker1 # [ 0.746384] Key type .fscrypt registered1683client # [ 0.753252] PM: Magic number: 10:315:8881684worker1 # [ 0.747104] Key type fscrypt-provisioning registered1685worker1 # [ 0.748087] ima: No TPM chip found, activating TPM-bypass!1686lb # [ 0.751183] ima: Allocated hash algorithm: sha11687client # [ 0.756930] RAS: Correctable Errors collector initialized.1688worker2 # [ 0.744948] Demotion targets for Node 0: null1689lb # [ 0.751939] ima: No architecture policies found1690worker2 # [ 0.745860] Key type .fscrypt registered1691lb # [ 0.752933] PM: Magic number: 10:315:8881692worker2 # [ 0.746578] Key type fscrypt-provisioning registered1693worker1 # [ 0.752350] ima: Allocated hash algorithm: sha11694worker1 # [ 0.753125] ima: No architecture policies found1695worker1 # [ 0.755342] PM: Magic number: 10:315:8881696worker2 # [ 0.750054] ima: No TPM chip found, activating TPM-bypass!1697lb # [ 0.756652] RAS: Correctable Errors collector initialized.1698worker2 # [ 0.750995] ima: Allocated hash algorithm: sha11699worker1 # [ 0.756968] RAS: Correctable Errors collector initialized.1700worker2 # [ 0.751850] ima: No architecture policies found1701client # [ 0.765571] clk: Disabling unused clocks1702client # [ 0.766281] PM: genpd: Disabling unused power domains1703worker2 # [ 0.754104] PM: Magic number: 10:315:8881704worker2 # [ 0.755791] RAS: Correctable Errors collector initialized.1705lb # [ 0.765353] clk: Disabling unused clocks1706lb # [ 0.766058] PM: genpd: Disabling unused power domains1707worker1 # [ 0.767007] clk: Disabling unused clocks1708worker1 # [ 0.769338] PM: genpd: Disabling unused power domains1709worker2 # [ 0.766644] clk: Disabling unused clocks1710worker2 # [ 0.768948] PM: genpd: Disabling unused power domains1711client # [ 0.933466] Freeing initrd memory: 29096K1712client # [ 0.936670] Freeing unused decrypted memory: 2028K1713worker1 # [ 0.930555] Freeing initrd memory: 29092K1714client # [ 0.939316] Freeing unused kernel image (initmem) memory: 3644K1715worker1 # [ 0.933730] Freeing unused decrypted memory: 2028K1716client # [ 0.940432] Write protecting the kernel read-only data: 32768k1717client # [ 0.942304] Freeing unused kernel image (text/rodata gap) memory: 1208K1718worker1 # [ 0.936345] Freeing unused kernel image (initmem) memory: 3644K1719client # [ 0.943850] Freeing unused kernel image (rodata/data gap) memory: 752K1720worker1 # [ 0.937450] Write protecting the kernel read-only data: 32768k1721lb # [ 0.940595] Freeing initrd memory: 29092K1722worker1 # [ 0.939290] Freeing unused kernel image (text/rodata gap) memory: 1208K1723worker1 # [ 0.940825] Freeing unused kernel image (rodata/data gap) memory: 752K1724worker2 # [ 0.936685] Freeing initrd memory: 29092K1725lb # [ 0.943817] Freeing unused decrypted memory: 2028K1726worker2 # [ 0.939884] Freeing unused decrypted memory: 2028K1727lb # [ 0.946521] Freeing unused kernel image (initmem) memory: 3644K1728lb # [ 0.947598] Write protecting the kernel read-only data: 32768k1729worker2 # [ 0.942639] Freeing unused kernel image (initmem) memory: 3644K1730lb # [ 0.949597] Freeing unused kernel image (text/rodata gap) memory: 1208K1731worker2 # [ 0.943762] Write protecting the kernel read-only data: 32768k1732lb # [ 0.951119] Freeing unused kernel image (rodata/data gap) memory: 752K1733worker2 # [ 0.945696] Freeing unused kernel image (text/rodata gap) memory: 1208K1734worker2 # [ 0.947290] Freeing unused kernel image (rodata/data gap) memory: 752K1735client # [ 0.994881] x86/mm: Checked W+X mappings: passed, no W+X pages found.1736client # [ 0.995981] Run /init as init process1737worker1 # [ 0.991729] x86/mm: Checked W+X mappings: passed, no W+X pages found.1738worker1 # [ 0.992869] Run /init as init process1739client # [ 1.006267] systemd[1]: Inserted module 'autofs4'1740lb # [ 1.002336] x86/mm: Checked W+X mappings: passed, no W+X pages found.1741lb # [ 1.003437] Run /init as init process1742worker1 # [ 1.003535] systemd[1]: Inserted module 'autofs4'1743worker2 # [ 0.998457] x86/mm: Checked W+X mappings: passed, no W+X pages found.1744worker2 # [ 0.999519] Run /init as init process1745lb # [ 1.013912] systemd[1]: Inserted module 'autofs4'1746worker2 # [ 1.010039] systemd[1]: Inserted module 'autofs4'1747client # [ 1.022547] fuse: init (API version 7.45)1748worker1 # [ 1.019749] fuse: init (API version 7.45)1749client # [ 1.028387] ACPI: \_SB_.GSIG: Enabled at IRQ 221750client # [ 1.030995] ACPI: \_SB_.GSIH: Enabled at IRQ 231751worker1 # [ 1.025556] ACPI: \_SB_.GSIG: Enabled at IRQ 221752client # [ 1.034269] ACPI: \_SB_.GSIE: Enabled at IRQ 201753worker1 # [ 1.028254] ACPI: \_SB_.GSIH: Enabled at IRQ 231754lb # [ 1.030180] fuse: init (API version 7.45)1755client # [ 1.036803] ACPI: \_SB_.GSIF: Enabled at IRQ 211756worker1 # [ 1.031588] ACPI: \_SB_.GSIE: Enabled at IRQ 201757worker2 # [ 1.026699] fuse: init (API version 7.45)1758worker1 # [ 1.034038] ACPI: \_SB_.GSIF: Enabled at IRQ 211759lb # [ 1.035986] ACPI: \_SB_.GSIG: Enabled at IRQ 221760lb # [ 1.038772] ACPI: \_SB_.GSIH: Enabled at IRQ 231761worker2 # [ 1.032521] ACPI: \_SB_.GSIG: Enabled at IRQ 221762worker2 # [ 1.035056] ACPI: \_SB_.GSIH: Enabled at IRQ 231763lb # [ 1.041917] ACPI: \_SB_.GSIE: Enabled at IRQ 201764lb # [ 1.044244] ACPI: \_SB_.GSIF: Enabled at IRQ 211765worker2 # [ 1.038091] ACPI: \_SB_.GSIE: Enabled at IRQ 201766worker2 # [ 1.040396] ACPI: \_SB_.GSIF: Enabled at IRQ 211767client # [ 1.063236] systemd[1]: Successfully made /usr/ read-only.1768worker1 # [ 1.061052] systemd[1]: Successfully made /usr/ read-only.1769lb # [ 1.071282] systemd[1]: Successfully made /usr/ read-only.1770worker2 # [ 1.065695] systemd[1]: Successfully made /usr/ read-only.1771worker1 # [ 1.396648] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1772worker1 # [ 1.401799] systemd[1]: Detected virtualization kvm.1773worker1 # [ 1.402654] systemd[1]: Detected architecture x86-64.1774worker1 # [ 1.403528] systemd[1]: Running in initrd.1775client # [ 1.399790] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1776worker1 # [ 1.404662] systemd[1]: Initializing machine ID from random generator.1777worker1 # [ 1.405892] systemd[1]: Hostname set to <worker1>.1778client # [ 1.411767] systemd[1]: Detected virtualization kvm.1779client # [ 1.413873] systemd[1]: Detected architecture x86-64.1780client # [ 1.416018] systemd[1]: Running in initrd.1781worker2 # [ 1.401231] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1782worker2 # [ 1.406263] systemd[1]: Detected virtualization kvm.1783worker2 # [ 1.407131] systemd[1]: Detected architecture x86-64.1784worker2 # [ 1.407993] systemd[1]: Running in initrd.1785client # [ 1.418455] systemd[1]: Initializing machine ID from random generator.1786worker2 # [ 1.409138] systemd[1]: Initializing machine ID from random generator.1787client # [ 1.421306] systemd[1]: Hostname set to <client>.1788worker2 # [ 1.410338] systemd[1]: Hostname set to <worker2>.1789lb # [ 1.408201] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1790lb # [ 1.420057] systemd[1]: Detected virtualization kvm.1791lb # [ 1.422214] systemd[1]: Detected architecture x86-64.1792lb # [ 1.424287] systemd[1]: Running in initrd.1793lb # [ 1.426943] systemd[1]: Initializing machine ID from random generator.1794lb # [ 1.429914] systemd[1]: Hostname set to <lb>.1795worker1 # [ 1.596487] systemd[1]: bpf-restrict-fs: LSM BPF program attached1796worker2 # [ 1.599101] systemd[1]: bpf-restrict-fs: LSM BPF program attached1797worker1 # [ 1.634636] systemd[1]: Queued start job for default target Initrd Default Target.1798worker1 # [ 1.638612] systemd[1]: Created slice Slice /system/modprobe.1799worker1 # [ 1.639806] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1800worker1 # [ 1.641235] systemd[1]: Expecting device /dev/disk/by-label/nixos...1801worker1 # [ 1.642429] systemd[1]: Reached target Path Units.1802worker1 # [ 1.643339] systemd[1]: Reached target Slice Units.1803worker2 # [ 1.637526] systemd[1]: Queued start job for default target Initrd Default Target.1804worker1 # [ 1.644184] systemd[1]: Reached target Swaps.1805client # [ 1.649486] systemd[1]: bpf-restrict-fs: LSM BPF program attached1806worker1 # [ 1.645000] systemd[1]: Reached target Timer Units.1807worker1 # [ 1.645956] systemd[1]: Listening on D-Bus System Message Bus Socket.1808worker2 # [ 1.641220] systemd[1]: Created slice Slice /system/modprobe.1809worker1 # [ 1.647202] systemd[1]: Listening on Journal Socket (/dev/log).1810worker1 # [ 1.648454] systemd[1]: Listening on Journal Sockets.1811worker2 # [ 1.642502] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1812worker1 # [ 1.649469] systemd[1]: Listening on udev Control Socket.1813worker2 # [ 1.643963] systemd[1]: Expecting device /dev/disk/by-label/nixos...1814worker1 # [ 1.650504] systemd[1]: Listening on udev Kernel Socket.1815worker2 # [ 1.645053] systemd[1]: Reached target Path Units.1816worker1 # [ 1.651481] systemd[1]: Reached target Socket Units.1817worker2 # [ 1.645956] systemd[1]: Reached target Slice Units.1818worker2 # [ 1.646829] systemd[1]: Reached target Swaps.1819worker2 # [ 1.647673] systemd[1]: Reached target Timer Units.1820worker1 # [ 1.653236] systemd[1]: Starting Create List of Static Device Nodes...1821lb # [ 1.653287] systemd[1]: bpf-restrict-fs: LSM BPF program attached1822worker2 # [ 1.648605] systemd[1]: Listening on D-Bus System Message Bus Socket.1823worker2 # [ 1.649805] systemd[1]: Listening on Journal Socket (/dev/log).1824worker2 # [ 1.651053] systemd[1]: Listening on Journal Sockets.1825worker1 # [ 1.657134] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1826worker2 # [ 1.652087] systemd[1]: Listening on udev Control Socket.1827worker2 # [ 1.653195] systemd[1]: Listening on udev Kernel Socket.1828worker2 # [ 1.654192] systemd[1]: Reached target Socket Units.1829worker2 # [ 1.656108] systemd[1]: Starting Create List of Static Device Nodes...1830worker1 # [ 1.664504] systemd[1]: Starting Load Kernel Module configfs...1831worker2 # [ 1.660005] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1832worker2 # [ 1.666608] systemd[1]: Starting Load Kernel Module configfs...1833worker1 # [ 1.677547] systemd[1]: Starting Journal Service...1834worker2 # [ 1.682410] systemd[1]: Starting Journal Service...1835worker1 # [ 1.691460] systemd[1]: Starting Load Kernel Modules...1836client # [ 1.704758] systemd[1]: Queued start job for default target Initrd Default Target.1837worker2 # [ 1.694033] systemd[1]: Starting Load Kernel Modules...1838client # [ 1.708910] systemd[1]: Created slice Slice /system/modprobe.1839client # [ 1.710095] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1840client # [ 1.711554] systemd[1]: Expecting device /dev/disk/by-label/nixos...1841client # [ 1.712702] systemd[1]: Reached target Path Units.1842client # [ 1.713606] systemd[1]: Reached target Slice Units.1843client # [ 1.714562] systemd[1]: Reached target Swaps.1844lb # [ 1.708905] systemd[1]: Queued start job for default target Initrd Default Target.1845client # [ 1.715315] systemd[1]: Reached target Timer Units.1846client # [ 1.716274] systemd[1]: Listening on D-Bus System Message Bus Socket.1847worker1 # [ 1.698397] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1848client # [ 1.717431] systemd[1]: Listening on Journal Socket (/dev/log).1849client # [ 1.718556] systemd[1]: Listening on Journal Sockets.1850lb # [ 1.713566] systemd[1]: Created slice Slice /system/modprobe.1851client # [ 1.719580] systemd[1]: Listening on udev Control Socket.1852client # [ 1.720565] systemd[1]: Listening on udev Kernel Socket.1853worker1 # [ 1.714126] netfs: FS-Cache loaded1854lb # [ 1.714784] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1855client # [ 1.721455] systemd[1]: Reached target Socket Units.1856lb # [ 1.716194] systemd[1]: Expecting device /dev/disk/by-label/nixos...1857lb # [ 1.717300] systemd[1]: Reached target Path Units.1858client # [ 1.723271] systemd[1]: Starting Create List of Static Device Nodes...1859lb # [ 1.718181] systemd[1]: Reached target Slice Units.1860lb # [ 1.719023] systemd[1]: Reached target Swaps.1861worker1 # [ 1.718784] 9pnet: Installing 9P2000 support1862worker2 # [ 1.701012] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1863lb # [ 1.719864] systemd[1]: Reached target Timer Units.1864lb # [ 1.720855] systemd[1]: Listening on D-Bus System Message Bus Socket.1865client # [ 1.727586] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1866lb # [ 1.722109] systemd[1]: Listening on Journal Socket (/dev/log).1867lb # [ 1.723324] systemd[1]: Listening on Journal Sockets.1868worker2 # [ 1.717218] netfs: FS-Cache loaded1869lb # [ 1.724342] systemd[1]: Listening on udev Control Socket.1870lb # [ 1.725369] systemd[1]: Listening on udev Kernel Socket.1871lb # [ 1.726330] systemd[1]: Reached target Socket Units.1872worker1 # [ 1.726589] systemd[1]: Starting Coldplug All udev Devices...1873lb # [ 1.728192] systemd[1]: Starting Create List of Static Device Nodes...1874worker2 # [ 1.721963] 9pnet: Installing 9P2000 support1875client # [ 1.734236] systemd[1]: Starting Load Kernel Module configfs...1876lb # [ 1.732005] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1877worker2 # [ 1.731116] systemd[1]: Starting Coldplug All udev Devices...1878lb # [ 1.738248] systemd[1]: Starting Load Kernel Module configfs...1879worker1 # [ 1.740418] systemd[1]: Finished Create List of Static Device Nodes.1880client # [ 1.749584] systemd[1]: Starting Journal Service...1881worker1 # [ 1.745950] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1882lb # [ 1.750808] systemd[1]: Starting Journal Service...1883worker2 # [ 1.745047] systemd[1]: Finished Create List of Static Device Nodes.1884worker1 # [ 1.754768] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1885client # [ 1.762909] systemd[1]: Starting Load Kernel Modules...1886worker2 # [ 1.751530] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1887worker1 # [ 1.759731] systemd[1]: modprobe@configfs.service: Deactivated successfully.1888worker1 # [ 1.762657] systemd-journald[67]: Collecting audit messages is disabled.1889lb # [ 1.764241] systemd[1]: Starting Load Kernel Modules...1890worker2 # [ 1.759310] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1891worker1 # [ 1.767669] systemd[1]: Finished Load Kernel Module configfs.1892worker2 # [ 1.765351] systemd[1]: modprobe@configfs.service: Deactivated successfully.1893worker2 # [ 1.766727] systemd-journald[67]: Collecting audit messages is disabled.1894client # [ 1.767906] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1895worker1 # [ 1.772758] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1896worker2 # [ 1.772276] systemd[1]: Finished Load Kernel Module configfs.1897worker1 # [ 1.777626] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1898client # [ 1.786082] netfs: FS-Cache loaded1899lb # [ 1.770229] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1900worker2 # [ 1.776391] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1901worker1 # [ 1.785398] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1902client # [ 1.793358] 9pnet: Installing 9P2000 support1903worker1 # [ 1.786784] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1904worker2 # [ 1.781558] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1905lb # [ 1.790275] netfs: FS-Cache loaded1906lb # [ 1.795012] 9pnet: Installing 9P2000 support1907worker2 # [ 1.788005] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1908worker2 # [ 1.791077] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1909lb # [ 1.799389] systemd[1]: Starting Coldplug All udev Devices...1910client # [ 1.808611] systemd[1]: Starting Coldplug All udev Devices...1911client # [ 1.814262] systemd-journald[67]: Collecting audit messages is disabled.1912worker1 # [ 1.815391] systemd[1]: Finished Load Kernel Modules.1913client # [ 1.822831] systemd[1]: Finished Create List of Static Device Nodes.1914lb # [ 1.817300] systemd[1]: Finished Create List of Static Device Nodes.1915lb # [ 1.820880] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1916worker2 # [ 1.817607] systemd[1]: Finished Load Kernel Modules.1917worker1 # [ 1.823363] systemd[1]: Starting Apply Kernel Variables...1918client # [ 1.830565] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1919lb # [ 1.829591] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1920client # [ 1.835942] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1921worker2 # [ 1.826116] systemd[1]: Starting Apply Kernel Variables...1922worker1 # [ 1.834474] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1923lb # [ 1.835596] systemd[1]: modprobe@configfs.service: Deactivated successfully.1924client # [ 1.841964] systemd[1]: modprobe@configfs.service: Deactivated successfully.1925lb # [ 1.838576] systemd-journald[67]: Collecting audit messages is disabled.1926lb # [ 1.842564] systemd[1]: Finished Load Kernel Module configfs.1927client # [ 1.848905] systemd[1]: Finished Load Kernel Module configfs.1928worker1 # [ 1.842565] systemd[1]: Starting Create Static Device Nodes in /dev...1929worker2 # [ 1.837031] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1930client # [ 1.852960] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1931lb # [ 1.847627] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1932client # [ 1.855676] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1933worker2 # [ 1.845227] systemd[1]: Starting Create Static Device Nodes in /dev...1934lb # [ 1.853035] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1935client # [ 1.864548] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1936client # [ 1.866399] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1937lb # [ 1.861383] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1938worker1 # [ 1.861626] systemd[1]: Finished Apply Kernel Variables.1939lb # [ 1.865326] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1940worker2 # [ 1.864239] systemd[1]: Finished Apply Kernel Variables.1941worker1 # [ 1.875775] systemd[1]: Started Journal Service.1942worker1 # [ 1.743509] systemd-modules-load[68]: Inserted module 'dm_mod'1943worker1 # [ 1.745308] systemd-modules-load[68]: Inserted module 'virtio_balloon'1944worker1 # [ 1.747837] systemd-modules-load[68]: Inserted module 'virtio_gpu'1945worker1 # [ 1.750149] systemd[1]: Finished Create Static Device Nodes in /dev.1946worker2 # [ 1.879081] systemd[1]: Started Journal Service.1947worker1 # [ 1.752339] systemd[1]: Reached target Preparation for Local File Systems.1948worker1 # [ 1.753500] systemd[1]: Reached target Local File Systems.1949worker2 # [ 1.747150] systemd-modules-load[68]: Inserted module 'dm_mod'1950worker2 # [ 1.748943] systemd-modules-load[68]: Inserted module 'virtio_balloon'1951client # [ 1.895616] systemd[1]: Finished Load Kernel Modules.1952worker2 # [ 1.750311] systemd-modules-load[68]: Inserted module 'virtio_gpu'1953worker1 # [ 1.757099] systemd[1]: Starting Create System Files and Directories...1954worker2 # [ 1.752277] systemd[1]: Finished Create Static Device Nodes in /dev.1955worker2 # [ 1.754391] systemd[1]: Reached target Preparation for Local File Systems.1956worker2 # [ 1.755762] systemd[1]: Reached target Local File Systems.1957worker1 # [ 1.762063] systemd[1]: Starting Rule-based Manager for Device Events and Files...1958lb # [ 1.896784] systemd[1]: Finished Load Kernel Modules.1959client # [ 1.904002] systemd[1]: Starting Apply Kernel Variables...1960worker2 # [ 1.759248] systemd[1]: Starting Create System Files and Directories...1961lb # [ 1.902773] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1962worker2 # [ 1.765984] systemd[1]: Starting Rule-based Manager for Device Events and Files...1963lb # [ 1.911391] systemd[1]: Starting Apply Kernel Variables...1964client # [ 1.920834] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1965worker1 # [ 1.788190] systemd[1]: Finished Create System Files and Directories.1966lb # [ 1.924243] systemd[1]: Starting Create Static Device Nodes in /dev...1967client # [ 1.930743] systemd[1]: Starting Create Static Device Nodes in /dev...1968client # [ 1.800149] systemd-modules-load[68]: Inserted module 'dm_mod'1969client # [ 1.803279] systemd-modules-load[68]: Inserted module 'virtio_balloon'1970client # [ 1.804337] systemd-modules-load[68]: Inserted module 'virtio_gpu'1971worker2 # [ 1.794373] systemd[1]: Finished Create System Files and Directories.1972client # [ 1.943733] systemd[1]: Started Journal Service.1973client # [ 1.815918] systemd[1]: Finished Apply Kernel Variables.1974worker1 # [ 1.816392] systemd-udevd[78]: Using default interface naming scheme 'v261'.1975lb # [ 1.952806] systemd[1]: Started Journal Service.1976lb # [ 1.822224] systemd-modules-load[68]: Inserted module 'dm_mod'1977client # [ 1.832194] systemd[1]: Finished Create Static Device Nodes in /dev.1978worker2 # [ 1.818653] systemd-udevd[77]: Using default interface naming scheme 'v261'.1979lb # [ 1.827132] systemd-modules-load[68]: Inserted module 'virtio_balloon'1980client # [ 1.833632] systemd[1]: Reached target Preparation for Local File Systems.1981lb # [ 1.828242] systemd-modules-load[68]: Inserted module 'virtio_gpu'1982client # [ 1.835401] systemd[1]: Reached target Local File Systems.1983lb # [ 1.829971] systemd[1]: Finished Apply Kernel Variables.1984lb # [ 1.831069] systemd[1]: Finished Create Static Device Nodes in /dev.1985client # [ 1.839058] systemd[1]: Starting Create System Files and Directories...1986lb # [ 1.838955] systemd[1]: Reached target Preparation for Local File Systems.1987client # [ 1.846075] systemd[1]: Starting Rule-based Manager for Device Events and Files...1988worker1 # [ 1.840363] systemd[1]: Started Rule-based Manager for Device Events and Files.1989lb # [ 1.841819] systemd[1]: Reached target Local File Systems.1990lb # [ 1.845067] systemd[1]: Starting Create System Files and Directories...1991worker2 # [ 1.842123] systemd[1]: Started Rule-based Manager for Device Events and Files.1992lb # [ 1.850156] systemd[1]: Starting Rule-based Manager for Device Events and Files...1993client # [ 1.870653] systemd[1]: Finished Create System Files and Directories.1994lb # [ 1.876316] systemd[1]: Finished Create System Files and Directories.1995worker1 # [ 1.881525] systemd[1]: Finished Coldplug All udev Devices.1996worker1 # [ 1.882396] systemd[1]: Reached target System Initialization.1997worker1 # [ 1.884146] systemd[1]: Reached target Basic System.1998client # [ 1.893278] systemd-udevd[83]: Using default interface naming scheme 'v261'.1999worker2 # [ 1.881335] systemd[1]: Finished Coldplug All udev Devices.2000worker2 # [ 1.884061] systemd[1]: Reached target System Initialization.
loading 2,360 hidden lines…
4361worker2 # [ 27.010098] nix-grpc-daemon[783]: ts=2026-09-21T13:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4362worker1 # [ 27.124543] nix-grpc-daemon[785]: ts=2026-09-21T13:54:04Z level=info event=scheduler_connected addr=lb:500514363worker1 # [ 27.128367] nix-grpc-daemon[785]: ts=2026-09-21T13:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4364lb: (finished: waiting for unit niks3.service, in 15.24 seconds)4365lb: waiting for TCP port 5751 on localhost4366worker2 # [ 27.211253] nix-grpc-daemon[783]: ts=2026-09-21T13:54:03Z level=info event=scheduler_connected addr=lb:500514367worker2 # [ 27.215091] nix-grpc-daemon[783]: ts=2026-09-21T13:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4368lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!4369lb: (finished: waiting for TCP port 5751 on localhost, in 0.09 seconds)4370lb: waiting for unit envoy.service4371worker1 # [ 27.329503] nix-grpc-daemon[785]: ts=2026-09-21T13:54:04Z level=info event=scheduler_connected addr=lb:500514372worker1 # [ 27.333345] nix-grpc-daemon[785]: ts=2026-09-21T13:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4373lb: (finished: waiting for unit envoy.service, in 0.07 seconds)4374lb: waiting for TCP port 50051 on localhost4375lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!4376lb # [ 27.398169] envoy[901]: conn peer=[::1]:33198 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=34377lb: (finished: waiting for TCP port 50051 on localhost, in 0.03 seconds)4378worker2 # [ 27.416503] nix-grpc-daemon[783]: ts=2026-09-21T13:54:03Z level=info event=scheduler_connected addr=lb:500514379worker2 # [ 27.420253] nix-grpc-daemon[783]: ts=2026-09-21T13:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4380worker1 # [ 27.534265] nix-grpc-daemon[785]: ts=2026-09-21T13:54:04Z level=info event=scheduler_connected addr=lb:500514381worker1 # [ 27.538126] nix-grpc-daemon[785]: ts=2026-09-21T13:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4382worker2 # [ 27.534804] systemd[1]: Starting scheduler-token.service...4383worker2 # [ 27.613183] systemd[1]: scheduler-token.service: Deactivated successfully.4384worker2 # [ 27.614766] systemd[1]: Finished scheduler-token.service.4385worker2 # [ 27.616520] systemd[1]: scheduler-token.service: Consumed 20ms CPU time over 81ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4386worker2 # [ 27.620823] nix-grpc-daemon[783]: ts=2026-09-21T13:54:03Z level=info event=scheduler_connected addr=lb:500514387worker2 # [ 27.622451] nix-grpc-daemon[783]: ts=2026-09-21T13:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4388lb: waiting until exactly no endpoints of x86_64-linux are unhealthy4389lb: must succeed: curl -sf localhost:9901/clusters4390lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4391lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.08 seconds)4392subtest: one node schedules and both builders hold a WorkerSession on it4393lb: must succeed: curl -sf localhost:9901/clusters4394worker1 # [ 27.739085] nix-grpc-daemon[785]: ts=2026-09-21T13:54:04Z level=info event=scheduler_connected addr=lb:500514395worker1 # [ 27.743206] nix-grpc-daemon[785]: ts=2026-09-21T13:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4396lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4397worker2 # [ 27.823531] nix-grpc-daemon[783]: ts=2026-09-21T13:54:04Z level=info event=scheduler_connected addr=lb:500514398worker2 # [ 27.827343] nix-grpc-daemon[783]: ts=2026-09-21T13:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4399worker1 # [ 27.944938] nix-grpc-daemon[785]: ts=2026-09-21T13:54:05Z level=info event=scheduler_connected addr=lb:500514400worker1 # [ 27.948771] nix-grpc-daemon[785]: ts=2026-09-21T13:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4401worker2 # [ 28.029621] nix-grpc-daemon[783]: ts=2026-09-21T13:54:04Z level=info event=scheduler_connected addr=lb:500514402worker2 # [ 28.032836] nix-grpc-daemon[783]: ts=2026-09-21T13:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4403worker1 # [ 28.146805] nix-grpc-daemon[785]: ts=2026-09-21T13:54:05Z level=info event=scheduler_connected addr=lb:500514404worker1 # [ 28.150702] nix-grpc-daemon[785]: ts=2026-09-21T13:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4405worker2 # [ 28.233843] nix-grpc-daemon[783]: ts=2026-09-21T13:54:04Z level=info event=scheduler_connected addr=lb:500514406worker2 # [ 28.237629] nix-grpc-daemon[783]: ts=2026-09-21T13:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4407worker1 # [ 28.351573] nix-grpc-daemon[785]: ts=2026-09-21T13:54:05Z level=info event=scheduler_connected addr=lb:500514408worker1 # [ 28.355524] nix-grpc-daemon[785]: ts=2026-09-21T13:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4409worker2 # [ 28.438823] nix-grpc-daemon[783]: ts=2026-09-21T13:54:04Z level=info event=scheduler_connected addr=lb:500514410worker2 # [ 28.442684] nix-grpc-daemon[783]: ts=2026-09-21T13:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4411worker1 # [ 28.556596] nix-grpc-daemon[785]: ts=2026-09-21T13:54:05Z level=info event=scheduler_connected addr=lb:500514412worker1 # [ 28.560578] nix-grpc-daemon[785]: ts=2026-09-21T13:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4413worker2 # [ 28.643915] nix-grpc-daemon[783]: ts=2026-09-21T13:54:04Z level=info event=scheduler_connected addr=lb:500514414worker2 # [ 28.647864] nix-grpc-daemon[783]: ts=2026-09-21T13:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4415lb: must succeed: curl -sf localhost:9901/clusters4416worker1 # [ 28.761537] nix-grpc-daemon[785]: ts=2026-09-21T13:54:05Z level=info event=scheduler_connected addr=lb:500514417worker1 # [ 28.765318] nix-grpc-daemon[785]: ts=2026-09-21T13:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4418lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4419worker2 # [ 28.849133] nix-grpc-daemon[783]: ts=2026-09-21T13:54:05Z level=info event=scheduler_connected addr=lb:500514420worker2 # [ 28.852956] nix-grpc-daemon[783]: ts=2026-09-21T13:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4421worker1 # [ 28.967444] nix-grpc-daemon[785]: ts=2026-09-21T13:54:06Z level=info event=scheduler_connected addr=lb:500514422worker1 # [ 28.970516] nix-grpc-daemon[785]: ts=2026-09-21T13:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4423worker2 # [ 29.055215] nix-grpc-daemon[783]: ts=2026-09-21T13:54:05Z level=info event=scheduler_connected addr=lb:500514424worker2 # [ 29.058419] nix-grpc-daemon[783]: ts=2026-09-21T13:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4425worker1 # [ 29.167981] nix-grpc-daemon[785]: ts=2026-09-21T13:54:06Z level=info event=scheduler_connected addr=lb:500514426worker1 # [ 29.171909] nix-grpc-daemon[785]: ts=2026-09-21T13:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4427worker2 # [ 29.256203] nix-grpc-daemon[783]: ts=2026-09-21T13:54:05Z level=info event=scheduler_connected addr=lb:500514428worker2 # [ 29.260085] nix-grpc-daemon[783]: ts=2026-09-21T13:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4429worker1 # [ 29.372702] nix-grpc-daemon[785]: ts=2026-09-21T13:54:06Z level=info event=scheduler_connected addr=lb:500514430worker1 # [ 29.376713] nix-grpc-daemon[785]: ts=2026-09-21T13:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4431worker2 # [ 29.461392] nix-grpc-daemon[783]: ts=2026-09-21T13:54:05Z level=info event=scheduler_connected addr=lb:500514432worker2 # [ 29.465192] nix-grpc-daemon[783]: ts=2026-09-21T13:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4433worker1 # [ 29.577763] nix-grpc-daemon[785]: ts=2026-09-21T13:54:06Z level=info event=scheduler_connected addr=lb:500514434worker1 # [ 29.581889] nix-grpc-daemon[785]: ts=2026-09-21T13:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4435worker2 # [ 29.666260] nix-grpc-daemon[783]: ts=2026-09-21T13:54:05Z level=info event=scheduler_connected addr=lb:500514436worker2 # [ 29.670610] nix-grpc-daemon[783]: ts=2026-09-21T13:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4437worker1 # [ 29.782880] nix-grpc-daemon[785]: ts=2026-09-21T13:54:06Z level=info event=scheduler_connected addr=lb:500514438worker1 # [ 29.786748] nix-grpc-daemon[785]: ts=2026-09-21T13:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4439lb: must succeed: curl -sf localhost:9901/clusters4440lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4441worker2 # [ 29.871660] nix-grpc-daemon[783]: ts=2026-09-21T13:54:06Z level=info event=scheduler_connected addr=lb:500514442worker2 # [ 29.875905] nix-grpc-daemon[783]: ts=2026-09-21T13:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4443worker1 # [ 29.988459] nix-grpc-daemon[785]: ts=2026-09-21T13:54:07Z level=info event=scheduler_connected addr=lb:500514444worker1 # [ 29.992232] nix-grpc-daemon[785]: ts=2026-09-21T13:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4445worker2 # [ 30.075392] nix-grpc-daemon[783]: ts=2026-09-21T13:54:06Z level=info event=scheduler_connected addr=lb:500514446worker2 # [ 30.080309] nix-grpc-daemon[783]: ts=2026-09-21T13:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4447worker1 # [ 30.189782] nix-grpc-daemon[785]: ts=2026-09-21T13:54:07Z level=info event=scheduler_connected addr=lb:500514448worker1 # [ 30.193491] nix-grpc-daemon[785]: ts=2026-09-21T13:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4449worker2 # [ 30.276183] nix-grpc-daemon[783]: ts=2026-09-21T13:54:06Z level=info event=scheduler_connected addr=lb:500514450worker2 # [ 30.279979] nix-grpc-daemon[783]: ts=2026-09-21T13:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4451worker1 # [ 30.394427] nix-grpc-daemon[785]: ts=2026-09-21T13:54:07Z level=info event=scheduler_connected addr=lb:500514452worker1 # [ 30.398360] nix-grpc-daemon[785]: ts=2026-09-21T13:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4453worker2 # [ 30.481348] nix-grpc-daemon[783]: ts=2026-09-21T13:54:06Z level=info event=scheduler_connected addr=lb:500514454worker2 # [ 30.485179] nix-grpc-daemon[783]: ts=2026-09-21T13:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4455worker1 # [ 30.599525] nix-grpc-daemon[785]: ts=2026-09-21T13:54:07Z level=info event=scheduler_connected addr=lb:500514456worker1 # [ 30.603338] nix-grpc-daemon[785]: ts=2026-09-21T13:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4457worker2 # [ 30.686446] nix-grpc-daemon[783]: ts=2026-09-21T13:54:06Z level=info event=scheduler_connected addr=lb:500514458worker2 # [ 30.690554] nix-grpc-daemon[783]: ts=2026-09-21T13:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4459worker1 # [ 30.803976] nix-grpc-daemon[785]: ts=2026-09-21T13:54:07Z level=info event=scheduler_connected addr=lb:500514460worker1 # [ 30.807850] nix-grpc-daemon[785]: ts=2026-09-21T13:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4461lb: must succeed: curl -sf localhost:9901/clusters4462worker2 # [ 30.891367] nix-grpc-daemon[783]: ts=2026-09-21T13:54:07Z level=info event=scheduler_connected addr=lb:500514463worker2 # [ 30.895260] nix-grpc-daemon[783]: ts=2026-09-21T13:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4464lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4465worker1 # [ 31.009732] nix-grpc-daemon[785]: ts=2026-09-21T13:54:08Z level=info event=scheduler_connected addr=lb:500514466worker1 # [ 31.012765] nix-grpc-daemon[785]: ts=2026-09-21T13:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4467worker2 # [ 31.096952] nix-grpc-daemon[783]: ts=2026-09-21T13:54:07Z level=info event=scheduler_connected addr=lb:500514468worker2 # [ 31.100696] nix-grpc-daemon[783]: ts=2026-09-21T13:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4469worker1 # [ 31.210727] nix-grpc-daemon[785]: ts=2026-09-21T13:54:08Z level=info event=scheduler_connected addr=lb:500514470worker1 # [ 31.214642] nix-grpc-daemon[785]: ts=2026-09-21T13:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4471worker2 # [ 31.301599] nix-grpc-daemon[783]: ts=2026-09-21T13:54:07Z level=info event=scheduler_connected addr=lb:500514472worker2 # [ 31.305503] nix-grpc-daemon[783]: ts=2026-09-21T13:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4473worker1 # [ 31.415528] nix-grpc-daemon[785]: ts=2026-09-21T13:54:08Z level=info event=scheduler_connected addr=lb:500514474worker1 # [ 31.419358] nix-grpc-daemon[785]: ts=2026-09-21T13:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4475worker2 # [ 31.506526] nix-grpc-daemon[783]: ts=2026-09-21T13:54:07Z level=info event=scheduler_connected addr=lb:500514476worker2 # [ 31.510422] nix-grpc-daemon[783]: ts=2026-09-21T13:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4477worker1 # [ 31.620320] nix-grpc-daemon[785]: ts=2026-09-21T13:54:08Z level=info event=scheduler_connected addr=lb:500514478worker1 # [ 31.624193] nix-grpc-daemon[785]: ts=2026-09-21T13:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4479worker2 # [ 31.711509] nix-grpc-daemon[783]: ts=2026-09-21T13:54:08Z level=info event=scheduler_connected addr=lb:500514480worker2 # [ 31.715366] nix-grpc-daemon[783]: ts=2026-09-21T13:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4481worker1 # [ 31.825127] nix-grpc-daemon[785]: ts=2026-09-21T13:54:08Z level=info event=scheduler_connected addr=lb:500514482worker1 # [ 31.828945] nix-grpc-daemon[785]: ts=2026-09-21T13:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4483worker2 # [ 31.916499] nix-grpc-daemon[783]: ts=2026-09-21T13:54:08Z level=info event=scheduler_connected addr=lb:500514484worker2 # [ 31.920259] nix-grpc-daemon[783]: ts=2026-09-21T13:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4485lb: must succeed: curl -sf localhost:9901/clusters4486lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4487worker1 # [ 32.030969] nix-grpc-daemon[785]: ts=2026-09-21T13:54:09Z level=info event=scheduler_connected addr=lb:500514488worker1 # [ 32.034157] nix-grpc-daemon[785]: ts=2026-09-21T13:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4489worker2 # [ 32.122585] nix-grpc-daemon[783]: ts=2026-09-21T13:54:08Z level=info event=scheduler_connected addr=lb:500514490worker2 # [ 32.125684] nix-grpc-daemon[783]: ts=2026-09-21T13:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4491worker1 # [ 32.231957] nix-grpc-daemon[785]: ts=2026-09-21T13:54:09Z level=info event=scheduler_connected addr=lb:500514492worker1 # [ 32.235857] nix-grpc-daemon[785]: ts=2026-09-21T13:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4493worker2 # [ 32.323694] nix-grpc-daemon[783]: ts=2026-09-21T13:54:08Z level=info event=scheduler_connected addr=lb:500514494worker2 # [ 32.327629] nix-grpc-daemon[783]: ts=2026-09-21T13:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4495worker1 # [ 32.436771] nix-grpc-daemon[785]: ts=2026-09-21T13:54:09Z level=info event=scheduler_connected addr=lb:500514496worker1 # [ 32.440519] nix-grpc-daemon[785]: ts=2026-09-21T13:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4497worker2 # [ 32.528698] nix-grpc-daemon[783]: ts=2026-09-21T13:54:08Z level=info event=scheduler_connected addr=lb:500514498worker2 # [ 32.532516] nix-grpc-daemon[783]: ts=2026-09-21T13:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4499worker1 # [ 32.640898] nix-grpc-daemon[785]: ts=2026-09-21T13:54:09Z level=info event=scheduler_connected addr=lb:500514500worker1 # [ 32.642722] nix-grpc-daemon[785]: ts=2026-09-21T13:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4501worker2 # [ 32.733470] nix-grpc-daemon[783]: ts=2026-09-21T13:54:09Z level=info event=scheduler_connected addr=lb:500514502worker2 # [ 32.737312] nix-grpc-daemon[783]: ts=2026-09-21T13:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4503worker1 # [ 32.843620] nix-grpc-daemon[785]: ts=2026-09-21T13:54:09Z level=info event=scheduler_connected addr=lb:500514504worker1 # [ 32.847562] nix-grpc-daemon[785]: ts=2026-09-21T13:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4505worker2 # [ 32.938146] nix-grpc-daemon[783]: ts=2026-09-21T13:54:09Z level=info event=scheduler_connected addr=lb:500514506worker2 # [ 32.941980] nix-grpc-daemon[783]: ts=2026-09-21T13:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4507lb: must succeed: curl -sf localhost:9901/clusters4508worker1 # [ 33.049972] nix-grpc-daemon[785]: ts=2026-09-21T13:54:10Z level=info event=scheduler_connected addr=lb:500514509worker1 # [ 33.053161] nix-grpc-daemon[785]: ts=2026-09-21T13:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4510lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4511worker2 # [ 33.143937] nix-grpc-daemon[783]: ts=2026-09-21T13:54:09Z level=info event=scheduler_connected addr=lb:500514512worker2 # [ 33.147623] nix-grpc-daemon[783]: ts=2026-09-21T13:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4513worker1 # [ 33.250953] nix-grpc-daemon[785]: ts=2026-09-21T13:54:10Z level=info event=scheduler_connected addr=lb:500514514worker1 # [ 33.254932] nix-grpc-daemon[785]: ts=2026-09-21T13:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4515worker2 # [ 33.348304] nix-grpc-daemon[783]: ts=2026-09-21T13:54:09Z level=info event=scheduler_connected addr=lb:500514516worker2 # [ 33.350270] nix-grpc-daemon[783]: ts=2026-09-21T13:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4517worker1 # [ 33.455688] nix-grpc-daemon[785]: ts=2026-09-21T13:54:10Z level=info event=scheduler_connected addr=lb:500514518worker1 # [ 33.459554] nix-grpc-daemon[785]: ts=2026-09-21T13:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4519worker2 # [ 33.551302] nix-grpc-daemon[783]: ts=2026-09-21T13:54:09Z level=info event=scheduler_connected addr=lb:500514520worker2 # [ 33.555214] nix-grpc-daemon[783]: ts=2026-09-21T13:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4521worker1 # [ 33.660455] nix-grpc-daemon[785]: ts=2026-09-21T13:54:10Z level=info event=scheduler_connected addr=lb:500514522worker1 # [ 33.664341] nix-grpc-daemon[785]: ts=2026-09-21T13:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4523worker2 # [ 33.756252] nix-grpc-daemon[783]: ts=2026-09-21T13:54:10Z level=info event=scheduler_connected addr=lb:500514524worker2 # [ 33.760111] nix-grpc-daemon[783]: ts=2026-09-21T13:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4525worker1 # [ 33.865240] nix-grpc-daemon[785]: ts=2026-09-21T13:54:11Z level=info event=scheduler_connected addr=lb:500514526worker1 # [ 33.869375] nix-grpc-daemon[785]: ts=2026-09-21T13:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4527worker2 # [ 33.960809] nix-grpc-daemon[783]: ts=2026-09-21T13:54:10Z level=info event=scheduler_connected addr=lb:500514528worker2 # [ 33.964564] nix-grpc-daemon[783]: ts=2026-09-21T13:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4529worker1 # [ 34.071611] nix-grpc-daemon[785]: ts=2026-09-21T13:54:11Z level=info event=scheduler_connected addr=lb:500514530worker1 # [ 34.074678] nix-grpc-daemon[785]: ts=2026-09-21T13:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4531lb: must succeed: curl -sf localhost:9901/clusters4532lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4533worker2 # [ 34.167297] nix-grpc-daemon[783]: ts=2026-09-21T13:54:10Z level=info event=scheduler_connected addr=lb:500514534worker2 # [ 34.170659] nix-grpc-daemon[783]: ts=2026-09-21T13:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4535worker1 # [ 34.272806] nix-grpc-daemon[785]: ts=2026-09-21T13:54:11Z level=info event=scheduler_connected addr=lb:500514536worker1 # [ 34.276637] nix-grpc-daemon[785]: ts=2026-09-21T13:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4537worker2 # [ 34.368365] nix-grpc-daemon[783]: ts=2026-09-21T13:54:10Z level=info event=scheduler_connected addr=lb:500514538worker2 # [ 34.372196] nix-grpc-daemon[783]: ts=2026-09-21T13:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4539worker1 # [ 34.477546] nix-grpc-daemon[785]: ts=2026-09-21T13:54:11Z level=info event=scheduler_connected addr=lb:500514540worker1 # [ 34.481385] nix-grpc-daemon[785]: ts=2026-09-21T13:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4541worker2 # [ 34.573311] nix-grpc-daemon[783]: ts=2026-09-21T13:54:10Z level=info event=scheduler_connected addr=lb:500514542worker2 # [ 34.577229] nix-grpc-daemon[783]: ts=2026-09-21T13:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4543worker1 # [ 34.682372] nix-grpc-daemon[785]: ts=2026-09-21T13:54:11Z level=info event=scheduler_connected addr=lb:500514544worker1 # [ 34.686254] nix-grpc-daemon[785]: ts=2026-09-21T13:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4545worker2 # [ 34.778189] nix-grpc-daemon[783]: ts=2026-09-21T13:54:11Z level=info event=scheduler_connected addr=lb:500514546worker2 # [ 34.782152] nix-grpc-daemon[783]: ts=2026-09-21T13:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4547worker1 # [ 34.886787] nix-grpc-daemon[785]: ts=2026-09-21T13:54:12Z level=info event=scheduler_connected addr=lb:500514548worker1 # [ 34.890537] nix-grpc-daemon[785]: ts=2026-09-21T13:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4549worker2 # [ 34.982973] nix-grpc-daemon[783]: ts=2026-09-21T13:54:11Z level=info event=scheduler_connected addr=lb:500514550worker2 # [ 34.986867] nix-grpc-daemon[783]: ts=2026-09-21T13:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4551worker1 # [ 35.092882] nix-grpc-daemon[785]: ts=2026-09-21T13:54:12Z level=info event=scheduler_connected addr=lb:500514552worker1 # [ 35.095915] nix-grpc-daemon[785]: ts=2026-09-21T13:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4553lb: must succeed: curl -sf localhost:9901/clusters4554worker2 # [ 35.189202] nix-grpc-daemon[783]: ts=2026-09-21T13:54:11Z level=info event=scheduler_connected addr=lb:500514555worker2 # [ 35.192429] nix-grpc-daemon[783]: ts=2026-09-21T13:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4556lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4557worker1 # [ 35.293917] nix-grpc-daemon[785]: ts=2026-09-21T13:54:12Z level=info event=scheduler_connected addr=lb:500514558worker1 # [ 35.297894] nix-grpc-daemon[785]: ts=2026-09-21T13:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4559worker2 # [ 35.390214] nix-grpc-daemon[783]: ts=2026-09-21T13:54:11Z level=info event=scheduler_connected addr=lb:500514560worker2 # [ 35.394108] nix-grpc-daemon[783]: ts=2026-09-21T13:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4561worker1 # [ 35.498796] nix-grpc-daemon[785]: ts=2026-09-21T13:54:12Z level=info event=scheduler_connected addr=lb:500514562worker1 # [ 35.502760] nix-grpc-daemon[785]: ts=2026-09-21T13:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4563worker2 # [ 35.595258] nix-grpc-daemon[783]: ts=2026-09-21T13:54:11Z level=info event=scheduler_connected addr=lb:500514564worker2 # [ 35.599159] nix-grpc-daemon[783]: ts=2026-09-21T13:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4565worker1 # [ 35.703615] nix-grpc-daemon[785]: ts=2026-09-21T13:54:12Z level=info event=scheduler_connected addr=lb:500514566worker1 # [ 35.707390] nix-grpc-daemon[785]: ts=2026-09-21T13:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4567worker2 # [ 35.800315] nix-grpc-daemon[783]: ts=2026-09-21T13:54:12Z level=info event=scheduler_connected addr=lb:500514568worker2 # [ 35.804257] nix-grpc-daemon[783]: ts=2026-09-21T13:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4569worker1 # [ 35.907746] nix-grpc-daemon[785]: ts=2026-09-21T13:54:13Z level=info event=scheduler_connected addr=lb:500514570worker1 # [ 35.910158] nix-grpc-daemon[785]: ts=2026-09-21T13:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4571worker2 # [ 36.004989] nix-grpc-daemon[783]: ts=2026-09-21T13:54:12Z level=info event=scheduler_connected addr=lb:500514572worker2 # [ 36.008832] nix-grpc-daemon[783]: ts=2026-09-21T13:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4573worker1 # [ 36.111784] nix-grpc-daemon[785]: ts=2026-09-21T13:54:13Z level=info event=scheduler_connected addr=lb:500514574worker1 # [ 36.115614] nix-grpc-daemon[785]: ts=2026-09-21T13:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4575worker2 # [ 36.211306] nix-grpc-daemon[783]: ts=2026-09-21T13:54:12Z level=info event=scheduler_connected addr=lb:500514576worker2 # [ 36.214423] nix-grpc-daemon[783]: ts=2026-09-21T13:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4577lb: must succeed: curl -sf localhost:9901/clusters4578lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4579worker1 # [ 36.316667] nix-grpc-daemon[785]: ts=2026-09-21T13:54:13Z level=info event=scheduler_connected addr=lb:500514580worker1 # [ 36.320500] nix-grpc-daemon[785]: ts=2026-09-21T13:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4581worker2 # [ 36.412717] nix-grpc-daemon[783]: ts=2026-09-21T13:54:12Z level=info event=scheduler_connected addr=lb:500514582worker2 # [ 36.416615] nix-grpc-daemon[783]: ts=2026-09-21T13:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4583worker1 # [ 36.461297] nix-grpc-daemon[785]: ts=2026-09-21T13:54:13Z level=info event=scheduler_take_over4584lb # [ 36.463524] niks3-server[1011]: 2026/09/21 13:54:13 INFO lead: acquired remote=192.168.1.3:520844585worker1 # [ 36.521456] nix-grpc-daemon[785]: ts=2026-09-21T13:54:13Z level=info event=scheduler_connected addr=lb:500514586worker1 # [ 36.525295] nix-grpc-daemon[785]: ts=2026-09-21T13:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4587worker2 # [ 36.617386] nix-grpc-daemon[783]: ts=2026-09-21T13:54:12Z level=info event=scheduler_connected addr=lb:500514588worker2 # [ 36.620554] nix-grpc-daemon[783]: ts=2026-09-21T13:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4589worker1 # [ 36.726268] nix-grpc-daemon[785]: ts=2026-09-21T13:54:13Z level=info event=scheduler_connected addr=lb:500514590worker1 # [ 36.730414] nix-grpc-daemon[785]: ts=2026-09-21T13:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4591worker2 # [ 36.821705] nix-grpc-daemon[783]: ts=2026-09-21T13:54:13Z level=info event=scheduler_connected addr=lb:500514592worker2 # [ 36.826197] nix-grpc-daemon[783]: ts=2026-09-21T13:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4593worker1 # [ 36.930669] nix-grpc-daemon[785]: ts=2026-09-21T13:54:14Z level=info event=scheduler_connected addr=lb:500514594worker1 # [ 36.949265] nix-grpc-daemon[785]: ts=2026-09-21T13:54:14Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:559564595worker1 # [ 36.954158] nix-grpc-daemon[785]: ts=2026-09-21T13:54:14Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04596worker2 # [ 37.026895] nix-grpc-daemon[783]: ts=2026-09-21T13:54:13Z level=info event=scheduler_connected addr=lb:500514597worker1 # [ 37.056217] nix-grpc-daemon[785]: ts=2026-09-21T13:54:14Z level=info event=oidc_jwks provider=mock keys=14598worker1 # [ 37.060614] nix-grpc-daemon[785]: ts=2026-09-21T13:54:14Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:559564599worker1 # [ 37.064495] nix-grpc-daemon[785]: ts=2026-09-21T13:54:14Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04600lb: must succeed: curl -sf localhost:9901/clusters4601lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4602worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4603worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds)4604lb: must succeed: curl -sf localhost:9901/clusters4605lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4606lb: must succeed: curl -sf localhost:9901/clusters4607lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4608lb: must succeed: curl -sf localhost:9901/clusters4609lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4610lb: must succeed: curl -sf localhost:9901/clusters4611lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4612worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4613worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.06 seconds)4614worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4615worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4616lb: must succeed: curl -sf localhost:9901/clusters4617lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4618lb: must succeed: curl -sf localhost:9901/clusters4619lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4620worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4621worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4622worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4623worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4624worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4625worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4626worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4627worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4628(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 10.42 seconds)4629subtest: balancer auth: client cert, bearer token, nothing, unknown CN4630worker2 # [ 38.546130] nix-grpc-daemon[783]: ts=2026-09-21T13:54:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:447084631worker2 # [ 38.558631] systemd[1]: Started Nix Daemon.4632worker2 # [ 38.790162] nix-daemon[853]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4633worker2 # [ 38.814276] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 92 ms (attempt 1/5)4634worker2 # [ 38.909158] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 156 ms (attempt 2/5)4635worker2 # [ 39.065617] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 28 ms (attempt 3/5)4636worker2 # [ 39.099365] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 783 ms (attempt 4/5)4637worker2 # [ 39.891951] nix-grpc-daemon[783]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4638worker2 # [ 39.896170] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4639worker1 # [ 39.941431] nix-grpc-daemon[785]: ts=2026-09-21T13:54:17Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:559704640worker1 # [ 39.953523] systemd[1]: Started Nix Daemon.4641worker1 # [ 40.191578] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4642worker1 # [ 40.218112] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 78 ms (attempt 1/5)4643worker1 # [ 40.298906] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 2/5)4644worker1 # [ 40.318112] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 155 ms (attempt 3/5)4645worker1 # [ 40.479122] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 161 ms (attempt 4/5)4646worker1 # [ 40.649355] nix-grpc-daemon[785]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4647worker1 # [ 40.653574] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4648worker1 # [ 40.673471] nix-grpc-daemon[785]: ts=2026-09-21T13:54:17Z level=debug event=path_info_miss path=6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test error="error: path '/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test' is required, but there is no substituter that can build it"4649worker2 # [ 40.688382] nix-grpc-daemon[783]: ts=2026-09-21T13:54:17Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:447084650worker2 # [ 40.706157] nix-grpc-daemon[783]: ts=2026-09-21T13:54:17Z level=debug event=path_info_miss path=6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test error="error: path '/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test' is required, but there is no substituter that can build it"4651client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4652client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.09 seconds)4653client: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4654client: (finished: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.05 seconds)4655worker2 # [ 41.153647] nix-grpc-daemon[783]: ts=2026-09-21T13:54:17Z level=info event=oidc_jwks provider=mock keys=14656worker2 # [ 41.157537] nix-grpc-daemon[783]: ts=2026-09-21T13:54:17Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:447084657worker2 # [ 41.165326] nix-grpc-daemon[783]: ts=2026-09-21T13:54:17Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:447084658worker2 # [ 41.174310] nix-grpc-daemon[783]: ts=2026-09-21T13:54:17Z level=debug event=path_info_miss path=6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test error="error: path '/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test' is required, but there is no substituter that can build it"4659worker2 # [ 41.183393] nix-grpc-daemon[783]: ts=2026-09-21T13:54:17Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:447084660worker2 # [ 41.192548] nix-grpc-daemon[783]: ts=2026-09-21T13:54:17Z level=debug event=path_info_miss path=6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test error="error: path '/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test' is required, but there is no substituter that can build it"4661client: must succeed: rm -r /var/lib/nix-grpc-store4662client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds)4663worker1 # [ 41.524547] nix-grpc-daemon[785]: ts=2026-09-21T13:54:18Z level=info event=denied method=QueryMissing cn=- role=none4664worker2 # [ 41.823036] nix-grpc-daemon[783]: ts=2026-09-21T13:54:18Z level=info event=denied method=QueryMissing cn=stranger role=none4665(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.73 seconds)4666subtest: DAG build is scheduled across workers and lands in the cache4667client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24668worker1 # [ 42.133759] nix-grpc-daemon[785]: ts=2026-09-21T13:54:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:559704669client # don't know how to build these paths:4670client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4671worker2 # [ 42.146600] nix-grpc-daemon[783]: ts=2026-09-21T13:54:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447084672worker1 # [ 42.179464] nix-grpc-daemon[785]: ts=2026-09-21T13:54:19Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04673worker1 # [ 42.184707] nix-grpc-daemon[785]: ts=2026-09-21T13:54:19Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14674worker1 # [ 42.226773] nix-grpc-daemon[785]: ts=2026-09-21T13:54:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559704675worker1 # [ 42.251867] nix-grpc-daemon[785]: ts=2026-09-21T13:54:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559704676client # copying 1 paths to 'lb:50051'...4677worker1 # [ 42.260120] nix-grpc-daemon[785]: ts=2026-09-21T13:54:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:559704678client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4679worker1 # [ 42.275306] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4680worker1 # [ 42.289504] nix-grpc-daemon[785]: ts=2026-09-21T13:54:19Z level=info event=niks3_push_started pid=8854681worker1 # [ 42.593647] nix-grpc-daemon[885]: time=2026-09-21T13:54:19.726Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""4682worker1 # [ 42.814681] nix-daemon[870]: accepted connection from pid 897, user nix-grpc-daemon (trusted)4683worker1 # [ 42.830575] nix-daemon[870]: reaped child process 900, status = succeeded4684worker1 # [ 43.021147] nix-daemon[870]: accepted connection from pid 902, user nix-grpc-daemon (trusted)4685worker1 # [ 43.032831] nix-daemon[870]: reaped child process 905, status = succeeded4686lb # [ 43.056740] niks3-server[1011]: 2026/09/21 13:54:19 INFO Received uploads request method=POST path=/api/pending_closures4687worker1 # [ 43.079407] nix-grpc-daemon[885]: time=2026-09-21T13:54:20.215Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4688worker1 # [ 43.081707] nix-grpc-daemon[885]: time=2026-09-21T13:54:20.218Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4689lb # [ 43.128914] niks3-server[1011]: 2026/09/21 13:54:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4690lb # [ 43.148760] niks3-server[1011]: 2026/09/21 13:54:20 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4691lb # [ 43.159094] niks3-server[1011]: 2026/09/21 13:54:20 INFO Signed narinfos id=1 count=14692worker1 # [ 43.158685] nix-grpc-daemon[885]: time=2026-09-21T13:54:20.294Z level=INFO msg="Uploading 1 narinfos"4693lb # [ 43.167148] niks3-server[1011]: 2026/09/21 13:54:20 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4694lb # [ 43.173094] niks3-server[1011]: 2026/09/21 13:54:20 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4695lb # [ 43.183114] niks3-server[1011]: 2026/09/21 13:54:20 INFO Completed upload id=14696worker1 # [ 43.182464] nix-grpc-daemon[885]: time=2026-09-21T13:54:20.318Z level=INFO msg="Upload complete. (564ms)"4697lb # [ 43.187663] niks3-server[1011]: 2026/09/21 13:54:20 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4698worker1 # [ 43.187958] nix-grpc-daemon[785]: ts=2026-09-21T13:54:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=0 paths=1 nar_bytes_in=5044699worker1 # [ 43.195901] nix-grpc-daemon[785]: ts=2026-09-21T13:54:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559704700worker1 # [ 43.210053] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4701worker1 # [ 43.215201] nix-daemon[870]: reaped child process 883, status = succeeded4702client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4703client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4704worker1 # [ 43.224152] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4705client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4706client # farm-a> LOG-farm-a-t14707worker1 # [ 44.439457] nix-daemon[870]: reaped child process 909, status = succeeded4708worker1 # [ 44.607569] nix-daemon[870]: accepted connection from pid 925, user nix-grpc-daemon (trusted)4709worker1 # [ 44.619487] nix-daemon[870]: reaped child process 928, status = succeeded4710lb # [ 44.638236] niks3-server[1011]: 2026/09/21 13:54:21 INFO Received uploads request method=POST path=/api/pending_closures4711worker1 # [ 44.645099] nix-grpc-daemon[885]: time=2026-09-21T13:54:21.781Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4712worker1 # [ 44.646840] nix-grpc-daemon[885]: time=2026-09-21T13:54:21.781Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4713lb # [ 44.670349] niks3-server[1011]: 2026/09/21 13:54:21 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4714lb # [ 44.673101] niks3-server[1011]: 2026/09/21 13:54:21 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4715lb # [ 44.677419] niks3-server[1011]: 2026/09/21 13:54:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4716lb # [ 44.680430] niks3-server[1011]: 2026/09/21 13:54:21 INFO Signed narinfos id=2 count=14717worker1 # [ 44.680104] nix-grpc-daemon[885]: time=2026-09-21T13:54:21.816Z level=INFO msg="Uploading 1 narinfos"4718lb # [ 44.681764] niks3-server[1011]: 2026/09/21 13:54:21 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4719lb # [ 44.687445] niks3-server[1011]: 2026/09/21 13:54:21 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4720lb # [ 44.692963] niks3-server[1011]: 2026/09/21 13:54:21 INFO Completed upload id=24721worker1 # [ 44.692126] nix-grpc-daemon[885]: time=2026-09-21T13:54:21.828Z level=INFO msg="Upload complete. (257ms)"4722lb # [ 44.695088] niks3-server[1011]: 2026/09/21 13:54:21 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4723worker1 # [ 44.695179] nix-grpc-daemon[785]: ts=2026-09-21T13:54:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14724worker1 # [ 44.704561] nix-daemon[870]: reaped child process 907, status = succeeded4725worker1 # [ 44.712591] nix-grpc-daemon[785]: ts=2026-09-21T13:54:21Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04726worker1 # [ 44.714261] nix-grpc-daemon[785]: ts=2026-09-21T13:54:21Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14727worker1 # [ 44.760895] nix-grpc-daemon[785]: ts=2026-09-21T13:54:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559704728worker1 # [ 44.773743] nix-grpc-daemon[785]: ts=2026-09-21T13:54:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559704729worker1 # [ 44.785258] nix-grpc-daemon[885]: time=2026-09-21T13:54:21.921Z level=INFO msg="All 1 paths already cached"4730client # copying 1 paths to 'lb:50051'...4731worker1 # [ 44.793137] nix-grpc-daemon[785]: ts=2026-09-21T13:54:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:559704732client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4733worker1 # [ 44.811281] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4734worker1 # [ 44.988564] nix-daemon[870]: accepted connection from pid 934, user nix-grpc-daemon (trusted)4735worker1 # [ 45.001616] nix-daemon[870]: reaped child process 937, status = succeeded4736worker1 # [ 45.165884] nix-daemon[870]: accepted connection from pid 940, user nix-grpc-daemon (trusted)4737worker1 # [ 45.177624] nix-daemon[870]: reaped child process 943, status = succeeded4738lb # [ 45.195607] niks3-server[1011]: 2026/09/21 13:54:22 INFO Received uploads request method=POST path=/api/pending_closures4739worker1 # [ 45.205421] nix-grpc-daemon[885]: time=2026-09-21T13:54:22.341Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4740worker1 # [ 45.206916] nix-grpc-daemon[885]: time=2026-09-21T13:54:22.343Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4741lb # [ 45.225489] niks3-server[1011]: 2026/09/21 13:54:22 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4742lb # [ 45.237612] niks3-server[1011]: 2026/09/21 13:54:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4743worker1 # [ 45.243540] nix-grpc-daemon[885]: time=2026-09-21T13:54:22.380Z level=INFO msg="Uploading 1 narinfos"4744lb # [ 45.244408] niks3-server[1011]: 2026/09/21 13:54:22 INFO Signed narinfos id=3 count=14745lb # [ 45.248345] niks3-server[1011]: 2026/09/21 13:54:22 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4746lb # [ 45.259653] niks3-server[1011]: 2026/09/21 13:54:22 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4747lb # [ 45.267404] niks3-server[1011]: 2026/09/21 13:54:22 INFO Completed upload id=34748worker1 # [ 45.266445] nix-grpc-daemon[885]: time=2026-09-21T13:54:22.402Z level=INFO msg="Upload complete. (441ms)"4749lb # [ 45.269913] niks3-server[1011]: 2026/09/21 13:54:22 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4750worker1 # [ 45.270159] nix-grpc-daemon[785]: ts=2026-09-21T13:54:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=0 paths=1 nar_bytes_in=6244751worker1 # [ 45.275240] nix-grpc-daemon[785]: ts=2026-09-21T13:54:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559704752worker1 # [ 45.286132] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4753worker1 # [ 45.289304] nix-daemon[870]: reaped child process 932, status = succeeded4754worker1 # [ 45.294949] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4755client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4756client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4757client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4758client # farm-b> LOG-farm-b-t14759lb # [ 46.488702] envoy[901]: conn peer=[2001:db8:1::1]:55262 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=407 ms=22164760lb # [ 46.492424] envoy[901]: conn peer=[2001:db8:1::1]:55270 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=624761lb # [ 46.495825] envoy[901]: conn peer=[2001:db8:1::1]:55278 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=134762lb # [ 46.499141] envoy[901]: conn peer=[2001:db8:1::1]:55290 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=114763worker1 # [ 46.509475] nix-daemon[870]: reaped child process 947, status = succeeded4764worker1 # [ 46.685342] nix-daemon[870]: accepted connection from pid 963, user nix-grpc-daemon (trusted)4765worker1 # [ 46.695947] nix-daemon[870]: reaped child process 966, status = succeeded4766lb # [ 46.716958] niks3-server[1011]: 2026/09/21 13:54:23 INFO Received uploads request method=POST path=/api/pending_closures4767worker1 # [ 46.725817] nix-grpc-daemon[885]: time=2026-09-21T13:54:23.862Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4768worker1 # [ 46.727291] nix-grpc-daemon[885]: time=2026-09-21T13:54:23.863Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4769lb # [ 46.760917] niks3-server[1011]: 2026/09/21 13:54:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4770worker1 # [ 46.763761] nix-grpc-daemon[885]: time=2026-09-21T13:54:23.900Z level=INFO msg="Uploading 1 narinfos"4771lb # [ 46.765553] niks3-server[1011]: 2026/09/21 13:54:23 INFO Signed narinfos id=4 count=14772lb # [ 46.773286] niks3-server[1011]: 2026/09/21 13:54:23 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4773lb # [ 46.781107] niks3-server[1011]: 2026/09/21 13:54:23 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4774lb # [ 46.788208] niks3-server[1011]: 2026/09/21 13:54:23 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4775lb # [ 46.794790] niks3-server[1011]: 2026/09/21 13:54:23 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4776lb # [ 46.799444] niks3-server[1011]: 2026/09/21 13:54:23 INFO Completed upload id=44777lb # [ 46.802190] niks3-server[1011]: 2026/09/21 13:54:23 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4778worker1 # [ 46.800186] nix-grpc-daemon[885]: time=2026-09-21T13:54:23.936Z level=INFO msg="Upload complete. (298ms)"4779worker1 # [ 46.807878] nix-grpc-daemon[785]: ts=2026-09-21T13:54:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14780worker1 # [ 46.818840] nix-grpc-daemon[785]: ts=2026-09-21T13:54:23Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04781worker1 # [ 46.821870] nix-grpc-daemon[785]: ts=2026-09-21T13:54:23Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14782worker1 # [ 46.830870] nix-daemon[870]: reaped child process 945, status = succeeded4783worker1 # [ 46.861962] nix-grpc-daemon[785]: ts=2026-09-21T13:54:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559704784worker1 # [ 46.874979] nix-grpc-daemon[785]: ts=2026-09-21T13:54:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559704785worker1 # [ 46.887519] nix-grpc-daemon[885]: time=2026-09-21T13:54:24.024Z level=INFO msg="All 2 paths already cached"4786client # copying 1 paths to 'lb:50051'...4787client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4788worker1 # [ 46.896129] nix-grpc-daemon[785]: ts=2026-09-21T13:54:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:559704789worker1 # [ 46.904900] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4790worker1 # [ 47.087456] nix-daemon[870]: accepted connection from pid 972, user nix-grpc-daemon (trusted)4791worker1 # [ 47.099195] nix-daemon[870]: reaped child process 975, status = succeeded4792worker1 # [ 47.288940] nix-daemon[870]: accepted connection from pid 977, user nix-grpc-daemon (trusted)4793worker1 # [ 47.301958] nix-daemon[870]: reaped child process 980, status = succeeded4794lb # [ 47.320170] niks3-server[1011]: 2026/09/21 13:54:24 INFO Received uploads request method=POST path=/api/pending_closures4795worker1 # [ 47.326947] nix-grpc-daemon[885]: time=2026-09-21T13:54:24.463Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4796worker1 # [ 47.328482] nix-grpc-daemon[885]: time=2026-09-21T13:54:24.465Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4797lb # [ 47.349131] niks3-server[1011]: 2026/09/21 13:54:24 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4798lb # [ 47.357301] niks3-server[1011]: 2026/09/21 13:54:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4799lb # [ 47.361726] niks3-server[1011]: 2026/09/21 13:54:24 INFO Signed narinfos id=5 count=14800worker1 # [ 47.363822] nix-grpc-daemon[885]: time=2026-09-21T13:54:24.500Z level=INFO msg="Uploading 1 narinfos"4801lb # [ 47.374334] niks3-server[1011]: 2026/09/21 13:54:24 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4802lb # [ 47.380981] niks3-server[1011]: 2026/09/21 13:54:24 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4803lb # [ 47.386332] niks3-server[1011]: 2026/09/21 13:54:24 INFO Completed upload id=54804worker1 # [ 47.385517] nix-grpc-daemon[885]: time=2026-09-21T13:54:24.522Z level=INFO msg="Upload complete. (468ms)"4805lb # [ 47.390763] niks3-server[1011]: 2026/09/21 13:54:24 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4806worker1 # [ 47.389916] nix-grpc-daemon[785]: ts=2026-09-21T13:54:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=0 paths=1 nar_bytes_in=7524807worker1 # [ 47.394314] nix-grpc-daemon[785]: ts=2026-09-21T13:54:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559704808worker1 # [ 47.402371] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4809worker1 # [ 47.406975] nix-daemon[870]: reaped child process 970, status = succeeded4810worker1 # [ 47.412051] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4811client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4812client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4813client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4814client # farm-top> LOG-farm-top-t14815worker1 # [ 48.837946] nix-daemon[870]: reaped child process 984, status = succeeded4816worker1 # [ 49.023086] nix-daemon[870]: accepted connection from pid 1000, user nix-grpc-daemon (trusted)4817worker1 # [ 49.033798] nix-daemon[870]: reaped child process 1003, status = succeeded4818lb # [ 49.056767] niks3-server[1011]: 2026/09/21 13:54:25 INFO Received uploads request method=POST path=/api/pending_closures4819worker1 # [ 49.064129] nix-grpc-daemon[885]: time=2026-09-21T13:54:26.200Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4820worker1 # [ 49.065662] nix-grpc-daemon[885]: time=2026-09-21T13:54:26.202Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4821lb # [ 49.090941] niks3-server[1011]: 2026/09/21 13:54:25 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4822lb # [ 49.102260] niks3-server[1011]: 2026/09/21 13:54:26 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4823lb # [ 49.116594] niks3-server[1011]: 2026/09/21 13:54:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4824lb # [ 49.121148] niks3-server[1011]: 2026/09/21 13:54:26 INFO Signed narinfos id=6 count=14825worker1 # [ 49.120689] nix-grpc-daemon[885]: time=2026-09-21T13:54:26.256Z level=INFO msg="Uploading 1 narinfos"4826lb # [ 49.126835] niks3-server[1011]: 2026/09/21 13:54:26 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4827lb # [ 49.136738] niks3-server[1011]: 2026/09/21 13:54:26 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4828lb # [ 49.139518] niks3-server[1011]: 2026/09/21 13:54:26 INFO Completed upload id=64829lb # [ 49.141066] niks3-server[1011]: 2026/09/21 13:54:26 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4830worker1 # [ 49.138883] nix-grpc-daemon[885]: time=2026-09-21T13:54:26.275Z level=INFO msg="Upload complete. (316ms)"4831worker1 # [ 49.147666] nix-grpc-daemon[785]: ts=2026-09-21T13:54:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14832worker1 # [ 49.159815] nix-daemon[870]: reaped child process 982, status = succeeded4833client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 7.33 seconds)4834client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4835client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.25 seconds)4836client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4837client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.02 seconds)4838client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4839client # copying 3 paths...4840client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4841client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4842client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4843client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.31 seconds)4844(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 8.00 seconds)4845subtest: repeat is answered Cached by the scheduler without building4846worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4847worker1 # finding garbage collector roots...4848worker1 # deleting specified paths...4849worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4850worker1 # deleting unused links...4851worker1 # note: hard linking is currently saving 0.0 KiB4852worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.30 seconds)4853worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4854worker2 # finding garbage collector roots...4855worker2 # deleting specified paths...4856worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4857worker2 # deleting unused links...4858worker2 # note: hard linking is currently saving 0.0 KiB4859worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.28 seconds)4860client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24861worker1 # [ 50.695869] nix-grpc-daemon[785]: ts=2026-09-21T13:54:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:559704862client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4863client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4864worker1 # [ 50.717077] nix-grpc-daemon[785]: ts=2026-09-21T13:54:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559704865worker1 # [ 50.728658] nix-grpc-daemon[885]: time=2026-09-21T13:54:27.864Z level=INFO msg="All 2 paths already cached"4866worker1 # [ 50.744629] nix-grpc-daemon[785]: ts=2026-09-21T13:54:27Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14867client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.34 seconds)4868client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4869client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.23 seconds)4870lb: must succeed: curl -sf localhost:9901/clusters4871lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4872lb: must succeed: curl -sf localhost:9901/clusters4873lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4874worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4875worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4876(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.44 seconds)4877subtest: inputs already in the cache are not sent to the scheduler4878lb: must succeed: curl -sf localhost:9901/clusters4879lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4880lb: must succeed: curl -sf localhost:9901/clusters4881lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4882worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4883worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4884client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24885client # don't know how to build these paths:4886client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4887worker1 # [ 51.729788] nix-grpc-daemon[785]: ts=2026-09-21T13:54:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:559704888worker1 # [ 51.736575] nix-grpc-daemon[785]: ts=2026-09-21T13:54:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559704889worker1 # [ 51.742552] nix-grpc-daemon[885]: time=2026-09-21T13:54:28.877Z level=INFO msg="All 2 paths already cached"4890worker1 # [ 51.753777] nix-grpc-daemon[785]: ts=2026-09-21T13:54:28Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04891worker1 # [ 51.759640] nix-grpc-daemon[785]: ts=2026-09-21T13:54:28Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14892worker1 # [ 51.797538] nix-grpc-daemon[785]: ts=2026-09-21T13:54:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559704893worker1 # [ 51.811290] nix-grpc-daemon[785]: ts=2026-09-21T13:54:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559704894worker1 # [ 51.824191] nix-grpc-daemon[885]: time=2026-09-21T13:54:28.960Z level=INFO msg="All 2 paths already cached"4895client # copying 1 paths to 'lb:50051'...4896client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4897worker1 # [ 51.832707] nix-grpc-daemon[785]: ts=2026-09-21T13:54:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:559704898worker1 # [ 51.845076] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4899worker1 # [ 52.039114] nix-daemon[870]: accepted connection from pid 1032, user nix-grpc-daemon (trusted)4900worker1 # [ 52.052300] nix-daemon[870]: reaped child process 1035, status = succeeded4901worker1 # [ 52.232414] nix-daemon[870]: accepted connection from pid 1037, user nix-grpc-daemon (trusted)4902worker1 # [ 52.244456] nix-daemon[870]: reaped child process 1040, status = succeeded4903lb # [ 52.265203] niks3-server[1011]: 2026/09/21 13:54:29 INFO Received uploads request method=POST path=/api/pending_closures4904worker1 # [ 52.272734] nix-grpc-daemon[885]: time=2026-09-21T13:54:29.409Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4905worker1 # [ 52.274178] nix-grpc-daemon[885]: time=2026-09-21T13:54:29.409Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4906lb # [ 52.289439] niks3-server[1011]: 2026/09/21 13:54:29 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4907lb # [ 52.301529] niks3-server[1011]: 2026/09/21 13:54:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4908worker1 # [ 52.307441] nix-grpc-daemon[885]: time=2026-09-21T13:54:29.443Z level=INFO msg="Uploading 1 narinfos"4909lb # [ 52.308586] niks3-server[1011]: 2026/09/21 13:54:29 INFO Signed narinfos id=7 count=14910lb # [ 52.313409] niks3-server[1011]: 2026/09/21 13:54:29 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4911lb # [ 52.324839] niks3-server[1011]: 2026/09/21 13:54:29 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4912lb # [ 52.331389] niks3-server[1011]: 2026/09/21 13:54:29 INFO Completed upload id=74913worker1 # [ 52.330285] nix-grpc-daemon[885]: time=2026-09-21T13:54:29.466Z level=INFO msg="Upload complete. (472ms)"4914lb # [ 52.333978] niks3-server[1011]: 2026/09/21 13:54:29 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4915worker1 # [ 52.333791] nix-grpc-daemon[785]: ts=2026-09-21T13:54:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=0 paths=1 nar_bytes_in=7524916worker1 # [ 52.339704] nix-grpc-daemon[785]: ts=2026-09-21T13:54:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559704917worker1 # [ 52.349564] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4918worker1 # [ 52.352277] nix-daemon[870]: reaped child process 1030, status = succeeded4919worker1 # [ 52.357574] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4920client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4921client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4922client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4923client # farm-top> LOG-farm-top-t1b4924worker1 # [ 53.170845] nix-daemon[870]: reaped child process 1044, status = succeeded4925worker1 # [ 53.331621] nix-daemon[870]: accepted connection from pid 1060, user nix-grpc-daemon (trusted)4926worker1 # [ 53.343832] nix-daemon[870]: reaped child process 1063, status = succeeded4927lb # [ 53.366492] niks3-server[1011]: 2026/09/21 13:54:30 INFO Received uploads request method=POST path=/api/pending_closures4928worker1 # [ 53.374565] nix-grpc-daemon[885]: time=2026-09-21T13:54:30.510Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4929worker1 # [ 53.376139] nix-grpc-daemon[885]: time=2026-09-21T13:54:30.512Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4930lb # [ 53.405599] niks3-server[1011]: 2026/09/21 13:54:30 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4931lb # [ 53.413130] niks3-server[1011]: 2026/09/21 13:54:30 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4932lb # [ 53.425886] niks3-server[1011]: 2026/09/21 13:54:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4933lb # [ 53.431093] niks3-server[1011]: 2026/09/21 13:54:30 INFO Signed narinfos id=8 count=14934worker1 # [ 53.432867] nix-grpc-daemon[885]: time=2026-09-21T13:54:30.569Z level=INFO msg="Uploading 1 narinfos"4935lb # [ 53.437115] niks3-server[1011]: 2026/09/21 13:54:30 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4936lb # [ 53.446082] niks3-server[1011]: 2026/09/21 13:54:30 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4937lb # [ 53.451063] niks3-server[1011]: 2026/09/21 13:54:30 INFO Completed upload id=84938worker1 # [ 53.450358] nix-grpc-daemon[885]: time=2026-09-21T13:54:30.586Z level=INFO msg="Upload complete. (286ms)"4939lb # [ 53.453394] niks3-server[1011]: 2026/09/21 13:54:30 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4940worker1 # [ 53.457975] nix-grpc-daemon[785]: ts=2026-09-21T13:54:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14941worker1 # [ 53.472647] nix-daemon[870]: reaped child process 1042, status = succeeded4942client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.04 seconds)4943client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4944client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.22 seconds)4945lb: must succeed: curl -sf localhost:9901/clusters4946lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4947lb: must succeed: curl -sf localhost:9901/clusters4948lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4949worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4950worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4951(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.58 seconds)4952subtest: two clients wanting the same drv build it once4953client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4954client # [ 53.938899] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4955client # Running as unit: dedup1.service; invocation ID: 7c5d60973a274d60928a18f685509ec74956client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.06 seconds)4957client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4958worker2 # [ 54.191429] nix-grpc-daemon[783]: ts=2026-09-21T13:54:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:447084959client # [ 54.211524] nix[891]: don't know how to build these paths:4960client # [ 54.212450] nix[891]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4961worker1 # [ 54.214697] nix-grpc-daemon[785]: ts=2026-09-21T13:54:31Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04962worker1 # [ 54.219439] nix-grpc-daemon[785]: ts=2026-09-21T13:54:31Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=14963worker2 # [ 54.252767] nix-grpc-daemon[783]: ts=2026-09-21T13:54:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447084964worker2 # [ 54.265175] nix-grpc-daemon[783]: ts=2026-09-21T13:54:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447084965client # [ 54.287212] nix[891]: copying 1 paths to 'lb:50051'...4966worker2 # [ 54.274707] nix-grpc-daemon[783]: ts=2026-09-21T13:54:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447084967client # [ 54.290169] nix[891]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4968worker2 # [ 54.288380] nix-daemon[853]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4969worker2 # [ 54.310435] nix-grpc-daemon[783]: ts=2026-09-21T13:54:30Z level=info event=niks3_push_started pid=8804970worker2 # [ 54.561631] nix-grpc-daemon[880]: time=2026-09-21T13:54:30.870Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""4971worker2 # [ 54.734599] nix-daemon[853]: accepted connection from pid 892, user nix-grpc-daemon (trusted)4972worker2 # [ 54.748434] nix-daemon[853]: reaped child process 895, status = succeeded4973worker2 # [ 54.890873] nix-daemon[853]: accepted connection from pid 897, user nix-grpc-daemon (trusted)4974worker2 # [ 54.903307] nix-daemon[853]: reaped child process 900, status = succeeded4975lb # [ 54.921738] niks3-server[1011]: 2026/09/21 13:54:31 INFO Received uploads request method=POST path=/api/pending_closures4976worker2 # [ 54.918964] nix-grpc-daemon[880]: time=2026-09-21T13:54:31.231Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4977worker2 # [ 54.921116] nix-grpc-daemon[880]: time=2026-09-21T13:54:31.233Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4978lb # [ 54.944252] niks3-server[1011]: 2026/09/21 13:54:31 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4979lb # [ 54.947027] niks3-server[1011]: 2026/09/21 13:54:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4980lb # [ 54.949055] niks3-server[1011]: 2026/09/21 13:54:31 INFO Signed narinfos id=9 count=14981worker2 # [ 54.941984] nix-grpc-daemon[880]: time=2026-09-21T13:54:31.254Z level=INFO msg="Uploading 1 narinfos"4982lb # [ 54.954762] niks3-server[1011]: 2026/09/21 13:54:31 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4983lb # [ 54.958051] niks3-server[1011]: 2026/09/21 13:54:31 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4984lb # [ 54.961377] niks3-server[1011]: 2026/09/21 13:54:31 INFO Completed upload id=94985worker2 # [ 54.954161] nix-grpc-daemon[880]: time=2026-09-21T13:54:31.266Z level=INFO msg="Upload complete. (378ms)"4986worker2 # [ 54.956173] nix-grpc-daemon[783]: ts=2026-09-21T13:54:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44708 duration_s=0 paths=1 nar_bytes_in=4324987lb # [ 54.964297] niks3-server[1011]: 2026/09/21 13:54:31 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4988worker2 # [ 54.959819] nix-grpc-daemon[783]: ts=2026-09-21T13:54:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447084989worker2 # [ 54.965568] nix-daemon[853]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4990worker2 # [ 54.968650] nix-daemon[853]: reaped child process 878, status = succeeded4991worker2 # [ 54.974990] nix-daemon[853]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4992client # [ 54.987323] nix[891]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4993client # [ 55.023631] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4994client # Running as unit: dedup2.service; invocation ID: 05062a44e5db4fc3a1593cc1c50f518d4995client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.10 seconds)4996worker1 # [ 55.339643] nix-grpc-daemon[785]: ts=2026-09-21T13:54:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:559704997client # [ 55.353050] nix[907]: don't know how to build these paths:4998client # [ 55.356067] nix[907]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4999worker1 # [ 55.359401] nix-grpc-daemon[785]: ts=2026-09-21T13:54:32Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05000worker2 # [ 55.357631] nix-grpc-daemon[783]: ts=2026-09-21T13:54:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447085001client # [ 55.379459] nix[907]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5002client: must succeed: sleep 25003lb # [ 56.489197] envoy[901]: conn peer=[2001:db8:1::1]:55294 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9498 tx=4041 ms=70585004lb # [ 56.492795] envoy[901]: conn peer=[2001:db8:1::1]:39162 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=476 ms=675005lb # [ 56.496247] envoy[901]: conn peer=[2001:db8:1::1]:39178 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4126 tx=1763 ms=17435006client: (finished: must succeed: sleep 2, in 2.06 seconds)5007client: waiting for success: ! systemctl is-active dedup1 dedup25008worker2 # [ 58.590822] nix-daemon[853]: reaped child process 904, status = succeeded5009worker2 # [ 58.738518] nix-daemon[853]: accepted connection from pid 935, user nix-grpc-daemon (trusted)5010worker2 # [ 58.750841] nix-daemon[853]: reaped child process 938, status = succeeded5011lb # [ 58.772720] niks3-server[1011]: 2026/09/21 13:54:35 INFO Received uploads request method=POST path=/api/pending_closures5012worker2 # [ 58.773753] nix-grpc-daemon[880]: time=2026-09-21T13:54:35.086Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5013worker2 # [ 58.775356] nix-grpc-daemon[880]: time=2026-09-21T13:54:35.087Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"5014lb # [ 58.806358] niks3-server[1011]: 2026/09/21 13:54:35 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5015lb # [ 58.823168] niks3-server[1011]: 2026/09/21 13:54:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign5016worker2 # [ 58.824579] nix-grpc-daemon[880]: time=2026-09-21T13:54:35.136Z level=INFO msg="Uploading 1 narinfos"5017lb # [ 58.832425] niks3-server[1011]: 2026/09/21 13:54:35 INFO Signed narinfos id=10 count=15018lb # [ 58.837875] niks3-server[1011]: 2026/09/21 13:54:35 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst5019lb # [ 58.842481] niks3-server[1011]: 2026/09/21 13:54:35 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls5020lb # [ 58.852451] niks3-server[1011]: 2026/09/21 13:54:35 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete5021lb # [ 58.856799] niks3-server[1011]: 2026/09/21 13:54:35 INFO Completed upload id=105022worker2 # [ 58.849693] nix-grpc-daemon[880]: time=2026-09-21T13:54:35.162Z level=INFO msg="Upload complete. (265ms)"5023lb # [ 58.858073] niks3-server[1011]: 2026/09/21 13:54:35 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo5024worker2 # [ 58.854731] nix-grpc-daemon[783]: ts=2026-09-21T13:54:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44708 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=15025worker2 # [ 58.860347] nix-grpc-daemon[783]: ts=2026-09-21T13:54:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44708 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=15026worker2 # [ 58.869949] nix-daemon[853]: reaped child process 902, status = succeeded5027client # [ 58.916455] systemd[1]: dedup2.service: Deactivated successfully.5028client # [ 58.921706] systemd[1]: dedup2.service: Consumed 162ms CPU time over 3.896s wall clock time, 35.2M memory peak, 5.9K incoming IP traffic, 5.7K outgoing IP traffic.5029client # [ 58.959111] systemd[1]: dedup1.service: Deactivated successfully.5030client # [ 58.960798] systemd[1]: dedup1.service: Consumed 139ms CPU time over 5.021s wall clock time, 43.5M memory peak, 7.3K incoming IP traffic, 8.5K outgoing IP traffic.5031client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.11 seconds)5032client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }5033client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.05 seconds)5034worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5035worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)5036worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5037worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)5038worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5039worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5040worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5041worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5042(finished: subtest: two clients wanting the same drv build it once, in 6.06 seconds)5043subtest: build hook: nix-daemon with builders = grpc://lb5044client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log5045client # [ 60.189351] systemd[1]: Started Nix Daemon.5046client # [ 60.406327] nix-daemon[949]: accepted connection from pid 946, user root (trusted)5047worker2 # [ 60.643690] nix-grpc-daemon[783]: ts=2026-09-21T13:54:36Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:447085048worker1 # [ 60.654466] nix-grpc-daemon[785]: ts=2026-09-21T13:54:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559705049worker1 # [ 60.671777] nix-grpc-daemon[785]: ts=2026-09-21T13:54:37Z level=debug event=want client=6 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05050worker1 # [ 60.676838] nix-grpc-daemon[785]: ts=2026-09-21T13:54:37Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=15051worker1 # [ 60.715492] nix-grpc-daemon[785]: ts=2026-09-21T13:54:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559705052worker1 # [ 60.725869] nix-grpc-daemon[785]: ts=2026-09-21T13:54:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559705053worker1 # [ 60.735376] nix-grpc-daemon[785]: ts=2026-09-21T13:54:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:559705054worker1 # [ 60.746321] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5055worker1 # [ 60.947960] nix-daemon[870]: accepted connection from pid 1099, user nix-grpc-daemon (trusted)5056worker1 # [ 60.960603] nix-daemon[870]: reaped child process 1102, status = succeeded5057worker1 # [ 61.152747] nix-daemon[870]: accepted connection from pid 1104, user nix-grpc-daemon (trusted)5058worker1 # [ 61.164640] nix-daemon[870]: reaped child process 1107, status = succeeded5059lb # [ 61.185735] niks3-server[1011]: 2026/09/21 13:54:38 INFO Received uploads request method=POST path=/api/pending_closures5060worker1 # [ 61.195117] nix-grpc-daemon[885]: time=2026-09-21T13:54:38.331Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5061worker1 # [ 61.196487] nix-grpc-daemon[885]: time=2026-09-21T13:54:38.331Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5062lb # [ 61.216140] niks3-server[1011]: 2026/09/21 13:54:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign5063lb # [ 61.222083] niks3-server[1011]: 2026/09/21 13:54:38 INFO Signed narinfos id=11 count=15064worker1 # [ 61.224832] nix-grpc-daemon[885]: time=2026-09-21T13:54:38.361Z level=INFO msg="Uploading 1 narinfos"5065lb # [ 61.232508] niks3-server[1011]: 2026/09/21 13:54:38 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5066lb # [ 61.237890] niks3-server[1011]: 2026/09/21 13:54:38 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5067lb # [ 61.250121] niks3-server[1011]: 2026/09/21 13:54:38 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete5068lb # [ 61.256813] niks3-server[1011]: 2026/09/21 13:54:38 INFO Completed upload id=115069worker1 # [ 61.255916] nix-grpc-daemon[885]: time=2026-09-21T13:54:38.392Z level=INFO msg="Upload complete. (497ms)"5070lb # [ 61.260219] niks3-server[1011]: 2026/09/21 13:54:38 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo5071worker1 # [ 61.260129] nix-grpc-daemon[785]: ts=2026-09-21T13:54:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=0 paths=1 nar_bytes_in=5125072worker1 # [ 61.264877] nix-grpc-daemon[785]: ts=2026-09-21T13:54:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559705073worker1 # [ 61.273840] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5074worker1 # [ 61.282120] nix-daemon[870]: reaped child process 1097, status = succeeded5075worker1 # [ 61.285426] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5076worker1 # [ 62.714811] nix-daemon[870]: reaped child process 1111, status = succeeded5077worker1 # [ 62.902371] nix-daemon[870]: accepted connection from pid 1127, user nix-grpc-daemon (trusted)5078worker1 # [ 62.913033] nix-daemon[870]: reaped child process 1130, status = succeeded5079lb # [ 62.933400] niks3-server[1011]: 2026/09/21 13:54:39 INFO Received uploads request method=POST path=/api/pending_closures5080worker1 # [ 62.940110] nix-grpc-daemon[885]: time=2026-09-21T13:54:40.076Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5081worker1 # [ 62.942031] nix-grpc-daemon[885]: time=2026-09-21T13:54:40.078Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5082lb # [ 62.970136] niks3-server[1011]: 2026/09/21 13:54:39 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5083lb # [ 62.981811] niks3-server[1011]: 2026/09/21 13:54:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign5084worker1 # [ 62.989892] nix-grpc-daemon[885]: time=2026-09-21T13:54:40.126Z level=INFO msg="Uploading 1 narinfos"5085lb # [ 62.990869] niks3-server[1011]: 2026/09/21 13:54:39 INFO Signed narinfos id=12 count=15086lb # [ 62.995963] niks3-server[1011]: 2026/09/21 13:54:39 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5087lb # [ 63.000901] niks3-server[1011]: 2026/09/21 13:54:39 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5088lb # [ 63.010926] niks3-server[1011]: 2026/09/21 13:54:39 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete5089lb # [ 63.016317] niks3-server[1011]: 2026/09/21 13:54:39 INFO Completed upload id=125090lb # [ 63.019128] niks3-server[1011]: 2026/09/21 13:54:39 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5091worker1 # [ 63.017147] nix-grpc-daemon[885]: time=2026-09-21T13:54:40.153Z level=INFO msg="Upload complete. (323ms)"5092worker1 # [ 63.025496] nix-grpc-daemon[785]: ts=2026-09-21T13:54:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=6 outputs=15093worker2 # [ 63.023457] nix-grpc-daemon[783]: ts=2026-09-21T13:54:39Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:447085094worker1 # [ 63.038610] nix-daemon[870]: reaped child process 1109, status = succeeded5095worker2 # [ 63.034232] nix-grpc-daemon[783]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5096worker2 # [ 63.063363] nix-grpc-daemon[783]: ts=2026-09-21T13:54:39Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:447085097worker2 # [ 63.069552] nix-grpc-daemon[783]: ts=2026-09-21T13:54:39Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:44708 duration_s=0 paths=1 nar_bytes_out=1285098worker2 # [ 63.367692] nix-grpc-daemon[783]: ts=2026-09-21T13:54:39Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:447085099worker1 # [ 63.377323] nix-grpc-daemon[785]: ts=2026-09-21T13:54:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559705100worker1 # [ 63.387742] nix-grpc-daemon[885]: time=2026-09-21T13:54:40.524Z level=INFO msg="All 1 paths already cached"5101worker1 # [ 63.403142] nix-grpc-daemon[785]: ts=2026-09-21T13:54:40Z level=debug event=want client=7 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05102worker1 # [ 63.409827] nix-grpc-daemon[785]: ts=2026-09-21T13:54:40Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=15103worker1 # [ 63.414923] nix-grpc-daemon[785]: ts=2026-09-21T13:54:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559705104worker1 # [ 63.425987] nix-grpc-daemon[785]: ts=2026-09-21T13:54:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559705105worker1 # [ 63.438702] nix-grpc-daemon[885]: time=2026-09-21T13:54:40.575Z level=INFO msg="All 1 paths already cached"5106worker1 # [ 63.446151] nix-grpc-daemon[785]: ts=2026-09-21T13:54:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:559705107worker1 # [ 63.456181] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5108worker1 # [ 63.656426] nix-daemon[870]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5109worker1 # [ 63.669273] nix-daemon[870]: reaped child process 1139, status = succeeded5110worker1 # [ 63.856506] nix-daemon[870]: accepted connection from pid 1141, user nix-grpc-daemon (trusted)5111worker1 # [ 63.868162] nix-daemon[870]: reaped child process 1144, status = succeeded5112lb # [ 63.888601] niks3-server[1011]: 2026/09/21 13:54:40 INFO Received uploads request method=POST path=/api/pending_closures5113worker1 # [ 63.896352] nix-grpc-daemon[885]: time=2026-09-21T13:54:41.032Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5114worker1 # [ 63.897888] nix-grpc-daemon[885]: time=2026-09-21T13:54:41.034Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5115lb # [ 63.919202] niks3-server[1011]: 2026/09/21 13:54:40 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5116lb # [ 63.930159] niks3-server[1011]: 2026/09/21 13:54:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign5117lb # [ 63.935680] niks3-server[1011]: 2026/09/21 13:54:40 INFO Signed narinfos id=13 count=15118worker1 # [ 63.937642] nix-grpc-daemon[885]: time=2026-09-21T13:54:41.073Z level=INFO msg="Uploading 1 narinfos"5119lb # [ 63.942810] niks3-server[1011]: 2026/09/21 13:54:40 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5120lb # [ 63.952460] niks3-server[1011]: 2026/09/21 13:54:40 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete5121worker1 # [ 63.958616] nix-grpc-daemon[885]: time=2026-09-21T13:54:41.094Z level=INFO msg="Upload complete. (488ms)"5122lb # [ 63.959354] niks3-server[1011]: 2026/09/21 13:54:40 INFO Completed upload id=135123worker1 # [ 63.961243] nix-grpc-daemon[785]: ts=2026-09-21T13:54:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=0 paths=1 nar_bytes_in=6245124lb # [ 63.964510] niks3-server[1011]: 2026/09/21 13:54:40 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5125worker1 # [ 63.967710] nix-grpc-daemon[785]: ts=2026-09-21T13:54:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559705126worker1 # [ 63.977422] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5127worker1 # [ 63.980088] nix-daemon[870]: reaped child process 1134, status = succeeded5128worker1 # [ 63.985162] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5129worker1 # [ 64.996978] nix-daemon[870]: reaped child process 1148, status = succeeded5130worker1 # [ 65.165570] nix-daemon[870]: accepted connection from pid 1164, user nix-grpc-daemon (trusted)5131worker1 # [ 65.177620] nix-daemon[870]: reaped child process 1167, status = succeeded5132lb # [ 65.193699] niks3-server[1011]: 2026/09/21 13:54:42 INFO Received uploads request method=POST path=/api/pending_closures5133worker1 # [ 65.201066] nix-grpc-daemon[885]: time=2026-09-21T13:54:42.337Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5134worker1 # [ 65.202496] nix-grpc-daemon[885]: time=2026-09-21T13:54:42.339Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5135lb # [ 65.235372] niks3-server[1011]: 2026/09/21 13:54:42 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5136lb # [ 65.243911] niks3-server[1011]: 2026/09/21 13:54:42 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5137lb # [ 65.251062] niks3-server[1011]: 2026/09/21 13:54:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign5138lb # [ 65.256914] niks3-server[1011]: 2026/09/21 13:54:42 INFO Signed narinfos id=14 count=15139worker1 # [ 65.259410] nix-grpc-daemon[885]: time=2026-09-21T13:54:42.395Z level=INFO msg="Uploading 1 narinfos"5140lb # [ 65.261894] niks3-server[1011]: 2026/09/21 13:54:42 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5141lb # [ 65.272211] niks3-server[1011]: 2026/09/21 13:54:42 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete5142lb # [ 65.279212] niks3-server[1011]: 2026/09/21 13:54:42 INFO Completed upload id=145143worker1 # [ 65.278182] nix-grpc-daemon[885]: time=2026-09-21T13:54:42.414Z level=INFO msg="Upload complete. (286ms)"5144lb # [ 65.280981] niks3-server[1011]: 2026/09/21 13:54:42 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5145worker1 # [ 65.284361] nix-grpc-daemon[785]: ts=2026-09-21T13:54:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=7 outputs=15146worker1 # [ 65.294943] nix-grpc-daemon[785]: ts=2026-09-21T13:54:42Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:559705147worker1 # [ 65.302234] nix-daemon[870]: reaped child process 1146, status = succeeded5148worker2 # [ 65.307864] nix-grpc-daemon[783]: ts=2026-09-21T13:54:41Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:447085149worker2 # [ 65.320903] nix-grpc-daemon[783]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...5150worker2 # [ 65.334548] nix-grpc-daemon[783]: ts=2026-09-21T13:54:41Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:44708 duration_s=0 paths=1 nar_bytes_out=1765151worker1 # [ 65.709218] nix-grpc-daemon[785]: ts=2026-09-21T13:54:42Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:559705152worker2 # [ 65.706688] nix-grpc-daemon[783]: ts=2026-09-21T13:54:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447085153worker2 # [ 65.718207] nix-grpc-daemon[880]: time=2026-09-21T13:54:42.030Z level=INFO msg="All 2 paths already cached"5154worker1 # [ 65.740095] nix-grpc-daemon[785]: ts=2026-09-21T13:54:42Z level=debug event=want client=8 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05155worker1 # [ 65.745501] nix-grpc-daemon[785]: ts=2026-09-21T13:54:42Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=15156worker1 # [ 65.785814] nix-grpc-daemon[785]: ts=2026-09-21T13:54:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559705157worker1 # [ 65.799477] nix-grpc-daemon[785]: ts=2026-09-21T13:54:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559705158worker1 # [ 65.813499] nix-grpc-daemon[885]: time=2026-09-21T13:54:42.949Z level=INFO msg="All 2 paths already cached"5159worker1 # [ 65.821220] nix-grpc-daemon[785]: ts=2026-09-21T13:54:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:559705160worker1 # [ 65.831163] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5161worker1 # [ 66.035511] nix-daemon[870]: accepted connection from pid 1173, user nix-grpc-daemon (trusted)5162worker1 # [ 66.048210] nix-daemon[870]: reaped child process 1176, status = succeeded5163worker1 # [ 66.218812] nix-daemon[870]: accepted connection from pid 1178, user nix-grpc-daemon (trusted)5164worker1 # [ 66.230726] nix-daemon[870]: reaped child process 1181, status = succeeded5165lb # [ 66.252642] niks3-server[1011]: 2026/09/21 13:54:43 INFO Received uploads request method=POST path=/api/pending_closures5166worker1 # [ 66.261223] nix-grpc-daemon[885]: time=2026-09-21T13:54:43.397Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5167worker1 # [ 66.262812] nix-grpc-daemon[885]: time=2026-09-21T13:54:43.399Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5168lb # [ 66.283454] niks3-server[1011]: 2026/09/21 13:54:43 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5169lb # [ 66.295621] niks3-server[1011]: 2026/09/21 13:54:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5170worker1 # [ 66.299922] nix-grpc-daemon[885]: time=2026-09-21T13:54:43.436Z level=INFO msg="Uploading 1 narinfos"5171lb # [ 66.300796] niks3-server[1011]: 2026/09/21 13:54:43 INFO Signed narinfos id=15 count=15172lb # [ 66.306240] niks3-server[1011]: 2026/09/21 13:54:43 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5173lb # [ 66.315575] niks3-server[1011]: 2026/09/21 13:54:43 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5174lb # [ 66.318819] niks3-server[1011]: 2026/09/21 13:54:43 INFO Completed upload id=155175worker1 # [ 66.318977] nix-grpc-daemon[885]: time=2026-09-21T13:54:43.455Z level=INFO msg="Upload complete. (475ms)"5176lb # [ 66.321621] niks3-server[1011]: 2026/09/21 13:54:43 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5177worker1 # [ 66.323487] nix-grpc-daemon[785]: ts=2026-09-21T13:54:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=0 paths=1 nar_bytes_in=7525178worker1 # [ 66.328332] nix-grpc-daemon[785]: ts=2026-09-21T13:54:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559705179worker1 # [ 66.337275] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5180worker1 # [ 66.341524] nix-daemon[870]: reaped child process 1171, status = succeeded5181worker1 # [ 66.346516] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5182lb # [ 66.489619] envoy[901]: conn peer=[2001:db8:1::1]:57178 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1440 tx=1082 ms=35375183lb # [ 66.493577] envoy[901]: conn peer=[2001:db8:1::1]:57174 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3011 tx=1449 ms=47205184lb # [ 66.497459] envoy[901]: conn peer=[2001:db8:1::1]:53770 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=333 ms=585185lb # [ 66.501322] envoy[901]: conn peer=[2001:db8:1::1]:57180 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3420 tx=1825 ms=24775186lb # [ 66.504868] envoy[901]: conn peer=[2001:db8:1::1]:53780 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3893 tx=1954 ms=20205187lb # [ 66.508338] envoy[901]: conn peer=[2001:db8:1::1]:53792 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=379 ms=805188worker1 # [ 67.158550] nix-daemon[870]: reaped child process 1185, status = succeeded5189worker1 # [ 67.309807] nix-daemon[870]: accepted connection from pid 1201, user nix-grpc-daemon (trusted)5190worker1 # [ 67.321798] nix-daemon[870]: reaped child process 1204, status = succeeded5191lb # [ 67.342592] niks3-server[1011]: 2026/09/21 13:54:44 INFO Received uploads request method=POST path=/api/pending_closures5192worker1 # [ 67.350101] nix-grpc-daemon[885]: time=2026-09-21T13:54:44.486Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5193worker1 # [ 67.351701] nix-grpc-daemon[885]: time=2026-09-21T13:54:44.488Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5194lb # [ 67.387176] niks3-server[1011]: 2026/09/21 13:54:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5195lb # [ 67.391313] niks3-server[1011]: 2026/09/21 13:54:44 INFO Signed narinfos id=16 count=15196worker1 # [ 67.393576] nix-grpc-daemon[885]: time=2026-09-21T13:54:44.529Z level=INFO msg="Uploading 1 narinfos"5197lb # [ 67.403210] niks3-server[1011]: 2026/09/21 13:54:44 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5198lb # [ 67.408117] niks3-server[1011]: 2026/09/21 13:54:44 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5199lb # [ 67.414620] niks3-server[1011]: 2026/09/21 13:54:44 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5200lb # [ 67.421447] niks3-server[1011]: 2026/09/21 13:54:44 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5201lb # [ 67.430212] niks3-server[1011]: 2026/09/21 13:54:44 INFO Completed upload id=165202worker1 # [ 67.429608] nix-grpc-daemon[885]: time=2026-09-21T13:54:44.565Z level=INFO msg="Upload complete. (276ms)"5203lb # [ 67.432386] niks3-server[1011]: 2026/09/21 13:54:44 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5204worker1 # [ 67.435680] nix-grpc-daemon[785]: ts=2026-09-21T13:54:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=8 outputs=15205worker1 # [ 67.446198] nix-grpc-daemon[785]: ts=2026-09-21T13:54:44Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:559705206worker1 # [ 67.454605] nix-daemon[870]: reaped child process 1183, status = succeeded5207worker1 # [ 67.460361] nix-grpc-daemon[785]: ts=2026-09-21T13:54:44Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:559705208worker1 # [ 67.464920] nix-grpc-daemon[785]: ts=2026-09-21T13:54:44Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=0 paths=1 nar_bytes_out=2325209client # [ 67.577234] nix-daemon[949]: reaped child process 951, status = succeeded5210client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.65 seconds)5211client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5212client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.03 seconds)5213client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5214client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.04 seconds)5215client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5216client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)5217(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.75 seconds)5218subtest: an input only one worker has still reaches the builder5219client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only5220client # this derivation will be built:5221client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5222client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5223client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.50 seconds)5224client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5225client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.23 seconds)5226worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5227worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.26 seconds)5228worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5229worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.03 seconds)5230client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&25231client # [ 68.912134] nix-daemon[949]: accepted connection from pid 1032, user root (trusted)5232client # this derivation will be built:5233client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5234worker1 # [ 69.126823] nix-grpc-daemon[785]: ts=2026-09-21T13:54:46Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:559705235client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5236worker2 # [ 69.124531] nix-grpc-daemon[783]: ts=2026-09-21T13:54:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447085237client # copying 1 paths...5238client # copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' to 'grpc://lb:50051'...5239worker2 # [ 69.137449] nix-grpc-daemon[783]: ts=2026-09-21T13:54:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447085240worker2 # [ 69.149179] nix-daemon[853]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5241worker2 # [ 69.335799] nix-daemon[853]: accepted connection from pid 956, user nix-grpc-daemon (trusted)5242worker2 # [ 69.348797] nix-daemon[853]: reaped child process 959, status = succeeded5243lb # [ 69.370336] niks3-server[1011]: 2026/09/21 13:54:46 INFO Received uploads request method=POST path=/api/pending_closures5244worker2 # [ 69.370938] nix-grpc-daemon[880]: time=2026-09-21T13:54:45.683Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5245worker2 # [ 69.372940] nix-grpc-daemon[880]: time=2026-09-21T13:54:45.685Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5246lb # [ 69.397681] niks3-server[1011]: 2026/09/21 13:54:46 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5247lb # [ 69.408163] niks3-server[1011]: 2026/09/21 13:54:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5248lb # [ 69.413370] niks3-server[1011]: 2026/09/21 13:54:46 INFO Signed narinfos id=17 count=15249worker2 # [ 69.409499] nix-grpc-daemon[880]: time=2026-09-21T13:54:45.721Z level=INFO msg="Uploading 1 narinfos"5250lb # [ 69.420334] niks3-server[1011]: 2026/09/21 13:54:46 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5251lb # [ 69.431655] niks3-server[1011]: 2026/09/21 13:54:46 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5252lb # [ 69.437347] niks3-server[1011]: 2026/09/21 13:54:46 INFO Completed upload id=175253worker2 # [ 69.429966] nix-grpc-daemon[880]: time=2026-09-21T13:54:45.742Z level=INFO msg="Upload complete. (269ms)"5254lb # [ 69.440901] niks3-server[1011]: 2026/09/21 13:54:46 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5255worker2 # [ 69.433327] nix-grpc-daemon[783]: ts=2026-09-21T13:54:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44708 duration_s=0 paths=1 nar_bytes_in=1205256worker2 # [ 69.444582] nix-daemon[853]: reaped child process 954, status = succeeded5257worker1 # [ 69.454921] nix-grpc-daemon[785]: ts=2026-09-21T13:54:46Z level=debug event=want client=9 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05258worker1 # [ 69.459672] nix-grpc-daemon[785]: ts=2026-09-21T13:54:46Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=15259worker2 # [ 69.492761] nix-grpc-daemon[783]: ts=2026-09-21T13:54:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447085260worker2 # [ 69.506111] nix-grpc-daemon[783]: ts=2026-09-21T13:54:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447085261client # copying 2 paths to 'lb:50051'...5262client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5263worker2 # [ 69.523141] nix-grpc-daemon[783]: ts=2026-09-21T13:54:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447085264client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5265worker2 # [ 69.536172] nix-daemon[853]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5266worker2 # [ 69.714358] nix-daemon[853]: accepted connection from pid 963, user nix-grpc-daemon (trusted)5267worker2 # [ 69.727188] nix-daemon[853]: reaped child process 966, status = succeeded5268worker2 # [ 69.900308] nix-daemon[853]: accepted connection from pid 968, user nix-grpc-daemon (trusted)5269worker2 # [ 69.913090] nix-daemon[853]: reaped child process 971, status = succeeded5270lb # [ 69.938180] niks3-server[1011]: 2026/09/21 13:54:46 INFO Received uploads request method=POST path=/api/pending_closures5271lb # [ 69.946677] niks3-server[1011]: 2026/09/21 13:54:46 INFO Received uploads request method=POST path=/api/pending_closures5272worker2 # [ 69.949088] nix-grpc-daemon[880]: time=2026-09-21T13:54:46.260Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5273worker2 # [ 69.950654] nix-grpc-daemon[880]: time=2026-09-21T13:54:46.260Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5274worker2 # [ 69.952496] nix-grpc-daemon[880]: time=2026-09-21T13:54:46.260Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5275lb # [ 69.985255] niks3-server[1011]: 2026/09/21 13:54:46 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5276lb # [ 69.989403] niks3-server[1011]: 2026/09/21 13:54:46 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5277lb # [ 69.997280] niks3-server[1011]: 2026/09/21 13:54:46 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5278lb # [ 70.005301] niks3-server[1011]: 2026/09/21 13:54:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5279lb # [ 70.008981] niks3-server[1011]: 2026/09/21 13:54:46 INFO Signed narinfos id=18 count=15280lb # [ 70.012463] niks3-server[1011]: 2026/09/21 13:54:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5281lb # [ 70.014479] niks3-server[1011]: 2026/09/21 13:54:46 INFO Signed narinfos id=19 count=25282lb # [ 70.016328] niks3-server[1011]: 2026/09/21 13:54:46 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5283worker2 # [ 70.008323] nix-grpc-daemon[880]: time=2026-09-21T13:54:46.320Z level=INFO msg="Uploading 3 narinfos"5284lb # [ 70.029553] niks3-server[1011]: 2026/09/21 13:54:46 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5285lb # [ 70.031823] niks3-server[1011]: 2026/09/21 13:54:46 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5286lb # [ 70.035505] niks3-server[1011]: 2026/09/21 13:54:46 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5287lb # [ 70.038609] niks3-server[1011]: 2026/09/21 13:54:46 INFO Completed upload id=185288lb # [ 70.040377] niks3-server[1011]: 2026/09/21 13:54:46 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5289lb # [ 70.042886] niks3-server[1011]: 2026/09/21 13:54:46 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5290lb # [ 70.045077] niks3-server[1011]: 2026/09/21 13:54:46 INFO Completed upload id=195291worker2 # [ 70.038269] nix-grpc-daemon[880]: time=2026-09-21T13:54:46.350Z level=INFO msg="Upload complete. (486ms)"5292worker2 # [ 70.043171] nix-grpc-daemon[783]: ts=2026-09-21T13:54:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44708 duration_s=0 paths=2 nar_bytes_in=9765293worker2 # [ 70.049530] nix-grpc-daemon[783]: ts=2026-09-21T13:54:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447085294worker2 # [ 70.060488] nix-daemon[853]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5295worker2 # [ 70.072233] nix-daemon[853]: reaped child process 961, status = succeeded5296client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5297worker2 # [ 70.076482] nix-daemon[853]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5298client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5299client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5300worker2 # [ 71.101743] nix-daemon[853]: reaped child process 975, status = succeeded5301worker2 # [ 71.274332] nix-daemon[853]: accepted connection from pid 991, user nix-grpc-daemon (trusted)5302worker2 # [ 71.285619] nix-daemon[853]: reaped child process 994, status = succeeded5303lb # [ 71.311201] niks3-server[1011]: 2026/09/21 13:54:48 INFO Received uploads request method=POST path=/api/pending_closures5304worker2 # [ 71.312477] nix-grpc-daemon[880]: time=2026-09-21T13:54:47.624Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5305lb # [ 71.342451] niks3-server[1011]: 2026/09/21 13:54:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5306lb # [ 71.346674] niks3-server[1011]: 2026/09/21 13:54:48 INFO Signed narinfos id=20 count=15307worker2 # [ 71.342650] nix-grpc-daemon[880]: time=2026-09-21T13:54:47.655Z level=INFO msg="Uploading 1 narinfos"5308lb # [ 71.355356] niks3-server[1011]: 2026/09/21 13:54:48 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5309lb # [ 71.364428] niks3-server[1011]: 2026/09/21 13:54:48 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5310lb # [ 71.374184] niks3-server[1011]: 2026/09/21 13:54:48 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5311lb # [ 71.379126] niks3-server[1011]: 2026/09/21 13:54:48 INFO Completed upload id=205312worker2 # [ 71.374222] nix-grpc-daemon[880]: time=2026-09-21T13:54:47.686Z level=INFO msg="Upload complete. (289ms)"5313lb # [ 71.381775] niks3-server[1011]: 2026/09/21 13:54:48 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5314client # copying 1 paths...5315worker2 # [ 71.378964] nix-grpc-daemon[783]: ts=2026-09-21T13:54:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44708 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=9 outputs=15316worker1 # [ 71.393922] nix-grpc-daemon[785]: ts=2026-09-21T13:54:48Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:559705317worker2 # [ 71.391251] nix-daemon[853]: reaped child process 973, status = succeeded5318worker1 # [ 71.406709] nix-grpc-daemon[785]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5319client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5320worker1 # [ 71.439973] nix-grpc-daemon[785]: ts=2026-09-21T13:54:48Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:559705321worker1 # [ 71.444941] nix-grpc-daemon[785]: ts=2026-09-21T13:54:48Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=0 paths=1 nar_bytes_out=1205322client # [ 71.561158] nix-daemon[949]: reaped child process 1035, status = succeeded5323client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.87 seconds)5324client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25325client # [ 71.786143] nix-daemon[949]: accepted connection from pid 1054, user root (trusted)5326client # this derivation will be built:5327client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5328worker1 # [ 71.995251] nix-grpc-daemon[785]: ts=2026-09-21T13:54:49Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:559705329client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5330worker1 # [ 72.001180] nix-grpc-daemon[785]: ts=2026-09-21T13:54:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559705331worker1 # [ 72.006716] nix-grpc-daemon[885]: time=2026-09-21T13:54:49.141Z level=INFO msg="All 1 paths already cached"5332worker1 # [ 72.017237] nix-grpc-daemon[785]: ts=2026-09-21T13:54:49Z level=debug event=want client=10 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05333worker1 # [ 72.023492] nix-grpc-daemon[785]: ts=2026-09-21T13:54:49Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15334worker1 # [ 72.028564] nix-grpc-daemon[785]: ts=2026-09-21T13:54:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559705335worker1 # [ 72.039668] nix-grpc-daemon[785]: ts=2026-09-21T13:54:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559705336client # copying 1 paths to 'lb:50051'...5337client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5338worker1 # [ 72.057106] nix-grpc-daemon[785]: ts=2026-09-21T13:54:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:559705339worker1 # [ 72.068545] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5340worker1 # [ 72.082350] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 17 ms (attempt 1/5)5341worker1 # [ 72.103928] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 46 ms (attempt 2/5)5342worker1 # [ 72.154514] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 8 ms (attempt 3/5)5343worker1 # [ 72.167796] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 490 ms (attempt 4/5)5344worker1 # [ 72.664543] nix-grpc-daemon[785]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5345worker1 # [ 72.668504] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5346worker1 # [ 72.674143] nix-grpc-daemon[785]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5347worker1 # [ 72.921410] nix-daemon[870]: accepted connection from pid 1223, user nix-grpc-daemon (trusted)5348worker1 # [ 72.934225] nix-daemon[870]: reaped child process 1226, status = succeeded5349worker1 # [ 73.132864] nix-daemon[870]: accepted connection from pid 1228, user nix-grpc-daemon (trusted)5350worker1 # [ 73.144713] nix-daemon[870]: reaped child process 1231, status = succeeded5351lb # [ 73.165829] niks3-server[1011]: 2026/09/21 13:54:50 INFO Received uploads request method=POST path=/api/pending_closures5352worker1 # [ 73.174462] nix-grpc-daemon[885]: time=2026-09-21T13:54:50.310Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5353worker1 # [ 73.176375] nix-grpc-daemon[885]: time=2026-09-21T13:54:50.312Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5354lb # [ 73.194497] niks3-server[1011]: 2026/09/21 13:54:50 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5355lb # [ 73.205289] niks3-server[1011]: 2026/09/21 13:54:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5356lb # [ 73.209265] niks3-server[1011]: 2026/09/21 13:54:50 INFO Signed narinfos id=21 count=15357worker1 # [ 73.213264] nix-grpc-daemon[885]: time=2026-09-21T13:54:50.349Z level=INFO msg="Uploading 1 narinfos"5358lb # [ 73.216070] niks3-server[1011]: 2026/09/21 13:54:50 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5359lb # [ 73.227261] niks3-server[1011]: 2026/09/21 13:54:50 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5360worker1 # [ 73.232516] nix-grpc-daemon[885]: time=2026-09-21T13:54:50.369Z level=INFO msg="Upload complete. (528ms)"5361lb # [ 73.233670] niks3-server[1011]: 2026/09/21 13:54:50 INFO Completed upload id=215362worker1 # [ 73.235093] nix-grpc-daemon[785]: ts=2026-09-21T13:54:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=1 paths=1 nar_bytes_in=5605363lb # [ 73.237381] niks3-server[1011]: 2026/09/21 13:54:50 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5364worker1 # [ 73.240959] nix-grpc-daemon[785]: ts=2026-09-21T13:54:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:559705365worker1 # [ 73.251058] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5366worker1 # [ 73.257138] nix-daemon[870]: reaped child process 1216, status = succeeded5367worker1 # [ 73.260554] nix-daemon[870]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5368client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5369client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5370client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5371worker1 # [ 74.674071] nix-daemon[870]: reaped child process 1235, status = succeeded5372worker1 # [ 74.843341] nix-daemon[870]: accepted connection from pid 1251, user nix-grpc-daemon (trusted)5373worker1 # [ 74.854263] nix-daemon[870]: reaped child process 1254, status = succeeded5374lb # [ 74.874704] niks3-server[1011]: 2026/09/21 13:54:51 INFO Received uploads request method=POST path=/api/pending_closures5375worker1 # [ 74.881808] nix-grpc-daemon[885]: time=2026-09-21T13:54:52.018Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5376lb # [ 74.906218] niks3-server[1011]: 2026/09/21 13:54:51 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5377lb # [ 74.910396] niks3-server[1011]: 2026/09/21 13:54:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5378worker1 # [ 74.917080] nix-grpc-daemon[885]: time=2026-09-21T13:54:52.053Z level=INFO msg="Uploading 1 narinfos"5379lb # [ 74.918108] niks3-server[1011]: 2026/09/21 13:54:51 INFO Signed narinfos id=22 count=15380lb # [ 74.922369] niks3-server[1011]: 2026/09/21 13:54:51 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5381lb # [ 74.934676] niks3-server[1011]: 2026/09/21 13:54:51 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5382lb # [ 74.940437] niks3-server[1011]: 2026/09/21 13:54:51 INFO Completed upload id=225383worker1 # [ 74.939686] nix-grpc-daemon[885]: time=2026-09-21T13:54:52.076Z level=INFO msg="Upload complete. (271ms)"5384lb # [ 74.943281] niks3-server[1011]: 2026/09/21 13:54:51 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5385worker1 # [ 74.946793] nix-grpc-daemon[785]: ts=2026-09-21T13:54:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=10 outputs=15386worker1 # [ 74.957257] nix-daemon[870]: reaped child process 1233, status = succeeded5387client # copying 1 paths...5388worker1 # [ 74.992800] nix-grpc-daemon[785]: ts=2026-09-21T13:54:52Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:559705389client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5390worker1 # [ 75.011142] nix-grpc-daemon[785]: ts=2026-09-21T13:54:52Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:559705391worker1 # [ 75.015564] nix-grpc-daemon[785]: ts=2026-09-21T13:54:52Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=0 paths=1 nar_bytes_out=1205392client # [ 75.121195] nix-daemon[949]: reaped child process 1057, status = succeeded5393client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.56 seconds)5394client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25395client # [ 75.312113] nix-daemon[949]: accepted connection from pid 1076, user root (trusted)5396client # this derivation will be built:5397client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5398worker2 # [ 75.522969] nix-grpc-daemon[783]: ts=2026-09-21T13:54:51Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:447085399client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5400worker1 # [ 75.531785] nix-grpc-daemon[785]: ts=2026-09-21T13:54:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:559705401worker1 # [ 75.542313] nix-grpc-daemon[885]: time=2026-09-21T13:54:52.678Z level=INFO msg="All 1 paths already cached"5402worker1 # [ 75.556541] nix-grpc-daemon[785]: ts=2026-09-21T13:54:52Z level=debug event=want client=11 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05403worker2 # [ 75.552941] nix-grpc-daemon[783]: ts=2026-09-21T13:54:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447085404worker1 # [ 75.561311] nix-grpc-daemon[785]: ts=2026-09-21T13:54:52Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15405worker2 # [ 75.564421] nix-grpc-daemon[783]: ts=2026-09-21T13:54:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447085406worker2 # [ 75.576727] nix-grpc-daemon[880]: time=2026-09-21T13:54:51.888Z level=INFO msg="All 1 paths already cached"5407client # copying 1 paths to 'lb:50051'...5408client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5409worker2 # [ 75.583820] nix-grpc-daemon[783]: ts=2026-09-21T13:54:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447085410worker2 # [ 75.592928] nix-daemon[853]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5411worker2 # [ 75.774874] nix-daemon[853]: accepted connection from pid 998, user nix-grpc-daemon (trusted)5412worker2 # [ 75.787384] nix-daemon[853]: reaped child process 1001, status = succeeded5413worker2 # [ 75.950178] nix-daemon[853]: accepted connection from pid 1003, user nix-grpc-daemon (trusted)5414worker2 # [ 75.962189] nix-daemon[853]: reaped child process 1006, status = succeeded5415lb # [ 75.985506] niks3-server[1011]: 2026/09/21 13:54:52 INFO Received uploads request method=POST path=/api/pending_closures5416worker2 # [ 75.988229] nix-grpc-daemon[880]: time=2026-09-21T13:54:52.300Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5417worker2 # [ 75.989729] nix-grpc-daemon[880]: time=2026-09-21T13:54:52.302Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5418lb # [ 76.017303] niks3-server[1011]: 2026/09/21 13:54:52 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5419lb # [ 76.025817] niks3-server[1011]: 2026/09/21 13:54:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5420lb # [ 76.032849] niks3-server[1011]: 2026/09/21 13:54:52 INFO Signed narinfos id=23 count=15421worker2 # [ 76.028228] nix-grpc-daemon[880]: time=2026-09-21T13:54:52.340Z level=INFO msg="Uploading 1 narinfos"5422lb # [ 76.040065] niks3-server[1011]: 2026/09/21 13:54:52 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5423lb # [ 76.050875] niks3-server[1011]: 2026/09/21 13:54:52 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5424lb # [ 76.053658] niks3-server[1011]: 2026/09/21 13:54:52 INFO Completed upload id=235425worker2 # [ 76.046593] nix-grpc-daemon[880]: time=2026-09-21T13:54:52.359Z level=INFO msg="Upload complete. (439ms)"5426lb # [ 76.055247] niks3-server[1011]: 2026/09/21 13:54:52 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5427worker2 # [ 76.050259] nix-grpc-daemon[783]: ts=2026-09-21T13:54:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44708 duration_s=0 paths=1 nar_bytes_in=5605428worker2 # [ 76.055264] nix-grpc-daemon[783]: ts=2026-09-21T13:54:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447085429worker2 # [ 76.064704] nix-daemon[853]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5430worker2 # [ 76.068854] nix-daemon[853]: reaped child process 996, status = succeeded5431worker2 # [ 76.074040] nix-daemon[853]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5432client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5433client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5434lb # [ 76.490657] envoy[901]: conn peer=[2001:db8:1::1]:53798 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4301 tx=2163 ms=18125435lb # [ 76.494421] envoy[901]: conn peer=[2001:db8:1::1]:53810 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=565436lb # [ 76.497836] envoy[901]: conn peer=[2001:db8:1::1]:53836 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=555437lb # [ 76.501792] envoy[901]: conn peer=[2001:db8:1::1]:53826 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4244 tx=1960 ms=23725438lb # [ 76.505336] envoy[901]: conn peer=[2001:db8:1::1]:53840 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3752 tx=1897 ms=30725439lb # [ 76.508804] envoy[901]: conn peer=[2001:db8:1::1]:56206 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=575440client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5441worker2 # [ 77.287252] nix-daemon[853]: reaped child process 1010, status = succeeded5442worker2 # [ 77.424571] nix-daemon[853]: accepted connection from pid 1026, user nix-grpc-daemon (trusted)5443worker2 # [ 77.436568] nix-daemon[853]: reaped child process 1029, status = succeeded5444lb # [ 77.459942] niks3-server[1011]: 2026/09/21 13:54:54 INFO Received uploads request method=POST path=/api/pending_closures5445worker2 # [ 77.461619] nix-grpc-daemon[880]: time=2026-09-21T13:54:53.773Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5446lb # [ 77.490219] niks3-server[1011]: 2026/09/21 13:54:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5447lb # [ 77.496490] niks3-server[1011]: 2026/09/21 13:54:54 INFO Signed narinfos id=24 count=15448worker2 # [ 77.492490] nix-grpc-daemon[880]: time=2026-09-21T13:54:53.804Z level=INFO msg="Uploading 1 narinfos"5449lb # [ 77.504233] niks3-server[1011]: 2026/09/21 13:54:54 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5450lb # [ 77.508481] niks3-server[1011]: 2026/09/21 13:54:54 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5451lb # [ 77.520519] niks3-server[1011]: 2026/09/21 13:54:54 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5452worker2 # [ 77.517992] nix-grpc-daemon[880]: time=2026-09-21T13:54:53.830Z level=INFO msg="Upload complete. (237ms)"5453lb # [ 77.525468] niks3-server[1011]: 2026/09/21 13:54:54 INFO Completed upload id=245454worker2 # [ 77.521288] nix-grpc-daemon[783]: ts=2026-09-21T13:54:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44708 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=11 outputs=15455lb # [ 77.531117] niks3-server[1011]: 2026/09/21 13:54:54 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5456worker2 # [ 77.528194] nix-daemon[853]: reaped child process 1008, status = succeeded5457client # copying 1 paths...5458worker2 # [ 77.529900] nix-grpc-daemon[783]: ts=2026-09-21T13:54:53Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:447085459client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5460worker1 # [ 77.557568] nix-grpc-daemon[785]: ts=2026-09-21T13:54:54Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:559705461worker1 # [ 77.568196] nix-grpc-daemon[785]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5462worker1 # [ 77.581429] nix-grpc-daemon[785]: ts=2026-09-21T13:54:54Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55970 duration_s=0 paths=1 nar_bytes_out=1205463client # [ 77.689380] nix-daemon[949]: reaped child process 1079, status = succeeded5464client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.57 seconds)5465(finished: subtest: an input only one worker has still reaches the builder, in 10.01 seconds)5466subtest: upload whose reference the worker lacks is completed from the cache5467client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5468client # this derivation will be built:5469client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5470client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5471client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.49 seconds)5472client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5473client # this derivation will be built:5474client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5475client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5476client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.47 seconds)5477client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5478client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.22 seconds)5479worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5480worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.24 seconds)5481worker2 # [ 79.159056] systemd[1]: Stopping Nix worker-protocol over gRPC...5482worker2 # [ 79.903519] nix-grpc-daemon[783]: ts=2026-09-21T13:54:56Z level=info event=signal_exit5483worker2 # [ 81.178913] nix-grpc-daemon[783]: ts=2026-09-21T13:54:57Z level=info event=scheduler_restarting5484worker1 # [ 81.192892] nix-grpc-daemon[785]: ts=2026-09-21T13:54:58Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:559565485worker2 # [ 81.189284] nix-grpc-daemon[783]: ts=2026-09-21T13:54:57Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5486worker1 # [ 81.196918] nix-grpc-daemon[785]: ts=2026-09-21T13:54:58Z level=info event=worker_gone addr=192.168.1.4:500515487worker2 # [ 81.205373] nix-daemon[853]: reaped child process 855, status = succeeded5488worker2 # [ 81.971566] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5489worker2 # [ 81.973703] systemd[1]: Stopped Nix worker-protocol over gRPC.5490worker2 # [ 81.974619] systemd[1]: nix-grpc-daemon.service: Consumed 1.496s CPU time over 1min 5.358s wall clock time, 49.5M memory peak, 59.4K incoming IP traffic, 175.6K outgoing IP traffic.5491worker2 # [ 81.979184] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5492worker2 # [ 81.980883] systemd[1]: Closed Nix worker-protocol over gRPC.5493worker2 # [ 81.982287] systemd[1]: nix-grpc-daemon.socket: Consumed 287us CPU time over 1min 11.840s wall clock time, 256K memory peak, 60.7K incoming IP traffic, 52.2K outgoing IP traffic.5494lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5495lb: must succeed: curl -sf localhost:9901/clusters5496lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5497lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.07 seconds)5498client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25499worker1 # [ 82.316286] nix-grpc-daemon[785]: ts=2026-09-21T13:54:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:559705500worker1 # [ 82.323095] nix-grpc-daemon[785]: ts=2026-09-21T13:54:59Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:559705501client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5502client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.30 seconds)5503worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5504worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.03 seconds)5505worker2 # [ 82.432715] systemd[1]: Listening on Nix worker-protocol over gRPC.5506worker1 # [ 82.500229] systemd[1]: Stopping Nix worker-protocol over gRPC...5507worker1 # [ 82.815271] nix-grpc-daemon[785]: ts=2026-09-21T13:54:59Z level=info event=signal_exit5508worker2 # [ 83.043689] systemd[1]: Starting scheduler-token.service...5509lb # [ 83.081515] niks3-server[1011]: 2026/09/21 13:54:59 INFO lead: released remote=192.168.1.3:520845510worker2 # [ 83.127685] systemd[1]: scheduler-token.service: Deactivated successfully.5511worker2 # [ 83.130651] systemd[1]: Finished scheduler-token.service.5512worker2 # [ 83.132699] systemd[1]: scheduler-token.service: Consumed 36ms CPU time over 87ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5513worker2 # [ 83.137540] systemd[1]: Starting Nix worker-protocol over gRPC...5514worker2 # [ 83.277385] nix-grpc-daemon[1053]: ts=2026-09-21T13:54:59Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5515worker2 # [ 83.279967] nix-grpc-daemon[1053]: ts=2026-09-21T13:54:59Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5516worker2 # [ 83.283628] nix-grpc-daemon[1053]: ts=2026-09-21T13:54:59Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15517worker2 # [ 83.290066] nix-grpc-daemon[1053]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5518worker2 # [ 83.293027] nix-grpc-daemon[1053]: E0000 00:00:1789998899.602366 1053 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5519worker2 # [ 83.296776] nix-grpc-daemon[1053]: E0000 00:00:1789998899.605346 1053 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5520worker2 # [ 83.302050] nix-grpc-daemon[1053]: ts=2026-09-21T13:54:59Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5521worker2 # [ 83.303627] systemd[1]: Started Nix worker-protocol over gRPC.5522worker2 # [ 83.309487] nix-grpc-daemon[1053]: ts=2026-09-21T13:54:59Z level=info event=scheduler_take_over5523lb # [ 83.317185] niks3-server[1011]: 2026/09/21 13:55:00 INFO lead: acquired remote=192.168.1.4:449925524worker2 # [ 83.313101] nix-grpc-daemon[1053]: E0000 00:00:1789998899.624245 1095 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5525worker2 # [ 83.325414] nix-grpc-daemon[1053]: E0000 00:00:1789998899.637868 1095 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5526worker2 # [ 83.329145] nix-grpc-daemon[1053]: E0000 00:00:1789998899.640377 1095 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5527worker2 # [ 83.340047] nix-grpc-daemon[1053]: E0000 00:00:1789998899.652475 1095 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5528worker2 # [ 83.349338] nix-grpc-daemon[1053]: ts=2026-09-21T13:54:59Z level=info event=scheduler_connected addr=lb:500515529worker2 # [ 83.351086] nix-grpc-daemon[1053]: ts=2026-09-21T13:54:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5530worker2 # [ 83.552602] nix-grpc-daemon[1053]: ts=2026-09-21T13:54:59Z level=info event=scheduler_connected addr=lb:500515531worker2 # [ 83.556818] nix-grpc-daemon[1053]: ts=2026-09-21T13:54:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5532worker2 # [ 83.757957] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:00Z level=info event=scheduler_connected addr=lb:500515533worker2 # [ 83.762377] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5534worker2 # [ 83.963630] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:00Z level=info event=scheduler_connected addr=lb:500515535worker2 # [ 83.967900] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5536worker1 # [ 84.085606] nix-grpc-daemon[785]: ts=2026-09-21T13:55:01Z level=info event=scheduler_restarting5537worker1 # [ 84.089967] nix-grpc-daemon[785]: ts=2026-09-21T13:55:01Z level=info event=scheduler_restarting addr=lb:500515538worker1 # [ 84.094244] nix-grpc-daemon[785]: ts=2026-09-21T13:55:01Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:559565539worker1 # [ 84.097491] nix-grpc-daemon[785]: ts=2026-09-21T13:55:01Z level=info event=worker_gone addr=192.168.1.3:500515540worker1 # [ 84.117363] nix-daemon[870]: reaped child process 872, status = succeeded5541worker2 # [ 84.169134] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:00Z level=info event=scheduler_connected addr=lb:500515542worker2 # [ 84.173071] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5543worker1 # [ 84.297564] nix-grpc-daemon[785]: ts=2026-09-21T13:55:01Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5544worker2 # [ 84.373885] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:00Z level=info event=scheduler_connected addr=lb:500515545worker2 # [ 84.381066] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:00Z level=info event=oidc_jwks provider=mock keys=15546worker2 # [ 84.382396] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:00Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:331265547worker2 # [ 84.384708] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:00Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05548worker1 # [ 84.967722] nix-grpc-daemon[785]: E0000 00:00:1789998902.104038 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5549worker1 # [ 84.972402] nix-grpc-daemon[785]: E0000 00:00:1789998902.104166 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5550worker1 # [ 85.933672] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5551worker1 # [ 85.938357] systemd[1]: Stopped Nix worker-protocol over gRPC.5552worker1 # [ 85.941995] systemd[1]: nix-grpc-daemon.service: Consumed 2.749s CPU time over 1min 9.450s wall clock time, 64.4M memory peak, 97.4K incoming IP traffic, 224.2K outgoing IP traffic.5553worker1 # [ 85.951216] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5554worker1 # [ 85.954537] systemd[1]: Closed Nix worker-protocol over gRPC.5555worker1 # [ 85.957229] systemd[1]: nix-grpc-daemon.socket: Consumed 281us CPU time over 1min 16.535s wall clock time, 256K memory peak, 99.8K incoming IP traffic, 83.2K outgoing IP traffic.5556lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5557lb: must succeed: curl -sf localhost:9901/clusters5558lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5559lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.08 seconds)5560client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25561worker2 # [ 86.315799] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:331285562worker2 # [ 86.330128] nix-daemon[853]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5563worker2 # [ 86.348134] nix-grpc-daemon[1053]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 33 ms (attempt 1/5)5564worker2 # [ 86.386872] nix-grpc-daemon[1053]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 87 ms (attempt 2/5)5565worker2 # [ 86.479672] nix-grpc-daemon[1053]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 309 ms (attempt 3/5)5566lb # [ 86.489719] envoy[901]: conn peer=[2001:db8:1::1]:56218 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3742 tx=1901 ms=21045567lb # [ 86.493754] envoy[901]: conn peer=[2001:db8:1::1]:56228 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=329 ms=775568lb # [ 86.497400] envoy[901]: conn peer=[2001:db8:1::4]:48608 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=82436 tx=3680 ms=642145569lb # [ 86.501093] envoy[901]: conn peer=[2001:db8:1::1]:56240 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=215570lb # [ 86.504513] envoy[901]: conn peer=192.168.1.3:41512 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=21618 tx=4115 ms=674085571worker2 # [ 86.794376] nix-grpc-daemon[1053]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 753 ms (attempt 4/5)5572worker2 # [ 87.554401] nix-grpc-daemon[1053]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5573worker2 # [ 87.559080] nix-grpc-daemon[1053]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5574client # copying 2 paths...5575client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5576worker2 # [ 87.581705] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:331285577client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5578worker2 # [ 87.594979] nix-daemon[853]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5579worker2 # [ 87.614225] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:03Z level=info event=niks3_push_started pid=11135580worker2 # [ 87.810726] nix-grpc-daemon[1113]: time=2026-09-21T13:55:04.122Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5581worker2 # [ 87.973595] nix-daemon[853]: accepted connection from pid 1126, user nix-grpc-daemon (trusted)5582worker2 # [ 87.985799] nix-daemon[853]: reaped child process 1129, status = succeeded5583lb # [ 88.011578] niks3-server[1011]: 2026/09/21 13:55:04 INFO Received uploads request method=POST path=/api/pending_closures5584lb # [ 88.021942] niks3-server[1011]: 2026/09/21 13:55:04 INFO Received uploads request method=POST path=/api/pending_closures5585worker2 # [ 88.021346] nix-grpc-daemon[1113]: time=2026-09-21T13:55:04.333Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5586worker2 # [ 88.022932] nix-grpc-daemon[1113]: time=2026-09-21T13:55:04.335Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5587worker2 # [ 88.025161] nix-grpc-daemon[1113]: time=2026-09-21T13:55:04.337Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5588lb # [ 88.074801] niks3-server[1011]: 2026/09/21 13:55:04 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5589lb # [ 88.085830] niks3-server[1011]: 2026/09/21 13:55:04 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5590lb # [ 88.094666] niks3-server[1011]: 2026/09/21 13:55:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5591lb # [ 88.098907] niks3-server[1011]: 2026/09/21 13:55:05 INFO Signed narinfos id=25 count=15592lb # [ 88.100519] niks3-server[1011]: 2026/09/21 13:55:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5593lb # [ 88.102979] niks3-server[1011]: 2026/09/21 13:55:05 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5594lb # [ 88.104926] niks3-server[1011]: 2026/09/21 13:55:05 INFO Signed narinfos id=26 count=25595worker2 # [ 88.099229] nix-grpc-daemon[1113]: time=2026-09-21T13:55:04.410Z level=INFO msg="Uploading 3 narinfos"5596lb # [ 88.111375] niks3-server[1011]: 2026/09/21 13:55:05 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5597lb # [ 88.120097] niks3-server[1011]: 2026/09/21 13:55:05 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5598lb # [ 88.123240] niks3-server[1011]: 2026/09/21 13:55:05 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5599lb # [ 88.127403] niks3-server[1011]: 2026/09/21 13:55:05 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5600lb # [ 88.130585] niks3-server[1011]: 2026/09/21 13:55:05 INFO Completed upload id=255601lb # [ 88.131956] niks3-server[1011]: 2026/09/21 13:55:05 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5602lb # [ 88.133706] niks3-server[1011]: 2026/09/21 13:55:05 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5603lb # [ 88.136341] niks3-server[1011]: 2026/09/21 13:55:05 INFO Completed upload id=265604worker2 # [ 88.130593] nix-grpc-daemon[1113]: time=2026-09-21T13:55:04.442Z level=INFO msg="Upload complete. (316ms)"5605worker2 # [ 88.135358] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33128 duration_s=0 paths=2 nar_bytes_in=3125606worker2 # [ 88.147981] nix-daemon[853]: reaped child process 1111, status = succeeded5607client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 2.13 seconds)5608worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5609worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.03 seconds)5610worker1 # [ 88.279478] systemd[1]: Listening on Nix worker-protocol over gRPC.5611worker1 # [ 88.285683] systemd[1]: Starting Nix worker-protocol over gRPC...5612worker1 # [ 88.471364] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:05Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5613worker1 # [ 88.473986] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:05Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5614worker1 # [ 88.477663] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:05Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15615worker1 # [ 88.485067] nix-grpc-daemon[1280]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5616worker1 # [ 88.486418] nix-grpc-daemon[1280]: E0000 00:00:1789998905.621365 1280 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5617worker1 # [ 88.492063] nix-grpc-daemon[1280]: E0000 00:00:1789998905.626578 1280 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5618worker1 # [ 88.495889] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:05Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5619worker1 # [ 88.498091] systemd[1]: Started Nix worker-protocol over gRPC.5620lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5621lb: must succeed: curl -sf localhost:9901/clusters5622worker1 # [ 88.505075] nix-grpc-daemon[1280]: E0000 00:00:1789998905.641156 1322 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5623worker1 # [ 88.516591] nix-grpc-daemon[1280]: E0000 00:00:1789998905.653101 1322 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5624worker1 # [ 88.521051] nix-grpc-daemon[1280]: E0000 00:00:1789998905.655352 1322 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5625worker1 # [ 88.530700] nix-grpc-daemon[1280]: E0000 00:00:1789998905.667145 1322 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5626worker1 # [ 88.540841] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:05Z level=info event=scheduler_connected addr=lb:500515627worker2 # [ 88.536429] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:04Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:331265628worker2 # [ 88.538254] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:04Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05629lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5630lb: must succeed: curl -sf localhost:9901/clusters5631lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5632lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.16 seconds)5633(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.96 seconds)5634subtest: a niks3 restart does not cost the leader its role5635lb: must succeed: curl -sf localhost:9901/clusters5636lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5637lb: must succeed: curl -sf localhost:9901/clusters5638lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5639lb # [ 89.800374] niks3-server[1011]: 2026/09/21 13:55:06 INFO Shutdown signal received, draining in-flight requests timeout=10s5640lb # [ 89.803343] systemd[1]: Stopping niks3 server...5641lb # [ 89.804473] niks3-server[1011]: 2026/09/21 13:55:06 INFO lead: released remote=192.168.1.4:449925642worker2 # [ 89.799343] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:06Z level=info event=scheduler_lock_lost status=2005643worker1 # [ 89.807981] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:06Z level=info event=scheduler_lock_lost status=2005644lb # [ 89.819394] systemd[1]: niks3.service: Deactivated successfully.5645lb # [ 89.821177] systemd[1]: Stopped niks3 server.5646lb # [ 89.821810] systemd[1]: niks3.service: Consumed 584ms CPU time over 1min 3.838s wall clock time, 20.2M memory peak, 67.4K incoming IP traffic, 97.9K outgoing IP traffic.5647lb # [ 89.830422] systemd[1]: Starting niks3 server...5648lb # [ 89.899455] niks3-server[1150]: 2026/09/21 13:55:06 goose: no migrations to run. current version: 202609200000005649lb # [ 89.905798] niks3-server[1150]: 2026/09/21 13:55:06 OK 1_commit_pending_closure.sql (4.44ms)5650lb # [ 89.909483] niks3-server[1150]: 2026/09/21 13:55:06 OK 2_object_stats_trigger.sql (2ms)5651lb # [ 89.910597] niks3-server[1150]: 2026/09/21 13:55:06 goose: up to current file version: 25652lb # [ 89.912671] niks3-server[1150]: 2026/09/21 13:55:06 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key5653lb # [ 89.925258] niks3-server[1150]: 2026/09/21 13:55:06 INFO Read proxy enabled — serving cache objects from S35654lb # [ 89.926899] niks3-server[1150]: 2026/09/21 13:55:06 INFO Using socket-activated listener address=0.0.0.0:57515655lb # [ 89.929213] niks3-server[1150]: 2026/09/21 13:55:06 INFO systemd watchdog enabled interval=15s5656lb # [ 89.930367] niks3-server[1150]: 2026/09/21 13:55:06 INFO Starting HTTP server address=0.0.0.0:57515657lb # [ 89.932206] systemd[1]: Started niks3 server.5658lb: waiting for unit niks3.service5659lb: (finished: waiting for unit niks3.service, in 0.04 seconds)5660lb: must succeed: curl -sf localhost:9901/clusters5661lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5662worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5663worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5664lb: must succeed: curl -sf localhost:9901/clusters5665lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5666lb: must succeed: curl -sf localhost:9901/clusters5667lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5668worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5669worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.06 seconds)5670client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&25671worker2 # [ 90.459785] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:331285672client # don't know how to build these paths:5673client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5674worker1 # [ 90.478176] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:536825675worker1 # [ 90.490258] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5676worker1 # [ 90.508124] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 76 ms (attempt 1/5)5677worker1 # [ 90.590204] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 5 ms (attempt 2/5)5678worker1 # [ 90.600853] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 107 ms (attempt 3/5)5679worker1 # [ 90.713813] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 648 ms (attempt 4/5)5680lb # [ 90.813470] niks3-server[1150]: 2026/09/21 13:55:07 INFO lead: acquired remote=192.168.1.4:450465681worker1 # [ 91.368643] nix-grpc-daemon[1280]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5682worker1 # [ 91.372667] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5683worker2 # [ 91.408090] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:07Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=05684worker2 # [ 91.412351] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:07Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.4:50051 clients=15685worker2 # [ 91.451168] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:331285686worker2 # [ 91.465380] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:331285687client # copying 1 paths to 'lb:50051'...5688client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...5689worker2 # [ 91.477644] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:331285690worker2 # [ 91.488771] nix-daemon[853]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5691worker2 # [ 91.677111] nix-daemon[853]: accepted connection from pid 1148, user nix-grpc-daemon (trusted)5692worker2 # [ 91.688862] nix-daemon[853]: reaped child process 1151, status = succeeded5693worker2 # [ 91.857577] nix-daemon[853]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5694worker2 # [ 91.869532] nix-daemon[853]: reaped child process 1156, status = succeeded5695lb # [ 91.893618] niks3-server[1150]: 2026/09/21 13:55:08 INFO Received uploads request method=POST path=/api/pending_closures5696worker2 # [ 91.898697] nix-grpc-daemon[1113]: time=2026-09-21T13:55:08.210Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5697worker2 # [ 91.900172] nix-grpc-daemon[1113]: time=2026-09-21T13:55:08.212Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"5698lb # [ 91.943139] niks3-server[1150]: 2026/09/21 13:55:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5699lb # [ 91.949980] niks3-server[1150]: 2026/09/21 13:55:08 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst5700lb # [ 91.966592] niks3-server[1150]: 2026/09/21 13:55:08 INFO Signed narinfos id=27 count=15701worker2 # [ 91.961326] nix-grpc-daemon[1113]: time=2026-09-21T13:55:08.273Z level=INFO msg="Uploading 1 narinfos"5702lb # [ 91.974367] niks3-server[1150]: 2026/09/21 13:55:08 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5703lb # [ 91.977382] niks3-server[1150]: 2026/09/21 13:55:08 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls5704lb # [ 91.980596] niks3-server[1150]: 2026/09/21 13:55:08 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo5705lb # [ 91.985464] niks3-server[1150]: 2026/09/21 13:55:08 INFO Completed upload id=275706worker2 # [ 91.978533] nix-grpc-daemon[1113]: time=2026-09-21T13:55:08.290Z level=INFO msg="Upload complete. (478ms)"5707worker2 # [ 91.983156] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33128 duration_s=0 paths=1 nar_bytes_in=5365708worker2 # [ 91.989356] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:331285709worker2 # [ 92.001258] nix-daemon[853]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5710worker2 # [ 92.008460] nix-daemon[853]: reaped child process 1146, status = succeeded5711worker2 # [ 92.016976] nix-daemon[853]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5712client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...5713client # farm-a> worker2: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5714client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5715client # farm-a> LOG-farm-a-after-niks3-restart5716worker2 # [ 92.828845] nix-daemon[853]: reaped child process 1160, status = succeeded5717worker2 # [ 92.962765] nix-daemon[853]: accepted connection from pid 1176, user nix-grpc-daemon (trusted)5718worker2 # [ 92.973467] nix-daemon[853]: reaped child process 1179, status = succeeded5719lb # [ 92.993479] niks3-server[1150]: 2026/09/21 13:55:09 INFO Received uploads request method=POST path=/api/pending_closures5720worker2 # [ 92.996358] nix-grpc-daemon[1113]: time=2026-09-21T13:55:09.308Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5721worker2 # [ 92.997834] nix-grpc-daemon[1113]: time=2026-09-21T13:55:09.310Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"5722lb # [ 93.039412] niks3-server[1150]: 2026/09/21 13:55:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5723lb # [ 93.047071] niks3-server[1150]: 2026/09/21 13:55:09 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5724lb # [ 93.051224] niks3-server[1150]: 2026/09/21 13:55:09 INFO Signed narinfos id=28 count=15725worker2 # [ 93.046679] nix-grpc-daemon[1113]: time=2026-09-21T13:55:09.358Z level=INFO msg="Uploading 1 narinfos"5726lb # [ 93.060371] niks3-server[1150]: 2026/09/21 13:55:09 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5727lb # [ 93.064728] niks3-server[1150]: 2026/09/21 13:55:09 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls5728lb # [ 93.072094] niks3-server[1150]: 2026/09/21 13:55:09 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5729lb # [ 93.076967] niks3-server[1150]: 2026/09/21 13:55:09 INFO Completed upload id=285730worker2 # [ 93.069985] nix-grpc-daemon[1113]: time=2026-09-21T13:55:09.382Z level=INFO msg="Upload complete. (247ms)"5731lb # [ 93.078668] niks3-server[1150]: 2026/09/21 13:55:09 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo5732worker2 # [ 93.079860] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33128 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=15733worker2 # [ 93.092966] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:09Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=05734worker2 # [ 93.095041] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:09Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.4:50051 clients=15735worker2 # [ 93.098502] nix-daemon[853]: reaped child process 1158, status = succeeded5736worker2 # [ 93.138458] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:331285737worker2 # [ 93.151265] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:331285738client # copying 1 paths to 'lb:50051'...5739worker2 # [ 93.192639] nix-grpc-daemon[1113]: time=2026-09-21T13:55:09.504Z level=INFO msg="All 1 paths already cached"5740client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...5741worker2 # [ 93.197452] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:331285742worker2 # [ 93.205525] nix-daemon[853]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5743worker2 # [ 93.363302] nix-daemon[853]: accepted connection from pid 1185, user nix-grpc-daemon (trusted)5744worker2 # [ 93.376783] nix-daemon[853]: reaped child process 1188, status = succeeded5745worker2 # [ 93.506425] nix-daemon[853]: accepted connection from pid 1190, user nix-grpc-daemon (trusted)5746worker2 # [ 93.518559] nix-daemon[853]: reaped child process 1193, status = succeeded5747lb # [ 93.535223] niks3-server[1150]: 2026/09/21 13:55:10 INFO Received uploads request method=POST path=/api/pending_closures5748worker2 # [ 93.537701] nix-grpc-daemon[1113]: time=2026-09-21T13:55:09.850Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5749worker2 # [ 93.539198] nix-grpc-daemon[1113]: time=2026-09-21T13:55:09.851Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"5750lb # [ 93.565894] niks3-server[1150]: 2026/09/21 13:55:10 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst5751lb # [ 93.577883] niks3-server[1150]: 2026/09/21 13:55:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5752lb # [ 93.582805] niks3-server[1150]: 2026/09/21 13:55:10 INFO Signed narinfos id=29 count=15753worker2 # [ 93.578898] nix-grpc-daemon[1113]: time=2026-09-21T13:55:09.891Z level=INFO msg="Uploading 1 narinfos"5754lb # [ 93.590175] niks3-server[1150]: 2026/09/21 13:55:10 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls5755lb # [ 93.599350] niks3-server[1150]: 2026/09/21 13:55:10 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5756worker2 # [ 93.599629] nix-grpc-daemon[1113]: time=2026-09-21T13:55:09.912Z level=INFO msg="Upload complete. (380ms)"5757lb # [ 93.606791] niks3-server[1150]: 2026/09/21 13:55:10 INFO Completed upload id=295758worker2 # [ 93.601555] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33128 duration_s=0 paths=1 nar_bytes_in=6565759lb # [ 93.610949] niks3-server[1150]: 2026/09/21 13:55:10 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo5760worker2 # [ 93.607757] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:331285761worker2 # [ 93.612154] nix-daemon[853]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5762worker2 # [ 93.614385] nix-daemon[853]: reaped child process 1183, status = succeeded5763worker2 # [ 93.619872] nix-daemon[853]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5764client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...5765client # farm-b> worker2: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5766client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5767client # farm-b> LOG-farm-b-after-niks3-restart5768worker2 # [ 95.044908] nix-daemon[853]: reaped child process 1197, status = succeeded5769worker2 # [ 95.212131] nix-daemon[853]: accepted connection from pid 1213, user nix-grpc-daemon (trusted)5770worker2 # [ 95.224201] nix-daemon[853]: reaped child process 1216, status = succeeded5771lb # [ 95.247680] niks3-server[1150]: 2026/09/21 13:55:12 INFO Received uploads request method=POST path=/api/pending_closures5772worker2 # [ 95.250183] nix-grpc-daemon[1113]: time=2026-09-21T13:55:11.562Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5773worker2 # [ 95.251672] nix-grpc-daemon[1113]: time=2026-09-21T13:55:11.564Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"5774lb # [ 95.284094] niks3-server[1150]: 2026/09/21 13:55:12 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5775lb # [ 95.301102] niks3-server[1150]: 2026/09/21 13:55:12 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst5776lb # [ 95.307322] niks3-server[1150]: 2026/09/21 13:55:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5777lb # [ 95.312678] niks3-server[1150]: 2026/09/21 13:55:12 INFO Signed narinfos id=30 count=15778worker2 # [ 95.308383] nix-grpc-daemon[1113]: time=2026-09-21T13:55:11.620Z level=INFO msg="Uploading 1 narinfos"5779lb # [ 95.319751] niks3-server[1150]: 2026/09/21 13:55:12 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls5780lb # [ 95.324147] niks3-server[1150]: 2026/09/21 13:55:12 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5781lb # [ 95.328398] niks3-server[1150]: 2026/09/21 13:55:12 INFO Completed upload id=305782worker2 # [ 95.321517] nix-grpc-daemon[1113]: time=2026-09-21T13:55:11.633Z level=INFO msg="Upload complete. (293ms)"5783lb # [ 95.330212] niks3-server[1150]: 2026/09/21 13:55:12 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo5784worker2 # [ 95.330523] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33128 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=15785worker2 # [ 95.340821] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:11Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=05786worker2 # [ 95.343890] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:11Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.4:50051 clients=15787worker2 # [ 95.353343] nix-daemon[853]: reaped child process 1195, status = succeeded5788worker2 # [ 95.385456] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:331285789worker2 # [ 95.398111] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:331285790worker2 # [ 95.411845] nix-grpc-daemon[1113]: time=2026-09-21T13:55:11.724Z level=INFO msg="All 2 paths already cached"5791client # copying 1 paths to 'lb:50051'...5792client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...5793worker2 # [ 95.420479] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:331285794worker2 # [ 95.429352] nix-daemon[853]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5795worker2 # [ 95.597227] nix-daemon[853]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5796worker2 # [ 95.609271] nix-daemon[853]: reaped child process 1225, status = succeeded5797worker2 # [ 95.765371] nix-daemon[853]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5798worker2 # [ 95.777139] nix-daemon[853]: reaped child process 1230, status = succeeded5799lb # [ 95.799370] niks3-server[1150]: 2026/09/21 13:55:12 INFO Received uploads request method=POST path=/api/pending_closures5800worker2 # [ 95.801202] nix-grpc-daemon[1113]: time=2026-09-21T13:55:12.113Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5801worker2 # [ 95.802614] nix-grpc-daemon[1113]: time=2026-09-21T13:55:12.113Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"5802lb # [ 95.827742] niks3-server[1150]: 2026/09/21 13:55:12 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst5803lb # [ 95.838557] niks3-server[1150]: 2026/09/21 13:55:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5804worker2 # [ 95.836682] nix-grpc-daemon[1113]: time=2026-09-21T13:55:12.148Z level=INFO msg="Uploading 1 narinfos"5805lb # [ 95.844108] niks3-server[1150]: 2026/09/21 13:55:12 INFO Signed narinfos id=31 count=15806lb # [ 95.848773] niks3-server[1150]: 2026/09/21 13:55:12 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls5807lb # [ 95.858750] niks3-server[1150]: 2026/09/21 13:55:12 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5808lb # [ 95.865891] niks3-server[1150]: 2026/09/21 13:55:12 INFO Completed upload id=315809worker2 # [ 95.859043] nix-grpc-daemon[1113]: time=2026-09-21T13:55:12.171Z level=INFO msg="Upload complete. (416ms)"5810lb # [ 95.868096] niks3-server[1150]: 2026/09/21 13:55:12 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo5811worker2 # [ 95.863120] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33128 duration_s=0 paths=1 nar_bytes_in=7845812worker2 # [ 95.867482] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:331285813worker2 # [ 95.875477] nix-daemon[853]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5814worker2 # [ 95.880471] nix-daemon[853]: reaped child process 1220, status = succeeded5815client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...5816worker2 # [ 95.886447] nix-daemon[853]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5817client # farm-top> worker2: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5818lb # [ 96.490303] envoy[901]: conn peer=[2001:db8:1::1]:47756 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1003 tx=282 ms=18335819client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5820client # farm-top> LOG-farm-top-after-niks3-restart5821worker2 # [ 97.110848] nix-daemon[853]: reaped child process 1234, status = succeeded5822worker2 # [ 97.281092] nix-daemon[853]: accepted connection from pid 1250, user nix-grpc-daemon (trusted)5823worker2 # [ 97.292590] nix-daemon[853]: reaped child process 1253, status = succeeded5824lb # [ 97.314651] niks3-server[1150]: 2026/09/21 13:55:14 INFO Received uploads request method=POST path=/api/pending_closures5825worker2 # [ 97.318097] nix-grpc-daemon[1113]: time=2026-09-21T13:55:13.630Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5826worker2 # [ 97.320769] nix-grpc-daemon[1113]: time=2026-09-21T13:55:13.632Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"5827lb # [ 97.344361] niks3-server[1150]: 2026/09/21 13:55:14 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5828lb # [ 97.361415] niks3-server[1150]: 2026/09/21 13:55:14 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst5829lb # [ 97.371320] niks3-server[1150]: 2026/09/21 13:55:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5830lb # [ 97.376255] niks3-server[1150]: 2026/09/21 13:55:14 INFO Signed narinfos id=32 count=15831worker2 # [ 97.371631] nix-grpc-daemon[1113]: time=2026-09-21T13:55:13.683Z level=INFO msg="Uploading 1 narinfos"5832lb # [ 97.381406] niks3-server[1150]: 2026/09/21 13:55:14 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls5833lb # [ 97.391050] niks3-server[1150]: 2026/09/21 13:55:14 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5834lb # [ 97.397354] niks3-server[1150]: 2026/09/21 13:55:14 INFO Completed upload id=325835worker2 # [ 97.390432] nix-grpc-daemon[1113]: time=2026-09-21T13:55:13.702Z level=INFO msg="Upload complete. (295ms)"5836lb # [ 97.399532] niks3-server[1150]: 2026/09/21 13:55:14 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo5837worker2 # [ 97.398358] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33128 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=15838worker2 # [ 97.411825] nix-daemon[853]: reaped child process 1232, status = succeeded5839client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&2, in 7.23 seconds)5840client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath5841client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath, in 0.25 seconds)5842(finished: subtest: a niks3 restart does not cost the leader its role, in 8.05 seconds)5843subtest: scheduler down: the other node takes the lock and keeps it5844lb: must succeed: curl -sf localhost:9901/clusters5845lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5846lb: must succeed: curl -sf localhost:9901/clusters5847lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5848lb: must succeed: curl -sf localhost:9901/clusters5849lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5850lb: must succeed: curl -sf localhost:9901/clusters5851lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5852worker2 # [ 98.006960] systemd[1]: Stopping Nix worker-protocol over gRPC...5853worker2 # [ 98.300314] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:14Z level=info event=signal_exit5854lb # [ 99.022865] niks3-server[1150]: 2026/09/21 13:55:15 INFO lead: released remote=192.168.1.4:450465855worker2 # [ 100.015516] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:16Z level=info event=scheduler_restarting5856worker1 # [ 100.023369] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:17Z level=info event=scheduler_restarting addr=lb:500515857worker2 # [ 100.018369] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:16Z level=info event=scheduler_restarting addr=lb:500515858worker2 # [ 100.020106] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:16Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:331265859worker2 # [ 100.021362] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:16Z level=info event=worker_gone addr=192.168.1.4:500515860worker2 # [ 100.022673] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:16Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:331265861worker2 # [ 100.025797] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:16Z level=info event=worker_gone addr=192.168.1.3:500515862worker2 # [ 100.040864] nix-daemon[853]: reaped child process 1105, status = succeeded5863worker1 # [ 100.223460] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:17Z level=info event=scheduler_connected addr=lb:500515864worker1 # [ 100.227252] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5865worker2 # [ 100.228557] nix-grpc-daemon[1053]: ts=2026-09-21T13:55:16Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5866worker2 # [ 100.346808] nix-grpc-daemon[1053]: E0000 00:00:1789998916.659029 1098 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5867worker2 # [ 100.352158] nix-grpc-daemon[1053]: E0000 00:00:1789998916.664350 1098 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5868worker1 # [ 100.428121] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:17Z level=info event=scheduler_connected addr=lb:500515869worker1 # [ 100.432396] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5870worker1 # [ 100.633332] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:17Z level=info event=scheduler_connected addr=lb:500515871worker1 # [ 100.637350] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5872lb # [ 100.815972] niks3-server[1150]: 2026/09/21 13:55:17 INFO lead: acquired remote=192.168.1.3:513005873worker1 # [ 100.816324] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:17Z level=info event=scheduler_take_over5874worker1 # [ 100.837897] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:17Z level=info event=scheduler_connected addr=lb:500515875worker1 # [ 100.841824] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5876worker1 # [ 101.043720] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:18Z level=info event=scheduler_connected addr=lb:500515877worker1 # [ 101.051156] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:18Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:458265878worker1 # [ 101.054761] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:18Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05879worker2 # [ 102.113888] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5880worker2 # [ 102.119320] systemd[1]: Stopped Nix worker-protocol over gRPC.5881worker2 # [ 102.120922] systemd[1]: nix-grpc-daemon.service: Consumed 1.117s CPU time over 18.975s wall clock time, 35.9M memory peak, 46.3K incoming IP traffic, 83.5K outgoing IP traffic.5882worker2 # [ 102.127958] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5883worker2 # [ 102.131340] systemd[1]: Closed Nix worker-protocol over gRPC.5884worker2 # [ 102.133826] systemd[1]: nix-grpc-daemon.socket: Consumed 275us CPU time over 19.702s wall clock time, 256K memory peak, 39.5K incoming IP traffic, 29.7K outgoing IP traffic.5885worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5886worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.07 seconds)5887worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5888worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5889client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&25890worker1 # [ 102.530953] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:536825891client # don't know how to build these paths:5892client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5893worker1 # [ 102.539265] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:536825894worker1 # [ 102.556474] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:19Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=05895worker1 # [ 102.561552] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:19Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=15896worker1 # [ 102.600693] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:536825897worker1 # [ 102.612244] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:536825898client # copying 1 paths to 'lb:50051'...5899client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...5900worker1 # [ 102.622958] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:536825901worker1 # [ 102.631866] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5902worker1 # [ 102.654185] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:19Z level=info event=niks3_push_started pid=13535903worker1 # [ 102.878557] nix-grpc-daemon[1353]: time=2026-09-21T13:55:20.014Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""5904worker1 # [ 103.055089] nix-daemon[870]: accepted connection from pid 1366, user nix-grpc-daemon (trusted)5905worker1 # [ 103.065882] nix-daemon[870]: reaped child process 1369, status = succeeded5906worker1 # [ 103.261363] nix-daemon[870]: accepted connection from pid 1371, user nix-grpc-daemon (trusted)5907worker1 # [ 103.273110] nix-daemon[870]: reaped child process 1374, status = succeeded5908lb # [ 103.290821] niks3-server[1150]: 2026/09/21 13:55:20 INFO Received uploads request method=POST path=/api/pending_closures5909worker1 # [ 103.298964] nix-grpc-daemon[1353]: time=2026-09-21T13:55:20.434Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5910worker1 # [ 103.300593] nix-grpc-daemon[1353]: time=2026-09-21T13:55:20.434Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"5911lb # [ 103.322545] niks3-server[1150]: 2026/09/21 13:55:20 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst5912lb # [ 103.334842] niks3-server[1150]: 2026/09/21 13:55:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5913lb # [ 103.340215] niks3-server[1150]: 2026/09/21 13:55:20 INFO Signed narinfos id=33 count=15914worker1 # [ 103.342513] nix-grpc-daemon[1353]: time=2026-09-21T13:55:20.478Z level=INFO msg="Uploading 1 narinfos"5915lb # [ 103.354668] niks3-server[1150]: 2026/09/21 13:55:20 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls5916lb # [ 103.360681] niks3-server[1150]: 2026/09/21 13:55:20 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5917lb # [ 103.367981] niks3-server[1150]: 2026/09/21 13:55:20 INFO Completed upload id=335918worker1 # [ 103.367297] nix-grpc-daemon[1353]: time=2026-09-21T13:55:20.503Z level=INFO msg="Upload complete. (485ms)"5919lb # [ 103.372522] niks3-server[1150]: 2026/09/21 13:55:20 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo5920worker1 # [ 103.371758] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53682 duration_s=0 paths=1 nar_bytes_in=5205921worker1 # [ 103.376696] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:536825922worker1 # [ 103.385822] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5923worker1 # [ 103.391211] nix-daemon[870]: reaped child process 1351, status = succeeded5924worker1 # [ 103.396152] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5925client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...5926client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5927client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5928client # farm-a> LOG-farm-a-on-standby5929worker1 # [ 104.809566] nix-daemon[870]: reaped child process 1378, status = succeeded5930worker1 # [ 104.968178] nix-daemon[870]: accepted connection from pid 1394, user nix-grpc-daemon (trusted)5931worker1 # [ 104.980604] nix-daemon[870]: reaped child process 1397, status = succeeded5932lb # [ 104.998380] niks3-server[1150]: 2026/09/21 13:55:21 INFO Received uploads request method=POST path=/api/pending_closures5933worker1 # [ 105.010294] nix-grpc-daemon[1353]: time=2026-09-21T13:55:22.145Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5934worker1 # [ 105.011629] nix-grpc-daemon[1353]: time=2026-09-21T13:55:22.145Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"5935lb # [ 105.034327] niks3-server[1150]: 2026/09/21 13:55:21 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5936lb # [ 105.049493] niks3-server[1150]: 2026/09/21 13:55:21 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5937lb # [ 105.059826] niks3-server[1150]: 2026/09/21 13:55:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5938worker1 # [ 105.063286] nix-grpc-daemon[1353]: time=2026-09-21T13:55:22.199Z level=INFO msg="Uploading 1 narinfos"5939lb # [ 105.063621] niks3-server[1150]: 2026/09/21 13:55:21 INFO Signed narinfos id=34 count=15940lb # [ 105.068678] niks3-server[1150]: 2026/09/21 13:55:21 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls5941lb # [ 105.079198] niks3-server[1150]: 2026/09/21 13:55:21 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5942lb # [ 105.085127] niks3-server[1150]: 2026/09/21 13:55:21 INFO Completed upload id=345943worker1 # [ 105.084410] nix-grpc-daemon[1353]: time=2026-09-21T13:55:22.220Z level=INFO msg="Upload complete. (280ms)"5944lb # [ 105.086895] niks3-server[1150]: 2026/09/21 13:55:21 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo5945worker1 # [ 105.090780] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53682 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=15946worker1 # [ 105.103478] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:22Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=05947worker1 # [ 105.107319] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:22Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=15948worker1 # [ 105.116139] nix-daemon[870]: reaped child process 1376, status = succeeded5949worker1 # [ 105.147830] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:536825950worker1 # [ 105.159829] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:536825951worker1 # [ 105.172453] nix-grpc-daemon[1353]: time=2026-09-21T13:55:22.308Z level=INFO msg="All 1 paths already cached"5952client # copying 1 paths to 'lb:50051'...5953client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...5954worker1 # [ 105.180622] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:536825955worker1 # [ 105.188948] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5956worker1 # [ 105.395392] nix-daemon[870]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)5957worker1 # [ 105.407431] nix-daemon[870]: reaped child process 1406, status = succeeded5958worker1 # [ 105.598629] nix-daemon[870]: accepted connection from pid 1408, user nix-grpc-daemon (trusted)5959worker1 # [ 105.610150] nix-daemon[870]: reaped child process 1411, status = succeeded5960lb # [ 105.631927] niks3-server[1150]: 2026/09/21 13:55:22 INFO Received uploads request method=POST path=/api/pending_closures5961worker1 # [ 105.638986] nix-grpc-daemon[1353]: time=2026-09-21T13:55:22.775Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5962worker1 # [ 105.640578] nix-grpc-daemon[1353]: time=2026-09-21T13:55:22.777Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"5963lb # [ 105.664246] niks3-server[1150]: 2026/09/21 13:55:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5964lb # [ 105.669412] niks3-server[1150]: 2026/09/21 13:55:22 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst5965worker1 # [ 105.677114] nix-grpc-daemon[1353]: time=2026-09-21T13:55:22.813Z level=INFO msg="Uploading 1 narinfos"5966lb # [ 105.677896] niks3-server[1150]: 2026/09/21 13:55:22 INFO Signed narinfos id=35 count=15967lb # [ 105.686130] niks3-server[1150]: 2026/09/21 13:55:22 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls5968lb # [ 105.694404] niks3-server[1150]: 2026/09/21 13:55:22 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5969lb # [ 105.700936] niks3-server[1150]: 2026/09/21 13:55:22 INFO Completed upload id=355970worker1 # [ 105.700031] nix-grpc-daemon[1353]: time=2026-09-21T13:55:22.836Z level=INFO msg="Upload complete. (498ms)"5971lb # [ 105.705121] niks3-server[1150]: 2026/09/21 13:55:22 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo5972worker1 # [ 105.703382] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53682 duration_s=0 paths=1 nar_bytes_in=6405973worker1 # [ 105.708647] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:536825974worker1 # [ 105.717963] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5975worker1 # [ 105.721621] nix-daemon[870]: reaped child process 1401, status = succeeded5976worker1 # [ 105.726549] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5977client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...5978client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5979lb # [ 106.490313] envoy[901]: conn peer=[2001:db8:1::1]:47764 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9782 tx=4190 ms=69545980lb # [ 106.494085] envoy[901]: conn peer=[2001:db8:1::4]:49632 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=5880 tx=648 ms=168735981client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5982client # farm-b> LOG-farm-b-on-standby5983worker1 # [ 107.145764] nix-daemon[870]: reaped child process 1415, status = succeeded5984worker1 # [ 107.341088] nix-daemon[870]: accepted connection from pid 1431, user nix-grpc-daemon (trusted)5985worker1 # [ 107.351545] nix-daemon[870]: reaped child process 1434, status = succeeded5986lb # [ 107.375292] niks3-server[1150]: 2026/09/21 13:55:24 INFO Received uploads request method=POST path=/api/pending_closures5987worker1 # [ 107.384775] nix-grpc-daemon[1353]: time=2026-09-21T13:55:24.521Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5988worker1 # [ 107.386299] nix-grpc-daemon[1353]: time=2026-09-21T13:55:24.522Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"5989lb # [ 107.420251] niks3-server[1150]: 2026/09/21 13:55:24 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst5990lb # [ 107.424493] niks3-server[1150]: 2026/09/21 13:55:24 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5991lb # [ 107.432129] niks3-server[1150]: 2026/09/21 13:55:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5992lb # [ 107.439450] niks3-server[1150]: 2026/09/21 13:55:24 INFO Signed narinfos id=36 count=15993worker1 # [ 107.439211] nix-grpc-daemon[1353]: time=2026-09-21T13:55:24.575Z level=INFO msg="Uploading 1 narinfos"5994lb # [ 107.445955] niks3-server[1150]: 2026/09/21 13:55:24 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls5995lb # [ 107.455137] niks3-server[1150]: 2026/09/21 13:55:24 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5996lb # [ 107.460116] niks3-server[1150]: 2026/09/21 13:55:24 INFO Completed upload id=365997lb # [ 107.462446] niks3-server[1150]: 2026/09/21 13:55:24 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo5998worker1 # [ 107.460704] nix-grpc-daemon[1353]: time=2026-09-21T13:55:24.597Z level=INFO msg="Upload complete. (324ms)"5999worker1 # [ 107.471162] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53682 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=16000worker1 # [ 107.480216] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:24Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=06001worker1 # [ 107.484209] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:24Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=16002worker1 # [ 107.492816] nix-daemon[870]: reaped child process 1413, status = succeeded6003worker1 # [ 107.522076] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:536826004worker1 # [ 107.534802] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:536826005worker1 # [ 107.548106] nix-grpc-daemon[1353]: time=2026-09-21T13:55:24.684Z level=INFO msg="All 2 paths already cached"6006client # copying 1 paths to 'lb:50051'...6007client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...6008worker1 # [ 107.556467] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:536826009worker1 # [ 107.564645] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)6010worker1 # [ 107.771590] nix-daemon[870]: accepted connection from pid 1440, user nix-grpc-daemon (trusted)6011worker1 # [ 107.784150] nix-daemon[870]: reaped child process 1443, status = succeeded6012worker1 # [ 107.969714] nix-daemon[870]: accepted connection from pid 1445, user nix-grpc-daemon (trusted)6013worker1 # [ 107.981566] nix-daemon[870]: reaped child process 1448, status = succeeded6014lb # [ 108.001150] niks3-server[1150]: 2026/09/21 13:55:24 INFO Received uploads request method=POST path=/api/pending_closures6015worker1 # [ 108.009909] nix-grpc-daemon[1353]: time=2026-09-21T13:55:25.146Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6016worker1 # [ 108.011865] nix-grpc-daemon[1353]: time=2026-09-21T13:55:25.148Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6017lb # [ 108.029807] niks3-server[1150]: 2026/09/21 13:55:24 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6018lb # [ 108.040927] niks3-server[1150]: 2026/09/21 13:55:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6019lb # [ 108.047379] niks3-server[1150]: 2026/09/21 13:55:24 INFO Signed narinfos id=37 count=16020worker1 # [ 108.049665] nix-grpc-daemon[1353]: time=2026-09-21T13:55:25.185Z level=INFO msg="Uploading 1 narinfos"6021lb # [ 108.055408] niks3-server[1150]: 2026/09/21 13:55:24 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6022lb # [ 108.066745] niks3-server[1150]: 2026/09/21 13:55:24 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6023lb # [ 108.071428] niks3-server[1150]: 2026/09/21 13:55:24 INFO Completed upload id=376024worker1 # [ 108.073055] nix-grpc-daemon[1353]: time=2026-09-21T13:55:25.209Z level=INFO msg="Upload complete. (494ms)"6025lb # [ 108.074440] niks3-server[1150]: 2026/09/21 13:55:24 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6026worker1 # [ 108.077151] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53682 duration_s=0 paths=1 nar_bytes_in=7686027worker1 # [ 108.081648] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:536826028worker1 # [ 108.093713] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)6029worker1 # [ 108.096193] nix-daemon[870]: reaped child process 1438, status = succeeded6030worker1 # [ 108.101605] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)6031client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6032client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6033client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6034client # farm-top> LOG-farm-top-on-standby6035worker1 # [ 109.314383] nix-daemon[870]: reaped child process 1452, status = succeeded6036worker1 # [ 109.473612] nix-daemon[870]: accepted connection from pid 1468, user nix-grpc-daemon (trusted)6037worker1 # [ 109.485399] nix-daemon[870]: reaped child process 1471, status = succeeded6038lb # [ 109.506932] niks3-server[1150]: 2026/09/21 13:55:26 INFO Received uploads request method=POST path=/api/pending_closures6039worker1 # [ 109.516699] nix-grpc-daemon[1353]: time=2026-09-21T13:55:26.653Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6040worker1 # [ 109.518209] nix-grpc-daemon[1353]: time=2026-09-21T13:55:26.654Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6041lb # [ 109.547912] niks3-server[1150]: 2026/09/21 13:55:26 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6042lb # [ 109.557873] niks3-server[1150]: 2026/09/21 13:55:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6043lb # [ 109.561760] niks3-server[1150]: 2026/09/21 13:55:26 INFO Signed narinfos id=38 count=16044worker1 # [ 109.563288] nix-grpc-daemon[1353]: time=2026-09-21T13:55:26.699Z level=INFO msg="Uploading 1 narinfos"6045lb # [ 109.568834] niks3-server[1150]: 2026/09/21 13:55:26 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6046lb # [ 109.574238] niks3-server[1150]: 2026/09/21 13:55:26 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6047lb # [ 109.578569] niks3-server[1150]: 2026/09/21 13:55:26 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6048lb # [ 109.583141] niks3-server[1150]: 2026/09/21 13:55:26 INFO Completed upload id=386049worker1 # [ 109.582447] nix-grpc-daemon[1353]: time=2026-09-21T13:55:26.718Z level=INFO msg="Upload complete. (274ms)"6050lb # [ 109.585142] niks3-server[1150]: 2026/09/21 13:55:26 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6051worker1 # [ 109.589607] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53682 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16052worker1 # [ 109.600660] nix-daemon[870]: reaped child process 1450, status = succeeded6053client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 7.37 seconds)6054client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath6055client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.24 seconds)6056worker2 # [ 109.940606] systemd[1]: Listening on Nix worker-protocol over gRPC.6057worker2 # [ 109.947121] systemd[1]: Starting scheduler-token.service...6058worker2 # [ 110.022122] systemd[1]: scheduler-token.service: Deactivated successfully.6059worker2 # [ 110.024573] systemd[1]: Finished scheduler-token.service.6060worker2 # [ 110.025381] systemd[1]: scheduler-token.service: Consumed 19ms CPU time over 76ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6061worker2 # [ 110.030313] systemd[1]: Starting Nix worker-protocol over gRPC...6062worker2 # [ 110.174394] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:26Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6063worker2 # [ 110.176296] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:26Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6064worker2 # [ 110.179617] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:26Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16065worker2 # [ 110.187066] nix-grpc-daemon[1274]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6066worker2 # [ 110.188499] nix-grpc-daemon[1274]: E0000 00:00:1789998926.498593 1274 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6067worker2 # [ 110.195112] nix-grpc-daemon[1274]: E0000 00:00:1789998926.503873 1274 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6068worker2 # [ 110.197178] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:26Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6069worker2 # [ 110.198727] systemd[1]: Started Nix worker-protocol over gRPC.6070lb: must succeed: curl -sf localhost:9901/clusters6071worker2 # [ 110.208071] nix-grpc-daemon[1274]: E0000 00:00:1789998926.520201 1315 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6072worker2 # [ 110.221977] nix-grpc-daemon[1274]: E0000 00:00:1789998926.534438 1315 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6073worker2 # [ 110.227053] nix-grpc-daemon[1274]: E0000 00:00:1789998926.536706 1315 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6074worker2 # [ 110.235929] nix-grpc-daemon[1274]: E0000 00:00:1789998926.548393 1315 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6075worker2 # [ 110.246138] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:26Z level=info event=scheduler_connected addr=lb:500516076worker1 # [ 110.261155] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:27Z level=info event=oidc_jwks provider=mock keys=16077worker1 # [ 110.265266] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:27Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:458266078worker1 # [ 110.269154] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:27Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06079lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6080worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6081worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6082lb: must succeed: curl -sf localhost:9901/clusters6083lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6084lb: must succeed: curl -sf localhost:9901/clusters6085lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6086worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6087worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6088(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.86 seconds)6089subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6090client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs6091client # [ 110.627846] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.6092client # Running as unit: rs.service; invocation ID: f851f80e459a4fdb91681266aebe88666093client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.07 seconds)6094worker1 # [ 110.853233] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:536826095client # [ 110.866821] nix[1202]: don't know how to build these paths:6096client # [ 110.868392] nix[1202]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6097worker1 # [ 110.869676] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:28Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06098worker1 # [ 110.874835] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:28Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=16099worker1 # [ 110.914365] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:536826100worker1 # [ 110.928489] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:536826101client # [ 110.946296] nix[1202]: copying 1 paths to 'lb:50051'...6102client # [ 110.949118] nix[1202]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6103worker1 # [ 110.942629] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:536826104worker1 # [ 110.953330] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)6105worker1 # [ 111.142327] nix-daemon[870]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6106worker1 # [ 111.154100] nix-daemon[870]: reaped child process 1487, status = succeeded6107worker1 # [ 111.330578] nix-daemon[870]: accepted connection from pid 1489, user nix-grpc-daemon (trusted)6108worker1 # [ 111.342319] nix-daemon[870]: reaped child process 1492, status = succeeded6109lb # [ 111.364901] niks3-server[1150]: 2026/09/21 13:55:28 INFO Received uploads request method=POST path=/api/pending_closures6110worker1 # [ 111.371131] nix-grpc-daemon[1353]: time=2026-09-21T13:55:28.507Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6111worker1 # [ 111.372934] nix-grpc-daemon[1353]: time=2026-09-21T13:55:28.509Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6112lb # [ 111.391292] niks3-server[1150]: 2026/09/21 13:55:28 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6113lb # [ 111.403468] niks3-server[1150]: 2026/09/21 13:55:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6114worker1 # [ 111.407239] nix-grpc-daemon[1353]: time=2026-09-21T13:55:28.543Z level=INFO msg="Uploading 1 narinfos"6115lb # [ 111.408135] niks3-server[1150]: 2026/09/21 13:55:28 INFO Signed narinfos id=39 count=16116lb # [ 111.414989] niks3-server[1150]: 2026/09/21 13:55:28 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6117lb # [ 111.424495] niks3-server[1150]: 2026/09/21 13:55:28 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6118lb # [ 111.431506] niks3-server[1150]: 2026/09/21 13:55:28 INFO Completed upload id=396119worker1 # [ 111.430727] nix-grpc-daemon[1353]: time=2026-09-21T13:55:28.567Z level=INFO msg="Upload complete. (465ms)"6120lb # [ 111.435884] niks3-server[1150]: 2026/09/21 13:55:28 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6121worker1 # [ 111.434705] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53682 duration_s=0 paths=1 nar_bytes_in=4166122worker1 # [ 111.439130] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:536826123worker1 # [ 111.448109] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)6124worker1 # [ 111.455167] nix-daemon[870]: reaped child process 1482, status = succeeded6125worker1 # [ 111.460279] nix-daemon[870]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)6126client # [ 111.467498] nix[1202]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6127lb: must succeed: curl -sf localhost:9901/clusters6128lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6129lb: must succeed: curl -sf localhost:9901/clusters6130lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6131lb: must succeed: curl -sf localhost:9901/clusters6132lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6133lb: must succeed: curl -sf localhost:9901/clusters6134lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6135worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6136worker2 # [ 113.557857] systemd-journald[286]: Received client request to rotate journal, rotating.6137worker2 # [ 113.575315] systemd-journald[286]: Vacuuming done, freed 0B of archived journals from /var/log/journal/b26a9da5a8f54e678a600ccc19ce9c82.6138worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.06 seconds)6139worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service6140worker1 # [ 113.499159] systemd[1]: Stopping Nix worker-protocol over gRPC...6141worker1 # [ 113.504082] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:30Z level=info event=draining6142worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)6143client: must succeed: sleep 5.06144lb # [ 116.490936] envoy[901]: conn peer=[2001:db8:1::1]:50318 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9659 tx=4104 ms=70696145client: (finished: must succeed: sleep 5.0, in 5.05 seconds)6146worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6147worker1 # [ 118.891809] nix-daemon[870]: reaped child process 1496, status = succeeded6148worker1 # [ 119.074369] nix-daemon[870]: accepted connection from pid 1537, user nix-grpc-daemon (trusted)6149worker1 # [ 119.085105] nix-daemon[870]: reaped child process 1540, status = succeeded6150lb # [ 119.106782] niks3-server[1150]: 2026/09/21 13:55:36 INFO Received uploads request method=POST path=/api/pending_closures6151worker1 # [ 119.115374] nix-grpc-daemon[1353]: time=2026-09-21T13:55:36.251Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6152worker1 # [ 119.116884] nix-grpc-daemon[1353]: time=2026-09-21T13:55:36.253Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6153lb # [ 119.145086] niks3-server[1150]: 2026/09/21 13:55:36 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6154lb # [ 119.161756] niks3-server[1150]: 2026/09/21 13:55:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6155worker1 # [ 119.169057] nix-grpc-daemon[1353]: time=2026-09-21T13:55:36.305Z level=INFO msg="Uploading 1 narinfos"6156lb # [ 119.169840] niks3-server[1150]: 2026/09/21 13:55:36 INFO Signed narinfos id=40 count=16157lb # [ 119.177807] niks3-server[1150]: 2026/09/21 13:55:36 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6158lb # [ 119.183820] niks3-server[1150]: 2026/09/21 13:55:36 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6159lb # [ 119.191960] niks3-server[1150]: 2026/09/21 13:55:36 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6160lb # [ 119.194838] niks3-server[1150]: 2026/09/21 13:55:36 INFO Completed upload id=406161worker1 # [ 119.195461] nix-grpc-daemon[1353]: time=2026-09-21T13:55:36.331Z level=INFO msg="Upload complete. (321ms)"6162lb # [ 119.198382] niks3-server[1150]: 2026/09/21 13:55:36 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6163worker1 # [ 119.203614] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53682 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16164worker1 # [ 119.217107] nix-daemon[870]: reaped child process 1494, status = succeeded6165client # [ 119.252622] systemd[1]: rs.service: Deactivated successfully.6166client # [ 119.256375] systemd[1]: rs.service: Consumed 138ms CPU time over 8.627s wall clock time, 43.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6167worker1 # [ 119.504895] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:36Z level=info event=signal_exit6168lb # [ 120.046075] niks3-server[1150]: 2026/09/21 13:55:36 INFO lead: released remote=192.168.1.3:513006169lb # [ 120.213776] niks3-server[1150]: 2026/09/21 13:55:37 INFO lead: acquired remote=192.168.1.4:469646170worker2 # [ 120.206732] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:36Z level=info event=scheduler_take_over6171worker2 # [ 121.038927] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:37Z level=info event=scheduler_restarting addr=lb:500516172worker1 # [ 121.049689] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:38Z level=info event=scheduler_restarting6173worker1 # [ 121.052618] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:38Z level=info event=scheduler_restarting addr=lb:500516174worker1 # [ 121.056187] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:38Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:458266175worker1 # [ 121.059799] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:38Z level=info event=worker_gone addr=192.168.1.4:500516176worker1 # [ 121.063384] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:38Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:458266177worker1 # [ 121.066988] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:38Z level=info event=worker_gone addr=192.168.1.3:500516178worker1 # [ 121.085627] nix-daemon[870]: reaped child process 1332, status = succeeded6179worker2 # [ 121.241420] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:37Z level=info event=scheduler_connected addr=lb:500516180worker1 # [ 121.258294] nix-grpc-daemon[1280]: ts=2026-09-21T13:55:38Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6181worker2 # [ 121.254133] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:37Z level=info event=oidc_jwks provider=mock keys=16182worker2 # [ 121.258188] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:37Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:460406183worker2 # [ 121.261936] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:37Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06184worker1 # [ 121.987143] nix-grpc-daemon[1280]: E0000 00:00:1789998939.123402 1325 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true6185worker1 # [ 121.991820] nix-grpc-daemon[1280]: E0000 00:00:1789998939.123546 1325 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6186worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.22 seconds)6187client: waiting for success: ! systemctl is-active rs6188client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)6189client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6190client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.04 seconds)6191client: must fail: journalctl -u rs | grep -q 'reconnecting'6192client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)6193lb: must succeed: curl -sf localhost:9901/clusters6194lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6195worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6196worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6197worker1 # [ 122.406553] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6198worker1 # [ 122.410415] systemd[1]: Stopped Nix worker-protocol over gRPC.6199worker1 # [ 122.412458] systemd[1]: nix-grpc-daemon.service: Consumed 1.323s CPU time over 34.119s wall clock time, 35.2M memory peak, 46K incoming IP traffic, 90.8K outgoing IP traffic.6200worker1 # [ 122.428300] systemd[1]: Starting Nix worker-protocol over gRPC...6201worker1 # [ 122.612349] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:39Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6202worker1 # [ 122.614896] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:39Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6203worker1 # [ 122.618606] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:39Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16204worker1 # [ 122.626063] nix-grpc-daemon[1547]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6205worker1 # [ 122.627388] nix-grpc-daemon[1547]: E0000 00:00:1789998939.762531 1547 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6206worker1 # [ 122.632985] nix-grpc-daemon[1547]: E0000 00:00:1789998939.767762 1547 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6207worker1 # [ 122.636167] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:39Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6208worker1 # [ 122.638847] systemd[1]: Started Nix worker-protocol over gRPC.6209worker1 # [ 122.644988] nix-grpc-daemon[1547]: E0000 00:00:1789998939.780485 1588 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6210worker1 # [ 122.656039] nix-grpc-daemon[1547]: E0000 00:00:1789998939.792505 1588 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6211worker1 # [ 122.661102] nix-grpc-daemon[1547]: E0000 00:00:1789998939.794846 1588 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6212worker1 # [ 122.670335] nix-grpc-daemon[1547]: E0000 00:00:1789998939.806858 1588 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6213worker1 # [ 122.679896] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:39Z level=info event=scheduler_connected addr=lb:500516214worker2 # [ 122.676270] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:38Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:460406215worker2 # [ 122.680565] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:38Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06216lb: must succeed: curl -sf localhost:9901/clusters6217lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6218worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6219worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6220(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.91 seconds)6221subtest: scheduler restart mid-build: build finishes, no double build6222client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid6223client # [ 123.541058] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.6224client # Running as unit: mid.service; invocation ID: 2dc597611fb64e1ab773b45d687013456225client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)6226worker1 # [ 123.776676] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:391186227worker1 # [ 123.788200] nix-daemon[870]: accepted connection from pid 1547, user nix-grpc-daemon (trusted)6228client # [ 123.808265] nix[1238]: don't know how to build these paths:6229client # [ 123.809286] nix[1238]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6230worker2 # [ 123.807842] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:40Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06231worker2 # [ 123.811767] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:40Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=16232worker2 # [ 123.859669] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:338626233worker2 # [ 123.870324] nix-daemon[853]: accepted connection from pid 1274, user nix-grpc-daemon (trusted)6234worker2 # [ 123.888442] nix-grpc-daemon[1274]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 24 ms (attempt 1/5)6235worker2 # [ 123.918178] nix-grpc-daemon[1274]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 34 ms (attempt 2/5)6236worker2 # [ 123.957704] nix-grpc-daemon[1274]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 233 ms (attempt 3/5)6237worker2 # [ 124.197293] nix-grpc-daemon[1274]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 95 ms (attempt 4/5)6238worker2 # [ 124.292804] nix-grpc-daemon[1274]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6239worker2 # [ 124.296718] nix-grpc-daemon[1274]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6240worker2 # [ 124.315590] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:338626241client # [ 124.340210] nix[1238]: copying 1 paths to 'lb:50051'...6242worker2 # [ 124.327752] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:338626243client # [ 124.344160] nix[1238]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6244worker2 # [ 124.340165] nix-daemon[853]: accepted connection from pid 1274, user nix-grpc-daemon (trusted)6245worker2 # [ 124.361125] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:40Z level=info event=niks3_push_started pid=14026246worker2 # [ 124.564317] nix-grpc-daemon[1402]: time=2026-09-21T13:55:40.876Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6247worker2 # [ 124.739464] nix-daemon[853]: accepted connection from pid 1415, user nix-grpc-daemon (trusted)6248worker2 # [ 124.752949] nix-daemon[853]: reaped child process 1418, status = succeeded6249worker2 # [ 124.923544] nix-daemon[853]: accepted connection from pid 1424, user nix-grpc-daemon (trusted)6250worker2 # [ 124.935576] nix-daemon[853]: reaped child process 1427, status = succeeded6251lb # [ 124.960334] niks3-server[1150]: 2026/09/21 13:55:41 INFO Received uploads request method=POST path=/api/pending_closures6252worker2 # [ 124.961812] nix-grpc-daemon[1402]: time=2026-09-21T13:55:41.274Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6253worker2 # [ 124.963292] nix-grpc-daemon[1402]: time=2026-09-21T13:55:41.275Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6254lb # [ 124.995929] niks3-server[1150]: 2026/09/21 13:55:41 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6255lb # [ 125.003969] niks3-server[1150]: 2026/09/21 13:55:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6256lb # [ 125.008566] niks3-server[1150]: 2026/09/21 13:55:41 INFO Signed narinfos id=41 count=16257worker2 # [ 125.004218] nix-grpc-daemon[1402]: time=2026-09-21T13:55:41.316Z level=INFO msg="Uploading 1 narinfos"6258lb # [ 125.019130] niks3-server[1150]: 2026/09/21 13:55:41 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6259lb # [ 125.028322] niks3-server[1150]: 2026/09/21 13:55:41 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6260worker2 # [ 125.029080] nix-grpc-daemon[1402]: time=2026-09-21T13:55:41.341Z level=INFO msg="Upload complete. (460ms)"6261lb # [ 125.037077] niks3-server[1150]: 2026/09/21 13:55:41 INFO Completed upload id=416262worker2 # [ 125.031368] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33862 duration_s=0 paths=1 nar_bytes_in=4246263worker2 # [ 125.034759] nix-grpc-daemon[1274]: ts=2026-09-21T13:55:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:338626264worker2 # [ 125.041133] nix-daemon[853]: accepted connection from pid 1274, user nix-grpc-daemon (trusted)6265lb # [ 125.049210] niks3-server[1150]: 2026/09/21 13:55:41 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6266worker2 # [ 125.044092] nix-daemon[853]: reaped child process 1400, status = succeeded6267worker2 # [ 125.048953] nix-daemon[853]: accepted connection from pid 1274, user nix-grpc-daemon (trusted)6268client # [ 125.062894] nix[1238]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6269lb: must succeed: curl -sf localhost:9901/clusters6270lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6271lb: must succeed: curl -sf localhost:9901/clusters6272lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6273worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6274worker2 # [ 126.210612] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1274 (nix-grpc-daemon) on client request.6275worker2 # [ 126.216276] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6276lb # [ 126.232048] niks3-server[1150]: 2026/09/21 13:55:43 INFO lead: released remote=192.168.1.4:469646277client # [ 126.248642] nix[1238]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6278worker1 # [ 126.239731] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:43Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6279worker2 # [ 126.239179] systemd[1]: Starting scheduler-token.service...6280client # [ 126.252657] nix[1238]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6281worker2 # [ 126.249386] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6282worker2 # [ 126.252192] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6283worker2 # [ 126.253442] systemd[1]: nix-grpc-daemon.service: Consumed 440ms CPU time over 16.218s wall clock time, 33.8M memory peak, 15K incoming IP traffic, 27.3K outgoing IP traffic.6284worker2 # [ 126.260486] nix-daemon[853]: reaped child process 1429, status = succeeded6285worker2 # [ 126.262561] nix-daemon[853]: reaped child process 1394, status = succeeded6286worker2 # [ 126.272299] nix-daemon[1431]: unexpected Nix daemon error: error: interrupted by the user6287worker2 # [ 126.276663] nix-daemon[853]: reaped child process 1431, status = failed with exit code 16288worker2 # [ 126.334839] systemd[1]: scheduler-token.service: Deactivated successfully.6289worker2 # [ 126.336161] systemd[1]: Finished scheduler-token.service.6290worker2 # [ 126.337757] systemd[1]: scheduler-token.service: Consumed 18ms CPU time over 95ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6291worker2 # [ 126.340170] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6292worker2 # [ 126.345095] systemd[1]: Starting scheduler-token.service...6293worker2 # [ 126.411706] systemd[1]: scheduler-token.service: Deactivated successfully.6294worker2 # [ 126.413270] systemd[1]: Finished scheduler-token.service.6295worker2 # [ 126.414564] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 68ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6296worker2 # [ 126.420140] systemd[1]: Starting Nix worker-protocol over gRPC...6297worker1 # [ 126.439928] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:43Z level=info event=scheduler_connected addr=lb:500516298lb # [ 126.492126] envoy[901]: conn peer=[2001:db8:1::1]:57644 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2973 tx=1403 ms=83606299lb # [ 126.495795] envoy[901]: conn peer=[2001:db8:1::3]:58466 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2392 tx=752 ms=327116300lb # [ 126.499498] envoy[901]: conn peer=[2001:db8:1::4]:41850 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=2046 tx=298 ms=159916301worker2 # [ 126.583756] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:42Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6302worker2 # [ 126.586150] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:42Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6303worker2 # [ 126.589844] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:42Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16304worker2 # [ 126.597065] nix-grpc-daemon[1470]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6305worker2 # [ 126.599875] nix-grpc-daemon[1470]: E0000 00:00:1789998942.909294 1470 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6306worker2 # [ 126.604103] nix-grpc-daemon[1470]: E0000 00:00:1789998942.912218 1470 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6307worker2 # [ 126.609949] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:42Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6308worker2 # [ 126.612588] systemd[1]: Started Nix worker-protocol over gRPC.6309worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.44 seconds)6310client: must succeed: sleep 5.06311worker2 # [ 126.619951] nix-grpc-daemon[1470]: E0000 00:00:1789998942.932464 1511 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6312lb # [ 126.628632] niks3-server[1150]: 2026/09/21 13:55:43 INFO lead: acquired remote=192.168.1.4:437646313worker2 # [ 126.624070] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:42Z level=info event=scheduler_take_over6314worker2 # [ 126.627138] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:42Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:338786315worker2 # [ 126.629669] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:42Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06316worker2 # [ 126.639987] nix-grpc-daemon[1470]: E0000 00:00:1789998942.952445 1511 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6317worker2 # [ 126.644050] nix-grpc-daemon[1470]: E0000 00:00:1789998942.954773 1511 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6318worker2 # [ 126.654270] nix-grpc-daemon[1470]: E0000 00:00:1789998942.966723 1511 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6319worker2 # [ 126.663182] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:42Z level=info event=scheduler_connected addr=lb:500516320worker2 # [ 126.668801] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:42Z level=info event=oidc_jwks provider=mock keys=16321worker2 # [ 126.670178] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:42Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:338786322worker2 # [ 126.671767] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:42Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06323worker2 # [ 126.743586] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:43Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06324worker2 # [ 126.747780] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:43Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16325worker1 # [ 126.795159] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:391186326worker1 # [ 126.807705] nix-grpc-daemon[1547]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 44 ms (attempt 1/5)6327worker1 # [ 126.857444] nix-grpc-daemon[1547]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 102 ms (attempt 2/5)6328worker1 # [ 126.965339] nix-grpc-daemon[1547]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 334 ms (attempt 3/5)6329worker1 # [ 127.306215] nix-grpc-daemon[1547]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 371 ms (attempt 4/5)6330worker1 # [ 127.677839] nix-grpc-daemon[1547]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6331worker1 # [ 127.681865] nix-grpc-daemon[1547]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6332worker1 # [ 127.703677] nix-grpc-daemon[1547]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' from 'http://lb:5751'...6333worker1 # [ 127.725560] nix-daemon[870]: accepted connection from pid 1547, user nix-grpc-daemon (trusted)6334worker1 # [ 127.741383] nix-daemon[870]: accepted connection from pid 1547, user nix-grpc-daemon (trusted)6335client # [ 127.748606] nix[1238]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6336client: (finished: must succeed: sleep 5.0, in 5.05 seconds)6337client: waiting for success: ! systemctl is-active mid6338worker1 # [ 131.961953] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:49Z level=info event=niks3_push_started pid=16496339worker1 # [ 131.985756] nix-daemon[870]: reaped child process 1624, status = succeeded6340worker1 # [ 132.185809] nix-grpc-daemon[1649]: time=2026-09-21T13:55:49.321Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6341worker1 # [ 132.362656] nix-daemon[870]: accepted connection from pid 1662, user nix-grpc-daemon (trusted)6342worker1 # [ 132.374539] nix-daemon[870]: reaped child process 1665, status = succeeded6343lb # [ 132.393670] niks3-server[1150]: 2026/09/21 13:55:49 INFO Received uploads request method=POST path=/api/pending_closures6344worker1 # [ 132.401646] nix-grpc-daemon[1649]: time=2026-09-21T13:55:49.538Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6345worker1 # [ 132.403107] nix-grpc-daemon[1649]: time=2026-09-21T13:55:49.539Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6346lb # [ 132.438284] niks3-server[1150]: 2026/09/21 13:55:49 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6347lb # [ 132.453878] niks3-server[1150]: 2026/09/21 13:55:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6348lb # [ 132.457310] niks3-server[1150]: 2026/09/21 13:55:49 INFO Signed narinfos id=42 count=16349worker1 # [ 132.458634] nix-grpc-daemon[1649]: time=2026-09-21T13:55:49.595Z level=INFO msg="Uploading 1 narinfos"6350lb # [ 132.472559] niks3-server[1150]: 2026/09/21 13:55:49 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6351lb # [ 132.478197] niks3-server[1150]: 2026/09/21 13:55:49 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6352lb # [ 132.480585] niks3-server[1150]: 2026/09/21 13:55:49 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6353lb # [ 132.483390] niks3-server[1150]: 2026/09/21 13:55:49 INFO Completed upload id=426354worker1 # [ 132.482447] nix-grpc-daemon[1649]: time=2026-09-21T13:55:49.618Z level=INFO msg="Upload complete. (292ms)"6355lb # [ 132.485339] niks3-server[1150]: 2026/09/21 13:55:49 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6356worker1 # [ 132.488123] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39118 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16357worker1 # [ 132.497878] nix-daemon[870]: reaped child process 1622, status = succeeded6358client # [ 132.527323] systemd[1]: mid.service: Deactivated successfully.6359client # [ 132.528921] systemd[1]: mid.service: Consumed 143ms CPU time over 8.988s wall clock time, 41.4M memory peak, 8.7K incoming IP traffic, 10.4K outgoing IP traffic.6360client: (finished: waiting for success: ! systemctl is-active mid, in 1.08 seconds)6361client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6362client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds)6363(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 9.58 seconds)6364subtest: interrupting the client stops the build on the worker6365client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6366client # [ 133.086795] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6367client # Running as unit: intr.service; invocation ID: 2f5de3131b704e9c9783e9fc80a8e8166368client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.04 seconds)6369worker2 # [ 133.322251] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:355206370worker2 # [ 133.333826] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6371client # [ 133.360161] nix[1274]: don't know how to build these paths:6372client # [ 133.361291] nix[1274]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6373worker2 # [ 133.355071] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:49Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06374worker2 # [ 133.359351] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:49Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16375worker2 # [ 133.409622] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355206376worker2 # [ 133.421909] nix-grpc-daemon[1470]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 55 ms (attempt 1/5)6377worker2 # [ 133.482757] nix-grpc-daemon[1470]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 120 ms (attempt 2/5)6378worker2 # [ 133.608664] nix-grpc-daemon[1470]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 228 ms (attempt 3/5)6379worker2 # [ 133.842567] nix-grpc-daemon[1470]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 141 ms (attempt 4/5)6380worker2 # [ 133.989192] nix-grpc-daemon[1470]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6381worker2 # [ 133.991454] nix-grpc-daemon[1470]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6382worker2 # [ 134.004256] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:355206383client # [ 134.025121] nix[1274]: copying 1 paths to 'lb:50051'...6384worker2 # [ 134.011672] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:355206385client # [ 134.025928] nix[1274]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6386worker2 # [ 134.017818] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6387worker2 # [ 134.029178] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:50Z level=info event=niks3_push_started pid=15466388worker2 # [ 134.222769] nix-grpc-daemon[1546]: time=2026-09-21T13:55:50.534Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6389worker2 # [ 134.392246] nix-daemon[853]: accepted connection from pid 1559, user nix-grpc-daemon (trusted)6390worker2 # [ 134.402952] nix-daemon[853]: reaped child process 1566, status = succeeded6391worker2 # [ 134.569568] nix-daemon[853]: accepted connection from pid 1568, user nix-grpc-daemon (trusted)6392worker2 # [ 134.581263] nix-daemon[853]: reaped child process 1571, status = succeeded6393lb # [ 134.605725] niks3-server[1150]: 2026/09/21 13:55:51 INFO Received uploads request method=POST path=/api/pending_closures6394worker2 # [ 134.610513] nix-grpc-daemon[1546]: time=2026-09-21T13:55:50.923Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6395worker2 # [ 134.613620] nix-grpc-daemon[1546]: time=2026-09-21T13:55:50.924Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6396lb # [ 134.643246] niks3-server[1150]: 2026/09/21 13:55:51 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6397lb # [ 134.651818] niks3-server[1150]: 2026/09/21 13:55:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6398worker2 # [ 134.653591] nix-grpc-daemon[1546]: time=2026-09-21T13:55:50.965Z level=INFO msg="Uploading 1 narinfos"6399lb # [ 134.661172] niks3-server[1150]: 2026/09/21 13:55:51 INFO Signed narinfos id=43 count=16400lb # [ 134.671114] niks3-server[1150]: 2026/09/21 13:55:51 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6401lb # [ 134.677401] niks3-server[1150]: 2026/09/21 13:55:51 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6402lb # [ 134.685356] niks3-server[1150]: 2026/09/21 13:55:51 INFO Completed upload id=436403lb # [ 134.687762] niks3-server[1150]: 2026/09/21 13:55:51 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6404worker2 # [ 134.678288] nix-grpc-daemon[1546]: time=2026-09-21T13:55:50.990Z level=INFO msg="Upload complete. (451ms)"6405worker2 # [ 134.682745] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=0 paths=1 nar_bytes_in=4246406worker2 # [ 134.688683] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355206407worker2 # [ 134.697627] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6408client # [ 134.719257] nix[1274]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6409worker2 # [ 134.705934] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6410worker2 # [ 134.708065] nix-daemon[853]: reaped child process 1544, status = succeeded6411client: must succeed: systemctl kill -s INT intr6412client # [ 135.520700] systemd[1]: intr.service: Sent signal SIGINT to main process 1274 (nix) on client request.6413worker2 # [ 135.508621] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:51Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6414client # [ 135.526442] nix[1274]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6415client: (finished: must succeed: systemctl kill -s INT intr, in 0.06 seconds)6416client # [ 135.534191] nix[1274]: error: interrupted by the user6417worker2 # [ 135.522936] nix-daemon[1575]: unexpected Nix daemon error:6418worker2 # [ 135.524341] nix-daemon[853]: reaped child process 1573, status = succeeded6419worker2 # [ 135.526121] nix-daemon[1575]: error: interrupted by the user6420worker2 # [ 135.529281] nix-daemon[853]: reaped child process 1575, status = failed with exit code 16421client # [ 135.568603] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6422client # [ 135.570199] systemd[1]: intr.service: Failed with result 'exit-code'.6423client # [ 135.572251] systemd[1]: intr.service: Consumed 123ms CPU time over 2.484s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.2K outgoing IP traffic.6424(finished: subtest: interrupting the client stops the build on the worker, in 2.58 seconds)6425subtest: a deploy mid-build drains: the build finishes, the new generation takes over6426client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6427client # [ 135.676841] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6428client # Running as unit: sw.service; invocation ID: 49482dd502bd4c41b20e1f825cd36a8d6429client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.04 seconds)6430client # [ 135.927839] nix[1295]: don't know how to build these paths:6431worker1 # [ 135.919172] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:391186432client # [ 135.929552] nix[1295]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6433worker2 # [ 135.924796] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:52Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06434worker2 # [ 135.928803] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:52Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16435worker1 # [ 135.974763] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:391186436worker1 # [ 135.987658] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:391186437client # [ 136.005208] nix[1295]: copying 1 paths to 'lb:50051'...6438client # [ 136.008067] nix[1295]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6439worker1 # [ 136.001554] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:391186440worker1 # [ 136.010704] nix-daemon[870]: accepted connection from pid 1547, user nix-grpc-daemon (trusted)6441worker1 # [ 136.228923] nix-daemon[870]: accepted connection from pid 1689, user nix-grpc-daemon (trusted)6442worker1 # [ 136.240858] nix-daemon[870]: reaped child process 1692, status = succeeded6443worker1 # [ 136.437657] nix-daemon[870]: accepted connection from pid 1694, user nix-grpc-daemon (trusted)6444worker1 # [ 136.449282] nix-daemon[870]: reaped child process 1697, status = succeeded6445lb # [ 136.471890] niks3-server[1150]: 2026/09/21 13:55:53 INFO Received uploads request method=POST path=/api/pending_closures6446worker1 # [ 136.480415] nix-grpc-daemon[1649]: time=2026-09-21T13:55:53.616Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6447worker1 # [ 136.481901] nix-grpc-daemon[1649]: time=2026-09-21T13:55:53.616Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6448lb # [ 136.493629] envoy[901]: conn peer=[2001:db8:1::1]:48178 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3921 tx=1854 ms=87266449lb # [ 136.499077] envoy[901]: conn peer=[2001:db8:1::1]:58034 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2967 tx=853 ms=22166450lb # [ 136.511492] niks3-server[1150]: 2026/09/21 13:55:53 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6451lb # [ 136.520959] niks3-server[1150]: 2026/09/21 13:55:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6452worker1 # [ 136.524875] nix-grpc-daemon[1649]: time=2026-09-21T13:55:53.661Z level=INFO msg="Uploading 1 narinfos"6453lb # [ 136.526170] niks3-server[1150]: 2026/09/21 13:55:53 INFO Signed narinfos id=44 count=16454lb # [ 136.532345] niks3-server[1150]: 2026/09/21 13:55:53 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6455lb # [ 136.540363] niks3-server[1150]: 2026/09/21 13:55:53 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6456lb # [ 136.546682] niks3-server[1150]: 2026/09/21 13:55:53 INFO Completed upload id=446457worker1 # [ 136.545896] nix-grpc-daemon[1649]: time=2026-09-21T13:55:53.682Z level=INFO msg="Upload complete. (520ms)"6458lb # [ 136.549512] niks3-server[1150]: 2026/09/21 13:55:53 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6459worker1 # [ 136.550141] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39118 duration_s=0 paths=1 nar_bytes_in=4166460worker1 # [ 136.557527] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:391186461worker1 # [ 136.565247] nix-daemon[870]: accepted connection from pid 1547, user nix-grpc-daemon (trusted)6462worker1 # [ 136.574037] nix-daemon[870]: reaped child process 1687, status = succeeded6463worker1 # [ 136.579144] nix-daemon[870]: accepted connection from pid 1547, user nix-grpc-daemon (trusted)6464client # [ 136.586226] nix[1295]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6465worker1: must succeed: systemctl reload nix-grpc-daemon.service6466worker1 # [ 138.239269] systemd[1]: Reloading Nix worker-protocol over gRPC...6467worker1 # [ 138.288052] systemd[1]: Reloaded Nix worker-protocol over gRPC.6468worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.09 seconds)6469worker1: must succeed: pgrep -f 'read -t [9]0 x'6470worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)6471client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26472client # don't know how to build these paths:6473worker2 # [ 138.611226] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:355206474client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6475worker1 # [ 138.621635] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:391186476worker1 # [ 138.637115] nix-grpc-daemon[1547]: ts=2026-09-21T13:55:55Z level=info event=draining6477worker2 # [ 138.634142] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:54Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06478worker2 # [ 138.639559] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:54Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16479worker2 # [ 138.687744] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355206480worker2 # [ 138.700323] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:355206481client # copying 1 paths to 'lb:50051'...6482worker2 # [ 138.712938] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:355206483client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6484worker2 # [ 138.734906] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6485worker2 # [ 138.932054] nix-daemon[853]: accepted connection from pid 1622, user nix-grpc-daemon (trusted)6486worker2 # [ 138.944296] nix-daemon[853]: reaped child process 1625, status = succeeded6487worker2 # [ 139.103496] nix-daemon[853]: accepted connection from pid 1627, user nix-grpc-daemon (trusted)6488worker2 # [ 139.116810] nix-daemon[853]: reaped child process 1630, status = succeeded6489lb # [ 139.145490] niks3-server[1150]: 2026/09/21 13:55:56 INFO Received uploads request method=POST path=/api/pending_closures6490worker2 # [ 139.148341] nix-grpc-daemon[1546]: time=2026-09-21T13:55:55.460Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6491worker2 # [ 139.149885] nix-grpc-daemon[1546]: time=2026-09-21T13:55:55.462Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6492lb # [ 139.175774] niks3-server[1150]: 2026/09/21 13:55:56 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6493lb # [ 139.188571] niks3-server[1150]: 2026/09/21 13:55:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6494lb # [ 139.194107] niks3-server[1150]: 2026/09/21 13:55:56 INFO Signed narinfos id=45 count=16495worker2 # [ 139.190327] nix-grpc-daemon[1546]: time=2026-09-21T13:55:55.502Z level=INFO msg="Uploading 1 narinfos"6496lb # [ 139.202447] niks3-server[1150]: 2026/09/21 13:55:56 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6497lb # [ 139.212142] niks3-server[1150]: 2026/09/21 13:55:56 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6498lb # [ 139.217246] niks3-server[1150]: 2026/09/21 13:55:56 INFO Completed upload id=456499worker2 # [ 139.210238] nix-grpc-daemon[1546]: time=2026-09-21T13:55:55.522Z level=INFO msg="Upload complete. (461ms)"6500lb # [ 139.221811] niks3-server[1150]: 2026/09/21 13:55:56 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6501worker2 # [ 139.214160] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=0 paths=1 nar_bytes_in=5286502worker2 # [ 139.221098] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355206503worker2 # [ 139.229147] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6504worker2 # [ 139.238642] nix-daemon[853]: reaped child process 1620, status = succeeded6505client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6506worker2 # [ 139.244182] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6507client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6508client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6509client # farm-a> LOG-farm-a-during-drain6510worker2 # [ 140.856946] nix-daemon[853]: reaped child process 1634, status = succeeded6511worker2 # [ 141.037882] nix-daemon[853]: accepted connection from pid 1650, user nix-grpc-daemon (trusted)6512worker2 # [ 141.049873] nix-daemon[853]: reaped child process 1653, status = succeeded6513lb # [ 141.079716] niks3-server[1150]: 2026/09/21 13:55:57 INFO Received uploads request method=POST path=/api/pending_closures6514worker2 # [ 141.083947] nix-grpc-daemon[1546]: time=2026-09-21T13:55:57.396Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6515worker2 # [ 141.086247] nix-grpc-daemon[1546]: time=2026-09-21T13:55:57.398Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6516lb # [ 141.121919] niks3-server[1150]: 2026/09/21 13:55:58 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6517lb # [ 141.134098] niks3-server[1150]: 2026/09/21 13:55:58 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6518lb # [ 141.139329] niks3-server[1150]: 2026/09/21 13:55:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6519lb # [ 141.141941] niks3-server[1150]: 2026/09/21 13:55:58 INFO Signed narinfos id=46 count=16520worker2 # [ 141.136606] nix-grpc-daemon[1546]: time=2026-09-21T13:55:57.448Z level=INFO msg="Uploading 1 narinfos"6521lb # [ 141.148526] niks3-server[1150]: 2026/09/21 13:55:58 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6522lb # [ 141.153069] niks3-server[1150]: 2026/09/21 13:55:58 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6523lb # [ 141.156062] niks3-server[1150]: 2026/09/21 13:55:58 INFO Completed upload id=466524worker2 # [ 141.149119] nix-grpc-daemon[1546]: time=2026-09-21T13:55:57.461Z level=INFO msg="Upload complete. (298ms)"6525lb # [ 141.159254] niks3-server[1150]: 2026/09/21 13:55:58 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6526worker2 # [ 141.153783] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16527worker2 # [ 141.164990] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:57Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06528worker2 # [ 141.167542] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:57Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16529worker2 # [ 141.172612] nix-daemon[853]: reaped child process 1632, status = succeeded6530worker2 # [ 141.207889] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355206531worker2 # [ 141.214880] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:355206532worker2 # [ 141.225417] nix-grpc-daemon[1546]: time=2026-09-21T13:55:57.537Z level=INFO msg="All 1 paths already cached"6533client # copying 1 paths to 'lb:50051'...6534client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6535worker2 # [ 141.232831] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:355206536worker2 # [ 141.247170] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6537worker2 # [ 141.445612] nix-daemon[853]: accepted connection from pid 1659, user nix-grpc-daemon (trusted)6538worker2 # [ 141.457711] nix-daemon[853]: reaped child process 1662, status = succeeded6539worker2 # [ 141.654512] nix-daemon[853]: accepted connection from pid 1664, user nix-grpc-daemon (trusted)6540worker2 # [ 141.667703] nix-daemon[853]: reaped child process 1667, status = succeeded6541lb # [ 141.694486] niks3-server[1150]: 2026/09/21 13:55:58 INFO Received uploads request method=POST path=/api/pending_closures6542worker2 # [ 141.698052] nix-grpc-daemon[1546]: time=2026-09-21T13:55:58.010Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6543worker2 # [ 141.699423] nix-grpc-daemon[1546]: time=2026-09-21T13:55:58.010Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6544lb # [ 141.726175] niks3-server[1150]: 2026/09/21 13:55:58 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6545lb # [ 141.734952] niks3-server[1150]: 2026/09/21 13:55:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6546worker2 # [ 141.732971] nix-grpc-daemon[1546]: time=2026-09-21T13:55:58.045Z level=INFO msg="Uploading 1 narinfos"6547lb # [ 141.740132] niks3-server[1150]: 2026/09/21 13:55:58 INFO Signed narinfos id=47 count=16548lb # [ 141.746326] niks3-server[1150]: 2026/09/21 13:55:58 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6549lb # [ 141.757277] niks3-server[1150]: 2026/09/21 13:55:58 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6550lb # [ 141.763521] niks3-server[1150]: 2026/09/21 13:55:58 INFO Completed upload id=476551worker2 # [ 141.757063] nix-grpc-daemon[1546]: time=2026-09-21T13:55:58.068Z level=INFO msg="Upload complete. (498ms)"6552lb # [ 141.766911] niks3-server[1150]: 2026/09/21 13:55:58 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6553worker2 # [ 141.760422] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=0 paths=1 nar_bytes_in=6406554worker2 # [ 141.763953] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355206555worker2 # [ 141.769070] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6556worker2 # [ 141.773970] nix-daemon[853]: reaped child process 1657, status = succeeded6557worker2 # [ 141.779052] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6558client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6559client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6560client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6561client # farm-b> LOG-farm-b-during-drain6562worker2 # [ 143.394111] nix-daemon[853]: reaped child process 1671, status = succeeded6563worker2 # [ 143.565309] nix-daemon[853]: accepted connection from pid 1687, user nix-grpc-daemon (trusted)6564worker2 # [ 143.577901] nix-daemon[853]: reaped child process 1690, status = succeeded6565lb # [ 143.605911] niks3-server[1150]: 2026/09/21 13:56:00 INFO Received uploads request method=POST path=/api/pending_closures6566worker2 # [ 143.608725] nix-grpc-daemon[1546]: time=2026-09-21T13:55:59.920Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6567worker2 # [ 143.610393] nix-grpc-daemon[1546]: time=2026-09-21T13:55:59.920Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6568lb # [ 143.641655] niks3-server[1150]: 2026/09/21 13:56:00 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6569lb # [ 143.647442] niks3-server[1150]: 2026/09/21 13:56:00 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6570lb # [ 143.659500] niks3-server[1150]: 2026/09/21 13:56:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6571lb # [ 143.666254] niks3-server[1150]: 2026/09/21 13:56:00 INFO Signed narinfos id=48 count=16572worker2 # [ 143.661664] nix-grpc-daemon[1546]: time=2026-09-21T13:55:59.973Z level=INFO msg="Uploading 1 narinfos"6573lb # [ 143.668903] niks3-server[1150]: 2026/09/21 13:56:00 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6574lb # [ 143.679788] niks3-server[1150]: 2026/09/21 13:56:00 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6575lb # [ 143.684141] niks3-server[1150]: 2026/09/21 13:56:00 INFO Completed upload id=486576lb # [ 143.685994] niks3-server[1150]: 2026/09/21 13:56:00 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6577worker2 # [ 143.678431] nix-grpc-daemon[1546]: time=2026-09-21T13:55:59.990Z level=INFO msg="Upload complete. (291ms)"6578worker2 # [ 143.683629] nix-grpc-daemon[1470]: ts=2026-09-21T13:55:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16579worker2 # [ 143.693975] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:00Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06580worker2 # [ 143.698327] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:00Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16581worker2 # [ 143.707601] nix-daemon[853]: reaped child process 1669, status = succeeded6582worker2 # [ 143.737997] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355206583worker2 # [ 143.751255] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:355206584worker2 # [ 143.764725] nix-grpc-daemon[1546]: time=2026-09-21T13:56:00.077Z level=INFO msg="All 2 paths already cached"6585client # copying 1 paths to 'lb:50051'...6586client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6587worker2 # [ 143.772189] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:355206588worker2 # [ 143.777592] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6589worker2 # [ 143.957582] nix-daemon[853]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)6590worker2 # [ 143.969774] nix-daemon[853]: reaped child process 1699, status = succeeded6591worker2 # [ 144.157879] nix-daemon[853]: accepted connection from pid 1701, user nix-grpc-daemon (trusted)6592worker2 # [ 144.171386] nix-daemon[853]: reaped child process 1704, status = succeeded6593lb # [ 144.197286] niks3-server[1150]: 2026/09/21 13:56:01 INFO Received uploads request method=POST path=/api/pending_closures6594worker2 # [ 144.198644] nix-grpc-daemon[1546]: time=2026-09-21T13:56:00.510Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6595worker2 # [ 144.200037] nix-grpc-daemon[1546]: time=2026-09-21T13:56:00.510Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6596lb # [ 144.222086] niks3-server[1150]: 2026/09/21 13:56:01 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6597lb # [ 144.234897] niks3-server[1150]: 2026/09/21 13:56:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6598worker2 # [ 144.232771] nix-grpc-daemon[1546]: time=2026-09-21T13:56:00.545Z level=INFO msg="Uploading 1 narinfos"6599lb # [ 144.241389] niks3-server[1150]: 2026/09/21 13:56:01 INFO Signed narinfos id=49 count=16600lb # [ 144.246619] niks3-server[1150]: 2026/09/21 13:56:01 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6601lb # [ 144.259055] niks3-server[1150]: 2026/09/21 13:56:01 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6602lb # [ 144.263987] niks3-server[1150]: 2026/09/21 13:56:01 INFO Completed upload id=496603worker2 # [ 144.256971] nix-grpc-daemon[1546]: time=2026-09-21T13:56:00.569Z level=INFO msg="Upload complete. (472ms)"6604lb # [ 144.267068] niks3-server[1150]: 2026/09/21 13:56:01 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6605worker2 # [ 144.261121] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=0 paths=1 nar_bytes_in=7686606worker2 # [ 144.266996] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355206607worker2 # [ 144.276761] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6608worker2 # [ 144.280219] nix-daemon[853]: reaped child process 1694, status = succeeded6609client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6610client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6611worker2 # [ 144.285540] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6612client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6613client # farm-top> LOG-farm-top-during-drain6614worker2 # [ 145.499561] nix-daemon[853]: reaped child process 1708, status = succeeded6615worker2 # [ 145.676284] nix-daemon[853]: accepted connection from pid 1724, user nix-grpc-daemon (trusted)6616worker2 # [ 145.688359] nix-daemon[853]: reaped child process 1727, status = succeeded6617lb # [ 145.716074] niks3-server[1150]: 2026/09/21 13:56:02 INFO Received uploads request method=POST path=/api/pending_closures6618worker2 # [ 145.718877] nix-grpc-daemon[1546]: time=2026-09-21T13:56:02.031Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6619worker2 # [ 145.720368] nix-grpc-daemon[1546]: time=2026-09-21T13:56:02.032Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6620lb # [ 145.751601] niks3-server[1150]: 2026/09/21 13:56:02 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6621lb # [ 145.766945] niks3-server[1150]: 2026/09/21 13:56:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6622lb # [ 145.773908] niks3-server[1150]: 2026/09/21 13:56:02 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6623worker2 # [ 145.771099] nix-grpc-daemon[1546]: time=2026-09-21T13:56:02.083Z level=INFO msg="Uploading 1 narinfos"6624lb # [ 145.778593] niks3-server[1150]: 2026/09/21 13:56:02 INFO Signed narinfos id=50 count=16625lb # [ 145.792503] niks3-server[1150]: 2026/09/21 13:56:02 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6626lb # [ 145.797315] niks3-server[1150]: 2026/09/21 13:56:02 INFO Completed upload id=506627worker2 # [ 145.790232] nix-grpc-daemon[1546]: time=2026-09-21T13:56:02.102Z level=INFO msg="Upload complete. (297ms)"6628lb # [ 145.799169] niks3-server[1150]: 2026/09/21 13:56:02 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6629lb # [ 145.804896] niks3-server[1150]: 2026/09/21 13:56:02 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6630worker2 # [ 145.797597] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16631worker2 # [ 145.809298] nix-daemon[853]: reaped child process 1706, status = succeeded6632client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 7.51 seconds)6633client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6634client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.24 seconds)6635worker1: must succeed: pkill -f 'read -t [9]0 x'6636worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6637client: waiting for success: ! systemctl is-active sw6638worker1 # [ 146.200218] nix-daemon[870]: reaped child process 1701, status = succeeded6639worker1 # [ 146.371575] nix-daemon[870]: accepted connection from pid 1743, user nix-grpc-daemon (trusted)6640worker1 # [ 146.383457] nix-daemon[870]: reaped child process 1746, status = succeeded6641lb # [ 146.404101] niks3-server[1150]: 2026/09/21 13:56:03 INFO Received uploads request method=POST path=/api/pending_closures6642worker1 # [ 146.416565] nix-grpc-daemon[1649]: time=2026-09-21T13:56:03.552Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6643worker1 # [ 146.418040] nix-grpc-daemon[1649]: time=2026-09-21T13:56:03.552Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6644lb # [ 146.440876] niks3-server[1150]: 2026/09/21 13:56:03 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6645lb # [ 146.454649] niks3-server[1150]: 2026/09/21 13:56:03 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6646lb # [ 146.463435] niks3-server[1150]: 2026/09/21 13:56:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6647lb # [ 146.469200] niks3-server[1150]: 2026/09/21 13:56:03 INFO Signed narinfos id=51 count=16648worker1 # [ 146.468762] nix-grpc-daemon[1649]: time=2026-09-21T13:56:03.604Z level=INFO msg="Uploading 1 narinfos"6649lb # [ 146.472707] niks3-server[1150]: 2026/09/21 13:56:03 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6650lb # [ 146.480746] niks3-server[1150]: 2026/09/21 13:56:03 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6651lb # [ 146.486375] niks3-server[1150]: 2026/09/21 13:56:03 INFO Completed upload id=516652worker1 # [ 146.485645] nix-grpc-daemon[1649]: time=2026-09-21T13:56:03.622Z level=INFO msg="Upload complete. (291ms)"6653lb # [ 146.488208] niks3-server[1150]: 2026/09/21 13:56:03 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6654lb # [ 146.492988] envoy[901]: conn peer=[2001:db8:1::1]:58064 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9628 tx=4087 ms=72016655worker1 # [ 146.492512] nix-grpc-daemon[1547]: ts=2026-09-21T13:56:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39118 duration_s=9 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16656worker1 # [ 146.503822] nix-daemon[870]: reaped child process 1699, status = succeeded6657client # [ 146.534978] systemd[1]: sw.service: Deactivated successfully.6658client # [ 146.537040] systemd[1]: sw.service: Consumed 131ms CPU time over 10.859s wall clock time, 41.4M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.6659worker1 # [ 146.640510] nix-grpc-daemon[1547]: ts=2026-09-21T13:56:03Z level=info event=signal_exit6660client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)6661client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6662client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds)6663worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6664worker2 # [ 147.665260] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:03Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:338786665worker2 # [ 147.668967] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:03Z level=info event=worker_gone addr=192.168.1.3:500516666worker1 # [ 147.678729] nix-grpc-daemon[1547]: ts=2026-09-21T13:56:04Z level=info event=scheduler_restarting6667worker1 # [ 147.685289] nix-grpc-daemon[1547]: ts=2026-09-21T13:56:04Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6668worker1 # [ 147.703377] nix-daemon[870]: reaped child process 1603, status = succeeded6669worker1 # [ 148.449962] nix-grpc-daemon[1547]: E0000 00:00:1789998965.586250 1592 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true6670worker1 # [ 148.455296] nix-grpc-daemon[1547]: E0000 00:00:1789998965.591474 1592 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6671worker1 # [ 148.682804] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6672worker1 # [ 148.686846] systemd[1]: nix-grpc-daemon.service: Consumed 636ms CPU time over 26.254s wall clock time, 32.6M memory peak, 20.6K incoming IP traffic, 27.2K outgoing IP traffic.6673worker1 # [ 148.702840] systemd[1]: Starting Nix worker-protocol over gRPC...6674worker1 # [ 148.905854] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:06Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6675worker1 # [ 148.908366] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:06Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6676worker1 # [ 148.912345] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:06Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16677worker1 # [ 148.920062] nix-grpc-daemon[1760]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6678worker1 # [ 148.922888] nix-grpc-daemon[1760]: E0000 00:00:1789998966.056345 1760 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6679worker1 # [ 148.926845] nix-grpc-daemon[1760]: E0000 00:00:1789998966.059284 1760 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6680worker1 # [ 148.930785] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:06Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6681worker1 # [ 148.933547] systemd[1]: Started Nix worker-protocol over gRPC.6682worker1 # [ 148.939097] nix-grpc-daemon[1760]: E0000 00:00:1789998966.074869 1802 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6683worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 1.65 seconds)6684lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6685lb: must succeed: curl -sf localhost:9901/clusters6686worker1 # [ 148.951035] nix-grpc-daemon[1760]: E0000 00:00:1789998966.087520 1802 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6687worker1 # [ 148.955178] nix-grpc-daemon[1760]: E0000 00:00:1789998966.089722 1802 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6688worker1 # [ 148.965165] nix-grpc-daemon[1760]: E0000 00:00:1789998966.101667 1802 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6689worker1 # [ 148.976169] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:06Z level=info event=scheduler_connected addr=lb:500516690worker2 # [ 148.973185] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:05Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:338786691worker2 # [ 148.977238] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:05Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06692lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6693lb: must succeed: curl -sf localhost:9901/clusters6694lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6695lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.16 seconds)6696lb: must succeed: curl -sf localhost:9901/clusters6697lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6698worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6699worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6700(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 14.60 seconds)6701subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6702client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16703worker2 # [ 150.471820] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:355206704worker1 # [ 150.492211] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334326705worker1 # [ 150.503857] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6706worker1 # [ 150.520916] nix-grpc-daemon[1760]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 1 ms (attempt 1/5)6707worker1 # [ 150.527681] nix-grpc-daemon[1760]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 142 ms (attempt 2/5)6708worker1 # [ 150.675640] nix-grpc-daemon[1760]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 125 ms (attempt 3/5)6709worker1 # [ 150.806701] nix-grpc-daemon[1760]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 231 ms (attempt 4/5)6710worker1 # [ 151.043823] nix-grpc-daemon[1760]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6711worker1 # [ 151.047757] nix-grpc-daemon[1760]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6712worker2 # [ 151.089181] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:07Z level=debug event=want client=5 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06713worker2 # [ 151.093545] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:07Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16714worker2 # [ 151.097768] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:07Z level=debug event=want client=5 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06715worker2 # [ 151.101553] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:07Z level=debug event=want client=5 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06716worker1 # [ 151.140879] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326717worker1 # [ 151.154937] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334326718worker1 # [ 151.166509] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334326719worker1 # [ 151.175139] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6720worker1 # [ 151.197987] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:08Z level=info event=niks3_push_started pid=18206721worker1 # [ 151.432106] nix-grpc-daemon[1820]: time=2026-09-21T13:56:08.568Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6722worker1 # [ 151.631394] nix-daemon[870]: accepted connection from pid 1833, user nix-grpc-daemon (trusted)6723worker1 # [ 151.641726] nix-daemon[870]: reaped child process 1836, status = succeeded6724worker1 # [ 151.853551] nix-daemon[870]: accepted connection from pid 1838, user nix-grpc-daemon (trusted)6725worker1 # [ 151.865270] nix-daemon[870]: reaped child process 1841, status = succeeded6726lb # [ 151.887820] niks3-server[1150]: 2026/09/21 13:56:08 INFO Received uploads request method=POST path=/api/pending_closures6727worker1 # [ 151.896591] nix-grpc-daemon[1820]: time=2026-09-21T13:56:09.032Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6728worker1 # [ 151.897972] nix-grpc-daemon[1820]: time=2026-09-21T13:56:09.032Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6729lb # [ 151.923182] niks3-server[1150]: 2026/09/21 13:56:08 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6730lb # [ 151.931338] niks3-server[1150]: 2026/09/21 13:56:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6731lb # [ 151.936340] niks3-server[1150]: 2026/09/21 13:56:08 INFO Signed narinfos id=52 count=16732worker1 # [ 151.939477] nix-grpc-daemon[1820]: time=2026-09-21T13:56:09.075Z level=INFO msg="Uploading 1 narinfos"6733lb # [ 151.948115] niks3-server[1150]: 2026/09/21 13:56:08 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6734lb # [ 151.956120] niks3-server[1150]: 2026/09/21 13:56:08 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6735lb # [ 151.962610] niks3-server[1150]: 2026/09/21 13:56:08 INFO Completed upload id=526736worker1 # [ 151.961529] nix-grpc-daemon[1820]: time=2026-09-21T13:56:09.098Z level=INFO msg="Upload complete. (524ms)"6737lb # [ 151.965381] niks3-server[1150]: 2026/09/21 13:56:08 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6738worker1 # [ 151.966503] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=0 paths=1 nar_bytes_in=5046739worker1 # [ 151.971514] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326740worker1 # [ 151.980532] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6741worker1 # [ 151.984559] nix-daemon[870]: reaped child process 1818, status = succeeded6742worker1 # [ 151.988630] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6743worker1 # [ 152.801084] nix-daemon[870]: reaped child process 1845, status = succeeded6744worker1 # [ 152.989258] nix-daemon[870]: accepted connection from pid 1861, user nix-grpc-daemon (trusted)6745worker1 # [ 152.999550] nix-daemon[870]: reaped child process 1864, status = succeeded6746lb # [ 153.020567] niks3-server[1150]: 2026/09/21 13:56:09 INFO Received uploads request method=POST path=/api/pending_closures6747worker1 # [ 153.028509] nix-grpc-daemon[1820]: time=2026-09-21T13:56:10.164Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6748worker1 # [ 153.030221] nix-grpc-daemon[1820]: time=2026-09-21T13:56:10.166Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6749lb # [ 153.060852] niks3-server[1150]: 2026/09/21 13:56:09 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6750lb # [ 153.071215] niks3-server[1150]: 2026/09/21 13:56:09 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6751lb # [ 153.079903] niks3-server[1150]: 2026/09/21 13:56:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6752worker1 # [ 153.085799] nix-grpc-daemon[1820]: time=2026-09-21T13:56:10.221Z level=INFO msg="Uploading 1 narinfos"6753lb # [ 153.086557] niks3-server[1150]: 2026/09/21 13:56:09 INFO Signed narinfos id=53 count=16754lb # [ 153.091128] niks3-server[1150]: 2026/09/21 13:56:09 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6755lb # [ 153.100344] niks3-server[1150]: 2026/09/21 13:56:10 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6756lb # [ 153.104901] niks3-server[1150]: 2026/09/21 13:56:10 INFO Completed upload id=536757worker1 # [ 153.105301] nix-grpc-daemon[1820]: time=2026-09-21T13:56:10.241Z level=INFO msg="Upload complete. (310ms)"6758lb # [ 153.107939] niks3-server[1150]: 2026/09/21 13:56:10 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6759worker2 # [ 153.105897] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:09Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16760worker1 # [ 153.113245] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=7 outputs=16761worker1 # [ 153.120813] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326762worker2 # [ 153.115884] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:09Z level=debug event=want client=5 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06763worker1 # [ 153.133665] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334326764worker1 # [ 153.139618] nix-daemon[870]: reaped child process 1843, status = succeeded6765worker1 # [ 153.145359] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334326766worker1 # [ 153.152774] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6767worker1 # [ 153.339755] nix-daemon[870]: accepted connection from pid 1868, user nix-grpc-daemon (trusted)6768worker1 # [ 153.351702] nix-daemon[870]: reaped child process 1871, status = succeeded6769worker1 # [ 153.557741] nix-daemon[870]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6770worker1 # [ 153.569502] nix-daemon[870]: reaped child process 1876, status = succeeded6771lb # [ 153.589129] niks3-server[1150]: 2026/09/21 13:56:10 INFO Received uploads request method=POST path=/api/pending_closures6772worker1 # [ 153.599027] nix-grpc-daemon[1820]: time=2026-09-21T13:56:10.735Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6773worker1 # [ 153.600548] nix-grpc-daemon[1820]: time=2026-09-21T13:56:10.737Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6774lb # [ 153.622700] niks3-server[1150]: 2026/09/21 13:56:10 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6775lb # [ 153.634243] niks3-server[1150]: 2026/09/21 13:56:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6776lb # [ 153.639734] niks3-server[1150]: 2026/09/21 13:56:10 INFO Signed narinfos id=54 count=16777worker1 # [ 153.641704] nix-grpc-daemon[1820]: time=2026-09-21T13:56:10.778Z level=INFO msg="Uploading 1 narinfos"6778lb # [ 153.646601] niks3-server[1150]: 2026/09/21 13:56:10 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6779lb # [ 153.657532] niks3-server[1150]: 2026/09/21 13:56:10 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6780lb # [ 153.666056] niks3-server[1150]: 2026/09/21 13:56:10 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6781lb # [ 153.669436] niks3-server[1150]: 2026/09/21 13:56:10 INFO Completed upload id=546782worker1 # [ 153.668656] nix-grpc-daemon[1820]: time=2026-09-21T13:56:10.805Z level=INFO msg="Upload complete. (508ms)"6783worker1 # [ 153.674182] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=0 paths=1 nar_bytes_in=5046784worker1 # [ 153.682105] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326785worker1 # [ 153.694161] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6786worker1 # [ 153.701231] nix-daemon[870]: reaped child process 1866, status = succeeded6787worker1 # [ 153.709879] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6788worker1 # [ 155.136892] nix-daemon[870]: reaped child process 1880, status = succeeded6789worker1 # [ 155.336336] nix-daemon[870]: accepted connection from pid 1896, user nix-grpc-daemon (trusted)6790worker1 # [ 155.348098] nix-daemon[870]: reaped child process 1899, status = succeeded6791lb # [ 155.371522] niks3-server[1150]: 2026/09/21 13:56:12 INFO Received uploads request method=POST path=/api/pending_closures6792worker1 # [ 155.379281] nix-grpc-daemon[1820]: time=2026-09-21T13:56:12.515Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6793worker1 # [ 155.380799] nix-grpc-daemon[1820]: time=2026-09-21T13:56:12.517Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6794lb # [ 155.406783] niks3-server[1150]: 2026/09/21 13:56:12 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6795lb # [ 155.416406] niks3-server[1150]: 2026/09/21 13:56:12 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6796lb # [ 155.426126] niks3-server[1150]: 2026/09/21 13:56:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6797lb # [ 155.430420] niks3-server[1150]: 2026/09/21 13:56:12 INFO Signed narinfos id=55 count=16798worker1 # [ 155.432698] nix-grpc-daemon[1820]: time=2026-09-21T13:56:12.568Z level=INFO msg="Uploading 1 narinfos"6799lb # [ 155.438923] niks3-server[1150]: 2026/09/21 13:56:12 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6800lb # [ 155.448149] niks3-server[1150]: 2026/09/21 13:56:12 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6801lb # [ 155.454564] niks3-server[1150]: 2026/09/21 13:56:12 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6802lb # [ 155.458073] niks3-server[1150]: 2026/09/21 13:56:12 INFO Completed upload id=556803worker1 # [ 155.458489] nix-grpc-daemon[1820]: time=2026-09-21T13:56:12.594Z level=INFO msg="Upload complete. (340ms)"6804worker1 # [ 155.464667] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=8 outputs=16805worker2 # [ 155.459360] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:11Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16806worker1 # [ 155.473879] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326807worker2 # [ 155.469093] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:11Z level=debug event=want client=5 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06808worker1 # [ 155.488599] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334326809worker1 # [ 155.493115] nix-daemon[870]: reaped child process 1878, status = succeeded6810worker1 # [ 155.499556] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334326811worker1 # [ 155.508136] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6812worker1 # [ 155.705595] nix-daemon[870]: accepted connection from pid 1903, user nix-grpc-daemon (trusted)6813worker1 # [ 155.717660] nix-daemon[870]: reaped child process 1906, status = succeeded6814worker1 # [ 155.903311] nix-daemon[870]: accepted connection from pid 1908, user nix-grpc-daemon (trusted)6815worker1 # [ 155.914804] nix-daemon[870]: reaped child process 1911, status = succeeded6816lb # [ 155.937605] niks3-server[1150]: 2026/09/21 13:56:12 INFO Received uploads request method=POST path=/api/pending_closures6817worker1 # [ 155.945800] nix-grpc-daemon[1820]: time=2026-09-21T13:56:13.082Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6818worker1 # [ 155.947616] nix-grpc-daemon[1820]: time=2026-09-21T13:56:13.083Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6819lb # [ 155.966235] niks3-server[1150]: 2026/09/21 13:56:12 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6820lb # [ 155.978972] niks3-server[1150]: 2026/09/21 13:56:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6821worker1 # [ 155.985117] nix-grpc-daemon[1820]: time=2026-09-21T13:56:13.121Z level=INFO msg="Uploading 1 narinfos"6822lb # [ 155.985475] niks3-server[1150]: 2026/09/21 13:56:12 INFO Signed narinfos id=56 count=16823lb # [ 155.990188] niks3-server[1150]: 2026/09/21 13:56:12 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6824lb # [ 156.001814] niks3-server[1150]: 2026/09/21 13:56:12 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6825lb # [ 156.008190] niks3-server[1150]: 2026/09/21 13:56:12 INFO Completed upload id=566826worker1 # [ 156.007172] nix-grpc-daemon[1820]: time=2026-09-21T13:56:13.143Z level=INFO msg="Upload complete. (491ms)"6827lb # [ 156.011202] niks3-server[1150]: 2026/09/21 13:56:12 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6828worker1 # [ 156.010769] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=0 paths=1 nar_bytes_in=5046829worker1 # [ 156.016810] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326830worker1 # [ 156.028060] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6831worker1 # [ 156.030359] nix-daemon[870]: reaped child process 1901, status = succeeded6832worker1 # [ 156.035942] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6833lb # [ 156.495249] envoy[901]: conn peer=[2001:db8:1::1]:58050 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2975 tx=1404 ms=105866834lb # [ 156.499348] envoy[901]: conn peer=[2001:db8:1::3]:37788 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1114 tx=448 ms=249986835worker1 # [ 156.848661] nix-daemon[870]: reaped child process 1915, status = succeeded6836worker1 # [ 157.025837] nix-daemon[870]: accepted connection from pid 1931, user nix-grpc-daemon (trusted)6837worker1 # [ 157.037541] nix-daemon[870]: reaped child process 1934, status = succeeded6838lb # [ 157.055603] niks3-server[1150]: 2026/09/21 13:56:13 INFO Received uploads request method=POST path=/api/pending_closures6839worker1 # [ 157.063367] nix-grpc-daemon[1820]: time=2026-09-21T13:56:14.199Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6840worker1 # [ 157.064864] nix-grpc-daemon[1820]: time=2026-09-21T13:56:14.201Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"6841lb # [ 157.097504] niks3-server[1150]: 2026/09/21 13:56:14 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6842lb # [ 157.106391] niks3-server[1150]: 2026/09/21 13:56:14 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv6843lb # [ 157.110398] niks3-server[1150]: 2026/09/21 13:56:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6844lb # [ 157.116301] niks3-server[1150]: 2026/09/21 13:56:14 INFO Signed narinfos id=57 count=16845worker1 # [ 157.117899] nix-grpc-daemon[1820]: time=2026-09-21T13:56:14.254Z level=INFO msg="Uploading 1 narinfos"6846lb # [ 157.128576] niks3-server[1150]: 2026/09/21 13:56:14 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6847lb # [ 157.132942] niks3-server[1150]: 2026/09/21 13:56:14 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls6848lb # [ 157.138897] niks3-server[1150]: 2026/09/21 13:56:14 INFO Completed upload id=576849worker1 # [ 157.138170] nix-grpc-daemon[1820]: time=2026-09-21T13:56:14.274Z level=INFO msg="Upload complete. (295ms)"6850lb # [ 157.141417] niks3-server[1150]: 2026/09/21 13:56:14 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo6851worker1 # [ 157.143655] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=9 outputs=16852worker2 # [ 157.138598] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:13Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=16853worker1 # [ 157.152163] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326854worker2 # [ 157.147867] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:13Z level=debug event=want client=5 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=06855worker1 # [ 157.163539] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334326856worker1 # [ 157.166639] nix-daemon[870]: reaped child process 1913, status = succeeded6857worker1 # [ 157.171118] nix-grpc-daemon[1820]: time=2026-09-21T13:56:14.307Z level=INFO msg="All 1 paths already cached"6858worker1 # [ 157.174914] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334326859worker1 # [ 157.178710] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6860worker1 # [ 157.363689] nix-daemon[870]: accepted connection from pid 1938, user nix-grpc-daemon (trusted)6861worker1 # [ 157.375547] nix-daemon[870]: reaped child process 1941, status = succeeded6862worker1 # [ 157.573092] nix-daemon[870]: accepted connection from pid 1943, user nix-grpc-daemon (trusted)6863worker1 # [ 157.584832] nix-daemon[870]: reaped child process 1946, status = succeeded6864lb # [ 157.605717] niks3-server[1150]: 2026/09/21 13:56:14 INFO Received uploads request method=POST path=/api/pending_closures6865worker1 # [ 157.614526] nix-grpc-daemon[1820]: time=2026-09-21T13:56:14.750Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6866worker1 # [ 157.615896] nix-grpc-daemon[1820]: time=2026-09-21T13:56:14.750Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"6867lb # [ 157.639161] niks3-server[1150]: 2026/09/21 13:56:14 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst6868lb # [ 157.644225] niks3-server[1150]: 2026/09/21 13:56:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6869lb # [ 157.649090] niks3-server[1150]: 2026/09/21 13:56:14 INFO Signed narinfos id=58 count=16870worker1 # [ 157.651307] nix-grpc-daemon[1820]: time=2026-09-21T13:56:14.787Z level=INFO msg="Uploading 1 narinfos"6871lb # [ 157.659468] niks3-server[1150]: 2026/09/21 13:56:14 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls6872lb # [ 157.668260] niks3-server[1150]: 2026/09/21 13:56:14 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6873lb # [ 157.676820] niks3-server[1150]: 2026/09/21 13:56:14 INFO Completed upload id=586874worker1 # [ 157.675847] nix-grpc-daemon[1820]: time=2026-09-21T13:56:14.812Z level=INFO msg="Upload complete. (491ms)"6875lb # [ 157.679246] niks3-server[1150]: 2026/09/21 13:56:14 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo6876worker1 # [ 157.679453] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=0 paths=1 nar_bytes_in=6246877worker1 # [ 157.685245] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326878worker1 # [ 157.696103] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6879worker1 # [ 157.700491] nix-daemon[870]: reaped child process 1936, status = succeeded6880worker1 # [ 157.705892] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6881worker1 # [ 159.132510] nix-daemon[870]: reaped child process 1950, status = succeeded6882worker1 # [ 159.325758] nix-daemon[870]: accepted connection from pid 1966, user nix-grpc-daemon (trusted)6883worker1 # [ 159.338117] nix-daemon[870]: reaped child process 1969, status = succeeded6884lb # [ 159.359851] niks3-server[1150]: 2026/09/21 13:56:16 INFO Received uploads request method=POST path=/api/pending_closures6885worker1 # [ 159.368062] nix-grpc-daemon[1820]: time=2026-09-21T13:56:16.504Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6886worker1 # [ 159.369548] nix-grpc-daemon[1820]: time=2026-09-21T13:56:16.506Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"6887lb # [ 159.393961] niks3-server[1150]: 2026/09/21 13:56:16 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv6888lb # [ 159.412767] niks3-server[1150]: 2026/09/21 13:56:16 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst6889lb # [ 159.420584] niks3-server[1150]: 2026/09/21 13:56:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6890lb # [ 159.424303] niks3-server[1150]: 2026/09/21 13:56:16 INFO Signed narinfos id=59 count=16891worker1 # [ 159.426037] nix-grpc-daemon[1820]: time=2026-09-21T13:56:16.562Z level=INFO msg="Uploading 1 narinfos"6892lb # [ 159.429745] niks3-server[1150]: 2026/09/21 13:56:16 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls6893lb # [ 159.438156] niks3-server[1150]: 2026/09/21 13:56:16 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6894lb # [ 159.442244] niks3-server[1150]: 2026/09/21 13:56:16 INFO Completed upload id=596895worker1 # [ 159.441492] nix-grpc-daemon[1820]: time=2026-09-21T13:56:16.578Z level=INFO msg="Upload complete. (327ms)"6896lb # [ 159.443936] niks3-server[1150]: 2026/09/21 13:56:16 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo6897worker1 # [ 159.447141] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=10 outputs=16898worker2 # [ 159.444558] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:15Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=16899worker2 # [ 159.450189] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:15Z level=debug event=want client=5 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=06900worker1 # [ 159.459914] nix-daemon[870]: reaped child process 1948, status = succeeded6901worker1 # [ 159.493177] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326902worker1 # [ 159.506392] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334326903worker1 # [ 159.520746] nix-grpc-daemon[1820]: time=2026-09-21T13:56:16.657Z level=INFO msg="All 1 paths already cached"6904worker1 # [ 159.528733] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334326905worker1 # [ 159.538165] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6906worker1 # [ 159.744909] nix-daemon[870]: accepted connection from pid 1973, user nix-grpc-daemon (trusted)6907worker1 # [ 159.756766] nix-daemon[870]: reaped child process 1976, status = succeeded6908worker1 # [ 159.957426] nix-daemon[870]: accepted connection from pid 1978, user nix-grpc-daemon (trusted)6909worker1 # [ 159.969191] nix-daemon[870]: reaped child process 1981, status = succeeded6910lb # [ 159.990915] niks3-server[1150]: 2026/09/21 13:56:16 INFO Received uploads request method=POST path=/api/pending_closures6911worker1 # [ 159.998660] nix-grpc-daemon[1820]: time=2026-09-21T13:56:17.135Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6912worker1 # [ 160.000488] nix-grpc-daemon[1820]: time=2026-09-21T13:56:17.136Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"6913lb # [ 160.021855] niks3-server[1150]: 2026/09/21 13:56:16 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst6914lb # [ 160.031134] niks3-server[1150]: 2026/09/21 13:56:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6915lb # [ 160.037946] niks3-server[1150]: 2026/09/21 13:56:16 INFO Signed narinfos id=60 count=16916worker1 # [ 160.040119] nix-grpc-daemon[1820]: time=2026-09-21T13:56:17.176Z level=INFO msg="Uploading 1 narinfos"6917lb # [ 160.043244] niks3-server[1150]: 2026/09/21 13:56:16 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls6918lb # [ 160.054822] niks3-server[1150]: 2026/09/21 13:56:16 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6919lb # [ 160.061372] niks3-server[1150]: 2026/09/21 13:56:16 INFO Completed upload id=606920worker1 # [ 160.060429] nix-grpc-daemon[1820]: time=2026-09-21T13:56:17.196Z level=INFO msg="Upload complete. (510ms)"6921lb # [ 160.065604] niks3-server[1150]: 2026/09/21 13:56:16 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo6922worker1 # [ 160.063550] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=0 paths=1 nar_bytes_in=6246923worker1 # [ 160.069160] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326924worker1 # [ 160.078174] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6925worker1 # [ 160.083294] nix-daemon[870]: reaped child process 1971, status = succeeded6926worker1 # [ 160.088428] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6927worker1 # [ 161.303337] nix-daemon[870]: reaped child process 1985, status = succeeded6928worker1 # [ 161.486486] nix-daemon[870]: accepted connection from pid 2001, user nix-grpc-daemon (trusted)6929worker1 # [ 161.498523] nix-daemon[870]: reaped child process 2004, status = succeeded6930lb # [ 161.519928] niks3-server[1150]: 2026/09/21 13:56:18 INFO Received uploads request method=POST path=/api/pending_closures6931worker1 # [ 161.528106] nix-grpc-daemon[1820]: time=2026-09-21T13:56:18.664Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6932worker1 # [ 161.529597] nix-grpc-daemon[1820]: time=2026-09-21T13:56:18.666Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"6933lb # [ 161.557163] niks3-server[1150]: 2026/09/21 13:56:18 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv6934lb # [ 161.570336] niks3-server[1150]: 2026/09/21 13:56:18 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst6935lb # [ 161.582216] niks3-server[1150]: 2026/09/21 13:56:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6936lb # [ 161.586873] niks3-server[1150]: 2026/09/21 13:56:18 INFO Signed narinfos id=61 count=16937worker1 # [ 161.586222] nix-grpc-daemon[1820]: time=2026-09-21T13:56:18.722Z level=INFO msg="Uploading 1 narinfos"6938lb # [ 161.592100] niks3-server[1150]: 2026/09/21 13:56:18 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls6939lb # [ 161.601448] niks3-server[1150]: 2026/09/21 13:56:18 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6940lb # [ 161.607132] niks3-server[1150]: 2026/09/21 13:56:18 INFO Completed upload id=616941lb # [ 161.608977] niks3-server[1150]: 2026/09/21 13:56:18 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo6942worker1 # [ 161.607598] nix-grpc-daemon[1820]: time=2026-09-21T13:56:18.744Z level=INFO msg="Upload complete. (311ms)"6943worker1 # [ 161.612781] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=11 outputs=16944worker2 # [ 161.607814] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:17Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=16945worker2 # [ 161.617794] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:17Z level=debug event=want client=5 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=06946worker1 # [ 161.627207] nix-daemon[870]: reaped child process 1983, status = succeeded6947worker1 # [ 161.657623] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326948worker1 # [ 161.669750] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334326949worker1 # [ 161.682211] nix-grpc-daemon[1820]: time=2026-09-21T13:56:18.818Z level=INFO msg="All 1 paths already cached"6950worker1 # [ 161.688769] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334326951worker1 # [ 161.698811] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6952worker1 # [ 161.907256] nix-daemon[870]: accepted connection from pid 2008, user nix-grpc-daemon (trusted)6953worker1 # [ 161.919338] nix-daemon[870]: reaped child process 2011, status = succeeded6954worker1 # [ 162.126240] nix-daemon[870]: accepted connection from pid 2013, user nix-grpc-daemon (trusted)6955worker1 # [ 162.137777] nix-daemon[870]: reaped child process 2016, status = succeeded6956lb # [ 162.158613] niks3-server[1150]: 2026/09/21 13:56:19 INFO Received uploads request method=POST path=/api/pending_closures6957worker1 # [ 162.166498] nix-grpc-daemon[1820]: time=2026-09-21T13:56:19.303Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6958worker1 # [ 162.168042] nix-grpc-daemon[1820]: time=2026-09-21T13:56:19.304Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"6959lb # [ 162.190176] niks3-server[1150]: 2026/09/21 13:56:19 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst6960lb # [ 162.200528] niks3-server[1150]: 2026/09/21 13:56:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6961worker1 # [ 162.204766] nix-grpc-daemon[1820]: time=2026-09-21T13:56:19.341Z level=INFO msg="Uploading 1 narinfos"6962lb # [ 162.205683] niks3-server[1150]: 2026/09/21 13:56:19 INFO Signed narinfos id=62 count=16963lb # [ 162.212994] niks3-server[1150]: 2026/09/21 13:56:19 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls6964lb # [ 162.222376] niks3-server[1150]: 2026/09/21 13:56:19 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6965lb # [ 162.229172] niks3-server[1150]: 2026/09/21 13:56:19 INFO Completed upload id=626966worker1 # [ 162.228280] nix-grpc-daemon[1820]: time=2026-09-21T13:56:19.364Z level=INFO msg="Upload complete. (517ms)"6967worker1 # [ 162.231891] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=0 paths=1 nar_bytes_in=6246968lb # [ 162.234722] niks3-server[1150]: 2026/09/21 13:56:19 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo6969worker1 # [ 162.237749] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326970worker1 # [ 162.249753] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6971worker1 # [ 162.251817] nix-daemon[870]: reaped child process 2006, status = succeeded6972worker1 # [ 162.256990] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6973worker1 # [ 163.080612] nix-daemon[870]: reaped child process 2020, status = succeeded6974worker1 # [ 163.272345] nix-daemon[870]: accepted connection from pid 2036, user nix-grpc-daemon (trusted)6975worker1 # [ 163.284506] nix-daemon[870]: reaped child process 2039, status = succeeded6976lb # [ 163.304821] niks3-server[1150]: 2026/09/21 13:56:20 INFO Received uploads request method=POST path=/api/pending_closures6977worker1 # [ 163.312981] nix-grpc-daemon[1820]: time=2026-09-21T13:56:20.449Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6978worker1 # [ 163.314520] nix-grpc-daemon[1820]: time=2026-09-21T13:56:20.451Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"6979lb # [ 163.345579] niks3-server[1150]: 2026/09/21 13:56:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6980worker1 # [ 163.350812] nix-grpc-daemon[1820]: time=2026-09-21T13:56:20.487Z level=INFO msg="Uploading 1 narinfos"6981lb # [ 163.351530] niks3-server[1150]: 2026/09/21 13:56:20 INFO Signed narinfos id=63 count=16982lb # [ 163.359309] niks3-server[1150]: 2026/09/21 13:56:20 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv6983lb # [ 163.370970] niks3-server[1150]: 2026/09/21 13:56:20 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst6984lb # [ 163.377869] niks3-server[1150]: 2026/09/21 13:56:20 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6985lb # [ 163.385528] niks3-server[1150]: 2026/09/21 13:56:20 INFO Completed upload id=636986worker1 # [ 163.384676] nix-grpc-daemon[1820]: time=2026-09-21T13:56:20.521Z level=INFO msg="Upload complete. (321ms)"6987lb # [ 163.387914] niks3-server[1150]: 2026/09/21 13:56:20 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo6988lb # [ 163.390982] niks3-server[1150]: 2026/09/21 13:56:20 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls6989worker2 # [ 163.385223] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:19Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=16990worker1 # [ 163.391187] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=12 outputs=16991worker1 # [ 163.399708] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334326992worker2 # [ 163.395368] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:19Z level=debug event=want client=5 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=06993worker1 # [ 163.409530] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334326994worker1 # [ 163.416913] nix-daemon[870]: reaped child process 2018, status = succeeded6995worker1 # [ 163.418819] nix-grpc-daemon[1820]: time=2026-09-21T13:56:20.555Z level=INFO msg="All 2 paths already cached"6996worker1 # [ 163.422899] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334326997worker1 # [ 163.427145] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6998worker1 # [ 163.619286] nix-daemon[870]: accepted connection from pid 2043, user nix-grpc-daemon (trusted)6999worker1 # [ 163.631663] nix-daemon[870]: reaped child process 2046, status = succeeded7000worker1 # [ 163.822092] nix-daemon[870]: accepted connection from pid 2048, user nix-grpc-daemon (trusted)7001worker1 # [ 163.834193] nix-daemon[870]: reaped child process 2051, status = succeeded7002lb # [ 163.853095] niks3-server[1150]: 2026/09/21 13:56:20 INFO Received uploads request method=POST path=/api/pending_closures7003worker1 # [ 163.860963] nix-grpc-daemon[1820]: time=2026-09-21T13:56:20.997Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7004worker1 # [ 163.862337] nix-grpc-daemon[1820]: time=2026-09-21T13:56:20.997Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7005lb # [ 163.878187] niks3-server[1150]: 2026/09/21 13:56:20 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7006lb # [ 163.891292] niks3-server[1150]: 2026/09/21 13:56:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7007worker1 # [ 163.896024] nix-grpc-daemon[1820]: time=2026-09-21T13:56:21.032Z level=INFO msg="Uploading 1 narinfos"7008lb # [ 163.896667] niks3-server[1150]: 2026/09/21 13:56:20 INFO Signed narinfos id=64 count=17009lb # [ 163.903671] niks3-server[1150]: 2026/09/21 13:56:20 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7010lb # [ 163.912266] niks3-server[1150]: 2026/09/21 13:56:20 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7011worker1 # [ 163.918520] nix-grpc-daemon[1820]: time=2026-09-21T13:56:21.055Z level=INFO msg="Upload complete. (485ms)"7012lb # [ 163.919470] niks3-server[1150]: 2026/09/21 13:56:20 INFO Completed upload id=647013worker1 # [ 163.920761] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=0 paths=1 nar_bytes_in=7527014lb # [ 163.924560] niks3-server[1150]: 2026/09/21 13:56:20 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7015worker1 # [ 163.927686] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334327016worker1 # [ 163.937600] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)7017worker1 # [ 163.939239] nix-daemon[870]: reaped child process 2041, status = succeeded7018worker1 # [ 163.944464] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)7019worker1 # [ 164.967660] nix-daemon[870]: reaped child process 2055, status = succeeded7020worker1 # [ 165.156713] nix-daemon[870]: accepted connection from pid 2071, user nix-grpc-daemon (trusted)7021worker1 # [ 165.167812] nix-daemon[870]: reaped child process 2074, status = succeeded7022lb # [ 165.186822] niks3-server[1150]: 2026/09/21 13:56:22 INFO Received uploads request method=POST path=/api/pending_closures7023worker1 # [ 165.194818] nix-grpc-daemon[1820]: time=2026-09-21T13:56:22.331Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7024worker1 # [ 165.196300] nix-grpc-daemon[1820]: time=2026-09-21T13:56:22.332Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7025lb # [ 165.228623] niks3-server[1150]: 2026/09/21 13:56:22 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7026lb # [ 165.235382] niks3-server[1150]: 2026/09/21 13:56:22 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7027lb # [ 165.248723] niks3-server[1150]: 2026/09/21 13:56:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7028lb # [ 165.255119] niks3-server[1150]: 2026/09/21 13:56:22 INFO Signed narinfos id=65 count=17029worker1 # [ 165.254438] nix-grpc-daemon[1820]: time=2026-09-21T13:56:22.390Z level=INFO msg="Uploading 1 narinfos"7030lb # [ 165.258287] niks3-server[1150]: 2026/09/21 13:56:22 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7031lb # [ 165.268458] niks3-server[1150]: 2026/09/21 13:56:22 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7032lb # [ 165.274464] niks3-server[1150]: 2026/09/21 13:56:22 INFO Completed upload id=657033worker1 # [ 165.273641] nix-grpc-daemon[1820]: time=2026-09-21T13:56:22.410Z level=INFO msg="Upload complete. (322ms)"7034lb # [ 165.277114] niks3-server[1150]: 2026/09/21 13:56:22 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7035worker2 # [ 165.274284] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:21Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17036worker1 # [ 165.280488] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=13 outputs=17037worker1 # [ 165.294724] nix-daemon[870]: reaped child process 2053, status = succeeded7038worker1 # [ 165.323634] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334327039worker1 # [ 165.336394] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334327040worker1 # [ 165.350117] nix-grpc-daemon[1820]: time=2026-09-21T13:56:22.486Z level=INFO msg="All 2 paths already cached"7041worker1 # [ 165.357292] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334327042worker1 # [ 165.368185] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)7043worker1 # [ 165.582899] nix-daemon[870]: accepted connection from pid 2078, user nix-grpc-daemon (trusted)7044worker1 # [ 165.594793] nix-daemon[870]: reaped child process 2081, status = succeeded7045worker1 # [ 165.807759] nix-daemon[870]: accepted connection from pid 2083, user nix-grpc-daemon (trusted)7046worker1 # [ 165.819312] nix-daemon[870]: reaped child process 2086, status = succeeded7047lb # [ 165.841313] niks3-server[1150]: 2026/09/21 13:56:22 INFO Received uploads request method=POST path=/api/pending_closures7048worker1 # [ 165.848618] nix-grpc-daemon[1820]: time=2026-09-21T13:56:22.985Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7049worker1 # [ 165.850453] nix-grpc-daemon[1820]: time=2026-09-21T13:56:22.986Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7050lb # [ 165.873205] niks3-server[1150]: 2026/09/21 13:56:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7051worker1 # [ 165.879327] nix-grpc-daemon[1820]: time=2026-09-21T13:56:23.015Z level=INFO msg="Uploading 1 narinfos"7052lb # [ 165.880739] niks3-server[1150]: 2026/09/21 13:56:22 INFO Signed narinfos id=66 count=17053lb # [ 165.884345] niks3-server[1150]: 2026/09/21 13:56:22 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7054lb # [ 165.890199] niks3-server[1150]: 2026/09/21 13:56:22 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7055lb # [ 165.901762] niks3-server[1150]: 2026/09/21 13:56:22 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7056worker1 # [ 165.905666] nix-grpc-daemon[1820]: time=2026-09-21T13:56:23.042Z level=INFO msg="Upload complete. (525ms)"7057lb # [ 165.906772] niks3-server[1150]: 2026/09/21 13:56:22 INFO Completed upload id=667058worker1 # [ 165.908290] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=0 paths=1 nar_bytes_in=7527059lb # [ 165.912396] niks3-server[1150]: 2026/09/21 13:56:22 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7060worker1 # [ 165.913568] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334327061worker1 # [ 165.922852] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)7062worker1 # [ 165.927185] nix-daemon[870]: reaped child process 2076, status = succeeded7063worker1 # [ 165.932598] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)7064worker1 # [ 166.557960] nix-daemon[870]: reaped child process 2090, status = succeeded7065worker1 # [ 166.767556] nix-daemon[870]: accepted connection from pid 2107, user nix-grpc-daemon (trusted)7066worker1 # [ 166.779241] nix-daemon[870]: reaped child process 2110, status = succeeded7067lb # [ 166.803721] niks3-server[1150]: 2026/09/21 13:56:23 INFO Received uploads request method=POST path=/api/pending_closures7068worker1 # [ 166.810193] nix-grpc-daemon[1820]: time=2026-09-21T13:56:23.946Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7069worker1 # [ 166.811665] nix-grpc-daemon[1820]: time=2026-09-21T13:56:23.948Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7070lb # [ 166.840953] niks3-server[1150]: 2026/09/21 13:56:23 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7071lb # [ 166.851080] niks3-server[1150]: 2026/09/21 13:56:23 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7072lb # [ 166.859926] niks3-server[1150]: 2026/09/21 13:56:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7073lb # [ 166.864906] niks3-server[1150]: 2026/09/21 13:56:23 INFO Signed narinfos id=67 count=17074worker1 # [ 166.866700] nix-grpc-daemon[1820]: time=2026-09-21T13:56:24.002Z level=INFO msg="Uploading 1 narinfos"7075lb # [ 166.870051] niks3-server[1150]: 2026/09/21 13:56:23 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7076lb # [ 166.879475] niks3-server[1150]: 2026/09/21 13:56:23 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7077lb # [ 166.884562] niks3-server[1150]: 2026/09/21 13:56:23 INFO Completed upload id=677078worker1 # [ 166.883885] nix-grpc-daemon[1820]: time=2026-09-21T13:56:24.020Z level=INFO msg="Upload complete. (343ms)"7079lb # [ 166.887307] niks3-server[1150]: 2026/09/21 13:56:23 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7080worker2 # [ 166.884656] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:23Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17081worker1 # [ 166.890753] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=0 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=14 outputs=17082worker1 # [ 166.905648] nix-daemon[870]: reaped child process 2088, status = succeeded7083worker1 # [ 166.933379] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334327084worker1 # [ 166.946238] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334327085worker1 # [ 166.961448] nix-grpc-daemon[1820]: time=2026-09-21T13:56:24.097Z level=INFO msg="All 2 paths already cached"7086worker1 # [ 166.969415] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334327087worker1 # [ 166.980175] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)7088worker1 # [ 167.197196] nix-daemon[870]: accepted connection from pid 2114, user nix-grpc-daemon (trusted)7089worker1 # [ 167.208964] nix-daemon[870]: reaped child process 2117, status = succeeded7090worker1 # [ 167.421076] nix-daemon[870]: accepted connection from pid 2119, user nix-grpc-daemon (trusted)7091worker1 # [ 167.432815] nix-daemon[870]: reaped child process 2122, status = succeeded7092lb # [ 167.453307] niks3-server[1150]: 2026/09/21 13:56:24 INFO Received uploads request method=POST path=/api/pending_closures7093worker1 # [ 167.461562] nix-grpc-daemon[1820]: time=2026-09-21T13:56:24.598Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7094worker1 # [ 167.463377] nix-grpc-daemon[1820]: time=2026-09-21T13:56:24.599Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7095lb # [ 167.482417] niks3-server[1150]: 2026/09/21 13:56:24 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7096lb # [ 167.492657] niks3-server[1150]: 2026/09/21 13:56:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7097lb # [ 167.498168] niks3-server[1150]: 2026/09/21 13:56:24 INFO Signed narinfos id=68 count=17098worker1 # [ 167.500322] nix-grpc-daemon[1820]: time=2026-09-21T13:56:24.636Z level=INFO msg="Uploading 1 narinfos"7099lb # [ 167.505561] niks3-server[1150]: 2026/09/21 13:56:24 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7100lb # [ 167.516040] niks3-server[1150]: 2026/09/21 13:56:24 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7101lb # [ 167.522780] niks3-server[1150]: 2026/09/21 13:56:24 INFO Completed upload id=687102worker1 # [ 167.521805] nix-grpc-daemon[1820]: time=2026-09-21T13:56:24.658Z level=INFO msg="Upload complete. (527ms)"7103lb # [ 167.526664] niks3-server[1150]: 2026/09/21 13:56:24 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7104worker1 # [ 167.525386] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=0 paths=1 nar_bytes_in=7527105worker1 # [ 167.530415] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334327106worker1 # [ 167.538158] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)7107worker1 # [ 167.544963] nix-daemon[870]: reaped child process 2112, status = succeeded7108worker1 # [ 167.550103] nix-daemon[870]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)7109worker1 # [ 168.977426] nix-daemon[870]: reaped child process 2126, status = succeeded7110worker1 # [ 169.185520] nix-daemon[870]: accepted connection from pid 2142, user nix-grpc-daemon (trusted)7111worker1 # [ 169.197388] nix-daemon[870]: reaped child process 2145, status = succeeded7112lb # [ 169.219601] niks3-server[1150]: 2026/09/21 13:56:26 INFO Received uploads request method=POST path=/api/pending_closures7113worker1 # [ 169.227646] nix-grpc-daemon[1820]: time=2026-09-21T13:56:26.363Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7114worker1 # [ 169.230057] nix-grpc-daemon[1820]: time=2026-09-21T13:56:26.365Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7115lb # [ 169.264753] niks3-server[1150]: 2026/09/21 13:56:26 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7116lb # [ 169.270911] niks3-server[1150]: 2026/09/21 13:56:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7117lb # [ 169.274318] niks3-server[1150]: 2026/09/21 13:56:26 INFO Signed narinfos id=69 count=17118worker1 # [ 169.275783] nix-grpc-daemon[1820]: time=2026-09-21T13:56:26.412Z level=INFO msg="Uploading 1 narinfos"7119lb # [ 169.278612] niks3-server[1150]: 2026/09/21 13:56:26 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7120lb # [ 169.283454] niks3-server[1150]: 2026/09/21 13:56:26 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7121lb # [ 169.291340] niks3-server[1150]: 2026/09/21 13:56:26 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7122lb # [ 169.297172] niks3-server[1150]: 2026/09/21 13:56:26 INFO Completed upload id=697123lb # [ 169.299029] niks3-server[1150]: 2026/09/21 13:56:26 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7124worker1 # [ 169.297567] nix-grpc-daemon[1820]: time=2026-09-21T13:56:26.434Z level=INFO msg="Upload complete. (337ms)"7125worker1 # [ 169.302944] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33432 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=15 outputs=17126worker1 # [ 169.316474] nix-daemon[870]: reaped child process 2124, status = succeeded7127client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 19.14 seconds)7128worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7129worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7130client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'7131client # [ 169.505481] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7132client # Running as unit: gpu.service; invocation ID: 319d3b00b7db47eabd878229c95704b17133client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.07 seconds)7134lb: must succeed: curl -sf localhost:9901/clusters7135lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)7136lb: must succeed: curl -sf localhost:9901/clusters7137lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7138worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7139worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7140worker1 # [ 169.738056] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:334327141client # [ 169.752157] nix[1375]: don't know how to build these paths:7142client # [ 169.753076] nix[1375]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7143worker2 # [ 169.740506] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:355207144client # [ 169.777417] nix[1375]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker7145worker2 # [ 169.763179] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:26Z level=debug event=want client=6 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07146lb: must succeed: curl -sf localhost:9901/clusters7147lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)7148lb: must succeed: curl -sf localhost:9901/clusters7149lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7150worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7151worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7152client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7153lb # [ 176.495499] envoy[901]: conn peer=[2001:db8:1::1]:59924 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27145 tx=11358 ms=188437154client # [ 199.779761] nix[1375]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}7155client # [ 199.787524] nix[1375]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7156client # [ 199.817774] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7157client # [ 199.819596] systemd[1]: gpu.service: Failed with result 'exit-code'.7158client # [ 199.821329] systemd[1]: gpu.service: Consumed 146ms CPU time over 30.314s wall clock time, 37.4M memory peak, 5.1K incoming IP traffic, 5.1K outgoing IP traffic.7159client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.91 seconds)7160client: must succeed: journalctl -u gpu -o cat7161client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)7162lb: must succeed: curl -sf localhost:9901/clusters7163lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)7164lb: must succeed: curl -sf localhost:9901/clusters7165lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)7166worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7167worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7168(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 50.84 seconds)7169subtest: low disk drains a worker and builds go to the other7170worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7171worker1 # [ 201.946134] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:59Z level=info event=unhealthy reason=min_free7172worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.94 seconds)7173worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7174worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.05 seconds)7175client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&27176worker2 # [ 202.316871] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:355207177client # don't know how to build these paths:7178client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7179worker1 # [ 202.332658] nix-grpc-daemon[1760]: ts=2026-09-21T13:56:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334327180worker2 # [ 202.349909] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:58Z level=debug event=want client=7 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07181worker2 # [ 202.355246] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:58Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17182worker2 # [ 202.395222] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355207183worker2 # [ 202.408234] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:355207184client # copying 1 paths to 'lb:50051'...7185client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7186worker2 # [ 202.421080] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:355207187worker2 # [ 202.433487] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)7188worker2 # [ 202.622405] nix-daemon[853]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)7189worker2 # [ 202.638099] nix-daemon[853]: reaped child process 1764, status = succeeded7190worker2 # [ 202.832471] nix-daemon[853]: accepted connection from pid 1766, user nix-grpc-daemon (trusted)7191worker2 # [ 202.846763] nix-daemon[853]: reaped child process 1769, status = succeeded7192lb # [ 202.873421] niks3-server[1150]: 2026/09/21 13:56:59 INFO Received uploads request method=POST path=/api/pending_closures7193worker2 # [ 202.875888] nix-grpc-daemon[1546]: time=2026-09-21T13:56:59.188Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7194worker2 # [ 202.877897] nix-grpc-daemon[1546]: time=2026-09-21T13:56:59.189Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7195lb # [ 202.911600] niks3-server[1150]: 2026/09/21 13:56:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7196lb # [ 202.918608] niks3-server[1150]: 2026/09/21 13:56:59 INFO Signed narinfos id=70 count=17197worker2 # [ 202.914534] nix-grpc-daemon[1546]: time=2026-09-21T13:56:59.226Z level=INFO msg="Uploading 1 narinfos"7198lb # [ 202.926408] niks3-server[1150]: 2026/09/21 13:56:59 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7199lb # [ 202.932179] niks3-server[1150]: 2026/09/21 13:56:59 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7200lb # [ 202.942824] niks3-server[1150]: 2026/09/21 13:56:59 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7201lb # [ 202.949367] niks3-server[1150]: 2026/09/21 13:56:59 INFO Completed upload id=707202worker2 # [ 202.942514] nix-grpc-daemon[1546]: time=2026-09-21T13:56:59.254Z level=INFO msg="Upload complete. (497ms)"7203lb # [ 202.954718] niks3-server[1150]: 2026/09/21 13:56:59 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7204worker2 # [ 202.946931] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=0 paths=1 nar_bytes_in=5127205worker2 # [ 202.954355] nix-grpc-daemon[1470]: ts=2026-09-21T13:56:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355207206worker2 # [ 202.963773] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)7207worker2 # [ 202.972343] nix-daemon[853]: reaped child process 1759, status = succeeded7208client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7209client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7210worker2 # [ 202.978778] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)7211client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7212client # farm-a> LOG-farm-a-drain7213worker2 # [ 204.392913] nix-daemon[853]: reaped child process 1773, status = succeeded7214worker2 # [ 204.574100] nix-daemon[853]: accepted connection from pid 1789, user nix-grpc-daemon (trusted)7215worker2 # [ 204.586602] nix-daemon[853]: reaped child process 1792, status = succeeded7216lb # [ 204.614445] niks3-server[1150]: 2026/09/21 13:57:01 INFO Received uploads request method=POST path=/api/pending_closures7217worker2 # [ 204.616115] nix-grpc-daemon[1546]: time=2026-09-21T13:57:00.928Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7218worker2 # [ 204.617690] nix-grpc-daemon[1546]: time=2026-09-21T13:57:00.930Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7219lb # [ 204.654684] niks3-server[1150]: 2026/09/21 13:57:01 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7220lb # [ 204.666271] niks3-server[1150]: 2026/09/21 13:57:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7221lb # [ 204.670771] niks3-server[1150]: 2026/09/21 13:57:01 INFO Signed narinfos id=71 count=17222worker2 # [ 204.666835] nix-grpc-daemon[1546]: time=2026-09-21T13:57:00.979Z level=INFO msg="Uploading 1 narinfos"7223lb # [ 204.678467] niks3-server[1150]: 2026/09/21 13:57:01 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7224lb # [ 204.686474] niks3-server[1150]: 2026/09/21 13:57:01 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7225lb # [ 204.694240] niks3-server[1150]: 2026/09/21 13:57:01 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7226lb # [ 204.700466] niks3-server[1150]: 2026/09/21 13:57:01 INFO Completed upload id=717227worker2 # [ 204.693745] nix-grpc-daemon[1546]: time=2026-09-21T13:57:01.006Z level=INFO msg="Upload complete. (308ms)"7228lb # [ 204.702234] niks3-server[1150]: 2026/09/21 13:57:01 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7229worker2 # [ 204.703640] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=16 outputs=17230worker2 # [ 204.712324] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:01Z level=debug event=want client=7 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07231worker2 # [ 204.717230] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:01Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17232worker2 # [ 204.726324] nix-daemon[853]: reaped child process 1771, status = succeeded7233worker2 # [ 204.757397] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355207234worker2 # [ 204.769680] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:355207235worker2 # [ 204.782827] nix-grpc-daemon[1546]: time=2026-09-21T13:57:01.095Z level=INFO msg="All 1 paths already cached"7236client # copying 1 paths to 'lb:50051'...7237client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7238worker2 # [ 204.790127] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:355207239worker2 # [ 204.801163] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)7240worker2 # [ 205.006571] nix-daemon[853]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7241worker2 # [ 205.021626] nix-daemon[853]: reaped child process 1801, status = succeeded7242worker2 # [ 205.210619] nix-daemon[853]: accepted connection from pid 1803, user nix-grpc-daemon (trusted)7243worker2 # [ 205.224942] nix-daemon[853]: reaped child process 1806, status = succeeded7244lb # [ 205.250917] niks3-server[1150]: 2026/09/21 13:57:02 INFO Received uploads request method=POST path=/api/pending_closures7245worker2 # [ 205.252134] nix-grpc-daemon[1546]: time=2026-09-21T13:57:01.564Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7246worker2 # [ 205.253740] nix-grpc-daemon[1546]: time=2026-09-21T13:57:01.566Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7247lb # [ 205.283300] niks3-server[1150]: 2026/09/21 13:57:02 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7248lb # [ 205.293100] niks3-server[1150]: 2026/09/21 13:57:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7249worker2 # [ 205.291789] nix-grpc-daemon[1546]: time=2026-09-21T13:57:01.604Z level=INFO msg="Uploading 1 narinfos"7250lb # [ 205.299433] niks3-server[1150]: 2026/09/21 13:57:02 INFO Signed narinfos id=72 count=17251lb # [ 205.304708] niks3-server[1150]: 2026/09/21 13:57:02 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7252lb # [ 205.316873] niks3-server[1150]: 2026/09/21 13:57:02 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7253lb # [ 205.321856] niks3-server[1150]: 2026/09/21 13:57:02 INFO Completed upload id=727254worker2 # [ 205.317132] nix-grpc-daemon[1546]: time=2026-09-21T13:57:01.629Z level=INFO msg="Upload complete. (503ms)"7255lb # [ 205.324936] niks3-server[1150]: 2026/09/21 13:57:02 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7256worker2 # [ 205.320848] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=0 paths=1 nar_bytes_in=6247257worker2 # [ 205.327297] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355207258worker2 # [ 205.335836] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)7259worker2 # [ 205.347476] nix-daemon[853]: reaped child process 1796, status = succeeded7260client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7261client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7262worker2 # [ 205.351785] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)7263client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7264client # farm-b> LOG-farm-b-drain7265worker2 # [ 206.374151] nix-daemon[853]: reaped child process 1810, status = succeeded7266lb # [ 206.497802] envoy[901]: conn peer=[2001:db8:1::1]:55032 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300507267worker2 # [ 206.554620] nix-daemon[853]: accepted connection from pid 1826, user nix-grpc-daemon (trusted)7268worker2 # [ 206.568800] nix-daemon[853]: reaped child process 1829, status = succeeded7269lb # [ 206.595920] niks3-server[1150]: 2026/09/21 13:57:03 INFO Received uploads request method=POST path=/api/pending_closures7270worker2 # [ 206.597974] nix-grpc-daemon[1546]: time=2026-09-21T13:57:02.910Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7271worker2 # [ 206.604168] nix-grpc-daemon[1546]: time=2026-09-21T13:57:02.912Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7272lb # [ 206.630287] niks3-server[1150]: 2026/09/21 13:57:03 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7273lb # [ 206.643602] niks3-server[1150]: 2026/09/21 13:57:03 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7274lb # [ 206.655596] niks3-server[1150]: 2026/09/21 13:57:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7275lb # [ 206.659105] niks3-server[1150]: 2026/09/21 13:57:03 INFO Signed narinfos id=73 count=17276worker2 # [ 206.654514] nix-grpc-daemon[1546]: time=2026-09-21T13:57:02.966Z level=INFO msg="Uploading 1 narinfos"7277lb # [ 206.665150] niks3-server[1150]: 2026/09/21 13:57:03 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7278lb # [ 206.672827] niks3-server[1150]: 2026/09/21 13:57:03 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7279lb # [ 206.676783] niks3-server[1150]: 2026/09/21 13:57:03 INFO Completed upload id=737280worker2 # [ 206.669851] nix-grpc-daemon[1546]: time=2026-09-21T13:57:02.982Z level=INFO msg="Upload complete. (311ms)"7281lb # [ 206.678577] niks3-server[1150]: 2026/09/21 13:57:03 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7282worker2 # [ 206.675295] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=17 outputs=17283worker2 # [ 206.686799] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:02Z level=debug event=want client=7 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07284worker2 # [ 206.690142] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:02Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17285worker2 # [ 206.696991] nix-daemon[853]: reaped child process 1808, status = succeeded7286worker2 # [ 206.731079] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355207287worker2 # [ 206.742932] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:355207288worker2 # [ 206.756618] nix-grpc-daemon[1546]: time=2026-09-21T13:57:03.069Z level=INFO msg="All 2 paths already cached"7289client # copying 1 paths to 'lb:50051'...7290client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7291worker2 # [ 206.765199] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:355207292worker2 # [ 206.773987] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)7293worker2 # [ 206.977370] nix-daemon[853]: accepted connection from pid 1835, user nix-grpc-daemon (trusted)7294worker2 # [ 206.992655] nix-daemon[853]: reaped child process 1838, status = succeeded7295worker2 # [ 207.183525] nix-daemon[853]: accepted connection from pid 1840, user nix-grpc-daemon (trusted)7296worker2 # [ 207.197760] nix-daemon[853]: reaped child process 1843, status = succeeded7297lb # [ 207.223966] niks3-server[1150]: 2026/09/21 13:57:04 INFO Received uploads request method=POST path=/api/pending_closures7298worker2 # [ 207.226073] nix-grpc-daemon[1546]: time=2026-09-21T13:57:03.538Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7299worker2 # [ 207.227990] nix-grpc-daemon[1546]: time=2026-09-21T13:57:03.540Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7300lb # [ 207.253760] niks3-server[1150]: 2026/09/21 13:57:04 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7301lb # [ 207.264336] niks3-server[1150]: 2026/09/21 13:57:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7302worker2 # [ 207.262120] nix-grpc-daemon[1546]: time=2026-09-21T13:57:03.574Z level=INFO msg="Uploading 1 narinfos"7303lb # [ 207.269086] niks3-server[1150]: 2026/09/21 13:57:04 INFO Signed narinfos id=74 count=17304lb # [ 207.275858] niks3-server[1150]: 2026/09/21 13:57:04 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7305lb # [ 207.286925] niks3-server[1150]: 2026/09/21 13:57:04 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7306lb # [ 207.293275] niks3-server[1150]: 2026/09/21 13:57:04 INFO Completed upload id=747307worker2 # [ 207.288395] nix-grpc-daemon[1546]: time=2026-09-21T13:57:03.600Z level=INFO msg="Upload complete. (501ms)"7308lb # [ 207.297109] niks3-server[1150]: 2026/09/21 13:57:04 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7309worker2 # [ 207.291703] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=0 paths=1 nar_bytes_in=7527310worker2 # [ 207.296690] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:355207311worker2 # [ 207.306536] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)7312worker2 # [ 207.312593] nix-daemon[853]: reaped child process 1833, status = succeeded7313worker2 # [ 207.317946] nix-daemon[853]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)7314client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7315client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7316client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7317client # farm-top> LOG-farm-top-drain7318worker2 # [ 208.749198] nix-daemon[853]: reaped child process 1847, status = succeeded7319worker2 # [ 208.956800] nix-daemon[853]: accepted connection from pid 1863, user nix-grpc-daemon (trusted)7320worker2 # [ 208.969389] nix-daemon[853]: reaped child process 1866, status = succeeded7321lb # [ 208.997282] niks3-server[1150]: 2026/09/21 13:57:05 INFO Received uploads request method=POST path=/api/pending_closures7322worker2 # [ 208.999862] nix-grpc-daemon[1546]: time=2026-09-21T13:57:05.312Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7323worker2 # [ 209.001486] nix-grpc-daemon[1546]: time=2026-09-21T13:57:05.313Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7324lb # [ 209.034307] niks3-server[1150]: 2026/09/21 13:57:05 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7325lb # [ 209.047211] niks3-server[1150]: 2026/09/21 13:57:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7326lb # [ 209.051891] niks3-server[1150]: 2026/09/21 13:57:05 INFO Signed narinfos id=75 count=17327worker2 # [ 209.047875] nix-grpc-daemon[1546]: time=2026-09-21T13:57:05.360Z level=INFO msg="Uploading 1 narinfos"7328lb # [ 209.061376] niks3-server[1150]: 2026/09/21 13:57:05 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7329lb # [ 209.065655] niks3-server[1150]: 2026/09/21 13:57:05 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7330lb # [ 209.074728] niks3-server[1150]: 2026/09/21 13:57:05 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7331lb # [ 209.079614] niks3-server[1150]: 2026/09/21 13:57:05 INFO Completed upload id=757332worker2 # [ 209.072788] nix-grpc-daemon[1546]: time=2026-09-21T13:57:05.385Z level=INFO msg="Upload complete. (344ms)"7333lb # [ 209.082229] niks3-server[1150]: 2026/09/21 13:57:05 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7334worker2 # [ 209.080588] nix-grpc-daemon[1470]: ts=2026-09-21T13:57:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35520 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=18 outputs=17335worker2 # [ 209.094403] nix-daemon[853]: reaped child process 1845, status = succeeded7336client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 7.07 seconds)7337client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7338client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.20 seconds)7339worker1: must succeed: rm /nix/.rw-store/fill7340worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.13 seconds)7341worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7342worker1 # [ 209.949861] nix-grpc-daemon[1760]: ts=2026-09-21T13:57:07Z level=info event=healthy reason=min_free7343worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.13 seconds)7344(finished: subtest: low disk drains a worker and builds go to the other, in 9.61 seconds)7345(finished: run the VM test script, in 211.59 seconds)7346test script finished in 211.85s7347cleanup7348kill QemuMachine (pid 49)7349client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7350kill QemuMachine (pid 50)7351lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7352kill QemuMachine (pid 51)7353worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7354kill QemuMachine (pid 52)7355worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7356(finished: cleanup, in 0.99 seconds)7357additionally exposed symbols:7358 client, lb, worker1, worker2,7359 vlan1,7360 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh