vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #487
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker2: QEMU running (pid 52)13worker2 # Disk image does not exist, creating the virtualisation disk image...14worker2 # Formatting '/build/vm-state-worker2/tmp.loQVMC8CK9', fmt=raw size=107374182415worker2 # mke2fs 1.47.4 (6-Mar-2025)16worker2 # Discarding device blocks: 0/262144 done17worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes18worker2 # Filesystem UUID: d4a9debd-c298-4c42-877d-df53317dcf7b19worker2 # Superblock backups stored on blocks:20worker2 # 32768, 98304, 163840, 22937621worker2 # 22worker2 # Allocating group tables: 0/8 done23worker2 # Writing inode tables: 0/8 done24worker2 # Creating journal (8192 blocks): done25worker2 # Writing superblocks and filesystem accounting information: 0/8 done26worker2 # 27worker2 # Virtualisation disk image created.28client # Disk image does not exist, creating the virtualisation disk image...29client: QEMU running (pid 49)30lb: QEMU running (pid 50)31lb # Disk image does not exist, creating the virtualisation disk image...32client # Formatting '/build/vm-state-client/tmp.9DDKJOuVXY', fmt=raw size=107374182433lb # Formatting '/build/vm-state-lb/tmp.CTZsPvLhLv', fmt=raw size=107374182434client # mke2fs 1.47.4 (6-Mar-2025)35lb # mke2fs 1.47.4 (6-Mar-2025)36client # Discarding device blocks: 0/262144 done37lb # Discarding device blocks: 0/262144 done38client # Creating filesystem with 262144 4k blocks and 65536 inodes39lb # Creating filesystem with 262144 4k blocks and 65536 inodes40client # Filesystem UUID: 2651a5d8-bf36-48d4-b2a8-6d14f6fe9d1841lb # Filesystem UUID: da4fcb73-cabc-4d40-a187-8b1538919ef442client # Superblock backups stored on blocks:43lb # Superblock backups stored on blocks:44client # 32768, 98304, 163840, 22937645lb # 32768, 98304, 163840, 22937646worker1 # Disk image does not exist, creating the virtualisation disk image...47worker1: QEMU running (pid 51)48client # 49lb # 50client # Allocating group tables: 0/8 done51lb # Allocating group tables: 0/8 done52worker1 # Formatting '/build/vm-state-worker1/tmp.6HCDNv5x4u', fmt=raw size=107374182453(finished: start all VMs, in 0.20 seconds)54client # Writing inode tables: 0/8 done55lb # Writing inode tables: 0/8 done56worker1 # mke2fs 1.47.4 (6-Mar-2025)57worker1 # Discarding device blocks: 0/262144 done58client: waiting for unit network-addresses-eth1.service59lb # Creating journal (8192 blocks): done60lb # Writing superblocks and filesystem accounting information: 0/8 done61client # Creating journal (8192 blocks): done62worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes63lb # 64client: waiting for the VM to finish booting65client # Writing superblocks and filesystem accounting information: 0/8 done66worker1 # Filesystem UUID: 48b7cf2f-c68a-43b8-a3e2-15233c382ca067client # 68worker1 # Superblock backups stored on blocks:69client # Virtualisation disk image created.70worker1 # 32768, 98304, 163840, 22937671lb # Virtualisation disk image created.72worker1 # 73worker1 # Allocating group tables: 0/8 done74worker1 # Writing inode tables: 0/8 done75worker1 # Creating journal (8192 blocks): done76worker1 # Writing superblocks and filesystem accounting information: 0/8 done77worker1 # 78worker1 # Virtualisation disk image created.79worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83worker2 # 84worker2 # 85worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0086worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...87worker2 # 88worker2 # 89worker2 # 90worker2 # 91worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB0092worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...93worker2 # 94worker2 # 95worker1 # 96worker1 # 97client # 98client # 99worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00100client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00101lb # 102lb # 103lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00104worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...105worker1 # 106worker1 # 107worker1 # 108worker1 # 109client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...110client # 111client # 112client # 113client # 114worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00115client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00116lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...117lb # 118lb # 119lb # 120lb # 121lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00122worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...123worker1 # 124worker1 # 125client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...126client # 127client # 128lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129lb # 130lb # 131worker2 # Booting from ROM...132client # Booting from ROM...133worker1 # Booting from ROM...134lb # Booting from ROM...135worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/y2d0s28bxdlycgqmymvml6a4g8hm32vg-nixos-system-worker2-test/init regInfo=/nix/store/2pxjmjvppdrxyd9pnpjdylzbhar9qj1i-closure-info/registration console=ttyS0,115200n8 console=tty0137worker2 # [ 0.000000] BIOS-provided physical RAM map:138worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved141worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable142worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved143worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved144worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved145worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved146worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved147worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved148worker2 # [ 0.000000] NX (Execute Disable) protection: active149worker2 # [ 0.000000] APIC: Static calls initialized150worker2 # [ 0.000000] SMBIOS 2.8 present.151worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014152worker2 # [ 0.000000] DMI: Memory slots populated: 1/1153worker2 # [ 0.000000] Hypervisor detected: KVM154worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000155worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00156worker2 # [ 0.000000] kvm-clock: using sched offset of 454923125 cycles157worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns158worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor159worker2 # [ 0.000811] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000160worker2 # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs161client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026162worker2 # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT163worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026164worker2 # [ 0.002736] found SMP MP-table at [mem 0x000f5450-0x000f545f]165worker2 # [ 0.002748] Using GB pages for direct mapping166worker2 # [ 0.002869] RAMDISK: [mem 0x5e367000-0x5ffcffff]167worker2 # [ 0.002876] ACPI: Early table checksum verification disabled168worker2 # [ 0.002879] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )169client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/jwz4idn9byb3rgqdqa3827m7srpcmdj8-nixos-system-client-test/init regInfo=/nix/store/k6c8bg1s150rn281il5cadignqaai16m-closure-info/registration console=ttyS0,115200n8 console=tty0170client # [ 0.000000] BIOS-provided physical RAM map:171worker2 # [ 0.002882] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)172worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/m861wijgs462ggfjaanhm99ifwyrs751-nixos-system-worker1-test/init regInfo=/nix/store/g9w4psxi6bzn798fqi8gw8a3hbjf7pi9-closure-info/registration console=ttyS0,115200n8 console=tty0173worker1 # [ 0.000000] BIOS-provided physical RAM map:174client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable175worker2 # [ 0.002886] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)176worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable177client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved178worker2 # [ 0.002893] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)179worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved180client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved181worker2 # [ 0.002895] ACPI: FACS 0x000000005FFE0000 000040182worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved183client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable184worker2 # [ 0.002896] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)185worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable186client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved187worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved188worker2 # [ 0.002898] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)189client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved190worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved191client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved192worker2 # [ 0.002899] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)193worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved194client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved195worker2 # [ 0.002901] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)196lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026197worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved198client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved199worker2 # [ 0.002902] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]200worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved201client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved202worker2 # [ 0.002903] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]203worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved204client # [ 0.000000] NX (Execute Disable) protection: active205worker2 # [ 0.002904] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]206worker1 # [ 0.000000] NX (Execute Disable) protection: active207client # [ 0.000000] APIC: Static calls initialized208client # [ 0.000000] SMBIOS 2.8 present.209worker1 # [ 0.000000] APIC: Static calls initialized210worker2 # [ 0.002904] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]211worker1 # [ 0.000000] SMBIOS 2.8 present.212lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/j1nn9bbhrn5xz154akfhw453f8i07i4d-nixos-system-lb-test/init regInfo=/nix/store/hq8v7nws787a9kwd3h009z1n9dqal4fl-closure-info/registration console=ttyS0,115200n8 console=tty0213client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014214worker2 # [ 0.002905] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]215lb # [ 0.000000] BIOS-provided physical RAM map:216worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014217client # [ 0.000000] DMI: Memory slots populated: 1/1218worker2 # [ 0.002905] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]219client # [ 0.000000] Hypervisor detected: KVM220lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable221worker1 # [ 0.000000] DMI: Memory slots populated: 1/1222worker1 # [ 0.000000] Hypervisor detected: KVM223client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000224worker2 # [ 0.002906] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]225lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved226worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000227worker2 # [ 0.003130] No NUMA configuration found228client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00229worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00230lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved231client # [ 0.000000] kvm-clock: using sched offset of 496613358 cycles232worker2 # [ 0.003131] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]233worker1 # [ 0.000001] kvm-clock: using sched offset of 472181619 cycles234lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable235worker2 # [ 0.003134] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]236worker2 # [ 0.005431] Zone ranges:237client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns238lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved239client # [ 0.000005] tsc: Detected 2400.012 MHz processor240worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns241worker2 # [ 0.005432] DMA [mem 0x0000000000001000-0x0000000000ffffff]242worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor243lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved244client # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000245worker2 # [ 0.005433] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]246worker1 # [ 0.000814] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000247worker2 # [ 0.005434] Normal empty248lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved249worker2 # [ 0.005435] Device empty250client # [ 0.000839] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs251worker2 # [ 0.005435] Movable zone start for each node252worker1 # [ 0.000841] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs253lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved254worker2 # [ 0.005436] Early memory node ranges255client # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT256worker1 # [ 0.000843] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT257lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved258client # [ 0.002736] found SMP MP-table at [mem 0x000f5450-0x000f545f]259worker2 # [ 0.005437] node 0: [mem 0x0000000000001000-0x000000000009efff]260client # [ 0.002747] Using GB pages for direct mapping261worker1 # [ 0.002738] found SMP MP-table at [mem 0x000f5450-0x000f545f]262lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved263worker2 # [ 0.005437] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]264worker1 # [ 0.002749] Using GB pages for direct mapping265client # [ 0.002850] RAMDISK: [mem 0x5e366000-0x5ffcffff]266lb # [ 0.000000] NX (Execute Disable) protection: active267worker2 # [ 0.005439] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]268worker1 # [ 0.002864] RAMDISK: [mem 0x5e367000-0x5ffcffff]269client # [ 0.002860] ACPI: Early table checksum verification disabled270lb # [ 0.000000] APIC: Static calls initialized271lb # [ 0.000000] SMBIOS 2.8 present.272worker2 # [ 0.005459] On node 0, zone DMA: 1 pages in unavailable ranges273worker1 # [ 0.002874] ACPI: Early table checksum verification disabled274client # [ 0.002863] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )275worker1 # [ 0.002877] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )276worker2 # [ 0.005723] On node 0, zone DMA: 97 pages in unavailable ranges277lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014278client # [ 0.002867] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)279worker2 # [ 0.032676] On node 0, zone DMA32: 40 pages in unavailable ranges280lb # [ 0.000000] DMI: Memory slots populated: 1/1281worker1 # [ 0.002881] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)282worker2 # [ 0.033132] ACPI: PM-Timer IO Port: 0x608283lb # [ 0.000000] Hypervisor detected: KVM284client # [ 0.002870] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)285worker1 # [ 0.002884] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)286worker2 # [ 0.033144] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])287lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000288client # [ 0.002877] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)289lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00290worker2 # [ 0.033171] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23291worker1 # [ 0.002891] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)292client # [ 0.002879] ACPI: FACS 0x000000005FFE0000 000040293lb # [ 0.000000] kvm-clock: using sched offset of 483091004 cycles294worker2 # [ 0.033173] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)295worker1 # [ 0.002893] ACPI: FACS 0x000000005FFE0000 000040296client # [ 0.002881] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)297worker2 # [ 0.033175] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)298lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns299worker1 # [ 0.002895] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)300client # [ 0.002882] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)301lb # [ 0.000005] tsc: Detected 2400.012 MHz processor302worker2 # [ 0.033176] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)303worker1 # [ 0.002896] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)304lb # [ 0.000815] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000305worker2 # [ 0.033177] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)306client # [ 0.002884] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)307worker1 # [ 0.002898] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)308lb # [ 0.000841] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs309worker2 # [ 0.033178] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)310client # [ 0.002885] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)311worker2 # [ 0.033180] ACPI: Using ACPI (MADT) for SMP configuration information312worker1 # [ 0.002899] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)313lb # [ 0.000844] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT314client # [ 0.002887] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]315worker2 # [ 0.033181] ACPI: HPET id: 0x8086a201 base: 0xfed00000316lb # [ 0.002737] found SMP MP-table at [mem 0x000f5450-0x000f545f]317worker1 # [ 0.002900] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]318worker2 # [ 0.033185] TSC deadline timer available319lb # [ 0.002748] Using GB pages for direct mapping320client # [ 0.002888] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]321worker2 # [ 0.033188] CPU topo: Max. logical packages: 1322worker1 # [ 0.002902] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]323lb # [ 0.002861] RAMDISK: [mem 0x5e367000-0x5ffcffff]324client # [ 0.002888] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]325worker2 # [ 0.033189] CPU topo: Max. logical dies: 1326lb # [ 0.002868] ACPI: Early table checksum verification disabled327worker1 # [ 0.002902] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]328worker2 # [ 0.033190] CPU topo: Max. dies per package: 1329client # [ 0.002889] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]330lb # [ 0.002871] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )331worker2 # [ 0.033193] CPU topo: Max. threads per core: 1332worker1 # [ 0.002903] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]333client # [ 0.002889] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]334worker2 # [ 0.033194] CPU topo: Num. cores per package: 1335lb # [ 0.002875] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)336worker1 # [ 0.002903] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]337worker2 # [ 0.033194] CPU topo: Num. threads per package: 1338client # [ 0.002890] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]339worker1 # [ 0.002904] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]340lb # [ 0.002878] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)341worker2 # [ 0.033194] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs342client # [ 0.002890] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]343client # [ 0.003122] No NUMA configuration found344worker1 # [ 0.002904] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]345worker2 # [ 0.033212] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()346lb # [ 0.002885] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)347worker1 # [ 0.003143] No NUMA configuration found348client # [ 0.003123] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]349lb # [ 0.002887] ACPI: FACS 0x000000005FFE0000 000040350worker2 # [ 0.033243] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]351worker1 # [ 0.003144] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]352client # [ 0.003125] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]353lb # [ 0.002888] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)354client # [ 0.005666] Zone ranges:355worker1 # [ 0.003146] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]356worker2 # [ 0.033244] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]357worker1 # [ 0.005863] Zone ranges:358client # [ 0.005667] DMA [mem 0x0000000000001000-0x0000000000ffffff]359worker2 # [ 0.033245] [mem 0x60000000-0xafffffff] available for PCI devices360lb # [ 0.002890] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)361worker1 # [ 0.005864] DMA [mem 0x0000000000001000-0x0000000000ffffff]362client # [ 0.005668] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]363worker2 # [ 0.033247] Booting paravirtualized kernel on KVM364client # [ 0.005669] Normal empty365worker1 # [ 0.005866] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]366lb # [ 0.002891] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)367client # [ 0.005670] Device empty368worker1 # [ 0.005867] Normal empty369worker1 # [ 0.005868] Device empty370client # [ 0.005671] Movable zone start for each node371worker2 # [ 0.033250] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns372lb # [ 0.002893] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)373client # [ 0.005671] Early memory node ranges374worker1 # [ 0.005869] Movable zone start for each node375worker2 # [ 0.037697] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1376worker1 # [ 0.005869] Early memory node ranges377client # [ 0.005672] node 0: [mem 0x0000000000001000-0x000000000009efff]378lb # [ 0.002894] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]379worker1 # [ 0.005870] node 0: [mem 0x0000000000001000-0x000000000009efff]380worker2 # [ 0.039897] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152381client # [ 0.005673] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]382lb # [ 0.002895] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]383worker2 # [ 0.039939] kvm-guest: PV spinlocks disabled, single CPU384worker1 # [ 0.005871] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]385client # [ 0.005674] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]386lb # [ 0.002896] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]387client # [ 0.005693] On node 0, zone DMA: 1 pages in unavailable ranges388worker1 # [ 0.005872] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]389lb # [ 0.002896] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]390worker1 # [ 0.005890] On node 0, zone DMA: 1 pages in unavailable ranges391client # [ 0.005967] On node 0, zone DMA: 97 pages in unavailable ranges392worker1 # [ 0.006156] On node 0, zone DMA: 97 pages in unavailable ranges393lb # [ 0.002897] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]394client # [ 0.032143] On node 0, zone DMA32: 40 pages in unavailable ranges395worker1 # [ 0.032385] On node 0, zone DMA32: 40 pages in unavailable ranges396client # [ 0.032608] ACPI: PM-Timer IO Port: 0x608397lb # [ 0.002897] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]398worker1 # [ 0.032846] ACPI: PM-Timer IO Port: 0x608399client # [ 0.032619] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])400worker2 # [ 0.039941] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/y2d0s28bxdlycgqmymvml6a4g8hm32vg-nixos-system-worker2-test/init regInfo=/nix/store/2pxjmjvppdrxyd9pnpjdylzbhar9qj1i-closure-info/registration console=ttyS0,115200n8 console=tty0401lb # [ 0.002898] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]402worker1 # [ 0.032858] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])403lb # [ 0.003122] No NUMA configuration found404client # [ 0.032646] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23405worker1 # [ 0.032889] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23406lb # [ 0.003123] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]407client # [ 0.032649] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)408worker2 # [ 0.040033] Unknown kernel command line parameters "regInfo=/nix/store/2pxjmjvppdrxyd9pnpjdylzbhar9qj1i-closure-info/registration", will be passed to user space.409worker1 # [ 0.032893] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)410lb # [ 0.003125] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]411worker2 # [ 0.040045] random: crng init done412client # [ 0.032651] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)413lb # [ 0.005446] Zone ranges:414worker1 # [ 0.032894] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)415worker2 # [ 0.040046] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes416client # [ 0.032652] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)417lb # [ 0.005446] DMA [mem 0x0000000000001000-0x0000000000ffffff]418worker1 # [ 0.032895] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)419lb # [ 0.005448] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]420client # [ 0.032653] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)421worker2 # [ 0.042194] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)422lb # [ 0.005449] Normal empty423worker1 # [ 0.032896] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)424lb # [ 0.005449] Device empty425client # [ 0.032654] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)426worker2 # [ 0.043272] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)427lb # [ 0.005450] Movable zone start for each node428worker1 # [ 0.032897] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)429worker2 # [ 0.043304] Fallback order for Node 0: 0430client # [ 0.032656] ACPI: Using ACPI (MADT) for SMP configuration information431lb # [ 0.005451] Early memory node ranges432worker1 # [ 0.032900] ACPI: Using ACPI (MADT) for SMP configuration information433client # [ 0.032657] ACPI: HPET id: 0x8086a201 base: 0xfed00000434worker2 # [ 0.043307] Built 1 zonelists, mobility grouping on. Total pages: 393078435lb # [ 0.005451] node 0: [mem 0x0000000000001000-0x000000000009efff]436worker2 # [ 0.043308] Policy zone: DMA32437client # [ 0.032661] TSC deadline timer available438worker1 # [ 0.032900] ACPI: HPET id: 0x8086a201 base: 0xfed00000439worker1 # [ 0.032904] TSC deadline timer available440lb # [ 0.005452] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]441client # [ 0.032665] CPU topo: Max. logical packages: 1442worker2 # [ 0.046087] mem auto-init: stack:all(zero), heap alloc:on, heap free:off443worker1 # [ 0.032908] CPU topo: Max. logical packages: 1444client # [ 0.032666] CPU topo: Max. logical dies: 1445lb # [ 0.005453] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]446worker2 # [ 0.049174] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1447worker1 # [ 0.032909] CPU topo: Max. logical dies: 1448client # [ 0.032667] CPU topo: Max. dies per package: 1449lb # [ 0.005472] On node 0, zone DMA: 1 pages in unavailable ranges450worker2 # [ 0.052563] allocated 3145728 bytes of page_ext451worker1 # [ 0.032910] CPU topo: Max. dies per package: 1452client # [ 0.032670] CPU topo: Max. threads per core: 1453worker1 # [ 0.032913] CPU topo: Max. threads per core: 1454worker2 # [ 0.062358] ftrace: allocating 48731 entries in 192 pages455lb # [ 0.005744] On node 0, zone DMA: 97 pages in unavailable ranges456client # [ 0.032671] CPU topo: Num. cores per package: 1457worker1 # [ 0.032914] CPU topo: Num. cores per package: 1458client # [ 0.032671] CPU topo: Num. threads per package: 1459worker2 # [ 0.062360] ftrace: allocated 192 pages with 2 groups460lb # [ 0.032873] On node 0, zone DMA32: 40 pages in unavailable ranges461worker2 # [ 0.063201] Dynamic Preempt: lazy462worker1 # [ 0.032914] CPU topo: Num. threads per package: 1463lb # [ 0.033339] ACPI: PM-Timer IO Port: 0x608464client # [ 0.032671] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs465worker2 # [ 0.063324] rcu: Preemptible hierarchical RCU implementation.466worker1 # [ 0.032914] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs467lb # [ 0.033350] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])468client # [ 0.032690] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()469worker2 # [ 0.063324] rcu: RCU event tracing is enabled.470worker1 # [ 0.032933] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()471lb # [ 0.033376] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23472client # [ 0.032723] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]473worker2 # [ 0.063325] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.474lb # [ 0.033379] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)475worker1 # [ 0.032964] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]476worker2 # [ 0.063327] Trampoline variant of Tasks RCU enabled.477client # [ 0.032725] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]478worker2 # [ 0.063327] Rude variant of Tasks RCU enabled.479lb # [ 0.033380] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)480worker1 # [ 0.032966] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]481client # [ 0.032726] [mem 0x60000000-0xafffffff] available for PCI devices482worker2 # [ 0.063327] Tracing variant of Tasks RCU enabled.483lb # [ 0.033381] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)484worker1 # [ 0.032968] [mem 0x60000000-0xafffffff] available for PCI devices485client # [ 0.032728] Booting paravirtualized kernel on KVM486worker2 # [ 0.063328] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.487worker1 # [ 0.032969] Booting paravirtualized kernel on KVM488lb # [ 0.033382] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)489worker2 # [ 0.063329] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1490client # [ 0.032731] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns491lb # [ 0.033383] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)492worker1 # [ 0.032973] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns493lb # [ 0.033385] ACPI: Using ACPI (MADT) for SMP configuration information494client # [ 0.037185] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1495worker2 # [ 0.063344] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.496worker1 # [ 0.037414] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1497lb # [ 0.033386] ACPI: HPET id: 0x8086a201 base: 0xfed00000498client # [ 0.039326] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152499lb # [ 0.033390] TSC deadline timer available500worker2 # [ 0.063346] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.501worker1 # [ 0.039550] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152502client # [ 0.039370] kvm-guest: PV spinlocks disabled, single CPU503lb # [ 0.033394] CPU topo: Max. logical packages: 1504worker1 # [ 0.039595] kvm-guest: PV spinlocks disabled, single CPU505lb # [ 0.033395] CPU topo: Max. logical dies: 1506worker2 # [ 0.063347] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.507lb # [ 0.033396] CPU topo: Max. dies per package: 1508worker2 # [ 0.067727] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16509lb # [ 0.033399] CPU topo: Max. threads per core: 1510worker2 # [ 0.068003] rcu: srcu_init: Setting srcu_struct sizes based on contention.511lb # [ 0.033399] CPU topo: Num. cores per package: 1512lb # [ 0.033400] CPU topo: Num. threads per package: 1513worker2 # [ 0.068009] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns514client # [ 0.039372] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/jwz4idn9byb3rgqdqa3827m7srpcmdj8-nixos-system-client-test/init regInfo=/nix/store/k6c8bg1s150rn281il5cadignqaai16m-closure-info/registration console=ttyS0,115200n8 console=tty0515lb # [ 0.033400] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs516worker1 # [ 0.039597] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/m861wijgs462ggfjaanhm99ifwyrs751-nixos-system-worker1-test/init regInfo=/nix/store/g9w4psxi6bzn798fqi8gw8a3hbjf7pi9-closure-info/registration console=ttyS0,115200n8 console=tty0517lb # [ 0.033418] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()518worker2 # [ 0.068110] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)519worker2 # [ 0.071679] Console: colour VGA+ 80x25520client # [ 0.039471] Unknown kernel command line parameters "regInfo=/nix/store/k6c8bg1s150rn281il5cadignqaai16m-closure-info/registration", will be passed to user space.521lb # [ 0.033449] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]522client # [ 0.039483] random: crng init done523worker2 # [ 0.071682] printk: legacy console [tty0] enabled524worker1 # [ 0.039690] Unknown kernel command line parameters "regInfo=/nix/store/g9w4psxi6bzn798fqi8gw8a3hbjf7pi9-closure-info/registration", will be passed to user space.525worker1 # [ 0.039702] random: crng init done526worker2 # [ 0.101040] printk: legacy console [ttyS0] enabled527lb # [ 0.033450] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]528client # [ 0.039484] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes529worker2 # [ 0.198828] ACPI: Core revision 20250807530worker1 # [ 0.039703] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes531lb # [ 0.033452] [mem 0x60000000-0xafffffff] available for PCI devices532client # [ 0.041588] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)533lb # [ 0.033453] Booting paravirtualized kernel on KVM534worker1 # [ 0.041806] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)535client # [ 0.042639] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)536worker2 # [ 0.199684] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns537lb # [ 0.033455] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns538worker1 # [ 0.042854] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)539client # [ 0.042670] Fallback order for Node 0: 0540worker1 # [ 0.042884] Fallback order for Node 0: 0541worker2 # [ 0.201178] APIC: Switch to symmetric I/O mode setup542client # [ 0.042673] Built 1 zonelists, mobility grouping on. Total pages: 393078543lb # [ 0.037890] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1544client # [ 0.042674] Policy zone: DMA32545worker1 # [ 0.042887] Built 1 zonelists, mobility grouping on. Total pages: 393078546worker2 # [ 0.202114] x2apic enabled547lb # [ 0.040362] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152548worker1 # [ 0.042888] Policy zone: DMA32549client # [ 0.045299] mem auto-init: stack:all(zero), heap alloc:on, heap free:off550lb # [ 0.040405] kvm-guest: PV spinlocks disabled, single CPU551worker2 # [ 0.202871] APIC: Switched APIC routing to: physical x2apic552worker1 # [ 0.045539] mem auto-init: stack:all(zero), heap alloc:on, heap free:off553client # [ 0.048407] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1554worker1 # [ 0.048626] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1555client # [ 0.051755] allocated 3145728 bytes of page_ext556worker1 # [ 0.051924] allocated 3145728 bytes of page_ext557client # [ 0.061659] ftrace: allocating 48731 entries in 192 pages558worker1 # [ 0.061843] ftrace: allocating 48731 entries in 192 pages559worker2 # [ 0.204671] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1560client # [ 0.061660] ftrace: allocated 192 pages with 2 groups561client # [ 0.062501] Dynamic Preempt: lazy562worker1 # [ 0.061844] ftrace: allocated 192 pages with 2 groups563worker1 # [ 0.062692] Dynamic Preempt: lazy564client # [ 0.062662] rcu: Preemptible hierarchical RCU implementation.565lb # [ 0.040406] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/j1nn9bbhrn5xz154akfhw453f8i07i4d-nixos-system-lb-test/init regInfo=/nix/store/hq8v7nws787a9kwd3h009z1n9dqal4fl-closure-info/registration console=ttyS0,115200n8 console=tty0566worker1 # [ 0.062863] rcu: Preemptible hierarchical RCU implementation.567client # [ 0.062663] rcu: RCU event tracing is enabled.568worker2 # [ 0.205625] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns569worker1 # [ 0.062864] rcu: RCU event tracing is enabled.570client # [ 0.062663] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.571worker2 # [ 0.207224] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)572worker1 # [ 0.062865] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.573client # [ 0.062665] Trampoline variant of Tasks RCU enabled.574lb # [ 0.040499] Unknown kernel command line parameters "regInfo=/nix/store/hq8v7nws787a9kwd3h009z1n9dqal4fl-closure-info/registration", will be passed to user space.575client # [ 0.062665] Rude variant of Tasks RCU enabled.576worker1 # [ 0.062866] Trampoline variant of Tasks RCU enabled.577lb # [ 0.040511] random: crng init done578worker2 # [ 0.208307] x86/cpu: User Mode Instruction Prevention (UMIP) activated579worker1 # [ 0.062867] Rude variant of Tasks RCU enabled.580client # [ 0.062665] Tracing variant of Tasks RCU enabled.581lb # [ 0.040512] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes582worker1 # [ 0.062867] Tracing variant of Tasks RCU enabled.583client # [ 0.062666] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.584worker2 # [ 0.209356] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127585lb # [ 0.042671] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)586worker1 # [ 0.062868] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.587client # [ 0.062667] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1588worker2 # [ 0.210222] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0589lb # [ 0.043755] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)590worker1 # [ 0.062869] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1591lb # [ 0.043786] Fallback order for Node 0: 0592client # [ 0.062688] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.593worker1 # [ 0.062890] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.594lb # [ 0.043789] Built 1 zonelists, mobility grouping on. Total pages: 393078595worker2 # [ 0.211225] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto596client # [ 0.062690] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.597lb # [ 0.043790] Policy zone: DMA32598worker1 # [ 0.062892] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.599lb # [ 0.046299] mem auto-init: stack:all(zero), heap alloc:on, heap free:off600client # [ 0.062691] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.601worker2 # [ 0.212222] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl602lb # [ 0.049416] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1603worker1 # [ 0.062893] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.604client # [ 0.067007] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16605worker2 # [ 0.213221] Transient Scheduler Attacks: Vulnerable: No microcode606lb # [ 0.053540] allocated 3145728 bytes of page_ext607worker1 # [ 0.067322] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16608client # [ 0.067296] rcu: srcu_init: Setting srcu_struct sizes based on contention.609lb # [ 0.063498] ftrace: allocating 48731 entries in 192 pages610worker2 # [ 0.214221] Spectre V2 : Mitigation: Enhanced / Automatic IBRS611worker1 # [ 0.067597] rcu: srcu_init: Setting srcu_struct sizes based on contention.612lb # [ 0.063500] ftrace: allocated 192 pages with 2 groups613lb # [ 0.064388] Dynamic Preempt: lazy614client # [ 0.067304] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns615worker2 # [ 0.215221] Speculative Return Stack Overflow: Mitigation: Safe RET616worker1 # [ 0.067605] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns617lb # [ 0.064567] rcu: Preemptible hierarchical RCU implementation.618client # [ 0.067409] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)619lb # [ 0.064567] rcu: RCU event tracing is enabled.620client # [ 0.070960] Console: colour VGA+ 80x25621worker1 # [ 0.067710] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)622lb # [ 0.064568] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.623worker1 # [ 0.071233] Console: colour VGA+ 80x25624client # [ 0.070963] printk: legacy console [tty0] enabled625lb # [ 0.064570] Trampoline variant of Tasks RCU enabled.626worker1 # [ 0.071236] printk: legacy console [tty0] enabled627client # [ 0.100417] printk: legacy console [ttyS0] enabled628lb # [ 0.064570] Rude variant of Tasks RCU enabled.629worker1 # [ 0.100671] printk: legacy console [ttyS0] enabled630client # [ 0.195979] ACPI: Core revision 20250807631lb # [ 0.064570] Tracing variant of Tasks RCU enabled.632worker1 # [ 0.196086] ACPI: Core revision 20250807633worker2 # [ 0.216221] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization634lb # [ 0.064571] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.635worker2 # [ 0.218228] Spectre V2 : Enabling IBPB for BPF636client # [ 0.196862] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns637lb # [ 0.064572] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1638worker1 # [ 0.197003] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns639client # [ 0.198359] APIC: Switch to symmetric I/O mode setup640worker2 # [ 0.218939] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier641lb # [ 0.064588] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.642worker1 # [ 0.198483] APIC: Switch to symmetric I/O mode setup643client # [ 0.199297] x2apic enabled644worker2 # [ 0.220222] active return thunk: srso_alias_return_thunk645worker1 # [ 0.199413] x2apic enabled646lb # [ 0.064590] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.647client # [ 0.200049] APIC: Switched APIC routing to: physical x2apic648worker1 # [ 0.200150] APIC: Switched APIC routing to: physical x2apic649worker2 # [ 0.221071] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'650lb # [ 0.064591] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.651worker2 # [ 0.222221] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'652lb # [ 0.069043] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16653client # [ 0.201844] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1654worker1 # [ 0.201955] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1655lb # [ 0.069332] rcu: srcu_init: Setting srcu_struct sizes based on contention.656worker2 # [ 0.223221] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'657worker2 # [ 0.224221] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'658client # [ 0.202782] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns659lb # [ 0.069339] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns660worker1 # [ 0.202911] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns661worker2 # [ 0.225221] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'662lb # [ 0.069442] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)663client # [ 0.204378] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)664lb # [ 0.072934] Console: colour VGA+ 80x25665worker1 # [ 0.204548] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)666worker2 # [ 0.226221] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'667lb # [ 0.072937] printk: legacy console [tty0] enabled668client # [ 0.205464] x86/cpu: User Mode Instruction Prevention (UMIP) activated669lb # [ 0.102350] printk: legacy console [ttyS0] enabled670worker1 # [ 0.205634] x86/cpu: User Mode Instruction Prevention (UMIP) activated671worker2 # [ 0.227221] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'672client # [ 0.206540] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127673lb # [ 0.197984] ACPI: Core revision 20250807674worker1 # [ 0.206678] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127675worker2 # [ 0.228222] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256676client # [ 0.207376] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0677worker1 # [ 0.207545] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0678worker2 # [ 0.229221] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64679lb # [ 0.198848] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns680worker2 # [ 0.230221] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512681lb # [ 0.200363] APIC: Switch to symmetric I/O mode setup682client # [ 0.208379] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto683lb # [ 0.201293] x2apic enabled684worker1 # [ 0.208549] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto685worker2 # [ 0.231221] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024686client # [ 0.209376] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl687worker2 # [ 0.232221] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8688lb # [ 0.202034] APIC: Switched APIC routing to: physical x2apic689worker1 # [ 0.209545] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl690client # [ 0.210376] Transient Scheduler Attacks: Vulnerable: No microcode691worker1 # [ 0.210545] Transient Scheduler Attacks: Vulnerable: No microcode692client # [ 0.211375] Spectre V2 : Mitigation: Enhanced / Automatic IBRS693worker2 # [ 0.233136] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.694lb # [ 0.203850] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1695worker1 # [ 0.211545] Spectre V2 : Mitigation: Enhanced / Automatic IBRS696client # [ 0.212376] Speculative Return Stack Overflow: Mitigation: Safe RET697worker1 # [ 0.212545] Speculative Return Stack Overflow: Mitigation: Safe RET698lb # [ 0.204872] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns699lb # [ 0.206626] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)700lb # [ 0.207710] x86/cpu: User Mode Instruction Prevention (UMIP) activated701client # [ 0.213375] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization702worker1 # [ 0.213545] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization703client # [ 0.215384] Spectre V2 : Enabling IBPB for BPF704lb # [ 0.209518] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127705worker1 # [ 0.215552] Spectre V2 : Enabling IBPB for BPF706lb # [ 0.210304] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0707client # [ 0.216110] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier708worker1 # [ 0.216546] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier709client # [ 0.217376] active return thunk: srso_alias_return_thunk710worker1 # [ 0.217545] active return thunk: srso_alias_return_thunk711lb # [ 0.211384] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto712client # [ 0.218396] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'713worker1 # [ 0.218559] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'714lb # [ 0.212623] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl715client # [ 0.219375] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'716worker1 # [ 0.219545] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'717lb # [ 0.213623] Transient Scheduler Attacks: Vulnerable: No microcode718client # [ 0.220375] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'719worker1 # [ 0.220545] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'720lb # [ 0.215336] Spectre V2 : Mitigation: Enhanced / Automatic IBRS721client # [ 0.221375] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'722worker1 # [ 0.221545] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'723lb # [ 0.216310] Speculative Return Stack Overflow: Mitigation: Safe RET724client # [ 0.222375] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'725worker1 # [ 0.222545] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'726client # [ 0.223375] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'727worker1 # [ 0.223545] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'728client # [ 0.224375] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'729worker1 # [ 0.224545] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'730client # [ 0.225376] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256731lb # [ 0.217346] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization732worker1 # [ 0.225545] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256733client # [ 0.226375] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64734lb # [ 0.218631] Spectre V2 : Enabling IBPB for BPF735worker1 # [ 0.226545] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64736client # [ 0.227375] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512737worker1 # [ 0.227545] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512738lb # [ 0.219624] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier739client # [ 0.228376] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024740lb # [ 0.220624] active return thunk: srso_alias_return_thunk741worker1 # [ 0.228545] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024742client # [ 0.229375] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8743worker1 # [ 0.229545] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8744lb # [ 0.222258] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'745client # [ 0.230376] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.746lb # [ 0.223511] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'747worker1 # [ 0.230545] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.748lb # [ 0.224366] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'749lb # [ 0.225430] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'750lb # [ 0.226457] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'751lb # [ 0.227623] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'752lb # [ 0.228623] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'753lb # [ 0.229624] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256754lb # [ 0.230623] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64755lb # [ 0.231623] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512756lb # [ 0.232623] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024757lb # [ 0.233623] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8758worker2 # [ 0.267160] Freeing SMP alternatives memory: 44K759worker2 # [ 0.267793] pid_max: default: 32768 minimum: 301760lb # [ 0.234623] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.761worker2 # [ 0.268310] LSM: initializing lsm=capability,landlock,yama,bpf,ima762worker2 # [ 0.269322] landlock: Up and running.763worker2 # [ 0.270222] Yama: becoming mindful.764worker2 # [ 0.271404] LSM support for eBPF active765worker2 # [ 0.272171] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)766worker2 # [ 0.273262] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)767worker2 # [ 0.275989] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)768worker2 # [ 0.276739] Performance Events: Fam17h+ core perfctr, AMD PMU driver.769worker2 # [ 0.277226] ... version: 2770worker2 # [ 0.277954] ... bit width: 48771worker2 # [ 0.278224] ... generic counters: 6772worker2 # [ 0.278951] ... generic bitmap: 000000000000003f773worker2 # [ 0.279269] ... fixed-purpose counters: 0774worker2 # [ 0.279985] ... fixed-purpose bitmap: 0000000000000000775worker2 # [ 0.280224] ... value mask: 0000ffffffffffff776worker2 # [ 0.281142] ... max period: 00007fffffffffff777worker2 # [ 0.281939] ... global_ctrl mask: 000000000000003f778worker2 # [ 0.282336] signal: max sigframe size: 3376779worker2 # [ 0.283183] rcu: Hierarchical SRCU implementation.780client # [ 0.264462] Freeing SMP alternatives memory: 44K781worker2 # [ 0.283841] rcu: Max phase no-delay instances is 400.782client # [ 0.265265] pid_max: default: 32768 minimum: 301783worker1 # [ 0.265401] Freeing SMP alternatives memory: 44K784client # [ 0.266045] LSM: initializing lsm=capability,landlock,yama,bpf,ima785worker1 # [ 0.266094] pid_max: default: 32768 minimum: 301786client # [ 0.266492] landlock: Up and running.787worker1 # [ 0.266639] LSM: initializing lsm=capability,landlock,yama,bpf,ima788client # [ 0.267846] Yama: becoming mindful.789worker1 # [ 0.267651] landlock: Up and running.790client # [ 0.268595] LSM support for eBPF active791worker1 # [ 0.268546] Yama: becoming mindful.792worker1 # [ 0.269595] LSM support for eBPF active793worker2 # [ 0.288744] smp: Bringing up secondary CPUs ...794client # [ 0.269524] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)795worker2 # [ 0.289237] smp: Brought up 1 node, 1 CPU796worker1 # [ 0.270360] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)797client # [ 0.270413] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)798worker2 # [ 0.289914] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)799worker1 # [ 0.271563] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)800client # [ 0.273020] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)801worker2 # [ 0.290388] Memory: 1454548K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)802worker2 # [ 0.291465] devtmpfs: initialized803worker1 # [ 0.273988] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)804client # [ 0.274130] Performance Events: Fam17h+ core perfctr, AMD PMU driver.805worker2 # [ 0.292262] x86/mm: Memory block size: 128MB806client # [ 0.275159] ... version: 2807worker1 # [ 0.275122] Performance Events: Fam17h+ core perfctr, AMD PMU driver.808client # [ 0.275378] ... bit width: 48809worker1 # [ 0.275555] ... version: 2810client # [ 0.276104] ... generic counters: 6811lb # [ 0.269876] Freeing SMP alternatives memory: 44K812worker2 # [ 0.294082] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)813worker1 # [ 0.276258] ... bit width: 48814client # [ 0.276431] ... generic bitmap: 000000000000003f815lb # [ 0.270626] pid_max: default: 32768 minimum: 301816worker1 # [ 0.276547] ... generic counters: 6817client # [ 0.277291] ... fixed-purpose counters: 0818worker2 # [ 0.295111] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).819worker1 # [ 0.277214] ... generic bitmap: 000000000000003f820lb # [ 0.271449] LSM: initializing lsm=capability,landlock,yama,bpf,ima821client # [ 0.277869] ... fixed-purpose bitmap: 0000000000000000822worker2 # [ 0.296301] pinctrl core: initialized pinctrl subsystem823worker1 # [ 0.277548] ... fixed-purpose counters: 0824lb # [ 0.272451] landlock: Up and running.825client # [ 0.278378] ... value mask: 0000ffffffffffff826lb # [ 0.273038] Yama: becoming mindful.827worker1 # [ 0.278252] ... fixed-purpose bitmap: 0000000000000000828worker2 # [ 0.297389] PM: RTC time: 10:34:28, date: 2026-09-22829client # [ 0.279236] ... max period: 00007fffffffffff830lb # [ 0.273836] LSM support for eBPF active831worker1 # [ 0.278547] ... value mask: 0000ffffffffffff832client # [ 0.280049] ... global_ctrl mask: 000000000000003f833worker1 # [ 0.279391] ... max period: 00007fffffffffff834lb # [ 0.274745] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)835client # [ 0.280509] signal: max sigframe size: 3376836worker1 # [ 0.280181] ... global_ctrl mask: 000000000000003f837client # [ 0.281270] rcu: Hierarchical SRCU implementation.838lb # [ 0.275660] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)839worker1 # [ 0.280660] signal: max sigframe size: 3376840worker2 # [ 0.300975] NET: Registered PF_NETLINK/PF_ROUTE protocol family841client # [ 0.281944] rcu: Max phase no-delay instances is 400.842worker1 # [ 0.281421] rcu: Hierarchical SRCU implementation.843worker1 # [ 0.282123] rcu: Max phase no-delay instances is 400.844worker2 # [ 0.301701] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations845lb # [ 0.278316] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)846worker2 # [ 0.302245] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations847lb # [ 0.279174] Performance Events: Fam17h+ core perfctr, AMD PMU driver.848lb # [ 0.279628] ... version: 2849worker2 # [ 0.303500] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations850lb # [ 0.280356] ... bit width: 48851worker2 # [ 0.304233] audit: initializing netlink subsys (disabled)852client # [ 0.286971] smp: Bringing up secondary CPUs ...853lb # [ 0.280625] ... generic counters: 6854client # [ 0.287391] smp: Brought up 1 node, 1 CPU855worker1 # [ 0.287046] smp: Bringing up secondary CPUs ...856lb # [ 0.281366] ... generic bitmap: 000000000000003f857worker2 # [ 0.305225] audit: type=2000 audit(1790073268.852:1): state=initialized audit_enabled=0 res=1858worker1 # [ 0.287562] smp: Brought up 1 node, 1 CPU859client # [ 0.288046] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)860lb # [ 0.281671] ... fixed-purpose counters: 0861worker2 # [ 0.306330] thermal_sys: Registered thermal governor 'fair_share'862worker1 # [ 0.288228] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)863lb # [ 0.282392] ... fixed-purpose bitmap: 0000000000000000864worker2 # [ 0.306332] thermal_sys: Registered thermal governor 'bang_bang'865lb # [ 0.282626] ... value mask: 0000ffffffffffff866client # [ 0.288587] Memory: 1454556K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)867worker2 # [ 0.307225] thermal_sys: Registered thermal governor 'step_wise'868lb # [ 0.283510] ... max period: 00007fffffffffff869client # [ 0.289630] devtmpfs: initialized870worker1 # [ 0.288731] Memory: 1454564K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)871worker2 # [ 0.308155] thermal_sys: Registered thermal governor 'user_space'872lb # [ 0.284307] ... global_ctrl mask: 000000000000003f873client # [ 0.290390] x86/mm: Memory block size: 128MB874worker1 # [ 0.289797] devtmpfs: initialized875worker2 # [ 0.308930] thermal_sys: Registered thermal governor 'power_allocator'876lb # [ 0.284735] signal: max sigframe size: 3376877worker1 # [ 0.290586] x86/mm: Memory block size: 128MB878worker2 # [ 0.309242] cpuidle: using governor menu879lb # [ 0.285531] rcu: Hierarchical SRCU implementation.880client # [ 0.292159] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)881lb # [ 0.286276] rcu: Max phase no-delay instances is 400.882worker2 # [ 0.311441] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5883worker1 # [ 0.292386] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)884client # [ 0.293296] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).885worker2 # [ 0.312479] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]886client # [ 0.294467] pinctrl core: initialized pinctrl subsystem887worker1 # [ 0.293460] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).888worker2 # [ 0.313227] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry889worker1 # [ 0.294630] pinctrl core: initialized pinctrl subsystem890client # [ 0.295495] PM: RTC time: 10:34:28, date: 2026-09-22891worker2 # [ 0.314235] PCI: Using configuration type 1 for base access892worker1 # [ 0.295748] PM: RTC time: 10:34:28, date: 2026-09-22893lb # [ 0.291108] smp: Bringing up secondary CPUs ...894worker2 # [ 0.315380] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.895lb # [ 0.291639] smp: Brought up 1 node, 1 CPU896client # [ 0.298995] NET: Registered PF_NETLINK/PF_ROUTE protocol family897lb # [ 0.292327] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)898worker1 # [ 0.299259] NET: Registered PF_NETLINK/PF_ROUTE protocol family899client # [ 0.299863] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations900lb # [ 0.292829] Memory: 1454612K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)901worker1 # [ 0.300051] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations902client # [ 0.300400] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations903lb # [ 0.293861] devtmpfs: initialized904worker1 # [ 0.300576] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations905client # [ 0.301656] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations906lb # [ 0.294716] x86/mm: Memory block size: 128MB907client # [ 0.302388] audit: initializing netlink subsys (disabled)908worker1 # [ 0.301831] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations909worker2 # [ 0.322253] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages910worker1 # [ 0.302558] audit: initializing netlink subsys (disabled)911lb # [ 0.296553] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)912client # [ 0.303380] audit: type=2000 audit(1790073268.876:1): state=initialized audit_enabled=0 res=1913worker2 # [ 0.323225] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page914worker1 # [ 0.303608] thermal_sys: Registered thermal governor 'fair_share'915worker2 # [ 0.324201] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages916client # [ 0.304544] thermal_sys: Registered thermal governor 'fair_share'917lb # [ 0.297517] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).918worker1 # [ 0.303610] thermal_sys: Registered thermal governor 'bang_bang'919client # [ 0.304546] thermal_sys: Registered thermal governor 'bang_bang'920lb # [ 0.298700] pinctrl core: initialized pinctrl subsystem921worker1 # [ 0.304523] thermal_sys: Registered thermal governor 'step_wise'922client # [ 0.305379] thermal_sys: Registered thermal governor 'step_wise'923lb # [ 0.299824] PM: RTC time: 10:34:28, date: 2026-09-22924worker1 # [ 0.305233] thermal_sys: Registered thermal governor 'user_space'925client # [ 0.306298] thermal_sys: Registered thermal governor 'user_space'926worker1 # [ 0.305552] audit: type=2000 audit(1790073268.876:1): state=initialized audit_enabled=0 res=1927client # [ 0.307119] thermal_sys: Registered thermal governor 'power_allocator'928client # [ 0.307390] cpuidle: using governor menu929worker1 # [ 0.307550] thermal_sys: Registered thermal governor 'power_allocator'930worker2 # [ 0.329226] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page931worker1 # [ 0.307564] cpuidle: using governor menu932client # [ 0.309589] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5933lb # [ 0.303493] NET: Registered PF_NETLINK/PF_ROUTE protocol family934worker1 # [ 0.309592] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5935client # [ 0.310622] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]936lb # [ 0.304859] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations937worker1 # [ 0.310780] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]938client # [ 0.311381] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry939lb # [ 0.305648] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations940worker1 # [ 0.311550] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry941client # [ 0.312390] PCI: Using configuration type 1 for base access942worker1 # [ 0.312552] PCI: Using configuration type 1 for base access943lb # [ 0.306916] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations944client # [ 0.313472] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.945lb # [ 0.307635] audit: initializing netlink subsys (disabled)946worker1 # [ 0.313592] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.947lb # [ 0.308628] audit: type=2000 audit(1790073268.890:1): state=initialized audit_enabled=0 res=1948worker2 # [ 0.337604] ACPI: Added _OSI(Module Device)949worker2 # [ 0.338226] ACPI: Added _OSI(Processor Device)950lb # [ 0.309747] thermal_sys: Registered thermal governor 'fair_share'951worker2 # [ 0.338954] ACPI: Added _OSI(Processor Aggregator Device)952lb # [ 0.309749] thermal_sys: Registered thermal governor 'bang_bang'953lb # [ 0.310627] thermal_sys: Registered thermal governor 'step_wise'954client # [ 0.318611] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages955lb # [ 0.311582] thermal_sys: Registered thermal governor 'user_space'956worker1 # [ 0.318787] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages957client # [ 0.319380] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page958worker1 # [ 0.319549] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page959lb # [ 0.312366] thermal_sys: Registered thermal governor 'power_allocator'960lb # [ 0.312639] cpuidle: using governor menu961lb # [ 0.314901] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5962lb # [ 0.315864] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]963client # [ 0.324380] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages964worker1 # [ 0.324550] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages965lb # [ 0.316629] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry966worker2 # [ 0.346788] ACPI: 1 ACPI AML tables successfully acquired and loaded967client # [ 0.325378] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page968lb # [ 0.317646] PCI: Using configuration type 1 for base access969worker1 # [ 0.325547] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page970lb # [ 0.318710] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.971worker2 # [ 0.350639] ACPI: Interpreter enabled972worker2 # [ 0.351240] ACPI: PM: (supports S0 S3 S4 S5)973worker2 # [ 0.351957] ACPI: Using IOAPIC for interrupt routing974worker2 # [ 0.354285] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug975lb # [ 0.325643] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages976worker2 # [ 0.355225] PCI: Using E820 reservations for host bridge windows977client # [ 0.335719] ACPI: Added _OSI(Module Device)978lb # [ 0.326626] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page979worker1 # [ 0.335933] ACPI: Added _OSI(Module Device)980client # [ 0.336380] ACPI: Added _OSI(Processor Device)981worker1 # [ 0.336549] ACPI: Added _OSI(Processor Device)982client # [ 0.337111] ACPI: Added _OSI(Processor Aggregator Device)983worker1 # [ 0.337283] ACPI: Added _OSI(Processor Aggregator Device)984worker2 # [ 0.358313] ACPI: Enabled 2 GPEs in block 00 to 3F985lb # [ 0.331627] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages986lb # [ 0.332627] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page987client # [ 0.345070] ACPI: 1 ACPI AML tables successfully acquired and loaded988worker2 # [ 0.366188] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])989worker1 # [ 0.345660] ACPI: 1 ACPI AML tables successfully acquired and loaded990worker2 # [ 0.367009] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]991worker2 # [ 0.368302] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]992client # [ 0.348797] ACPI: Interpreter enabled993worker2 # [ 0.369347] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]994client # [ 0.349393] ACPI: PM: (supports S0 S3 S4 S5)995worker1 # [ 0.349953] ACPI: Interpreter enabled996worker2 # [ 0.370585] PCI host bridge to bus 0000:00997client # [ 0.350112] ACPI: Using IOAPIC for interrupt routing998worker1 # [ 0.350563] ACPI: PM: (supports S0 S3 S4 S5)999worker1 # [ 0.351320] ACPI: Using IOAPIC for interrupt routing1000worker2 # [ 0.371229] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1001lb # [ 0.343960] ACPI: Added _OSI(Module Device)1002client # [ 0.350460] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1003worker2 # [ 0.372224] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1004lb # [ 0.344627] ACPI: Added _OSI(Processor Device)1005worker1 # [ 0.351581] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1006client # [ 0.351379] PCI: Using E820 reservations for host bridge windows1007lb # [ 0.345362] ACPI: Added _OSI(Processor Aggregator Device)1008worker2 # [ 0.373224] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1009worker1 # [ 0.352548] PCI: Using E820 reservations for host bridge windows1010worker2 # [ 0.374224] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1011client # [ 0.354425] ACPI: Enabled 2 GPEs in block 00 to 3F1012worker2 # [ 0.375225] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1013worker1 # [ 0.355614] ACPI: Enabled 2 GPEs in block 00 to 3F1014worker2 # [ 0.376224] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1015worker2 # [ 0.377225] pci_bus 0000:00: root bus resource [bus 00-ff]1016worker2 # [ 0.378198] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1017lb # [ 0.351347] ACPI: 1 ACPI AML tables successfully acquired and loaded1018worker2 # [ 0.379674] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1019lb # [ 0.355025] ACPI: Interpreter enabled1020client # [ 0.363060] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1021lb # [ 0.355641] ACPI: PM: (supports S0 S3 S4 S5)1022worker1 # [ 0.364243] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1023lb # [ 0.356387] ACPI: Using IOAPIC for interrupt routing1024client # [ 0.363384] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1025worker1 # [ 0.364554] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1026client # [ 0.365780] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1027worker1 # [ 0.366328] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1028lb # [ 0.358691] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1029client # [ 0.366501] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1030worker2 # [ 0.382294] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1031worker1 # [ 0.367551] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1032lb # [ 0.359627] PCI: Using E820 reservations for host bridge windows1033client # [ 0.367741] PCI host bridge to bus 0000:001034worker2 # [ 0.383236] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1035worker1 # [ 0.368948] PCI host bridge to bus 0000:001036client # [ 0.368382] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1037worker2 # [ 0.384214] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1038worker1 # [ 0.369553] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1039client # [ 0.369379] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1040worker1 # [ 0.370547] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1041lb # [ 0.362739] ACPI: Enabled 2 GPEs in block 00 to 3F1042worker2 # [ 0.385839] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1043client # [ 0.370378] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1044worker1 # [ 0.371548] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1045client # [ 0.371378] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1046worker1 # [ 0.372547] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1047worker2 # [ 0.386971] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1048client # [ 0.372379] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1049worker1 # [ 0.373548] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1050client # [ 0.373378] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1051worker1 # [ 0.374547] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1052client # [ 0.374378] pci_bus 0000:00: root bus resource [bus 00-ff]1053worker1 # [ 0.375548] pci_bus 0000:00: root bus resource [bus 00-ff]1054worker2 # [ 0.390258] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1055client # [ 0.375336] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1056worker1 # [ 0.376538] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1057worker2 # [ 0.391106] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1058lb # [ 0.370214] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1059worker2 # [ 0.391956] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1060worker1 # [ 0.378024] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1061client # [ 0.376814] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1062worker2 # [ 0.392230] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1063lb # [ 0.370632] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1064lb # [ 0.371718] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1065lb # [ 0.372758] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1066worker2 # [ 0.394477] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1067lb # [ 0.373993] PCI host bridge to bus 0000:001068lb # [ 0.374631] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1069worker2 # [ 0.397155] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1070lb # [ 0.375626] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1071worker2 # [ 0.397951] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1072lb # [ 0.376626] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1073client # [ 0.379443] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1074worker1 # [ 0.380644] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1075worker2 # [ 0.398247] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1076client # [ 0.380390] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1077lb # [ 0.377626] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1078worker1 # [ 0.381560] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1079client # [ 0.381325] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1080worker1 # [ 0.382472] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1081worker2 # [ 0.399794] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1082lb # [ 0.378626] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1083lb # [ 0.379626] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1084client # [ 0.382329] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1085worker1 # [ 0.383445] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1086lb # [ 0.380626] pci_bus 0000:00: root bus resource [bus 00-ff]1087lb # [ 0.381628] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1088client # [ 0.384103] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1089worker1 # [ 0.385257] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1090worker2 # [ 0.402231] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1091lb # [ 0.383072] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1092worker2 # [ 0.403927] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1093client # [ 0.387387] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1094worker2 # [ 0.404268] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1095worker1 # [ 0.388578] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1096client # [ 0.388214] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1097worker1 # [ 0.389403] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1098worker2 # [ 0.405763] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1099client # [ 0.389103] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1100worker1 # [ 0.390255] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1101client # [ 0.389385] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1102worker1 # [ 0.390554] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1103worker2 # [ 0.408161] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1104client # [ 0.391083] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1105worker1 # [ 0.392080] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1106worker2 # [ 0.408891] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1107worker2 # [ 0.409267] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1108lb # [ 0.385762] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1109client # [ 0.393386] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1110worker1 # [ 0.394464] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1111lb # [ 0.386707] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1112client # [ 0.394228] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1113worker1 # [ 0.395190] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1114worker2 # [ 0.411546] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1115lb # [ 0.387649] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1116client # [ 0.395101] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1117worker1 # [ 0.395569] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1118lb # [ 0.388984] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1119client # [ 0.395916] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1120worker1 # [ 0.397126] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1121worker2 # [ 0.414143] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1122worker2 # [ 0.414858] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1123lb # [ 0.390330] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1124worker2 # [ 0.415246] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1125client # [ 0.398394] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1126worker2 # [ 0.416822] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1127worker1 # [ 0.399565] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1128lb # [ 0.393661] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1129client # [ 0.400038] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1130worker1 # [ 0.400555] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1131lb # [ 0.394476] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1132client # [ 0.400403] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1133worker1 # [ 0.401482] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1134lb # [ 0.395334] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1135client # [ 0.401938] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1136worker1 # [ 0.402919] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1137worker2 # [ 0.419580] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1138lb # [ 0.395653] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1139worker2 # [ 0.420876] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1140lb # [ 0.397228] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1141client # [ 0.403386] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1142worker2 # [ 0.421275] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1143worker1 # [ 0.405391] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1144client # [ 0.404208] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1145worker1 # [ 0.406164] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1146client # [ 0.405127] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1147worker2 # [ 0.422813] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1148worker1 # [ 0.406571] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1149lb # [ 0.399662] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1150lb # [ 0.400539] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1151client # [ 0.405937] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1152worker2 # [ 0.425009] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1153lb # [ 0.401408] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1154worker1 # [ 0.408099] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1155worker2 # [ 0.425848] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1156worker2 # [ 0.426246] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1157lb # [ 0.403023] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1158client # [ 0.407388] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1159worker1 # [ 0.410406] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1160client # [ 0.408223] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1161worker2 # [ 0.427230] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1162worker1 # [ 0.411181] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1163client # [ 0.409135] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1164worker1 # [ 0.411570] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1165lb # [ 0.406572] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1166worker2 # [ 0.429057] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1167client # [ 0.410015] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1168lb # [ 0.407278] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1169worker1 # [ 0.413131] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1170lb # [ 0.407648] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1171worker2 # [ 0.431258] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1172worker2 # [ 0.432240] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1173client # [ 0.412387] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1174lb # [ 0.409208] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1175client # [ 0.413961] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1176worker1 # [ 0.415565] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1177worker2 # [ 0.433781] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1178client # [ 0.414400] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1179worker1 # [ 0.416554] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1180lb # [ 0.410660] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1181worker1 # [ 0.417501] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1182lb # [ 0.411547] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1183client # [ 0.415936] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1184worker2 # [ 0.436088] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1185worker1 # [ 0.418958] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1186worker2 # [ 0.436895] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1187lb # [ 0.412394] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1188client # [ 0.417388] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1189worker2 # [ 0.437247] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1190client # [ 0.418225] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1191worker1 # [ 0.421328] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1192lb # [ 0.413175] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1193client # [ 0.419124] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1194worker2 # [ 0.438807] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1195worker1 # [ 0.422190] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1196client # [ 0.419385] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1197worker1 # [ 0.422570] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1198worker1 # [ 0.423554] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1199lb # [ 0.415545] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1200worker2 # [ 0.440233] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1201lb # [ 0.416347] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1202worker2 # [ 0.441150] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1203client # [ 0.420916] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1204lb # [ 0.416648] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1205worker2 # [ 0.441922] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1206worker1 # [ 0.425923] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1207client # [ 0.422391] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1208lb # [ 0.418206] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1209client # [ 0.423368] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1210worker2 # [ 0.442815] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1211worker1 # [ 0.427584] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1212client # [ 0.424761] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1213worker2 # [ 0.443769] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1214worker1 # [ 0.428563] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1215worker2 # [ 0.444440] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1216worker1 # [ 0.430089] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1217lb # [ 0.420670] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1218client # [ 0.426396] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1219worker2 # [ 0.445752] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1220lb # [ 0.421520] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1221client # [ 0.427230] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1222lb # [ 0.422385] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1223worker1 # [ 0.432442] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1224client # [ 0.428099] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1225lb # [ 0.423251] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1226worker1 # [ 0.433167] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1227client # [ 0.428938] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1228worker1 # [ 0.433570] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1229lb # [ 0.424660] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1230client # [ 0.430386] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1231lb # [ 0.425535] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1232worker1 # [ 0.435113] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1233client # [ 0.431221] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1234client # [ 0.432097] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1235lb # [ 0.426344] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1236worker1 # [ 0.437247] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1237lb # [ 0.426632] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1238worker1 # [ 0.437555] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1239client # [ 0.432947] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1240worker1 # [ 0.438502] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1241lb # [ 0.428172] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1242client # [ 0.434010] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1243worker2 # [ 0.446427] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1244worker1 # [ 0.439974] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1245lb # [ 0.429664] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1246client # [ 0.434578] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1247worker2 # [ 0.448792] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1248lb # [ 0.430648] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1249client # [ 0.437071] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1250worker1 # [ 0.441925] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1251worker2 # [ 0.449472] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1252lb # [ 0.432156] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1253worker1 # [ 0.442748] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1254worker2 # [ 0.450763] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1255worker1 # [ 0.444272] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1256worker2 # [ 0.451529] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1257worker2 # [ 0.452527] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1258lb # [ 0.435108] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1259lb # [ 0.436309] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1260worker2 # [ 0.453513] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1261lb # [ 0.436648] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1262worker2 # [ 0.455272] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1263lb # [ 0.438230] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1264worker2 # [ 0.456171] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1265worker2 # [ 0.457606] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1266lb # [ 0.440357] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1267client # [ 0.437581] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1268worker2 # [ 0.458737] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1269lb # [ 0.440633] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1270client # [ 0.440030] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1271lb # [ 0.441612] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1272client # [ 0.440692] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1273worker1 # [ 0.444755] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1274lb # [ 0.443894] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1275client # [ 0.441971] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1276worker2 # [ 0.462356] ACPI: PCI: Interrupt link LNKA configured for IRQ 101277worker1 # [ 0.447315] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1278worker2 # [ 0.463340] ACPI: PCI: Interrupt link LNKB configured for IRQ 101279lb # [ 0.445464] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1280client # [ 0.442597] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1281worker1 # [ 0.447780] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1282worker2 # [ 0.464326] ACPI: PCI: Interrupt link LNKC configured for IRQ 111283lb # [ 0.446421] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1284client # [ 0.443663] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1285worker2 # [ 0.465323] ACPI: PCI: Interrupt link LNKD configured for IRQ 111286worker1 # [ 0.449554] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1287worker2 # [ 0.466333] ACPI: PCI: Interrupt link LNKE configured for IRQ 101288lb # [ 0.448353] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1289client # [ 0.444631] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1290worker2 # [ 0.467346] ACPI: PCI: Interrupt link LNKF configured for IRQ 101291worker1 # [ 0.450787] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1292worker2 # [ 0.468327] ACPI: PCI: Interrupt link LNKG configured for IRQ 111293client # [ 0.446400] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1294worker1 # [ 0.451840] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1295worker2 # [ 0.469329] ACPI: PCI: Interrupt link LNKH configured for IRQ 111296client # [ 0.447230] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1297worker1 # [ 0.452808] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1298worker2 # [ 0.470269] ACPI: PCI: Interrupt link GSIA configured for IRQ 161299client # [ 0.448405] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1300worker2 # [ 0.471236] ACPI: PCI: Interrupt link GSIB configured for IRQ 171301worker1 # [ 0.454576] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1302worker2 # [ 0.472236] ACPI: PCI: Interrupt link GSIC configured for IRQ 181303client # [ 0.450016] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1304worker1 # [ 0.455413] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1305worker2 # [ 0.473210] ACPI: PCI: Interrupt link GSID configured for IRQ 191306worker2 # [ 0.474007] ACPI: PCI: Interrupt link GSIE configured for IRQ 201307worker1 # [ 0.456574] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1308worker2 # [ 0.475002] ACPI: PCI: Interrupt link GSIF configured for IRQ 211309worker2 # [ 0.476135] ACPI: PCI: Interrupt link GSIG configured for IRQ 221310client # [ 0.453750] ACPI: PCI: Interrupt link LNKA configured for IRQ 101311worker1 # [ 0.458352] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1312worker2 # [ 0.477022] ACPI: PCI: Interrupt link GSIH configured for IRQ 231313client # [ 0.454485] ACPI: PCI: Interrupt link LNKB configured for IRQ 101314client # [ 0.455480] ACPI: PCI: Interrupt link LNKC configured for IRQ 111315lb # [ 0.448822] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1316worker2 # [ 0.479234] iommu: Default domain type: Translated1317client # [ 0.456472] ACPI: PCI: Interrupt link LNKD configured for IRQ 111318worker2 # [ 0.480054] iommu: DMA domain TLB invalidation policy: lazy mode1319worker1 # [ 0.462184] ACPI: PCI: Interrupt link LNKA configured for IRQ 101320client # [ 0.457476] ACPI: PCI: Interrupt link LNKE configured for IRQ 101321lb # [ 0.451411] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1322worker2 # [ 0.481304] ACPI: bus type USB registered1323client # [ 0.458471] ACPI: PCI: Interrupt link LNKF configured for IRQ 101324worker1 # [ 0.462655] ACPI: PCI: Interrupt link LNKB configured for IRQ 101325worker2 # [ 0.482071] usbcore: registered new interface driver usbfs1326lb # [ 0.451877] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1327client # [ 0.459477] ACPI: PCI: Interrupt link LNKG configured for IRQ 111328worker1 # [ 0.463648] ACPI: PCI: Interrupt link LNKC configured for IRQ 111329worker2 # [ 0.482911] usbcore: registered new interface driver hub1330worker1 # [ 0.464648] ACPI: PCI: Interrupt link LNKD configured for IRQ 111331worker2 # [ 0.483237] usbcore: registered new device driver usb1332lb # [ 0.453319] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1333client # [ 0.460474] ACPI: PCI: Interrupt link LNKH configured for IRQ 111334worker1 # [ 0.465643] ACPI: PCI: Interrupt link LNKE configured for IRQ 101335client # [ 0.461416] ACPI: PCI: Interrupt link GSIA configured for IRQ 161336lb # [ 0.453887] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1337worker2 # [ 0.485057] NetLabel: Initializing1338client # [ 0.462393] ACPI: PCI: Interrupt link GSIB configured for IRQ 171339worker1 # [ 0.466647] ACPI: PCI: Interrupt link LNKF configured for IRQ 101340worker2 # [ 0.485621] NetLabel: domain hash size = 1281341client # [ 0.463339] ACPI: PCI: Interrupt link GSIC configured for IRQ 181342worker1 # [ 0.467649] ACPI: PCI: Interrupt link LNKG configured for IRQ 111343lb # [ 0.454909] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1344worker2 # [ 0.486224] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1345client # [ 0.464101] ACPI: PCI: Interrupt link GSID configured for IRQ 191346worker1 # [ 0.468647] ACPI: PCI: Interrupt link LNKH configured for IRQ 111347worker2 # [ 0.487212] NetLabel: unlabeled traffic allowed by default1348lb # [ 0.455904] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1349client # [ 0.464391] ACPI: PCI: Interrupt link GSIE configured for IRQ 201350worker2 # [ 0.487929] PCI: Using ACPI for IRQ routing1351worker1 # [ 0.469583] ACPI: PCI: Interrupt link GSIA configured for IRQ 161352client # [ 0.465301] ACPI: PCI: Interrupt link GSIF configured for IRQ 211353worker1 # [ 0.470510] ACPI: PCI: Interrupt link GSIB configured for IRQ 171354lb # [ 0.457655] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1355client # [ 0.466121] ACPI: PCI: Interrupt link GSIG configured for IRQ 221356worker1 # [ 0.471287] ACPI: PCI: Interrupt link GSIC configured for IRQ 181357client # [ 0.466396] ACPI: PCI: Interrupt link GSIH configured for IRQ 231358worker1 # [ 0.471563] ACPI: PCI: Interrupt link GSID configured for IRQ 191359lb # [ 0.458551] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1360worker1 # [ 0.472503] ACPI: PCI: Interrupt link GSIE configured for IRQ 201361client # [ 0.468429] iommu: Default domain type: Translated1362worker1 # [ 0.473277] ACPI: PCI: Interrupt link GSIF configured for IRQ 211363lb # [ 0.459683] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1364client # [ 0.469193] iommu: DMA domain TLB invalidation policy: lazy mode1365worker1 # [ 0.473559] ACPI: PCI: Interrupt link GSIG configured for IRQ 221366client # [ 0.470412] ACPI: bus type USB registered1367lb # [ 0.461453] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1368worker1 # [ 0.474513] ACPI: PCI: Interrupt link GSIH configured for IRQ 231369client # [ 0.471155] usbcore: registered new interface driver usbfs1370client # [ 0.471393] usbcore: registered new interface driver hub1371worker1 # [ 0.476575] iommu: Default domain type: Translated1372client # [ 0.472293] usbcore: registered new device driver usb1373worker1 # [ 0.477361] iommu: DMA domain TLB invalidation policy: lazy mode1374lb # [ 0.465290] ACPI: PCI: Interrupt link LNKA configured for IRQ 101375worker1 # [ 0.478539] ACPI: bus type USB registered1376client # [ 0.473946] NetLabel: Initializing1377client # [ 0.474379] NetLabel: domain hash size = 1281378worker1 # [ 0.479277] usbcore: registered new interface driver usbfs1379lb # [ 0.465732] ACPI: PCI: Interrupt link LNKB configured for IRQ 101380client # [ 0.475071] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1381worker1 # [ 0.479564] usbcore: registered new interface driver hub1382lb # [ 0.466725] ACPI: PCI: Interrupt link LNKC configured for IRQ 111383client # [ 0.475412] NetLabel: unlabeled traffic allowed by default1384worker1 # [ 0.480419] usbcore: registered new device driver usb1385client # [ 0.476320] PCI: Using ACPI for IRQ routing1386lb # [ 0.467726] ACPI: PCI: Interrupt link LNKD configured for IRQ 111387worker1 # [ 0.482212] NetLabel: Initializing1388lb # [ 0.468738] ACPI: PCI: Interrupt link LNKE configured for IRQ 101389worker1 # [ 0.482548] NetLabel: domain hash size = 1281390lb # [ 0.469722] ACPI: PCI: Interrupt link LNKF configured for IRQ 101391worker1 # [ 0.483270] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1392lb # [ 0.470726] ACPI: PCI: Interrupt link LNKG configured for IRQ 111393worker1 # [ 0.483607] NetLabel: unlabeled traffic allowed by default1394lb # [ 0.471725] ACPI: PCI: Interrupt link LNKH configured for IRQ 111395worker1 # [ 0.484560] PCI: Using ACPI for IRQ routing1396lb # [ 0.472663] ACPI: PCI: Interrupt link GSIA configured for IRQ 161397lb # [ 0.473641] ACPI: PCI: Interrupt link GSIB configured for IRQ 171398lb # [ 0.474638] ACPI: PCI: Interrupt link GSIC configured for IRQ 181399lb # [ 0.475640] ACPI: PCI: Interrupt link GSID configured for IRQ 191400lb # [ 0.476637] ACPI: PCI: Interrupt link GSIE configured for IRQ 201401lb # [ 0.477641] ACPI: PCI: Interrupt link GSIF configured for IRQ 211402lb # [ 0.478637] ACPI: PCI: Interrupt link GSIG configured for IRQ 221403lb # [ 0.479637] ACPI: PCI: Interrupt link GSIH configured for IRQ 231404lb # [ 0.481639] iommu: Default domain type: Translated1405lb # [ 0.482461] iommu: DMA domain TLB invalidation policy: lazy mode1406lb # [ 0.483703] ACPI: bus type USB registered1407lb # [ 0.484483] usbcore: registered new interface driver usbfs1408lb # [ 0.485366] usbcore: registered new interface driver hub1409lb # [ 0.485646] usbcore: registered new device driver usb1410lb # [ 0.487551] NetLabel: Initializing1411lb # [ 0.488633] NetLabel: domain hash size = 1281412lb # [ 0.489493] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413lb # [ 0.490416] NetLabel: unlabeled traffic allowed by default1414lb # [ 0.490643] PCI: Using ACPI for IRQ routing1415worker2 # [ 0.531549] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416worker2 # [ 0.532220] pci 0000:00:01.0: vgaarb: bridge control possible1417worker2 # [ 0.532220] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418worker2 # [ 0.532231] vgaarb: loaded1419worker2 # [ 0.533014] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01420worker2 # [ 0.533869] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1421worker2 # [ 0.537349] clocksource: Switched to clocksource kvm-clock1422worker2 # [ 0.539129] VFS: Disk quotas dquot_6.6.01423client # [ 0.519724] pci 0000:00:01.0: vgaarb: setting as boot VGA device1424client # [ 0.520374] pci 0000:00:01.0: vgaarb: bridge control possible1425worker2 # [ 0.542078] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1426worker2 # [ 0.543595] pnp: PnP ACPI init1427client # [ 0.520374] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1428client # [ 0.520382] vgaarb: loaded1429worker2 # [ 0.544455] ACPI: IRQ 4 override to edge(!), high(!)1430client # [ 0.521115] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01431worker2 # [ 0.545523] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1432client # [ 0.521378] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1433worker2 # [ 0.547127] pnp: PnP ACPI: found 6 devices1434worker1 # [ 0.529489] pci 0000:00:01.0: vgaarb: setting as boot VGA device1435worker1 # [ 0.529544] pci 0000:00:01.0: vgaarb: bridge control possible1436worker1 # [ 0.529544] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1437worker1 # [ 0.529552] vgaarb: loaded1438worker1 # [ 0.530301] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01439client # [ 0.525471] clocksource: Switched to clocksource kvm-clock1440worker1 # [ 0.530548] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1441client # [ 0.529270] VFS: Disk quotas dquot_6.6.01442worker2 # [ 0.554847] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1443client # [ 0.530006] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1444worker1 # [ 0.533637] clocksource: Switched to clocksource kvm-clock1445worker2 # [ 0.556343] clocksource: Switched to clocksource acpi_pm1446client # [ 0.531376] pnp: PnP ACPI init1447worker2 # [ 0.557373] NET: Registered PF_INET protocol family1448client # [ 0.532178] ACPI: IRQ 4 override to edge(!), high(!)1449worker2 # [ 0.558606] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1450client # [ 0.533181] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1451worker1 # [ 0.537493] VFS: Disk quotas dquot_6.6.01452client # [ 0.534712] pnp: PnP ACPI: found 6 devices1453worker1 # [ 0.538200] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1454worker1 # [ 0.539479] pnp: PnP ACPI init1455worker1 # [ 0.540244] ACPI: IRQ 4 override to edge(!), high(!)1456worker1 # [ 0.541190] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1457worker1 # [ 0.542690] pnp: PnP ACPI: found 6 devices1458lb # [ 0.534917] pci 0000:00:01.0: vgaarb: setting as boot VGA device1459client # [ 0.542332] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1460client # [ 0.543799] clocksource: Switched to clocksource acpi_pm1461lb # [ 0.535622] pci 0000:00:01.0: vgaarb: bridge control possible1462client # [ 0.544744] NET: Registered PF_INET protocol family1463lb # [ 0.535622] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1464lb # [ 0.535630] vgaarb: loaded1465client # [ 0.545974] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1466lb # [ 0.536388] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01467worker1 # [ 0.550308] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1468lb # [ 0.536626] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1469worker1 # [ 0.551711] clocksource: Switched to clocksource acpi_pm1470worker1 # [ 0.552658] NET: Registered PF_INET protocol family1471worker1 # [ 0.553849] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1472worker2 # [ 0.576177] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1473lb # [ 0.539711] clocksource: Switched to clocksource kvm-clock1474worker2 # [ 0.577789] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1475worker2 # [ 0.579118] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1476worker2 # [ 0.580552] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1477lb # [ 0.543177] VFS: Disk quotas dquot_6.6.01478worker2 # [ 0.581821] TCP: Hash tables configured (established 16384 bind 16384)1479lb # [ 0.543856] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1480lb # [ 0.545216] pnp: PnP ACPI init1481worker2 # [ 0.582938] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1482lb # [ 0.545986] ACPI: IRQ 4 override to edge(!), high(!)1483worker2 # [ 0.584275] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1484lb # [ 0.547027] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1485worker2 # [ 0.585369] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1486lb # [ 0.548564] pnp: PnP ACPI: found 6 devices1487worker2 # [ 0.586584] NET: Registered PF_UNIX/PF_LOCAL protocol family1488worker2 # [ 0.587603] NET: Registered PF_XDP protocol family1489client # [ 0.562370] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1490worker2 # [ 0.588500] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1491worker2 # [ 0.589523] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1492client # [ 0.563912] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1493worker2 # [ 0.590564] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1494client # [ 0.565245] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1495worker2 # [ 0.591695] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1496client # [ 0.566579] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1497worker2 # [ 0.592822] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1498worker1 # [ 0.570409] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1499client # [ 0.567839] TCP: Hash tables configured (established 16384 bind 16384)1500worker2 # [ 0.593898] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1501worker1 # [ 0.571906] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1502lb # [ 0.556131] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1503client # [ 0.568924] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1504lb # [ 0.557563] clocksource: Switched to clocksource acpi_pm1505worker2 # [ 0.595657] ACPI: \_SB_.GSIA: Enabled at IRQ 161506worker1 # [ 0.573157] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1507client # [ 0.570141] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1508lb # [ 0.558495] NET: Registered PF_INET protocol family1509client # [ 0.571168] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1510worker1 # [ 0.574415] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1511client # [ 0.572334] NET: Registered PF_UNIX/PF_LOCAL protocol family1512lb # [ 0.559643] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1513worker1 # [ 0.575606] TCP: Hash tables configured (established 16384 bind 16384)1514worker2 # [ 0.598357] ACPI: \_SB_.GSIB: Enabled at IRQ 171515client # [ 0.573271] NET: Registered PF_XDP protocol family1516worker1 # [ 0.576667] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1517client # [ 0.574106] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1518worker2 # [ 0.600316] ACPI: \_SB_.GSIC: Enabled at IRQ 181519worker1 # [ 0.577883] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1520client # [ 0.575177] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1521worker1 # [ 0.578929] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1522client # [ 0.576202] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1523worker2 # [ 0.602279] ACPI: \_SB_.GSID: Enabled at IRQ 191524worker1 # [ 0.580141] NET: Registered PF_UNIX/PF_LOCAL protocol family1525client # [ 0.577359] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1526worker1 # [ 0.581078] NET: Registered PF_XDP protocol family1527worker2 # [ 0.604323] PCI: CLS 0 bytes, default 641528client # [ 0.578435] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1529worker1 # [ 0.581897] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1530client # [ 0.579506] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1531worker1 # [ 0.582887] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1532worker2 # [ 0.605165] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1533worker1 # [ 0.583862] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1534client # [ 0.581195] ACPI: \_SB_.GSIA: Enabled at IRQ 161535worker2 # [ 0.606856] Trying to unpack rootfs image as initramfs...1536worker1 # [ 0.584989] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1537worker1 # [ 0.586116] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1538client # [ 0.583403] ACPI: \_SB_.GSIB: Enabled at IRQ 171539worker1 # [ 0.587233] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1540worker1 # [ 0.589080] ACPI: \_SB_.GSIA: Enabled at IRQ 161541client # [ 0.585775] ACPI: \_SB_.GSIC: Enabled at IRQ 181542client # [ 0.587817] ACPI: \_SB_.GSID: Enabled at IRQ 191543worker1 # [ 0.591562] ACPI: \_SB_.GSIB: Enabled at IRQ 171544lb # [ 0.576256] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1545client # [ 0.589570] PCI: CLS 0 bytes, default 641546worker1 # [ 0.593568] ACPI: \_SB_.GSIC: Enabled at IRQ 181547lb # [ 0.577807] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1548client # [ 0.590522] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1549lb # [ 0.579084] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1550worker1 # [ 0.595604] ACPI: \_SB_.GSID: Enabled at IRQ 191551client # [ 0.592232] Trying to unpack rootfs image as initramfs...1552lb # [ 0.580476] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1553worker1 # [ 0.597279] PCI: CLS 0 bytes, default 641554lb # [ 0.581771] TCP: Hash tables configured (established 16384 bind 16384)1555worker1 # [ 0.598140] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1556lb # [ 0.582925] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1557worker1 # [ 0.599854] Trying to unpack rootfs image as initramfs...1558lb # [ 0.584198] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1559lb # [ 0.585273] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1560lb # [ 0.586585] NET: Registered PF_UNIX/PF_LOCAL protocol family1561lb # [ 0.587604] NET: Registered PF_XDP protocol family1562lb # [ 0.588488] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1563lb # [ 0.589533] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1564lb # [ 0.590598] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1565lb # [ 0.591764] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1566lb # [ 0.592906] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1567lb # [ 0.594043] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1568lb # [ 0.595803] ACPI: \_SB_.GSIA: Enabled at IRQ 161569lb # [ 0.598470] ACPI: \_SB_.GSIB: Enabled at IRQ 171570lb # [ 0.600428] ACPI: \_SB_.GSIC: Enabled at IRQ 181571lb # [ 0.602415] ACPI: \_SB_.GSID: Enabled at IRQ 191572lb # [ 0.604067] PCI: CLS 0 bytes, default 641573lb # [ 0.605001] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1574lb # [ 0.606815] Trying to unpack rootfs image as initramfs...1575worker2 # [ 0.651725] Initialise system trusted keyrings1576worker2 # [ 0.655296] workingset: timestamp_bits=40 max_order=19 bucket_order=01577client # [ 0.635495] Initialise system trusted keyrings1578client # [ 0.639139] workingset: timestamp_bits=40 max_order=19 bucket_order=01579worker1 # [ 0.644022] Initialise system trusted keyrings1580worker1 # [ 0.647005] workingset: timestamp_bits=40 max_order=19 bucket_order=01581worker2 # [ 0.676520] Key type asymmetric registered1582worker2 # [ 0.679271] Asymmetric key parser 'x509' registered1583worker2 # [ 0.680097] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1584worker2 # [ 0.683389] io scheduler mq-deadline registered1585worker2 # [ 0.684120] io scheduler kyber registered1586client # [ 0.660328] Key type asymmetric registered1587worker2 # [ 0.686880] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1588client # [ 0.662102] Asymmetric key parser 'x509' registered1589worker2 # [ 0.688132] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1590client # [ 0.663003] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1591lb # [ 0.651381] Initialise system trusted keyrings1592worker1 # [ 0.667914] Key type asymmetric registered1593client # [ 0.666257] io scheduler mq-deadline registered1594worker1 # [ 0.669982] Asymmetric key parser 'x509' registered1595client # [ 0.667054] io scheduler kyber registered1596lb # [ 0.654351] workingset: timestamp_bits=40 max_order=19 bucket_order=01597worker2 # [ 0.693565] Linux agpgart interface v0.1031598worker1 # [ 0.670829] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1599worker2 # [ 0.694332] ACPI: bus type drm_connector registered1600worker1 # [ 0.674128] io scheduler mq-deadline registered1601client # [ 0.671253] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1602worker1 # [ 0.674891] io scheduler kyber registered1603worker2 # [ 0.697624] usbcore: registered new interface driver usbserial_generic1604client # [ 0.672580] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1605worker2 # [ 0.698656] usbserial: USB Serial support registered for generic1606worker1 # [ 0.678606] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1607worker2 # [ 0.701261] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1608worker2 # [ 0.701261] Please enable it if your BIOS has the CPPC option.1609client # [ 0.677906] Linux agpgart interface v0.1031610worker2 # [ 0.703384] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1611client # [ 0.678726] ACPI: bus type drm_connector registered1612worker1 # [ 0.679934] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1613worker2 # [ 0.707441] drop_monitor: Initializing network drop monitor service1614worker1 # [ 0.685581] Linux agpgart interface v0.1031615worker2 # [ 0.708591] NET: Registered PF_INET6 protocol family1616worker1 # [ 0.686373] ACPI: bus type drm_connector registered1617client # [ 0.683525] usbcore: registered new interface driver usbserial_generic1618client # [ 0.684635] usbserial: USB Serial support registered for generic1619worker2 # [ 0.711732] Segment Routing with IPv61620worker1 # [ 0.689442] usbcore: registered new interface driver usbserial_generic1621worker1 # [ 0.690531] usbserial: USB Serial support registered for generic1622lb # [ 0.675667] Key type asymmetric registered1623worker2 # [ 0.714269] In-situ OAM (IOAM) with IPv61624client # [ 0.688101] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1625lb # [ 0.676427] Asymmetric key parser 'x509' registered1626client # [ 0.688101] Please enable it if your BIOS has the CPPC option.1627worker2 # [ 0.715320] IPI shorthand broadcast: enabled1628lb # [ 0.677236] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1629client # [ 0.690276] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1630worker1 # [ 0.694002] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1631client # [ 0.691633] drop_monitor: Initializing network drop monitor service1632worker1 # [ 0.694002] Please enable it if your BIOS has the CPPC option.1633client # [ 0.692795] NET: Registered PF_INET6 protocol family1634worker1 # [ 0.696264] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1635lb # [ 0.682436] io scheduler mq-deadline registered1636lb # [ 0.683220] io scheduler kyber registered1637worker1 # [ 0.700109] drop_monitor: Initializing network drop monitor service1638client # [ 0.697596] Segment Routing with IPv61639worker1 # [ 0.701309] NET: Registered PF_INET6 protocol family1640worker2 # [ 0.723646] sched_clock: Marking stable (588022951, 135211504)->(797432892, -74198437)1641lb # [ 0.685851] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1642client # [ 0.700115] In-situ OAM (IOAM) with IPv61643lb # [ 0.687164] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1644worker1 # [ 0.704385] Segment Routing with IPv61645client # [ 0.701198] IPI shorthand broadcast: enabled1646worker2 # [ 0.727519] registered taskstats version 11647worker2 # [ 0.728482] Loading compiled-in X.509 certificates1648worker1 # [ 0.706994] In-situ OAM (IOAM) with IPv61649worker1 # [ 0.708088] IPI shorthand broadcast: enabled1650lb # [ 0.692581] Linux agpgart interface v0.1031651lb # [ 0.693324] ACPI: bus type drm_connector registered1652lb # [ 0.696589] usbcore: registered new interface driver usbserial_generic1653client # [ 0.709367] sched_clock: Marking stable (576014158, 133070825)->(784915905, -75830922)1654lb # [ 0.697693] usbserial: USB Serial support registered for generic1655lb # [ 0.700323] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1656worker1 # [ 0.716361] sched_clock: Marking stable (583015650, 132938826)->(788164269, -72209793)1657lb # [ 0.700323] Please enable it if your BIOS has the CPPC option.1658client # [ 0.714348] registered taskstats version 11659client # [ 0.715333] Loading compiled-in X.509 certificates1660lb # [ 0.702560] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1661worker1 # [ 0.721060] registered taskstats version 11662worker1 # [ 0.722082] Loading compiled-in X.509 certificates1663lb # [ 0.706590] drop_monitor: Initializing network drop monitor service1664lb # [ 0.707798] NET: Registered PF_INET6 protocol family1665worker2 # [ 0.748248] Demotion targets for Node 0: null1666lb # [ 0.710726] Segment Routing with IPv61667worker2 # [ 0.749146] Key type .fscrypt registered1668worker2 # [ 0.749819] Key type fscrypt-provisioning registered1669worker2 # [ 0.750741] ima: No TPM chip found, activating TPM-bypass!1670lb # [ 0.713335] In-situ OAM (IOAM) with IPv61671lb # [ 0.714410] IPI shorthand broadcast: enabled1672worker2 # [ 0.755255] ima: Allocated hash algorithm: sha11673worker2 # [ 0.756037] ima: No architecture policies found1674worker2 # [ 0.756989] PM: Magic number: 10:760:5781675worker2 # [ 0.757727] platform pcspkr: hash matches1676client # [ 0.734109] Demotion targets for Node 0: null1677client # [ 0.735029] Key type .fscrypt registered1678client # [ 0.735769] Key type fscrypt-provisioning registered1679worker2 # [ 0.761634] RAS: Correctable Errors collector initialized.1680worker1 # [ 0.739974] Demotion targets for Node 0: null1681lb # [ 0.723757] sched_clock: Marking stable (590015598, 133283524)->(802345698, -79046576)1682client # [ 0.738209] ima: No TPM chip found, activating TPM-bypass!1683worker1 # [ 0.742008] Key type .fscrypt registered1684client # [ 0.739183] ima: Allocated hash algorithm: sha11685worker1 # [ 0.742696] Key type fscrypt-provisioning registered1686client # [ 0.739991] ima: No architecture policies found1687worker1 # [ 0.743699] ima: No TPM chip found, activating TPM-bypass!1688lb # [ 0.728405] registered taskstats version 11689lb # [ 0.729356] Loading compiled-in X.509 certificates1690client # [ 0.744282] PM: Magic number: 10:760:5781691worker1 # [ 0.747970] ima: Allocated hash algorithm: sha11692client # [ 0.745063] platform pcspkr: hash matches1693worker2 # [ 0.770958] clk: Disabling unused clocks1694worker1 # [ 0.748800] ima: No architecture policies found1695worker1 # [ 0.750973] PM: Magic number: 10:760:5781696worker2 # [ 0.773259] PM: genpd: Disabling unused power domains1697worker1 # [ 0.751723] platform pcspkr: hash matches1698client # [ 0.748726] RAS: Correctable Errors collector initialized.1699worker1 # [ 0.755372] RAS: Correctable Errors collector initialized.1700client # [ 0.757453] clk: Disabling unused clocks1701lb # [ 0.745851] Demotion targets for Node 0: null1702client # [ 0.760097] PM: genpd: Disabling unused power domains1703worker1 # [ 0.764117] clk: Disabling unused clocks1704lb # [ 0.749355] Key type .fscrypt registered1705worker1 # [ 0.764858] PM: genpd: Disabling unused power domains1706lb # [ 0.749995] Key type fscrypt-provisioning registered1707lb # [ 0.751001] ima: No TPM chip found, activating TPM-bypass!1708lb # [ 0.753314] ima: Allocated hash algorithm: sha11709lb # [ 0.754110] ima: No architecture policies found1710lb # [ 0.756316] PM: Magic number: 10:760:5781711lb # [ 0.757043] platform pcspkr: hash matches1712lb # [ 0.760732] RAS: Correctable Errors collector initialized.1713lb # [ 0.769455] clk: Disabling unused clocks1714lb # [ 0.770154] PM: genpd: Disabling unused power domains1715worker2 # [ 0.942263] Freeing initrd memory: 29092K1716client # [ 0.919063] Freeing initrd memory: 29096K1717worker2 # [ 0.945859] Freeing unused decrypted memory: 2028K1718client # [ 0.922251] Freeing unused decrypted memory: 2028K1719worker1 # [ 0.926011] Freeing initrd memory: 29092K1720worker2 # [ 0.948882] Freeing unused kernel image (initmem) memory: 3644K1721worker2 # [ 0.949950] Write protecting the kernel read-only data: 32768k1722client # [ 0.924871] Freeing unused kernel image (initmem) memory: 3644K1723worker1 # [ 0.929176] Freeing unused decrypted memory: 2028K1724client # [ 0.926004] Write protecting the kernel read-only data: 32768k1725worker2 # [ 0.951936] Freeing unused kernel image (text/rodata gap) memory: 1208K1726client # [ 0.927863] Freeing unused kernel image (text/rodata gap) memory: 1208K1727worker2 # [ 0.953565] Freeing unused kernel image (rodata/data gap) memory: 752K1728worker1 # [ 0.931731] Freeing unused kernel image (initmem) memory: 3644K1729client # [ 0.929376] Freeing unused kernel image (rodata/data gap) memory: 752K1730worker1 # [ 0.932836] Write protecting the kernel read-only data: 32768k1731worker1 # [ 0.934731] Freeing unused kernel image (text/rodata gap) memory: 1208K1732worker1 # [ 0.936305] Freeing unused kernel image (rodata/data gap) memory: 752K1733lb # [ 0.936885] Freeing initrd memory: 29092K1734lb # [ 0.940057] Freeing unused decrypted memory: 2028K1735lb # [ 0.942720] Freeing unused kernel image (initmem) memory: 3644K1736lb # [ 0.943841] Write protecting the kernel read-only data: 32768k1737lb # [ 0.945759] Freeing unused kernel image (text/rodata gap) memory: 1208K1738lb # [ 0.947339] Freeing unused kernel image (rodata/data gap) memory: 752K1739worker2 # [ 1.004777] x86/mm: Checked W+X mappings: passed, no W+X pages found.1740worker2 # [ 1.005827] Run /init as init process1741client # [ 0.980640] x86/mm: Checked W+X mappings: passed, no W+X pages found.1742client # [ 0.981760] Run /init as init process1743worker1 # [ 0.987573] x86/mm: Checked W+X mappings: passed, no W+X pages found.1744worker1 # [ 0.988699] Run /init as init process1745worker2 # [ 1.016608] systemd[1]: Inserted module 'autofs4'1746client # [ 0.992160] systemd[1]: Inserted module 'autofs4'1747worker1 # [ 0.998984] systemd[1]: Inserted module 'autofs4'1748worker2 # [ 1.033547] fuse: init (API version 7.45)1749client # [ 1.008394] fuse: init (API version 7.45)1750lb # [ 0.998475] x86/mm: Checked W+X mappings: passed, no W+X pages found.1751worker1 # [ 1.015205] fuse: init (API version 7.45)1752lb # [ 0.999569] Run /init as init process1753worker2 # [ 1.039507] ACPI: \_SB_.GSIG: Enabled at IRQ 221754client # [ 1.014285] ACPI: \_SB_.GSIG: Enabled at IRQ 221755worker2 # [ 1.041893] ACPI: \_SB_.GSIH: Enabled at IRQ 231756client # [ 1.016577] ACPI: \_SB_.GSIH: Enabled at IRQ 231757worker1 # [ 1.020963] ACPI: \_SB_.GSIG: Enabled at IRQ 221758worker2 # [ 1.045403] ACPI: \_SB_.GSIE: Enabled at IRQ 201759client # [ 1.020146] ACPI: \_SB_.GSIE: Enabled at IRQ 201760worker1 # [ 1.023721] ACPI: \_SB_.GSIH: Enabled at IRQ 231761lb # [ 1.009604] systemd[1]: Inserted module 'autofs4'1762worker2 # [ 1.047953] ACPI: \_SB_.GSIF: Enabled at IRQ 211763client # [ 1.022823] ACPI: \_SB_.GSIF: Enabled at IRQ 211764worker1 # [ 1.026776] ACPI: \_SB_.GSIE: Enabled at IRQ 201765worker1 # [ 1.029068] ACPI: \_SB_.GSIF: Enabled at IRQ 211766lb # [ 1.026010] fuse: init (API version 7.45)1767lb # [ 1.031870] ACPI: \_SB_.GSIG: Enabled at IRQ 221768lb # [ 1.034639] ACPI: \_SB_.GSIH: Enabled at IRQ 231769lb # [ 1.037881] ACPI: \_SB_.GSIE: Enabled at IRQ 201770worker2 # [ 1.076201] systemd[1]: Successfully made /usr/ read-only.1771client # [ 1.050955] systemd[1]: Successfully made /usr/ read-only.1772lb # [ 1.040629] ACPI: \_SB_.GSIF: Enabled at IRQ 211773worker1 # [ 1.056697] systemd[1]: Successfully made /usr/ read-only.1774lb # [ 1.068573] systemd[1]: Successfully made /usr/ read-only.1775worker2 # [ 1.412747] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1776client # [ 1.387537] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1777worker2 # [ 1.424623] systemd[1]: Detected virtualization kvm.1778client # [ 1.399590] systemd[1]: Detected virtualization kvm.1779worker1 # [ 1.393244] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1780worker2 # [ 1.426698] systemd[1]: Detected architecture x86-64.1781client # [ 1.401761] systemd[1]: Detected architecture x86-64.1782worker1 # [ 1.405229] systemd[1]: Detected virtualization kvm.1783worker2 # [ 1.428844] systemd[1]: Running in initrd.1784client # [ 1.403966] systemd[1]: Running in initrd.1785worker1 # [ 1.407387] systemd[1]: Detected architecture x86-64.1786worker1 # [ 1.409544] systemd[1]: Running in initrd.1787worker2 # [ 1.431374] systemd[1]: Initializing machine ID from random generator.1788client # [ 1.406507] systemd[1]: Initializing machine ID from random generator.1789worker2 # [ 1.434163] systemd[1]: Hostname set to <worker2>.1790client # [ 1.409416] systemd[1]: Hostname set to <client>.1791worker1 # [ 1.412137] systemd[1]: Initializing machine ID from random generator.1792worker1 # [ 1.415160] systemd[1]: Hostname set to <worker1>.1793lb # [ 1.405425] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1794lb # [ 1.417260] systemd[1]: Detected virtualization kvm.1795lb # [ 1.419413] systemd[1]: Detected architecture x86-64.1796lb # [ 1.421553] systemd[1]: Running in initrd.1797lb # [ 1.424069] systemd[1]: Initializing machine ID from random generator.1798lb # [ 1.426944] systemd[1]: Hostname set to <lb>.1799worker2 # [ 1.646543] systemd[1]: bpf-restrict-fs: LSM BPF program attached1800client # [ 1.633444] systemd[1]: bpf-restrict-fs: LSM BPF program attached1801worker1 # [ 1.640719] systemd[1]: bpf-restrict-fs: LSM BPF program attached1802worker2 # [ 1.685917] systemd[1]: Queued start job for default target Initrd Default Target.1803worker2 # [ 1.690563] systemd[1]: Created slice Slice /system/modprobe.1804lb # [ 1.651643] systemd[1]: bpf-restrict-fs: LSM BPF program attached1805worker2 # [ 1.691717] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1806worker2 # [ 1.693133] systemd[1]: Expecting device /dev/disk/by-label/nixos...1807worker2 # [ 1.694298] systemd[1]: Reached target Path Units.1808worker2 # [ 1.695159] systemd[1]: Reached target Slice Units.1809worker2 # [ 1.696103] systemd[1]: Reached target Swaps.1810worker2 # [ 1.696913] systemd[1]: Reached target Timer Units.1811worker2 # [ 1.697908] systemd[1]: Listening on D-Bus System Message Bus Socket.1812worker2 # [ 1.699156] systemd[1]: Listening on Journal Socket (/dev/log).1813worker2 # [ 1.700394] systemd[1]: Listening on Journal Sockets.1814worker2 # [ 1.701405] systemd[1]: Listening on udev Control Socket.1815worker2 # [ 1.702447] systemd[1]: Listening on udev Kernel Socket.1816worker2 # [ 1.703439] systemd[1]: Reached target Socket Units.1817worker2 # [ 1.705245] systemd[1]: Starting Create List of Static Device Nodes...1818worker2 # [ 1.709045] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1819client # [ 1.687945] systemd[1]: Queued start job for default target Initrd Default Target.1820worker2 # [ 1.716316] systemd[1]: Starting Load Kernel Module configfs...1821worker1 # [ 1.694842] systemd[1]: Queued start job for default target Initrd Default Target.1822client # [ 1.692448] systemd[1]: Created slice Slice /system/modprobe.1823client # [ 1.693598] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1824client # [ 1.695049] systemd[1]: Expecting device /dev/disk/by-label/nixos...1825worker1 # [ 1.699302] systemd[1]: Created slice Slice /system/modprobe.1826client # [ 1.696222] systemd[1]: Reached target Path Units.1827client # [ 1.697120] systemd[1]: Reached target Slice Units.1828worker1 # [ 1.700497] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1829client # [ 1.698003] systemd[1]: Reached target Swaps.1830client # [ 1.698848] systemd[1]: Reached target Timer Units.1831worker1 # [ 1.701966] systemd[1]: Expecting device /dev/disk/by-label/nixos...1832worker1 # [ 1.703101] systemd[1]: Reached target Path Units.1833client # [ 1.699847] systemd[1]: Listening on D-Bus System Message Bus Socket.1834worker1 # [ 1.703979] systemd[1]: Reached target Slice Units.1835client # [ 1.701129] systemd[1]: Listening on Journal Socket (/dev/log).1836worker1 # [ 1.704848] systemd[1]: Reached target Swaps.1837worker2 # [ 1.727306] systemd[1]: Starting Journal Service...1838worker1 # [ 1.705661] systemd[1]: Reached target Timer Units.1839client # [ 1.702354] systemd[1]: Listening on Journal Sockets.1840client # [ 1.703360] systemd[1]: Listening on udev Control Socket.1841worker1 # [ 1.706636] systemd[1]: Listening on D-Bus System Message Bus Socket.1842client # [ 1.704399] systemd[1]: Listening on udev Kernel Socket.1843worker1 # [ 1.707880] systemd[1]: Listening on Journal Socket (/dev/log).1844client # [ 1.705376] systemd[1]: Reached target Socket Units.1845worker1 # [ 1.709092] systemd[1]: Listening on Journal Sockets.1846worker1 # [ 1.710146] systemd[1]: Listening on udev Control Socket.1847client # [ 1.707229] systemd[1]: Starting Create List of Static Device Nodes...1848worker1 # [ 1.711163] systemd[1]: Listening on udev Kernel Socket.1849worker1 # [ 1.712135] systemd[1]: Reached target Socket Units.1850worker1 # [ 1.713891] systemd[1]: Starting Create List of Static Device Nodes...1851client # [ 1.711156] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1852worker1 # [ 1.717755] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1853client # [ 1.717650] systemd[1]: Starting Load Kernel Module configfs...1854lb # [ 1.706919] systemd[1]: Queued start job for default target Initrd Default Target.1855worker1 # [ 1.725135] systemd[1]: Starting Load Kernel Module configfs...1856worker2 # [ 1.737403] systemd[1]: Starting Load Kernel Modules...1857lb # [ 1.711694] systemd[1]: Created slice Slice /system/modprobe.1858lb # [ 1.712906] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1859lb # [ 1.714360] systemd[1]: Expecting device /dev/disk/by-label/nixos...1860lb # [ 1.715496] systemd[1]: Reached target Path Units.1861lb # [ 1.716395] systemd[1]: Reached target Slice Units.1862lb # [ 1.717265] systemd[1]: Reached target Swaps.1863lb # [ 1.718133] systemd[1]: Reached target Timer Units.1864client # [ 1.731277] systemd[1]: Starting Journal Service...1865lb # [ 1.719122] systemd[1]: Listening on D-Bus System Message Bus Socket.1866lb # [ 1.720358] systemd[1]: Listening on Journal Socket (/dev/log).1867lb # [ 1.721587] systemd[1]: Listening on Journal Sockets.1868lb # [ 1.722611] systemd[1]: Listening on udev Control Socket.1869worker1 # [ 1.739242] systemd[1]: Starting Journal Service...1870lb # [ 1.723663] systemd[1]: Listening on udev Kernel Socket.1871worker2 # [ 1.762522] netfs: FS-Cache loaded1872lb # [ 1.724627] systemd[1]: Reached target Socket Units.1873lb # [ 1.726587] systemd[1]: Starting Create List of Static Device Nodes...1874lb # [ 1.730371] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1875worker2 # [ 1.770461] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1876worker2 # [ 1.773013] 9pnet: Installing 9P2000 support1877lb # [ 1.739389] systemd[1]: Starting Load Kernel Module configfs...1878client # [ 1.743526] systemd[1]: Starting Load Kernel Modules...1879worker2 # [ 1.781311] systemd[1]: Starting Coldplug All udev Devices...1880worker1 # [ 1.752874] systemd[1]: Starting Load Kernel Modules...1881client # [ 1.764440] netfs: FS-Cache loaded1882lb # [ 1.753373] systemd[1]: Starting Journal Service...1883worker1 # [ 1.771575] netfs: FS-Cache loaded1884client # [ 1.769219] 9pnet: Installing 9P2000 support1885worker2 # [ 1.796851] systemd[1]: Finished Create List of Static Device Nodes.1886worker1 # [ 1.776251] 9pnet: Installing 9P2000 support1887worker2 # [ 1.802864] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1888client # [ 1.780220] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1889worker1 # [ 1.783596] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1890worker2 # [ 1.811653] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1891worker2 # [ 1.813578] systemd-journald[67]: Collecting audit messages is disabled.1892worker1 # [ 1.793119] systemd[1]: Starting Coldplug All udev Devices...1893lb # [ 1.767411] systemd[1]: Starting Load Kernel Modules...1894client # [ 1.791179] systemd[1]: Starting Coldplug All udev Devices...1895worker2 # [ 1.817659] systemd[1]: modprobe@configfs.service: Deactivated successfully.1896lb # [ 1.786558] netfs: FS-Cache loaded1897worker2 # [ 1.824392] systemd[1]: Finished Load Kernel Module configfs.1898client # [ 1.802243] systemd[1]: Finished Create List of Static Device Nodes.1899worker1 # [ 1.806039] systemd[1]: Finished Create List of Static Device Nodes.1900worker2 # [ 1.827545] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1901lb # [ 1.791270] 9pnet: Installing 9P2000 support1902worker1 # [ 1.808570] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1903worker2 # [ 1.831729] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1904client # [ 1.808740] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1905worker1 # [ 1.812759] systemd-journald[67]: Collecting audit messages is disabled.1906worker2 # [ 1.836278] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1907lb # [ 1.798508] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1908client # [ 1.811678] systemd-journald[67]: Collecting audit messages is disabled.1909worker1 # [ 1.817411] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1910worker2 # [ 1.841313] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1911client # [ 1.816481] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1912lb # [ 1.808427] systemd[1]: Starting Coldplug All udev Devices...1913worker1 # [ 1.824354] systemd[1]: modprobe@configfs.service: Deactivated successfully.1914client # [ 1.821450] systemd[1]: modprobe@configfs.service: Deactivated successfully.1915worker1 # [ 1.830331] systemd[1]: Finished Load Kernel Module configfs.1916client # [ 1.826043] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1917client # [ 1.830443] systemd[1]: Finished Load Kernel Module configfs.1918worker1 # [ 1.834107] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1919lb # [ 1.821496] systemd[1]: Finished Create List of Static Device Nodes.1920worker1 # [ 1.836683] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1921client # [ 1.834559] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1922lb # [ 1.826952] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1923worker1 # [ 1.842972] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1924client # [ 1.840136] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1925worker2 # [ 1.866787] systemd[1]: Finished Load Kernel Modules.1926worker1 # [ 1.847775] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1927client # [ 1.846577] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1928lb # [ 1.834726] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1929worker2 # [ 1.875339] systemd[1]: Starting Apply Kernel Variables...1930lb # [ 1.838494] systemd-journald[67]: Collecting audit messages is disabled.1931lb # [ 1.841730] systemd[1]: modprobe@configfs.service: Deactivated successfully.1932lb # [ 1.848695] systemd[1]: Finished Load Kernel Module configfs.1933worker2 # [ 1.888395] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1934lb # [ 1.851802] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1935lb # [ 1.854507] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1936client # [ 1.869619] systemd[1]: Finished Load Kernel Modules.1937worker1 # [ 1.873897] systemd[1]: Finished Load Kernel Modules.1938worker2 # [ 1.899481] systemd[1]: Starting Create Static Device Nodes in /dev...1939lb # [ 1.862008] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1940worker1 # [ 1.881131] systemd[1]: Starting Apply Kernel Variables...1941client # [ 1.878278] systemd[1]: Starting Apply Kernel Variables...1942lb # [ 1.866391] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1943worker2 # [ 1.911832] systemd[1]: Finished Apply Kernel Variables.1944worker1 # [ 1.893087] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1945client # [ 1.892404] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1946worker1 # [ 1.903148] systemd[1]: Starting Create Static Device Nodes in /dev...1947client # [ 1.903285] systemd[1]: Starting Create Static Device Nodes in /dev...1948lb # [ 1.892906] systemd[1]: Finished Load Kernel Modules.1949worker2 # [ 1.796368] systemd-modules-load[68]: Inserted module 'dm_mod'[ 1.932692] systemd[1]: Started Journal Service.1950worker2 # 1951worker2 # [ 1.799303] systemd-modules-load[68]: Inserted module 'virtio_balloon'1952worker2 # [ 1.802293] systemd-modules-load[68]: Inserted module 'virtio_gpu'1953worker2 # [ 1.803353] systemd[1]: Finished Create Static Device Nodes in /dev.1954lb # [ 1.901524] systemd[1]: Starting Apply Kernel Variables...1955client # [ 1.914637] systemd[1]: Finished Apply Kernel Variables.1956worker2 # [ 1.805357] systemd[1]: Reached target Preparation for Local File Systems.1957worker1 # [ 1.919108] systemd[1]: Finished Apply Kernel Variables.1958worker2 # [ 1.806780] systemd[1]: Reached target Local File Systems.1959worker2 # [ 1.809279] systemd[1]: Starting Create System Files and Directories...1960worker2 # [ 1.817048] systemd[1]: Starting Rule-based Manager for Device Events and Files...1961worker1 # [ 1.931494] systemd[1]: Started Journal Service.1962lb # [ 1.915418] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1963client # [ 1.928486] systemd[1]: Started Journal Service.1964worker1 # [ 1.799564] systemd-modules-load[68]: Inserted module 'dm_mod'1965client # [ 1.796484] systemd-modules-load[68]: Inserted module 'dm_mod'1966worker1 # [ 1.803486] systemd-modules-load[68]: Inserted module 'virtio_balloon'1967client # [ 1.800307] systemd-modules-load[68]: Inserted module 'virtio_balloon'1968worker1 # [ 1.804566] systemd-modules-load[68]: Inserted module 'virtio_gpu'1969client # [ 1.801368] systemd-modules-load[68]: Inserted module 'virtio_gpu'1970worker1 # [ 1.807150] systemd[1]: Finished Create Static Device Nodes in /dev.1971worker1 # [ 1.808287] systemd[1]: Reached target Preparation for Local File Systems.1972lb # [ 1.925526] systemd[1]: Starting Create Static Device Nodes in /dev...1973worker1 # [ 1.810365] systemd[1]: Reached target Local File Systems.1974client # [ 1.810947] systemd[1]: Finished Create Static Device Nodes in /dev.1975worker1 # [ 1.814922] systemd[1]: Starting Create System Files and Directories...1976client # [ 1.813880] systemd[1]: Reached target Preparation for Local File Systems.1977client # [ 1.816793] systemd[1]: Reached target Local File Systems.1978worker1 # [ 1.820133] systemd[1]: Starting Rule-based Manager for Device Events and Files...1979client # [ 1.817857] systemd[1]: Starting Create System Files and Directories...1980worker2 # [ 1.843429] systemd[1]: Finished Create System Files and Directories.1981lb # [ 1.945887] systemd[1]: Finished Apply Kernel Variables.1982client # [ 1.826071] systemd[1]: Starting Rule-based Manager for Device Events and Files...1983lb # [ 1.956419] systemd[1]: Started Journal Service.1984lb # [ 1.825237] systemd-modules-load[68]: Inserted module 'dm_mod'1985lb # [ 1.826627] systemd-modules-load[68]: Inserted module 'virtio_balloon'1986lb # [ 1.829914] systemd-modules-load[68]: Inserted module 'virtio_gpu'1987worker1 # [ 1.846076] systemd[1]: Finished Create System Files and Directories.1988lb # [ 1.834233] systemd[1]: Finished Create Static Device Nodes in /dev.1989worker2 # [ 1.872262] systemd-udevd[76]: Using default interface naming scheme 'v261'.1990client # [ 1.848890] systemd[1]: Finished Create System Files and Directories.1991lb # [ 1.840808] systemd[1]: Reached target Preparation for Local File Systems.1992lb # [ 1.841815] systemd[1]: Reached target Local File Systems.1993lb # [ 1.845082] systemd[1]: Starting Create System Files and Directories...1994lb # [ 1.852072] systemd[1]: Starting Rule-based Manager for Device Events and Files...1995worker1 # [ 1.872232] systemd-udevd[78]: Using default interface naming scheme 'v261'.1996worker2 # [ 1.894629] systemd[1]: Started Rule-based Manager for Device Events and Files.1997client # [ 1.872250] systemd-udevd[83]: Using default interface naming scheme 'v261'.1998lb # [ 1.877097] systemd[1]: Finished Create System Files and Directories.1999worker1 # [ 1.896576] systemd[1]: Started Rule-based Manager for Device Events and Files.2000client # [ 1.894324] systemd[1]: Started Rule-based Manager for Device Events and Files.
loading 2,584 hidden lines…
4585worker2 # [ 38.176878] nix-grpc-daemon[787]: ts=2026-09-22T10:35:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:521524586worker2 # [ 38.189636] systemd[1]: Started Nix Daemon.4587worker2 # [ 38.385945] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4588worker2 # [ 38.412310] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 7 ms (attempt 1/5)4589worker2 # [ 38.421845] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 165 ms (attempt 2/5)4590worker2 # [ 38.589884] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 317 ms (attempt 3/5)4591worker2 # [ 38.906895] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 578 ms (attempt 4/5)4592worker2 # [ 39.490839] nix-grpc-daemon[787]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4593worker2 # [ 39.495265] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4594worker2 # [ 39.530194] nix-grpc-daemon[787]: ts=2026-09-22T10:35:08Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:521524595worker2 # [ 39.548864] nix-grpc-daemon[787]: ts=2026-09-22T10:35:08Z level=debug event=path_info_miss path=jwz4idn9byb3rgqdqa3827m7srpcmdj8-nixos-system-client-test error="error: path '/nix/store/jwz4idn9byb3rgqdqa3827m7srpcmdj8-nixos-system-client-test' is required, but there is no substituter that can build it"4596worker2 # [ 39.560160] nix-grpc-daemon[787]: ts=2026-09-22T10:35:08Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:521524597worker2 # [ 39.565305] nix-grpc-daemon[787]: ts=2026-09-22T10:35:08Z level=debug event=path_info_miss path=jwz4idn9byb3rgqdqa3827m7srpcmdj8-nixos-system-client-test error="error: path '/nix/store/jwz4idn9byb3rgqdqa3827m7srpcmdj8-nixos-system-client-test' is required, but there is no substituter that can build it"4598client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4599client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.09 seconds)4600client: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4601client: (finished: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4602worker1 # [ 39.931635] nix-grpc-daemon[784]: ts=2026-09-22T10:35:08Z level=info event=oidc_jwks provider=mock keys=14603worker1 # [ 39.936866] nix-grpc-daemon[784]: ts=2026-09-22T10:35:08Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:434624604worker1 # [ 39.949891] systemd[1]: Started Nix Daemon.4605worker1 # [ 40.157063] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4606worker1 # [ 40.182066] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 63 ms (attempt 1/5)4607worker1 # [ 40.247607] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 3 ms (attempt 2/5)4608worker1 # [ 40.253255] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 156 ms (attempt 3/5)4609worker1 # [ 40.413894] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 373 ms (attempt 4/5)4610worker1 # [ 40.797698] nix-grpc-daemon[784]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4611worker1 # [ 40.801736] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4612worker1 # [ 40.825307] nix-grpc-daemon[784]: ts=2026-09-22T10:35:09Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:434624613worker1 # [ 40.841593] nix-grpc-daemon[784]: ts=2026-09-22T10:35:09Z level=debug event=path_info_miss path=jwz4idn9byb3rgqdqa3827m7srpcmdj8-nixos-system-client-test error="error: path '/nix/store/jwz4idn9byb3rgqdqa3827m7srpcmdj8-nixos-system-client-test' is required, but there is no substituter that can build it"4614worker2 # [ 40.868389] nix-grpc-daemon[787]: ts=2026-09-22T10:35:09Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:521524615worker2 # [ 40.878163] nix-grpc-daemon[787]: ts=2026-09-22T10:35:09Z level=debug event=path_info_miss path=jwz4idn9byb3rgqdqa3827m7srpcmdj8-nixos-system-client-test error="error: path '/nix/store/jwz4idn9byb3rgqdqa3827m7srpcmdj8-nixos-system-client-test' is required, but there is no substituter that can build it"4616client: must succeed: rm -r /var/lib/nix-grpc-store4617client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.04 seconds)4618worker2 # [ 41.137437] nix-grpc-daemon[787]: ts=2026-09-22T10:35:10Z level=info event=denied method=QueryMissing cn=- role=none4619worker1 # [ 41.339125] nix-grpc-daemon[784]: ts=2026-09-22T10:35:09Z level=info event=denied method=QueryMissing cn=stranger role=none4620(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.51 seconds)4621subtest: DAG build is scheduled across workers and lands in the cache4622client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24623worker1 # [ 41.560837] nix-grpc-daemon[784]: ts=2026-09-22T10:35:09Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:434624624client # don't know how to build these paths:4625client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4626worker1 # [ 41.573476] nix-grpc-daemon[784]: ts=2026-09-22T10:35:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434624627worker2 # [ 41.622480] nix-grpc-daemon[787]: ts=2026-09-22T10:35:10Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04628worker2 # [ 41.627806] nix-grpc-daemon[787]: ts=2026-09-22T10:35:10Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14629worker1 # [ 41.658371] nix-grpc-daemon[784]: ts=2026-09-22T10:35:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434624630worker1 # [ 41.679551] nix-grpc-daemon[784]: ts=2026-09-22T10:35:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434624631client # copying 1 paths to 'lb:50051'...4632worker1 # [ 41.691619] nix-grpc-daemon[784]: ts=2026-09-22T10:35:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434624633client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4634worker1 # [ 41.708852] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4635worker1 # [ 41.736461] nix-grpc-daemon[784]: ts=2026-09-22T10:35:09Z level=info event=niks3_push_started pid=8634636worker1 # [ 42.027150] nix-grpc-daemon[863]: time=2026-09-22T10:35:10.245Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""4637worker1 # [ 42.208319] nix-daemon[851]: accepted connection from pid 877, user nix-grpc-daemon (trusted)4638worker1 # [ 42.222433] nix-daemon[851]: reaped child process 880, status = succeeded4639worker1 # [ 42.373211] nix-daemon[851]: accepted connection from pid 882, user nix-grpc-daemon (trusted)4640worker1 # [ 42.384858] nix-daemon[851]: reaped child process 885, status = succeeded4641lb # [ 42.383783] niks3-server[1016]: 2026/09/22 10:35:11 INFO Received uploads request method=POST path=/api/pending_closures4642worker1 # [ 42.423901] nix-grpc-daemon[863]: time=2026-09-22T10:35:10.645Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4643worker1 # [ 42.426278] nix-grpc-daemon[863]: time=2026-09-22T10:35:10.647Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4644lb # [ 42.454430] niks3-server[1016]: 2026/09/22 10:35:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4645lb # [ 42.465041] niks3-server[1016]: 2026/09/22 10:35:11 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4646lb # [ 42.481509] niks3-server[1016]: 2026/09/22 10:35:11 INFO Signed narinfos id=1 count=14647worker1 # [ 42.499953] nix-grpc-daemon[863]: time=2026-09-22T10:35:10.721Z level=INFO msg="Uploading 1 narinfos"4648lb # [ 42.492509] niks3-server[1016]: 2026/09/22 10:35:11 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4649lb # [ 42.497054] niks3-server[1016]: 2026/09/22 10:35:11 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4650lb # [ 42.507257] niks3-server[1016]: 2026/09/22 10:35:11 INFO Completed upload id=14651worker1 # [ 42.523830] nix-grpc-daemon[863]: time=2026-09-22T10:35:10.745Z level=INFO msg="Upload complete. (482ms)"4652lb # [ 42.512112] niks3-server[1016]: 2026/09/22 10:35:11 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4653worker1 # [ 42.531074] nix-grpc-daemon[784]: ts=2026-09-22T10:35:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=1 nar_bytes_in=5044654worker1 # [ 42.538280] nix-grpc-daemon[784]: ts=2026-09-22T10:35:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434624655worker1 # [ 42.550108] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4656worker1 # [ 42.557346] nix-daemon[851]: reaped child process 861, status = succeeded4657client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4658worker1 # [ 42.566587] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4659client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4660client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4661client # farm-a> LOG-farm-a-t14662worker1 # [ 43.795544] nix-daemon[851]: reaped child process 889, status = succeeded4663worker1 # [ 43.953412] nix-daemon[851]: accepted connection from pid 905, user nix-grpc-daemon (trusted)4664worker1 # [ 43.964233] nix-daemon[851]: reaped child process 908, status = succeeded4665lb # [ 43.962979] niks3-server[1016]: 2026/09/22 10:35:12 INFO Received uploads request method=POST path=/api/pending_closures4666worker1 # [ 43.992219] nix-grpc-daemon[863]: time=2026-09-22T10:35:12.213Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4667worker1 # [ 43.993921] nix-grpc-daemon[863]: time=2026-09-22T10:35:12.215Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4668lb # [ 44.016897] niks3-server[1016]: 2026/09/22 10:35:12 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4669lb # [ 44.022452] niks3-server[1016]: 2026/09/22 10:35:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4670lb # [ 44.029251] niks3-server[1016]: 2026/09/22 10:35:12 INFO Signed narinfos id=2 count=14671worker1 # [ 44.046165] nix-grpc-daemon[863]: time=2026-09-22T10:35:12.267Z level=INFO msg="Uploading 1 narinfos"4672lb # [ 44.032351] niks3-server[1016]: 2026/09/22 10:35:12 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4673lb # [ 44.039625] niks3-server[1016]: 2026/09/22 10:35:12 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4674lb # [ 44.047810] niks3-server[1016]: 2026/09/22 10:35:12 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4675lb # [ 44.054027] niks3-server[1016]: 2026/09/22 10:35:12 INFO Completed upload id=24676worker1 # [ 44.070500] nix-grpc-daemon[863]: time=2026-09-22T10:35:12.291Z level=INFO msg="Upload complete. (291ms)"4677lb # [ 44.056397] niks3-server[1016]: 2026/09/22 10:35:12 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4678worker1 # [ 44.078456] nix-grpc-daemon[784]: ts=2026-09-22T10:35:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14679worker1 # [ 44.089656] nix-daemon[851]: reaped child process 887, status = succeeded4680worker2 # [ 44.121962] nix-grpc-daemon[787]: ts=2026-09-22T10:35:13Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04681worker2 # [ 44.126234] nix-grpc-daemon[787]: ts=2026-09-22T10:35:13Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14682worker1 # [ 44.146073] nix-grpc-daemon[784]: ts=2026-09-22T10:35:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434624683worker1 # [ 44.159388] nix-grpc-daemon[784]: ts=2026-09-22T10:35:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434624684worker1 # [ 44.173135] nix-grpc-daemon[863]: time=2026-09-22T10:35:12.394Z level=INFO msg="All 1 paths already cached"4685client # copying 1 paths to 'lb:50051'...4686worker1 # [ 44.181503] nix-grpc-daemon[784]: ts=2026-09-22T10:35:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434624687client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4688worker1 # [ 44.198491] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4689worker1 # [ 44.353725] nix-daemon[851]: accepted connection from pid 912, user nix-grpc-daemon (trusted)4690worker1 # [ 44.366782] nix-daemon[851]: reaped child process 915, status = succeeded4691worker1 # [ 44.522155] nix-daemon[851]: accepted connection from pid 917, user nix-grpc-daemon (trusted)4692worker1 # [ 44.533721] nix-daemon[851]: reaped child process 920, status = succeeded4693lb # [ 44.533387] niks3-server[1016]: 2026/09/22 10:35:13 INFO Received uploads request method=POST path=/api/pending_closures4694worker1 # [ 44.559914] nix-grpc-daemon[863]: time=2026-09-22T10:35:12.781Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4695worker1 # [ 44.561440] nix-grpc-daemon[863]: time=2026-09-22T10:35:12.782Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4696lb # [ 44.562151] niks3-server[1016]: 2026/09/22 10:35:13 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4697lb # [ 44.566848] niks3-server[1016]: 2026/09/22 10:35:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4698lb # [ 44.568957] niks3-server[1016]: 2026/09/22 10:35:13 INFO Signed narinfos id=3 count=14699worker1 # [ 44.586477] nix-grpc-daemon[863]: time=2026-09-22T10:35:12.807Z level=INFO msg="Uploading 1 narinfos"4700lb # [ 44.575484] niks3-server[1016]: 2026/09/22 10:35:13 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4701lb # [ 44.579801] niks3-server[1016]: 2026/09/22 10:35:13 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4702lb # [ 44.582345] niks3-server[1016]: 2026/09/22 10:35:13 INFO Completed upload id=34703worker1 # [ 44.599743] nix-grpc-daemon[863]: time=2026-09-22T10:35:12.821Z level=INFO msg="Upload complete. (393ms)"4704lb # [ 44.584388] niks3-server[1016]: 2026/09/22 10:35:13 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4705worker1 # [ 44.601491] nix-grpc-daemon[784]: ts=2026-09-22T10:35:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=1 nar_bytes_in=6244706worker1 # [ 44.606475] nix-grpc-daemon[784]: ts=2026-09-22T10:35:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434624707worker1 # [ 44.612328] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4708worker1 # [ 44.614491] nix-daemon[851]: reaped child process 910, status = succeeded4709worker1 # [ 44.619945] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4710client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4711client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4712client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4713client # farm-b> LOG-farm-b-t14714worker1 # [ 46.048700] nix-daemon[851]: reaped child process 924, status = succeeded4715worker1 # [ 46.209996] nix-daemon[851]: accepted connection from pid 940, user nix-grpc-daemon (trusted)4716worker1 # [ 46.222097] nix-daemon[851]: reaped child process 943, status = succeeded4717lb # [ 46.220612] niks3-server[1016]: 2026/09/22 10:35:15 INFO Received uploads request method=POST path=/api/pending_closures4718worker1 # [ 46.241450] nix-grpc-daemon[863]: time=2026-09-22T10:35:14.462Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4719worker1 # [ 46.243331] nix-grpc-daemon[863]: time=2026-09-22T10:35:14.464Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4720lb # [ 46.239143] niks3-server[1016]: 2026/09/22 10:35:15 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4721lb # [ 46.243151] niks3-server[1016]: 2026/09/22 10:35:15 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4722lb # [ 46.245228] niks3-server[1016]: 2026/09/22 10:35:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4723lb # [ 46.247215] niks3-server[1016]: 2026/09/22 10:35:15 INFO Signed narinfos id=4 count=14724worker1 # [ 46.264876] nix-grpc-daemon[863]: time=2026-09-22T10:35:14.486Z level=INFO msg="Uploading 1 narinfos"4725lb # [ 46.251046] niks3-server[1016]: 2026/09/22 10:35:15 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4726lb # [ 46.255556] niks3-server[1016]: 2026/09/22 10:35:15 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4727lb # [ 46.258414] niks3-server[1016]: 2026/09/22 10:35:15 INFO Completed upload id=44728worker1 # [ 46.275941] nix-grpc-daemon[863]: time=2026-09-22T10:35:14.497Z level=INFO msg="Upload complete. (248ms)"4729lb # [ 46.260261] niks3-server[1016]: 2026/09/22 10:35:15 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4730worker1 # [ 46.278729] nix-grpc-daemon[784]: ts=2026-09-22T10:35:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14731worker1 # [ 46.285219] nix-daemon[851]: reaped child process 922, status = succeeded4732worker2 # [ 46.306853] nix-grpc-daemon[787]: ts=2026-09-22T10:35:15Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04733worker2 # [ 46.311398] nix-grpc-daemon[787]: ts=2026-09-22T10:35:15Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14734worker1 # [ 46.335588] nix-grpc-daemon[784]: ts=2026-09-22T10:35:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434624735worker1 # [ 46.345599] nix-grpc-daemon[784]: ts=2026-09-22T10:35:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434624736worker1 # [ 46.355643] nix-grpc-daemon[863]: time=2026-09-22T10:35:14.577Z level=INFO msg="All 2 paths already cached"4737client # copying 1 paths to 'lb:50051'...4738worker1 # [ 46.363618] nix-grpc-daemon[784]: ts=2026-09-22T10:35:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434624739client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4740worker1 # [ 46.381077] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4741worker1 # [ 46.507598] nix-daemon[851]: accepted connection from pid 947, user nix-grpc-daemon (trusted)4742worker1 # [ 46.519405] nix-daemon[851]: reaped child process 950, status = succeeded4743worker1 # [ 46.645870] nix-daemon[851]: accepted connection from pid 952, user nix-grpc-daemon (trusted)4744worker1 # [ 46.657410] nix-daemon[851]: reaped child process 955, status = succeeded4745lb # [ 46.652134] niks3-server[1016]: 2026/09/22 10:35:15 INFO Received uploads request method=POST path=/api/pending_closures4746worker1 # [ 46.674210] nix-grpc-daemon[863]: time=2026-09-22T10:35:14.895Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4747worker1 # [ 46.675569] nix-grpc-daemon[863]: time=2026-09-22T10:35:14.895Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4748lb # [ 46.666809] niks3-server[1016]: 2026/09/22 10:35:15 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4749lb # [ 46.669073] niks3-server[1016]: 2026/09/22 10:35:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4750lb # [ 46.671139] niks3-server[1016]: 2026/09/22 10:35:15 INFO Signed narinfos id=5 count=14751worker1 # [ 46.687621] nix-grpc-daemon[863]: time=2026-09-22T10:35:14.908Z level=INFO msg="Uploading 1 narinfos"4752lb # [ 46.674786] niks3-server[1016]: 2026/09/22 10:35:15 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4753lb # [ 46.679255] niks3-server[1016]: 2026/09/22 10:35:15 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4754lb # [ 46.682532] niks3-server[1016]: 2026/09/22 10:35:15 INFO Completed upload id=54755worker1 # [ 46.699998] nix-grpc-daemon[863]: time=2026-09-22T10:35:14.921Z level=INFO msg="Upload complete. (313ms)"4756lb # [ 46.684267] niks3-server[1016]: 2026/09/22 10:35:15 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4757worker1 # [ 46.702128] nix-grpc-daemon[784]: ts=2026-09-22T10:35:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=1 nar_bytes_in=7524758worker1 # [ 46.705050] nix-grpc-daemon[784]: ts=2026-09-22T10:35:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434624759worker1 # [ 46.709255] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4760worker1 # [ 46.714049] nix-daemon[851]: reaped child process 945, status = succeeded4761worker1 # [ 46.718965] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4762client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4763client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4764client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4765client # farm-top> LOG-farm-top-t14766worker1 # [ 48.132383] nix-daemon[851]: reaped child process 959, status = succeeded4767worker1 # [ 48.274240] nix-daemon[851]: accepted connection from pid 975, user nix-grpc-daemon (trusted)4768worker1 # [ 48.284977] nix-daemon[851]: reaped child process 978, status = succeeded4769lb # [ 48.283360] niks3-server[1016]: 2026/09/22 10:35:17 INFO Received uploads request method=POST path=/api/pending_closures4770worker1 # [ 48.308938] nix-grpc-daemon[863]: time=2026-09-22T10:35:16.530Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4771worker1 # [ 48.311813] nix-grpc-daemon[863]: time=2026-09-22T10:35:16.531Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4772lb # [ 48.310055] niks3-server[1016]: 2026/09/22 10:35:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4773lb # [ 48.312288] niks3-server[1016]: 2026/09/22 10:35:17 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4774lb # [ 48.315525] niks3-server[1016]: 2026/09/22 10:35:17 INFO Signed narinfos id=6 count=14775worker1 # [ 48.331937] nix-grpc-daemon[863]: time=2026-09-22T10:35:16.553Z level=INFO msg="Uploading 1 narinfos"4776lb # [ 48.317247] niks3-server[1016]: 2026/09/22 10:35:17 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4777lb # [ 48.320935] niks3-server[1016]: 2026/09/22 10:35:17 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4778lb # [ 48.325685] niks3-server[1016]: 2026/09/22 10:35:17 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4779lb # [ 48.328450] niks3-server[1016]: 2026/09/22 10:35:17 INFO Completed upload id=64780worker1 # [ 48.344746] nix-grpc-daemon[863]: time=2026-09-22T10:35:16.566Z level=INFO msg="Upload complete. (217ms)"4781lb # [ 48.330632] niks3-server[1016]: 2026/09/22 10:35:17 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4782worker1 # [ 48.347651] nix-grpc-daemon[784]: ts=2026-09-22T10:35:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14783worker1 # [ 48.354305] nix-daemon[851]: reaped child process 957, status = succeeded4784client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 7.03 seconds)4785client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4786client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.19 seconds)4787client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4788client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.02 seconds)4789client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4790lb # [ 48.640447] envoy[923]: conn peer=[2001:db8:1::1]:51972 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=715 tx=364 ms=14344791lb # [ 48.641970] envoy[923]: conn peer=[2001:db8:1::1]:51988 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=365 ms=9574792lb # [ 48.643334] envoy[923]: conn peer=[2001:db8:1::1]:51992 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=114793lb # [ 48.644687] envoy[923]: conn peer=[2001:db8:1::1]:51994 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=94794lb # [ 48.646136] envoy[923]: conn peer=[2001:db8:1::1]:52002 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9567 tx=4420 ms=68294795client # copying 3 paths...4796client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4797client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4798client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4799client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.23 seconds)4800(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.52 seconds)4801subtest: repeat is answered Cached by the scheduler without building4802worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4803worker1 # [ 48.961086] nix-grpc-daemon[784]: E0000 00:00:1790073317.174651 825 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4804worker1 # [ 48.965313] nix-grpc-daemon[784]: E0000 00:00:1790073317.186821 825 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4805worker1 # [ 48.969056] nix-grpc-daemon[784]: E0000 00:00:1790073317.189442 825 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4806worker1 # [ 48.980729] nix-grpc-daemon[784]: E0000 00:00:1790073317.201696 825 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4807worker1 # finding garbage collector roots...4808worker1 # deleting specified paths...4809worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4810worker1 # deleting unused links...4811worker1 # note: hard linking is currently saving 0.0 KiB4812worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.24 seconds)4813worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4814worker2 # finding garbage collector roots...4815worker2 # deleting specified paths...4816worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4817worker2 # deleting unused links...4818worker2 # note: hard linking is currently saving 0.0 KiB4819worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.22 seconds)4820client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24821worker2 # [ 49.570437] nix-grpc-daemon[787]: ts=2026-09-22T10:35:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:521524822client # don't know how to build these paths:4823client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4824worker2 # [ 49.578782] nix-grpc-daemon[787]: ts=2026-09-22T10:35:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:521524825worker2 # [ 49.606575] nix-grpc-daemon[787]: ts=2026-09-22T10:35:18Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14826client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.28 seconds)4827client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4828client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.16 seconds)4829lb: must succeed: curl -sf localhost:9901/clusters4830lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4831lb: must succeed: curl -sf localhost:9901/clusters4832lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4833worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4834worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4835(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.10 seconds)4836subtest: inputs already in the cache are not sent to the scheduler4837lb: must succeed: curl -sf localhost:9901/clusters4838lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4839lb: must succeed: curl -sf localhost:9901/clusters4840lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4841worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4842worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.03 seconds)4843client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24844worker1 # [ 50.311293] nix-grpc-daemon[784]: ts=2026-09-22T10:35:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:434624845client # don't know how to build these paths:4846client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4847worker1 # [ 50.319696] nix-grpc-daemon[784]: ts=2026-09-22T10:35:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434624848worker1 # [ 50.328409] nix-grpc-daemon[863]: time=2026-09-22T10:35:18.549Z level=INFO msg="All 2 paths already cached"4849worker2 # [ 50.362189] nix-grpc-daemon[787]: ts=2026-09-22T10:35:19Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04850worker2 # [ 50.366727] nix-grpc-daemon[787]: ts=2026-09-22T10:35:19Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14851worker1 # [ 50.385954] nix-grpc-daemon[784]: ts=2026-09-22T10:35:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434624852worker1 # [ 50.398442] nix-grpc-daemon[784]: ts=2026-09-22T10:35:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434624853worker1 # [ 50.410764] nix-grpc-daemon[863]: time=2026-09-22T10:35:18.632Z level=INFO msg="All 2 paths already cached"4854client # copying 1 paths to 'lb:50051'...4855client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4856worker1 # [ 50.419510] nix-grpc-daemon[784]: ts=2026-09-22T10:35:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434624857worker1 # [ 50.435182] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4858worker1 # [ 50.606250] nix-daemon[851]: accepted connection from pid 995, user nix-grpc-daemon (trusted)4859worker1 # [ 50.619471] nix-daemon[851]: reaped child process 998, status = succeeded4860worker1 # [ 50.780996] nix-daemon[851]: accepted connection from pid 1000, user nix-grpc-daemon (trusted)4861worker1 # [ 50.792677] nix-daemon[851]: reaped child process 1003, status = succeeded4862lb # [ 50.793290] niks3-server[1016]: 2026/09/22 10:35:19 INFO Received uploads request method=POST path=/api/pending_closures4863worker1 # [ 50.817710] nix-grpc-daemon[863]: time=2026-09-22T10:35:19.038Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4864worker1 # [ 50.819158] nix-grpc-daemon[863]: time=2026-09-22T10:35:19.038Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4865lb # [ 50.821274] niks3-server[1016]: 2026/09/22 10:35:19 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4866lb # [ 50.829662] niks3-server[1016]: 2026/09/22 10:35:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4867lb # [ 50.833252] niks3-server[1016]: 2026/09/22 10:35:19 INFO Signed narinfos id=7 count=14868worker1 # [ 50.853272] nix-grpc-daemon[863]: time=2026-09-22T10:35:19.074Z level=INFO msg="Uploading 1 narinfos"4869lb # [ 50.838517] niks3-server[1016]: 2026/09/22 10:35:19 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4870lb # [ 50.847938] niks3-server[1016]: 2026/09/22 10:35:19 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4871lb # [ 50.854923] niks3-server[1016]: 2026/09/22 10:35:19 INFO Completed upload id=74872worker1 # [ 50.871250] nix-grpc-daemon[863]: time=2026-09-22T10:35:19.092Z level=INFO msg="Upload complete. (424ms)"4873lb # [ 50.856517] niks3-server[1016]: 2026/09/22 10:35:19 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4874worker1 # [ 50.873499] nix-grpc-daemon[784]: ts=2026-09-22T10:35:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=1 nar_bytes_in=7524875worker1 # [ 50.877141] nix-grpc-daemon[784]: ts=2026-09-22T10:35:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434624876worker1 # [ 50.884140] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4877worker1 # [ 50.887104] nix-daemon[851]: reaped child process 993, status = succeeded4878worker1 # [ 50.892130] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4879client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4880client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4881client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4882client # farm-top> LOG-farm-top-t1b4883worker1 # [ 51.718730] nix-daemon[851]: reaped child process 1007, status = succeeded4884worker1 # [ 51.879514] nix-daemon[851]: accepted connection from pid 1023, user nix-grpc-daemon (trusted)4885worker1 # [ 51.891892] nix-daemon[851]: reaped child process 1026, status = succeeded4886lb # [ 51.887635] niks3-server[1016]: 2026/09/22 10:35:20 INFO Received uploads request method=POST path=/api/pending_closures4887worker1 # [ 51.912883] nix-grpc-daemon[863]: time=2026-09-22T10:35:20.134Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4888worker1 # [ 51.914359] nix-grpc-daemon[863]: time=2026-09-22T10:35:20.135Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4889lb # [ 51.923417] niks3-server[1016]: 2026/09/22 10:35:20 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4890lb # [ 51.933439] niks3-server[1016]: 2026/09/22 10:35:20 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4891lb # [ 51.943102] niks3-server[1016]: 2026/09/22 10:35:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4892worker1 # [ 51.965340] nix-grpc-daemon[863]: time=2026-09-22T10:35:20.186Z level=INFO msg="Uploading 1 narinfos"4893lb # [ 51.949077] niks3-server[1016]: 2026/09/22 10:35:20 INFO Signed narinfos id=8 count=14894lb # [ 51.953806] niks3-server[1016]: 2026/09/22 10:35:20 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4895lb # [ 51.962756] niks3-server[1016]: 2026/09/22 10:35:20 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4896lb # [ 51.969771] niks3-server[1016]: 2026/09/22 10:35:20 INFO Completed upload id=84897worker1 # [ 51.986311] nix-grpc-daemon[863]: time=2026-09-22T10:35:20.207Z level=INFO msg="Upload complete. (285ms)"4898lb # [ 51.972304] niks3-server[1016]: 2026/09/22 10:35:20 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4899worker1 # [ 51.991863] nix-grpc-daemon[784]: ts=2026-09-22T10:35:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14900worker1 # [ 52.004953] nix-daemon[851]: reaped child process 1005, status = succeeded4901client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.94 seconds)4902client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4903client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.17 seconds)4904lb: must succeed: curl -sf localhost:9901/clusters4905lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4906lb: must succeed: curl -sf localhost:9901/clusters4907lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4908worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4909worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4910(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.39 seconds)4911subtest: outputs read back through any worker carry the cache signature4912client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4913worker1 # [ 52.539583] nix-grpc-daemon[784]: ts=2026-09-22T10:35:20Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:579004914client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4915client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4916worker1 # [ 52.584497] nix-grpc-daemon[784]: ts=2026-09-22T10:35:20Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:579004917client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.24 seconds)4918client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4919worker2 # [ 52.792270] nix-grpc-daemon[787]: ts=2026-09-22T10:35:21Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:507684920client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):4921client # /nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a4922client # /nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b4923client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4924worker2 # [ 52.835120] nix-grpc-daemon[787]: ts=2026-09-22T10:35:21Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:507684925worker2 # [ 52.843938] nix-grpc-daemon[787]: copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4926worker2 # [ 52.866732] nix-grpc-daemon[787]: copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4927worker2 # [ 52.874426] nix-grpc-daemon[787]: copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4928client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.28 seconds)4929client: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4930client # finding garbage collector roots...4931client # removing stale temporary roots file "/nix/var/nix/temproots/823"4932client # removing stale temporary roots file "/nix/var/nix/temproots/864"4933client # removing stale temporary roots file "/nix/var/nix/temproots/856"4934client # removing stale temporary roots file "/nix/var/nix/temproots/807"4935client # removing stale temporary roots file "/nix/var/nix/temproots/877"4936client # removing stale temporary roots file "/nix/var/nix/temproots/844"4937client # deleting specified paths...4938client # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4939client # deleting unused links...4940client # note: hard linking is currently saving 0.0 KiB4941worker1 # [ 53.230668] nix-grpc-daemon[784]: ts=2026-09-22T10:35:21Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:434624942client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4943client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4944worker1 # [ 53.238812] nix-grpc-daemon[784]: ts=2026-09-22T10:35:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434624945worker1 # [ 53.246299] nix-grpc-daemon[784]: copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4946worker1 # [ 53.268468] nix-grpc-daemon[863]: time=2026-09-22T10:35:21.489Z level=INFO msg="All 1 paths already cached"4947worker1 # [ 53.275546] nix-grpc-daemon[784]: ts=2026-09-22T10:35:21Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:434624948worker2 # [ 53.303659] nix-grpc-daemon[787]: ts=2026-09-22T10:35:22Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:521524949client # copying 1 paths...4950worker2 # [ 53.318747] nix-grpc-daemon[787]: ts=2026-09-22T10:35:22Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:521524951client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'grpc://lb:50051'...4952worker1 # [ 53.322424] nix-grpc-daemon[784]: ts=2026-09-22T10:35:21Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:434624953worker1 # [ 53.326910] nix-grpc-daemon[784]: ts=2026-09-22T10:35:21Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=1 nar_bytes_out=2324954client: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.52 seconds)4955(finished: subtest: outputs read back through any worker carry the cache signature, in 1.04 seconds)4956subtest: two clients wanting the same drv build it once4957client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4958client # [ 53.437895] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4959client # Running as unit: dedup1.service; invocation ID: 8e48cde9ddac4a5ca06f66de6f59e0104960client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.04 seconds)4961client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4962client # [ 53.610225] nix[928]: don't know how to build these paths:4963client # [ 53.611610] nix[928]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4964worker2 # [ 53.632556] nix-grpc-daemon[787]: ts=2026-09-22T10:35:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:521524965worker2 # [ 53.643908] nix-grpc-daemon[787]: ts=2026-09-22T10:35:22Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04966worker2 # [ 53.648657] nix-grpc-daemon[787]: ts=2026-09-22T10:35:22Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=14967worker2 # [ 53.688172] nix-grpc-daemon[787]: ts=2026-09-22T10:35:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:521524968worker2 # [ 53.703671] nix-grpc-daemon[787]: ts=2026-09-22T10:35:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:521524969client # [ 53.690254] nix[928]: copying 1 paths to 'lb:50051'...4970worker2 # [ 53.714499] nix-grpc-daemon[787]: ts=2026-09-22T10:35:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:521524971client # [ 53.692960] nix[928]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4972worker2 # [ 53.727877] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4973worker2 # [ 53.745930] nix-grpc-daemon[787]: ts=2026-09-22T10:35:22Z level=info event=niks3_push_started pid=9284974worker2 # [ 54.001453] nix-grpc-daemon[928]: time=2026-09-22T10:35:23.025Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""4975worker2 # [ 54.152389] nix-daemon[873]: accepted connection from pid 941, user nix-grpc-daemon (trusted)4976worker2 # [ 54.168614] nix-daemon[873]: reaped child process 944, status = succeeded4977worker2 # [ 54.298853] nix-daemon[873]: accepted connection from pid 946, user nix-grpc-daemon (trusted)4978worker2 # [ 54.310930] nix-daemon[873]: reaped child process 949, status = succeeded4979lb # [ 54.286470] niks3-server[1016]: 2026/09/22 10:35:23 INFO Received uploads request method=POST path=/api/pending_closures4980worker2 # [ 54.332151] nix-grpc-daemon[928]: time=2026-09-22T10:35:23.359Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4981worker2 # [ 54.334724] nix-grpc-daemon[928]: time=2026-09-22T10:35:23.362Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4982lb # [ 54.322113] niks3-server[1016]: 2026/09/22 10:35:23 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4983lb # [ 54.333983] niks3-server[1016]: 2026/09/22 10:35:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4984lb # [ 54.339365] niks3-server[1016]: 2026/09/22 10:35:23 INFO Signed narinfos id=9 count=14985worker2 # [ 54.378698] nix-grpc-daemon[928]: time=2026-09-22T10:35:23.406Z level=INFO msg="Uploading 1 narinfos"4986lb # [ 54.349915] niks3-server[1016]: 2026/09/22 10:35:23 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4987lb # [ 54.359385] niks3-server[1016]: 2026/09/22 10:35:23 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4988worker2 # [ 54.402287] nix-grpc-daemon[928]: time=2026-09-22T10:35:23.429Z level=INFO msg="Upload complete. (391ms)"4989lb # [ 54.366075] niks3-server[1016]: 2026/09/22 10:35:23 INFO Completed upload id=94990worker2 # [ 54.405344] nix-grpc-daemon[787]: ts=2026-09-22T10:35:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=0 paths=1 nar_bytes_in=4324991lb # [ 54.368685] niks3-server[1016]: 2026/09/22 10:35:23 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4992worker2 # [ 54.411966] nix-grpc-daemon[787]: ts=2026-09-22T10:35:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:521524993worker2 # [ 54.413659] nix-daemon[873]: reaped child process 926, status = succeeded4994worker2 # [ 54.416503] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4995worker2 # [ 54.422979] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4996client # [ 54.398901] nix[928]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4997client # [ 54.500858] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4998client # Running as unit: dedup2.service; invocation ID: 6650a6ab299a41b89740429b030d1b984999client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.07 seconds)5000worker1 # [ 54.755185] nix-grpc-daemon[784]: ts=2026-09-22T10:35:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:434625001client # [ 54.757757] nix[944]: don't know how to build these paths:5002client # [ 54.761050] nix[944]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5003worker2 # [ 54.795223] nix-grpc-daemon[787]: ts=2026-09-22T10:35:23Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05004client # [ 54.777555] nix[944]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5005worker2 # [ 54.800768] nix-grpc-daemon[787]: ts=2026-09-22T10:35:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:521525006client: must succeed: sleep 25007client: (finished: must succeed: sleep 2, in 2.05 seconds)5008client: waiting for success: ! systemctl is-active dedup1 dedup25009worker2 # [ 58.039756] nix-daemon[873]: reaped child process 953, status = succeeded5010worker2 # [ 58.159904] nix-daemon[873]: accepted connection from pid 985, user nix-grpc-daemon (trusted)5011worker2 # [ 58.172220] nix-daemon[873]: reaped child process 988, status = succeeded5012lb # [ 58.147593] niks3-server[1016]: 2026/09/22 10:35:27 INFO Received uploads request method=POST path=/api/pending_closures5013worker2 # [ 58.195596] nix-grpc-daemon[928]: time=2026-09-22T10:35:27.223Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5014worker2 # [ 58.197167] nix-grpc-daemon[928]: time=2026-09-22T10:35:27.224Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"5015lb # [ 58.194529] niks3-server[1016]: 2026/09/22 10:35:27 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5016lb # [ 58.204489] niks3-server[1016]: 2026/09/22 10:35:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign5017worker2 # [ 58.247947] nix-grpc-daemon[928]: time=2026-09-22T10:35:27.275Z level=INFO msg="Uploading 1 narinfos"5018lb # [ 58.210964] niks3-server[1016]: 2026/09/22 10:35:27 INFO Signed narinfos id=10 count=15019lb # [ 58.219969] niks3-server[1016]: 2026/09/22 10:35:27 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst5020lb # [ 58.229147] niks3-server[1016]: 2026/09/22 10:35:27 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete5021lb # [ 58.237264] niks3-server[1016]: 2026/09/22 10:35:27 INFO Completed upload id=105022lb # [ 58.238623] niks3-server[1016]: 2026/09/22 10:35:27 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls5023worker2 # [ 58.275102] nix-grpc-daemon[928]: time=2026-09-22T10:35:27.302Z level=INFO msg="Upload complete. (240ms)"5024lb # [ 58.241605] niks3-server[1016]: 2026/09/22 10:35:27 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo5025worker2 # [ 58.280867] nix-grpc-daemon[787]: ts=2026-09-22T10:35:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=15026worker2 # [ 58.287250] nix-grpc-daemon[787]: ts=2026-09-22T10:35:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=15027worker2 # [ 58.298289] nix-daemon[873]: reaped child process 951, status = succeeded5028client # [ 58.305408] systemd[1]: dedup1.service: Deactivated successfully.5029client # [ 58.308957] systemd[1]: dedup1.service: Consumed 116ms CPU time over 4.870s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.5K outgoing IP traffic.5030client # [ 58.313498] systemd[1]: dedup2.service: Deactivated successfully.5031client # [ 58.316996] systemd[1]: dedup2.service: Consumed 143ms CPU time over 3.815s wall clock time, 37.1M memory peak, 5.9K incoming IP traffic, 5.7K outgoing IP traffic.5032lb # [ 58.640070] envoy[923]: conn peer=[2001:db8:1::1]:42234 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=505033lb # [ 58.641703] envoy[923]: conn peer=[2001:db8:1::1]:42236 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4169 tx=1895 ms=16895034lb # [ 58.643316] envoy[923]: conn peer=[2001:db8:1::1]:33104 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=969 tx=2185 ms=1565035lb # [ 58.644865] envoy[923]: conn peer=[2001:db8:1::1]:33114 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=409 ms=655036lb # [ 58.646412] envoy[923]: conn peer=[2001:db8:1::1]:33120 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2998 tx=1563 ms=46675037lb # [ 58.647969] envoy[923]: conn peer=[2001:db8:1::1]:33124 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1427 tx=1197 ms=35345038client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.11 seconds)5039client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }5040client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)5041worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5042worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5043worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5044worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5045worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5046worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5047worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5048worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5049(finished: subtest: two clients wanting the same drv build it once, in 5.90 seconds)5050subtest: the first client leaving does not fail the second5051client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share5052client # [ 59.347451] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share.5053client # Running as unit: share1.service; invocation ID: fe602764e9e74c0fa22638f9760c73ac5054client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share, in 0.05 seconds)5055client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share5056worker2 # [ 59.569396] nix-grpc-daemon[787]: ts=2026-09-22T10:35:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:521525057client # [ 59.552323] nix[984]: don't know how to build these paths:5058client # [ 59.553165] nix[984]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5059worker2 # [ 59.583262] nix-grpc-daemon[787]: ts=2026-09-22T10:35:28Z level=debug event=want client=6 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05060worker2 # [ 59.587678] nix-grpc-daemon[787]: ts=2026-09-22T10:35:28Z level=debug event=assigned drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv worker=192.168.1.3:50051 clients=15061worker1 # [ 59.608779] nix-grpc-daemon[784]: ts=2026-09-22T10:35:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434625062worker1 # [ 59.622624] nix-grpc-daemon[784]: ts=2026-09-22T10:35:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434625063client # [ 59.629310] nix[984]: copying 1 paths to 'lb:50051'...5064worker1 # [ 59.634207] nix-grpc-daemon[784]: ts=2026-09-22T10:35:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434625065client # [ 59.631914] nix[984]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' to 'lb:50051'...5066worker1 # [ 59.647117] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5067worker1 # [ 59.834129] nix-daemon[851]: accepted connection from pid 1052, user nix-grpc-daemon (trusted)5068worker1 # [ 59.847312] nix-daemon[851]: reaped child process 1055, status = succeeded5069worker1 # [ 60.009270] nix-daemon[851]: accepted connection from pid 1057, user nix-grpc-daemon (trusted)5070worker1 # [ 60.021091] nix-daemon[851]: reaped child process 1060, status = succeeded5071lb # [ 60.024939] niks3-server[1016]: 2026/09/22 10:35:28 INFO Received uploads request method=POST path=/api/pending_closures5072worker1 # [ 60.048271] nix-grpc-daemon[863]: time=2026-09-22T10:35:28.269Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5073worker1 # [ 60.050092] nix-grpc-daemon[863]: time=2026-09-22T10:35:28.271Z level=INFO msg="Uploading j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv (432B)"5074lb # [ 60.053304] niks3-server[1016]: 2026/09/22 10:35:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign5075lb # [ 60.055395] niks3-server[1016]: 2026/09/22 10:35:28 INFO Signed narinfos id=11 count=15076worker1 # [ 60.076381] nix-grpc-daemon[863]: time=2026-09-22T10:35:28.297Z level=INFO msg="Uploading 1 narinfos"5077lb # [ 60.063465] niks3-server[1016]: 2026/09/22 10:35:28 INFO Registered completed upload object_key=nar/0s0b0gfhn36p0aylk1qa64cgw01bi7a8ca93iv4zg2q7wqcjcqhs.nar.zst5078lb # [ 60.066227] niks3-server[1016]: 2026/09/22 10:35:28 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.ls5079lb # [ 60.070250] niks3-server[1016]: 2026/09/22 10:35:28 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete5080lb # [ 60.073150] niks3-server[1016]: 2026/09/22 10:35:28 INFO Completed upload id=115081worker1 # [ 60.090524] nix-grpc-daemon[863]: time=2026-09-22T10:35:28.312Z level=INFO msg="Upload complete. (431ms)"5082lb # [ 60.075571] niks3-server[1016]: 2026/09/22 10:35:28 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.narinfo5083worker1 # [ 60.092642] nix-grpc-daemon[784]: ts=2026-09-22T10:35:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=1 nar_bytes_in=4325084worker1 # [ 60.096564] nix-grpc-daemon[784]: ts=2026-09-22T10:35:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434625085worker1 # [ 60.102730] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5086worker1 # [ 60.104979] nix-daemon[851]: reaped child process 1050, status = succeeded5087worker1 # [ 60.110243] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5088client # [ 60.106224] nix[984]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5089client # [ 60.411414] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share.5090client # Running as unit: share2.service; invocation ID: 2a59157fe4f248188688d7c57be083675091client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share, in 1.08 seconds)5092worker2 # [ 60.654831] nix-grpc-daemon[787]: ts=2026-09-22T10:35:29Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:521525093client # [ 60.637103] nix[1000]: don't know how to build these paths:5094client # [ 60.637944] nix[1000]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5095worker2 # [ 60.669925] nix-grpc-daemon[787]: ts=2026-09-22T10:35:29Z level=debug event=want client=7 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05096worker1 # [ 60.653592] nix-grpc-daemon[784]: ts=2026-09-22T10:35:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434625097client # [ 60.655194] nix[1000]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5098client: must succeed: sleep 2; systemctl kill -s INT share15099client # [ 63.734718] systemd[1]: share1.service: Sent signal SIGINT to main process 984 (nix) on client request.5100client # [ 63.742782] nix[984]: shutting downj2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5101client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.07 seconds)5102client: waiting for success: ! systemctl is-active share15103client # [ 63.751430] nix[984]: error: interrupted by the user5104client # [ 63.794599] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE5105client # [ 63.795846] systemd[1]: share1.service: Failed with result 'exit-code'.5106client # [ 63.798412] systemd[1]: share1.service: Consumed 116ms CPU time over 4.447s wall clock time, 41.4M memory peak, 6.2K incoming IP traffic, 8.3K outgoing IP traffic.5107client: (finished: waiting for success: ! systemctl is-active share1, in 0.06 seconds)5108client: waiting for success: ! systemctl is-active share25109worker1 # [ 64.136663] nix-daemon[851]: reaped child process 1064, status = succeeded5110worker1 # [ 64.293612] nix-daemon[851]: accepted connection from pid 1098, user nix-grpc-daemon (trusted)5111worker1 # [ 64.305275] nix-daemon[851]: reaped child process 1101, status = succeeded5112lb # [ 64.304489] niks3-server[1016]: 2026/09/22 10:35:33 INFO Received uploads request method=POST path=/api/pending_closures5113worker1 # [ 64.329287] nix-grpc-daemon[863]: time=2026-09-22T10:35:32.549Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5114worker1 # [ 64.330665] nix-grpc-daemon[863]: time=2026-09-22T10:35:32.549Z level=INFO msg="Uploading b9lg6znd0rinl36gy6ixq1dcj6b8bhhw-slow-share (120B)"5115lb # [ 64.349291] niks3-server[1016]: 2026/09/22 10:35:33 INFO Registered completed upload object_key=log/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5116lb # [ 64.354532] niks3-server[1016]: 2026/09/22 10:35:33 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst5117lb # [ 64.366456] niks3-server[1016]: 2026/09/22 10:35:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign5118lb # [ 64.371495] niks3-server[1016]: 2026/09/22 10:35:33 INFO Signed narinfos id=12 count=15119worker1 # [ 64.390732] nix-grpc-daemon[863]: time=2026-09-22T10:35:32.611Z level=INFO msg="Uploading 1 narinfos"5120lb # [ 64.376881] niks3-server[1016]: 2026/09/22 10:35:33 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.ls5121lb # [ 64.385230] niks3-server[1016]: 2026/09/22 10:35:33 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete5122lb # [ 64.391103] niks3-server[1016]: 2026/09/22 10:35:33 INFO Completed upload id=125123worker1 # [ 64.407487] nix-grpc-daemon[863]: time=2026-09-22T10:35:32.628Z level=INFO msg="Upload complete. (285ms)"5124lb # [ 64.393727] niks3-server[1016]: 2026/09/22 10:35:33 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.narinfo5125worker1 # [ 64.413820] nix-grpc-daemon[784]: ts=2026-09-22T10:35:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=3 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=6 outputs=15126worker1 # [ 64.420431] nix-grpc-daemon[784]: ts=2026-09-22T10:35:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=4 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=6 outputs=15127worker1 # [ 64.432436] nix-daemon[851]: reaped child process 1062, status = succeeded5128client # [ 64.488603] systemd[1]: share2.service: Deactivated successfully.5129client # [ 64.492347] systemd[1]: share2.service: Consumed 116ms CPU time over 4.080s wall clock time, 33.3M memory peak, 6.1K incoming IP traffic, 5.7K outgoing IP traffic.5130client: (finished: waiting for success: ! systemctl is-active share2, in 1.10 seconds)5131client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }5132client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.04 seconds)5133(finished: subtest: the first client leaving does not fail the second, in 5.88 seconds)5134subtest: build hook: nix-daemon with builders = grpc://lb5135client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log5136client # [ 65.328773] systemd[1]: Started Nix Daemon.5137client # [ 65.479258] nix-daemon[1047]: accepted connection from pid 1044, user root (trusted)5138worker2 # [ 65.681828] nix-grpc-daemon[787]: ts=2026-09-22T10:35:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:521525139worker2 # [ 65.687301] nix-grpc-daemon[787]: ts=2026-09-22T10:35:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:521525140worker2 # [ 65.698192] nix-grpc-daemon[787]: ts=2026-09-22T10:35:34Z level=debug event=want client=8 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05141worker2 # [ 65.704499] nix-grpc-daemon[787]: ts=2026-09-22T10:35:34Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.4:50051 clients=15142worker2 # [ 65.709299] nix-grpc-daemon[787]: ts=2026-09-22T10:35:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:521525143worker2 # [ 65.720109] nix-grpc-daemon[787]: ts=2026-09-22T10:35:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:521525144worker2 # [ 65.731483] nix-grpc-daemon[787]: ts=2026-09-22T10:35:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:521525145worker2 # [ 65.742074] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5146worker2 # [ 65.901571] nix-daemon[873]: accepted connection from pid 1023, user nix-grpc-daemon (trusted)5147worker2 # [ 65.915558] nix-daemon[873]: reaped child process 1026, status = succeeded5148worker2 # [ 66.066098] nix-daemon[873]: accepted connection from pid 1028, user nix-grpc-daemon (trusted)5149worker2 # [ 66.076813] nix-daemon[873]: reaped child process 1031, status = succeeded5150lb # [ 66.057076] niks3-server[1016]: 2026/09/22 10:35:34 INFO Received uploads request method=POST path=/api/pending_closures5151worker2 # [ 66.230238] nix-grpc-daemon[928]: time=2026-09-22T10:35:35.257Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5152worker2 # [ 66.235896] nix-grpc-daemon[928]: time=2026-09-22T10:35:35.261Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5153lb # [ 66.209091] niks3-server[1016]: 2026/09/22 10:35:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign5154lb # [ 66.211271] niks3-server[1016]: 2026/09/22 10:35:35 INFO Signed narinfos id=13 count=15155worker2 # [ 66.248479] nix-grpc-daemon[928]: time=2026-09-22T10:35:35.275Z level=INFO msg="Uploading 1 narinfos"5156lb # [ 66.216931] niks3-server[1016]: 2026/09/22 10:35:35 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5157lb # [ 66.220877] niks3-server[1016]: 2026/09/22 10:35:35 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete5158lb # [ 66.223984] niks3-server[1016]: 2026/09/22 10:35:35 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5159lb # [ 66.226309] niks3-server[1016]: 2026/09/22 10:35:35 INFO Completed upload id=135160worker2 # [ 66.262603] nix-grpc-daemon[928]: time=2026-09-22T10:35:35.290Z level=INFO msg="Upload complete. (509ms)"5161lb # [ 66.229698] niks3-server[1016]: 2026/09/22 10:35:35 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo5162worker2 # [ 66.267515] nix-grpc-daemon[787]: ts=2026-09-22T10:35:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=0 paths=1 nar_bytes_in=5125163worker2 # [ 66.274110] nix-grpc-daemon[787]: ts=2026-09-22T10:35:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:521525164worker2 # [ 66.283586] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5165worker2 # [ 66.296436] nix-daemon[873]: reaped child process 1021, status = succeeded5166worker2 # [ 66.303842] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5167worker2 # [ 67.523253] nix-daemon[873]: reaped child process 1035, status = succeeded5168worker2 # [ 67.660395] nix-daemon[873]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5169worker2 # [ 67.674128] nix-daemon[873]: reaped child process 1054, status = succeeded5170lb # [ 67.653494] niks3-server[1016]: 2026/09/22 10:35:36 INFO Received uploads request method=POST path=/api/pending_closures5171worker2 # [ 67.697173] nix-grpc-daemon[928]: time=2026-09-22T10:35:36.724Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5172worker2 # [ 67.699259] nix-grpc-daemon[928]: time=2026-09-22T10:35:36.726Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5173lb # [ 67.685107] niks3-server[1016]: 2026/09/22 10:35:36 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5174lb # [ 67.693208] niks3-server[1016]: 2026/09/22 10:35:36 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5175lb # [ 67.705399] niks3-server[1016]: 2026/09/22 10:35:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign5176lb # [ 67.709577] niks3-server[1016]: 2026/09/22 10:35:36 INFO Signed narinfos id=14 count=15177worker2 # [ 67.749328] nix-grpc-daemon[928]: time=2026-09-22T10:35:36.776Z level=INFO msg="Uploading 1 narinfos"5178lb # [ 67.716119] niks3-server[1016]: 2026/09/22 10:35:36 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5179lb # [ 67.724902] niks3-server[1016]: 2026/09/22 10:35:36 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete5180lb # [ 67.731098] niks3-server[1016]: 2026/09/22 10:35:36 INFO Completed upload id=145181worker2 # [ 67.767343] nix-grpc-daemon[928]: time=2026-09-22T10:35:36.795Z level=INFO msg="Upload complete. (252ms)"5182lb # [ 67.733504] niks3-server[1016]: 2026/09/22 10:35:36 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5183worker2 # [ 67.774744] nix-grpc-daemon[787]: ts=2026-09-22T10:35:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=7 outputs=15184worker2 # [ 67.782313] nix-grpc-daemon[787]: ts=2026-09-22T10:35:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:521525185worker2 # [ 67.789623] nix-grpc-daemon[787]: ts=2026-09-22T10:35:36Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:521525186worker2 # [ 67.795577] nix-grpc-daemon[787]: ts=2026-09-22T10:35:36Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=0 paths=1 nar_bytes_out=1285187worker2 # [ 67.798336] nix-daemon[873]: reaped child process 1033, status = succeeded5188worker2 # [ 68.061728] nix-grpc-daemon[787]: ts=2026-09-22T10:35:37Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:521525189worker1 # [ 68.050917] nix-grpc-daemon[784]: ts=2026-09-22T10:35:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434625190worker2 # [ 68.093811] nix-grpc-daemon[787]: ts=2026-09-22T10:35:37Z level=debug event=want client=9 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05191worker2 # [ 68.098751] nix-grpc-daemon[787]: ts=2026-09-22T10:35:37Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.4:50051 clients=15192worker2 # [ 68.105341] nix-grpc-daemon[787]: ts=2026-09-22T10:35:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:521525193worker2 # [ 68.116320] nix-grpc-daemon[787]: ts=2026-09-22T10:35:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:521525194worker2 # [ 68.129616] nix-grpc-daemon[928]: time=2026-09-22T10:35:37.157Z level=INFO msg="All 1 paths already cached"5195worker2 # [ 68.137167] nix-grpc-daemon[787]: ts=2026-09-22T10:35:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:521525196worker2 # [ 68.145527] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5197worker2 # [ 68.300349] nix-daemon[873]: accepted connection from pid 1060, user nix-grpc-daemon (trusted)5198worker2 # [ 68.315379] nix-daemon[873]: reaped child process 1063, status = succeeded5199worker2 # [ 68.473647] nix-daemon[873]: accepted connection from pid 1065, user nix-grpc-daemon (trusted)5200worker2 # [ 68.487702] nix-daemon[873]: reaped child process 1068, status = succeeded5201lb # [ 68.467394] niks3-server[1016]: 2026/09/22 10:35:37 INFO Received uploads request method=POST path=/api/pending_closures5202worker2 # [ 68.512376] nix-grpc-daemon[928]: time=2026-09-22T10:35:37.540Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5203worker2 # [ 68.514359] nix-grpc-daemon[928]: time=2026-09-22T10:35:37.541Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5204lb # [ 68.495353] niks3-server[1016]: 2026/09/22 10:35:37 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5205lb # [ 68.507582] niks3-server[1016]: 2026/09/22 10:35:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5206worker2 # [ 68.550482] nix-grpc-daemon[928]: time=2026-09-22T10:35:37.577Z level=INFO msg="Uploading 1 narinfos"5207lb # [ 68.513880] niks3-server[1016]: 2026/09/22 10:35:37 INFO Signed narinfos id=15 count=15208lb # [ 68.518506] niks3-server[1016]: 2026/09/22 10:35:37 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5209lb # [ 68.530781] niks3-server[1016]: 2026/09/22 10:35:37 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5210worker2 # [ 68.573782] nix-grpc-daemon[928]: time=2026-09-22T10:35:37.601Z level=INFO msg="Upload complete. (411ms)"5211lb # [ 68.537286] niks3-server[1016]: 2026/09/22 10:35:37 INFO Completed upload id=155212lb # [ 68.539491] niks3-server[1016]: 2026/09/22 10:35:37 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5213worker2 # [ 68.576642] nix-grpc-daemon[787]: ts=2026-09-22T10:35:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=0 paths=1 nar_bytes_in=6245214worker2 # [ 68.582789] nix-grpc-daemon[787]: ts=2026-09-22T10:35:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:521525215worker2 # [ 68.592676] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5216worker2 # [ 68.597747] nix-daemon[873]: reaped child process 1058, status = succeeded5217worker2 # [ 68.601392] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5218lb # [ 68.640732] envoy[923]: conn peer=[2001:db8:1::1]:33126 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2995 tx=788 ms=42195219lb # [ 68.642472] envoy[923]: conn peer=[2001:db8:1::1]:33128 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1453 tx=1194 ms=38325220lb # [ 68.644226] envoy[923]: conn peer=[2001:db8:1::1]:38986 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3429 tx=1965 ms=21665221lb # [ 68.645942] envoy[923]: conn peer=[2001:db8:1::1]:38988 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=333 ms=545222worker2 # [ 69.625418] nix-daemon[873]: reaped child process 1072, status = succeeded5223worker2 # [ 69.772638] nix-daemon[873]: accepted connection from pid 1088, user nix-grpc-daemon (trusted)5224worker2 # [ 69.786228] nix-daemon[873]: reaped child process 1091, status = succeeded5225lb # [ 69.765597] niks3-server[1016]: 2026/09/22 10:35:38 INFO Received uploads request method=POST path=/api/pending_closures5226worker2 # [ 69.806688] nix-grpc-daemon[928]: time=2026-09-22T10:35:38.834Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5227worker2 # [ 69.808455] nix-grpc-daemon[928]: time=2026-09-22T10:35:38.835Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5228lb # [ 69.788526] niks3-server[1016]: 2026/09/22 10:35:38 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5229lb # [ 69.793224] niks3-server[1016]: 2026/09/22 10:35:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5230lb # [ 69.797096] niks3-server[1016]: 2026/09/22 10:35:38 INFO Signed narinfos id=16 count=15231worker2 # [ 69.835699] nix-grpc-daemon[928]: time=2026-09-22T10:35:38.863Z level=INFO msg="Uploading 1 narinfos"5232lb # [ 69.800980] niks3-server[1016]: 2026/09/22 10:35:38 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5233lb # [ 69.808757] niks3-server[1016]: 2026/09/22 10:35:38 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5234lb # [ 69.812345] niks3-server[1016]: 2026/09/22 10:35:38 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5235lb # [ 69.816896] niks3-server[1016]: 2026/09/22 10:35:38 INFO Completed upload id=165236worker2 # [ 69.854402] nix-grpc-daemon[928]: time=2026-09-22T10:35:38.882Z level=INFO msg="Upload complete. (246ms)"5237lb # [ 69.818374] niks3-server[1016]: 2026/09/22 10:35:38 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5238worker2 # [ 69.858418] nix-grpc-daemon[787]: ts=2026-09-22T10:35:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=8 outputs=15239worker1 # [ 69.841491] nix-grpc-daemon[784]: ts=2026-09-22T10:35:38Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:434625240worker2 # [ 69.866618] nix-daemon[873]: reaped child process 1070, status = succeeded5241worker1 # [ 69.847152] nix-grpc-daemon[784]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5242worker1 # [ 69.852886] nix-grpc-daemon[784]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...5243worker1 # [ 69.867891] nix-grpc-daemon[784]: ts=2026-09-22T10:35:38Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:434625244worker1 # [ 69.870090] nix-grpc-daemon[784]: ts=2026-09-22T10:35:38Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=1 nar_bytes_out=1765245worker1 # [ 70.132441] nix-grpc-daemon[784]: ts=2026-09-22T10:35:38Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:434625246worker1 # [ 70.138940] nix-grpc-daemon[784]: ts=2026-09-22T10:35:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434625247worker1 # [ 70.144938] nix-grpc-daemon[863]: time=2026-09-22T10:35:38.364Z level=INFO msg="All 2 paths already cached"5248worker2 # [ 70.173192] nix-grpc-daemon[787]: ts=2026-09-22T10:35:39Z level=debug event=want client=10 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05249worker2 # [ 70.177189] nix-grpc-daemon[787]: ts=2026-09-22T10:35:39Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.4:50051 clients=15250worker2 # [ 70.186327] nix-grpc-daemon[787]: ts=2026-09-22T10:35:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:521525251worker2 # [ 70.196496] nix-grpc-daemon[787]: ts=2026-09-22T10:35:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:521525252worker2 # [ 70.209219] nix-grpc-daemon[928]: time=2026-09-22T10:35:39.236Z level=INFO msg="All 2 paths already cached"5253worker2 # [ 70.216985] nix-grpc-daemon[787]: ts=2026-09-22T10:35:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:521525254worker2 # [ 70.225298] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5255worker2 # [ 70.387121] nix-daemon[873]: accepted connection from pid 1097, user nix-grpc-daemon (trusted)5256worker2 # [ 70.400304] nix-daemon[873]: reaped child process 1100, status = succeeded5257worker2 # [ 70.552389] nix-daemon[873]: accepted connection from pid 1102, user nix-grpc-daemon (trusted)5258worker2 # [ 70.565790] nix-daemon[873]: reaped child process 1105, status = succeeded5259lb # [ 70.545761] niks3-server[1016]: 2026/09/22 10:35:39 INFO Received uploads request method=POST path=/api/pending_closures5260worker2 # [ 70.590464] nix-grpc-daemon[928]: time=2026-09-22T10:35:39.617Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5261worker2 # [ 70.591957] nix-grpc-daemon[928]: time=2026-09-22T10:35:39.617Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5262lb # [ 70.570567] niks3-server[1016]: 2026/09/22 10:35:39 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5263lb # [ 70.583945] niks3-server[1016]: 2026/09/22 10:35:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5264worker2 # [ 70.624865] nix-grpc-daemon[928]: time=2026-09-22T10:35:39.652Z level=INFO msg="Uploading 1 narinfos"5265lb # [ 70.588440] niks3-server[1016]: 2026/09/22 10:35:39 INFO Signed narinfos id=17 count=15266lb # [ 70.600084] niks3-server[1016]: 2026/09/22 10:35:39 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5267lb # [ 70.603389] niks3-server[1016]: 2026/09/22 10:35:39 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5268lb # [ 70.610630] niks3-server[1016]: 2026/09/22 10:35:39 INFO Completed upload id=175269worker2 # [ 70.647096] nix-grpc-daemon[928]: time=2026-09-22T10:35:39.674Z level=INFO msg="Upload complete. (408ms)"5270lb # [ 70.612177] niks3-server[1016]: 2026/09/22 10:35:39 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5271worker2 # [ 70.649222] nix-grpc-daemon[787]: ts=2026-09-22T10:35:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=0 paths=1 nar_bytes_in=7525272worker2 # [ 70.653277] nix-grpc-daemon[787]: ts=2026-09-22T10:35:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:521525273worker2 # [ 70.659937] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5274worker2 # [ 70.662330] nix-daemon[873]: reaped child process 1095, status = succeeded5275worker2 # [ 70.667797] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5276worker2 # [ 71.694632] nix-daemon[873]: reaped child process 1109, status = succeeded5277worker2 # [ 71.836209] nix-daemon[873]: accepted connection from pid 1125, user nix-grpc-daemon (trusted)5278worker2 # [ 71.849150] nix-daemon[873]: reaped child process 1128, status = succeeded5279lb # [ 71.828364] niks3-server[1016]: 2026/09/22 10:35:40 INFO Received uploads request method=POST path=/api/pending_closures5280worker2 # [ 71.874101] nix-grpc-daemon[928]: time=2026-09-22T10:35:40.901Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5281worker2 # [ 71.875585] nix-grpc-daemon[928]: time=2026-09-22T10:35:40.903Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5282lb # [ 71.866771] niks3-server[1016]: 2026/09/22 10:35:40 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5283lb # [ 71.878346] niks3-server[1016]: 2026/09/22 10:35:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5284lb # [ 71.881658] niks3-server[1016]: 2026/09/22 10:35:40 INFO Signed narinfos id=18 count=15285worker2 # [ 71.920950] nix-grpc-daemon[928]: time=2026-09-22T10:35:40.948Z level=INFO msg="Uploading 1 narinfos"5286lb # [ 71.888761] niks3-server[1016]: 2026/09/22 10:35:40 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5287lb # [ 71.893459] niks3-server[1016]: 2026/09/22 10:35:40 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5288lb # [ 71.900261] niks3-server[1016]: 2026/09/22 10:35:40 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5289lb # [ 71.903314] niks3-server[1016]: 2026/09/22 10:35:40 INFO Completed upload id=185290worker2 # [ 71.939569] nix-grpc-daemon[928]: time=2026-09-22T10:35:40.967Z level=INFO msg="Upload complete. (264ms)"5291lb # [ 71.906706] niks3-server[1016]: 2026/09/22 10:35:40 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5292worker2 # [ 71.944448] nix-grpc-daemon[787]: ts=2026-09-22T10:35:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=9 outputs=15293worker2 # [ 71.956937] nix-daemon[873]: reaped child process 1107, status = succeeded5294worker1 # [ 71.972424] nix-grpc-daemon[784]: ts=2026-09-22T10:35:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:434625295worker1 # [ 71.984335] nix-grpc-daemon[784]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...5296worker1 # [ 72.013143] nix-grpc-daemon[784]: ts=2026-09-22T10:35:40Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:434625297worker1 # [ 72.017455] nix-grpc-daemon[784]: ts=2026-09-22T10:35:40Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=1 nar_bytes_out=2325298client # [ 72.124351] nix-daemon[1047]: reaped child process 1049, status = succeeded5299client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.96 seconds)5300client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5301client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.03 seconds)5302client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5303client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.03 seconds)5304client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5305client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)5306(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.05 seconds)5307subtest: an input only one worker has still reaches the builder5308client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only5309client # this derivation will be built:5310client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5311client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5312client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.37 seconds)5313client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5314client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.17 seconds)5315worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5316worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.22 seconds)5317worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5318worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)5319client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&25320client # [ 73.199257] nix-daemon[1047]: accepted connection from pid 1130, user root (trusted)5321client # this derivation will be built:5322client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5323worker2 # [ 73.406658] nix-grpc-daemon[787]: ts=2026-09-22T10:35:42Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:521525324client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5325worker1 # [ 73.391235] nix-grpc-daemon[784]: ts=2026-09-22T10:35:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434625326worker1 # [ 73.562525] nix-daemon[851]: accepted connection from pid 1111, user nix-grpc-daemon (trusted)5327worker1 # [ 73.573116] nix-daemon[851]: reaped child process 1114, status = succeeded5328lb # [ 73.573450] niks3-server[1016]: 2026/09/22 10:35:42 INFO Received uploads request method=POST path=/api/pending_closures5329worker1 # [ 73.599843] nix-grpc-daemon[863]: time=2026-09-22T10:35:41.820Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5330worker1 # [ 73.601212] nix-grpc-daemon[863]: time=2026-09-22T10:35:41.820Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5331lb # [ 73.605073] niks3-server[1016]: 2026/09/22 10:35:42 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5332lb # [ 73.614387] niks3-server[1016]: 2026/09/22 10:35:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5333worker1 # [ 73.636230] nix-grpc-daemon[863]: time=2026-09-22T10:35:41.857Z level=INFO msg="Uploading 1 narinfos"5334lb # [ 73.619732] niks3-server[1016]: 2026/09/22 10:35:42 INFO Signed narinfos id=19 count=15335lb # [ 73.626112] niks3-server[1016]: 2026/09/22 10:35:42 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5336lb # [ 73.635834] niks3-server[1016]: 2026/09/22 10:35:42 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5337lb # [ 73.642382] niks3-server[1016]: 2026/09/22 10:35:42 INFO Completed upload id=195338worker1 # [ 73.658581] nix-grpc-daemon[863]: time=2026-09-22T10:35:41.879Z level=INFO msg="Upload complete. (268ms)"5339lb # [ 73.645109] niks3-server[1016]: 2026/09/22 10:35:42 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5340worker2 # [ 73.700262] nix-grpc-daemon[787]: ts=2026-09-22T10:35:42Z level=debug event=want client=11 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05341worker2 # [ 73.705153] nix-grpc-daemon[787]: ts=2026-09-22T10:35:42Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=15342worker1 # [ 73.724736] nix-grpc-daemon[784]: ts=2026-09-22T10:35:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434625343worker1 # [ 73.737505] nix-grpc-daemon[784]: ts=2026-09-22T10:35:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434625344client # copying 2 paths to 'lb:50051'...5345client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5346worker1 # [ 73.751129] nix-grpc-daemon[784]: ts=2026-09-22T10:35:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434625347client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5348worker1 # [ 73.764203] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5349worker1 # [ 73.944203] nix-daemon[851]: accepted connection from pid 1118, user nix-grpc-daemon (trusted)5350worker1 # [ 73.958085] nix-daemon[851]: reaped child process 1121, status = succeeded5351worker1 # [ 74.117850] nix-daemon[851]: accepted connection from pid 1123, user nix-grpc-daemon (trusted)5352worker1 # [ 74.129817] nix-daemon[851]: reaped child process 1126, status = succeeded5353lb # [ 74.130800] niks3-server[1016]: 2026/09/22 10:35:43 INFO Received uploads request method=POST path=/api/pending_closures5354lb # [ 74.136637] niks3-server[1016]: 2026/09/22 10:35:43 INFO Received uploads request method=POST path=/api/pending_closures5355worker1 # [ 74.156752] nix-grpc-daemon[863]: time=2026-09-22T10:35:42.378Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5356worker1 # [ 74.158966] nix-grpc-daemon[863]: time=2026-09-22T10:35:42.379Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5357worker1 # [ 74.161024] nix-grpc-daemon[863]: time=2026-09-22T10:35:42.379Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5358lb # [ 74.158300] niks3-server[1016]: 2026/09/22 10:35:43 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5359lb # [ 74.162247] niks3-server[1016]: 2026/09/22 10:35:43 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5360lb # [ 74.166860] niks3-server[1016]: 2026/09/22 10:35:43 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5361lb # [ 74.169735] niks3-server[1016]: 2026/09/22 10:35:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5362lb # [ 74.172237] niks3-server[1016]: 2026/09/22 10:35:43 INFO Signed narinfos id=20 count=15363lb # [ 74.173684] niks3-server[1016]: 2026/09/22 10:35:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5364lb # [ 74.175713] niks3-server[1016]: 2026/09/22 10:35:43 INFO Signed narinfos id=21 count=25365worker1 # [ 74.193797] nix-grpc-daemon[863]: time=2026-09-22T10:35:42.414Z level=INFO msg="Uploading 3 narinfos"5366lb # [ 74.178414] niks3-server[1016]: 2026/09/22 10:35:43 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5367lb # [ 74.189907] niks3-server[1016]: 2026/09/22 10:35:43 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5368lb # [ 74.193549] niks3-server[1016]: 2026/09/22 10:35:43 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5369lb # [ 74.199771] niks3-server[1016]: 2026/09/22 10:35:43 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5370lb # [ 74.202834] niks3-server[1016]: 2026/09/22 10:35:43 INFO Completed upload id=205371lb # [ 74.204269] niks3-server[1016]: 2026/09/22 10:35:43 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5372lb # [ 74.206420] niks3-server[1016]: 2026/09/22 10:35:43 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5373lb # [ 74.208854] niks3-server[1016]: 2026/09/22 10:35:43 INFO Completed upload id=215374worker1 # [ 74.225333] nix-grpc-daemon[863]: time=2026-09-22T10:35:42.446Z level=INFO msg="Upload complete. (445ms)"5375worker1 # [ 74.227920] nix-grpc-daemon[784]: ts=2026-09-22T10:35:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=2 nar_bytes_in=9765376worker1 # [ 74.231175] nix-grpc-daemon[784]: ts=2026-09-22T10:35:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434625377worker1 # [ 74.238249] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5378worker1 # [ 74.240629] nix-daemon[851]: reaped child process 1116, status = succeeded5379worker1 # [ 74.246134] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5380client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5381client # uses-input-w1onlya> node-a: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5382client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5383worker1 # [ 75.270521] nix-daemon[851]: reaped child process 1130, status = succeeded5384worker1 # [ 75.427047] nix-daemon[851]: accepted connection from pid 1146, user nix-grpc-daemon (trusted)5385worker1 # [ 75.437928] nix-daemon[851]: reaped child process 1149, status = succeeded5386lb # [ 75.436764] niks3-server[1016]: 2026/09/22 10:35:44 INFO Received uploads request method=POST path=/api/pending_closures5387worker1 # [ 75.461404] nix-grpc-daemon[863]: time=2026-09-22T10:35:43.682Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5388lb # [ 75.468257] niks3-server[1016]: 2026/09/22 10:35:44 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5389lb # [ 75.476835] niks3-server[1016]: 2026/09/22 10:35:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5390lb # [ 75.482534] niks3-server[1016]: 2026/09/22 10:35:44 INFO Signed narinfos id=22 count=15391worker1 # [ 75.501849] nix-grpc-daemon[863]: time=2026-09-22T10:35:43.723Z level=INFO msg="Uploading 1 narinfos"5392lb # [ 75.489129] niks3-server[1016]: 2026/09/22 10:35:44 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5393lb # [ 75.503522] niks3-server[1016]: 2026/09/22 10:35:44 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5394lb # [ 75.508581] niks3-server[1016]: 2026/09/22 10:35:44 INFO Completed upload id=225395worker1 # [ 75.527094] nix-grpc-daemon[863]: time=2026-09-22T10:35:43.748Z level=INFO msg="Upload complete. (273ms)"5396lb # [ 75.512232] niks3-server[1016]: 2026/09/22 10:35:44 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5397client # copying 1 paths...5398worker1 # [ 75.531454] nix-grpc-daemon[784]: ts=2026-09-22T10:35:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=10 outputs=15399worker2 # [ 75.557629] nix-grpc-daemon[787]: ts=2026-09-22T10:35:44Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:521525400worker1 # [ 75.543730] nix-daemon[851]: reaped child process 1128, status = succeeded5401worker2 # [ 75.568430] nix-grpc-daemon[787]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5402client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5403worker1 # [ 75.573410] nix-grpc-daemon[784]: ts=2026-09-22T10:35:43Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:434625404worker1 # [ 75.578460] nix-grpc-daemon[784]: ts=2026-09-22T10:35:43Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=1 nar_bytes_out=1205405client # [ 75.623682] nix-daemon[1047]: reaped child process 1133, status = succeeded5406client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.63 seconds)5407client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25408client # [ 75.820563] nix-daemon[1047]: accepted connection from pid 1152, user root (trusted)5409client # this derivation will be built:5410client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5411worker1 # [ 75.994271] nix-grpc-daemon[784]: ts=2026-09-22T10:35:44Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:434625412client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5413worker2 # [ 76.017401] nix-grpc-daemon[787]: ts=2026-09-22T10:35:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:521525414worker2 # [ 76.035161] nix-grpc-daemon[787]: ts=2026-09-22T10:35:45Z level=debug event=want client=12 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05415worker2 # [ 76.040620] nix-grpc-daemon[787]: ts=2026-09-22T10:35:45Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=15416worker2 # [ 76.046623] nix-grpc-daemon[787]: ts=2026-09-22T10:35:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:521525417worker2 # [ 76.055195] nix-grpc-daemon[787]: ts=2026-09-22T10:35:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:521525418client # copying 1 paths to 'lb:50051'...5419client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5420worker2 # [ 76.066542] nix-grpc-daemon[787]: ts=2026-09-22T10:35:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:521525421worker2 # [ 76.077359] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5422worker2 # [ 76.092162] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 54 ms (attempt 1/5)5423worker2 # [ 76.146610] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 102 ms (attempt 2/5)5424worker2 # [ 76.254654] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 188 ms (attempt 3/5)5425worker2 # [ 76.449572] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 48 ms (attempt 4/5)5426worker2 # [ 76.503699] nix-grpc-daemon[787]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5427worker2 # [ 76.507167] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5428worker2 # [ 76.512802] nix-grpc-daemon[787]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5429worker2 # [ 76.685101] nix-daemon[873]: accepted connection from pid 1145, user nix-grpc-daemon (trusted)5430worker2 # [ 76.697379] nix-daemon[873]: reaped child process 1148, status = succeeded5431worker2 # [ 76.830435] nix-daemon[873]: accepted connection from pid 1150, user nix-grpc-daemon (trusted)5432worker2 # [ 76.843486] nix-daemon[873]: reaped child process 1153, status = succeeded5433lb # [ 76.816492] niks3-server[1016]: 2026/09/22 10:35:45 INFO Received uploads request method=POST path=/api/pending_closures5434worker2 # [ 76.857337] nix-grpc-daemon[928]: time=2026-09-22T10:35:45.884Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5435worker2 # [ 76.859223] nix-grpc-daemon[928]: time=2026-09-22T10:35:45.886Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5436lb # [ 76.833031] niks3-server[1016]: 2026/09/22 10:35:45 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5437lb # [ 76.834714] niks3-server[1016]: 2026/09/22 10:35:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5438lb # [ 76.836802] niks3-server[1016]: 2026/09/22 10:35:45 INFO Signed narinfos id=23 count=15439worker2 # [ 76.874442] nix-grpc-daemon[928]: time=2026-09-22T10:35:45.902Z level=INFO msg="Uploading 1 narinfos"5440lb # [ 76.841661] niks3-server[1016]: 2026/09/22 10:35:45 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5441lb # [ 76.845146] niks3-server[1016]: 2026/09/22 10:35:45 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5442lb # [ 76.847675] niks3-server[1016]: 2026/09/22 10:35:45 INFO Completed upload id=235443worker2 # [ 76.883948] nix-grpc-daemon[928]: time=2026-09-22T10:35:45.911Z level=INFO msg="Upload complete. (355ms)"5444worker2 # [ 76.886296] nix-grpc-daemon[787]: ts=2026-09-22T10:35:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=0 paths=1 nar_bytes_in=5605445lb # [ 76.850575] niks3-server[1016]: 2026/09/22 10:35:45 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5446worker2 # [ 76.889203] nix-grpc-daemon[787]: ts=2026-09-22T10:35:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:521525447worker2 # [ 76.894866] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5448worker2 # [ 76.901847] nix-daemon[873]: reaped child process 1138, status = succeeded5449worker2 # [ 76.902813] nix-grpc-daemon[787]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5450worker2 # [ 76.909853] nix-daemon[873]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5451client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5452client # uses-input-w1onlyb> worker2: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5453client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5454worker2 # [ 77.723829] nix-daemon[873]: reaped child process 1158, status = succeeded5455worker2 # [ 77.858775] nix-daemon[873]: accepted connection from pid 1174, user nix-grpc-daemon (trusted)5456worker2 # [ 77.871331] nix-daemon[873]: reaped child process 1177, status = succeeded5457lb # [ 77.851469] niks3-server[1016]: 2026/09/22 10:35:46 INFO Received uploads request method=POST path=/api/pending_closures5458worker2 # [ 77.897127] nix-grpc-daemon[928]: time=2026-09-22T10:35:46.924Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5459lb # [ 77.878913] niks3-server[1016]: 2026/09/22 10:35:46 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5460lb # [ 77.888470] niks3-server[1016]: 2026/09/22 10:35:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5461lb # [ 77.893115] niks3-server[1016]: 2026/09/22 10:35:46 INFO Signed narinfos id=24 count=15462worker2 # [ 77.932438] nix-grpc-daemon[928]: time=2026-09-22T10:35:46.960Z level=INFO msg="Uploading 1 narinfos"5463lb # [ 77.903739] niks3-server[1016]: 2026/09/22 10:35:46 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5464lb # [ 77.913303] niks3-server[1016]: 2026/09/22 10:35:46 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5465worker2 # [ 77.955249] nix-grpc-daemon[928]: time=2026-09-22T10:35:46.982Z level=INFO msg="Upload complete. (239ms)"5466lb # [ 77.918960] niks3-server[1016]: 2026/09/22 10:35:46 INFO Completed upload id=245467worker2 # [ 77.957933] nix-grpc-daemon[787]: ts=2026-09-22T10:35:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=11 outputs=15468lb # [ 77.922887] niks3-server[1016]: 2026/09/22 10:35:46 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5469client # copying 1 paths...5470worker2 # [ 77.966243] nix-grpc-daemon[787]: ts=2026-09-22T10:35:46Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:521525471worker2 # [ 77.968086] nix-daemon[873]: reaped child process 1155, status = succeeded5472client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5473worker1 # [ 77.964560] nix-grpc-daemon[784]: ts=2026-09-22T10:35:46Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:434625474worker1 # [ 77.976380] nix-grpc-daemon[784]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5475worker1 # [ 77.989951] nix-grpc-daemon[784]: ts=2026-09-22T10:35:46Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=1 nar_bytes_out=1205476client # [ 78.092811] nix-daemon[1047]: reaped child process 1155, status = succeeded5477client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.47 seconds)5478client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25479client # [ 78.276781] nix-daemon[1047]: accepted connection from pid 1174, user root (trusted)5480client # this derivation will be built:5481client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5482worker1 # [ 78.472972] nix-grpc-daemon[784]: ts=2026-09-22T10:35:46Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:434625483client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5484worker2 # [ 78.495764] nix-grpc-daemon[787]: ts=2026-09-22T10:35:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:521525485worker2 # [ 78.505820] nix-grpc-daemon[928]: time=2026-09-22T10:35:47.533Z level=INFO msg="All 1 paths already cached"5486worker2 # [ 78.519883] nix-grpc-daemon[787]: ts=2026-09-22T10:35:47Z level=debug event=want client=13 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05487worker2 # [ 78.524391] nix-grpc-daemon[787]: ts=2026-09-22T10:35:47Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=15488worker1 # [ 78.543781] nix-grpc-daemon[784]: ts=2026-09-22T10:35:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434625489worker1 # [ 78.556562] nix-grpc-daemon[784]: ts=2026-09-22T10:35:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434625490worker1 # [ 78.570298] nix-grpc-daemon[863]: time=2026-09-22T10:35:46.791Z level=INFO msg="All 1 paths already cached"5491client # copying 1 paths to 'lb:50051'...5492client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5493worker1 # [ 78.579405] nix-grpc-daemon[784]: ts=2026-09-22T10:35:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434625494worker1 # [ 78.590440] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5495lb # [ 78.642302] envoy[923]: conn peer=[2001:db8:1::1]:38994 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=515496lb # [ 78.646112] envoy[923]: conn peer=[2001:db8:1::1]:38990 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3880 tx=2177 ms=18815497lb # [ 78.649624] envoy[923]: conn peer=[2001:db8:1::1]:38998 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4342 tx=2412 ms=19375498lb # [ 78.653242] envoy[923]: conn peer=[2001:db8:1::1]:39008 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=595499lb # [ 78.656634] envoy[923]: conn peer=[2001:db8:1::1]:47454 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3853 tx=2065 ms=22015500lb # [ 78.660128] envoy[923]: conn peer=[2001:db8:1::1]:47458 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=460 tx=328 ms=135501lb # [ 78.663523] envoy[923]: conn peer=[2001:db8:1::1]:47478 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=329 ms=785502lb # [ 78.666910] envoy[923]: conn peer=[2001:db8:1::1]:47470 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3729 tx=2123 ms=20465503worker1 # [ 78.768909] nix-daemon[851]: accepted connection from pid 1154, user nix-grpc-daemon (trusted)5504worker1 # [ 78.781928] nix-daemon[851]: reaped child process 1157, status = succeeded5505worker1 # [ 78.949655] nix-daemon[851]: accepted connection from pid 1159, user nix-grpc-daemon (trusted)5506worker1 # [ 78.961845] nix-daemon[851]: reaped child process 1162, status = succeeded5507lb # [ 78.962449] niks3-server[1016]: 2026/09/22 10:35:47 INFO Received uploads request method=POST path=/api/pending_closures5508worker1 # [ 78.986776] nix-grpc-daemon[863]: time=2026-09-22T10:35:47.208Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5509worker1 # [ 78.988167] nix-grpc-daemon[863]: time=2026-09-22T10:35:47.208Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5510lb # [ 78.986937] niks3-server[1016]: 2026/09/22 10:35:47 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5511lb # [ 78.999123] niks3-server[1016]: 2026/09/22 10:35:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5512lb # [ 79.005682] niks3-server[1016]: 2026/09/22 10:35:47 INFO Signed narinfos id=25 count=15513worker1 # [ 79.025286] nix-grpc-daemon[863]: time=2026-09-22T10:35:47.246Z level=INFO msg="Uploading 1 narinfos"5514lb # [ 79.011088] niks3-server[1016]: 2026/09/22 10:35:47 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5515lb # [ 79.023455] niks3-server[1016]: 2026/09/22 10:35:47 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5516lb # [ 79.028269] niks3-server[1016]: 2026/09/22 10:35:47 INFO Completed upload id=255517worker1 # [ 79.044896] nix-grpc-daemon[863]: time=2026-09-22T10:35:47.266Z level=INFO msg="Upload complete. (443ms)"5518worker1 # [ 79.047730] nix-grpc-daemon[784]: ts=2026-09-22T10:35:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=0 paths=1 nar_bytes_in=5605519lb # [ 79.032322] niks3-server[1016]: 2026/09/22 10:35:47 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5520worker1 # [ 79.053411] nix-grpc-daemon[784]: ts=2026-09-22T10:35:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434625521worker1 # [ 79.058160] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5522worker1 # [ 79.060406] nix-daemon[851]: reaped child process 1152, status = succeeded5523worker1 # [ 79.065906] nix-daemon[851]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5524client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5525client # uses-input-w1onlyc> node-a: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5526client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5527worker1 # [ 79.892896] nix-daemon[851]: reaped child process 1166, status = succeeded5528worker1 # [ 80.055092] nix-daemon[851]: accepted connection from pid 1182, user nix-grpc-daemon (trusted)5529worker1 # [ 80.067315] nix-daemon[851]: reaped child process 1185, status = succeeded5530lb # [ 80.068843] niks3-server[1016]: 2026/09/22 10:35:48 INFO Received uploads request method=POST path=/api/pending_closures5531worker1 # [ 80.093125] nix-grpc-daemon[863]: time=2026-09-22T10:35:48.314Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5532lb # [ 80.098146] niks3-server[1016]: 2026/09/22 10:35:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5533lb # [ 80.106109] niks3-server[1016]: 2026/09/22 10:35:48 INFO Signed narinfos id=26 count=15534worker1 # [ 80.125441] nix-grpc-daemon[863]: time=2026-09-22T10:35:48.346Z level=INFO msg="Uploading 1 narinfos"5535lb # [ 80.111661] niks3-server[1016]: 2026/09/22 10:35:49 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5536lb # [ 80.126333] niks3-server[1016]: 2026/09/22 10:35:49 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5537lb # [ 80.130524] niks3-server[1016]: 2026/09/22 10:35:49 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5538lb # [ 80.136342] niks3-server[1016]: 2026/09/22 10:35:49 INFO Completed upload id=265539worker1 # [ 80.152609] nix-grpc-daemon[863]: time=2026-09-22T10:35:48.374Z level=INFO msg="Upload complete. (280ms)"5540lb # [ 80.139077] niks3-server[1016]: 2026/09/22 10:35:49 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5541worker1 # [ 80.156733] nix-grpc-daemon[784]: ts=2026-09-22T10:35:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43462 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=12 outputs=15542worker1 # [ 80.168560] nix-daemon[851]: reaped child process 1164, status = succeeded5543client # copying 1 paths...5544worker2 # [ 80.226617] nix-grpc-daemon[787]: ts=2026-09-22T10:35:49Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:521525545worker2 # [ 80.237993] nix-grpc-daemon[787]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5546client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5547worker2 # [ 80.269415] nix-grpc-daemon[787]: ts=2026-09-22T10:35:49Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:521525548worker2 # [ 80.274381] nix-grpc-daemon[787]: ts=2026-09-22T10:35:49Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:52152 duration_s=0 paths=1 nar_bytes_out=1205549client # [ 80.359649] nix-daemon[1047]: reaped child process 1177, status = succeeded5550client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.27 seconds)5551(finished: subtest: an input only one worker has still reaches the builder, in 8.14 seconds)5552subtest: upload whose reference the worker lacks is completed from the cache5553client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5554client # this derivation will be built:5555client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5556client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5557client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.43 seconds)5558client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5559client # this derivation will be built:5560client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5561client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5562client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.41 seconds)5563client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5564client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.18 seconds)5565worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5566worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.17 seconds)5567worker2 # [ 81.624662] systemd[1]: Stopping Nix worker-protocol over gRPC...5568worker2 # [ 82.062515] nix-grpc-daemon[787]: ts=2026-09-22T10:35:51Z level=info event=signal_exit5569lb # [ 82.299817] niks3-server[1016]: 2026/09/22 10:35:51 INFO lead: released remote=192.168.1.4:564465570worker1 # [ 83.319070] nix-grpc-daemon[784]: ts=2026-09-22T10:35:51Z level=info event=scheduler_restarting addr=lb:500515571worker2 # [ 83.341835] nix-grpc-daemon[787]: ts=2026-09-22T10:35:52Z level=info event=scheduler_restarting5572worker2 # [ 83.344839] nix-grpc-daemon[787]: ts=2026-09-22T10:35:52Z level=info event=scheduler_restarting addr=lb:500515573worker2 # [ 83.348341] nix-grpc-daemon[787]: ts=2026-09-22T10:35:52Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:521465574worker2 # [ 83.351858] nix-grpc-daemon[787]: ts=2026-09-22T10:35:52Z level=info event=worker_gone addr=192.168.1.4:500515575worker2 # [ 83.355132] nix-grpc-daemon[787]: ts=2026-09-22T10:35:52Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:521465576worker2 # [ 83.358590] nix-grpc-daemon[787]: ts=2026-09-22T10:35:52Z level=info event=worker_gone addr=192.168.1.3:500515577worker2 # [ 83.375126] nix-daemon[873]: reaped child process 875, status = succeeded5578worker1 # [ 83.518548] nix-grpc-daemon[784]: ts=2026-09-22T10:35:51Z level=info event=scheduler_connected addr=lb:500515579worker1 # [ 83.520282] nix-grpc-daemon[784]: ts=2026-09-22T10:35:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5580worker2 # [ 83.543654] nix-grpc-daemon[787]: ts=2026-09-22T10:35:52Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5581worker1 # [ 83.721109] nix-grpc-daemon[784]: ts=2026-09-22T10:35:51Z level=info event=scheduler_connected addr=lb:500515582worker1 # [ 83.723144] nix-grpc-daemon[784]: ts=2026-09-22T10:35:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5583worker1 # [ 83.923571] nix-grpc-daemon[784]: ts=2026-09-22T10:35:52Z level=info event=scheduler_connected addr=lb:500515584worker1 # [ 83.925548] nix-grpc-daemon[784]: ts=2026-09-22T10:35:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5585worker1 # [ 84.126502] nix-grpc-daemon[784]: ts=2026-09-22T10:35:52Z level=info event=scheduler_connected addr=lb:500515586worker1 # [ 84.130581] nix-grpc-daemon[784]: ts=2026-09-22T10:35:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5587worker2 # [ 84.187932] nix-grpc-daemon[787]: E0000 00:00:1790073353.215434 832 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true5588worker2 # [ 84.192938] nix-grpc-daemon[787]: E0000 00:00:1790073353.215562 832 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5589worker1 # [ 84.331402] nix-grpc-daemon[784]: ts=2026-09-22T10:35:52Z level=info event=scheduler_connected addr=lb:500515590worker1 # [ 84.333070] nix-grpc-daemon[784]: ts=2026-09-22T10:35:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5591worker1 # [ 84.533543] nix-grpc-daemon[784]: ts=2026-09-22T10:35:52Z level=info event=scheduler_connected addr=lb:500515592worker1 # [ 84.535288] nix-grpc-daemon[784]: ts=2026-09-22T10:35:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5593worker1 # [ 84.736353] nix-grpc-daemon[784]: ts=2026-09-22T10:35:52Z level=info event=scheduler_connected addr=lb:500515594worker1 # [ 84.740280] nix-grpc-daemon[784]: ts=2026-09-22T10:35:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5595worker1 # [ 84.941228] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=info event=scheduler_connected addr=lb:500515596worker1 # [ 84.945261] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5597worker1 # [ 85.145959] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=info event=scheduler_connected addr=lb:500515598worker1 # [ 85.150185] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5599worker1 # [ 85.351808] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=info event=scheduler_connected addr=lb:500515600worker1 # [ 85.355536] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5601worker2 # [ 85.383944] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5602worker2 # [ 85.389521] systemd[1]: Stopped Nix worker-protocol over gRPC.5603worker2 # [ 85.391620] systemd[1]: nix-grpc-daemon.service: Consumed 1.972s CPU time over 1min 6.598s wall clock time, 57.1M memory peak, 74.8K incoming IP traffic, 229.5K outgoing IP traffic.5604worker2 # [ 85.402773] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5605worker2 # [ 85.406361] systemd[1]: Closed Nix worker-protocol over gRPC.5606worker2 # [ 85.409263] systemd[1]: nix-grpc-daemon.socket: Consumed 299us CPU time over 1min 14.994s wall clock time, 256K memory peak, 92.2K incoming IP traffic, 80K outgoing IP traffic.5607lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5608lb: must succeed: curl -sf localhost:9901/clusters5609lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5610lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.06 seconds)5611client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25612worker1 # [ 85.556510] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=info event=scheduler_connected addr=lb:500515613worker1 # [ 85.560522] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5614worker1 # [ 85.687350] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:434625615client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5616worker1 # [ 85.692435] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:434625617client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.26 seconds)5618worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5619worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.02 seconds)5620worker1 # [ 85.761315] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=info event=scheduler_connected addr=lb:500515621worker1 # [ 85.765104] nix-grpc-daemon[784]: ts=2026-09-22T10:35:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5622worker2 # [ 85.816871] systemd[1]: Listening on Nix worker-protocol over gRPC.5623worker1 # [ 85.855493] systemd[1]: Stopping Nix worker-protocol over gRPC...5624worker1 # [ 85.952506] nix-grpc-daemon[784]: ts=2026-09-22T10:35:54Z level=info event=signal_exit5625worker1 # [ 85.965781] nix-grpc-daemon[784]: ts=2026-09-22T10:35:54Z level=info event=scheduler_connected addr=lb:500515626worker1 # [ 85.969680] nix-grpc-daemon[784]: ts=2026-09-22T10:35:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5627worker1 # [ 86.170498] nix-grpc-daemon[784]: ts=2026-09-22T10:35:54Z level=info event=scheduler_connected addr=lb:500515628worker1 # [ 86.174501] nix-grpc-daemon[784]: ts=2026-09-22T10:35:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5629worker2 # [ 86.371956] systemd[1]: Starting scheduler-token.service...5630worker1 # [ 86.375656] nix-grpc-daemon[784]: ts=2026-09-22T10:35:54Z level=info event=scheduler_connected addr=lb:500515631worker1 # [ 86.376996] nix-grpc-daemon[784]: ts=2026-09-22T10:35:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5632worker2 # [ 86.445737] systemd[1]: scheduler-token.service: Deactivated successfully.5633worker2 # [ 86.447506] systemd[1]: Finished scheduler-token.service.5634worker2 # [ 86.448810] systemd[1]: scheduler-token.service: Consumed 33ms CPU time over 75ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5635worker2 # [ 86.453580] systemd[1]: Starting Nix worker-protocol over gRPC...5636worker2 # [ 86.591376] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:55Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5637worker2 # [ 86.593645] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:55Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5638worker1 # [ 86.576332] nix-grpc-daemon[784]: ts=2026-09-22T10:35:54Z level=info event=scheduler_connected addr=lb:500515639worker2 # [ 86.597467] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:55Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15640worker1 # [ 86.577997] nix-grpc-daemon[784]: ts=2026-09-22T10:35:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5641worker2 # [ 86.604071] nix-grpc-daemon[1204]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5642worker2 # [ 86.606177] nix-grpc-daemon[1204]: E0000 00:00:1790073355.631254 1204 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5643worker2 # [ 86.611075] nix-grpc-daemon[1204]: E0000 00:00:1790073355.636223 1204 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5644worker2 # [ 86.614551] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:55Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5645worker2 # [ 86.619928] systemd[1]: Started Nix worker-protocol over gRPC.5646lb # [ 86.585719] niks3-server[1016]: 2026/09/22 10:35:55 INFO lead: acquired remote=192.168.1.4:382605647worker2 # [ 86.623873] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:55Z level=info event=scheduler_take_over5648worker2 # [ 86.626143] nix-grpc-daemon[1204]: E0000 00:00:1790073355.652983 1246 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5649worker2 # [ 86.637172] nix-grpc-daemon[1204]: E0000 00:00:1790073355.664841 1246 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5650worker2 # [ 86.641131] nix-grpc-daemon[1204]: E0000 00:00:1790073355.667042 1246 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5651worker2 # [ 86.651132] nix-grpc-daemon[1204]: E0000 00:00:1790073355.678813 1246 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5652worker2 # [ 86.658116] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:55Z level=info event=scheduler_connected addr=lb:500515653worker2 # [ 86.659579] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5654worker1 # [ 86.778415] nix-grpc-daemon[784]: ts=2026-09-22T10:35:54Z level=info event=scheduler_connected addr=lb:500515655worker1 # [ 86.780067] nix-grpc-daemon[784]: ts=2026-09-22T10:35:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5656worker2 # [ 86.860888] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:55Z level=info event=scheduler_connected addr=lb:500515657worker2 # [ 86.864637] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5658worker1 # [ 86.980617] nix-grpc-daemon[784]: ts=2026-09-22T10:35:55Z level=info event=scheduler_connected addr=lb:500515659worker1 # [ 86.982406] nix-grpc-daemon[784]: ts=2026-09-22T10:35:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5660worker2 # [ 87.066144] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:56Z level=info event=scheduler_connected addr=lb:500515661worker2 # [ 87.069974] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5662worker1 # [ 87.182965] nix-grpc-daemon[784]: ts=2026-09-22T10:35:55Z level=info event=scheduler_connected addr=lb:500515663worker1 # [ 87.184634] nix-grpc-daemon[784]: ts=2026-09-22T10:35:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5664worker1 # [ 87.232676] nix-grpc-daemon[784]: ts=2026-09-22T10:35:55Z level=info event=scheduler_restarting5665worker1 # [ 87.251624] nix-daemon[851]: reaped child process 853, status = succeeded5666worker2 # [ 87.270849] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:56Z level=info event=scheduler_connected addr=lb:500515667worker2 # [ 87.274722] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5668worker1 # [ 87.400590] nix-grpc-daemon[784]: ts=2026-09-22T10:35:55Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5669worker2 # [ 87.475830] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:56Z level=info event=scheduler_connected addr=lb:500515670worker2 # [ 87.479731] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5671worker2 # [ 87.684989] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:56Z level=info event=scheduler_connected addr=lb:500515672worker2 # [ 87.693149] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:56Z level=info event=oidc_jwks provider=mock keys=15673worker2 # [ 87.696114] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:56Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:479765674worker2 # [ 87.699907] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:56Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05675worker1 # [ 88.088068] nix-grpc-daemon[784]: E0000 00:00:1790073356.309318 829 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5676worker1 # [ 88.093519] nix-grpc-daemon[784]: E0000 00:00:1790073356.314698 829 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5677lb # [ 88.642412] envoy[923]: conn peer=[2001:db8:1::1]:47490 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=585678lb # [ 88.643894] envoy[923]: conn peer=[2001:db8:1::1]:47486 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3742 tx=2101 ms=18345679lb # [ 88.645310] envoy[923]: conn peer=[2001:db8:1::4]:57826 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=75593 tx=3571 ms=644175680lb # [ 88.646815] envoy[923]: conn peer=[2001:db8:1::1]:45278 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=165681lb # [ 88.648284] envoy[923]: conn peer=[2001:db8:1::3]:40088 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=22051 tx=4131 ms=674925682worker1 # [ 89.298434] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5683worker1 # [ 89.300950] systemd[1]: Stopped Nix worker-protocol over gRPC.5684worker1 # [ 89.301910] systemd[1]: nix-grpc-daemon.service: Consumed 2.554s CPU time over 1min 10.648s wall clock time, 61.9M memory peak, 93.5K incoming IP traffic, 176.4K outgoing IP traffic.5685worker1 # [ 89.305735] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5686worker1 # [ 89.307491] systemd[1]: Closed Nix worker-protocol over gRPC.5687worker1 # [ 89.308639] systemd[1]: nix-grpc-daemon.socket: Consumed 289us CPU time over 1min 20.163s wall clock time, 256K memory peak, 86.7K incoming IP traffic, 76K outgoing IP traffic.5688lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5689lb: must succeed: curl -sf localhost:9901/clusters5690lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5691lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.07 seconds)5692client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25693worker2 # [ 89.635547] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:479785694worker2 # [ 89.646474] nix-daemon[873]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5695worker2 # [ 89.662582] nix-grpc-daemon[1204]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 63 ms (attempt 1/5)5696worker2 # [ 89.731385] nix-grpc-daemon[1204]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 66 ms (attempt 2/5)5697worker2 # [ 89.803543] nix-grpc-daemon[1204]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 242 ms (attempt 3/5)5698worker2 # [ 90.051763] nix-grpc-daemon[1204]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 222 ms (attempt 4/5)5699worker2 # [ 90.279948] nix-grpc-daemon[1204]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5700worker2 # [ 90.283749] nix-grpc-daemon[1204]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5701client # copying 2 paths...5702client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5703worker2 # [ 90.302221] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:479785704client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5705worker2 # [ 90.312110] nix-daemon[873]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5706worker2 # [ 90.334896] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:59Z level=info event=niks3_push_started pid=12645707worker2 # [ 90.509715] nix-grpc-daemon[1264]: time=2026-09-22T10:35:59.537Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5708worker2 # [ 90.663475] nix-daemon[873]: accepted connection from pid 1277, user nix-grpc-daemon (trusted)5709worker2 # [ 90.676304] nix-daemon[873]: reaped child process 1280, status = succeeded5710lb # [ 90.655314] niks3-server[1016]: 2026/09/22 10:35:59 INFO Received uploads request method=POST path=/api/pending_closures5711lb # [ 90.660102] niks3-server[1016]: 2026/09/22 10:35:59 INFO Received uploads request method=POST path=/api/pending_closures5712worker2 # [ 90.700032] nix-grpc-daemon[1264]: time=2026-09-22T10:35:59.727Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5713worker2 # [ 90.701565] nix-grpc-daemon[1264]: time=2026-09-22T10:35:59.729Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5714worker2 # [ 90.703459] nix-grpc-daemon[1264]: time=2026-09-22T10:35:59.731Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5715lb # [ 90.687046] niks3-server[1016]: 2026/09/22 10:35:59 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5716lb # [ 90.690084] niks3-server[1016]: 2026/09/22 10:35:59 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5717lb # [ 90.695561] niks3-server[1016]: 2026/09/22 10:35:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5718lb # [ 90.697878] niks3-server[1016]: 2026/09/22 10:35:59 INFO Signed narinfos id=27 count=15719lb # [ 90.699601] niks3-server[1016]: 2026/09/22 10:35:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5720lb # [ 90.702843] niks3-server[1016]: 2026/09/22 10:35:59 INFO Signed narinfos id=28 count=25721worker2 # [ 90.739718] nix-grpc-daemon[1264]: time=2026-09-22T10:35:59.766Z level=INFO msg="Uploading 3 narinfos"5722lb # [ 90.704758] niks3-server[1016]: 2026/09/22 10:35:59 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5723lb # [ 90.722814] niks3-server[1016]: 2026/09/22 10:35:59 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5724lb # [ 90.725155] niks3-server[1016]: 2026/09/22 10:35:59 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5725lb # [ 90.730802] niks3-server[1016]: 2026/09/22 10:35:59 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5726lb # [ 90.734794] niks3-server[1016]: 2026/09/22 10:35:59 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5727lb # [ 90.738067] niks3-server[1016]: 2026/09/22 10:35:59 INFO Completed upload id=275728lb # [ 90.739644] niks3-server[1016]: 2026/09/22 10:35:59 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5729lb # [ 90.741508] niks3-server[1016]: 2026/09/22 10:35:59 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5730lb # [ 90.743669] niks3-server[1016]: 2026/09/22 10:35:59 INFO Completed upload id=285731worker2 # [ 90.781525] nix-grpc-daemon[1264]: time=2026-09-22T10:35:59.809Z level=INFO msg="Upload complete. (267ms)"5732worker2 # [ 90.786302] nix-grpc-daemon[1204]: ts=2026-09-22T10:35:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47978 duration_s=0 paths=2 nar_bytes_in=3125733worker2 # [ 90.796220] nix-daemon[873]: reaped child process 1262, status = succeeded5734client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.42 seconds)5735worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5736worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.02 seconds)5737worker1 # [ 90.884735] systemd[1]: Listening on Nix worker-protocol over gRPC.5738worker1 # [ 90.892791] systemd[1]: Starting Nix worker-protocol over gRPC...5739worker1 # [ 91.047364] nix-grpc-daemon[1208]: ts=2026-09-22T10:35:59Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5740worker1 # [ 91.049629] nix-grpc-daemon[1208]: ts=2026-09-22T10:35:59Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5741worker1 # [ 91.053419] nix-grpc-daemon[1208]: ts=2026-09-22T10:35:59Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15742worker1 # [ 91.060067] nix-grpc-daemon[1208]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5743worker1 # [ 91.061459] nix-grpc-daemon[1208]: E0000 00:00:1790073359.281040 1208 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5744worker1 # [ 91.068307] nix-grpc-daemon[1208]: E0000 00:00:1790073359.286326 1208 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5745worker1 # [ 91.071870] nix-grpc-daemon[1208]: ts=2026-09-22T10:35:59Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5746worker1 # [ 91.074332] systemd[1]: Started Nix worker-protocol over gRPC.5747worker1 # [ 91.079128] nix-grpc-daemon[1208]: E0000 00:00:1790073359.299901 1250 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5748lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5749lb: must succeed: curl -sf localhost:9901/clusters5750worker1 # [ 91.091164] nix-grpc-daemon[1208]: E0000 00:00:1790073359.312666 1250 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5751worker1 # [ 91.095054] nix-grpc-daemon[1208]: E0000 00:00:1790073359.315043 1250 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5752worker1 # [ 91.105438] nix-grpc-daemon[1208]: E0000 00:00:1790073359.326901 1250 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5753worker1 # [ 91.118742] nix-grpc-daemon[1208]: ts=2026-09-22T10:35:59Z level=info event=scheduler_connected addr=lb:500515754worker2 # [ 91.140159] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:00Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:479765755worker2 # [ 91.144804] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:00Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05756lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5757lb: must succeed: curl -sf localhost:9901/clusters5758lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5759lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.14 seconds)5760(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.84 seconds)5761subtest: a niks3 restart does not cost the leader its role5762lb: must succeed: curl -sf localhost:9901/clusters5763lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5764lb: must succeed: curl -sf localhost:9901/clusters5765lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5766lb # [ 92.329685] niks3-server[1016]: 2026/09/22 10:36:01 INFO Shutdown signal received, draining in-flight requests timeout=10s5767lb # [ 92.331250] systemd[1]: Stopping niks3 server...5768lb # [ 92.334785] niks3-server[1016]: 2026/09/22 10:36:01 INFO lead: released remote=192.168.1.4:382605769worker1 # [ 92.351624] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:00Z level=info event=scheduler_lock_lost status=2005770worker2 # [ 92.372424] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:01Z level=info event=scheduler_lock_lost status=2005771lb # [ 92.348255] systemd[1]: niks3.service: Deactivated successfully.5772lb # [ 92.349712] systemd[1]: Stopped niks3 server.5773lb # [ 92.351082] systemd[1]: niks3.service: Consumed 577ms CPU time over 1min 6.229s wall clock time, 21.1M memory peak, 83.6K incoming IP traffic, 116.9K outgoing IP traffic.5774lb # [ 92.358810] systemd[1]: Starting niks3 server...5775lb # [ 92.429208] niks3-server[1156]: 2026/09/22 10:36:01 goose: no migrations to run. current version: 202609200000005776lb # [ 92.434993] niks3-server[1156]: 2026/09/22 10:36:01 OK 1_commit_pending_closure.sql (4.26ms)5777lb # [ 92.438481] niks3-server[1156]: 2026/09/22 10:36:01 OK 2_object_stats_trigger.sql (1.91ms)5778lb # [ 92.439620] niks3-server[1156]: 2026/09/22 10:36:01 goose: up to current file version: 25779lb # [ 92.441286] niks3-server[1156]: 2026/09/22 10:36:01 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key5780lb # [ 92.453502] niks3-server[1156]: 2026/09/22 10:36:01 INFO Read proxy enabled — serving cache objects from S35781lb # [ 92.454780] niks3-server[1156]: 2026/09/22 10:36:01 INFO Using socket-activated listener address=0.0.0.0:57515782lb # [ 92.456672] niks3-server[1156]: 2026/09/22 10:36:01 INFO systemd watchdog enabled interval=15s5783lb # [ 92.457798] systemd[1]: Started niks3 server.5784lb # [ 92.460070] niks3-server[1156]: 2026/09/22 10:36:01 INFO Starting HTTP server address=0.0.0.0:57515785lb: waiting for unit niks3.service5786lb: (finished: waiting for unit niks3.service, in 0.04 seconds)5787lb: must succeed: curl -sf localhost:9901/clusters5788lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5789worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5790worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5791lb: must succeed: curl -sf localhost:9901/clusters5792lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5793lb: must succeed: curl -sf localhost:9901/clusters5794lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5795worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5796worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.06 seconds)5797client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&25798worker1 # [ 92.997258] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:596125799worker1 # [ 93.008712] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5800client # don't know how to build these paths:5801client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5802worker2 # [ 93.043510] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:479785803worker2 # [ 93.063742] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:02Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=05804worker2 # [ 93.068767] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:02Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.4:50051 clients=15805worker2 # [ 93.108410] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:479785806worker2 # [ 93.120788] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:479785807client # copying 1 paths to 'lb:50051'...5808client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...5809worker2 # [ 93.132866] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:479785810worker2 # [ 93.144500] nix-daemon[873]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5811worker2 # [ 93.317632] nix-daemon[873]: accepted connection from pid 1297, user nix-grpc-daemon (trusted)5812worker2 # [ 93.330282] nix-daemon[873]: reaped child process 1300, status = succeeded5813lb # [ 93.340330] niks3-server[1156]: 2026/09/22 10:36:02 INFO lead: acquired remote=192.168.1.4:404005814worker2 # [ 93.494103] nix-daemon[873]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5815worker2 # [ 93.506804] nix-daemon[873]: reaped child process 1307, status = succeeded5816lb # [ 93.487732] niks3-server[1156]: 2026/09/22 10:36:02 INFO Received uploads request method=POST path=/api/pending_closures5817worker2 # [ 93.544835] nix-grpc-daemon[1264]: time=2026-09-22T10:36:02.572Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5818worker2 # [ 93.546260] nix-grpc-daemon[1264]: time=2026-09-22T10:36:02.572Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"5819lb # [ 93.536635] niks3-server[1156]: 2026/09/22 10:36:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5820lb # [ 93.543960] niks3-server[1156]: 2026/09/22 10:36:02 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst5821lb # [ 93.560146] niks3-server[1156]: 2026/09/22 10:36:02 INFO Signed narinfos id=29 count=15822worker2 # [ 93.598259] nix-grpc-daemon[1264]: time=2026-09-22T10:36:02.625Z level=INFO msg="Uploading 1 narinfos"5823lb # [ 93.567661] niks3-server[1156]: 2026/09/22 10:36:02 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls5824lb # [ 93.569739] niks3-server[1156]: 2026/09/22 10:36:02 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5825lb # [ 93.574862] niks3-server[1156]: 2026/09/22 10:36:02 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo5826lb # [ 93.578655] niks3-server[1156]: 2026/09/22 10:36:02 INFO Completed upload id=295827worker2 # [ 93.615217] nix-grpc-daemon[1264]: time=2026-09-22T10:36:02.642Z level=INFO msg="Upload complete. (459ms)"5828worker2 # [ 93.620392] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47978 duration_s=0 paths=1 nar_bytes_in=5365829worker2 # [ 93.627374] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:479785830worker2 # [ 93.637478] nix-daemon[873]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5831worker2 # [ 93.649154] nix-daemon[873]: reaped child process 1295, status = succeeded5832worker2 # [ 93.652568] nix-daemon[873]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5833client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...5834client # farm-a> worker2: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5835client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5836client # farm-a> LOG-farm-a-after-niks3-restart5837worker2 # [ 94.878386] nix-daemon[873]: reaped child process 1311, status = succeeded5838worker2 # [ 95.042103] nix-daemon[873]: accepted connection from pid 1327, user nix-grpc-daemon (trusted)5839worker2 # [ 95.052989] nix-daemon[873]: reaped child process 1330, status = succeeded5840lb # [ 95.032483] niks3-server[1156]: 2026/09/22 10:36:03 INFO Received uploads request method=POST path=/api/pending_closures5841worker2 # [ 95.074403] nix-grpc-daemon[1264]: time=2026-09-22T10:36:04.101Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5842worker2 # [ 95.075864] nix-grpc-daemon[1264]: time=2026-09-22T10:36:04.103Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"5843lb # [ 95.053756] niks3-server[1156]: 2026/09/22 10:36:03 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5844lb # [ 95.056300] niks3-server[1156]: 2026/09/22 10:36:03 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5845lb # [ 95.061283] niks3-server[1156]: 2026/09/22 10:36:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5846lb # [ 95.063786] niks3-server[1156]: 2026/09/22 10:36:03 INFO Signed narinfos id=30 count=15847worker2 # [ 95.100134] nix-grpc-daemon[1264]: time=2026-09-22T10:36:04.127Z level=INFO msg="Uploading 1 narinfos"5848lb # [ 95.065435] niks3-server[1156]: 2026/09/22 10:36:03 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls5849lb # [ 95.070823] niks3-server[1156]: 2026/09/22 10:36:03 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5850lb # [ 95.075754] niks3-server[1156]: 2026/09/22 10:36:03 INFO Completed upload id=305851worker2 # [ 95.112125] nix-grpc-daemon[1264]: time=2026-09-22T10:36:04.139Z level=INFO msg="Upload complete. (250ms)"5852lb # [ 95.077301] niks3-server[1156]: 2026/09/22 10:36:03 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo5853worker2 # [ 95.114860] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47978 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=15854worker2 # [ 95.121754] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:04Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=05855worker2 # [ 95.123963] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:04Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.4:50051 clients=15856worker2 # [ 95.128801] nix-daemon[873]: reaped child process 1309, status = succeeded5857worker2 # [ 95.165770] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:479785858worker2 # [ 95.174059] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:479785859worker2 # [ 95.182719] nix-grpc-daemon[1264]: time=2026-09-22T10:36:04.210Z level=INFO msg="All 1 paths already cached"5860client # copying 1 paths to 'lb:50051'...5861client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...5862worker2 # [ 95.189435] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:479785863worker2 # [ 95.198111] nix-daemon[873]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5864worker2 # [ 95.361543] nix-daemon[873]: accepted connection from pid 1336, user nix-grpc-daemon (trusted)5865worker2 # [ 95.376326] nix-daemon[873]: reaped child process 1339, status = succeeded5866worker2 # [ 95.536640] nix-daemon[873]: accepted connection from pid 1341, user nix-grpc-daemon (trusted)5867worker2 # [ 95.549266] nix-daemon[873]: reaped child process 1344, status = succeeded5868lb # [ 95.528028] niks3-server[1156]: 2026/09/22 10:36:04 INFO Received uploads request method=POST path=/api/pending_closures5869worker2 # [ 95.568802] nix-grpc-daemon[1264]: time=2026-09-22T10:36:04.596Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5870worker2 # [ 95.570319] nix-grpc-daemon[1264]: time=2026-09-22T10:36:04.598Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"5871lb # [ 95.544067] niks3-server[1156]: 2026/09/22 10:36:04 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst5872lb # [ 95.547513] niks3-server[1156]: 2026/09/22 10:36:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5873lb # [ 95.550464] niks3-server[1156]: 2026/09/22 10:36:04 INFO Signed narinfos id=31 count=15874worker2 # [ 95.588327] nix-grpc-daemon[1264]: time=2026-09-22T10:36:04.615Z level=INFO msg="Uploading 1 narinfos"5875lb # [ 95.553599] niks3-server[1156]: 2026/09/22 10:36:04 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls5876lb # [ 95.557569] niks3-server[1156]: 2026/09/22 10:36:04 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5877lb # [ 95.561515] niks3-server[1156]: 2026/09/22 10:36:04 INFO Completed upload id=315878worker2 # [ 95.597901] nix-grpc-daemon[1264]: time=2026-09-22T10:36:04.625Z level=INFO msg="Upload complete. (392ms)"5879lb # [ 95.563126] niks3-server[1156]: 2026/09/22 10:36:04 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo5880worker2 # [ 95.599899] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47978 duration_s=0 paths=1 nar_bytes_in=6565881worker2 # [ 95.603629] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:479785882worker2 # [ 95.610255] nix-daemon[873]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5883worker2 # [ 95.613311] nix-daemon[873]: reaped child process 1334, status = succeeded5884client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...5885worker2 # [ 95.619054] nix-daemon[873]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5886client # farm-b> worker2: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5887client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5888client # farm-b> LOG-farm-b-after-niks3-restart5889worker2 # [ 96.833912] nix-daemon[873]: reaped child process 1348, status = succeeded5890worker2 # [ 96.979434] nix-daemon[873]: accepted connection from pid 1364, user nix-grpc-daemon (trusted)5891worker2 # [ 96.992451] nix-daemon[873]: reaped child process 1367, status = succeeded5892lb # [ 96.968032] niks3-server[1156]: 2026/09/22 10:36:05 INFO Received uploads request method=POST path=/api/pending_closures5893worker2 # [ 97.009806] nix-grpc-daemon[1264]: time=2026-09-22T10:36:06.037Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5894worker2 # [ 97.011306] nix-grpc-daemon[1264]: time=2026-09-22T10:36:06.039Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"5895lb # [ 96.986865] niks3-server[1156]: 2026/09/22 10:36:05 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5896lb # [ 96.991592] niks3-server[1156]: 2026/09/22 10:36:05 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst5897lb # [ 96.996252] niks3-server[1156]: 2026/09/22 10:36:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5898lb # [ 96.998921] niks3-server[1156]: 2026/09/22 10:36:05 INFO Signed narinfos id=32 count=15899worker2 # [ 97.035261] nix-grpc-daemon[1264]: time=2026-09-22T10:36:06.062Z level=INFO msg="Uploading 1 narinfos"5900lb # [ 97.002965] niks3-server[1156]: 2026/09/22 10:36:05 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls5901lb # [ 97.007735] niks3-server[1156]: 2026/09/22 10:36:05 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5902lb # [ 97.011107] niks3-server[1156]: 2026/09/22 10:36:05 INFO Completed upload id=325903worker2 # [ 97.047490] nix-grpc-daemon[1264]: time=2026-09-22T10:36:06.075Z level=INFO msg="Upload complete. (221ms)"5904lb # [ 97.013088] niks3-server[1156]: 2026/09/22 10:36:05 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo5905worker2 # [ 97.050391] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47978 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=15906worker2 # [ 97.058536] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:06Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=05907worker2 # [ 97.060452] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:06Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.4:50051 clients=15908worker2 # [ 97.063568] nix-daemon[873]: reaped child process 1346, status = succeeded5909worker2 # [ 97.101863] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:479785910worker2 # [ 97.110610] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:479785911worker2 # [ 97.119334] nix-grpc-daemon[1264]: time=2026-09-22T10:36:06.146Z level=INFO msg="All 2 paths already cached"5912client # copying 1 paths to 'lb:50051'...5913client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...5914worker2 # [ 97.125251] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:479785915worker2 # [ 97.131790] nix-daemon[873]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5916worker2 # [ 97.263083] nix-daemon[873]: accepted connection from pid 1373, user nix-grpc-daemon (trusted)5917worker2 # [ 97.275779] nix-daemon[873]: reaped child process 1376, status = succeeded5918worker2 # [ 97.407204] nix-daemon[873]: accepted connection from pid 1378, user nix-grpc-daemon (trusted)5919worker2 # [ 97.419931] nix-daemon[873]: reaped child process 1381, status = succeeded5920lb # [ 97.394953] niks3-server[1156]: 2026/09/22 10:36:06 INFO Received uploads request method=POST path=/api/pending_closures5921worker2 # [ 97.436262] nix-grpc-daemon[1264]: time=2026-09-22T10:36:06.463Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5922worker2 # [ 97.437687] nix-grpc-daemon[1264]: time=2026-09-22T10:36:06.463Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"5923lb # [ 97.408061] niks3-server[1156]: 2026/09/22 10:36:06 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst5924lb # [ 97.410834] niks3-server[1156]: 2026/09/22 10:36:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5925lb # [ 97.412840] niks3-server[1156]: 2026/09/22 10:36:06 INFO Signed narinfos id=33 count=15926worker2 # [ 97.450484] nix-grpc-daemon[1264]: time=2026-09-22T10:36:06.478Z level=INFO msg="Uploading 1 narinfos"5927lb # [ 97.416831] niks3-server[1156]: 2026/09/22 10:36:06 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls5928lb # [ 97.421397] niks3-server[1156]: 2026/09/22 10:36:06 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5929lb # [ 97.424385] niks3-server[1156]: 2026/09/22 10:36:06 INFO Completed upload id=335930worker2 # [ 97.462143] nix-grpc-daemon[1264]: time=2026-09-22T10:36:06.489Z level=INFO msg="Upload complete. (322ms)"5931lb # [ 97.426657] niks3-server[1156]: 2026/09/22 10:36:06 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo5932worker2 # [ 97.464272] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47978 duration_s=0 paths=1 nar_bytes_in=7845933worker2 # [ 97.466750] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:479785934worker2 # [ 97.473965] nix-daemon[873]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5935worker2 # [ 97.476511] nix-daemon[873]: reaped child process 1371, status = succeeded5936worker2 # [ 97.482044] nix-daemon[873]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5937client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...5938client # farm-top> worker2: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5939client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5940client # farm-top> LOG-farm-top-after-niks3-restart5941worker2 # [ 98.295777] nix-daemon[873]: reaped child process 1385, status = succeeded5942worker2 # [ 98.439367] nix-daemon[873]: accepted connection from pid 1401, user nix-grpc-daemon (trusted)5943worker2 # [ 98.452645] nix-daemon[873]: reaped child process 1404, status = succeeded5944lb # [ 98.433707] niks3-server[1156]: 2026/09/22 10:36:07 INFO Received uploads request method=POST path=/api/pending_closures5945worker2 # [ 98.478291] nix-grpc-daemon[1264]: time=2026-09-22T10:36:07.505Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5946worker2 # [ 98.479780] nix-grpc-daemon[1264]: time=2026-09-22T10:36:07.507Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"5947lb # [ 98.478255] niks3-server[1156]: 2026/09/22 10:36:07 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst5948lb # [ 98.484839] niks3-server[1156]: 2026/09/22 10:36:07 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5949lb # [ 98.492827] niks3-server[1156]: 2026/09/22 10:36:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5950lb # [ 98.498129] niks3-server[1156]: 2026/09/22 10:36:07 INFO Signed narinfos id=34 count=15951worker2 # [ 98.537178] nix-grpc-daemon[1264]: time=2026-09-22T10:36:07.564Z level=INFO msg="Uploading 1 narinfos"5952lb # [ 98.501406] niks3-server[1156]: 2026/09/22 10:36:07 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls5953lb # [ 98.511444] niks3-server[1156]: 2026/09/22 10:36:07 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5954lb # [ 98.518068] niks3-server[1156]: 2026/09/22 10:36:07 INFO Completed upload id=345955worker2 # [ 98.554453] nix-grpc-daemon[1264]: time=2026-09-22T10:36:07.582Z level=INFO msg="Upload complete. (266ms)"5956lb # [ 98.519820] niks3-server[1156]: 2026/09/22 10:36:07 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo5957worker2 # [ 98.563167] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47978 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=15958worker2 # [ 98.575987] nix-daemon[873]: reaped child process 1383, status = succeeded5959client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&2, in 5.82 seconds)5960client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath5961lb # [ 98.643911] envoy[923]: conn peer=[2001:db8:1::1]:45280 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1005 tx=282 ms=11675962lb # [ 98.647720] envoy[923]: conn peer=[2001:db8:1::1]:60906 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9735 tx=4467 ms=55585963client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath, in 0.21 seconds)5964(finished: subtest: a niks3 restart does not cost the leader its role, in 6.58 seconds)5965subtest: scheduler down: the other node takes the lock and keeps it5966lb: must succeed: curl -sf localhost:9901/clusters5967lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5968lb: must succeed: curl -sf localhost:9901/clusters5969lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5970lb: must succeed: curl -sf localhost:9901/clusters5971lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5972lb: must succeed: curl -sf localhost:9901/clusters5973lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5974worker2 # [ 99.077890] systemd[1]: Stopping Nix worker-protocol over gRPC...5975worker2 # [ 99.614657] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:08Z level=info event=signal_exit5976lb # [ 100.547840] niks3-server[1156]: 2026/09/22 10:36:09 INFO lead: released remote=192.168.1.4:404005977worker2 # [ 101.584870] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:10Z level=info event=scheduler_restarting5978worker2 # [ 101.586424] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:10Z level=info event=scheduler_restarting addr=lb:500515979worker1 # [ 101.565401] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:09Z level=info event=scheduler_restarting addr=lb:500515980worker2 # [ 101.587825] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:10Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:479765981worker2 # [ 101.589635] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:10Z level=info event=worker_gone addr=192.168.1.4:500515982worker2 # [ 101.590880] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:10Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:479765983worker2 # [ 101.592229] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:10Z level=info event=worker_gone addr=192.168.1.3:500515984worker2 # [ 101.607471] nix-daemon[873]: reaped child process 1256, status = succeeded5985worker2 # [ 101.649690] nix-grpc-daemon[1204]: E0000 00:00:1790073370.677102 1249 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5986worker2 # [ 101.654526] nix-grpc-daemon[1204]: E0000 00:00:1790073370.677332 1249 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true5987worker1 # [ 101.764740] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:09Z level=info event=scheduler_connected addr=lb:500515988worker1 # [ 101.768678] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5989worker2 # [ 101.794741] nix-grpc-daemon[1204]: ts=2026-09-22T10:36:10Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5990worker1 # [ 101.969672] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:10Z level=info event=scheduler_connected addr=lb:500515991worker1 # [ 101.973706] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5992worker1 # [ 102.174444] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:10Z level=info event=scheduler_connected addr=lb:500515993worker1 # [ 102.178298] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5994worker1 # [ 102.378569] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:10Z level=info event=scheduler_connected addr=lb:500515995worker1 # [ 102.380337] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5996worker1 # [ 102.581320] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:10Z level=info event=scheduler_connected addr=lb:500515997worker1 # [ 102.582889] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5998worker2 # [ 102.671116] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5999worker2 # [ 102.673346] systemd[1]: Stopped Nix worker-protocol over gRPC.6000worker2 # [ 102.674842] systemd[1]: nix-grpc-daemon.service: Consumed 1.068s CPU time over 16.217s wall clock time, 35M memory peak, 45.9K incoming IP traffic, 83.5K outgoing IP traffic.6001worker2 # [ 102.680484] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.6002worker2 # [ 102.682733] systemd[1]: Closed Nix worker-protocol over gRPC.6003worker2 # [ 102.683897] systemd[1]: nix-grpc-daemon.socket: Consumed 295us CPU time over 16.870s wall clock time, 256K memory peak, 38.3K incoming IP traffic, 28.2K outgoing IP traffic.6004worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over6005worker1 # [ 102.783579] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:11Z level=info event=scheduler_connected addr=lb:500516006worker1 # [ 102.785457] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6007worker1 # [ 102.986127] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:11Z level=info event=scheduler_connected addr=lb:500516008worker1 # [ 102.987874] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6009worker1 # [ 103.188564] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:11Z level=info event=scheduler_connected addr=lb:500516010worker1 # [ 103.190421] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6011worker1 # [ 103.357247] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:11Z level=info event=scheduler_take_over6012lb # [ 103.341174] niks3-server[1156]: 2026/09/22 10:36:12 INFO lead: acquired remote=192.168.1.3:516206013worker1 # [ 103.391080] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:11Z level=info event=scheduler_connected addr=lb:500516014worker1 # [ 103.394745] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6015worker1 # [ 103.596888] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:11Z level=info event=scheduler_connected addr=lb:500516016worker1 # [ 103.600333] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6017worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 1.11 seconds)6018worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6019worker1 # [ 103.800469] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:12Z level=info event=scheduler_connected addr=lb:500516020worker1 # [ 103.802316] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6021worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6022worker1 # [ 104.003331] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:12Z level=info event=scheduler_connected addr=lb:500516023worker1 # [ 104.007211] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6024worker1 # [ 104.207933] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:12Z level=info event=scheduler_connected addr=lb:500516025worker1 # [ 104.211857] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6026worker1 # [ 104.412966] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:12Z level=info event=scheduler_connected addr=lb:500516027worker1 # [ 104.420344] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:12Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:328766028worker1 # [ 104.425341] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:12Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06029worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6030worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6031client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&26032worker1 # [ 105.088741] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:596126033client # don't know how to build these paths:6034client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6035worker1 # [ 105.097432] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596126036worker1 # [ 105.108693] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 67 ms (attempt 1/5)6037worker1 # [ 105.181658] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 86 ms (attempt 2/5)6038worker1 # [ 105.273628] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 17 ms (attempt 3/5)6039worker1 # [ 105.296568] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 672 ms (attempt 4/5)6040worker1 # [ 105.974279] nix-grpc-daemon[1208]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6041worker1 # [ 105.976444] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6042worker1 # [ 105.999701] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:14Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=06043worker1 # [ 106.002270] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:14Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=16044worker1 # [ 106.049398] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596126045worker1 # [ 106.057899] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596126046client # copying 1 paths to 'lb:50051'...6047worker1 # [ 106.066862] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596126048client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...6049worker1 # [ 106.073928] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6050worker1 # [ 106.084467] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:14Z level=info event=niks3_push_started pid=12906051worker1 # [ 106.262636] nix-grpc-daemon[1290]: time=2026-09-22T10:36:14.483Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6052worker1 # [ 106.426649] nix-daemon[851]: accepted connection from pid 1303, user nix-grpc-daemon (trusted)6053worker1 # [ 106.437522] nix-daemon[851]: reaped child process 1306, status = succeeded6054worker1 # [ 106.608482] nix-daemon[851]: accepted connection from pid 1308, user nix-grpc-daemon (trusted)6055worker1 # [ 106.620655] nix-daemon[851]: reaped child process 1311, status = succeeded6056lb # [ 106.619529] niks3-server[1156]: 2026/09/22 10:36:15 INFO Received uploads request method=POST path=/api/pending_closures6057worker1 # [ 106.640200] nix-grpc-daemon[1290]: time=2026-09-22T10:36:14.861Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6058worker1 # [ 106.641850] nix-grpc-daemon[1290]: time=2026-09-22T10:36:14.863Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"6059lb # [ 106.639946] niks3-server[1156]: 2026/09/22 10:36:15 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst6060lb # [ 106.643614] niks3-server[1156]: 2026/09/22 10:36:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign6061lb # [ 106.646199] niks3-server[1156]: 2026/09/22 10:36:15 INFO Signed narinfos id=35 count=16062worker1 # [ 106.662260] nix-grpc-daemon[1290]: time=2026-09-22T10:36:14.883Z level=INFO msg="Uploading 1 narinfos"6063lb # [ 106.650404] niks3-server[1156]: 2026/09/22 10:36:15 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls6064lb # [ 106.654146] niks3-server[1156]: 2026/09/22 10:36:15 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete6065lb # [ 106.657360] niks3-server[1156]: 2026/09/22 10:36:15 INFO Completed upload id=356066worker1 # [ 106.674818] nix-grpc-daemon[1290]: time=2026-09-22T10:36:14.896Z level=INFO msg="Upload complete. (407ms)"6067lb # [ 106.661518] niks3-server[1156]: 2026/09/22 10:36:15 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo6068worker1 # [ 106.677926] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59612 duration_s=0 paths=1 nar_bytes_in=5206069worker1 # [ 106.680529] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596126070worker1 # [ 106.686964] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6071worker1 # [ 106.689138] nix-daemon[851]: reaped child process 1288, status = succeeded6072worker1 # [ 106.694287] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6073client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...6074client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6075client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6076client # farm-a> LOG-farm-a-on-standby6077worker1 # [ 107.907244] nix-daemon[851]: reaped child process 1315, status = succeeded6078worker1 # [ 108.047333] nix-daemon[851]: accepted connection from pid 1331, user nix-grpc-daemon (trusted)6079worker1 # [ 108.057889] nix-daemon[851]: reaped child process 1334, status = succeeded6080lb # [ 108.057160] niks3-server[1156]: 2026/09/22 10:36:16 INFO Received uploads request method=POST path=/api/pending_closures6081worker1 # [ 108.082662] nix-grpc-daemon[1290]: time=2026-09-22T10:36:16.303Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6082worker1 # [ 108.084151] nix-grpc-daemon[1290]: time=2026-09-22T10:36:16.305Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"6083lb # [ 108.091979] niks3-server[1156]: 2026/09/22 10:36:16 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6084lb # [ 108.107149] niks3-server[1156]: 2026/09/22 10:36:16 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst6085lb # [ 108.116306] niks3-server[1156]: 2026/09/22 10:36:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign6086lb # [ 108.121906] niks3-server[1156]: 2026/09/22 10:36:17 INFO Signed narinfos id=36 count=16087worker1 # [ 108.141025] nix-grpc-daemon[1290]: time=2026-09-22T10:36:16.361Z level=INFO msg="Uploading 1 narinfos"6088lb # [ 108.126313] niks3-server[1156]: 2026/09/22 10:36:17 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls6089lb # [ 108.135571] niks3-server[1156]: 2026/09/22 10:36:17 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete6090lb # [ 108.141838] niks3-server[1156]: 2026/09/22 10:36:17 INFO Completed upload id=366091worker1 # [ 108.158341] nix-grpc-daemon[1290]: time=2026-09-22T10:36:16.379Z level=INFO msg="Upload complete. (257ms)"6092lb # [ 108.143461] niks3-server[1156]: 2026/09/22 10:36:17 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo6093worker1 # [ 108.166516] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59612 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=16094worker1 # [ 108.182479] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:16Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=06095worker1 # [ 108.185345] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:16Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=16096worker1 # [ 108.188177] nix-daemon[851]: reaped child process 1313, status = succeeded6097worker1 # [ 108.227236] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596126098worker1 # [ 108.239669] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596126099worker1 # [ 108.253446] nix-grpc-daemon[1290]: time=2026-09-22T10:36:16.474Z level=INFO msg="All 1 paths already cached"6100client # copying 1 paths to 'lb:50051'...6101client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...6102worker1 # [ 108.261117] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596126103worker1 # [ 108.269300] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6104worker1 # [ 108.432409] nix-daemon[851]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)6105worker1 # [ 108.444320] nix-daemon[851]: reaped child process 1343, status = succeeded6106worker1 # [ 108.603055] nix-daemon[851]: accepted connection from pid 1345, user nix-grpc-daemon (trusted)6107worker1 # [ 108.614892] nix-daemon[851]: reaped child process 1348, status = succeeded6108lb # [ 108.615059] niks3-server[1156]: 2026/09/22 10:36:17 INFO Received uploads request method=POST path=/api/pending_closures6109worker1 # [ 108.640061] nix-grpc-daemon[1290]: time=2026-09-22T10:36:16.861Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6110worker1 # [ 108.641423] nix-grpc-daemon[1290]: time=2026-09-22T10:36:16.861Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"6111lb # [ 108.641228] niks3-server[1156]: 2026/09/22 10:36:17 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst6112lb # [ 108.646993] envoy[923]: conn peer=[2001:db8:1::4]:34398 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=5880 tx=648 ms=151306113lb # [ 108.658336] niks3-server[1156]: 2026/09/22 10:36:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6114lb # [ 108.664423] niks3-server[1156]: 2026/09/22 10:36:17 INFO Signed narinfos id=37 count=16115worker1 # [ 108.683998] nix-grpc-daemon[1290]: time=2026-09-22T10:36:16.905Z level=INFO msg="Uploading 1 narinfos"6116lb # [ 108.669583] niks3-server[1156]: 2026/09/22 10:36:17 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls6117lb # [ 108.679435] niks3-server[1156]: 2026/09/22 10:36:17 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6118lb # [ 108.687985] niks3-server[1156]: 2026/09/22 10:36:17 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo6119lb # [ 108.693541] niks3-server[1156]: 2026/09/22 10:36:17 INFO Completed upload id=376120worker1 # [ 108.709966] nix-grpc-daemon[1290]: time=2026-09-22T10:36:16.931Z level=INFO msg="Upload complete. (428ms)"6121worker1 # [ 108.712326] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59612 duration_s=0 paths=1 nar_bytes_in=6406122worker1 # [ 108.714729] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596126123worker1 # [ 108.720292] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6124worker1 # [ 108.724588] nix-daemon[851]: reaped child process 1338, status = succeeded6125worker1 # [ 108.729361] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6126client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...6127client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6128client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6129client # farm-b> LOG-farm-b-on-standby6130worker1 # [ 109.743064] nix-daemon[851]: reaped child process 1352, status = succeeded6131worker1 # [ 109.889309] nix-daemon[851]: accepted connection from pid 1368, user nix-grpc-daemon (trusted)6132worker1 # [ 109.901211] nix-daemon[851]: reaped child process 1371, status = succeeded6133lb # [ 109.901588] niks3-server[1156]: 2026/09/22 10:36:18 INFO Received uploads request method=POST path=/api/pending_closures6134worker1 # [ 109.926200] nix-grpc-daemon[1290]: time=2026-09-22T10:36:18.147Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6135worker1 # [ 109.928051] nix-grpc-daemon[1290]: time=2026-09-22T10:36:18.149Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"6136lb # [ 109.944624] niks3-server[1156]: 2026/09/22 10:36:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6137lb # [ 109.952324] niks3-server[1156]: 2026/09/22 10:36:18 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst6138lb # [ 109.961894] niks3-server[1156]: 2026/09/22 10:36:18 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls6139lb # [ 109.966923] niks3-server[1156]: 2026/09/22 10:36:18 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6140lb # [ 109.974508] niks3-server[1156]: 2026/09/22 10:36:18 INFO Signed narinfos id=38 count=16141worker1 # [ 109.993529] nix-grpc-daemon[1290]: time=2026-09-22T10:36:18.214Z level=INFO msg="Uploading 1 narinfos"6142lb # [ 109.983860] niks3-server[1156]: 2026/09/22 10:36:18 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6143lb # [ 109.992275] niks3-server[1156]: 2026/09/22 10:36:18 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo6144lb # [ 109.995666] niks3-server[1156]: 2026/09/22 10:36:18 INFO Completed upload id=386145worker1 # [ 110.012181] nix-grpc-daemon[1290]: time=2026-09-22T10:36:18.233Z level=INFO msg="Upload complete. (276ms)"6146worker1 # [ 110.017629] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59612 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=16147worker1 # [ 110.027534] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:18Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=06148worker1 # [ 110.031405] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:18Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=16149worker1 # [ 110.039799] nix-daemon[851]: reaped child process 1350, status = succeeded6150worker1 # [ 110.071842] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596126151worker1 # [ 110.082421] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596126152worker1 # [ 110.094328] nix-grpc-daemon[1290]: time=2026-09-22T10:36:18.315Z level=INFO msg="All 2 paths already cached"6153client # copying 1 paths to 'lb:50051'...6154client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...6155worker1 # [ 110.101148] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596126156worker1 # [ 110.107792] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6157worker1 # [ 110.272970] nix-daemon[851]: accepted connection from pid 1377, user nix-grpc-daemon (trusted)6158worker1 # [ 110.285246] nix-daemon[851]: reaped child process 1380, status = succeeded6159worker1 # [ 110.429809] nix-daemon[851]: accepted connection from pid 1382, user nix-grpc-daemon (trusted)6160worker1 # [ 110.441576] nix-daemon[851]: reaped child process 1385, status = succeeded6161lb # [ 110.440175] niks3-server[1156]: 2026/09/22 10:36:19 INFO Received uploads request method=POST path=/api/pending_closures6162worker1 # [ 110.466158] nix-grpc-daemon[1290]: time=2026-09-22T10:36:18.687Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6163worker1 # [ 110.467724] nix-grpc-daemon[1290]: time=2026-09-22T10:36:18.689Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6164lb # [ 110.471492] niks3-server[1156]: 2026/09/22 10:36:19 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6165lb # [ 110.483160] niks3-server[1156]: 2026/09/22 10:36:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6166lb # [ 110.488364] niks3-server[1156]: 2026/09/22 10:36:19 INFO Signed narinfos id=39 count=16167worker1 # [ 110.507414] nix-grpc-daemon[1290]: time=2026-09-22T10:36:18.728Z level=INFO msg="Uploading 1 narinfos"6168lb # [ 110.494625] niks3-server[1156]: 2026/09/22 10:36:19 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6169lb # [ 110.508081] niks3-server[1156]: 2026/09/22 10:36:19 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6170lb # [ 110.512531] niks3-server[1156]: 2026/09/22 10:36:19 INFO Completed upload id=396171worker1 # [ 110.531062] nix-grpc-daemon[1290]: time=2026-09-22T10:36:18.752Z level=INFO msg="Upload complete. (411ms)"6172lb # [ 110.515082] niks3-server[1156]: 2026/09/22 10:36:19 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6173worker1 # [ 110.534560] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59612 duration_s=0 paths=1 nar_bytes_in=7686174worker1 # [ 110.539136] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596126175worker1 # [ 110.548605] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6176worker1 # [ 110.553237] nix-daemon[851]: reaped child process 1375, status = succeeded6177client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6178worker1 # [ 110.559256] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6179client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6180client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6181client # farm-top> LOG-farm-top-on-standby6182worker1 # [ 111.984405] nix-daemon[851]: reaped child process 1389, status = succeeded6183worker1 # [ 112.131491] nix-daemon[851]: accepted connection from pid 1405, user nix-grpc-daemon (trusted)6184worker1 # [ 112.142209] nix-daemon[851]: reaped child process 1408, status = succeeded6185lb # [ 112.142347] niks3-server[1156]: 2026/09/22 10:36:21 INFO Received uploads request method=POST path=/api/pending_closures6186worker1 # [ 112.166768] nix-grpc-daemon[1290]: time=2026-09-22T10:36:20.388Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6187worker1 # [ 112.168378] nix-grpc-daemon[1290]: time=2026-09-22T10:36:20.389Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6188lb # [ 112.178691] niks3-server[1156]: 2026/09/22 10:36:21 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6189lb # [ 112.186212] niks3-server[1156]: 2026/09/22 10:36:21 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6190lb # [ 112.197742] niks3-server[1156]: 2026/09/22 10:36:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6191lb # [ 112.202959] niks3-server[1156]: 2026/09/22 10:36:21 INFO Signed narinfos id=40 count=16192worker1 # [ 112.222341] nix-grpc-daemon[1290]: time=2026-09-22T10:36:20.443Z level=INFO msg="Uploading 1 narinfos"6193lb # [ 112.208048] niks3-server[1156]: 2026/09/22 10:36:21 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6194lb # [ 112.216337] niks3-server[1156]: 2026/09/22 10:36:21 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6195lb # [ 112.223365] niks3-server[1156]: 2026/09/22 10:36:21 INFO Completed upload id=406196worker1 # [ 112.239819] nix-grpc-daemon[1290]: time=2026-09-22T10:36:20.461Z level=INFO msg="Upload complete. (272ms)"6197lb # [ 112.225933] niks3-server[1156]: 2026/09/22 10:36:21 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6198worker1 # [ 112.247524] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59612 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16199worker1 # [ 112.260313] nix-daemon[851]: reaped child process 1387, status = succeeded6200client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 7.40 seconds)6201client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath6202client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.19 seconds)6203worker2 # [ 112.555592] systemd[1]: Listening on Nix worker-protocol over gRPC.6204worker2 # [ 112.562374] systemd[1]: Starting scheduler-token.service...6205worker2 # [ 112.637628] systemd[1]: scheduler-token.service: Deactivated successfully.6206worker2 # [ 112.639535] systemd[1]: Finished scheduler-token.service.6207worker2 # [ 112.642185] systemd[1]: scheduler-token.service: Consumed 19ms CPU time over 77ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6208worker2 # [ 112.647262] systemd[1]: Starting Nix worker-protocol over gRPC...6209worker2 # [ 112.795985] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:21Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6210worker2 # [ 112.798342] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:21Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6211worker2 # [ 112.801832] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:21Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16212worker2 # [ 112.808128] nix-grpc-daemon[1425]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6213worker2 # [ 112.812524] nix-grpc-daemon[1425]: E0000 00:00:1790073381.835606 1425 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6214worker2 # [ 112.816148] nix-grpc-daemon[1425]: E0000 00:00:1790073381.838567 1425 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6215worker2 # [ 112.818186] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:21Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6216worker2 # [ 112.819704] systemd[1]: Started Nix worker-protocol over gRPC.6217lb: must succeed: curl -sf localhost:9901/clusters6218worker2 # [ 112.830123] nix-grpc-daemon[1425]: E0000 00:00:1790073381.857853 1466 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6219worker2 # [ 112.843495] nix-grpc-daemon[1425]: E0000 00:00:1790073381.871166 1466 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6220worker2 # [ 112.847217] nix-grpc-daemon[1425]: E0000 00:00:1790073381.873269 1466 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6221worker2 # [ 112.857380] nix-grpc-daemon[1425]: E0000 00:00:1790073381.885053 1466 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6222worker2 # [ 112.866774] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:21Z level=info event=scheduler_connected addr=lb:500516223worker1 # [ 112.853095] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:21Z level=info event=oidc_jwks provider=mock keys=16224worker1 # [ 112.855265] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:21Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:328766225worker1 # [ 112.857138] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:21Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06226lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6227worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6228worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6229lb: must succeed: curl -sf localhost:9901/clusters6230lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6231lb: must succeed: curl -sf localhost:9901/clusters6232lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6233worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6234worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6235(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 14.30 seconds)6236subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6237client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs6238client # [ 113.133816] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.6239client # Running as unit: rs.service; invocation ID: af825dace6294988acd1b2cbfe554a686240client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.04 seconds)6241worker1 # [ 113.344192] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:21Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:596126242client # [ 113.346591] nix[1301]: don't know how to build these paths:6243client # [ 113.348130] nix[1301]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6244worker1 # [ 113.357220] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:21Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06245worker1 # [ 113.362516] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:21Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=16246worker1 # [ 113.405172] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596126247worker1 # [ 113.417557] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596126248client # [ 113.424238] nix[1301]: copying 1 paths to 'lb:50051'...6249client # [ 113.427959] nix[1301]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6250worker1 # [ 113.431497] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596126251worker1 # [ 113.439936] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6252worker1 # [ 113.617844] nix-daemon[851]: accepted connection from pid 1421, user nix-grpc-daemon (trusted)6253worker1 # [ 113.629915] nix-daemon[851]: reaped child process 1424, status = succeeded6254worker1 # [ 113.791577] nix-daemon[851]: accepted connection from pid 1426, user nix-grpc-daemon (trusted)6255worker1 # [ 113.803104] nix-daemon[851]: reaped child process 1429, status = succeeded6256lb # [ 113.803344] niks3-server[1156]: 2026/09/22 10:36:22 INFO Received uploads request method=POST path=/api/pending_closures6257worker1 # [ 113.832747] nix-grpc-daemon[1290]: time=2026-09-22T10:36:22.053Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6258worker1 # [ 113.834247] nix-grpc-daemon[1290]: time=2026-09-22T10:36:22.053Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6259lb # [ 113.845145] niks3-server[1156]: 2026/09/22 10:36:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6260lb # [ 113.850657] niks3-server[1156]: 2026/09/22 10:36:22 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6261lb # [ 113.858683] niks3-server[1156]: 2026/09/22 10:36:22 INFO Signed narinfos id=41 count=16262worker1 # [ 113.878611] nix-grpc-daemon[1290]: time=2026-09-22T10:36:22.099Z level=INFO msg="Uploading 1 narinfos"6263lb # [ 113.873991] niks3-server[1156]: 2026/09/22 10:36:22 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6264lb # [ 113.879617] niks3-server[1156]: 2026/09/22 10:36:22 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6265lb # [ 113.895044] niks3-server[1156]: 2026/09/22 10:36:22 INFO Completed upload id=416266worker1 # [ 113.912825] nix-grpc-daemon[1290]: time=2026-09-22T10:36:22.134Z level=INFO msg="Upload complete. (458ms)"6267lb # [ 113.903041] niks3-server[1156]: 2026/09/22 10:36:22 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6268worker1 # [ 113.917635] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59612 duration_s=0 paths=1 nar_bytes_in=4166269worker1 # [ 113.923710] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596126270worker1 # [ 113.934236] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6271worker1 # [ 113.944959] nix-daemon[851]: reaped child process 1419, status = succeeded6272client # [ 113.947359] nix[1301]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6273worker1 # [ 113.952250] nix-daemon[851]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)6274lb: must succeed: curl -sf localhost:9901/clusters6275lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6276lb: must succeed: curl -sf localhost:9901/clusters6277lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6278lb: must succeed: curl -sf localhost:9901/clusters6279lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6280lb: must succeed: curl -sf localhost:9901/clusters6281lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6282worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6283worker2 # [ 115.908979] systemd-journald[288]: Received client request to rotate journal, rotating.6284worker2 # [ 115.919682] systemd-journald[288]: Vacuuming done, freed 0B of archived journals from /var/log/journal/3e66d44f52864b08ba93cecc2997050c.6285worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.04 seconds)6286worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service6287worker1 # [ 115.808727] systemd[1]: Stopping Nix worker-protocol over gRPC...6288worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)6289client: must succeed: sleep 5.06290worker1 # [ 116.078513] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:24Z level=info event=draining6291lb # [ 118.647924] envoy[923]: conn peer=[2001:db8:1::1]:52172 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9620 tx=4460 ms=71696292client: (finished: must succeed: sleep 5.0, in 5.05 seconds)6293worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6294worker1 # [ 121.176527] nix-daemon[851]: reaped child process 1433, status = succeeded6295worker1 # [ 121.311282] nix-daemon[851]: accepted connection from pid 1474, user nix-grpc-daemon (trusted)6296worker1 # [ 121.322105] nix-daemon[851]: reaped child process 1477, status = succeeded6297lb # [ 121.313729] niks3-server[1156]: 2026/09/22 10:36:30 INFO Received uploads request method=POST path=/api/pending_closures6298worker1 # [ 121.335230] nix-grpc-daemon[1290]: time=2026-09-22T10:36:29.556Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6299worker1 # [ 121.336802] nix-grpc-daemon[1290]: time=2026-09-22T10:36:29.558Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6300lb # [ 121.341394] niks3-server[1156]: 2026/09/22 10:36:30 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6301lb # [ 121.345847] niks3-server[1156]: 2026/09/22 10:36:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6302worker1 # [ 121.365904] nix-grpc-daemon[1290]: time=2026-09-22T10:36:29.587Z level=INFO msg="Uploading 1 narinfos"6303lb # [ 121.350227] niks3-server[1156]: 2026/09/22 10:36:30 INFO Signed narinfos id=42 count=16304lb # [ 121.353163] niks3-server[1156]: 2026/09/22 10:36:30 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6305lb # [ 121.356671] niks3-server[1156]: 2026/09/22 10:36:30 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6306lb # [ 121.362139] niks3-server[1156]: 2026/09/22 10:36:30 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6307lb # [ 121.366114] niks3-server[1156]: 2026/09/22 10:36:30 INFO Completed upload id=426308worker1 # [ 121.382254] nix-grpc-daemon[1290]: time=2026-09-22T10:36:29.603Z level=INFO msg="Upload complete. (212ms)"6309lb # [ 121.367739] niks3-server[1156]: 2026/09/22 10:36:30 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6310worker1 # [ 121.384590] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59612 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16311worker1 # [ 121.392036] nix-daemon[851]: reaped child process 1431, status = succeeded6312client # [ 121.417934] systemd[1]: rs.service: Deactivated successfully.6313client # [ 121.421654] systemd[1]: rs.service: Consumed 122ms CPU time over 8.287s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6314worker1 # [ 122.078976] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:30Z level=info event=signal_exit6315lb # [ 122.573557] niks3-server[1156]: 2026/09/22 10:36:31 INFO lead: released remote=192.168.1.3:516206316lb # [ 122.791145] niks3-server[1156]: 2026/09/22 10:36:31 INFO lead: acquired remote=192.168.1.4:463806317worker2 # [ 122.830312] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:31Z level=info event=scheduler_take_over6318worker2 # [ 123.609569] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:32Z level=info event=scheduler_restarting addr=lb:500516319worker1 # [ 123.594142] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:31Z level=info event=scheduler_restarting6320worker1 # [ 123.597630] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:31Z level=info event=scheduler_restarting addr=lb:500516321worker1 # [ 123.600929] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:31Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:328766322worker1 # [ 123.604790] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:31Z level=info event=worker_gone addr=192.168.1.4:500516323worker1 # [ 123.608290] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:31Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:328766324worker1 # [ 123.611927] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:31Z level=info event=worker_gone addr=192.168.1.3:500516325worker1 # [ 123.627620] nix-daemon[851]: reaped child process 1260, status = succeeded6326worker2 # [ 123.810925] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:32Z level=info event=scheduler_connected addr=lb:500516327worker1 # [ 123.797258] nix-grpc-daemon[1208]: ts=2026-09-22T10:36:32Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6328worker2 # [ 123.815192] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6329worker2 # [ 124.017241] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:33Z level=info event=scheduler_connected addr=lb:500516330worker2 # [ 124.029165] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:33Z level=info event=oidc_jwks provider=mock keys=16331worker2 # [ 124.033148] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:33Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:366206332worker2 # [ 124.037114] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:33Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06333worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.20 seconds)6334client: waiting for success: ! systemctl is-active rs6335client: (finished: waiting for success: ! systemctl is-active rs, in 0.05 seconds)6336client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6337worker1 # [ 124.359204] nix-grpc-daemon[1208]: E0000 00:00:1790073392.580415 1253 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true6338worker1 # [ 124.364062] nix-grpc-daemon[1208]: E0000 00:00:1790073392.580563 1253 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6339client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)6340client: must fail: journalctl -u rs | grep -q 'reconnecting'6341client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.02 seconds)6342lb: must succeed: curl -sf localhost:9901/clusters6343lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6344worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6345worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6346worker1 # [ 124.970878] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6347worker1 # [ 124.973347] systemd[1]: Stopped Nix worker-protocol over gRPC.6348worker1 # [ 124.975439] systemd[1]: nix-grpc-daemon.service: Consumed 1.288s CPU time over 34.077s wall clock time, 35.6M memory peak, 47.5K incoming IP traffic, 94.5K outgoing IP traffic.6349worker1 # [ 124.991277] systemd[1]: Starting Nix worker-protocol over gRPC...6350worker1 # [ 125.156831] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:33Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6351worker1 # [ 125.158543] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:33Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6352worker1 # [ 125.162482] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:33Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16353worker1 # [ 125.169063] nix-grpc-daemon[1484]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6354worker1 # [ 125.170368] nix-grpc-daemon[1484]: E0000 00:00:1790073393.389934 1484 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6355worker1 # [ 125.175305] nix-grpc-daemon[1484]: E0000 00:00:1790073393.394945 1484 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6356worker1 # [ 125.178373] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:33Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6357worker1 # [ 125.180352] systemd[1]: Started Nix worker-protocol over gRPC.6358worker1 # [ 125.186066] nix-grpc-daemon[1484]: E0000 00:00:1790073393.407059 1523 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6359worker1 # [ 125.197471] nix-grpc-daemon[1484]: E0000 00:00:1790073393.418927 1523 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6360worker1 # [ 125.202053] nix-grpc-daemon[1484]: E0000 00:00:1790073393.421124 1523 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6361worker1 # [ 125.211289] nix-grpc-daemon[1484]: E0000 00:00:1790073393.432752 1523 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6362worker1 # [ 125.218703] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:33Z level=info event=scheduler_connected addr=lb:500516363worker2 # [ 125.241241] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:34Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:366206364worker2 # [ 125.245708] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:34Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06365lb: must succeed: curl -sf localhost:9901/clusters6366lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6367worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6368worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6369(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.52 seconds)6370subtest: scheduler restart mid-build: build finishes, no double build6371client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid6372client # [ 125.673879] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.6373client # Running as unit: mid.service; invocation ID: e6b773f992b54f59908dd84fee46b4f66374client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)6375worker2 # [ 125.931169] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:366306376worker2 # [ 125.943774] nix-daemon[873]: accepted connection from pid 1425, user nix-grpc-daemon (trusted)6377client # [ 125.932449] nix[1337]: don't know how to build these paths:6378client # [ 125.933343] nix[1337]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6379worker2 # [ 125.962686] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:34Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06380worker2 # [ 125.967673] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:34Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=16381worker2 # [ 126.012718] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366306382worker2 # [ 126.025881] nix-grpc-daemon[1425]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 59 ms (attempt 1/5)6383worker2 # [ 126.091504] nix-grpc-daemon[1425]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 155 ms (attempt 2/5)6384worker2 # [ 126.252669] nix-grpc-daemon[1425]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 149 ms (attempt 3/5)6385worker2 # [ 126.401510] nix-grpc-daemon[1425]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 182 ms (attempt 4/5)6386worker2 # [ 126.586579] nix-grpc-daemon[1425]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6387worker2 # [ 126.590173] nix-grpc-daemon[1425]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6388worker2 # [ 126.604821] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366306389client # [ 126.590324] nix[1337]: copying 1 paths to 'lb:50051'...6390worker2 # [ 126.614469] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366306391client # [ 126.594184] nix[1337]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6392worker2 # [ 126.622919] nix-daemon[873]: accepted connection from pid 1425, user nix-grpc-daemon (trusted)6393worker2 # [ 126.643521] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:35Z level=info event=niks3_push_started pid=15536394worker2 # [ 126.819559] nix-grpc-daemon[1553]: time=2026-09-22T10:36:35.846Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6395worker2 # [ 126.975801] nix-daemon[873]: accepted connection from pid 1566, user nix-grpc-daemon (trusted)6396worker2 # [ 126.987418] nix-daemon[873]: reaped child process 1573, status = succeeded6397worker2 # [ 127.140960] nix-daemon[873]: accepted connection from pid 1575, user nix-grpc-daemon (trusted)6398worker2 # [ 127.153911] nix-daemon[873]: reaped child process 1578, status = succeeded6399lb # [ 127.133552] niks3-server[1156]: 2026/09/22 10:36:36 INFO Received uploads request method=POST path=/api/pending_closures6400worker2 # [ 127.174140] nix-grpc-daemon[1553]: time=2026-09-22T10:36:36.201Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6401worker2 # [ 127.176423] nix-grpc-daemon[1553]: time=2026-09-22T10:36:36.203Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6402lb # [ 127.152674] niks3-server[1156]: 2026/09/22 10:36:36 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6403lb # [ 127.156102] niks3-server[1156]: 2026/09/22 10:36:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6404lb # [ 127.157926] niks3-server[1156]: 2026/09/22 10:36:36 INFO Signed narinfos id=43 count=16405worker2 # [ 127.195845] nix-grpc-daemon[1553]: time=2026-09-22T10:36:36.223Z level=INFO msg="Uploading 1 narinfos"6406lb # [ 127.162522] niks3-server[1156]: 2026/09/22 10:36:36 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6407lb # [ 127.166539] niks3-server[1156]: 2026/09/22 10:36:36 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6408lb # [ 127.169128] niks3-server[1156]: 2026/09/22 10:36:36 INFO Completed upload id=436409worker2 # [ 127.205408] nix-grpc-daemon[1553]: time=2026-09-22T10:36:36.233Z level=INFO msg="Upload complete. (381ms)"6410worker2 # [ 127.207566] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36630 duration_s=0 paths=1 nar_bytes_in=4246411lb # [ 127.172046] niks3-server[1156]: 2026/09/22 10:36:36 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6412worker2 # [ 127.210551] nix-grpc-daemon[1425]: ts=2026-09-22T10:36:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366306413worker2 # [ 127.217520] nix-daemon[873]: accepted connection from pid 1425, user nix-grpc-daemon (trusted)6414worker2 # [ 127.220320] nix-daemon[873]: reaped child process 1551, status = succeeded6415worker2 # [ 127.225082] nix-daemon[873]: accepted connection from pid 1425, user nix-grpc-daemon (trusted)6416client # [ 127.201649] nix[1337]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6417lb # [ 128.648657] envoy[923]: conn peer=[2001:db8:1::1]:41858 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2973 tx=1515 ms=80556418lb # [ 128.652551] envoy[923]: conn peer=[2001:db8:1::3]:57316 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=4183 tx=1027 ms=326826419lb: must succeed: curl -sf localhost:9901/clusters6420lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6421lb: must succeed: curl -sf localhost:9901/clusters6422lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6423worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6424worker2 # [ 129.349445] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1425 (nix-grpc-daemon) on client request.6425worker2 # [ 129.353375] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6426lb # [ 129.322441] niks3-server[1156]: 2026/09/22 10:36:38 INFO lead: released remote=192.168.1.4:463806427worker1 # [ 129.342906] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:37Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6428client # [ 129.340083] nix[1337]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6429client # [ 129.342108] nix[1337]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6430worker2 # [ 129.372803] systemd[1]: Starting scheduler-token.service...6431worker2 # [ 129.381045] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6432worker2 # [ 129.382729] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6433worker2 # [ 129.384630] systemd[1]: nix-grpc-daemon.service: Consumed 428ms CPU time over 16.729s wall clock time, 33.8M memory peak, 14.8K incoming IP traffic, 28.4K outgoing IP traffic.6434worker2 # [ 129.402254] nix-daemon[873]: reaped child process 1543, status = succeeded6435worker2 # [ 129.404317] nix-daemon[873]: reaped child process 1580, status = succeeded6436worker2 # [ 129.415275] nix-daemon[1582]: unexpected Nix daemon error: error: interrupted by the user6437worker2 # [ 129.420692] nix-daemon[873]: reaped child process 1582, status = failed with exit code 16438worker2 # [ 129.445187] systemd[1]: scheduler-token.service: Deactivated successfully.6439worker2 # [ 129.447239] systemd[1]: Finished scheduler-token.service.6440worker2 # [ 129.448176] systemd[1]: scheduler-token.service: Consumed 18ms CPU time over 73ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6441worker2 # [ 129.451287] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6442worker2 # [ 129.455850] systemd[1]: Starting scheduler-token.service...6443worker2 # [ 129.512824] systemd[1]: scheduler-token.service: Deactivated successfully.6444worker2 # [ 129.513859] systemd[1]: Finished scheduler-token.service.6445worker2 # [ 129.516073] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 57ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6446worker2 # [ 129.520683] systemd[1]: Starting Nix worker-protocol over gRPC...6447worker1 # [ 129.543273] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:37Z level=info event=scheduler_connected addr=lb:500516448worker2 # [ 129.644704] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6449worker2 # [ 129.647369] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6450worker2 # [ 129.650385] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16451worker2 # [ 129.655071] nix-grpc-daemon[1625]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6452worker2 # [ 129.656343] nix-grpc-daemon[1625]: E0000 00:00:1790073398.682681 1625 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6453worker2 # [ 129.662071] nix-grpc-daemon[1625]: E0000 00:00:1790073398.687843 1625 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6454worker2 # [ 129.666383] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6455worker2 # [ 129.668033] systemd[1]: Started Nix worker-protocol over gRPC.6456lb # [ 129.639030] niks3-server[1156]: 2026/09/22 10:36:38 INFO lead: acquired remote=192.168.1.4:598686457worker1 # [ 129.654881] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:37Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6458worker2 # [ 129.677251] nix-grpc-daemon[1625]: E0000 00:00:1790073398.703940 1666 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6459worker2 # [ 129.681185] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=info event=scheduler_take_over6460worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.37 seconds)6461client: must succeed: sleep 5.06462worker2 # [ 129.690578] nix-grpc-daemon[1625]: E0000 00:00:1790073398.718288 1666 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6463worker2 # [ 129.694153] nix-grpc-daemon[1625]: E0000 00:00:1790073398.720434 1666 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6464worker2 # [ 129.704726] nix-grpc-daemon[1625]: E0000 00:00:1790073398.732381 1666 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6465worker2 # [ 129.712287] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=info event=scheduler_connected addr=lb:500516466worker2 # [ 129.716652] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=info event=oidc_jwks provider=mock keys=16467worker2 # [ 129.718120] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:366366468worker2 # [ 129.719792] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06469worker2 # [ 129.870649] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06470worker1 # [ 129.855616] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:38Z level=info event=scheduler_connected addr=lb:500516471worker2 # [ 129.874534] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=16472worker2 # [ 129.879489] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:366366473worker2 # [ 129.884604] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06474worker2 # [ 129.890425] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366606475worker2 # [ 129.897590] nix-daemon[873]: accepted connection from pid 1625, user nix-grpc-daemon (trusted)6476worker2 # [ 129.903811] nix-daemon[873]: accepted connection from pid 1625, user nix-grpc-daemon (trusted)6477worker2 # [ 129.910988] nix-daemon[873]: accepted connection from pid 1625, user nix-grpc-daemon (trusted)6478client # [ 129.886650] nix[1337]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6479client: (finished: must succeed: sleep 5.0, in 5.02 seconds)6480client: waiting for success: ! systemctl is-active mid6481worker2 # [ 134.924784] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:43Z level=info event=niks3_push_started pid=17086482worker2 # [ 134.937737] nix-daemon[873]: reaped child process 1683, status = succeeded6483worker2 # [ 135.078969] nix-grpc-daemon[1708]: time=2026-09-22T10:36:44.106Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6484worker2 # [ 135.221159] nix-daemon[873]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6485worker2 # [ 135.232390] nix-daemon[873]: reaped child process 1724, status = succeeded6486lb # [ 135.207608] niks3-server[1156]: 2026/09/22 10:36:44 INFO Received uploads request method=POST path=/api/pending_closures6487worker2 # [ 135.250416] nix-grpc-daemon[1708]: time=2026-09-22T10:36:44.278Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6488worker2 # [ 135.252939] nix-grpc-daemon[1708]: time=2026-09-22T10:36:44.280Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6489lb # [ 135.242106] niks3-server[1156]: 2026/09/22 10:36:44 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6490lb # [ 135.244680] niks3-server[1156]: 2026/09/22 10:36:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6491lb # [ 135.258411] niks3-server[1156]: 2026/09/22 10:36:44 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6492lb # [ 135.265616] niks3-server[1156]: 2026/09/22 10:36:44 INFO Signed narinfos id=44 count=16493worker2 # [ 135.301818] nix-grpc-daemon[1708]: time=2026-09-22T10:36:44.329Z level=INFO msg="Uploading 1 narinfos"6494lb # [ 135.274674] niks3-server[1156]: 2026/09/22 10:36:44 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6495lb # [ 135.277972] niks3-server[1156]: 2026/09/22 10:36:44 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6496lb # [ 135.286640] niks3-server[1156]: 2026/09/22 10:36:44 INFO Completed upload id=446497worker2 # [ 135.322981] nix-grpc-daemon[1708]: time=2026-09-22T10:36:44.350Z level=INFO msg="Upload complete. (240ms)"6498worker2 # [ 135.325664] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36660 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16499lb # [ 135.292048] niks3-server[1156]: 2026/09/22 10:36:44 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6500worker2 # [ 135.335460] nix-daemon[873]: reaped child process 1681, status = succeeded6501client # [ 135.334453] systemd[1]: mid.service: Deactivated successfully.6502client # [ 135.336788] systemd[1]: mid.service: Consumed 149ms CPU time over 9.662s wall clock time, 41.4M memory peak, 8.6K incoming IP traffic, 10.4K outgoing IP traffic.6503client: (finished: waiting for success: ! systemctl is-active mid, in 1.09 seconds)6504client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6505client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds)6506(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.37 seconds)6507subtest: interrupting the client stops the build on the worker6508client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6509client # [ 136.020088] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6510client # Running as unit: intr.service; invocation ID: 569dc4ba7fb5431fb6582a9919b9866a6511client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.03 seconds)6512client # [ 136.209029] nix[1373]: don't know how to build these paths:6513client # [ 136.209888] nix[1373]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6514worker2 # [ 136.231435] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:366606515worker2 # [ 136.243148] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:45Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06516worker2 # [ 136.247705] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:45Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.3:50051 clients=16517worker1 # [ 136.286621] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480906518worker1 # [ 136.297729] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6519worker1 # [ 136.315226] nix-grpc-daemon[1484]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 19 ms (attempt 1/5)6520worker1 # [ 136.340142] nix-grpc-daemon[1484]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 2 ms (attempt 2/5)6521worker1 # [ 136.347607] nix-grpc-daemon[1484]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 253 ms (attempt 3/5)6522worker1 # [ 136.606521] nix-grpc-daemon[1484]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 34 ms (attempt 4/5)6523worker1 # [ 136.646738] nix-grpc-daemon[1484]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6524worker1 # [ 136.650207] nix-grpc-daemon[1484]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6525worker1 # [ 136.667403] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480906526client # [ 136.675399] nix[1373]: copying 1 paths to 'lb:50051'...6527client # [ 136.679161] nix[1373]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6528worker1 # [ 136.683122] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480906529worker1 # [ 136.691586] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6530worker1 # [ 136.713965] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:44Z level=info event=niks3_push_started pid=15766531worker1 # [ 136.915358] nix-grpc-daemon[1576]: time=2026-09-22T10:36:45.136Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6532worker1 # [ 137.075138] nix-daemon[851]: accepted connection from pid 1589, user nix-grpc-daemon (trusted)6533worker1 # [ 137.086878] nix-daemon[851]: reaped child process 1592, status = succeeded6534worker1 # [ 137.258544] nix-daemon[851]: accepted connection from pid 1594, user nix-grpc-daemon (trusted)6535worker1 # [ 137.270360] nix-daemon[851]: reaped child process 1601, status = succeeded6536lb # [ 137.265543] niks3-server[1156]: 2026/09/22 10:36:46 INFO Received uploads request method=POST path=/api/pending_closures6537worker1 # [ 137.288406] nix-grpc-daemon[1576]: time=2026-09-22T10:36:45.509Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6538worker1 # [ 137.291498] nix-grpc-daemon[1576]: time=2026-09-22T10:36:45.511Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6539lb # [ 137.299378] niks3-server[1156]: 2026/09/22 10:36:46 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6540lb # [ 137.307955] niks3-server[1156]: 2026/09/22 10:36:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6541lb # [ 137.313498] niks3-server[1156]: 2026/09/22 10:36:46 INFO Signed narinfos id=45 count=16542worker1 # [ 137.332891] nix-grpc-daemon[1576]: time=2026-09-22T10:36:45.554Z level=INFO msg="Uploading 1 narinfos"6543lb # [ 137.326560] niks3-server[1156]: 2026/09/22 10:36:46 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6544lb # [ 137.332649] niks3-server[1156]: 2026/09/22 10:36:46 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6545lb # [ 137.339807] niks3-server[1156]: 2026/09/22 10:36:46 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6546lb # [ 137.343207] niks3-server[1156]: 2026/09/22 10:36:46 INFO Completed upload id=456547worker1 # [ 137.359506] nix-grpc-daemon[1576]: time=2026-09-22T10:36:45.580Z level=INFO msg="Upload complete. (440ms)"6548worker1 # [ 137.362476] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=4246549worker1 # [ 137.368297] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480906550worker1 # [ 137.378846] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6551worker1 # [ 137.383077] nix-daemon[851]: reaped child process 1574, status = succeeded6552worker1 # [ 137.387318] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6553client # [ 137.384245] nix[1373]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6554lb # [ 138.649804] envoy[923]: conn peer=[2001:db8:1::4]:42988 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=2871 tx=363 ms=165006555lb # [ 138.653665] envoy[923]: conn peer=[2001:db8:1::1]:45678 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3921 tx=1942 ms=94106556client: must succeed: systemctl kill -s INT intr6557client # [ 139.545596] systemd[1]: intr.service: Sent signal SIGINT to main process 1373 (nix) on client request.6558worker1 # [ 139.551538] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:47Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6559client # [ 139.549704] nix[1373]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6560client: (finished: must succeed: systemctl kill -s INT intr, in 0.05 seconds)6561client # [ 139.556878] nix[1373]: error: interrupted by the user6562client # [ 139.589777] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6563client # [ 139.591255] systemd[1]: intr.service: Failed with result 'exit-code'.6564client # [ 139.593605] systemd[1]: intr.service: Consumed 117ms CPU time over 3.573s wall clock time, 39.4M memory peak, 6.4K incoming IP traffic, 8.3K outgoing IP traffic.6565worker1 # [ 139.612071] nix-daemon[851]: reaped child process 1603, status = succeeded6566worker1 # [ 139.616530] nix-daemon[1605]: unexpected Nix daemon error: error: interrupted by the user6567worker1 # [ 139.621406] nix-daemon[851]: reaped child process 1605, status = failed with exit code 16568(finished: subtest: interrupting the client stops the build on the worker, in 3.68 seconds)6569subtest: a deploy mid-build drains: the build finishes, the new generation takes over6570client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6571client # [ 139.716745] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6572client # Running as unit: sw.service; invocation ID: 72b68ec2c2364ee5a739b57cc098b0bf6573client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.05 seconds)6574client # [ 139.911315] nix[1393]: don't know how to build these paths:6575client # [ 139.912227] nix[1393]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6576worker1 # [ 139.913293] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:48Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:480906577worker2 # [ 139.944883] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:48Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06578worker2 # [ 139.948857] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:48Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.4:50051 clients=16579worker2 # [ 139.988126] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366606580worker2 # [ 140.000709] nix-grpc-daemon[1625]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 41 ms (attempt 1/5)6581worker2 # [ 140.047302] nix-grpc-daemon[1625]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 34 ms (attempt 2/5)6582worker2 # [ 140.083959] nix-grpc-daemon[1625]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 272 ms (attempt 3/5)6583worker2 # [ 140.361819] nix-grpc-daemon[1625]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 668 ms (attempt 4/5)6584worker2 # [ 141.033834] nix-grpc-daemon[1625]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6585worker2 # [ 141.037714] nix-grpc-daemon[1625]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6586worker2 # [ 141.054863] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366606587client # [ 141.041826] nix[1393]: copying 1 paths to 'lb:50051'...6588client # [ 141.045973] nix[1393]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6589worker2 # [ 141.069146] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366606590worker2 # [ 141.080354] nix-daemon[873]: accepted connection from pid 1625, user nix-grpc-daemon (trusted)6591worker2 # [ 141.232510] nix-daemon[873]: accepted connection from pid 1764, user nix-grpc-daemon (trusted)6592worker2 # [ 141.244795] nix-daemon[873]: reaped child process 1767, status = succeeded6593worker2 # [ 141.393666] nix-daemon[873]: accepted connection from pid 1769, user nix-grpc-daemon (trusted)6594worker2 # [ 141.405893] nix-daemon[873]: reaped child process 1772, status = succeeded6595lb # [ 141.386688] niks3-server[1156]: 2026/09/22 10:36:50 INFO Received uploads request method=POST path=/api/pending_closures6596worker2 # [ 141.432561] nix-grpc-daemon[1708]: time=2026-09-22T10:36:50.459Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6597worker2 # [ 141.433990] nix-grpc-daemon[1708]: time=2026-09-22T10:36:50.459Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6598lb # [ 141.413571] niks3-server[1156]: 2026/09/22 10:36:50 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6599lb # [ 141.425906] niks3-server[1156]: 2026/09/22 10:36:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6600worker2 # [ 141.467538] nix-grpc-daemon[1708]: time=2026-09-22T10:36:50.495Z level=INFO msg="Uploading 1 narinfos"6601lb # [ 141.431127] niks3-server[1156]: 2026/09/22 10:36:50 INFO Signed narinfos id=46 count=16602lb # [ 141.437039] niks3-server[1156]: 2026/09/22 10:36:50 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6603lb # [ 141.447686] niks3-server[1156]: 2026/09/22 10:36:50 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6604lb # [ 141.453687] niks3-server[1156]: 2026/09/22 10:36:50 INFO Completed upload id=466605worker2 # [ 141.490134] nix-grpc-daemon[1708]: time=2026-09-22T10:36:50.517Z level=INFO msg="Upload complete. (400ms)"6606lb # [ 141.455852] niks3-server[1156]: 2026/09/22 10:36:50 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6607worker2 # [ 141.492520] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36660 duration_s=0 paths=1 nar_bytes_in=4166608worker2 # [ 141.496806] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366606609worker2 # [ 141.503130] nix-daemon[873]: accepted connection from pid 1625, user nix-grpc-daemon (trusted)6610worker2 # [ 141.505606] nix-daemon[873]: reaped child process 1762, status = succeeded6611worker2 # [ 141.510971] nix-daemon[873]: accepted connection from pid 1625, user nix-grpc-daemon (trusted)6612client # [ 141.486743] nix[1393]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6613worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&26614worker2 # Checking switch inhibitors... done6615worker2 # [ 142.918210] nixos[1804]: switching to system configuration /nix/store/xz1z051hg672gb0phndbq50fjibp6hjy-nixos-system-worker2-test6616worker2 # [ 142.920553] systemd[1]: Stopped target Remote File Systems.6617worker2 # activating the configuration...6618worker2 # [ 142.923308] systemd[1]: Stopped target Local File Systems.6619worker2 # setting up /etc...6620worker2 # [ 143.574705] systemd[1]: Reload requested from client PID 1804 ('.switch-to-conf') (unit backdoor.service)...6621worker2 # [ 143.576582] systemd[1]: Reloading...6622worker2 # [ 144.295309] systemd[1]: Reloading finished in 715 ms.6623worker2 # restarting sysinit-reactivation.target6624worker2 # [ 144.353198] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....6625worker2 # [ 144.496092] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..6626worker2 # [ 144.498174] systemd[1]: Reached target Reactivate sysinit units.6627worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service6628worker2 # [ 144.504057] systemd[1]: Reloading D-Bus System Message Bus...6629worker2 # [ 144.504970] dbus-broker-launch[534]: Caught SIGHUP, trigger reload.6630worker2 # [ 144.509525] systemd[1]: Reloading Nix worker-protocol over gRPC...6631worker2 # [ 144.538244] dbus-broker-launch[534]: Looking up NSS user entry for 'systemd-timesync'...6632worker2 # [ 144.540066] dbus-broker-launch[534]: NSS returned no entry for 'systemd-timesync'6633worker2 # [ 144.541076] dbus-broker-launch[534]: Invalid user-name in /nix/store/s93qka0z548yjbxqnmj4c4b32apww6jv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"6634worker2 # [ 144.551854] systemd[1]: Reloaded Nix worker-protocol over gRPC.6635worker2 # [ 144.572341] systemd[1]: Reloaded D-Bus System Message Bus.6636worker2 # [ 144.575851] systemd[1]: Reached target Remote File Systems.6637worker2 # [ 144.578364] systemd[1]: Reached target Local File Systems.6638worker2 # [ 144.579162] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore6639worker2 # [ 144.584822] systemd[1]: Starting Create SUID/SGID Wrappers...6640worker2 # [ 144.585875] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.6641worker2 # [ 144.588583] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc676642worker2 # [ 144.590537] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore6643worker2 # [ 144.591954] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6644worker2 # [ 144.594888] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6645worker2 # [ 144.602953] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os6646worker2 # [ 144.604218] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki6647worker2 # [ 144.667845] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:53Z level=info event=draining6648worker2 # [ 145.021940] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.6649worker2 # [ 145.025172] systemd[1]: Finished Create SUID/SGID Wrappers.6650worker2 # [ 145.028883] systemd[1]: Starting linger-users.service...6651worker2 # [ 145.064748] systemd[1]: linger-users.service: Deactivated successfully.6652worker2 # [ 145.068192] systemd[1]: Finished linger-users.service.6653worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service6654worker2 # [ 145.322911] nixos[1804]: finished switching to system configuration /nix/store/xz1z051hg672gb0phndbq50fjibp6hjy-nixos-system-worker2-test6655worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 3.10 seconds)6656worker2: must succeed: pgrep -f 'read -t [9]0 x'6657worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)6658client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26659client # don't know how to build these paths:6660worker1 # [ 145.612812] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:480906661client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6662worker1 # [ 145.618590] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480906663worker2 # [ 145.661596] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:54Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06664worker2 # [ 145.663532] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:54Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.3:50051 clients=16665worker1 # [ 145.697694] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480906666worker1 # [ 145.710519] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480906667client # copying 1 paths to 'lb:50051'...6668worker1 # [ 145.723203] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480906669client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6670worker1 # [ 145.744671] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6671worker1 # [ 145.929995] nix-daemon[851]: accepted connection from pid 1652, user nix-grpc-daemon (trusted)6672worker1 # [ 145.941752] nix-daemon[851]: reaped child process 1655, status = succeeded6673worker1 # [ 146.106238] nix-daemon[851]: accepted connection from pid 1657, user nix-grpc-daemon (trusted)6674worker1 # [ 146.118302] nix-daemon[851]: reaped child process 1660, status = succeeded6675lb # [ 146.116712] niks3-server[1156]: 2026/09/22 10:36:55 INFO Received uploads request method=POST path=/api/pending_closures6676worker1 # [ 146.144390] nix-grpc-daemon[1576]: time=2026-09-22T10:36:54.365Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6677worker1 # [ 146.145890] nix-grpc-daemon[1576]: time=2026-09-22T10:36:54.367Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6678lb # [ 146.150527] niks3-server[1156]: 2026/09/22 10:36:55 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6679lb # [ 146.160461] niks3-server[1156]: 2026/09/22 10:36:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6680worker1 # [ 146.182211] nix-grpc-daemon[1576]: time=2026-09-22T10:36:54.403Z level=INFO msg="Uploading 1 narinfos"6681lb # [ 146.167129] niks3-server[1156]: 2026/09/22 10:36:55 INFO Signed narinfos id=47 count=16682lb # [ 146.172952] niks3-server[1156]: 2026/09/22 10:36:55 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6683lb # [ 146.184308] niks3-server[1156]: 2026/09/22 10:36:55 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6684lb # [ 146.190526] niks3-server[1156]: 2026/09/22 10:36:55 INFO Completed upload id=476685worker1 # [ 146.209155] nix-grpc-daemon[1576]: time=2026-09-22T10:36:54.430Z level=INFO msg="Upload complete. (452ms)"6686lb # [ 146.193970] niks3-server[1156]: 2026/09/22 10:36:55 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6687worker1 # [ 146.213140] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=5286688worker1 # [ 146.218268] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480906689worker1 # [ 146.228240] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6690worker1 # [ 146.236206] nix-daemon[851]: reaped child process 1650, status = succeeded6691worker1 # [ 146.240193] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6692client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6693client # farm-a> node-a: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6694client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6695client # farm-a> LOG-farm-a-during-drain6696worker1 # [ 147.266448] nix-daemon[851]: reaped child process 1664, status = succeeded6697worker1 # [ 147.414343] nix-daemon[851]: accepted connection from pid 1680, user nix-grpc-daemon (trusted)6698worker1 # [ 147.425403] nix-daemon[851]: reaped child process 1683, status = succeeded6699lb # [ 147.422690] niks3-server[1156]: 2026/09/22 10:36:56 INFO Received uploads request method=POST path=/api/pending_closures6700worker1 # [ 147.450330] nix-grpc-daemon[1576]: time=2026-09-22T10:36:55.671Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6701worker1 # [ 147.452182] nix-grpc-daemon[1576]: time=2026-09-22T10:36:55.673Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6702lb # [ 147.464524] niks3-server[1156]: 2026/09/22 10:36:56 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6703lb # [ 147.469701] niks3-server[1156]: 2026/09/22 10:36:56 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6704lb # [ 147.483307] niks3-server[1156]: 2026/09/22 10:36:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6705worker1 # [ 147.504469] nix-grpc-daemon[1576]: time=2026-09-22T10:36:55.725Z level=INFO msg="Uploading 1 narinfos"6706lb # [ 147.487673] niks3-server[1156]: 2026/09/22 10:36:56 INFO Signed narinfos id=48 count=16707lb # [ 147.493213] niks3-server[1156]: 2026/09/22 10:36:56 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6708lb # [ 147.503702] niks3-server[1156]: 2026/09/22 10:36:56 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6709lb # [ 147.511184] niks3-server[1156]: 2026/09/22 10:36:56 INFO Completed upload id=486710lb # [ 147.513058] niks3-server[1156]: 2026/09/22 10:36:56 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6711worker1 # [ 147.527797] nix-grpc-daemon[1576]: time=2026-09-22T10:36:55.749Z level=INFO msg="Upload complete. (280ms)"6712worker1 # [ 147.535265] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16713worker2 # [ 147.558304] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:56Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06714worker2 # [ 147.561242] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:56Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.3:50051 clients=16715worker1 # [ 147.548667] nix-daemon[851]: reaped child process 1662, status = succeeded6716worker1 # [ 147.582261] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480906717worker1 # [ 147.594139] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480906718worker1 # [ 147.607473] nix-grpc-daemon[1576]: time=2026-09-22T10:36:55.828Z level=INFO msg="All 1 paths already cached"6719client # copying 1 paths to 'lb:50051'...6720client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6721worker1 # [ 147.614628] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480906722worker1 # [ 147.624707] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6723worker1 # [ 147.796591] nix-daemon[851]: accepted connection from pid 1687, user nix-grpc-daemon (trusted)6724worker1 # [ 147.809572] nix-daemon[851]: reaped child process 1690, status = succeeded6725worker1 # [ 147.966232] nix-daemon[851]: accepted connection from pid 1692, user nix-grpc-daemon (trusted)6726worker1 # [ 147.978395] nix-daemon[851]: reaped child process 1695, status = succeeded6727lb # [ 147.976668] niks3-server[1156]: 2026/09/22 10:36:56 INFO Received uploads request method=POST path=/api/pending_closures6728worker1 # [ 148.001548] nix-grpc-daemon[1576]: time=2026-09-22T10:36:56.223Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6729worker1 # [ 148.004064] nix-grpc-daemon[1576]: time=2026-09-22T10:36:56.225Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6730lb # [ 148.008991] niks3-server[1156]: 2026/09/22 10:36:56 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6731lb # [ 148.018986] niks3-server[1156]: 2026/09/22 10:36:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6732worker1 # [ 148.040448] nix-grpc-daemon[1576]: time=2026-09-22T10:36:56.261Z level=INFO msg="Uploading 1 narinfos"6733lb # [ 148.024124] niks3-server[1156]: 2026/09/22 10:36:56 INFO Signed narinfos id=49 count=16734lb # [ 148.033194] niks3-server[1156]: 2026/09/22 10:36:56 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6735lb # [ 148.041477] niks3-server[1156]: 2026/09/22 10:36:56 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6736lb # [ 148.048296] niks3-server[1156]: 2026/09/22 10:36:56 INFO Completed upload id=496737worker1 # [ 148.064770] nix-grpc-daemon[1576]: time=2026-09-22T10:36:56.286Z level=INFO msg="Upload complete. (425ms)"6738lb # [ 148.051743] niks3-server[1156]: 2026/09/22 10:36:56 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6739worker1 # [ 148.068966] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=6406740worker1 # [ 148.074520] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480906741worker1 # [ 148.086117] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6742worker1 # [ 148.088766] nix-daemon[851]: reaped child process 1685, status = succeeded6743client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6744client # farm-b> node-a: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6745worker1 # [ 148.094067] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6746lb # [ 148.649696] envoy[923]: conn peer=[2001:db8:1::1]:52506 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3006 tx=851 ms=33626747client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6748client # farm-b> LOG-farm-b-during-drain6749worker1 # [ 149.519650] nix-daemon[851]: reaped child process 1699, status = succeeded6750worker1 # [ 149.665792] nix-daemon[851]: accepted connection from pid 1715, user nix-grpc-daemon (trusted)6751worker1 # [ 149.678069] nix-daemon[851]: reaped child process 1718, status = succeeded6752lb # [ 149.675306] niks3-server[1156]: 2026/09/22 10:36:58 INFO Received uploads request method=POST path=/api/pending_closures6753worker1 # [ 149.697893] nix-grpc-daemon[1576]: time=2026-09-22T10:36:57.919Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6754worker1 # [ 149.699992] nix-grpc-daemon[1576]: time=2026-09-22T10:36:57.920Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6755lb # [ 149.693978] niks3-server[1156]: 2026/09/22 10:36:58 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6756lb # [ 149.699044] niks3-server[1156]: 2026/09/22 10:36:58 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6757lb # [ 149.701724] niks3-server[1156]: 2026/09/22 10:36:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6758lb # [ 149.703400] niks3-server[1156]: 2026/09/22 10:36:58 INFO Signed narinfos id=50 count=16759worker1 # [ 149.721135] nix-grpc-daemon[1576]: time=2026-09-22T10:36:57.942Z level=INFO msg="Uploading 1 narinfos"6760lb # [ 149.706181] niks3-server[1156]: 2026/09/22 10:36:58 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6761lb # [ 149.710509] niks3-server[1156]: 2026/09/22 10:36:58 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6762lb # [ 149.714737] niks3-server[1156]: 2026/09/22 10:36:58 INFO Completed upload id=506763worker1 # [ 149.730996] nix-grpc-daemon[1576]: time=2026-09-22T10:36:57.952Z level=INFO msg="Upload complete. (228ms)"6764lb # [ 149.716500] niks3-server[1156]: 2026/09/22 10:36:58 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6765worker1 # [ 149.733489] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16766worker1 # [ 149.740132] nix-daemon[851]: reaped child process 1697, status = succeeded6767worker2 # [ 149.760768] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:58Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06768worker2 # [ 149.762548] nix-grpc-daemon[1625]: ts=2026-09-22T10:36:58Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.3:50051 clients=16769worker1 # [ 149.784554] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480906770worker1 # [ 149.794264] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480906771worker1 # [ 149.805424] nix-grpc-daemon[1576]: time=2026-09-22T10:36:58.026Z level=INFO msg="All 2 paths already cached"6772client # copying 1 paths to 'lb:50051'...6773worker1 # [ 149.813383] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480906774client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6775worker1 # [ 149.832128] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6776worker1 # [ 149.998318] nix-daemon[851]: accepted connection from pid 1722, user nix-grpc-daemon (trusted)6777worker1 # [ 150.011279] nix-daemon[851]: reaped child process 1725, status = succeeded6778worker1 # [ 150.170439] nix-daemon[851]: accepted connection from pid 1727, user nix-grpc-daemon (trusted)6779worker1 # [ 150.182274] nix-daemon[851]: reaped child process 1730, status = succeeded6780lb # [ 150.182502] niks3-server[1156]: 2026/09/22 10:36:59 INFO Received uploads request method=POST path=/api/pending_closures6781worker1 # [ 150.207037] nix-grpc-daemon[1576]: time=2026-09-22T10:36:58.428Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6782worker1 # [ 150.208414] nix-grpc-daemon[1576]: time=2026-09-22T10:36:58.428Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6783lb # [ 150.209517] niks3-server[1156]: 2026/09/22 10:36:59 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6784lb # [ 150.219189] niks3-server[1156]: 2026/09/22 10:36:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6785lb # [ 150.224345] niks3-server[1156]: 2026/09/22 10:36:59 INFO Signed narinfos id=51 count=16786worker1 # [ 150.243662] nix-grpc-daemon[1576]: time=2026-09-22T10:36:58.465Z level=INFO msg="Uploading 1 narinfos"6787lb # [ 150.231114] niks3-server[1156]: 2026/09/22 10:36:59 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6788lb # [ 150.241442] niks3-server[1156]: 2026/09/22 10:36:59 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6789worker1 # [ 150.265281] nix-grpc-daemon[1576]: time=2026-09-22T10:36:58.486Z level=INFO msg="Upload complete. (423ms)"6790lb # [ 150.249121] niks3-server[1156]: 2026/09/22 10:36:59 INFO Completed upload id=516791worker1 # [ 150.267525] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=7686792lb # [ 150.252431] niks3-server[1156]: 2026/09/22 10:36:59 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6793worker1 # [ 150.272931] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480906794worker1 # [ 150.280869] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6795worker1 # [ 150.285349] nix-daemon[851]: reaped child process 1720, status = succeeded6796worker1 # [ 150.290523] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6797client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6798client # farm-top> node-a: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6799client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6800client # farm-top> LOG-farm-top-during-drain6801worker1 # [ 151.510928] nix-daemon[851]: reaped child process 1734, status = succeeded6802worker1 # [ 151.670090] nix-daemon[851]: accepted connection from pid 1750, user nix-grpc-daemon (trusted)6803worker1 # [ 151.681252] nix-daemon[851]: reaped child process 1753, status = succeeded6804lb # [ 151.682802] niks3-server[1156]: 2026/09/22 10:37:00 INFO Received uploads request method=POST path=/api/pending_closures6805worker1 # [ 151.707827] nix-grpc-daemon[1576]: time=2026-09-22T10:36:59.929Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6806worker1 # [ 151.709412] nix-grpc-daemon[1576]: time=2026-09-22T10:36:59.930Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6807lb # [ 151.714881] niks3-server[1156]: 2026/09/22 10:37:00 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6808lb # [ 151.732211] niks3-server[1156]: 2026/09/22 10:37:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6809lb # [ 151.739977] niks3-server[1156]: 2026/09/22 10:37:00 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6810lb # [ 151.742291] niks3-server[1156]: 2026/09/22 10:37:00 INFO Signed narinfos id=52 count=16811lb # [ 151.744658] niks3-server[1156]: 2026/09/22 10:37:00 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6812worker1 # [ 151.760238] nix-grpc-daemon[1576]: time=2026-09-22T10:36:59.981Z level=INFO msg="Uploading 1 narinfos"6813lb # [ 151.750260] niks3-server[1156]: 2026/09/22 10:37:00 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6814lb # [ 151.753520] niks3-server[1156]: 2026/09/22 10:37:00 INFO Completed upload id=526815worker1 # [ 151.769946] nix-grpc-daemon[1576]: time=2026-09-22T10:36:59.991Z level=INFO msg="Upload complete. (272ms)"6816lb # [ 151.755575] niks3-server[1156]: 2026/09/22 10:37:00 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6817worker1 # [ 151.776186] nix-grpc-daemon[1484]: ts=2026-09-22T10:36:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16818worker1 # [ 151.786726] nix-daemon[851]: reaped child process 1732, status = succeeded6819client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 6.45 seconds)6820client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6821client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.16 seconds)6822worker2: must succeed: pkill -f 'read -t [9]0 x'6823worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6824client: waiting for success: ! systemctl is-active sw6825worker2 # [ 152.139443] nix-daemon[873]: reaped child process 1776, status = succeeded6826worker2 # [ 152.264543] nix-daemon[873]: accepted connection from pid 1950, user nix-grpc-daemon (trusted)6827worker2 # [ 152.276962] nix-daemon[873]: reaped child process 1953, status = succeeded6828lb # [ 152.253563] niks3-server[1156]: 2026/09/22 10:37:01 INFO Received uploads request method=POST path=/api/pending_closures6829worker2 # [ 152.300172] nix-grpc-daemon[1708]: time=2026-09-22T10:37:01.327Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6830worker2 # [ 152.301534] nix-grpc-daemon[1708]: time=2026-09-22T10:37:01.327Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6831lb # [ 152.284513] niks3-server[1156]: 2026/09/22 10:37:01 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6832lb # [ 152.302714] niks3-server[1156]: 2026/09/22 10:37:01 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6833lb # [ 152.306093] niks3-server[1156]: 2026/09/22 10:37:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6834lb # [ 152.310215] niks3-server[1156]: 2026/09/22 10:37:01 INFO Signed narinfos id=53 count=16835worker2 # [ 152.349346] nix-grpc-daemon[1708]: time=2026-09-22T10:37:01.376Z level=INFO msg="Uploading 1 narinfos"6836lb # [ 152.318961] niks3-server[1156]: 2026/09/22 10:37:01 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6837lb # [ 152.322658] niks3-server[1156]: 2026/09/22 10:37:01 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6838lb # [ 152.326624] niks3-server[1156]: 2026/09/22 10:37:01 INFO Completed upload id=536839worker2 # [ 152.362673] nix-grpc-daemon[1708]: time=2026-09-22T10:37:01.390Z level=INFO msg="Upload complete. (231ms)"6840lb # [ 152.328805] niks3-server[1156]: 2026/09/22 10:37:01 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6841worker2 # [ 152.369288] nix-grpc-daemon[1625]: ts=2026-09-22T10:37:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36660 duration_s=10 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16842worker2 # [ 152.380377] nix-daemon[873]: reaped child process 1774, status = succeeded6843client # [ 152.385288] systemd[1]: sw.service: Deactivated successfully.6844client # [ 152.389886] systemd[1]: sw.service: Consumed 120ms CPU time over 12.670s wall clock time, 41.4M memory peak, 7.3K incoming IP traffic, 8.3K outgoing IP traffic.6845worker2 # [ 152.671614] nix-grpc-daemon[1625]: ts=2026-09-22T10:37:01Z level=info event=signal_exit6846lb # [ 152.863413] niks3-server[1156]: 2026/09/22 10:37:01 INFO lead: released remote=192.168.1.4:598686847client: (finished: waiting for success: ! systemctl is-active sw, in 1.08 seconds)6848client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6849client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds)6850worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6851worker2 # [ 153.216142] systemd[1]: Starting scheduler-token.service...6852worker2 # [ 153.268947] systemd[1]: scheduler-token.service: Deactivated successfully.6853worker2 # [ 153.270586] systemd[1]: Finished scheduler-token.service.6854worker2 # [ 153.272470] systemd[1]: scheduler-token.service: Consumed 16ms CPU time over 54ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6855worker1 # [ 153.879402] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:02Z level=info event=scheduler_restarting addr=lb:500516856worker2 # [ 153.903957] nix-grpc-daemon[1625]: ts=2026-09-22T10:37:02Z level=info event=scheduler_restarting6857worker2 # [ 153.907482] nix-grpc-daemon[1625]: ts=2026-09-22T10:37:02Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:366366858worker2 # [ 153.911441] nix-grpc-daemon[1625]: ts=2026-09-22T10:37:02Z level=info event=worker_gone addr=192.168.1.3:500516859worker2 # [ 153.915255] nix-grpc-daemon[1625]: ts=2026-09-22T10:37:02Z level=info event=scheduler_restarting addr=lb:500516860worker2 # [ 153.918338] nix-grpc-daemon[1625]: ts=2026-09-22T10:37:02Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:366366861worker2 # [ 153.921763] nix-grpc-daemon[1625]: ts=2026-09-22T10:37:02Z level=info event=worker_gone addr=192.168.1.4:500516862worker2 # [ 153.937702] nix-daemon[873]: reaped child process 1679, status = succeeded6863worker1 # [ 154.079345] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:02Z level=info event=scheduler_connected addr=lb:500516864worker1 # [ 154.083328] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6865worker2 # [ 154.114718] nix-grpc-daemon[1625]: ts=2026-09-22T10:37:03Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6866worker1 # [ 154.283821] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:02Z level=info event=scheduler_connected addr=lb:500516867worker1 # [ 154.287736] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6868worker1 # [ 154.488654] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:02Z level=info event=scheduler_connected addr=lb:500516869worker1 # [ 154.492515] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6870worker1 # [ 154.693457] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:02Z level=info event=scheduler_connected addr=lb:500516871worker1 # [ 154.697287] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6872worker2 # [ 154.900239] nix-grpc-daemon[1625]: E0000 00:00:1790073423.927084 1670 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true6873worker2 # [ 154.905573] nix-grpc-daemon[1625]: E0000 00:00:1790073423.932971 1670 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6874worker1 # [ 154.898086] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:03Z level=info event=scheduler_connected addr=lb:500516875worker1 # [ 154.899670] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6876worker1 # [ 155.100704] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:03Z level=info event=scheduler_connected addr=lb:500516877worker1 # [ 155.104645] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6878lb # [ 155.169431] niks3-server[1156]: 2026/09/22 10:37:04 INFO lead: acquired remote=192.168.1.3:378066879worker1 # [ 155.188921] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:03Z level=info event=scheduler_take_over6880worker1 # [ 155.305497] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:03Z level=info event=scheduler_connected addr=lb:500516881worker1 # [ 155.309364] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6882worker1 # [ 155.510341] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:03Z level=info event=scheduler_connected addr=lb:500516883worker1 # [ 155.514232] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6884worker2 # [ 155.707750] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6885worker2 # [ 155.711385] systemd[1]: nix-grpc-daemon.service: Consumed 639ms CPU time over 26.329s wall clock time, 33.5M memory peak, 24.6K incoming IP traffic, 33K outgoing IP traffic.6886worker2 # [ 155.728164] systemd[1]: Starting Nix worker-protocol over gRPC...6887worker1 # [ 155.715085] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:03Z level=info event=scheduler_connected addr=lb:500516888worker1 # [ 155.718877] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6889worker2 # [ 155.880824] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:04Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6890worker2 # [ 155.882593] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:04Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6891worker2 # [ 155.885963] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:04Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16892worker2 # [ 155.893084] nix-grpc-daemon[1970]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6893worker2 # [ 155.894537] nix-grpc-daemon[1970]: E0000 00:00:1790073424.919853 1970 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6894worker2 # [ 155.899932] nix-grpc-daemon[1970]: E0000 00:00:1790073424.925096 1970 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6895worker2 # [ 155.904144] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:04Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6896worker2 # [ 155.905960] systemd[1]: Started Nix worker-protocol over gRPC.6897worker2 # [ 155.911282] nix-grpc-daemon[1970]: E0000 00:00:1790073424.939003 2012 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6898worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.77 seconds)6899lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6900lb: must succeed: curl -sf localhost:9901/clusters6901worker2 # [ 155.924412] nix-grpc-daemon[1970]: E0000 00:00:1790073424.952083 2012 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6902worker2 # [ 155.928135] nix-grpc-daemon[1970]: E0000 00:00:1790073424.954260 2012 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6903worker2 # [ 155.938540] nix-grpc-daemon[1970]: E0000 00:00:1790073424.966216 2012 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6904worker1 # [ 155.920283] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:04Z level=info event=scheduler_connected addr=lb:500516905worker1 # [ 155.923976] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6906worker2 # [ 155.949080] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:04Z level=info event=scheduler_connected addr=lb:500516907worker2 # [ 155.950635] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6908lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6909worker1 # [ 156.124418] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:04Z level=info event=scheduler_connected addr=lb:500516910worker2 # [ 156.152192] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:05Z level=info event=scheduler_connected addr=lb:500516911worker1 # [ 156.136142] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:04Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:408766912worker1 # [ 156.140618] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:04Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06913worker1 # [ 156.150138] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:04Z level=info event=oidc_jwks provider=mock keys=16914worker1 # [ 156.154313] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:04Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:408766915worker1 # [ 156.158175] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:04Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06916lb: must succeed: curl -sf localhost:9901/clusters6917lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6918lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.15 seconds)6919lb: must succeed: curl -sf localhost:9901/clusters6920lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6921worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6922worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6923(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 17.48 seconds)6924subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6925client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16926worker1 # [ 157.382228] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:480906927worker1 # [ 157.390436] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480906928worker1 # [ 157.437156] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:05Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06929worker1 # [ 157.441109] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:05Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16930worker1 # [ 157.445331] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:05Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06931worker1 # [ 157.449668] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:05Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06932worker1 # [ 157.494388] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480906933worker1 # [ 157.506941] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480906934worker1 # [ 157.517966] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480906935worker1 # [ 157.526894] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6936worker1 # [ 157.704660] nix-daemon[851]: accepted connection from pid 1762, user nix-grpc-daemon (trusted)6937worker1 # [ 157.716915] nix-daemon[851]: reaped child process 1765, status = succeeded6938worker1 # [ 157.874287] nix-daemon[851]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6939worker1 # [ 157.885971] nix-daemon[851]: reaped child process 1770, status = succeeded6940lb # [ 157.886609] niks3-server[1156]: 2026/09/22 10:37:06 INFO Received uploads request method=POST path=/api/pending_closures6941worker1 # [ 157.912229] nix-grpc-daemon[1576]: time=2026-09-22T10:37:06.133Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6942worker1 # [ 157.913729] nix-grpc-daemon[1576]: time=2026-09-22T10:37:06.135Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6943lb # [ 157.920393] niks3-server[1156]: 2026/09/22 10:37:06 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6944lb # [ 157.928390] niks3-server[1156]: 2026/09/22 10:37:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6945worker1 # [ 157.949720] nix-grpc-daemon[1576]: time=2026-09-22T10:37:06.171Z level=INFO msg="Uploading 1 narinfos"6946lb # [ 157.933663] niks3-server[1156]: 2026/09/22 10:37:06 INFO Signed narinfos id=54 count=16947lb # [ 157.940149] niks3-server[1156]: 2026/09/22 10:37:06 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6948lb # [ 157.948944] niks3-server[1156]: 2026/09/22 10:37:06 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6949lb # [ 157.958240] niks3-server[1156]: 2026/09/22 10:37:06 INFO Completed upload id=546950worker1 # [ 157.974717] nix-grpc-daemon[1576]: time=2026-09-22T10:37:06.196Z level=INFO msg="Upload complete. (434ms)"6951lb # [ 157.961244] niks3-server[1156]: 2026/09/22 10:37:06 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6952worker1 # [ 157.979074] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=5046953worker1 # [ 157.984333] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480906954worker1 # [ 157.993484] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6955worker1 # [ 157.999923] nix-daemon[851]: reaped child process 1760, status = succeeded6956worker1 # [ 158.004392] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6957lb # [ 158.651276] envoy[923]: conn peer=[2001:db8:1::1]:37476 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9598 tx=4395 ms=61766958lb # [ 158.656730] envoy[923]: conn peer=[2001:db8:1::1]:52512 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2949 tx=1567 ms=124496959lb # [ 158.661252] envoy[923]: conn peer=[2001:db8:1::4]:41224 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=1627 tx=426 ms=243956960worker1 # [ 159.031521] nix-daemon[851]: reaped child process 1774, status = succeeded6961worker1 # [ 159.193794] nix-daemon[851]: accepted connection from pid 1790, user nix-grpc-daemon (trusted)6962worker1 # [ 159.204377] nix-daemon[851]: reaped child process 1793, status = succeeded6963lb # [ 159.205432] niks3-server[1156]: 2026/09/22 10:37:08 INFO Received uploads request method=POST path=/api/pending_closures6964worker1 # [ 159.230424] nix-grpc-daemon[1576]: time=2026-09-22T10:37:07.451Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6965worker1 # [ 159.231867] nix-grpc-daemon[1576]: time=2026-09-22T10:37:07.453Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6966lb # [ 159.243284] niks3-server[1156]: 2026/09/22 10:37:08 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6967lb # [ 159.254610] niks3-server[1156]: 2026/09/22 10:37:08 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6968lb # [ 159.263817] niks3-server[1156]: 2026/09/22 10:37:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6969worker1 # [ 159.285140] nix-grpc-daemon[1576]: time=2026-09-22T10:37:07.506Z level=INFO msg="Uploading 1 narinfos"6970lb # [ 159.270120] niks3-server[1156]: 2026/09/22 10:37:08 INFO Signed narinfos id=55 count=16971lb # [ 159.274327] niks3-server[1156]: 2026/09/22 10:37:08 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6972lb # [ 159.282558] niks3-server[1156]: 2026/09/22 10:37:08 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6973lb # [ 159.290397] niks3-server[1156]: 2026/09/22 10:37:08 INFO Completed upload id=556974worker1 # [ 159.306863] nix-grpc-daemon[1576]: time=2026-09-22T10:37:07.528Z level=INFO msg="Upload complete. (295ms)"6975lb # [ 159.292345] niks3-server[1156]: 2026/09/22 10:37:08 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6976worker1 # [ 159.313644] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=16977worker1 # [ 159.326076] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:07Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16978worker1 # [ 159.330307] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:07Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06979worker1 # [ 159.339340] nix-daemon[851]: reaped child process 1772, status = succeeded6980worker1 # [ 159.369126] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480906981worker1 # [ 159.381545] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480906982worker1 # [ 159.392696] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480906983worker1 # [ 159.401553] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6984worker1 # [ 159.564426] nix-daemon[851]: accepted connection from pid 1799, user nix-grpc-daemon (trusted)6985worker1 # [ 159.576619] nix-daemon[851]: reaped child process 1802, status = succeeded6986worker1 # [ 159.739446] nix-daemon[851]: accepted connection from pid 1804, user nix-grpc-daemon (trusted)6987worker1 # [ 159.751611] nix-daemon[851]: reaped child process 1807, status = succeeded6988lb # [ 159.750822] niks3-server[1156]: 2026/09/22 10:37:08 INFO Received uploads request method=POST path=/api/pending_closures6989worker1 # [ 159.775667] nix-grpc-daemon[1576]: time=2026-09-22T10:37:07.997Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6990worker1 # [ 159.777210] nix-grpc-daemon[1576]: time=2026-09-22T10:37:07.998Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6991lb # [ 159.780715] niks3-server[1156]: 2026/09/22 10:37:08 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6992lb # [ 159.791490] niks3-server[1156]: 2026/09/22 10:37:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6993worker1 # [ 159.812458] nix-grpc-daemon[1576]: time=2026-09-22T10:37:08.033Z level=INFO msg="Uploading 1 narinfos"6994lb # [ 159.797260] niks3-server[1156]: 2026/09/22 10:37:08 INFO Signed narinfos id=56 count=16995lb # [ 159.802543] niks3-server[1156]: 2026/09/22 10:37:08 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6996lb # [ 159.812575] niks3-server[1156]: 2026/09/22 10:37:08 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6997lb # [ 159.818583] niks3-server[1156]: 2026/09/22 10:37:08 INFO Completed upload id=566998worker1 # [ 159.834948] nix-grpc-daemon[1576]: time=2026-09-22T10:37:08.056Z level=INFO msg="Upload complete. (421ms)"6999worker1 # [ 159.837259] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=5047000lb # [ 159.823418] niks3-server[1156]: 2026/09/22 10:37:08 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo7001worker1 # [ 159.843700] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907002worker1 # [ 159.853383] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7003worker1 # [ 159.856402] nix-daemon[851]: reaped child process 1797, status = succeeded7004worker1 # [ 159.861387] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7005worker1 # [ 161.275875] nix-daemon[851]: reaped child process 1811, status = succeeded7006worker1 # [ 161.418331] nix-daemon[851]: accepted connection from pid 1827, user nix-grpc-daemon (trusted)7007worker1 # [ 161.429497] nix-daemon[851]: reaped child process 1830, status = succeeded7008lb # [ 161.427781] niks3-server[1156]: 2026/09/22 10:37:10 INFO Received uploads request method=POST path=/api/pending_closures7009worker1 # [ 161.450298] nix-grpc-daemon[1576]: time=2026-09-22T10:37:09.670Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7010worker1 # [ 161.451711] nix-grpc-daemon[1576]: time=2026-09-22T10:37:09.670Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"7011lb # [ 161.451193] niks3-server[1156]: 2026/09/22 10:37:10 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst7012lb # [ 161.454415] niks3-server[1156]: 2026/09/22 10:37:10 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv7013lb # [ 161.458684] niks3-server[1156]: 2026/09/22 10:37:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign7014lb # [ 161.461259] niks3-server[1156]: 2026/09/22 10:37:10 INFO Signed narinfos id=57 count=17015lb # [ 161.462587] niks3-server[1156]: 2026/09/22 10:37:10 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls7016worker1 # [ 161.478956] nix-grpc-daemon[1576]: time=2026-09-22T10:37:09.700Z level=INFO msg="Uploading 1 narinfos"7017lb # [ 161.468639] niks3-server[1156]: 2026/09/22 10:37:10 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete7018lb # [ 161.471927] niks3-server[1156]: 2026/09/22 10:37:10 INFO Completed upload id=577019worker1 # [ 161.488156] nix-grpc-daemon[1576]: time=2026-09-22T10:37:09.709Z level=INFO msg="Upload complete. (219ms)"7020lb # [ 161.473522] niks3-server[1156]: 2026/09/22 10:37:10 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo7021worker1 # [ 161.490729] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=17022worker1 # [ 161.496202] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:09Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=17023worker1 # [ 161.498424] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907024worker1 # [ 161.501197] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:09Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=07025worker1 # [ 161.507298] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480907026worker1 # [ 161.510372] nix-daemon[851]: reaped child process 1809, status = succeeded7027worker1 # [ 161.513358] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480907028worker1 # [ 161.517464] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7029worker1 # [ 161.666111] nix-daemon[851]: accepted connection from pid 1836, user nix-grpc-daemon (trusted)7030worker1 # [ 161.678175] nix-daemon[851]: reaped child process 1839, status = succeeded7031worker1 # [ 161.805617] nix-daemon[851]: accepted connection from pid 1841, user nix-grpc-daemon (trusted)7032worker1 # [ 161.817274] nix-daemon[851]: reaped child process 1844, status = succeeded7033lb # [ 161.811453] niks3-server[1156]: 2026/09/22 10:37:10 INFO Received uploads request method=POST path=/api/pending_closures7034worker1 # [ 161.832735] nix-grpc-daemon[1576]: time=2026-09-22T10:37:10.053Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7035worker1 # [ 161.834146] nix-grpc-daemon[1576]: time=2026-09-22T10:37:10.054Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"7036lb # [ 161.824716] niks3-server[1156]: 2026/09/22 10:37:10 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst7037lb # [ 161.829652] niks3-server[1156]: 2026/09/22 10:37:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign7038lb # [ 161.831756] niks3-server[1156]: 2026/09/22 10:37:10 INFO Signed narinfos id=58 count=17039worker1 # [ 161.847907] nix-grpc-daemon[1576]: time=2026-09-22T10:37:10.069Z level=INFO msg="Uploading 1 narinfos"7040lb # [ 161.835088] niks3-server[1156]: 2026/09/22 10:37:10 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls7041lb # [ 161.839341] niks3-server[1156]: 2026/09/22 10:37:10 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete7042lb # [ 161.841954] niks3-server[1156]: 2026/09/22 10:37:10 INFO Completed upload id=587043worker1 # [ 161.858252] nix-grpc-daemon[1576]: time=2026-09-22T10:37:10.079Z level=INFO msg="Upload complete. (335ms)"7044worker1 # [ 161.860408] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=5047045lb # [ 161.844884] niks3-server[1156]: 2026/09/22 10:37:10 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo7046worker1 # [ 161.863459] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907047worker1 # [ 161.867735] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7048worker1 # [ 161.872547] nix-daemon[851]: reaped child process 1834, status = succeeded7049worker1 # [ 161.877550] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7050worker1 # [ 162.689725] nix-daemon[851]: reaped child process 1848, status = succeeded7051worker1 # [ 162.811534] nix-daemon[851]: accepted connection from pid 1864, user nix-grpc-daemon (trusted)7052worker1 # [ 162.823565] nix-daemon[851]: reaped child process 1867, status = succeeded7053lb # [ 162.817678] niks3-server[1156]: 2026/09/22 10:37:11 INFO Received uploads request method=POST path=/api/pending_closures7054worker1 # [ 162.838497] nix-grpc-daemon[1576]: time=2026-09-22T10:37:11.059Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7055worker1 # [ 162.840120] nix-grpc-daemon[1576]: time=2026-09-22T10:37:11.061Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"7056lb # [ 162.838725] niks3-server[1156]: 2026/09/22 10:37:11 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst7057lb # [ 162.842256] niks3-server[1156]: 2026/09/22 10:37:11 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv7058lb # [ 162.844133] niks3-server[1156]: 2026/09/22 10:37:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign7059lb # [ 162.846378] niks3-server[1156]: 2026/09/22 10:37:11 INFO Signed narinfos id=59 count=17060worker1 # [ 162.864047] nix-grpc-daemon[1576]: time=2026-09-22T10:37:11.085Z level=INFO msg="Uploading 1 narinfos"7061lb # [ 162.849292] niks3-server[1156]: 2026/09/22 10:37:11 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls7062lb # [ 162.853990] niks3-server[1156]: 2026/09/22 10:37:11 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete7063lb # [ 162.856677] niks3-server[1156]: 2026/09/22 10:37:11 INFO Completed upload id=597064worker1 # [ 162.873030] nix-grpc-daemon[1576]: time=2026-09-22T10:37:11.094Z level=INFO msg="Upload complete. (190ms)"7065lb # [ 162.858814] niks3-server[1156]: 2026/09/22 10:37:11 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo7066worker1 # [ 162.875580] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=17067worker1 # [ 162.879410] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:11Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=17068worker1 # [ 162.881744] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907069worker1 # [ 162.887188] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:11Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=07070worker1 # [ 162.892316] nix-daemon[851]: reaped child process 1846, status = succeeded7071worker1 # [ 162.924429] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480907072worker1 # [ 162.930937] nix-grpc-daemon[1576]: time=2026-09-22T10:37:11.152Z level=INFO msg="All 1 paths already cached"7073worker1 # [ 162.934414] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480907074worker1 # [ 162.938230] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7075worker1 # [ 163.088077] nix-daemon[851]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)7076worker1 # [ 163.100261] nix-daemon[851]: reaped child process 1876, status = succeeded7077worker1 # [ 163.255439] nix-daemon[851]: accepted connection from pid 1878, user nix-grpc-daemon (trusted)7078worker1 # [ 163.267297] nix-daemon[851]: reaped child process 1881, status = succeeded7079lb # [ 163.266179] niks3-server[1156]: 2026/09/22 10:37:12 INFO Received uploads request method=POST path=/api/pending_closures7080worker1 # [ 163.286770] nix-grpc-daemon[1576]: time=2026-09-22T10:37:11.508Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7081worker1 # [ 163.288306] nix-grpc-daemon[1576]: time=2026-09-22T10:37:11.509Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"7082lb # [ 163.282966] niks3-server[1156]: 2026/09/22 10:37:12 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst7083lb # [ 163.285877] niks3-server[1156]: 2026/09/22 10:37:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign7084lb # [ 163.288429] niks3-server[1156]: 2026/09/22 10:37:12 INFO Signed narinfos id=60 count=17085worker1 # [ 163.304688] nix-grpc-daemon[1576]: time=2026-09-22T10:37:11.526Z level=INFO msg="Uploading 1 narinfos"7086lb # [ 163.291232] niks3-server[1156]: 2026/09/22 10:37:12 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls7087lb # [ 163.296026] niks3-server[1156]: 2026/09/22 10:37:12 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete7088lb # [ 163.299679] niks3-server[1156]: 2026/09/22 10:37:12 INFO Completed upload id=607089worker1 # [ 163.315988] nix-grpc-daemon[1576]: time=2026-09-22T10:37:11.537Z level=INFO msg="Upload complete. (371ms)"7090lb # [ 163.301302] niks3-server[1156]: 2026/09/22 10:37:12 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo7091worker1 # [ 163.317961] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=6247092worker1 # [ 163.321468] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907093worker1 # [ 163.327867] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7094worker1 # [ 163.330209] nix-daemon[851]: reaped child process 1871, status = succeeded7095worker1 # [ 163.335582] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7096worker1 # [ 164.548641] nix-daemon[851]: reaped child process 1885, status = succeeded7097worker1 # [ 164.690576] nix-daemon[851]: accepted connection from pid 1901, user nix-grpc-daemon (trusted)7098worker1 # [ 164.702611] nix-daemon[851]: reaped child process 1904, status = succeeded7099lb # [ 164.702593] niks3-server[1156]: 2026/09/22 10:37:13 INFO Received uploads request method=POST path=/api/pending_closures7100worker1 # [ 164.727623] nix-grpc-daemon[1576]: time=2026-09-22T10:37:12.948Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7101worker1 # [ 164.732518] nix-grpc-daemon[1576]: time=2026-09-22T10:37:12.950Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"7102lb # [ 164.743414] niks3-server[1156]: 2026/09/22 10:37:13 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv7103lb # [ 164.748742] niks3-server[1156]: 2026/09/22 10:37:13 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst7104lb # [ 164.763119] niks3-server[1156]: 2026/09/22 10:37:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign7105lb # [ 164.768234] niks3-server[1156]: 2026/09/22 10:37:13 INFO Signed narinfos id=61 count=17106worker1 # [ 164.787223] nix-grpc-daemon[1576]: time=2026-09-22T10:37:13.008Z level=INFO msg="Uploading 1 narinfos"7107lb # [ 164.773491] niks3-server[1156]: 2026/09/22 10:37:13 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls7108lb # [ 164.783468] niks3-server[1156]: 2026/09/22 10:37:13 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete7109lb # [ 164.789723] niks3-server[1156]: 2026/09/22 10:37:13 INFO Completed upload id=617110worker1 # [ 164.806108] nix-grpc-daemon[1576]: time=2026-09-22T10:37:13.027Z level=INFO msg="Upload complete. (262ms)"7111lb # [ 164.791299] niks3-server[1156]: 2026/09/22 10:37:13 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo7112worker1 # [ 164.808695] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=17113worker1 # [ 164.813166] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:13Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=17114worker1 # [ 164.816621] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907115worker1 # [ 164.820273] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:13Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=07116worker1 # [ 164.825476] nix-daemon[851]: reaped child process 1883, status = succeeded7117worker1 # [ 164.826671] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480907118worker1 # [ 164.832596] nix-grpc-daemon[1576]: time=2026-09-22T10:37:13.054Z level=INFO msg="All 1 paths already cached"7119worker1 # [ 164.836374] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480907120worker1 # [ 164.840105] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7121worker1 # [ 164.983746] nix-daemon[851]: accepted connection from pid 1910, user nix-grpc-daemon (trusted)7122worker1 # [ 164.995875] nix-daemon[851]: reaped child process 1913, status = succeeded7123worker1 # [ 165.132759] nix-daemon[851]: accepted connection from pid 1915, user nix-grpc-daemon (trusted)7124worker1 # [ 165.144377] nix-daemon[851]: reaped child process 1918, status = succeeded7125lb # [ 165.141585] niks3-server[1156]: 2026/09/22 10:37:14 INFO Received uploads request method=POST path=/api/pending_closures7126worker1 # [ 165.162583] nix-grpc-daemon[1576]: time=2026-09-22T10:37:13.383Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7127worker1 # [ 165.164042] nix-grpc-daemon[1576]: time=2026-09-22T10:37:13.383Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"7128lb # [ 165.155938] niks3-server[1156]: 2026/09/22 10:37:14 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst7129lb # [ 165.158074] niks3-server[1156]: 2026/09/22 10:37:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign7130lb # [ 165.159952] niks3-server[1156]: 2026/09/22 10:37:14 INFO Signed narinfos id=62 count=17131worker1 # [ 165.177610] nix-grpc-daemon[1576]: time=2026-09-22T10:37:13.398Z level=INFO msg="Uploading 1 narinfos"7132lb # [ 165.163708] niks3-server[1156]: 2026/09/22 10:37:14 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls7133lb # [ 165.168295] niks3-server[1156]: 2026/09/22 10:37:14 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete7134lb # [ 165.171909] niks3-server[1156]: 2026/09/22 10:37:14 INFO Completed upload id=627135worker1 # [ 165.188185] nix-grpc-daemon[1576]: time=2026-09-22T10:37:13.409Z level=INFO msg="Upload complete. (342ms)"7136lb # [ 165.173249] niks3-server[1156]: 2026/09/22 10:37:14 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo7137worker1 # [ 165.190462] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=6247138worker1 # [ 165.193642] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907139worker1 # [ 165.199926] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7140worker1 # [ 165.202391] nix-daemon[851]: reaped child process 1908, status = succeeded7141worker1 # [ 165.208050] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7142worker1 # [ 166.434438] nix-daemon[851]: reaped child process 1922, status = succeeded7143worker1 # [ 166.583685] nix-daemon[851]: accepted connection from pid 1938, user nix-grpc-daemon (trusted)7144worker1 # [ 166.596159] nix-daemon[851]: reaped child process 1941, status = succeeded7145lb # [ 166.595205] niks3-server[1156]: 2026/09/22 10:37:15 INFO Received uploads request method=POST path=/api/pending_closures7146worker1 # [ 166.617582] nix-grpc-daemon[1576]: time=2026-09-22T10:37:14.838Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7147worker1 # [ 166.619143] nix-grpc-daemon[1576]: time=2026-09-22T10:37:14.840Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"7148lb # [ 166.619067] niks3-server[1156]: 2026/09/22 10:37:15 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv7149lb # [ 166.625821] niks3-server[1156]: 2026/09/22 10:37:15 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst7150lb # [ 166.632386] niks3-server[1156]: 2026/09/22 10:37:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign7151worker1 # [ 166.651614] nix-grpc-daemon[1576]: time=2026-09-22T10:37:14.872Z level=INFO msg="Uploading 1 narinfos"7152lb # [ 166.635124] niks3-server[1156]: 2026/09/22 10:37:15 INFO Signed narinfos id=63 count=17153lb # [ 166.639166] niks3-server[1156]: 2026/09/22 10:37:15 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls7154lb # [ 166.645151] niks3-server[1156]: 2026/09/22 10:37:15 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete7155lb # [ 166.647877] niks3-server[1156]: 2026/09/22 10:37:15 INFO Completed upload id=637156worker1 # [ 166.664181] nix-grpc-daemon[1576]: time=2026-09-22T10:37:14.885Z level=INFO msg="Upload complete. (249ms)"7157lb # [ 166.650123] niks3-server[1156]: 2026/09/22 10:37:15 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo7158worker1 # [ 166.666723] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=17159worker1 # [ 166.671636] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:14Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=17160worker1 # [ 166.673740] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907161worker1 # [ 166.677138] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:14Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=07162worker1 # [ 166.683278] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480907163worker1 # [ 166.685497] nix-daemon[851]: reaped child process 1920, status = succeeded7164worker1 # [ 166.689090] nix-grpc-daemon[1576]: time=2026-09-22T10:37:14.910Z level=INFO msg="All 1 paths already cached"7165worker1 # [ 166.692937] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480907166worker1 # [ 166.697940] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7167worker1 # [ 166.853587] nix-daemon[851]: accepted connection from pid 1947, user nix-grpc-daemon (trusted)7168worker1 # [ 166.866491] nix-daemon[851]: reaped child process 1950, status = succeeded7169worker1 # [ 167.020620] nix-daemon[851]: accepted connection from pid 1952, user nix-grpc-daemon (trusted)7170worker1 # [ 167.032630] nix-daemon[851]: reaped child process 1955, status = succeeded7171lb # [ 167.030782] niks3-server[1156]: 2026/09/22 10:37:15 INFO Received uploads request method=POST path=/api/pending_closures7172worker1 # [ 167.055776] nix-grpc-daemon[1576]: time=2026-09-22T10:37:15.277Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7173worker1 # [ 167.057161] nix-grpc-daemon[1576]: time=2026-09-22T10:37:15.277Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"7174lb # [ 167.054631] niks3-server[1156]: 2026/09/22 10:37:15 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst7175lb # [ 167.067549] niks3-server[1156]: 2026/09/22 10:37:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7176worker1 # [ 167.089234] nix-grpc-daemon[1576]: time=2026-09-22T10:37:15.310Z level=INFO msg="Uploading 1 narinfos"7177lb # [ 167.073393] niks3-server[1156]: 2026/09/22 10:37:15 INFO Signed narinfos id=64 count=17178lb # [ 167.078897] niks3-server[1156]: 2026/09/22 10:37:15 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls7179lb # [ 167.088676] niks3-server[1156]: 2026/09/22 10:37:15 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7180worker1 # [ 167.111133] nix-grpc-daemon[1576]: time=2026-09-22T10:37:15.332Z level=INFO msg="Upload complete. (408ms)"7181lb # [ 167.095374] niks3-server[1156]: 2026/09/22 10:37:15 INFO Completed upload id=647182worker1 # [ 167.114190] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=6247183lb # [ 167.098815] niks3-server[1156]: 2026/09/22 10:37:15 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo7184worker1 # [ 167.119232] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907185worker1 # [ 167.125539] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7186worker1 # [ 167.129357] nix-daemon[851]: reaped child process 1945, status = succeeded7187worker1 # [ 167.134389] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7188worker1 # [ 168.362594] nix-daemon[851]: reaped child process 1959, status = succeeded7189worker1 # [ 168.505091] nix-daemon[851]: accepted connection from pid 1975, user nix-grpc-daemon (trusted)7190worker1 # [ 168.516132] nix-daemon[851]: reaped child process 1978, status = succeeded7191lb # [ 168.513383] niks3-server[1156]: 2026/09/22 10:37:17 INFO Received uploads request method=POST path=/api/pending_closures7192worker1 # [ 168.540594] nix-grpc-daemon[1576]: time=2026-09-22T10:37:16.761Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7193worker1 # [ 168.543062] nix-grpc-daemon[1576]: time=2026-09-22T10:37:16.763Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"7194lb # [ 168.555675] niks3-server[1156]: 2026/09/22 10:37:17 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst7195lb # [ 168.565322] niks3-server[1156]: 2026/09/22 10:37:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7196lb # [ 168.571952] niks3-server[1156]: 2026/09/22 10:37:17 INFO Signed narinfos id=65 count=17197worker1 # [ 168.588754] nix-grpc-daemon[1576]: time=2026-09-22T10:37:16.809Z level=INFO msg="Uploading 1 narinfos"7198lb # [ 168.580171] niks3-server[1156]: 2026/09/22 10:37:17 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv7199lb # [ 168.592945] niks3-server[1156]: 2026/09/22 10:37:17 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7200lb # [ 168.596452] niks3-server[1156]: 2026/09/22 10:37:17 INFO Completed upload id=657201worker1 # [ 168.612734] nix-grpc-daemon[1576]: time=2026-09-22T10:37:16.834Z level=INFO msg="Upload complete. (270ms)"7202lb # [ 168.598247] niks3-server[1156]: 2026/09/22 10:37:17 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo7203lb # [ 168.602646] niks3-server[1156]: 2026/09/22 10:37:17 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls7204worker1 # [ 168.621733] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=17205worker1 # [ 168.628474] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:16Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=17206worker1 # [ 168.638246] nix-daemon[851]: reaped child process 1957, status = succeeded7207worker1 # [ 168.663739] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907208worker1 # [ 168.675518] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:16Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=07209worker1 # [ 168.682630] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480907210worker1 # [ 168.695221] nix-grpc-daemon[1576]: time=2026-09-22T10:37:16.916Z level=INFO msg="All 2 paths already cached"7211worker1 # [ 168.702352] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480907212worker1 # [ 168.709582] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7213worker1 # [ 168.872658] nix-daemon[851]: accepted connection from pid 1984, user nix-grpc-daemon (trusted)7214worker1 # [ 168.885162] nix-daemon[851]: reaped child process 1987, status = succeeded7215worker1 # [ 169.046060] nix-daemon[851]: accepted connection from pid 1989, user nix-grpc-daemon (trusted)7216worker1 # [ 169.058218] nix-daemon[851]: reaped child process 1992, status = succeeded7217lb # [ 169.057705] niks3-server[1156]: 2026/09/22 10:37:17 INFO Received uploads request method=POST path=/api/pending_closures7218worker1 # [ 169.080806] nix-grpc-daemon[1576]: time=2026-09-22T10:37:17.301Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7219worker1 # [ 169.082495] nix-grpc-daemon[1576]: time=2026-09-22T10:37:17.302Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7220lb # [ 169.074605] niks3-server[1156]: 2026/09/22 10:37:17 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7221lb # [ 169.078690] niks3-server[1156]: 2026/09/22 10:37:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7222lb # [ 169.081707] niks3-server[1156]: 2026/09/22 10:37:17 INFO Signed narinfos id=66 count=17223worker1 # [ 169.098285] nix-grpc-daemon[1576]: time=2026-09-22T10:37:17.319Z level=INFO msg="Uploading 1 narinfos"7224lb # [ 169.085552] niks3-server[1156]: 2026/09/22 10:37:17 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7225lb # [ 169.089628] niks3-server[1156]: 2026/09/22 10:37:17 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7226lb # [ 169.094781] niks3-server[1156]: 2026/09/22 10:37:17 INFO Completed upload id=667227worker1 # [ 169.111105] nix-grpc-daemon[1576]: time=2026-09-22T10:37:17.332Z level=INFO msg="Upload complete. (390ms)"7228lb # [ 169.096382] niks3-server[1156]: 2026/09/22 10:37:17 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7229worker1 # [ 169.113305] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=7527230worker1 # [ 169.117178] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907231worker1 # [ 169.124217] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7232worker1 # [ 169.126946] nix-daemon[851]: reaped child process 1982, status = succeeded7233worker1 # [ 169.133040] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7234worker1 # [ 169.545371] nix-daemon[851]: reaped child process 1996, status = succeeded7235worker1 # [ 169.667889] nix-daemon[851]: accepted connection from pid 2012, user nix-grpc-daemon (trusted)7236worker1 # [ 169.679117] nix-daemon[851]: reaped child process 2015, status = succeeded7237lb # [ 169.672912] niks3-server[1156]: 2026/09/22 10:37:18 INFO Received uploads request method=POST path=/api/pending_closures7238worker1 # [ 169.699329] nix-grpc-daemon[1576]: time=2026-09-22T10:37:17.920Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7239worker1 # [ 169.700898] nix-grpc-daemon[1576]: time=2026-09-22T10:37:17.922Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7240lb # [ 169.713814] niks3-server[1156]: 2026/09/22 10:37:18 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7241lb # [ 169.720120] niks3-server[1156]: 2026/09/22 10:37:18 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7242lb # [ 169.731166] niks3-server[1156]: 2026/09/22 10:37:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7243lb # [ 169.735855] niks3-server[1156]: 2026/09/22 10:37:18 INFO Signed narinfos id=67 count=17244worker1 # [ 169.755662] nix-grpc-daemon[1576]: time=2026-09-22T10:37:17.976Z level=INFO msg="Uploading 1 narinfos"7245lb # [ 169.748147] niks3-server[1156]: 2026/09/22 10:37:18 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7246lb # [ 169.754854] niks3-server[1156]: 2026/09/22 10:37:18 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7247lb # [ 169.759393] niks3-server[1156]: 2026/09/22 10:37:18 INFO Completed upload id=677248worker1 # [ 169.775943] nix-grpc-daemon[1576]: time=2026-09-22T10:37:17.997Z level=INFO msg="Upload complete. (237ms)"7249lb # [ 169.761176] niks3-server[1156]: 2026/09/22 10:37:18 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7250worker1 # [ 169.783614] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=17251worker1 # [ 169.789570] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:18Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17252worker1 # [ 169.800770] nix-daemon[851]: reaped child process 1994, status = succeeded7253worker1 # [ 169.826349] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907254worker1 # [ 169.839352] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480907255worker1 # [ 169.852143] nix-grpc-daemon[1576]: time=2026-09-22T10:37:18.073Z level=INFO msg="All 2 paths already cached"7256worker1 # [ 169.859633] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480907257worker1 # [ 169.869156] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7258worker1 # [ 170.004968] nix-daemon[851]: accepted connection from pid 2019, user nix-grpc-daemon (trusted)7259worker1 # [ 170.017920] nix-daemon[851]: reaped child process 2022, status = succeeded7260worker1 # [ 170.162627] nix-daemon[851]: accepted connection from pid 2024, user nix-grpc-daemon (trusted)7261worker1 # [ 170.174973] nix-daemon[851]: reaped child process 2027, status = succeeded7262lb # [ 170.167689] niks3-server[1156]: 2026/09/22 10:37:19 INFO Received uploads request method=POST path=/api/pending_closures7263worker1 # [ 170.193271] nix-grpc-daemon[1576]: time=2026-09-22T10:37:18.414Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7264worker1 # [ 170.194578] nix-grpc-daemon[1576]: time=2026-09-22T10:37:18.414Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7265lb # [ 170.193680] niks3-server[1156]: 2026/09/22 10:37:19 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7266lb # [ 170.204360] niks3-server[1156]: 2026/09/22 10:37:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7267lb # [ 170.208836] niks3-server[1156]: 2026/09/22 10:37:19 INFO Signed narinfos id=68 count=17268worker1 # [ 170.228218] nix-grpc-daemon[1576]: time=2026-09-22T10:37:18.449Z level=INFO msg="Uploading 1 narinfos"7269lb # [ 170.220708] niks3-server[1156]: 2026/09/22 10:37:19 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7270lb # [ 170.228437] niks3-server[1156]: 2026/09/22 10:37:19 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7271worker1 # [ 170.251208] nix-grpc-daemon[1576]: time=2026-09-22T10:37:18.472Z level=INFO msg="Upload complete. (371ms)"7272lb # [ 170.234995] niks3-server[1156]: 2026/09/22 10:37:19 INFO Completed upload id=687273worker1 # [ 170.253782] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=7527274lb # [ 170.238738] niks3-server[1156]: 2026/09/22 10:37:19 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7275worker1 # [ 170.259095] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907276worker1 # [ 170.266059] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7277worker1 # [ 170.273098] nix-daemon[851]: reaped child process 2017, status = succeeded7278worker1 # [ 170.276166] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7279worker1 # [ 171.289441] nix-daemon[851]: reaped child process 2031, status = succeeded7280worker1 # [ 171.412954] nix-daemon[851]: accepted connection from pid 2047, user nix-grpc-daemon (trusted)7281worker1 # [ 171.424870] nix-daemon[851]: reaped child process 2050, status = succeeded7282lb # [ 171.420912] niks3-server[1156]: 2026/09/22 10:37:20 INFO Received uploads request method=POST path=/api/pending_closures7283worker1 # [ 171.445447] nix-grpc-daemon[1576]: time=2026-09-22T10:37:19.666Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7284worker1 # [ 171.447048] nix-grpc-daemon[1576]: time=2026-09-22T10:37:19.668Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7285lb # [ 171.458670] niks3-server[1156]: 2026/09/22 10:37:20 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7286lb # [ 171.464394] niks3-server[1156]: 2026/09/22 10:37:20 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7287lb # [ 171.478766] niks3-server[1156]: 2026/09/22 10:37:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7288worker1 # [ 171.498950] nix-grpc-daemon[1576]: time=2026-09-22T10:37:19.720Z level=INFO msg="Uploading 1 narinfos"7289lb # [ 171.482971] niks3-server[1156]: 2026/09/22 10:37:20 INFO Signed narinfos id=69 count=17290lb # [ 171.488236] niks3-server[1156]: 2026/09/22 10:37:20 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7291lb # [ 171.499116] niks3-server[1156]: 2026/09/22 10:37:20 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7292lb # [ 171.503382] niks3-server[1156]: 2026/09/22 10:37:20 INFO Completed upload id=697293worker1 # [ 171.519889] nix-grpc-daemon[1576]: time=2026-09-22T10:37:19.741Z level=INFO msg="Upload complete. (237ms)"7294lb # [ 171.505401] niks3-server[1156]: 2026/09/22 10:37:20 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7295worker1 # [ 171.527427] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=17296worker1 # [ 171.533309] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:19Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17297worker1 # [ 171.542925] nix-daemon[851]: reaped child process 2029, status = succeeded7298worker1 # [ 171.569604] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907299worker1 # [ 171.581475] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:480907300worker1 # [ 171.594079] nix-grpc-daemon[1576]: time=2026-09-22T10:37:19.815Z level=INFO msg="All 2 paths already cached"7301worker1 # [ 171.601676] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:480907302worker1 # [ 171.611157] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7303worker1 # [ 171.761560] nix-daemon[851]: accepted connection from pid 2054, user nix-grpc-daemon (trusted)7304worker1 # [ 171.773548] nix-daemon[851]: reaped child process 2057, status = succeeded7305worker1 # [ 171.918479] nix-daemon[851]: accepted connection from pid 2059, user nix-grpc-daemon (trusted)7306worker1 # [ 171.930332] nix-daemon[851]: reaped child process 2062, status = succeeded7307lb # [ 171.927620] niks3-server[1156]: 2026/09/22 10:37:20 INFO Received uploads request method=POST path=/api/pending_closures7308worker1 # [ 171.952350] nix-grpc-daemon[1576]: time=2026-09-22T10:37:20.173Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7309worker1 # [ 171.953710] nix-grpc-daemon[1576]: time=2026-09-22T10:37:20.173Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7310lb # [ 171.953418] niks3-server[1156]: 2026/09/22 10:37:20 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7311lb # [ 171.966160] niks3-server[1156]: 2026/09/22 10:37:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7312worker1 # [ 171.987365] nix-grpc-daemon[1576]: time=2026-09-22T10:37:20.208Z level=INFO msg="Uploading 1 narinfos"7313lb # [ 171.972251] niks3-server[1156]: 2026/09/22 10:37:20 INFO Signed narinfos id=70 count=17314lb # [ 171.977525] niks3-server[1156]: 2026/09/22 10:37:20 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7315lb # [ 171.990350] niks3-server[1156]: 2026/09/22 10:37:20 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7316lb # [ 171.995325] niks3-server[1156]: 2026/09/22 10:37:20 INFO Completed upload id=707317worker1 # [ 172.011888] nix-grpc-daemon[1576]: time=2026-09-22T10:37:20.233Z level=INFO msg="Upload complete. (389ms)"7318worker1 # [ 172.015067] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=0 paths=1 nar_bytes_in=7527319lb # [ 171.999351] niks3-server[1156]: 2026/09/22 10:37:20 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7320worker1 # [ 172.020412] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:480907321worker1 # [ 172.024499] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7322worker1 # [ 172.026716] nix-daemon[851]: reaped child process 2052, status = succeeded7323worker1 # [ 172.032049] nix-daemon[851]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)7324worker1 # [ 172.857922] nix-daemon[851]: reaped child process 2066, status = succeeded7325worker1 # [ 172.986368] nix-daemon[851]: accepted connection from pid 2082, user nix-grpc-daemon (trusted)7326worker1 # [ 172.998694] nix-daemon[851]: reaped child process 2085, status = succeeded7327lb # [ 172.996922] niks3-server[1156]: 2026/09/22 10:37:21 INFO Received uploads request method=POST path=/api/pending_closures7328worker1 # [ 173.024268] nix-grpc-daemon[1576]: time=2026-09-22T10:37:21.245Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7329worker1 # [ 173.025750] nix-grpc-daemon[1576]: time=2026-09-22T10:37:21.247Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7330lb # [ 173.033524] niks3-server[1156]: 2026/09/22 10:37:21 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7331lb # [ 173.050408] niks3-server[1156]: 2026/09/22 10:37:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7332worker1 # [ 173.072820] nix-grpc-daemon[1576]: time=2026-09-22T10:37:21.294Z level=INFO msg="Uploading 1 narinfos"7333lb # [ 173.056612] niks3-server[1156]: 2026/09/22 10:37:21 INFO Signed narinfos id=71 count=17334lb # [ 173.067095] niks3-server[1156]: 2026/09/22 10:37:21 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7335lb # [ 173.072242] niks3-server[1156]: 2026/09/22 10:37:21 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7336lb # [ 173.076261] niks3-server[1156]: 2026/09/22 10:37:21 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7337lb # [ 173.078713] niks3-server[1156]: 2026/09/22 10:37:21 INFO Completed upload id=717338worker1 # [ 173.094561] nix-grpc-daemon[1576]: time=2026-09-22T10:37:21.315Z level=INFO msg="Upload complete. (255ms)"7339lb # [ 173.081321] niks3-server[1156]: 2026/09/22 10:37:21 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7340worker1 # [ 173.101087] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48090 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=17341worker1 # [ 173.112908] nix-daemon[851]: reaped child process 2064, status = succeeded7342client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 15.99 seconds)7343worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7344worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7345client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'7346client # [ 173.239828] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7347client # Running as unit: gpu.service; invocation ID: c18ad625856c4c0884c9cc41612939ce7348client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.06 seconds)7349lb: must succeed: curl -sf localhost:9901/clusters7350lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)7351lb: must succeed: curl -sf localhost:9901/clusters7352lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7353worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7354worker2 # [ 173.446558] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:372407355worker2 # [ 173.458441] nix-daemon[873]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7356client # [ 173.446596] nix[1473]: don't know how to build these paths:7357client # [ 173.447487] nix[1473]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7358worker2 # [ 173.473448] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372407359worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)7360worker2 # [ 173.485161] nix-grpc-daemon[1970]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 78 ms (attempt 1/5)7361worker2 # [ 173.569214] nix-grpc-daemon[1970]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 186 ms (attempt 2/5)7362worker2 # [ 173.761384] nix-grpc-daemon[1970]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 209 ms (attempt 3/5)7363worker2 # [ 173.976246] nix-grpc-daemon[1970]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 247 ms (attempt 4/5)7364worker2 # [ 174.229421] nix-grpc-daemon[1970]: disabling binary cache 'https://cache.nixos.org' for 60 seconds7365worker2 # [ 174.233242] nix-grpc-daemon[1970]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org7366worker1 # [ 174.239607] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:22Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07367client # [ 174.236877] nix[1473]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker7368lb: must succeed: curl -sf localhost:9901/clusters7369lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)7370lb: must succeed: curl -sf localhost:9901/clusters7371lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7372worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7373worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)7374client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7375lb # [ 178.651589] envoy[923]: conn peer=[2001:db8:1::1]:38130 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27107 tx=12267 ms=157407376client # [ 204.237519] nix[1473]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}7377client # [ 204.243407] nix[1473]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7378client # [ 204.268124] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7379client # [ 204.269945] systemd[1]: gpu.service: Failed with result 'exit-code'.7380client # [ 204.271663] systemd[1]: gpu.service: Consumed 139ms CPU time over 31.030s wall clock time, 37.4M memory peak, 5.4K incoming IP traffic, 5.1K outgoing IP traffic.7381client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 30.57 seconds)7382client: must succeed: journalctl -u gpu -o cat7383client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)7384lb: must succeed: curl -sf localhost:9901/clusters7385lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)7386lb: must succeed: curl -sf localhost:9901/clusters7387lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7388worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7389worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)7390(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 48.26 seconds)7391subtest: low disk drains a worker and builds go to the other7392worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7393worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.93 seconds)7394worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7395worker1 # [ 207.202335] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:55Z level=info event=unhealthy reason=min_free7396worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.11 seconds)7397client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&27398client # don't know how to build these paths:7399worker1 # [ 207.686028] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:55Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:480907400client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7401worker2 # [ 207.711765] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372407402worker1 # [ 207.710638] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:55Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07403worker1 # [ 207.715417] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:55Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17404worker2 # [ 207.775666] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372407405worker2 # [ 207.789274] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372407406client # copying 1 paths to 'lb:50051'...7407client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7408worker2 # [ 207.802778] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:372407409worker2 # [ 207.815172] nix-daemon[873]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7410worker2 # [ 207.833483] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:56Z level=info event=niks3_push_started pid=20307411worker2 # [ 208.023674] nix-grpc-daemon[2030]: time=2026-09-22T10:37:57.050Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""7412worker2 # [ 208.177108] nix-daemon[873]: accepted connection from pid 2043, user nix-grpc-daemon (trusted)7413worker2 # [ 208.189555] nix-daemon[873]: reaped child process 2046, status = succeeded7414worker2 # [ 208.382214] nix-daemon[873]: accepted connection from pid 2048, user nix-grpc-daemon (trusted)7415worker2 # [ 208.397209] nix-daemon[873]: reaped child process 2051, status = succeeded7416lb # [ 208.374670] niks3-server[1156]: 2026/09/22 10:37:57 INFO Received uploads request method=POST path=/api/pending_closures7417worker2 # [ 208.420087] nix-grpc-daemon[2030]: time=2026-09-22T10:37:57.446Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7418worker2 # [ 208.421472] nix-grpc-daemon[2030]: time=2026-09-22T10:37:57.446Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7419lb # [ 208.406070] niks3-server[1156]: 2026/09/22 10:37:57 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7420lb # [ 208.416156] niks3-server[1156]: 2026/09/22 10:37:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7421worker2 # [ 208.456717] nix-grpc-daemon[2030]: time=2026-09-22T10:37:57.483Z level=INFO msg="Uploading 1 narinfos"7422lb # [ 208.420669] niks3-server[1156]: 2026/09/22 10:37:57 INFO Signed narinfos id=72 count=17423lb # [ 208.431684] niks3-server[1156]: 2026/09/22 10:37:57 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7424lb # [ 208.441396] niks3-server[1156]: 2026/09/22 10:37:57 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7425worker2 # [ 208.484486] nix-grpc-daemon[2030]: time=2026-09-22T10:37:57.512Z level=INFO msg="Upload complete. (458ms)"7426lb # [ 208.448179] niks3-server[1156]: 2026/09/22 10:37:57 INFO Completed upload id=727427lb # [ 208.450980] niks3-server[1156]: 2026/09/22 10:37:57 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7428worker2 # [ 208.487514] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37240 duration_s=0 paths=1 nar_bytes_in=5127429worker2 # [ 208.494300] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372407430worker2 # [ 208.503898] nix-daemon[873]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7431worker2 # [ 208.510171] nix-daemon[873]: reaped child process 2028, status = succeeded7432worker2 # [ 208.513248] nix-daemon[873]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7433client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7434client # farm-a> worker2-next: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7435lb # [ 208.653629] envoy[923]: conn peer=[2001:db8:1::1]:38640 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=912 tx=517 ms=308367436client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7437client # farm-a> LOG-farm-a-drain7438worker2 # [ 209.127866] nix-daemon[873]: reaped child process 2055, status = succeeded7439worker2 # [ 209.283572] nix-daemon[873]: accepted connection from pid 2071, user nix-grpc-daemon (trusted)7440worker2 # [ 209.297360] nix-daemon[873]: reaped child process 2074, status = succeeded7441lb # [ 209.277773] niks3-server[1156]: 2026/09/22 10:37:58 INFO Received uploads request method=POST path=/api/pending_closures7442worker2 # [ 209.322528] nix-grpc-daemon[2030]: time=2026-09-22T10:37:58.349Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7443worker2 # [ 209.324345] nix-grpc-daemon[2030]: time=2026-09-22T10:37:58.351Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7444lb # [ 209.312594] niks3-server[1156]: 2026/09/22 10:37:58 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7445lb # [ 209.325340] niks3-server[1156]: 2026/09/22 10:37:58 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7446lb # [ 209.337611] niks3-server[1156]: 2026/09/22 10:37:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7447worker2 # [ 209.379502] nix-grpc-daemon[2030]: time=2026-09-22T10:37:58.406Z level=INFO msg="Uploading 1 narinfos"7448lb # [ 209.342964] niks3-server[1156]: 2026/09/22 10:37:58 INFO Signed narinfos id=73 count=17449lb # [ 209.347388] niks3-server[1156]: 2026/09/22 10:37:58 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7450lb # [ 209.356193] niks3-server[1156]: 2026/09/22 10:37:58 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7451lb # [ 209.361969] niks3-server[1156]: 2026/09/22 10:37:58 INFO Completed upload id=737452worker2 # [ 209.398475] nix-grpc-daemon[2030]: time=2026-09-22T10:37:58.426Z level=INFO msg="Upload complete. (279ms)"7453lb # [ 209.364755] niks3-server[1156]: 2026/09/22 10:37:58 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7454worker2 # [ 209.404696] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37240 duration_s=0 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=17455worker1 # [ 209.389078] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:57Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07456worker1 # [ 209.390799] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:57Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17457worker2 # [ 209.418209] nix-daemon[873]: reaped child process 2053, status = succeeded7458worker2 # [ 209.453705] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372407459worker2 # [ 209.466505] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372407460worker2 # [ 209.479515] nix-grpc-daemon[2030]: time=2026-09-22T10:37:58.507Z level=INFO msg="All 1 paths already cached"7461client # copying 1 paths to 'lb:50051'...7462client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7463worker2 # [ 209.488100] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:372407464worker2 # [ 209.498169] nix-daemon[873]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7465worker2 # [ 209.670457] nix-daemon[873]: accepted connection from pid 2078, user nix-grpc-daemon (trusted)7466worker2 # [ 209.684394] nix-daemon[873]: reaped child process 2081, status = succeeded7467worker2 # [ 209.859650] nix-daemon[873]: accepted connection from pid 2083, user nix-grpc-daemon (trusted)7468worker2 # [ 209.873170] nix-daemon[873]: reaped child process 2086, status = succeeded7469lb # [ 209.853190] niks3-server[1156]: 2026/09/22 10:37:58 INFO Received uploads request method=POST path=/api/pending_closures7470worker2 # [ 209.894435] nix-grpc-daemon[2030]: time=2026-09-22T10:37:58.921Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7471worker2 # [ 209.895862] nix-grpc-daemon[2030]: time=2026-09-22T10:37:58.921Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7472lb # [ 209.867155] niks3-server[1156]: 2026/09/22 10:37:58 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7473lb # [ 209.871256] niks3-server[1156]: 2026/09/22 10:37:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7474lb # [ 209.873896] niks3-server[1156]: 2026/09/22 10:37:58 INFO Signed narinfos id=74 count=17475worker2 # [ 209.910511] nix-grpc-daemon[2030]: time=2026-09-22T10:37:58.938Z level=INFO msg="Uploading 1 narinfos"7476lb # [ 209.875782] niks3-server[1156]: 2026/09/22 10:37:58 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7477lb # [ 209.881941] niks3-server[1156]: 2026/09/22 10:37:58 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7478lb # [ 209.884307] niks3-server[1156]: 2026/09/22 10:37:58 INFO Completed upload id=747479worker2 # [ 209.921932] nix-grpc-daemon[2030]: time=2026-09-22T10:37:58.949Z level=INFO msg="Upload complete. (411ms)"7480lb # [ 209.886147] niks3-server[1156]: 2026/09/22 10:37:58 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7481worker2 # [ 209.924230] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37240 duration_s=0 paths=1 nar_bytes_in=6247482worker2 # [ 209.927373] nix-grpc-daemon[1970]: ts=2026-09-22T10:37:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372407483worker2 # [ 209.932557] nix-daemon[873]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7484worker2 # [ 209.937395] nix-daemon[873]: reaped child process 2076, status = succeeded7485worker2 # [ 209.942310] nix-daemon[873]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7486client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7487client # farm-b> worker2-next: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7488client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7489client # farm-b> LOG-farm-b-drain7490worker2 # [ 210.964408] nix-daemon[873]: reaped child process 2090, status = succeeded7491worker2 # [ 211.123208] nix-daemon[873]: accepted connection from pid 2106, user nix-grpc-daemon (trusted)7492worker2 # [ 211.137209] nix-daemon[873]: reaped child process 2109, status = succeeded7493lb # [ 211.117075] niks3-server[1156]: 2026/09/22 10:38:00 INFO Received uploads request method=POST path=/api/pending_closures7494worker2 # [ 211.157783] nix-grpc-daemon[2030]: time=2026-09-22T10:38:00.185Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7495worker2 # [ 211.159355] nix-grpc-daemon[2030]: time=2026-09-22T10:38:00.187Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7496lb # [ 211.139326] niks3-server[1156]: 2026/09/22 10:38:00 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7497lb # [ 211.140918] niks3-server[1156]: 2026/09/22 10:38:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7498lb # [ 211.143580] niks3-server[1156]: 2026/09/22 10:38:00 INFO Signed narinfos id=75 count=17499worker2 # [ 211.180245] nix-grpc-daemon[2030]: time=2026-09-22T10:38:00.207Z level=INFO msg="Uploading 1 narinfos"7500lb # [ 211.145641] niks3-server[1156]: 2026/09/22 10:38:00 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7501lb # [ 211.149048] niks3-server[1156]: 2026/09/22 10:38:00 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7502lb # [ 211.153051] niks3-server[1156]: 2026/09/22 10:38:00 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7503lb # [ 211.156178] niks3-server[1156]: 2026/09/22 10:38:00 INFO Completed upload id=757504worker2 # [ 211.192494] nix-grpc-daemon[2030]: time=2026-09-22T10:38:00.220Z level=INFO msg="Upload complete. (242ms)"7505lb # [ 211.158278] niks3-server[1156]: 2026/09/22 10:38:00 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7506worker2 # [ 211.195262] nix-grpc-daemon[1970]: ts=2026-09-22T10:38:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37240 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=17507worker2 # [ 211.202448] nix-daemon[873]: reaped child process 2088, status = succeeded7508worker1 # [ 211.184765] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:59Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07509worker1 # [ 211.189435] nix-grpc-daemon[1484]: ts=2026-09-22T10:37:59Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17510worker2 # [ 211.253137] nix-grpc-daemon[1970]: ts=2026-09-22T10:38:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372407511worker2 # [ 211.262903] nix-grpc-daemon[1970]: ts=2026-09-22T10:38:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372407512worker2 # [ 211.273266] nix-grpc-daemon[2030]: time=2026-09-22T10:38:00.300Z level=INFO msg="All 2 paths already cached"7513client # copying 1 paths to 'lb:50051'...7514client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7515worker2 # [ 211.281746] nix-grpc-daemon[1970]: ts=2026-09-22T10:38:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:372407516worker2 # [ 211.291137] nix-daemon[873]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7517worker2 # [ 211.451543] nix-daemon[873]: accepted connection from pid 2113, user nix-grpc-daemon (trusted)7518worker2 # [ 211.465483] nix-daemon[873]: reaped child process 2116, status = succeeded7519worker2 # [ 211.622585] nix-daemon[873]: accepted connection from pid 2118, user nix-grpc-daemon (trusted)7520worker2 # [ 211.636493] nix-daemon[873]: reaped child process 2121, status = succeeded7521lb # [ 211.612906] niks3-server[1156]: 2026/09/22 10:38:00 INFO Received uploads request method=POST path=/api/pending_closures7522worker2 # [ 211.656838] nix-grpc-daemon[2030]: time=2026-09-22T10:38:00.684Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7523worker2 # [ 211.658484] nix-grpc-daemon[2030]: time=2026-09-22T10:38:00.686Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7524lb # [ 211.643718] niks3-server[1156]: 2026/09/22 10:38:00 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7525lb # [ 211.652374] niks3-server[1156]: 2026/09/22 10:38:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign7526lb # [ 211.658239] niks3-server[1156]: 2026/09/22 10:38:00 INFO Signed narinfos id=76 count=17527worker2 # [ 211.697563] nix-grpc-daemon[2030]: time=2026-09-22T10:38:00.725Z level=INFO msg="Uploading 1 narinfos"7528lb # [ 211.662195] niks3-server[1156]: 2026/09/22 10:38:00 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7529lb # [ 211.674139] niks3-server[1156]: 2026/09/22 10:38:00 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete7530lb # [ 211.679756] niks3-server[1156]: 2026/09/22 10:38:00 INFO Completed upload id=767531worker2 # [ 211.716225] nix-grpc-daemon[2030]: time=2026-09-22T10:38:00.743Z level=INFO msg="Upload complete. (414ms)"7532lb # [ 211.683775] niks3-server[1156]: 2026/09/22 10:38:00 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7533worker2 # [ 211.720186] nix-grpc-daemon[1970]: ts=2026-09-22T10:38:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37240 duration_s=0 paths=1 nar_bytes_in=7527534worker2 # [ 211.726224] nix-grpc-daemon[1970]: ts=2026-09-22T10:38:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372407535worker2 # [ 211.734109] nix-daemon[873]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7536worker2 # [ 211.739203] nix-daemon[873]: reaped child process 2111, status = succeeded7537worker2 # [ 211.744192] nix-daemon[873]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7538client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7539client # farm-top> worker2-next: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7540client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7541client # farm-top> LOG-farm-top-drain7542worker2 # [ 212.959534] nix-daemon[873]: reaped child process 2125, status = succeeded7543worker2 # [ 213.108691] nix-daemon[873]: accepted connection from pid 2141, user nix-grpc-daemon (trusted)7544worker2 # [ 213.122672] nix-daemon[873]: reaped child process 2144, status = succeeded7545lb # [ 213.102767] niks3-server[1156]: 2026/09/22 10:38:01 INFO Received uploads request method=POST path=/api/pending_closures7546worker2 # [ 213.147564] nix-grpc-daemon[2030]: time=2026-09-22T10:38:02.174Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7547worker2 # [ 213.150061] nix-grpc-daemon[2030]: time=2026-09-22T10:38:02.176Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7548lb # [ 213.138109] niks3-server[1156]: 2026/09/22 10:38:02 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7549lb # [ 213.146567] niks3-server[1156]: 2026/09/22 10:38:02 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7550lb # [ 213.155923] niks3-server[1156]: 2026/09/22 10:38:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign7551lb # [ 213.161506] niks3-server[1156]: 2026/09/22 10:38:02 INFO Signed narinfos id=77 count=17552worker2 # [ 213.200839] nix-grpc-daemon[2030]: time=2026-09-22T10:38:02.228Z level=INFO msg="Uploading 1 narinfos"7553lb # [ 213.166995] niks3-server[1156]: 2026/09/22 10:38:02 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7554lb # [ 213.175825] niks3-server[1156]: 2026/09/22 10:38:02 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete7555lb # [ 213.181230] niks3-server[1156]: 2026/09/22 10:38:02 INFO Completed upload id=777556worker2 # [ 213.217680] nix-grpc-daemon[2030]: time=2026-09-22T10:38:02.245Z level=INFO msg="Upload complete. (266ms)"7557lb # [ 213.185216] niks3-server[1156]: 2026/09/22 10:38:02 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7558worker2 # [ 213.222844] nix-grpc-daemon[1970]: ts=2026-09-22T10:38:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37240 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=17559worker2 # [ 213.235671] nix-daemon[873]: reaped child process 2123, status = succeeded7560client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 5.76 seconds)7561client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7562client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.14 seconds)7563worker1: must succeed: rm /nix/.rw-store/fill7564worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.10 seconds)7565worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7566worker1 # [ 214.210148] nix-grpc-daemon[1484]: ts=2026-09-22T10:38:02Z level=info event=healthy reason=min_free7567worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.11 seconds)7568(finished: subtest: low disk drains a worker and builds go to the other, in 9.22 seconds)7569(finished: run the VM test script, in 215.52 seconds)7570test script finished in 215.77s7571cleanup7572kill QemuMachine (pid 49)7573client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7574kill QemuMachine (pid 50)7575lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7576kill QemuMachine (pid 51)7577worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7578kill QemuMachine (pid 52)7579worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7580(finished: cleanup, in 0.98 seconds)7581additionally exposed symbols:7582 client, lb, worker1, worker2,7583 vlan1,7584 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh