vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #465
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12lb # Disk image does not exist, creating the virtualisation disk image...13lb: QEMU running (pid 50)14client # Disk image does not exist, creating the virtualisation disk image...15lb # Formatting '/build/vm-state-lb/tmp.x0joBTv29W', fmt=raw size=107374182416client # Formatting '/build/vm-state-client/tmp.BzyKuyO1bx', fmt=raw size=107374182417lb # mke2fs 1.47.4 (6-Mar-2025)18worker2 # Disk image does not exist, creating the virtualisation disk image...19worker2: QEMU running (pid 52)20lb # Discarding device blocks: 0/262144 done21worker2 # Formatting '/build/vm-state-worker2/tmp.DetQJ02clW', fmt=raw size=107374182422client # mke2fs 1.47.4 (6-Mar-2025)23lb # Creating filesystem with 262144 4k blocks and 65536 inodes24worker1: QEMU running (pid 51)25worker1 # Disk image does not exist, creating the virtualisation disk image...26client: QEMU running (pid 49)27worker2 # mke2fs 1.47.4 (6-Mar-2025)28worker1 # Formatting '/build/vm-state-worker1/tmp.YQUgaNxuKg', fmt=raw size=107374182429client # Discarding device blocks: 0/262144 done30lb # Filesystem UUID: 84c6e6da-ded6-4164-8cf6-fbfbf6fd2e7631worker2 # Discarding device blocks: 0/262144 done32worker1 # mke2fs 1.47.4 (6-Mar-2025)33lb # Superblock backups stored on blocks:34worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes35worker1 # Discarding device blocks: 0/262144 done36lb # 32768, 98304, 163840, 22937637worker2 # Filesystem UUID: 7353f272-2f7f-4e8d-911d-734d826d63c638worker2 # Superblock backups stored on blocks:39(finished: start all VMs, in 0.22 seconds)40worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes41worker2 # 32768, 98304, 163840, 22937642client # Creating filesystem with 262144 4k blocks and 65536 inodes43worker1 # Filesystem UUID: d6460152-30df-4be2-92a9-6bfc3d096bd644worker1 # Superblock backups stored on blocks:45worker2 # 46client # Filesystem UUID: ce1a57c7-7c64-4218-9beb-2feb1b67a06047worker1 # 32768, 98304, 163840, 22937648lb # 49client: waiting for unit network-addresses-eth1.service50worker2 # Allocating group tables: 0/8 done51lb # Allocating group tables: 0/8 done52client: waiting for the VM to finish booting53worker2 # Writing inode tables: 0/8 done54lb # Writing inode tables: 0/8 done55worker1 # 56client # Superblock backups stored on blocks:57worker2 # Creating journal (8192 blocks): done58lb # Creating journal (8192 blocks): done59worker1 # Allocating group tables: 0/8 done60client # 32768, 98304, 163840, 22937661worker2 # Writing superblocks and filesystem accounting information: 0/8 done62lb # Writing superblocks and filesystem accounting information: 0/8 done63worker1 # Writing inode tables: 0/8 done64client # 65worker2 # 66lb # 67worker1 # Creating journal (8192 blocks): done68client # Allocating group tables: 0/8 done69worker2 # Virtualisation disk image created.70lb # Virtualisation disk image created.71worker1 # Writing superblocks and filesystem accounting information: 0/8 done72client # Writing inode tables: 0/8 done73worker1 # 74client # Creating journal (8192 blocks): done75worker1 # Virtualisation disk image created.76client # Writing superblocks and filesystem accounting information: 0/8 done77client # 78client # Virtualisation disk image created.79worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83worker2 # 84worker2 # 85worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0086client # 87client # 88client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0089worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...90worker2 # 91worker2 # 92worker2 # 93worker2 # 94worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB0095client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...96client # 97client # 98client # 99client # 100client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00101worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...102worker2 # 103worker2 # 104client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...105client # 106client # 107lb # 108lb # 109lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00110worker1 # 111worker1 # 112worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00113lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...114lb # 115lb # 116lb # 117lb # 118lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00119worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...120worker1 # 121worker1 # 122worker1 # 123worker1 # 124worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00125lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...126lb # 127lb # 128worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129worker1 # 130worker1 # 131worker2 # Booting from ROM...132client # Booting from ROM...133lb # Booting from ROM...134worker1 # Booting from ROM...135worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/mjmhix2nlchxpsyc8ymc8ijbl9kw4847-nixos-system-worker2-test/init regInfo=/nix/store/f92hdajrqvjpsw7r18x533lxyg4b61q5-closure-info/registration console=ttyS0,115200n8 console=tty0137worker2 # [ 0.000000] BIOS-provided physical RAM map:138worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026141worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved142worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable143worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved144worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved145worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved146client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/0v4f337qkyj5dmshnqw4pzwr5c8f8pgy-nixos-system-client-test/init regInfo=/nix/store/f9gdw55mpfid0rm8r2v7250yzv4pkjal-closure-info/registration console=ttyS0,115200n8 console=tty0147worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved148client # [ 0.000000] BIOS-provided physical RAM map:149worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved150client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable151worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved152client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved153worker2 # [ 0.000000] NX (Execute Disable) protection: active154client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved155worker2 # [ 0.000000] APIC: Static calls initialized156worker2 # [ 0.000000] SMBIOS 2.8 present.157client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable158client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved159worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014160worker2 # [ 0.000000] DMI: Memory slots populated: 1/1161client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved162worker2 # [ 0.000000] Hypervisor detected: KVM163client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved164worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000165worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00166client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved167worker2 # [ 0.000001] kvm-clock: using sched offset of 460995715 cycles168client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved169client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved170worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns171client # [ 0.000000] NX (Execute Disable) protection: active172worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor173client # [ 0.000000] APIC: Static calls initialized174worker2 # [ 0.000810] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000175client # [ 0.000000] SMBIOS 2.8 present.176worker2 # [ 0.000837] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs177client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014178worker2 # [ 0.000840] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT179client # [ 0.000000] DMI: Memory slots populated: 1/1180client # [ 0.000000] Hypervisor detected: KVM181worker2 # [ 0.002737] found SMP MP-table at [mem 0x000f5450-0x000f545f]182client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000183worker2 # [ 0.002748] Using GB pages for direct mapping184worker2 # [ 0.002875] RAMDISK: [mem 0x5e367000-0x5ffcffff]185client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00186worker2 # [ 0.002884] ACPI: Early table checksum verification disabled187client # [ 0.000000] kvm-clock: using sched offset of 460520547 cycles188worker2 # [ 0.002887] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )189client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns190worker2 # [ 0.002890] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)191client # [ 0.000004] tsc: Detected 2400.012 MHz processor192client # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000193worker2 # [ 0.002894] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)194client # [ 0.000839] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs195worker2 # [ 0.002900] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)196worker2 # [ 0.002903] ACPI: FACS 0x000000005FFE0000 000040197client # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT198client # [ 0.002736] found SMP MP-table at [mem 0x000f5450-0x000f545f]199worker2 # [ 0.002904] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)200client # [ 0.002747] Using GB pages for direct mapping201client # [ 0.002887] RAMDISK: [mem 0x5e366000-0x5ffcffff]202worker2 # [ 0.002906] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)203client # [ 0.002896] ACPI: Early table checksum verification disabled204worker2 # [ 0.002907] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)205client # [ 0.002898] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )206worker2 # [ 0.002909] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)207client # [ 0.002902] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)208lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026209worker2 # [ 0.002910] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]210client # [ 0.002906] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)211worker2 # [ 0.002911] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]212client # [ 0.002913] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)213worker2 # [ 0.002912] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]214client # [ 0.002916] ACPI: FACS 0x000000005FFE0000 000040215worker2 # [ 0.002912] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]216client # [ 0.002918] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)217lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/dlr5gkwdzlpfiks0wm9k3cgbj9l1v40b-nixos-system-lb-test/init regInfo=/nix/store/493p756s0f7jvdvcriannaqb5380jad0-closure-info/registration console=ttyS0,115200n8 console=tty0218worker2 # [ 0.002913] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]219lb # [ 0.000000] BIOS-provided physical RAM map:220client # [ 0.002920] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)221worker2 # [ 0.002913] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]222lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable223worker2 # [ 0.002913] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]224client # [ 0.002921] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)225lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved226worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026227worker2 # [ 0.003141] No NUMA configuration found228client # [ 0.002923] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)229lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved230worker2 # [ 0.003142] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]231client # [ 0.002924] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]232worker2 # [ 0.003144] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]233lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable234worker2 # [ 0.005436] Zone ranges:235lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved236client # [ 0.002925] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]237worker2 # [ 0.005437] DMA [mem 0x0000000000001000-0x0000000000ffffff]238lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved239client # [ 0.002926] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]240worker2 # [ 0.005438] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]241worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/k4mxfnh5a0g0aavqsj8h0grq5gn5cki6-nixos-system-worker1-test/init regInfo=/nix/store/j5nrz0mrj1ngq2p5dwyv942lpc0ifh7p-closure-info/registration console=ttyS0,115200n8 console=tty0242worker2 # [ 0.005440] Normal empty243client # [ 0.002926] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]244worker1 # [ 0.000000] BIOS-provided physical RAM map:245lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved246worker2 # [ 0.005440] Device empty247worker2 # [ 0.005441] Movable zone start for each node248worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable249client # [ 0.002926] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]250worker2 # [ 0.005442] Early memory node ranges251lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved252lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved253worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved254client # [ 0.002927] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]255worker2 # [ 0.005442] node 0: [mem 0x0000000000001000-0x000000000009efff]256lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved257worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved258client # [ 0.002927] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]259worker2 # [ 0.005443] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]260client # [ 0.003159] No NUMA configuration found261lb # [ 0.000000] NX (Execute Disable) protection: active262worker2 # [ 0.005444] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]263worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable264lb # [ 0.000000] APIC: Static calls initialized265client # [ 0.003160] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]266lb # [ 0.000000] SMBIOS 2.8 present.267worker2 # [ 0.005463] On node 0, zone DMA: 1 pages in unavailable ranges268worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved269client # [ 0.003162] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]270worker2 # [ 0.005729] On node 0, zone DMA: 97 pages in unavailable ranges271client # [ 0.005488] Zone ranges:272worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved273lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014274worker2 # [ 0.033083] On node 0, zone DMA32: 40 pages in unavailable ranges275client # [ 0.005488] DMA [mem 0x0000000000001000-0x0000000000ffffff]276lb # [ 0.000000] DMI: Memory slots populated: 1/1277worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved278worker2 # [ 0.033531] ACPI: PM-Timer IO Port: 0x608279lb # [ 0.000000] Hypervisor detected: KVM280client # [ 0.005490] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]281worker2 # [ 0.033541] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])282client # [ 0.005491] Normal empty283worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved284lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000285client # [ 0.005492] Device empty286lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00287worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved288client # [ 0.005492] Movable zone start for each node289worker2 # [ 0.033567] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23290client # [ 0.005493] Early memory node ranges291lb # [ 0.000001] kvm-clock: using sched offset of 486881885 cycles292worker2 # [ 0.033569] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)293worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved294client # [ 0.005493] node 0: [mem 0x0000000000001000-0x000000000009efff]295worker1 # [ 0.000000] NX (Execute Disable) protection: active296worker2 # [ 0.033571] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)297lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns298worker1 # [ 0.000000] APIC: Static calls initialized299client # [ 0.005494] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]300worker2 # [ 0.033572] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)301lb # [ 0.000005] tsc: Detected 2400.012 MHz processor302worker1 # [ 0.000000] SMBIOS 2.8 present.303client # [ 0.005495] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]304lb # [ 0.000810] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000305worker2 # [ 0.033573] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)306client # [ 0.005519] On node 0, zone DMA: 1 pages in unavailable ranges307worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014308worker2 # [ 0.033574] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)309lb # [ 0.000837] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs310worker1 # [ 0.000000] DMI: Memory slots populated: 1/1311client # [ 0.005789] On node 0, zone DMA: 97 pages in unavailable ranges312worker1 # [ 0.000000] Hypervisor detected: KVM313worker2 # [ 0.033577] ACPI: Using ACPI (MADT) for SMP configuration information314lb # [ 0.000840] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT315client # [ 0.032947] On node 0, zone DMA32: 40 pages in unavailable ranges316worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000317worker2 # [ 0.033578] ACPI: HPET id: 0x8086a201 base: 0xfed00000318client # [ 0.033409] ACPI: PM-Timer IO Port: 0x608319lb # [ 0.002736] found SMP MP-table at [mem 0x000f5450-0x000f545f]320worker2 # [ 0.033581] TSC deadline timer available321worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00322lb # [ 0.002747] Using GB pages for direct mapping323client # [ 0.033420] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])324worker2 # [ 0.033585] CPU topo: Max. logical packages: 1325worker1 # [ 0.000001] kvm-clock: using sched offset of 485418643 cycles326lb # [ 0.002877] RAMDISK: [mem 0x5e367000-0x5ffcffff]327worker2 # [ 0.033586] CPU topo: Max. logical dies: 1328client # [ 0.033446] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23329lb # [ 0.002886] ACPI: Early table checksum verification disabled330worker2 # [ 0.033586] CPU topo: Max. dies per package: 1331client # [ 0.033448] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)332worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns333worker2 # [ 0.033590] CPU topo: Max. threads per core: 1334lb # [ 0.002888] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )335worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor336client # [ 0.033450] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)337worker2 # [ 0.033590] CPU topo: Num. cores per package: 1338worker1 # [ 0.000817] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000339lb # [ 0.002893] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)340worker2 # [ 0.033591] CPU topo: Num. threads per package: 1341client # [ 0.033451] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)342worker2 # [ 0.033591] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs343worker1 # [ 0.000844] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs344client # [ 0.033452] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)345lb # [ 0.002897] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)346worker2 # [ 0.033609] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()347worker1 # [ 0.000847] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT348client # [ 0.033453] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)349lb # [ 0.002904] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)350worker1 # [ 0.002743] found SMP MP-table at [mem 0x000f5450-0x000f545f]351worker2 # [ 0.033639] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]352lb # [ 0.002906] ACPI: FACS 0x000000005FFE0000 000040353client # [ 0.033455] ACPI: Using ACPI (MADT) for SMP configuration information354worker1 # [ 0.002754] Using GB pages for direct mapping355client # [ 0.033456] ACPI: HPET id: 0x8086a201 base: 0xfed00000356worker2 # [ 0.033641] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]357lb # [ 0.002907] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)358worker1 # [ 0.002877] RAMDISK: [mem 0x5e367000-0x5ffcffff]359client # [ 0.033459] TSC deadline timer available360worker1 # [ 0.002887] ACPI: Early table checksum verification disabled361worker2 # [ 0.033642] [mem 0x60000000-0xafffffff] available for PCI devices362client # [ 0.033463] CPU topo: Max. logical packages: 1363lb # [ 0.002909] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)364worker2 # [ 0.033643] Booting paravirtualized kernel on KVM365client # [ 0.033464] CPU topo: Max. logical dies: 1366worker1 # [ 0.002889] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )367client # [ 0.033464] CPU topo: Max. dies per package: 1368lb # [ 0.002911] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)369worker1 # [ 0.002893] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)370worker2 # [ 0.033646] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns371client # [ 0.033468] CPU topo: Max. threads per core: 1372lb # [ 0.002912] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)373client # [ 0.033468] CPU topo: Num. cores per package: 1374worker2 # [ 0.038093] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1375worker1 # [ 0.002897] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)376client # [ 0.033468] CPU topo: Num. threads per package: 1377lb # [ 0.002914] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]378worker2 # [ 0.040286] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152379client # [ 0.033469] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs380worker1 # [ 0.002904] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)381lb # [ 0.002915] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]382worker2 # [ 0.040328] kvm-guest: PV spinlocks disabled, single CPU383worker1 # [ 0.002906] ACPI: FACS 0x000000005FFE0000 000040384client # [ 0.033487] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()385lb # [ 0.002915] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]386worker1 # [ 0.002907] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)387client # [ 0.033517] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]388lb # [ 0.002916] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]389client # [ 0.033519] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]390worker1 # [ 0.002909] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)391lb # [ 0.002916] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]392client # [ 0.033520] [mem 0x60000000-0xafffffff] available for PCI devices393worker1 # [ 0.002911] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)394lb # [ 0.002917] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]395client # [ 0.033522] Booting paravirtualized kernel on KVM396worker2 # [ 0.040330] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/mjmhix2nlchxpsyc8ymc8ijbl9kw4847-nixos-system-worker2-test/init regInfo=/nix/store/f92hdajrqvjpsw7r18x533lxyg4b61q5-closure-info/registration console=ttyS0,115200n8 console=tty0397lb # [ 0.002917] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]398worker1 # [ 0.002912] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)399lb # [ 0.003155] No NUMA configuration found400client # [ 0.033524] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns401worker1 # [ 0.002913] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]402lb # [ 0.003156] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]403worker2 # [ 0.040421] Unknown kernel command line parameters "regInfo=/nix/store/f92hdajrqvjpsw7r18x533lxyg4b61q5-closure-info/registration", will be passed to user space.404client # [ 0.037967] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1405worker2 # [ 0.040433] random: crng init done406worker1 # [ 0.002914] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]407lb # [ 0.003158] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]408lb # [ 0.005518] Zone ranges:409client # [ 0.040176] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152410worker2 # [ 0.040434] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes411worker1 # [ 0.002915] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]412lb # [ 0.005519] DMA [mem 0x0000000000001000-0x0000000000ffffff]413client # [ 0.040219] kvm-guest: PV spinlocks disabled, single CPU414worker2 # [ 0.042844] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)415worker1 # [ 0.002915] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]416lb # [ 0.005520] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]417lb # [ 0.005522] Normal empty418worker1 # [ 0.002916] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]419worker2 # [ 0.043918] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)420lb # [ 0.005523] Device empty421worker2 # [ 0.043949] Fallback order for Node 0: 0422lb # [ 0.005523] Movable zone start for each node423worker1 # [ 0.002916] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]424lb # [ 0.005524] Early memory node ranges425worker2 # [ 0.043952] Built 1 zonelists, mobility grouping on. Total pages: 393078426worker2 # [ 0.043953] Policy zone: DMA32427lb # [ 0.005524] node 0: [mem 0x0000000000001000-0x000000000009efff]428worker1 # [ 0.002917] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]429worker1 # [ 0.003146] No NUMA configuration found430client # [ 0.040221] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/0v4f337qkyj5dmshnqw4pzwr5c8f8pgy-nixos-system-client-test/init regInfo=/nix/store/f9gdw55mpfid0rm8r2v7250yzv4pkjal-closure-info/registration console=ttyS0,115200n8 console=tty0431lb # [ 0.005525] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]432worker2 # [ 0.046473] mem auto-init: stack:all(zero), heap alloc:on, heap free:off433worker1 # [ 0.003147] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]434worker2 # [ 0.049554] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1435lb # [ 0.005526] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]436worker1 # [ 0.003149] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]437worker2 # [ 0.052940] allocated 3145728 bytes of page_ext438lb # [ 0.005547] On node 0, zone DMA: 1 pages in unavailable ranges439worker1 # [ 0.005532] Zone ranges:440client # [ 0.040311] Unknown kernel command line parameters "regInfo=/nix/store/f9gdw55mpfid0rm8r2v7250yzv4pkjal-closure-info/registration", will be passed to user space.441worker2 # [ 0.063001] ftrace: allocating 48731 entries in 192 pages442client # [ 0.040324] random: crng init done443worker1 # [ 0.005533] DMA [mem 0x0000000000001000-0x0000000000ffffff]444lb # [ 0.005823] On node 0, zone DMA: 97 pages in unavailable ranges445worker2 # [ 0.063003] ftrace: allocated 192 pages with 2 groups446client # [ 0.040324] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes447worker2 # [ 0.063846] Dynamic Preempt: lazy448lb # [ 0.034021] On node 0, zone DMA32: 40 pages in unavailable ranges449worker1 # [ 0.005534] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]450worker1 # [ 0.005536] Normal empty451lb # [ 0.034480] ACPI: PM-Timer IO Port: 0x608452worker2 # [ 0.064023] rcu: Preemptible hierarchical RCU implementation.453worker1 # [ 0.005536] Device empty454client # [ 0.042488] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)455lb # [ 0.034491] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])456worker2 # [ 0.064023] rcu: RCU event tracing is enabled.457worker1 # [ 0.005537] Movable zone start for each node458worker1 # [ 0.005538] Early memory node ranges459client # [ 0.043835] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)460lb # [ 0.034520] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23461worker2 # [ 0.064024] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.462client # [ 0.043865] Fallback order for Node 0: 0463worker1 # [ 0.005538] node 0: [mem 0x0000000000001000-0x000000000009efff]464worker2 # [ 0.064025] Trampoline variant of Tasks RCU enabled.465lb # [ 0.034523] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)466client # [ 0.043868] Built 1 zonelists, mobility grouping on. Total pages: 393078467worker1 # [ 0.005539] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]468worker2 # [ 0.064026] Rude variant of Tasks RCU enabled.469client # [ 0.043868] Policy zone: DMA32470lb # [ 0.034524] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)471worker2 # [ 0.064026] Tracing variant of Tasks RCU enabled.472worker1 # [ 0.005540] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]473client # [ 0.046401] mem auto-init: stack:all(zero), heap alloc:on, heap free:off474lb # [ 0.034526] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)475worker1 # [ 0.005563] On node 0, zone DMA: 1 pages in unavailable ranges476worker2 # [ 0.064027] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.477client # [ 0.049494] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1478lb # [ 0.034527] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)479worker2 # [ 0.064027] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1480worker1 # [ 0.005840] On node 0, zone DMA: 97 pages in unavailable ranges481client # [ 0.052840] allocated 3145728 bytes of page_ext482lb # [ 0.034527] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)483worker1 # [ 0.034330] On node 0, zone DMA32: 40 pages in unavailable ranges484client # [ 0.062613] ftrace: allocating 48731 entries in 192 pages485worker2 # [ 0.064052] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.486worker1 # [ 0.034803] ACPI: PM-Timer IO Port: 0x608487lb # [ 0.034530] ACPI: Using ACPI (MADT) for SMP configuration information488client # [ 0.062616] ftrace: allocated 192 pages with 2 groups489client # [ 0.063456] Dynamic Preempt: lazy490lb # [ 0.034531] ACPI: HPET id: 0x8086a201 base: 0xfed00000491worker1 # [ 0.034814] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])492worker2 # [ 0.064054] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.493lb # [ 0.034535] TSC deadline timer available494client # [ 0.063623] rcu: Preemptible hierarchical RCU implementation.495worker1 # [ 0.034845] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23496lb # [ 0.034539] CPU topo: Max. logical packages: 1497client # [ 0.063623] rcu: RCU event tracing is enabled.498worker2 # [ 0.064055] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.499lb # [ 0.034539] CPU topo: Max. logical dies: 1500worker1 # [ 0.034847] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)501client # [ 0.063624] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.502worker2 # [ 0.068401] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16503lb # [ 0.034540] CPU topo: Max. dies per package: 1504worker1 # [ 0.034849] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)505client # [ 0.063626] Trampoline variant of Tasks RCU enabled.506lb # [ 0.034543] CPU topo: Max. threads per core: 1507worker2 # [ 0.068667] rcu: srcu_init: Setting srcu_struct sizes based on contention.508client # [ 0.063626] Rude variant of Tasks RCU enabled.509worker1 # [ 0.034850] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)510lb # [ 0.034544] CPU topo: Num. cores per package: 1511client # [ 0.063627] Tracing variant of Tasks RCU enabled.512lb # [ 0.034545] CPU topo: Num. threads per package: 1513worker1 # [ 0.034851] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)514worker2 # [ 0.068674] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns515client # [ 0.063627] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.516lb # [ 0.034545] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs517worker1 # [ 0.034852] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)518worker2 # [ 0.068774] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)519client # [ 0.063628] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1520lb # [ 0.034563] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()521worker1 # [ 0.034854] ACPI: Using ACPI (MADT) for SMP configuration information522worker2 # [ 0.072306] Console: colour VGA+ 80x25523worker1 # [ 0.034856] ACPI: HPET id: 0x8086a201 base: 0xfed00000524worker2 # [ 0.072309] printk: legacy console [tty0] enabled525lb # [ 0.034595] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]526client # [ 0.063644] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.527worker1 # [ 0.034859] TSC deadline timer available528worker2 # [ 0.101776] printk: legacy console [ttyS0] enabled529lb # [ 0.034597] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]530worker1 # [ 0.034864] CPU topo: Max. logical packages: 1531client # [ 0.063646] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.532worker2 # [ 0.199245] ACPI: Core revision 20250807533worker1 # [ 0.034864] CPU topo: Max. logical dies: 1534lb # [ 0.034598] [mem 0x60000000-0xafffffff] available for PCI devices535worker1 # [ 0.034865] CPU topo: Max. dies per package: 1536lb # [ 0.034600] Booting paravirtualized kernel on KVM537client # [ 0.063647] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.538worker1 # [ 0.034868] CPU topo: Max. threads per core: 1539worker2 # [ 0.200160] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns540client # [ 0.068324] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16541worker1 # [ 0.034869] CPU topo: Num. cores per package: 1542lb # [ 0.034603] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns543worker1 # [ 0.034869] CPU topo: Num. threads per package: 1544worker2 # [ 0.201620] APIC: Switch to symmetric I/O mode setup545client # [ 0.068607] rcu: srcu_init: Setting srcu_struct sizes based on contention.546lb # [ 0.039082] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1547worker1 # [ 0.034869] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs548worker2 # [ 0.202583] x2apic enabled549client # [ 0.068613] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns550lb # [ 0.041348] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152551worker1 # [ 0.034890] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()552worker2 # [ 0.203314] APIC: Switched APIC routing to: physical x2apic553lb # [ 0.041395] kvm-guest: PV spinlocks disabled, single CPU554worker1 # [ 0.034925] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]555client # [ 0.068714] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)556client # [ 0.072267] Console: colour VGA+ 80x25557worker1 # [ 0.034927] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]558client # [ 0.072270] printk: legacy console [tty0] enabled559worker2 # [ 0.205053] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1560client # [ 0.101780] printk: legacy console [ttyS0] enabled561worker1 # [ 0.034928] [mem 0x60000000-0xafffffff] available for PCI devices562worker1 # [ 0.034930] Booting paravirtualized kernel on KVM563client # [ 0.199308] ACPI: Core revision 20250807564worker2 # [ 0.206007] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns565lb # [ 0.041397] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/dlr5gkwdzlpfiks0wm9k3cgbj9l1v40b-nixos-system-lb-test/init regInfo=/nix/store/493p756s0f7jvdvcriannaqb5380jad0-closure-info/registration console=ttyS0,115200n8 console=tty0566worker1 # [ 0.034933] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns567client # [ 0.200169] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns568worker2 # [ 0.208609] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)569worker1 # [ 0.039395] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1570lb # [ 0.041491] Unknown kernel command line parameters "regInfo=/nix/store/493p756s0f7jvdvcriannaqb5380jad0-closure-info/registration", will be passed to user space.571client # [ 0.201788] APIC: Switch to symmetric I/O mode setup572worker1 # [ 0.041740] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152573lb # [ 0.041503] random: crng init done574worker2 # [ 0.209696] x86/cpu: User Mode Instruction Prevention (UMIP) activated575client # [ 0.202763] x2apic enabled576worker1 # [ 0.041786] kvm-guest: PV spinlocks disabled, single CPU577lb # [ 0.041504] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes578worker2 # [ 0.210745] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127579client # [ 0.203543] APIC: Switched APIC routing to: physical x2apic580lb # [ 0.043724] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)581worker2 # [ 0.211610] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0582lb # [ 0.045121] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)583lb # [ 0.045152] Fallback order for Node 0: 0584client # [ 0.205329] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1585lb # [ 0.045155] Built 1 zonelists, mobility grouping on. Total pages: 393078586worker2 # [ 0.212613] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto587lb # [ 0.045157] Policy zone: DMA32588worker1 # [ 0.041788] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/k4mxfnh5a0g0aavqsj8h0grq5gn5cki6-nixos-system-worker1-test/init regInfo=/nix/store/j5nrz0mrj1ngq2p5dwyv942lpc0ifh7p-closure-info/registration console=ttyS0,115200n8 console=tty0589lb # [ 0.047752] mem auto-init: stack:all(zero), heap alloc:on, heap free:off590client # [ 0.206339] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns591worker2 # [ 0.213609] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl592lb # [ 0.051062] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1593worker1 # [ 0.041882] Unknown kernel command line parameters "regInfo=/nix/store/j5nrz0mrj1ngq2p5dwyv942lpc0ifh7p-closure-info/registration", will be passed to user space.594worker2 # [ 0.214610] Transient Scheduler Attacks: Vulnerable: No microcode595worker1 # [ 0.041894] random: crng init done596lb # [ 0.054552] allocated 3145728 bytes of page_ext597client # [ 0.207974] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)598lb # [ 0.064593] ftrace: allocating 48731 entries in 192 pages599worker2 # [ 0.215609] Spectre V2 : Mitigation: Enhanced / Automatic IBRS600worker1 # [ 0.041895] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes601lb # [ 0.064595] ftrace: allocated 192 pages with 2 groups602client # [ 0.209057] x86/cpu: User Mode Instruction Prevention (UMIP) activated603lb # [ 0.065475] Dynamic Preempt: lazy604worker2 # [ 0.216609] Speculative Return Stack Overflow: Mitigation: Safe RET605worker1 # [ 0.044207] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)606lb # [ 0.065585] rcu: Preemptible hierarchical RCU implementation.607client # [ 0.210161] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127608worker1 # [ 0.045611] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)609lb # [ 0.065586] rcu: RCU event tracing is enabled.610worker1 # [ 0.045643] Fallback order for Node 0: 0611client # [ 0.211664] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0612lb # [ 0.065587] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.613worker1 # [ 0.045646] Built 1 zonelists, mobility grouping on. Total pages: 393078614lb # [ 0.065588] Trampoline variant of Tasks RCU enabled.615worker1 # [ 0.045647] Policy zone: DMA32616lb # [ 0.065589] Rude variant of Tasks RCU enabled.617client # [ 0.211975] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto618worker1 # [ 0.048273] mem auto-init: stack:all(zero), heap alloc:on, heap free:off619lb # [ 0.065589] Tracing variant of Tasks RCU enabled.620worker2 # [ 0.217609] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization621worker1 # [ 0.051688] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1622lb # [ 0.065590] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.623worker2 # [ 0.219616] Spectre V2 : Enabling IBPB for BPF624worker1 # [ 0.055174] allocated 3145728 bytes of page_ext625client # [ 0.213971] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl626lb # [ 0.065590] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1627worker1 # [ 0.065127] ftrace: allocating 48731 entries in 192 pages628client # [ 0.214971] Transient Scheduler Attacks: Vulnerable: No microcode629worker1 # [ 0.065129] ftrace: allocated 192 pages with 2 groups630worker2 # [ 0.220321] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier631lb # [ 0.065611] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.632worker1 # [ 0.066022] Dynamic Preempt: lazy633client # [ 0.216662] Spectre V2 : Mitigation: Enhanced / Automatic IBRS634worker2 # [ 0.221609] active return thunk: srso_alias_return_thunk635worker1 # [ 0.066182] rcu: Preemptible hierarchical RCU implementation.636lb # [ 0.065613] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.637worker1 # [ 0.066183] rcu: RCU event tracing is enabled.638client # [ 0.216971] Speculative Return Stack Overflow: Mitigation: Safe RET639worker2 # [ 0.222631] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'640worker1 # [ 0.066183] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.641lb # [ 0.065614] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.642worker1 # [ 0.066185] Trampoline variant of Tasks RCU enabled.643worker2 # [ 0.223609] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'644lb # [ 0.070311] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16645worker1 # [ 0.066185] Rude variant of Tasks RCU enabled.646lb # [ 0.070579] rcu: srcu_init: Setting srcu_struct sizes based on contention.647worker2 # [ 0.224609] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'648worker1 # [ 0.066186] Tracing variant of Tasks RCU enabled.649worker2 # [ 0.225609] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'650worker1 # [ 0.066186] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.651lb # [ 0.070587] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns652client # [ 0.217971] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization653worker1 # [ 0.066187] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1654worker2 # [ 0.226609] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'655lb # [ 0.070696] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)656client # [ 0.219978] Spectre V2 : Enabling IBPB for BPF657lb # [ 0.074249] Console: colour VGA+ 80x25658worker1 # [ 0.066213] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.659worker2 # [ 0.227609] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'660lb # [ 0.074252] printk: legacy console [tty0] enabled661client # [ 0.220972] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier662lb # [ 0.103594] printk: legacy console [ttyS0] enabled663worker1 # [ 0.066215] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.664worker2 # [ 0.228609] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'665lb # [ 0.198612] ACPI: Core revision 20250807666client # [ 0.221971] active return thunk: srso_alias_return_thunk667worker1 # [ 0.066216] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.668worker2 # [ 0.229609] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256669worker1 # [ 0.070991] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16670client # [ 0.222992] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'671worker2 # [ 0.230609] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64672lb # [ 0.199543] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns673worker1 # [ 0.071277] rcu: srcu_init: Setting srcu_struct sizes based on contention.674client # [ 0.223971] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'675worker2 # [ 0.231609] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512676lb # [ 0.201027] APIC: Switch to symmetric I/O mode setup677worker1 # [ 0.071285] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns678client # [ 0.224971] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'679lb # [ 0.201988] x2apic enabled680worker2 # [ 0.232609] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024681client # [ 0.225971] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'682worker2 # [ 0.233517] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8683worker1 # [ 0.071392] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)684lb # [ 0.202756] APIC: Switched APIC routing to: physical x2apic685worker1 # [ 0.074941] Console: colour VGA+ 80x25686client # [ 0.226971] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'687worker1 # [ 0.074944] printk: legacy console [tty0] enabled688worker2 # [ 0.234304] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.689worker1 # [ 0.104583] printk: legacy console [ttyS0] enabled690client # [ 0.227971] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'691lb # [ 0.204538] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1692worker1 # [ 0.200617] ACPI: Core revision 20250807693client # [ 0.228971] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'694lb # [ 0.205505] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns695worker1 # [ 0.201575] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns696client # [ 0.229971] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256697worker1 # [ 0.203119] APIC: Switch to symmetric I/O mode setup698lb # [ 0.207180] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)699client # [ 0.231703] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64700worker1 # [ 0.204081] x2apic enabled701client # [ 0.232673] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512702lb # [ 0.208265] x86/cpu: User Mode Instruction Prevention (UMIP) activated703worker1 # [ 0.204871] APIC: Switched APIC routing to: physical x2apic704client # [ 0.232971] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024705lb # [ 0.209329] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127706client # [ 0.233971] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8707lb # [ 0.210178] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0708worker1 # [ 0.206700] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1709client # [ 0.234971] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.710lb # [ 0.211181] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto711worker1 # [ 0.207690] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns712lb # [ 0.213177] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl713worker1 # [ 0.209369] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)714lb # [ 0.214177] Transient Scheduler Attacks: Vulnerable: No microcode715worker1 # [ 0.210455] x86/cpu: User Mode Instruction Prevention (UMIP) activated716lb # [ 0.215932] Spectre V2 : Mitigation: Enhanced / Automatic IBRS717worker1 # [ 0.211491] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127718lb # [ 0.216907] Speculative Return Stack Overflow: Mitigation: Safe RET719worker1 # [ 0.213084] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0720worker1 # [ 0.214186] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto721lb # [ 0.217942] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization722worker1 # [ 0.215366] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl723lb # [ 0.219185] Spectre V2 : Enabling IBPB for BPF724worker1 # [ 0.217366] Transient Scheduler Attacks: Vulnerable: No microcode725worker1 # [ 0.218366] Spectre V2 : Mitigation: Enhanced / Automatic IBRS726lb # [ 0.220178] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier727worker1 # [ 0.219366] Speculative Return Stack Overflow: Mitigation: Safe RET728lb # [ 0.221178] active return thunk: srso_alias_return_thunk729lb # [ 0.222862] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'730lb # [ 0.224177] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'731lb # [ 0.225177] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'732worker1 # [ 0.220366] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization733lb # [ 0.226177] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'734worker1 # [ 0.222374] Spectre V2 : Enabling IBPB for BPF735lb # [ 0.227177] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'736worker1 # [ 0.223121] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier737lb # [ 0.228177] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'738worker1 # [ 0.224366] active return thunk: srso_alias_return_thunk739lb # [ 0.229177] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'740worker1 # [ 0.225388] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'741lb # [ 0.230177] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256742worker1 # [ 0.226366] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'743lb # [ 0.231177] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64744worker1 # [ 0.227366] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'745lb # [ 0.232177] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512746worker1 # [ 0.228366] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'747lb # [ 0.233177] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024748worker1 # [ 0.229366] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'749lb # [ 0.234177] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8750worker1 # [ 0.230366] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'751worker2 # [ 0.268465] Freeing SMP alternatives memory: 44K752worker2 # [ 0.269167] pid_max: default: 32768 minimum: 301753lb # [ 0.235177] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.754worker1 # [ 0.231366] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'755worker2 # [ 0.269716] LSM: initializing lsm=capability,landlock,yama,bpf,ima756worker1 # [ 0.232367] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256757worker2 # [ 0.270712] landlock: Up and running.758worker1 # [ 0.234116] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64759worker2 # [ 0.271610] Yama: becoming mindful.760worker2 # [ 0.272706] LSM support for eBPF active761worker1 # [ 0.235122] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512762worker1 # [ 0.236123] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024763worker2 # [ 0.273490] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)764worker1 # [ 0.237131] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8765worker2 # [ 0.274646] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)766worker1 # [ 0.238137] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.767client # [ 0.270135] Freeing SMP alternatives memory: 44K768worker2 # [ 0.277001] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)769client # [ 0.270921] pid_max: default: 32768 minimum: 301770client # [ 0.271607] LSM: initializing lsm=capability,landlock,yama,bpf,ima771worker2 # [ 0.278168] Performance Events: Fam17h+ core perfctr, AMD PMU driver.772client # [ 0.272849] landlock: Up and running.773worker2 # [ 0.278619] ... version: 2774client # [ 0.273414] Yama: becoming mindful.775worker2 # [ 0.279346] ... bit width: 48776client # [ 0.274195] LSM support for eBPF active777worker2 # [ 0.279612] ... generic counters: 6778client # [ 0.275091] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)779worker2 # [ 0.280327] ... generic bitmap: 000000000000003f780worker2 # [ 0.280611] ... fixed-purpose counters: 0781client # [ 0.276011] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)782worker2 # [ 0.281347] ... fixed-purpose bitmap: 0000000000000000783worker2 # [ 0.281611] ... value mask: 0000ffffffffffff784worker2 # [ 0.282558] ... max period: 00007fffffffffff785client # [ 0.278821] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)786worker2 # [ 0.283289] ... global_ctrl mask: 000000000000003f787worker2 # [ 0.283727] signal: max sigframe size: 3376788client # [ 0.279488] Performance Events: Fam17h+ core perfctr, AMD PMU driver.789worker2 # [ 0.284479] rcu: Hierarchical SRCU implementation.790client # [ 0.279976] ... version: 2791client # [ 0.280693] ... bit width: 48792worker2 # [ 0.285205] rcu: Max phase no-delay instances is 400.793client # [ 0.280973] ... generic counters: 6794client # [ 0.281717] ... generic bitmap: 000000000000003f795client # [ 0.282019] ... fixed-purpose counters: 0796client # [ 0.282742] ... fixed-purpose bitmap: 0000000000000000797client # [ 0.282973] ... value mask: 0000ffffffffffff798client # [ 0.283909] ... max period: 00007fffffffffff799worker2 # [ 0.290181] smp: Bringing up secondary CPUs ...800worker2 # [ 0.290625] smp: Brought up 1 node, 1 CPU801client # [ 0.284681] ... global_ctrl mask: 000000000000003f802client # [ 0.285095] signal: max sigframe size: 3376803worker2 # [ 0.291286] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)804client # [ 0.285901] rcu: Hierarchical SRCU implementation.805client # [ 0.286608] rcu: Max phase no-delay instances is 400.806worker2 # [ 0.291788] Memory: 1454548K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)807worker2 # [ 0.292839] devtmpfs: initialized808worker2 # [ 0.293665] x86/mm: Memory block size: 128MB809lb # [ 0.270178] Freeing SMP alternatives memory: 44K810worker2 # [ 0.295501] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)811client # [ 0.291486] smp: Bringing up secondary CPUs ...812lb # [ 0.270966] pid_max: default: 32768 minimum: 301813client # [ 0.291987] smp: Brought up 1 node, 1 CPU814worker2 # [ 0.296540] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).815lb # [ 0.271277] LSM: initializing lsm=capability,landlock,yama,bpf,ima816client # [ 0.292670] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)817worker2 # [ 0.297690] pinctrl core: initialized pinctrl subsystem818lb # [ 0.272292] landlock: Up and running.819lb # [ 0.273178] Yama: becoming mindful.820worker2 # [ 0.298826] PM: RTC time: 09:26:44, date: 2026-09-21821lb # [ 0.274365] LSM support for eBPF active822client # [ 0.293239] Memory: 1454552K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)823lb # [ 0.275159] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)824client # [ 0.294218] devtmpfs: initialized825client # [ 0.295033] x86/mm: Memory block size: 128MB826lb # [ 0.276218] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)827worker2 # [ 0.302439] NET: Registered PF_NETLINK/PF_ROUTE protocol family828worker1 # [ 0.273691] Freeing SMP alternatives memory: 44K829client # [ 0.296796] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)830worker1 # [ 0.274370] pid_max: default: 32768 minimum: 301831worker2 # [ 0.303924] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations832lb # [ 0.278733] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)833worker1 # [ 0.275382] LSM: initializing lsm=capability,landlock,yama,bpf,ima834client # [ 0.297870] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).835worker2 # [ 0.304633] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations836worker1 # [ 0.276450] landlock: Up and running.837lb # [ 0.279736] Performance Events: Fam17h+ core perfctr, AMD PMU driver.838client # [ 0.299071] pinctrl core: initialized pinctrl subsystem839worker1 # [ 0.277056] Yama: becoming mindful.840lb # [ 0.280182] ... version: 2841worker2 # [ 0.305901] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations842worker1 # [ 0.277595] LSM support for eBPF active843client # [ 0.300164] PM: RTC time: 09:26:44, date: 2026-09-21844lb # [ 0.280894] ... bit width: 48845worker2 # [ 0.306622] audit: initializing netlink subsys (disabled)846lb # [ 0.281180] ... generic counters: 6847worker1 # [ 0.278532] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)848lb # [ 0.281885] ... generic bitmap: 000000000000003f849worker2 # [ 0.307727] thermal_sys: Registered thermal governor 'fair_share'850lb # [ 0.282190] ... fixed-purpose counters: 0851worker1 # [ 0.279407] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)852worker2 # [ 0.307729] thermal_sys: Registered thermal governor 'bang_bang'853lb # [ 0.282956] ... fixed-purpose bitmap: 0000000000000000854worker2 # [ 0.308613] thermal_sys: Registered thermal governor 'step_wise'855client # [ 0.303773] NET: Registered PF_NETLINK/PF_ROUTE protocol family856lb # [ 0.283180] ... value mask: 0000ffffffffffff857worker2 # [ 0.309551] thermal_sys: Registered thermal governor 'user_space'858worker1 # [ 0.282218] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)859lb # [ 0.284065] ... max period: 00007fffffffffff860client # [ 0.305223] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations861lb # [ 0.284823] ... global_ctrl mask: 000000000000003f862worker2 # [ 0.310316] audit: type=2000 audit(1789982805.034:1): state=initialized audit_enabled=0 res=1863worker1 # [ 0.282952] Performance Events: Fam17h+ core perfctr, AMD PMU driver.864client # [ 0.306001] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations865lb # [ 0.285297] signal: max sigframe size: 3376866worker2 # [ 0.311615] thermal_sys: Registered thermal governor 'power_allocator'867worker1 # [ 0.283372] ... version: 2868lb # [ 0.286055] rcu: Hierarchical SRCU implementation.869worker2 # [ 0.311638] cpuidle: using governor menu870worker1 # [ 0.284071] ... bit width: 48871client # [ 0.307256] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations872lb # [ 0.286767] rcu: Max phase no-delay instances is 400.873worker1 # [ 0.284368] ... generic counters: 6874client # [ 0.307983] audit: initializing netlink subsys (disabled)875worker2 # [ 0.313618] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5876worker1 # [ 0.285079] ... generic bitmap: 000000000000003f877worker1 # [ 0.285380] ... fixed-purpose counters: 0878client # [ 0.308976] audit: type=2000 audit(1789982805.042:1): state=initialized audit_enabled=0 res=1879worker2 # [ 0.314848] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]880worker1 # [ 0.286092] ... fixed-purpose bitmap: 0000000000000000881client # [ 0.310126] thermal_sys: Registered thermal governor 'fair_share'882worker2 # [ 0.315615] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry883worker1 # [ 0.286369] ... value mask: 0000ffffffffffff884client # [ 0.310128] thermal_sys: Registered thermal governor 'bang_bang'885worker2 # [ 0.316624] PCI: Using configuration type 1 for base access886worker1 # [ 0.287274] ... max period: 00007fffffffffff887client # [ 0.310974] thermal_sys: Registered thermal governor 'step_wise'888lb # [ 0.291758] smp: Bringing up secondary CPUs ...889worker1 # [ 0.288051] ... global_ctrl mask: 000000000000003f890worker2 # [ 0.317697] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.891lb # [ 0.292193] smp: Brought up 1 node, 1 CPU892client # [ 0.311915] thermal_sys: Registered thermal governor 'user_space'893worker1 # [ 0.288483] signal: max sigframe size: 3376894lb # [ 0.292853] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)895client # [ 0.312691] thermal_sys: Registered thermal governor 'power_allocator'896worker1 # [ 0.289284] rcu: Hierarchical SRCU implementation.897client # [ 0.312996] cpuidle: using governor menu898worker1 # [ 0.289952] rcu: Max phase no-delay instances is 400.899client # [ 0.315198] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5900lb # [ 0.293376] Memory: 1454628K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)901lb # [ 0.294433] devtmpfs: initialized902client # [ 0.316216] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]903worker2 # [ 0.322740] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages904lb # [ 0.295214] x86/mm: Memory block size: 128MB905client # [ 0.316976] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry906worker2 # [ 0.323613] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page907client # [ 0.317985] PCI: Using configuration type 1 for base access908lb # [ 0.296981] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)909worker1 # [ 0.294990] smp: Bringing up secondary CPUs ...910worker1 # [ 0.295383] smp: Brought up 1 node, 1 CPU911client # [ 0.319106] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.912lb # [ 0.298061] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).913worker1 # [ 0.296030] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)914lb # [ 0.299260] pinctrl core: initialized pinctrl subsystem915lb # [ 0.300376] PM: RTC time: 09:26:44, date: 2026-09-21916worker2 # [ 0.328613] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages917worker1 # [ 0.296581] Memory: 1454592K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)918worker2 # [ 0.329611] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page919worker1 # [ 0.297646] devtmpfs: initialized920worker1 # [ 0.298448] x86/mm: Memory block size: 128MB921lb # [ 0.304056] NET: Registered PF_NETLINK/PF_ROUTE protocol family922client # [ 0.325990] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages923worker1 # [ 0.300377] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)924lb # [ 0.305458] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations925client # [ 0.326974] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page926worker1 # [ 0.301400] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).927lb # [ 0.306204] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations928worker1 # [ 0.302460] pinctrl core: initialized pinctrl subsystem929lb # [ 0.307487] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations930worker1 # [ 0.303630] PM: RTC time: 09:26:44, date: 2026-09-21931lb # [ 0.308189] audit: initializing netlink subsys (disabled)932lb # [ 0.309353] thermal_sys: Registered thermal governor 'fair_share'933client # [ 0.331975] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages934worker2 # [ 0.339974] ACPI: Added _OSI(Module Device)935lb # [ 0.309355] thermal_sys: Registered thermal governor 'bang_bang'936client # [ 0.332974] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page937worker2 # [ 0.340613] ACPI: Added _OSI(Processor Device)938lb # [ 0.310181] thermal_sys: Registered thermal governor 'step_wise'939worker1 # [ 0.307296] NET: Registered PF_NETLINK/PF_ROUTE protocol family940worker2 # [ 0.341303] ACPI: Added _OSI(Processor Aggregator Device)941lb # [ 0.311092] thermal_sys: Registered thermal governor 'user_space'942worker1 # [ 0.308600] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations943lb # [ 0.311882] audit: type=2000 audit(1789982805.071:1): state=initialized audit_enabled=0 res=1944worker1 # [ 0.309396] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations945lb # [ 0.313183] thermal_sys: Registered thermal governor 'power_allocator'946lb # [ 0.313208] cpuidle: using governor menu947worker1 # [ 0.310664] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations948worker1 # [ 0.311380] audit: initializing netlink subsys (disabled)949lb # [ 0.315419] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5950worker1 # [ 0.312524] thermal_sys: Registered thermal governor 'fair_share'951lb # [ 0.316398] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]952worker1 # [ 0.312527] thermal_sys: Registered thermal governor 'bang_bang'953lb # [ 0.317183] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry954worker1 # [ 0.313370] thermal_sys: Registered thermal governor 'step_wise'955lb # [ 0.318192] PCI: Using configuration type 1 for base access956worker2 # [ 0.349849] ACPI: 1 ACPI AML tables successfully acquired and loaded957worker1 # [ 0.314352] thermal_sys: Registered thermal governor 'user_space'958client # [ 0.344240] ACPI: Added _OSI(Module Device)959lb # [ 0.319280] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.960worker1 # [ 0.315117] audit: type=2000 audit(1789982805.077:1): state=initialized audit_enabled=0 res=1961client # [ 0.344949] ACPI: Added _OSI(Processor Device)962worker1 # [ 0.316373] thermal_sys: Registered thermal governor 'power_allocator'963client # [ 0.345533] ACPI: Added _OSI(Processor Aggregator Device)964worker1 # [ 0.316386] cpuidle: using governor menu965worker2 # [ 0.354012] ACPI: Interpreter enabled966worker1 # [ 0.318647] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5967worker2 # [ 0.354627] ACPI: PM: (supports S0 S3 S4 S5)968worker2 # [ 0.355335] ACPI: Using IOAPIC for interrupt routing969worker1 # [ 0.319625] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]970worker1 # [ 0.320372] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry971lb # [ 0.326473] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages972worker1 # [ 0.321381] PCI: Using configuration type 1 for base access973client # [ 0.351407] ACPI: 1 ACPI AML tables successfully acquired and loaded974worker2 # [ 0.357673] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug975lb # [ 0.327181] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page976worker2 # [ 0.358612] PCI: Using E820 reservations for host bridge windows977worker1 # [ 0.322488] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.978worker2 # [ 0.361677] ACPI: Enabled 2 GPEs in block 00 to 3F979client # [ 0.355399] ACPI: Interpreter enabled980client # [ 0.355989] ACPI: PM: (supports S0 S3 S4 S5)981lb # [ 0.332181] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages982client # [ 0.356741] ACPI: Using IOAPIC for interrupt routing983lb # [ 0.333180] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page984worker1 # [ 0.329666] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages985client # [ 0.359031] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug986worker1 # [ 0.330371] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page987client # [ 0.359974] PCI: Using E820 reservations for host bridge windows988worker2 # [ 0.369472] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])989client # [ 0.363112] ACPI: Enabled 2 GPEs in block 00 to 3F990worker2 # [ 0.370411] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]991worker1 # [ 0.335370] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages992worker2 # [ 0.371702] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]993lb # [ 0.341840] ACPI: Added _OSI(Module Device)994lb # [ 0.342181] ACPI: Added _OSI(Processor Device)995worker1 # [ 0.336370] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page996worker2 # [ 0.372738] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]997lb # [ 0.342896] ACPI: Added _OSI(Processor Aggregator Device)998worker2 # [ 0.373963] PCI host bridge to bus 0000:00999worker2 # [ 0.374616] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1000worker2 # [ 0.375612] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1001worker2 # [ 0.376611] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1002client # [ 0.370568] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1003worker2 # [ 0.377612] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1004client # [ 0.370980] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1005worker2 # [ 0.378612] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1006client # [ 0.372053] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1007lb # [ 0.349768] ACPI: 1 ACPI AML tables successfully acquired and loaded1008worker2 # [ 0.379616] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1009client # [ 0.373096] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1010worker1 # [ 0.345843] ACPI: Added _OSI(Module Device)1011worker2 # [ 0.380612] pci_bus 0000:00: root bus resource [bus 00-ff]1012worker1 # [ 0.346370] ACPI: Added _OSI(Processor Device)1013client # [ 0.374338] PCI host bridge to bus 0000:001014worker2 # [ 0.381566] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1015worker1 # [ 0.347136] ACPI: Added _OSI(Processor Aggregator Device)1016client # [ 0.374978] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1017lb # [ 0.353123] ACPI: Interpreter enabled1018client # [ 0.375975] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1019worker2 # [ 0.383156] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1020lb # [ 0.353648] ACPI: PM: (supports S0 S3 S4 S5)1021client # [ 0.376974] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1022lb # [ 0.354180] ACPI: Using IOAPIC for interrupt routing1023client # [ 0.377974] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1024lb # [ 0.355076] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1025client # [ 0.378974] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1026client # [ 0.379974] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1027client # [ 0.380974] pci_bus 0000:00: root bus resource [bus 00-ff]1028worker1 # [ 0.353979] ACPI: 1 ACPI AML tables successfully acquired and loaded1029client # [ 0.382026] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1030lb # [ 0.360180] PCI: Using E820 reservations for host bridge windows1031lb # [ 0.361251] ACPI: Enabled 2 GPEs in block 00 to 3F1032worker2 # [ 0.385694] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1033client # [ 0.383431] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1034worker2 # [ 0.386624] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1035worker1 # [ 0.357794] ACPI: Interpreter enabled1036worker2 # [ 0.387595] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1037worker1 # [ 0.358385] ACPI: PM: (supports S0 S3 S4 S5)1038worker1 # [ 0.359086] ACPI: Using IOAPIC for interrupt routing1039worker2 # [ 0.388628] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1040worker2 # [ 0.390310] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1041worker1 # [ 0.361423] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1042worker1 # [ 0.362370] PCI: Using E820 reservations for host bridge windows1043lb # [ 0.369920] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1044worker2 # [ 0.392642] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1045client # [ 0.386095] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1046worker2 # [ 0.393501] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1047worker1 # [ 0.365528] ACPI: Enabled 2 GPEs in block 00 to 3F1048lb # [ 0.370668] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1049client # [ 0.386986] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1050worker2 # [ 0.394369] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1051worker2 # [ 0.394617] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1052lb # [ 0.371257] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1053client # [ 0.387924] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1054lb # [ 0.372301] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1055client # [ 0.389146] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1056worker2 # [ 0.396154] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1057lb # [ 0.373532] PCI host bridge to bus 0000:001058lb # [ 0.374185] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1059client # [ 0.390881] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1060lb # [ 0.375181] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1061worker2 # [ 0.398645] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1062lb # [ 0.376180] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1063worker2 # [ 0.399558] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1064worker2 # [ 0.400392] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1065lb # [ 0.377179] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1066worker1 # [ 0.373685] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1067client # [ 0.394008] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1068lb # [ 0.378180] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1069worker1 # [ 0.374376] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1070client # [ 0.394858] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1071worker2 # [ 0.402004] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1072lb # [ 0.379180] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1073client # [ 0.395683] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1074worker1 # [ 0.375449] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1075lb # [ 0.380180] pci_bus 0000:00: root bus resource [bus 00-ff]1076worker1 # [ 0.376510] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1077client # [ 0.395979] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1078lb # [ 0.381204] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1079worker1 # [ 0.377737] PCI host bridge to bus 0000:001080client # [ 0.397517] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1081worker1 # [ 0.378376] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1082worker2 # [ 0.404619] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1083lb # [ 0.382617] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1084worker2 # [ 0.406213] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1085worker1 # [ 0.379369] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1086worker2 # [ 0.406633] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1087worker1 # [ 0.380369] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1088worker1 # [ 0.381369] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1089client # [ 0.399974] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1090client # [ 0.400902] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1091client # [ 0.401653] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1092worker1 # [ 0.382369] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1093worker1 # [ 0.383370] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1094worker2 # [ 0.408705] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1095worker1 # [ 0.384370] pci_bus 0000:00: root bus resource [bus 00-ff]1096client # [ 0.403238] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1097worker1 # [ 0.385406] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1098worker2 # [ 0.410640] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1099lb # [ 0.385246] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1100lb # [ 0.386192] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1101worker1 # [ 0.386830] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1102client # [ 0.405983] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1103worker2 # [ 0.411612] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1104lb # [ 0.387147] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1105client # [ 0.406817] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1106worker2 # [ 0.412634] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1107client # [ 0.407698] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1108lb # [ 0.388545] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1109worker2 # [ 0.414187] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1110client # [ 0.408548] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1111lb # [ 0.389898] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1112worker2 # [ 0.415641] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1113worker2 # [ 0.416546] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1114lb # [ 0.392188] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1115worker2 # [ 0.417406] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1116lb # [ 0.393030] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1117client # [ 0.410994] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1118worker1 # [ 0.389505] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1119lb # [ 0.393954] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1120client # [ 0.412623] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1121worker2 # [ 0.419136] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1122worker1 # [ 0.390383] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1123client # [ 0.412996] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1124lb # [ 0.394981] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1125worker1 # [ 0.391334] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1126client # [ 0.414531] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1127worker2 # [ 0.420642] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1128lb # [ 0.396722] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1129worker1 # [ 0.392541] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1130worker2 # [ 0.421445] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1131worker2 # [ 0.422379] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1132lb # [ 0.398187] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1133client # [ 0.416722] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1134worker1 # [ 0.394118] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1135client # [ 0.416980] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1136lb # [ 0.398994] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1137worker2 # [ 0.423170] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1138client # [ 0.417927] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1139lb # [ 0.399924] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1140worker1 # [ 0.397392] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1141lb # [ 0.400757] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1142worker1 # [ 0.398276] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1143client # [ 0.419367] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1144worker2 # [ 0.425595] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1145worker1 # [ 0.399184] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1146worker2 # [ 0.426619] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1147worker1 # [ 0.400251] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1148worker2 # [ 0.427594] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1149client # [ 0.421816] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1150worker2 # [ 0.428487] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1151lb # [ 0.403364] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1152client # [ 0.422681] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1153worker1 # [ 0.401679] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1154lb # [ 0.404186] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1155client # [ 0.422996] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1156worker2 # [ 0.429916] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1157lb # [ 0.405149] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1158worker1 # [ 0.403392] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1159client # [ 0.424568] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1160worker2 # [ 0.431651] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1161lb # [ 0.406560] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1162worker1 # [ 0.404241] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1163worker2 # [ 0.432587] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1164worker1 # [ 0.405137] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1165lb # [ 0.408213] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1166worker2 # [ 0.434013] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1167worker1 # [ 0.406878] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1168lb # [ 0.409128] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1169client # [ 0.427010] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1170lb # [ 0.409906] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1171client # [ 0.428586] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1172worker2 # [ 0.435651] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1173client # [ 0.428995] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1174worker2 # [ 0.436488] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1175client # [ 0.429979] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1176lb # [ 0.410752] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1177worker1 # [ 0.409390] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1178worker2 # [ 0.437323] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1179worker1 # [ 0.410238] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1180client # [ 0.431472] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1181lb # [ 0.413188] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1182worker1 # [ 0.411140] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1183worker2 # [ 0.438179] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1184lb # [ 0.413982] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1185client # [ 0.432988] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1186lb # [ 0.414940] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1187worker1 # [ 0.412757] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1188client # [ 0.433932] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1189worker2 # [ 0.439643] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1190lb # [ 0.415806] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1191worker2 # [ 0.440510] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1192worker1 # [ 0.415270] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1193client # [ 0.435382] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1194worker2 # [ 0.441338] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1195worker1 # [ 0.416058] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1196lb # [ 0.418189] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1197worker2 # [ 0.443103] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1198worker1 # [ 0.416391] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1199client # [ 0.437736] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1200lb # [ 0.419036] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1201worker2 # [ 0.444302] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1202client # [ 0.437981] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1203lb # [ 0.419943] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1204worker1 # [ 0.418002] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1205client # [ 0.438922] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1206worker2 # [ 0.444823] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1207lb # [ 0.420761] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1208worker1 # [ 0.419388] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1209worker2 # [ 0.446224] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1210client # [ 0.440379] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1211worker1 # [ 0.420289] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1212lb # [ 0.422188] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1213worker1 # [ 0.421148] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1214lb # [ 0.423061] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1215client # [ 0.442001] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1216lb # [ 0.423930] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1217worker1 # [ 0.422806] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1218lb # [ 0.424186] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1219client # [ 0.442836] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1220client # [ 0.444034] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1221lb # [ 0.425733] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1222worker1 # [ 0.425376] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1223worker1 # [ 0.426256] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1224client # [ 0.445722] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1225lb # [ 0.426982] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1226worker1 # [ 0.427107] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1227lb # [ 0.427943] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1228client # [ 0.447769] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1229worker1 # [ 0.427963] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1230client # [ 0.448895] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1231lb # [ 0.428751] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1232worker2 # [ 0.446839] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1233client # [ 0.450646] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1234worker1 # [ 0.429391] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1235lb # [ 0.430202] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1236worker2 # [ 0.449401] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1237worker1 # [ 0.430262] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1238lb # [ 0.431041] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1239worker1 # [ 0.431092] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1240lb # [ 0.431896] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1241worker2 # [ 0.449875] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1242worker1 # [ 0.431375] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1243worker2 # [ 0.451231] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1244lb # [ 0.433196] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1245worker1 # [ 0.432920] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1246worker2 # [ 0.451828] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1247lb # [ 0.435187] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1248worker2 # [ 0.452897] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1249worker1 # [ 0.434384] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1250lb # [ 0.436049] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1251worker1 # [ 0.435337] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1252lb # [ 0.436901] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1253worker2 # [ 0.453858] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1254worker1 # [ 0.436720] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1255lb # [ 0.437738] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1256worker2 # [ 0.455582] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1257worker2 # [ 0.456239] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1258lb # [ 0.439196] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1259client # [ 0.451183] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1260worker2 # [ 0.456923] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1261worker1 # [ 0.438389] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1262worker1 # [ 0.439375] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1263lb # [ 0.440264] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1264worker2 # [ 0.458335] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1265worker1 # [ 0.440390] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1266lb # [ 0.441696] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1267worker1 # [ 0.441938] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1268client # [ 0.454694] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1269worker2 # [ 0.461666] ACPI: PCI: Interrupt link LNKA configured for IRQ 101270client # [ 0.455238] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1271worker2 # [ 0.462714] ACPI: PCI: Interrupt link LNKB configured for IRQ 101272worker1 # [ 0.443389] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1273worker1 # [ 0.444318] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1274worker2 # [ 0.463889] ACPI: PCI: Interrupt link LNKC configured for IRQ 111275client # [ 0.456982] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1276worker1 # [ 0.445140] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1277worker2 # [ 0.464708] ACPI: PCI: Interrupt link LNKD configured for IRQ 111278worker2 # [ 0.465716] ACPI: PCI: Interrupt link LNKE configured for IRQ 101279client # [ 0.458221] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1280worker1 # [ 0.446776] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1281worker2 # [ 0.466717] ACPI: PCI: Interrupt link LNKF configured for IRQ 101282client # [ 0.459268] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1283worker2 # [ 0.467715] ACPI: PCI: Interrupt link LNKG configured for IRQ 111284worker1 # [ 0.448047] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1285worker2 # [ 0.468714] ACPI: PCI: Interrupt link LNKH configured for IRQ 111286client # [ 0.460241] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1287worker1 # [ 0.448583] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1288worker2 # [ 0.469648] ACPI: PCI: Interrupt link GSIA configured for IRQ 161289worker2 # [ 0.470627] ACPI: PCI: Interrupt link GSIB configured for IRQ 171290lb # [ 0.442382] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1291worker1 # [ 0.449996] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1292client # [ 0.462049] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1293worker2 # [ 0.471601] ACPI: PCI: Interrupt link GSIC configured for IRQ 181294lb # [ 0.444718] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1295client # [ 0.462927] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1296worker2 # [ 0.472379] ACPI: PCI: Interrupt link GSID configured for IRQ 191297worker2 # [ 0.472624] ACPI: PCI: Interrupt link GSIE configured for IRQ 201298lb # [ 0.445441] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1299client # [ 0.464042] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1300worker2 # [ 0.473623] ACPI: PCI: Interrupt link GSIF configured for IRQ 211301lb # [ 0.446779] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1302worker2 # [ 0.474583] ACPI: PCI: Interrupt link GSIG configured for IRQ 221303client # [ 0.466095] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1304worker2 # [ 0.475354] ACPI: PCI: Interrupt link GSIH configured for IRQ 231305lb # [ 0.447444] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1306worker2 # [ 0.476686] iommu: Default domain type: Translated1307lb # [ 0.448467] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1308worker2 # [ 0.477503] iommu: DMA domain TLB invalidation policy: lazy mode1309worker2 # [ 0.478637] ACPI: bus type USB registered1310client # [ 0.470042] ACPI: PCI: Interrupt link LNKA configured for IRQ 101311lb # [ 0.449425] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1312worker2 # [ 0.479339] usbcore: registered new interface driver usbfs1313client # [ 0.471088] ACPI: PCI: Interrupt link LNKB configured for IRQ 101314worker2 # [ 0.479627] usbcore: registered new interface driver hub1315lb # [ 0.450978] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1316client # [ 0.472072] ACPI: PCI: Interrupt link LNKC configured for IRQ 111317worker2 # [ 0.480466] usbcore: registered new device driver usb1318lb # [ 0.451801] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1319client # [ 0.473076] ACPI: PCI: Interrupt link LNKD configured for IRQ 111320worker1 # [ 0.450579] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1321worker2 # [ 0.482177] NetLabel: Initializing1322client # [ 0.474076] ACPI: PCI: Interrupt link LNKE configured for IRQ 101323lb # [ 0.452534] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1324worker2 # [ 0.482613] NetLabel: domain hash size = 1281325worker1 # [ 0.453010] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1326client # [ 0.475072] ACPI: PCI: Interrupt link LNKF configured for IRQ 101327worker2 # [ 0.483338] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1328lb # [ 0.453830] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1329client # [ 0.476078] ACPI: PCI: Interrupt link LNKG configured for IRQ 111330worker1 # [ 0.453608] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1331worker2 # [ 0.483675] NetLabel: unlabeled traffic allowed by default1332client # [ 0.477072] ACPI: PCI: Interrupt link LNKH configured for IRQ 111333worker2 # [ 0.484550] PCI: Using ACPI for IRQ routing1334worker1 # [ 0.454984] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1335client # [ 0.478012] ACPI: PCI: Interrupt link GSIA configured for IRQ 161336client # [ 0.478978] ACPI: PCI: Interrupt link GSIB configured for IRQ 171337worker1 # [ 0.455643] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1338lb # [ 0.457499] ACPI: PCI: Interrupt link LNKA configured for IRQ 101339client # [ 0.479993] ACPI: PCI: Interrupt link GSIC configured for IRQ 181340lb # [ 0.458291] ACPI: PCI: Interrupt link LNKB configured for IRQ 101341worker1 # [ 0.456671] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1342client # [ 0.480992] ACPI: PCI: Interrupt link GSID configured for IRQ 191343lb # [ 0.459280] ACPI: PCI: Interrupt link LNKC configured for IRQ 111344client # [ 0.481987] ACPI: PCI: Interrupt link GSIE configured for IRQ 201345worker1 # [ 0.457641] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1346lb # [ 0.460289] ACPI: PCI: Interrupt link LNKD configured for IRQ 111347client # [ 0.482938] ACPI: PCI: Interrupt link GSIF configured for IRQ 211348lb # [ 0.461280] ACPI: PCI: Interrupt link LNKE configured for IRQ 101349client # [ 0.483774] ACPI: PCI: Interrupt link GSIG configured for IRQ 221350worker1 # [ 0.459358] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1351lb # [ 0.462274] ACPI: PCI: Interrupt link LNKF configured for IRQ 101352client # [ 0.484716] ACPI: PCI: Interrupt link GSIH configured for IRQ 231353worker1 # [ 0.459992] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1354lb # [ 0.463277] ACPI: PCI: Interrupt link LNKG configured for IRQ 111355worker1 # [ 0.460706] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1356client # [ 0.486241] iommu: Default domain type: Translated1357lb # [ 0.464292] ACPI: PCI: Interrupt link LNKH configured for IRQ 111358client # [ 0.486983] iommu: DMA domain TLB invalidation policy: lazy mode1359lb # [ 0.465218] ACPI: PCI: Interrupt link GSIA configured for IRQ 161360worker1 # [ 0.462057] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1361client # [ 0.488314] ACPI: bus type USB registered1362lb # [ 0.466196] ACPI: PCI: Interrupt link GSIB configured for IRQ 171363client # [ 0.489063] usbcore: registered new interface driver usbfs1364lb # [ 0.467195] ACPI: PCI: Interrupt link GSIC configured for IRQ 181365client # [ 0.489995] usbcore: registered new interface driver hub1366lb # [ 0.468192] ACPI: PCI: Interrupt link GSID configured for IRQ 191367client # [ 0.490900] usbcore: registered new device driver usb1368worker1 # [ 0.465388] ACPI: PCI: Interrupt link LNKA configured for IRQ 101369lb # [ 0.469196] ACPI: PCI: Interrupt link GSIE configured for IRQ 201370worker1 # [ 0.466450] ACPI: PCI: Interrupt link LNKB configured for IRQ 101371lb # [ 0.470193] ACPI: PCI: Interrupt link GSIF configured for IRQ 211372client # [ 0.492771] NetLabel: Initializing1373client # [ 0.492976] NetLabel: domain hash size = 1281374lb # [ 0.471172] ACPI: PCI: Interrupt link GSIG configured for IRQ 221375worker1 # [ 0.467473] ACPI: PCI: Interrupt link LNKC configured for IRQ 111376client # [ 0.493680] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1377lb # [ 0.472146] ACPI: PCI: Interrupt link GSIH configured for IRQ 231378worker1 # [ 0.468469] ACPI: PCI: Interrupt link LNKD configured for IRQ 111379client # [ 0.494028] NetLabel: unlabeled traffic allowed by default1380worker1 # [ 0.469478] ACPI: PCI: Interrupt link LNKE configured for IRQ 101381client # [ 0.494961] PCI: Using ACPI for IRQ routing1382lb # [ 0.474022] iommu: Default domain type: Translated1383worker1 # [ 0.470477] ACPI: PCI: Interrupt link LNKF configured for IRQ 101384worker1 # [ 0.471474] ACPI: PCI: Interrupt link LNKG configured for IRQ 111385worker1 # [ 0.472471] ACPI: PCI: Interrupt link LNKH configured for IRQ 111386lb # [ 0.474821] iommu: DMA domain TLB invalidation policy: lazy mode1387worker1 # [ 0.473404] ACPI: PCI: Interrupt link GSIA configured for IRQ 161388lb # [ 0.475452] ACPI: bus type USB registered1389worker1 # [ 0.474387] ACPI: PCI: Interrupt link GSIB configured for IRQ 171390lb # [ 0.476239] usbcore: registered new interface driver usbfs1391worker1 # [ 0.475384] ACPI: PCI: Interrupt link GSIC configured for IRQ 181392lb # [ 0.477185] usbcore: registered new interface driver hub1393worker1 # [ 0.476387] ACPI: PCI: Interrupt link GSID configured for IRQ 191394lb # [ 0.478095] usbcore: registered new device driver usb1395worker1 # [ 0.477382] ACPI: PCI: Interrupt link GSIE configured for IRQ 201396lb # [ 0.479675] NetLabel: Initializing1397lb # [ 0.480181] NetLabel: domain hash size = 1281398lb # [ 0.480951] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1399worker1 # [ 0.478382] ACPI: PCI: Interrupt link GSIF configured for IRQ 211400lb # [ 0.481261] NetLabel: unlabeled traffic allowed by default1401worker1 # [ 0.479396] ACPI: PCI: Interrupt link GSIG configured for IRQ 221402lb # [ 0.482192] PCI: Using ACPI for IRQ routing1403worker1 # [ 0.480383] ACPI: PCI: Interrupt link GSIH configured for IRQ 231404worker1 # [ 0.482454] iommu: Default domain type: Translated1405worker1 # [ 0.483332] iommu: DMA domain TLB invalidation policy: lazy mode1406worker1 # [ 0.484466] ACPI: bus type USB registered1407worker1 # [ 0.485290] usbcore: registered new interface driver usbfs1408worker1 # [ 0.486119] usbcore: registered new interface driver hub1409worker1 # [ 0.486382] usbcore: registered new device driver usb1410worker1 # [ 0.488192] NetLabel: Initializing1411worker1 # [ 0.488856] NetLabel: domain hash size = 1281412worker1 # [ 0.489369] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413worker1 # [ 0.490427] NetLabel: unlabeled traffic allowed by default1414worker2 # [ 0.527776] pci 0000:00:01.0: vgaarb: setting as boot VGA device1415worker1 # [ 0.491340] PCI: Using ACPI for IRQ routing1416worker2 # [ 0.528608] pci 0000:00:01.0: vgaarb: bridge control possible1417worker2 # [ 0.528608] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418worker2 # [ 0.528615] vgaarb: loaded1419worker2 # [ 0.529345] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01420worker2 # [ 0.529614] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1421worker2 # [ 0.533698] clocksource: Switched to clocksource kvm-clock1422worker2 # [ 0.537149] VFS: Disk quotas dquot_6.6.01423worker2 # [ 0.537865] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1424worker2 # [ 0.539269] pnp: PnP ACPI init1425worker2 # [ 0.540084] ACPI: IRQ 4 override to edge(!), high(!)1426worker2 # [ 0.541057] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1427worker2 # [ 0.542611] pnp: PnP ACPI: found 6 devices1428client # [ 0.539212] pci 0000:00:01.0: vgaarb: setting as boot VGA device1429client # [ 0.539970] pci 0000:00:01.0: vgaarb: bridge control possible1430client # [ 0.539970] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1431client # [ 0.539979] vgaarb: loaded1432client # [ 0.540774] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01433client # [ 0.541604] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1434worker2 # [ 0.550205] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1435worker2 # [ 0.551704] clocksource: Switched to clocksource acpi_pm1436worker2 # [ 0.552685] NET: Registered PF_INET protocol family1437client # [ 0.545082] clocksource: Switched to clocksource kvm-clock1438worker2 # [ 0.553875] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1439lb # [ 0.526194] pci 0000:00:01.0: vgaarb: setting as boot VGA device1440lb # [ 0.527176] pci 0000:00:01.0: vgaarb: bridge control possible1441client # [ 0.548999] VFS: Disk quotas dquot_6.6.01442lb # [ 0.527176] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1443lb # [ 0.527183] vgaarb: loaded1444client # [ 0.549745] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1445client # [ 0.551168] pnp: PnP ACPI init1446lb # [ 0.527977] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01447client # [ 0.552023] ACPI: IRQ 4 override to edge(!), high(!)1448lb # [ 0.528836] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1449client # [ 0.553086] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1450client # [ 0.554700] pnp: PnP ACPI: found 6 devices1451lb # [ 0.532271] clocksource: Switched to clocksource kvm-clock1452lb # [ 0.535807] VFS: Disk quotas dquot_6.6.01453worker1 # [ 0.535088] pci 0000:00:01.0: vgaarb: setting as boot VGA device1454lb # [ 0.536534] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1455worker1 # [ 0.535365] pci 0000:00:01.0: vgaarb: bridge control possible1456lb # [ 0.537916] pnp: PnP ACPI init1457worker1 # [ 0.535365] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1458worker2 # [ 0.570494] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1459lb # [ 0.538718] ACPI: IRQ 4 override to edge(!), high(!)1460worker1 # [ 0.535374] vgaarb: loaded1461client # [ 0.562398] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1462worker2 # [ 0.571996] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1463worker1 # [ 0.536098] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01464client # [ 0.563817] clocksource: Switched to clocksource acpi_pm1465lb # [ 0.539670] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1466worker1 # [ 0.536371] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1467client # [ 0.564761] NET: Registered PF_INET protocol family1468worker2 # [ 0.573243] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1469lb # [ 0.541146] pnp: PnP ACPI: found 6 devices1470worker2 # [ 0.574514] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1471client # [ 0.565933] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1472worker2 # [ 0.575763] TCP: Hash tables configured (established 16384 bind 16384)1473worker2 # [ 0.576891] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1474worker1 # [ 0.539461] clocksource: Switched to clocksource kvm-clock1475worker2 # [ 0.578187] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1476worker2 # [ 0.579306] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1477worker2 # [ 0.580531] NET: Registered PF_UNIX/PF_LOCAL protocol family1478worker1 # [ 0.543352] VFS: Disk quotas dquot_6.6.01479worker2 # [ 0.581517] NET: Registered PF_XDP protocol family1480lb # [ 0.548773] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1481worker2 # [ 0.582330] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1482worker1 # [ 0.544101] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1483lb # [ 0.550195] clocksource: Switched to clocksource acpi_pm1484worker1 # [ 0.545471] pnp: PnP ACPI init1485worker2 # [ 0.583285] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1486lb # [ 0.551179] NET: Registered PF_INET protocol family1487worker1 # [ 0.546360] ACPI: IRQ 4 override to edge(!), high(!)1488worker2 # [ 0.584270] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1489lb # [ 0.552359] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1490worker2 # [ 0.585356] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1491worker1 # [ 0.547342] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1492worker2 # [ 0.586421] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1493worker1 # [ 0.548871] pnp: PnP ACPI: found 6 devices1494worker2 # [ 0.587509] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1495worker2 # [ 0.589307] ACPI: \_SB_.GSIA: Enabled at IRQ 161496worker2 # [ 0.591537] ACPI: \_SB_.GSIB: Enabled at IRQ 171497client # [ 0.583499] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1498worker2 # [ 0.593588] ACPI: \_SB_.GSIC: Enabled at IRQ 181499client # [ 0.585119] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1500worker1 # [ 0.556521] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1501worker2 # [ 0.595558] ACPI: \_SB_.GSID: Enabled at IRQ 191502client # [ 0.586496] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1503worker1 # [ 0.557984] clocksource: Switched to clocksource acpi_pm1504worker2 # [ 0.597217] PCI: CLS 0 bytes, default 641505client # [ 0.587875] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1506worker1 # [ 0.559066] NET: Registered PF_INET protocol family1507client # [ 0.589118] TCP: Hash tables configured (established 16384 bind 16384)1508worker1 # [ 0.560305] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1509worker2 # [ 0.598086] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1510client # [ 0.590310] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1511worker2 # [ 0.599762] Trying to unpack rootfs image as initramfs...1512client # [ 0.591590] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1513client # [ 0.592733] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1514lb # [ 0.569149] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1515client # [ 0.594020] NET: Registered PF_UNIX/PF_LOCAL protocol family1516client # [ 0.595016] NET: Registered PF_XDP protocol family1517lb # [ 0.570721] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1518client # [ 0.595885] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1519lb # [ 0.572060] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1520client # [ 0.596912] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1521lb # [ 0.573473] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1522client # [ 0.597954] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1523lb # [ 0.574787] TCP: Hash tables configured (established 16384 bind 16384)1524client # [ 0.599107] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1525lb # [ 0.575962] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1526client # [ 0.600245] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1527lb # [ 0.577286] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1528client # [ 0.601403] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1529lb # [ 0.578438] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1530client # [ 0.603293] ACPI: \_SB_.GSIA: Enabled at IRQ 161531lb # [ 0.579745] NET: Registered PF_UNIX/PF_LOCAL protocol family1532lb # [ 0.580782] NET: Registered PF_XDP protocol family1533lb # [ 0.581670] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1534client # [ 0.605782] ACPI: \_SB_.GSIB: Enabled at IRQ 171535lb # [ 0.582728] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1536worker1 # [ 0.577021] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1537client # [ 0.607748] ACPI: \_SB_.GSIC: Enabled at IRQ 181538lb # [ 0.583744] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1539worker1 # [ 0.578586] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1540lb # [ 0.584855] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1541worker1 # [ 0.579909] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1542client # [ 0.609719] ACPI: \_SB_.GSID: Enabled at IRQ 191543lb # [ 0.586005] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1544worker1 # [ 0.581333] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1545lb # [ 0.587111] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1546client # [ 0.611481] PCI: CLS 0 bytes, default 641547worker1 # [ 0.582597] TCP: Hash tables configured (established 16384 bind 16384)1548client # [ 0.612523] Trying to unpack rootfs image as initramfs...1549lb # [ 0.588872] ACPI: \_SB_.GSIA: Enabled at IRQ 161550worker1 # [ 0.583784] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1551worker1 # [ 0.585106] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1552lb # [ 0.591108] ACPI: \_SB_.GSIB: Enabled at IRQ 171553worker1 # [ 0.586244] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1554worker1 # [ 0.587506] NET: Registered PF_UNIX/PF_LOCAL protocol family1555lb # [ 0.593357] ACPI: \_SB_.GSIC: Enabled at IRQ 181556client # [ 0.616497] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1557worker1 # [ 0.588470] NET: Registered PF_XDP protocol family1558worker1 # [ 0.589347] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1559lb # [ 0.595435] ACPI: \_SB_.GSID: Enabled at IRQ 191560worker1 # [ 0.590419] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1561worker1 # [ 0.591459] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1562lb # [ 0.597190] PCI: CLS 0 bytes, default 641563worker1 # [ 0.592578] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1564lb # [ 0.598114] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1565worker1 # [ 0.593708] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1566lb # [ 0.599862] Trying to unpack rootfs image as initramfs...1567worker1 # [ 0.594838] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1568worker1 # [ 0.596593] ACPI: \_SB_.GSIA: Enabled at IRQ 161569worker1 # [ 0.598770] ACPI: \_SB_.GSIB: Enabled at IRQ 171570worker1 # [ 0.600785] ACPI: \_SB_.GSIC: Enabled at IRQ 181571worker1 # [ 0.602796] ACPI: \_SB_.GSID: Enabled at IRQ 191572worker1 # [ 0.604564] PCI: CLS 0 bytes, default 641573worker2 # [ 0.643850] Initialise system trusted keyrings1574worker1 # [ 0.605475] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1575worker1 # [ 0.607218] Trying to unpack rootfs image as initramfs...1576worker2 # [ 0.647006] workingset: timestamp_bits=40 max_order=19 bucket_order=01577client # [ 0.658689] Initialise system trusted keyrings1578worker2 # [ 0.668177] Key type asymmetric registered1579worker2 # [ 0.668879] Asymmetric key parser 'x509' registered1580worker2 # [ 0.669776] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1581client # [ 0.661579] workingset: timestamp_bits=40 max_order=19 bucket_order=01582worker2 # [ 0.675107] io scheduler mq-deadline registered1583worker2 # [ 0.675905] io scheduler kyber registered1584lb # [ 0.645997] Initialise system trusted keyrings1585worker2 # [ 0.681113] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1586lb # [ 0.649657] workingset: timestamp_bits=40 max_order=19 bucket_order=01587worker2 # [ 0.682408] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1588worker2 # [ 0.686253] Linux agpgart interface v0.1031589worker2 # [ 0.687023] ACPI: bus type drm_connector registered1590worker1 # [ 0.651312] Initialise system trusted keyrings1591worker2 # [ 0.690335] usbcore: registered new interface driver usbserial_generic1592worker2 # [ 0.691411] usbserial: USB Serial support registered for generic1593worker1 # [ 0.654152] workingset: timestamp_bits=40 max_order=19 bucket_order=01594client # [ 0.685467] Key type asymmetric registered1595client # [ 0.686200] Asymmetric key parser 'x509' registered1596worker2 # [ 0.694966] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1597worker2 # [ 0.694966] Please enable it if your BIOS has the CPPC option.1598client # [ 0.687085] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1599worker2 # [ 0.697200] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1600client # [ 0.689506] io scheduler mq-deadline registered1601client # [ 0.690296] io scheduler kyber registered1602worker2 # [ 0.700084] drop_monitor: Initializing network drop monitor service1603worker2 # [ 0.701266] NET: Registered PF_INET6 protocol family1604client # [ 0.694159] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1605lb # [ 0.670828] Key type asymmetric registered1606worker2 # [ 0.704412] Segment Routing with IPv61607lb # [ 0.671581] Asymmetric key parser 'x509' registered1608lb # [ 0.672464] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1609worker2 # [ 0.706980] In-situ OAM (IOAM) with IPv61610client # [ 0.697571] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1611worker2 # [ 0.708001] IPI shorthand broadcast: enabled1612lb # [ 0.676716] io scheduler mq-deadline registered1613lb # [ 0.677515] io scheduler kyber registered1614client # [ 0.701689] Linux agpgart interface v0.1031615client # [ 0.702493] ACPI: bus type drm_connector registered1616lb # [ 0.681180] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1617worker1 # [ 0.676323] Key type asymmetric registered1618client # [ 0.705880] usbcore: registered new interface driver usbserial_generic1619worker1 # [ 0.677076] Asymmetric key parser 'x509' registered1620lb # [ 0.682510] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1621client # [ 0.706976] usbserial: USB Serial support registered for generic1622worker1 # [ 0.677961] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1623worker2 # [ 0.716190] sched_clock: Marking stable (581013948, 134934815)->(791879110, -75930347)1624client # [ 0.709402] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1625client # [ 0.709402] Please enable it if your BIOS has the CPPC option.1626worker2 # [ 0.720190] registered taskstats version 11627lb # [ 0.687772] Linux agpgart interface v0.1031628client # [ 0.711702] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1629worker2 # [ 0.721135] Loading compiled-in X.509 certificates1630lb # [ 0.688533] ACPI: bus type drm_connector registered1631worker1 # [ 0.683233] io scheduler mq-deadline registered1632worker1 # [ 0.683974] io scheduler kyber registered1633client # [ 0.715588] drop_monitor: Initializing network drop monitor service1634lb # [ 0.691748] usbcore: registered new interface driver usbserial_generic1635client # [ 0.716820] NET: Registered PF_INET6 protocol family1636lb # [ 0.692897] usbserial: USB Serial support registered for generic1637worker1 # [ 0.687664] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1638worker1 # [ 0.688929] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1639client # [ 0.720905] Segment Routing with IPv61640lb # [ 0.696625] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1641lb # [ 0.696625] Please enable it if your BIOS has the CPPC option.1642worker1 # [ 0.692909] Linux agpgart interface v0.1031643worker1 # [ 0.693692] ACPI: bus type drm_connector registered1644lb # [ 0.698935] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1645client # [ 0.723412] In-situ OAM (IOAM) with IPv61646client # [ 0.724551] IPI shorthand broadcast: enabled1647lb # [ 0.702756] drop_monitor: Initializing network drop monitor service1648lb # [ 0.703980] NET: Registered PF_INET6 protocol family1649worker1 # [ 0.698544] usbcore: registered new interface driver usbserial_generic1650worker1 # [ 0.699601] usbserial: USB Serial support registered for generic1651lb # [ 0.706925] Segment Routing with IPv61652worker2 # [ 0.739964] Demotion targets for Node 0: null1653lb # [ 0.707643] In-situ OAM (IOAM) with IPv61654worker2 # [ 0.740854] Key type .fscrypt registered1655worker1 # [ 0.703112] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1656worker2 # [ 0.741965] Key type fscrypt-provisioning registered1657client # [ 0.733006] sched_clock: Marking stable (597016074, 135357074)->(810150160, -77777012)1658lb # [ 0.709889] IPI shorthand broadcast: enabled1659worker1 # [ 0.703112] Please enable it if your BIOS has the CPPC option.1660worker2 # [ 0.742885] ima: No TPM chip found, activating TPM-bypass!1661worker2 # [ 0.743803] ima: Allocated hash algorithm: sha11662worker1 # [ 0.705331] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1663worker1 # [ 0.708264] drop_monitor: Initializing network drop monitor service1664client # [ 0.738606] registered taskstats version 11665worker1 # [ 0.709484] NET: Registered PF_INET6 protocol family1666worker2 # [ 0.747998] ima: No architecture policies found1667client # [ 0.739636] Loading compiled-in X.509 certificates1668worker2 # [ 0.749965] PM: Magic number: 10:923:4241669worker1 # [ 0.712551] Segment Routing with IPv61670lb # [ 0.718095] sched_clock: Marking stable (585013155, 132588947)->(793232403, -75630301)1671worker2 # [ 0.751610] RAS: Correctable Errors collector initialized.1672worker1 # [ 0.714122] In-situ OAM (IOAM) with IPv61673worker1 # [ 0.716384] IPI shorthand broadcast: enabled1674lb # [ 0.722891] registered taskstats version 11675lb # [ 0.723898] Loading compiled-in X.509 certificates1676worker2 # [ 0.761628] clk: Disabling unused clocks1677worker1 # [ 0.724579] sched_clock: Marking stable (590015123, 134080259)->(800567867, -76472485)1678worker2 # [ 0.763968] PM: genpd: Disabling unused power domains1679worker1 # [ 0.728223] registered taskstats version 11680client # [ 0.757386] Demotion targets for Node 0: null1681worker1 # [ 0.729212] Loading compiled-in X.509 certificates1682client # [ 0.759434] Key type .fscrypt registered1683client # [ 0.760135] Key type fscrypt-provisioning registered1684client # [ 0.761117] ima: No TPM chip found, activating TPM-bypass!1685client # [ 0.765407] ima: Allocated hash algorithm: sha11686lb # [ 0.741621] Demotion targets for Node 0: null1687client # [ 0.766207] ima: No architecture policies found1688lb # [ 0.743671] Key type .fscrypt registered1689lb # [ 0.744346] Key type fscrypt-provisioning registered1690client # [ 0.768389] PM: Magic number: 10:923:4241691lb # [ 0.745342] ima: No TPM chip found, activating TPM-bypass!1692client # [ 0.770192] RAS: Correctable Errors collector initialized.1693lb # [ 0.747628] ima: Allocated hash algorithm: sha11694lb # [ 0.748456] ima: No architecture policies found1695lb # [ 0.751620] PM: Magic number: 10:923:4241696lb # [ 0.753307] RAS: Correctable Errors collector initialized.1697worker1 # [ 0.749120] Demotion targets for Node 0: null1698worker1 # [ 0.750116] Key type .fscrypt registered1699client # [ 0.780102] clk: Disabling unused clocks1700worker1 # [ 0.750833] Key type fscrypt-provisioning registered1701worker1 # [ 0.751817] ima: No TPM chip found, activating TPM-bypass!1702client # [ 0.782386] PM: genpd: Disabling unused power domains1703worker1 # [ 0.755121] ima: Allocated hash algorithm: sha11704worker1 # [ 0.755965] ima: No architecture policies found1705lb # [ 0.762313] clk: Disabling unused clocks1706worker1 # [ 0.759118] PM: Magic number: 10:923:4241707worker1 # [ 0.760997] RAS: Correctable Errors collector initialized.1708lb # [ 0.766625] PM: genpd: Disabling unused power domains1709worker1 # [ 0.769853] clk: Disabling unused clocks1710worker1 # [ 0.774126] PM: genpd: Disabling unused power domains1711worker2 # [ 0.922874] Freeing initrd memory: 29092K1712worker2 # [ 0.926076] Freeing unused decrypted memory: 2028K1713worker2 # [ 0.928680] Freeing unused kernel image (initmem) memory: 3644K1714worker2 # [ 0.929778] Write protecting the kernel read-only data: 32768k1715worker2 # [ 0.931636] Freeing unused kernel image (text/rodata gap) memory: 1208K1716worker2 # [ 0.933199] Freeing unused kernel image (rodata/data gap) memory: 752K1717lb # [ 0.933726] Freeing initrd memory: 29092K1718client # [ 0.959260] Freeing initrd memory: 29096K1719lb # [ 0.937110] Freeing unused decrypted memory: 2028K1720client # [ 0.962894] Freeing unused decrypted memory: 2028K1721lb # [ 0.939847] Freeing unused kernel image (initmem) memory: 3644K1722lb # [ 0.940987] Write protecting the kernel read-only data: 32768k1723client # [ 0.965905] Freeing unused kernel image (initmem) memory: 3644K1724worker1 # [ 0.937441] Freeing initrd memory: 29092K1725lb # [ 0.942970] Freeing unused kernel image (text/rodata gap) memory: 1208K1726client # [ 0.967048] Write protecting the kernel read-only data: 32768k1727lb # [ 0.944526] Freeing unused kernel image (rodata/data gap) memory: 752K1728client # [ 0.969031] Freeing unused kernel image (text/rodata gap) memory: 1208K1729worker1 # [ 0.940654] Freeing unused decrypted memory: 2028K1730client # [ 0.970701] Freeing unused kernel image (rodata/data gap) memory: 752K1731worker1 # [ 0.943331] Freeing unused kernel image (initmem) memory: 3644K1732worker1 # [ 0.944466] Write protecting the kernel read-only data: 32768k1733worker2 # [ 0.984421] x86/mm: Checked W+X mappings: passed, no W+X pages found.1734worker1 # [ 0.946376] Freeing unused kernel image (text/rodata gap) memory: 1208K1735worker2 # [ 0.985490] Run /init as init process1736worker1 # [ 0.947922] Freeing unused kernel image (rodata/data gap) memory: 752K1737worker2 # [ 0.995869] systemd[1]: Inserted module 'autofs4'1738worker2 # [ 1.012037] fuse: init (API version 7.45)1739worker2 # [ 1.017911] ACPI: \_SB_.GSIG: Enabled at IRQ 221740worker2 # [ 1.020247] ACPI: \_SB_.GSIH: Enabled at IRQ 231741worker2 # [ 1.023227] ACPI: \_SB_.GSIE: Enabled at IRQ 201742worker2 # [ 1.025417] ACPI: \_SB_.GSIF: Enabled at IRQ 211743lb # [ 0.995721] x86/mm: Checked W+X mappings: passed, no W+X pages found.1744lb # [ 0.996862] Run /init as init process1745client # [ 1.021981] x86/mm: Checked W+X mappings: passed, no W+X pages found.1746client # [ 1.023147] Run /init as init process1747worker1 # [ 0.999118] x86/mm: Checked W+X mappings: passed, no W+X pages found.1748worker1 # [ 1.000253] Run /init as init process1749lb # [ 1.007428] systemd[1]: Inserted module 'autofs4'1750client # [ 1.034034] systemd[1]: Inserted module 'autofs4'1751worker1 # [ 1.010948] systemd[1]: Inserted module 'autofs4'1752worker2 # [ 1.050767] systemd[1]: Successfully made /usr/ read-only.1753lb # [ 1.023898] fuse: init (API version 7.45)1754client # [ 1.050885] fuse: init (API version 7.45)1755lb # [ 1.029755] ACPI: \_SB_.GSIG: Enabled at IRQ 221756lb # [ 1.032175] ACPI: \_SB_.GSIH: Enabled at IRQ 231757worker1 # [ 1.027322] fuse: init (API version 7.45)1758client # [ 1.056977] ACPI: \_SB_.GSIG: Enabled at IRQ 221759lb # [ 1.035396] ACPI: \_SB_.GSIE: Enabled at IRQ 201760client # [ 1.059491] ACPI: \_SB_.GSIH: Enabled at IRQ 231761lb # [ 1.037703] ACPI: \_SB_.GSIF: Enabled at IRQ 211762worker1 # [ 1.033225] ACPI: \_SB_.GSIG: Enabled at IRQ 221763client # [ 1.062595] ACPI: \_SB_.GSIE: Enabled at IRQ 201764client # [ 1.064801] ACPI: \_SB_.GSIF: Enabled at IRQ 211765worker1 # [ 1.035685] ACPI: \_SB_.GSIH: Enabled at IRQ 231766worker1 # [ 1.038627] ACPI: \_SB_.GSIE: Enabled at IRQ 201767worker1 # [ 1.040918] ACPI: \_SB_.GSIF: Enabled at IRQ 211768lb # [ 1.065405] systemd[1]: Successfully made /usr/ read-only.1769client # [ 1.091318] systemd[1]: Successfully made /usr/ read-only.1770worker1 # [ 1.068856] systemd[1]: Successfully made /usr/ read-only.1771worker2 # [ 1.387214] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1772worker2 # [ 1.399091] systemd[1]: Detected virtualization kvm.1773worker2 # [ 1.401183] systemd[1]: Detected architecture x86-64.1774worker2 # [ 1.403268] systemd[1]: Running in initrd.1775worker2 # [ 1.405657] systemd[1]: Initializing machine ID from random generator.1776worker2 # [ 1.408516] systemd[1]: Hostname set to <worker2>.1777lb # [ 1.401876] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1778worker1 # [ 1.404342] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1779worker1 # [ 1.409246] systemd[1]: Detected virtualization kvm.1780client # [ 1.427945] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1781lb # [ 1.413906] systemd[1]: Detected virtualization kvm.1782worker1 # [ 1.410112] systemd[1]: Detected architecture x86-64.1783worker1 # [ 1.410953] systemd[1]: Running in initrd.1784lb # [ 1.415993] systemd[1]: Detected architecture x86-64.1785client # [ 1.439909] systemd[1]: Detected virtualization kvm.1786worker1 # [ 1.412140] systemd[1]: Initializing machine ID from random generator.1787worker1 # [ 1.413383] systemd[1]: Hostname set to <worker1>.1788lb # [ 1.418087] systemd[1]: Running in initrd.1789client # [ 1.442094] systemd[1]: Detected architecture x86-64.1790client # [ 1.444256] systemd[1]: Running in initrd.1791lb # [ 1.420819] systemd[1]: Initializing machine ID from random generator.1792client # [ 1.446912] systemd[1]: Initializing machine ID from random generator.1793lb # [ 1.423913] systemd[1]: Hostname set to <lb>.1794client # [ 1.449804] systemd[1]: Hostname set to <client>.1795worker2 # [ 1.623296] systemd[1]: bpf-restrict-fs: LSM BPF program attached1796worker1 # [ 1.622132] systemd[1]: bpf-restrict-fs: LSM BPF program attached1797worker2 # [ 1.671713] systemd[1]: Queued start job for default target Initrd Default Target.1798worker2 # [ 1.676246] systemd[1]: Created slice Slice /system/modprobe.1799lb # [ 1.643399] systemd[1]: bpf-restrict-fs: LSM BPF program attached1800worker2 # [ 1.677429] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1801worker2 # [ 1.678797] systemd[1]: Expecting device /dev/disk/by-label/nixos...1802worker2 # [ 1.679962] systemd[1]: Reached target Path Units.1803worker2 # [ 1.680844] systemd[1]: Reached target Slice Units.1804worker2 # [ 1.681828] systemd[1]: Reached target Swaps.1805worker2 # [ 1.682663] systemd[1]: Reached target Timer Units.1806worker2 # [ 1.683637] systemd[1]: Listening on D-Bus System Message Bus Socket.1807worker2 # [ 1.684884] systemd[1]: Listening on Journal Socket (/dev/log).1808worker2 # [ 1.686078] systemd[1]: Listening on Journal Sockets.1809client # [ 1.676508] systemd[1]: bpf-restrict-fs: LSM BPF program attached1810worker2 # [ 1.687085] systemd[1]: Listening on udev Control Socket.1811worker2 # [ 1.688109] systemd[1]: Listening on udev Kernel Socket.1812worker2 # [ 1.689082] systemd[1]: Reached target Socket Units.1813worker2 # [ 1.690828] systemd[1]: Starting Create List of Static Device Nodes...1814worker2 # [ 1.694762] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1815worker2 # [ 1.702136] systemd[1]: Starting Load Kernel Module configfs...1816worker1 # [ 1.670104] systemd[1]: Queued start job for default target Initrd Default Target.1817worker1 # [ 1.674391] systemd[1]: Created slice Slice /system/modprobe.1818worker1 # [ 1.675580] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1819worker2 # [ 1.715023] systemd[1]: Starting Journal Service...1820worker1 # [ 1.676993] systemd[1]: Expecting device /dev/disk/by-label/nixos...1821worker1 # [ 1.678110] systemd[1]: Reached target Path Units.1822worker1 # [ 1.678980] systemd[1]: Reached target Slice Units.1823worker1 # [ 1.679898] systemd[1]: Reached target Swaps.1824worker1 # [ 1.680729] systemd[1]: Reached target Timer Units.1825worker1 # [ 1.681710] systemd[1]: Listening on D-Bus System Message Bus Socket.1826worker1 # [ 1.682995] systemd[1]: Listening on Journal Socket (/dev/log).1827worker1 # [ 1.684205] systemd[1]: Listening on Journal Sockets.1828worker1 # [ 1.685296] systemd[1]: Listening on udev Control Socket.1829worker1 # [ 1.686361] systemd[1]: Listening on udev Kernel Socket.1830worker1 # [ 1.687341] systemd[1]: Reached target Socket Units.1831worker2 # [ 1.726042] systemd[1]: Starting Load Kernel Modules...1832worker1 # [ 1.689123] systemd[1]: Starting Create List of Static Device Nodes...1833lb # [ 1.696517] systemd[1]: Queued start job for default target Initrd Default Target.1834worker1 # [ 1.692898] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1835lb # [ 1.700908] systemd[1]: Created slice Slice /system/modprobe.1836lb # [ 1.702130] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1837lb # [ 1.703603] systemd[1]: Expecting device /dev/disk/by-label/nixos...1838lb # [ 1.704738] systemd[1]: Reached target Path Units.1839worker1 # [ 1.699649] systemd[1]: Starting Load Kernel Module configfs...1840lb # [ 1.705692] systemd[1]: Reached target Slice Units.1841lb # [ 1.706617] systemd[1]: Reached target Swaps.1842client # [ 1.730668] systemd[1]: Queued start job for default target Initrd Default Target.1843lb # [ 1.707406] systemd[1]: Reached target Timer Units.1844lb # [ 1.708357] systemd[1]: Listening on D-Bus System Message Bus Socket.1845lb # [ 1.709677] systemd[1]: Listening on Journal Socket (/dev/log).1846lb # [ 1.710864] systemd[1]: Listening on Journal Sockets.1847client # [ 1.734757] systemd[1]: Created slice Slice /system/modprobe.1848lb # [ 1.711897] systemd[1]: Listening on udev Control Socket.1849client # [ 1.735948] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1850lb # [ 1.712952] systemd[1]: Listening on udev Kernel Socket.1851worker2 # [ 1.734069] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1852client # [ 1.737331] systemd[1]: Expecting device /dev/disk/by-label/nixos...1853lb # [ 1.713897] systemd[1]: Reached target Socket Units.1854client # [ 1.738437] systemd[1]: Reached target Path Units.1855client # [ 1.739318] systemd[1]: Reached target Slice Units.1856lb # [ 1.715671] systemd[1]: Starting Create List of Static Device Nodes...1857client # [ 1.740221] systemd[1]: Reached target Swaps.1858client # [ 1.741040] systemd[1]: Reached target Timer Units.1859worker2 # [ 1.751356] netfs: FS-Cache loaded1860client # [ 1.742059] systemd[1]: Listening on D-Bus System Message Bus Socket.1861worker1 # [ 1.713279] systemd[1]: Starting Journal Service...1862client # [ 1.743316] systemd[1]: Listening on Journal Socket (/dev/log).1863lb # [ 1.719425] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1864client # [ 1.744545] systemd[1]: Listening on Journal Sockets.1865client # [ 1.745587] systemd[1]: Listening on udev Control Socket.1866client # [ 1.746641] systemd[1]: Listening on udev Kernel Socket.1867worker2 # [ 1.755895] 9pnet: Installing 9P2000 support1868client # [ 1.747622] systemd[1]: Reached target Socket Units.1869client # [ 1.749558] systemd[1]: Starting Create List of Static Device Nodes...1870lb # [ 1.726178] systemd[1]: Starting Load Kernel Module configfs...1871client # [ 1.753441] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1872worker2 # [ 1.764210] systemd[1]: Starting Coldplug All udev Devices...1873worker1 # [ 1.727162] systemd[1]: Starting Load Kernel Modules...1874client # [ 1.760096] systemd[1]: Starting Load Kernel Module configfs...1875lb # [ 1.739870] systemd[1]: Starting Journal Service...1876worker2 # [ 1.777426] systemd[1]: Finished Create List of Static Device Nodes.1877worker1 # [ 1.732173] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1878client # [ 1.774601] systemd[1]: Starting Journal Service...1879worker2 # [ 1.783576] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1880lb # [ 1.753710] systemd[1]: Starting Load Kernel Modules...1881worker1 # [ 1.750054] netfs: FS-Cache loaded1882worker2 # [ 1.791419] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1883worker1 # [ 1.754802] 9pnet: Installing 9P2000 support1884client # [ 1.787634] systemd[1]: Starting Load Kernel Modules...1885worker2 # [ 1.798355] systemd[1]: modprobe@configfs.service: Deactivated successfully.1886worker2 # [ 1.801475] systemd-journald[67]: Collecting audit messages is disabled.1887worker1 # [ 1.763847] systemd[1]: Starting Coldplug All udev Devices...1888worker2 # [ 1.804028] systemd[1]: Finished Load Kernel Module configfs.1889lb # [ 1.760678] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1890lb # [ 1.777076] netfs: FS-Cache loaded1891worker2 # [ 1.810398] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1892client # [ 1.792448] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1893lb # [ 1.781800] 9pnet: Installing 9P2000 support1894worker2 # [ 1.813553] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1895worker1 # [ 1.777668] systemd[1]: Finished Create List of Static Device Nodes.1896client # [ 1.810547] netfs: FS-Cache loaded1897worker2 # [ 1.821044] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1898worker1 # [ 1.783687] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1899worker2 # [ 1.822505] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1900lb # [ 1.790672] systemd[1]: Starting Coldplug All udev Devices...1901client # [ 1.815429] 9pnet: Installing 9P2000 support1902worker1 # [ 1.791503] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1903client # [ 1.825292] systemd[1]: Starting Coldplug All udev Devices...1904worker1 # [ 1.797516] systemd[1]: modprobe@configfs.service: Deactivated successfully.1905lb # [ 1.804758] systemd[1]: Finished Create List of Static Device Nodes.1906worker1 # [ 1.800667] systemd-journald[67]: Collecting audit messages is disabled.1907worker1 # [ 1.803178] systemd[1]: Finished Load Kernel Module configfs.1908lb # [ 1.810202] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1909worker1 # [ 1.809546] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1910client # [ 1.839854] systemd[1]: Finished Create List of Static Device Nodes.1911lb # [ 1.818033] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1912worker2 # [ 1.851531] systemd[1]: Finished Load Kernel Modules.1913worker1 # [ 1.812768] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1914client # [ 1.844977] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1915lb # [ 1.823017] systemd[1]: modprobe@configfs.service: Deactivated successfully.1916lb # [ 1.824726] systemd-journald[67]: Collecting audit messages is disabled.1917worker2 # [ 1.859137] systemd[1]: Starting Apply Kernel Variables...1918worker1 # [ 1.821187] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1919worker1 # [ 1.822627] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1920client # [ 1.853786] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1921lb # [ 1.829984] systemd[1]: Finished Load Kernel Module configfs.1922client # [ 1.859168] systemd-journald[67]: Collecting audit messages is disabled.1923lb # [ 1.836105] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1924client # [ 1.860776] systemd[1]: modprobe@configfs.service: Deactivated successfully.1925worker2 # [ 1.870056] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1926lb # [ 1.840198] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1927client # [ 1.867753] systemd[1]: Finished Load Kernel Module configfs.1928worker2 # [ 1.879228] systemd[1]: Starting Create Static Device Nodes in /dev...1929client # [ 1.870847] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1930lb # [ 1.847674] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1931lb # [ 1.849763] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1932client # [ 1.875817] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1933worker1 # [ 1.851632] systemd[1]: Finished Load Kernel Modules.1934client # [ 1.883591] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1935client # [ 1.884963] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1936worker1 # [ 1.859319] systemd[1]: Starting Apply Kernel Variables...1937worker2 # [ 1.898266] systemd[1]: Finished Apply Kernel Variables.1938worker1 # [ 1.870222] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1939lb # [ 1.880111] systemd[1]: Finished Load Kernel Modules.1940worker2 # [ 1.913020] systemd[1]: Started Journal Service.1941worker2 # [ 1.779212] systemd-modules-load[68]: Inserted module 'dm_mod'1942worker2 # [ 1.781892] systemd-modules-load[68]: Inserted module 'virtio_balloon'1943worker2 # [ 1.783419] systemd-modules-load[68]: Inserted module 'virtio_gpu'1944worker1 # [ 1.880293] systemd[1]: Starting Create Static Device Nodes in /dev...1945lb # [ 1.887688] systemd[1]: Starting Apply Kernel Variables...1946worker2 # [ 1.785674] systemd[1]: Finished Create Static Device Nodes in /dev.1947client # [ 1.914911] systemd[1]: Finished Load Kernel Modules.1948worker2 # [ 1.793618] systemd[1]: Reached target Preparation for Local File Systems.1949client # [ 1.921575] systemd[1]: Starting Apply Kernel Variables...1950worker1 # [ 1.892682] systemd[1]: Finished Apply Kernel Variables.1951worker2 # [ 1.796215] systemd[1]: Reached target Local File Systems.1952lb # [ 1.897713] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1953worker2 # [ 1.798080] systemd[1]: Starting Create System Files and Directories...1954lb # [ 1.906795] systemd[1]: Starting Create Static Device Nodes in /dev...1955worker2 # [ 1.805079] systemd[1]: Starting Rule-based Manager for Device Events and Files...1956client # [ 1.931460] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1957client # [ 1.941454] systemd[1]: Starting Create Static Device Nodes in /dev...1958worker1 # [ 1.779602] systemd-modules-load[68]: Inserted module 'dm_mod'[ 1.914714] systemd[1]: Started Journal Service.1959worker1 # 1960worker1 # [ 1.783294] systemd-modules-load[68]: Inserted module 'virtio_balloon'1961worker1 # [ 1.785334] systemd-modules-load[68]: Inserted module 'virtio_gpu'1962worker1 # [ 1.786418] systemd[1]: Finished Create Static Device Nodes in /dev.1963worker1 # [ 1.787584] systemd[1]: Reached target Preparation for Local File Systems.1964lb # [ 1.926898] systemd[1]: Finished Apply Kernel Variables.1965worker1 # [ 1.789398] systemd[1]: Reached target Local File Systems.1966worker1 # [ 1.791198] systemd[1]: Starting Create System Files and Directories...1967worker2 # [ 1.830497] systemd[1]: Finished Create System Files and Directories.1968client # [ 1.959533] systemd[1]: Finished Apply Kernel Variables.1969worker1 # [ 1.799925] systemd[1]: Starting Rule-based Manager for Device Events and Files...1970lb # [ 1.939504] systemd[1]: Started Journal Service.1971lb # [ 1.809205] systemd-modules-load[68]: Inserted module 'dm_mod'1972lb # [ 1.812283] systemd-modules-load[68]: Inserted module 'virtio_balloon'1973lb # [ 1.814285] systemd-modules-load[68]: Inserted module 'virtio_gpu'1974lb # [ 1.815380] systemd[1]: Finished Create Static Device Nodes in /dev.1975lb # [ 1.817393] systemd[1]: Reached target Preparation for Local File Systems.1976client # [ 1.839349] systemd-modules-load[68]: Inserted module 'dm_mod'[ 1.975821] systemd[1]: Started Journal Service.1977lb # [ 1.820058] systemd[1]: Reached target Local File Systems.1978client # 1979lb # [ 1.822055] systemd[1]: Starting Create System Files and Directories...1980client # [ 1.843262] systemd-modules-load[68]: Inserted module 'virtio_balloon'1981worker2 # [ 1.853252] systemd-udevd[84]: Using default interface naming scheme 'v261'.1982client # [ 1.844429] systemd-modules-load[68]: Inserted module 'virtio_gpu'1983lb # [ 1.829069] systemd[1]: Starting Rule-based Manager for Device Events and Files...1984worker1 # [ 1.823196] systemd[1]: Finished Create System Files and Directories.1985client # [ 1.852115] systemd[1]: Finished Create Static Device Nodes in /dev.1986client # [ 1.856111] systemd[1]: Reached target Preparation for Local File Systems.1987client # [ 1.858082] systemd[1]: Reached target Local File Systems.1988client # [ 1.861080] systemd[1]: Starting Create System Files and Directories...1989worker2 # [ 1.875698] systemd[1]: Started Rule-based Manager for Device Events and Files.1990client # [ 1.868072] systemd[1]: Starting Rule-based Manager for Device Events and Files...1991worker1 # [ 1.851721] systemd-udevd[76]: Using default interface naming scheme 'v261'.1992lb # [ 1.859321] systemd[1]: Finished Create System Files and Directories.1993client # [ 1.891645] systemd[1]: Finished Create System Files and Directories.1994lb # [ 1.881352] systemd-udevd[78]: Using default interface naming scheme 'v261'.1995worker1 # [ 1.876237] systemd[1]: Started Rule-based Manager for Device Events and Files.1996worker2 # [ 1.920080] systemd[1]: Finished Coldplug All udev Devices.1997worker2 # [ 1.920903] systemd[1]: Reached target System Initialization.1998worker2 # [ 1.921702] systemd[1]: Reached target Basic System.1999client # [ 1.913376] systemd-udevd[83]: Using default interface naming scheme 'v261'.2000lb # [ 1.906318] systemd[1]: Started Rule-based Manager for Device Events and Files.
loading 2,070 hidden lines…
4071lb # [ 16.052156] envoy[902]: [2026-09-21 09:27:00.872][902][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.cel, envoy.tracers.opentelemetry.samplers.dynatrace, envoy.tracers.opentelemetry.samplers.parent_based, envoy.tracers.opentelemetry.samplers.trace_id_ratio_based4072lb # [ 16.056051] envoy[902]: [2026-09-21 09:27:00.872][902][info][main] [source/server/server.cc:471] envoy.srds_factory: envoy.srds_factory.default4073lb # [ 16.057536] envoy[902]: [2026-09-21 09:27:00.872][902][info][main] [source/server/server.cc:471] quic.http_server_connection: quic.http_server_connection.default4074lb # [ 16.059199] envoy[902]: [2026-09-21 09:27:00.872][902][info][main] [source/server/server.cc:471] envoy.retry_priorities: envoy.retry_priorities.previous_priorities4075lb # [ 16.060820] envoy[902]: [2026-09-21 09:27:00.872][902][info][main] [source/server/server.cc:471] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher4076lb # [ 16.063059] envoy[902]: [2026-09-21 09:27:00.872][902][info][main] [source/server/server.cc:471] envoy.upstreams: envoy.filters.connection_pools.tcp.generic4077lb # [ 16.064767] envoy[902]: [2026-09-21 09:27:00.872][902][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.serializers: dubbo.hessian24078lb # [ 16.066693] envoy[902]: [2026-09-21 09:27:00.872][902][info][main] [source/server/server.cc:471] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme4079lb # [ 16.069735] envoy[902]: [2026-09-21 09:27:00.998][902][info][main] [source/server/server.cc:531] HTTP header map info:4080lb # [ 16.071427] envoy[902]: [2026-09-21 09:27:01.038][902][info][main] [source/server/server.cc:534] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-host,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id4081lb # [ 16.084055] envoy[902]: [2026-09-21 09:27:01.038][902][info][main] [source/server/server.cc:534] request trailer map: 120 bytes:4082lb # [ 16.085671] envoy[902]: [2026-09-21 09:27:01.038][902][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id4083lb # [ 16.093104] envoy[902]: [2026-09-21 09:27:01.038][902][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status4084worker1 # [ 16.105539] systemd[1]: Started DHCP Client.4085worker1 # [ 16.107535] systemd[1]: Reached target Multi-User System.4086worker1 # [ 16.108551] systemd[1]: Startup finished in 869ms (kernel) + 4.018s (initrd) + 11.219s (userspace) = 16.107s.4087lb # [ 16.120150] envoy[902]: [2026-09-21 09:27:01.222][902][info][main] [source/server/server.cc:968] runtime: {}4088lb # [ 16.125079] envoy[902]: [2026-09-21 09:27:01.227][902][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288.4089lb # [ 16.127145] envoy[902]: [2026-09-21 09:27:01.229][902][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:99014090lb # [ 16.129474] envoy[902]: [2026-09-21 09:27:01.231][902][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:99014091lb # [ 16.131912] envoy[902]: [2026-09-21 09:27:01.234][902][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration4092lb # [ 16.133502] envoy[902]: [2026-09-21 09:27:01.235][902][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s)4093lb # [ 16.135122] envoy[902]: [2026-09-21 09:27:01.235][902][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s)4094lb # [ 16.187950] envoy[902]: [2026-09-21 09:27:01.290][902][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s)4095worker2 # [ 16.219296] systemd[1]: Reached target Network is Online.4096worker1 # [ 16.183208] systemd[1]: Starting Nix worker-protocol over gRPC...4097worker2 # [ 16.223253] systemd[1]: Starting scheduler-token.service...4098lb # [ 16.206935] envoy[902]: [2026-09-21 09:27:01.309][902][info][config] [source/server/configuration_impl.cc:164] loading stats configuration4099lb # [ 16.209194] envoy[902]: [2026-09-21 09:27:01.311][902][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor.4100lb # [ 16.212196] envoy[902]: [2026-09-21 09:27:01.311][902][info][main] [source/server/server.cc:1077] starting main dispatch loop4101worker2 # [ 16.315426] systemd[1]: scheduler-token.service: Deactivated successfully.4102worker2 # [ 16.316527] systemd[1]: Finished scheduler-token.service.4103worker2 # [ 16.318209] systemd[1]: scheduler-token.service: Consumed 20ms CPU time over 92ms wall clock time, 3.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4104worker2 # [ 16.322933] systemd[1]: Starting Nix worker-protocol over gRPC...4105worker1 # [ 16.427571] nix-grpc-daemon[781]: ts=2026-09-21T09:27:01Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4106worker1 # [ 16.438113] nix-grpc-daemon[781]: ts=2026-09-21T09:27:01Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4107worker1 # [ 16.444201] nix-grpc-daemon[781]: ts=2026-09-21T09:27:01Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=14108worker2 # [ 16.509409] nix-grpc-daemon[786]: ts=2026-09-21T09:27:00Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4109worker2 # [ 16.518563] nix-grpc-daemon[786]: ts=2026-09-21T09:27:00Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4110worker1 # [ 16.482589] nix-grpc-daemon[781]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4111worker1 # [ 16.484019] nix-grpc-daemon[781]: E0000 00:00:1789982821.643728 781 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4112worker2 # [ 16.523550] nix-grpc-daemon[786]: ts=2026-09-21T09:27:00Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14113worker1 # [ 16.501568] nix-grpc-daemon[781]: E0000 00:00:1789982821.664063 781 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4114worker1 # [ 16.505368] nix-grpc-daemon[781]: ts=2026-09-21T09:27:01Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4115worker1 # [ 16.508394] systemd[1]: Started Nix worker-protocol over gRPC.4116worker2 # [ 16.557878] nix-grpc-daemon[786]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4117worker2 # [ 16.559385] nix-grpc-daemon[786]: E0000 00:00:1789982820.915786 786 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4118worker2 # [ 16.575062] nix-grpc-daemon[786]: E0000 00:00:1789982820.934082 786 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4119worker2 # [ 16.578739] nix-grpc-daemon[786]: ts=2026-09-21T09:27:00Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4120worker1 # [ 16.541831] nix-grpc-daemon[781]: E0000 00:00:1789982821.704264 822 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4121worker2 # [ 16.581905] systemd[1]: Started Nix worker-protocol over gRPC.4122worker1 # [ 16.563620] nix-grpc-daemon[781]: E0000 00:00:1789982821.726079 822 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4123worker1 # [ 16.567191] nix-grpc-daemon[781]: E0000 00:00:1789982821.729688 822 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4124lb # [ 16.577299] envoy[902]: [2026-09-21 09:27:01.679][902][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization4125worker2 # [ 16.611888] nix-grpc-daemon[786]: E0000 00:00:1789982820.970288 826 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4126lb # [ 16.581669] envoy[902]: [2026-09-21 09:27:01.679][902][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized4127lb # [ 16.585744] envoy[902]: [2026-09-21 09:27:01.679][902][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager4128worker1 # [ 16.580947] nix-grpc-daemon[781]: E0000 00:00:1789982821.743429 822 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4129lb # [ 16.594388] envoy[902]: [2026-09-21 09:27:01.696][902][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers4130worker2 # [ 16.628632] nix-grpc-daemon[786]: E0000 00:00:1789982820.987668 826 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4131worker1 # [ 16.594124] nix-grpc-daemon[781]: ts=2026-09-21T09:27:01Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4132worker2 # [ 16.632122] nix-grpc-daemon[786]: E0000 00:00:1789982820.990867 826 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4133worker2 # [ 16.650062] nix-grpc-daemon[786]: E0000 00:00:1789982821.009059 826 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4134worker2 # [ 16.672142] nix-grpc-daemon[786]: ts=2026-09-21T09:27:01Z level=info event=scheduler_connected addr=lb:500514135worker2 # [ 16.683538] nix-grpc-daemon[786]: ts=2026-09-21T09:27:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4136worker1 # [ 16.795509] nix-grpc-daemon[781]: ts=2026-09-21T09:27:01Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4137worker2 # [ 16.886218] nix-grpc-daemon[786]: ts=2026-09-21T09:27:01Z level=info event=scheduler_connected addr=lb:500514138worker2 # [ 16.890314] nix-grpc-daemon[786]: ts=2026-09-21T09:27:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4139client # [ 17.057802] dhcpcd[566]: eth0: leased 10.0.2.15 for 86400 seconds4140client # [ 17.061852] dhcpcd[566]: eth0: adding route to 10.0.2.0/244141client # [ 17.063710] dhcpcd[566]: eth0: adding default route via 10.0.2.24142worker2 # [ 17.091258] nix-grpc-daemon[786]: ts=2026-09-21T09:27:01Z level=info event=scheduler_connected addr=lb:500514143worker2 # [ 17.095186] nix-grpc-daemon[786]: ts=2026-09-21T09:27:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4144client # [ 17.200173] systemd[1]: Started DHCP Client.4145client # [ 17.202274] systemd[1]: Reached target Multi-User System.4146client # [ 17.203357] systemd[1]: Startup finished in 891ms (kernel) + 4.097s (initrd) + 12.212s (userspace) = 17.202s.4147worker1 # [ 17.196295] nix-grpc-daemon[781]: ts=2026-09-21T09:27:02Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4148worker2 # [ 17.296287] nix-grpc-daemon[786]: ts=2026-09-21T09:27:01Z level=info event=scheduler_connected addr=lb:500514149worker2 # [ 17.300158] nix-grpc-daemon[786]: ts=2026-09-21T09:27:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4150worker2 # [ 17.500679] nix-grpc-daemon[786]: ts=2026-09-21T09:27:01Z level=info event=scheduler_connected addr=lb:500514151worker2 # [ 17.502425] nix-grpc-daemon[786]: ts=2026-09-21T09:27:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4152worker2 # [ 17.704352] nix-grpc-daemon[786]: ts=2026-09-21T09:27:02Z level=info event=scheduler_connected addr=lb:500514153worker2 # [ 17.707970] nix-grpc-daemon[786]: ts=2026-09-21T09:27:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4154worker2 # [ 17.909281] nix-grpc-daemon[786]: ts=2026-09-21T09:27:02Z level=info event=scheduler_connected addr=lb:500514155worker2 # [ 17.913136] nix-grpc-daemon[786]: ts=2026-09-21T09:27:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4156worker1 # [ 18.000915] nix-grpc-daemon[781]: ts=2026-09-21T09:27:03Z level=info event=scheduler_connected addr=lb:500514157worker1 # [ 18.005950] nix-grpc-daemon[781]: ts=2026-09-21T09:27:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4158worker2 # [ 18.114134] nix-grpc-daemon[786]: ts=2026-09-21T09:27:02Z level=info event=scheduler_connected addr=lb:500514159worker2 # [ 18.118450] nix-grpc-daemon[786]: ts=2026-09-21T09:27:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4160worker1 # [ 18.207150] nix-grpc-daemon[781]: ts=2026-09-21T09:27:03Z level=info event=scheduler_connected addr=lb:500514161worker1 # [ 18.211272] nix-grpc-daemon[781]: ts=2026-09-21T09:27:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4162worker2 # [ 18.319331] nix-grpc-daemon[786]: ts=2026-09-21T09:27:02Z level=info event=scheduler_connected addr=lb:500514163worker2 # [ 18.323123] nix-grpc-daemon[786]: ts=2026-09-21T09:27:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4164worker1 # [ 18.412154] nix-grpc-daemon[781]: ts=2026-09-21T09:27:03Z level=info event=scheduler_connected addr=lb:500514165worker1 # [ 18.416158] nix-grpc-daemon[781]: ts=2026-09-21T09:27:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4166worker2 # [ 18.524268] nix-grpc-daemon[786]: ts=2026-09-21T09:27:02Z level=info event=scheduler_connected addr=lb:500514167worker2 # [ 18.528139] nix-grpc-daemon[786]: ts=2026-09-21T09:27:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4168worker1 # [ 18.617232] nix-grpc-daemon[781]: ts=2026-09-21T09:27:03Z level=info event=scheduler_connected addr=lb:500514169worker1 # [ 18.620830] nix-grpc-daemon[781]: ts=2026-09-21T09:27:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4170worker2 # [ 18.730310] nix-grpc-daemon[786]: ts=2026-09-21T09:27:03Z level=info event=scheduler_connected addr=lb:500514171worker2 # [ 18.733394] nix-grpc-daemon[786]: ts=2026-09-21T09:27:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4172worker1 # [ 18.819138] nix-grpc-daemon[781]: ts=2026-09-21T09:27:03Z level=info event=scheduler_connected addr=lb:500514173worker1 # [ 18.823110] nix-grpc-daemon[781]: ts=2026-09-21T09:27:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4174worker2 # [ 18.931457] nix-grpc-daemon[786]: ts=2026-09-21T09:27:03Z level=info event=scheduler_connected addr=lb:500514175worker2 # [ 18.935236] nix-grpc-daemon[786]: ts=2026-09-21T09:27:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4176worker1 # [ 19.024943] nix-grpc-daemon[781]: ts=2026-09-21T09:27:04Z level=info event=scheduler_connected addr=lb:500514177worker1 # [ 19.028811] nix-grpc-daemon[781]: ts=2026-09-21T09:27:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4178worker2 # [ 19.136244] nix-grpc-daemon[786]: ts=2026-09-21T09:27:03Z level=info event=scheduler_connected addr=lb:500514179worker2 # [ 19.139968] nix-grpc-daemon[786]: ts=2026-09-21T09:27:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4180worker1 # [ 19.229748] nix-grpc-daemon[781]: ts=2026-09-21T09:27:04Z level=info event=scheduler_connected addr=lb:500514181worker1 # [ 19.233692] nix-grpc-daemon[781]: ts=2026-09-21T09:27:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4182worker2 # [ 19.341173] nix-grpc-daemon[786]: ts=2026-09-21T09:27:03Z level=info event=scheduler_connected addr=lb:500514183worker2 # [ 19.344979] nix-grpc-daemon[786]: ts=2026-09-21T09:27:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4184worker1 # [ 19.434681] nix-grpc-daemon[781]: ts=2026-09-21T09:27:04Z level=info event=scheduler_connected addr=lb:500514185worker1 # [ 19.438708] nix-grpc-daemon[781]: ts=2026-09-21T09:27:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4186worker2 # [ 19.546210] nix-grpc-daemon[786]: ts=2026-09-21T09:27:03Z level=info event=scheduler_connected addr=lb:500514187worker2 # [ 19.550172] nix-grpc-daemon[786]: ts=2026-09-21T09:27:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4188worker1 # [ 19.639361] nix-grpc-daemon[781]: ts=2026-09-21T09:27:04Z level=info event=scheduler_connected addr=lb:500514189worker1 # [ 19.642799] nix-grpc-daemon[781]: ts=2026-09-21T09:27:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4190worker2 # [ 19.752321] nix-grpc-daemon[786]: ts=2026-09-21T09:27:04Z level=info event=scheduler_connected addr=lb:500514191worker2 # [ 19.755451] nix-grpc-daemon[786]: ts=2026-09-21T09:27:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4192worker1 # [ 19.843753] nix-grpc-daemon[781]: ts=2026-09-21T09:27:05Z level=info event=scheduler_connected addr=lb:500514193worker1 # [ 19.847728] nix-grpc-daemon[781]: ts=2026-09-21T09:27:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4194worker2 # [ 19.953686] nix-grpc-daemon[786]: ts=2026-09-21T09:27:04Z level=info event=scheduler_connected addr=lb:500514195worker2 # [ 19.957474] nix-grpc-daemon[786]: ts=2026-09-21T09:27:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4196worker1 # [ 20.049920] nix-grpc-daemon[781]: ts=2026-09-21T09:27:05Z level=info event=scheduler_connected addr=lb:500514197worker1 # [ 20.053718] nix-grpc-daemon[781]: ts=2026-09-21T09:27:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4198worker2 # [ 20.158367] nix-grpc-daemon[786]: ts=2026-09-21T09:27:04Z level=info event=scheduler_connected addr=lb:500514199worker2 # [ 20.162205] nix-grpc-daemon[786]: ts=2026-09-21T09:27:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4200worker1 # [ 20.254611] nix-grpc-daemon[781]: ts=2026-09-21T09:27:05Z level=info event=scheduler_connected addr=lb:500514201worker1 # [ 20.258550] nix-grpc-daemon[781]: ts=2026-09-21T09:27:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4202worker2 # [ 20.363152] nix-grpc-daemon[786]: ts=2026-09-21T09:27:04Z level=info event=scheduler_connected addr=lb:500514203worker2 # [ 20.366969] nix-grpc-daemon[786]: ts=2026-09-21T09:27:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4204worker1 # [ 20.459748] nix-grpc-daemon[781]: ts=2026-09-21T09:27:05Z level=info event=scheduler_connected addr=lb:500514205worker1 # [ 20.463699] nix-grpc-daemon[781]: ts=2026-09-21T09:27:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4206worker2 # [ 20.568124] nix-grpc-daemon[786]: ts=2026-09-21T09:27:04Z level=info event=scheduler_connected addr=lb:500514207worker2 # [ 20.571966] nix-grpc-daemon[786]: ts=2026-09-21T09:27:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4208worker1 # [ 20.664693] nix-grpc-daemon[781]: ts=2026-09-21T09:27:05Z level=info event=scheduler_connected addr=lb:500514209worker1 # [ 20.668699] nix-grpc-daemon[781]: ts=2026-09-21T09:27:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4210worker2 # [ 20.773737] nix-grpc-daemon[786]: ts=2026-09-21T09:27:05Z level=info event=scheduler_connected addr=lb:500514211worker2 # [ 20.777517] nix-grpc-daemon[786]: ts=2026-09-21T09:27:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4212worker1 # [ 20.869776] nix-grpc-daemon[781]: ts=2026-09-21T09:27:06Z level=info event=scheduler_connected addr=lb:500514213worker1 # [ 20.873762] nix-grpc-daemon[781]: ts=2026-09-21T09:27:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4214worker2 # [ 20.978471] nix-grpc-daemon[786]: ts=2026-09-21T09:27:05Z level=info event=scheduler_connected addr=lb:500514215worker2 # [ 20.982328] nix-grpc-daemon[786]: ts=2026-09-21T09:27:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4216worker1 # [ 21.075693] nix-grpc-daemon[781]: ts=2026-09-21T09:27:06Z level=info event=scheduler_connected addr=lb:500514217worker1 # [ 21.079475] nix-grpc-daemon[781]: ts=2026-09-21T09:27:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4218worker2 # [ 21.183650] nix-grpc-daemon[786]: ts=2026-09-21T09:27:05Z level=info event=scheduler_connected addr=lb:500514219worker2 # [ 21.187449] nix-grpc-daemon[786]: ts=2026-09-21T09:27:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4220worker1 # [ 21.280456] nix-grpc-daemon[781]: ts=2026-09-21T09:27:06Z level=info event=scheduler_connected addr=lb:500514221worker1 # [ 21.284470] nix-grpc-daemon[781]: ts=2026-09-21T09:27:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4222worker2 # [ 21.388450] nix-grpc-daemon[786]: ts=2026-09-21T09:27:05Z level=info event=scheduler_connected addr=lb:500514223worker2 # [ 21.392269] nix-grpc-daemon[786]: ts=2026-09-21T09:27:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4224worker1 # [ 21.485518] nix-grpc-daemon[781]: ts=2026-09-21T09:27:06Z level=info event=scheduler_connected addr=lb:500514225worker1 # [ 21.489535] nix-grpc-daemon[781]: ts=2026-09-21T09:27:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4226worker2 # [ 21.592898] nix-grpc-daemon[786]: ts=2026-09-21T09:27:05Z level=info event=scheduler_connected addr=lb:500514227worker2 # [ 21.596780] nix-grpc-daemon[786]: ts=2026-09-21T09:27:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4228worker1 # [ 21.690513] nix-grpc-daemon[781]: ts=2026-09-21T09:27:06Z level=info event=scheduler_connected addr=lb:500514229worker1 # [ 21.694428] nix-grpc-daemon[781]: ts=2026-09-21T09:27:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4230worker2 # [ 21.798841] nix-grpc-daemon[786]: ts=2026-09-21T09:27:06Z level=info event=scheduler_connected addr=lb:500514231worker2 # [ 21.802597] nix-grpc-daemon[786]: ts=2026-09-21T09:27:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4232worker1 # [ 21.895327] nix-grpc-daemon[781]: ts=2026-09-21T09:27:07Z level=info event=scheduler_connected addr=lb:500514233worker1 # [ 21.899344] nix-grpc-daemon[781]: ts=2026-09-21T09:27:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4234worker2 # [ 22.003522] nix-grpc-daemon[786]: ts=2026-09-21T09:27:06Z level=info event=scheduler_connected addr=lb:500514235worker2 # [ 22.007491] nix-grpc-daemon[786]: ts=2026-09-21T09:27:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4236worker1 # [ 22.101635] nix-grpc-daemon[781]: ts=2026-09-21T09:27:07Z level=info event=scheduler_connected addr=lb:500514237worker1 # [ 22.104878] nix-grpc-daemon[781]: ts=2026-09-21T09:27:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4238worker2 # [ 22.208575] nix-grpc-daemon[786]: ts=2026-09-21T09:27:06Z level=info event=scheduler_connected addr=lb:500514239worker2 # [ 22.212389] nix-grpc-daemon[786]: ts=2026-09-21T09:27:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4240worker1 # [ 22.302782] nix-grpc-daemon[781]: ts=2026-09-21T09:27:07Z level=info event=scheduler_connected addr=lb:500514241worker1 # [ 22.306751] nix-grpc-daemon[781]: ts=2026-09-21T09:27:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4242worker2 # [ 22.413568] nix-grpc-daemon[786]: ts=2026-09-21T09:27:06Z level=info event=scheduler_connected addr=lb:500514243worker2 # [ 22.417418] nix-grpc-daemon[786]: ts=2026-09-21T09:27:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4244worker1 # [ 22.507592] nix-grpc-daemon[781]: ts=2026-09-21T09:27:07Z level=info event=scheduler_connected addr=lb:500514245worker1 # [ 22.511898] nix-grpc-daemon[781]: ts=2026-09-21T09:27:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4246worker2 # [ 22.618903] nix-grpc-daemon[786]: ts=2026-09-21T09:27:06Z level=info event=scheduler_connected addr=lb:500514247worker2 # [ 22.621965] nix-grpc-daemon[786]: ts=2026-09-21T09:27:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4248worker1 # [ 22.712694] nix-grpc-daemon[781]: ts=2026-09-21T09:27:07Z level=info event=scheduler_connected addr=lb:500514249worker1 # [ 22.716538] nix-grpc-daemon[781]: ts=2026-09-21T09:27:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4250worker2 # [ 22.824370] nix-grpc-daemon[786]: ts=2026-09-21T09:27:07Z level=info event=scheduler_connected addr=lb:500514251worker2 # [ 22.827474] nix-grpc-daemon[786]: ts=2026-09-21T09:27:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4252worker1 # [ 22.917082] nix-grpc-daemon[781]: ts=2026-09-21T09:27:08Z level=info event=scheduler_connected addr=lb:500514253worker1 # [ 22.919969] nix-grpc-daemon[781]: ts=2026-09-21T09:27:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4254worker2 # [ 23.025369] nix-grpc-daemon[786]: ts=2026-09-21T09:27:07Z level=info event=scheduler_connected addr=lb:500514255worker2 # [ 23.029220] nix-grpc-daemon[786]: ts=2026-09-21T09:27:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4256worker1 # [ 23.121705] nix-grpc-daemon[781]: ts=2026-09-21T09:27:08Z level=info event=scheduler_connected addr=lb:500514257worker1 # [ 23.125488] nix-grpc-daemon[781]: ts=2026-09-21T09:27:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4258worker2 # [ 23.230390] nix-grpc-daemon[786]: ts=2026-09-21T09:27:07Z level=info event=scheduler_connected addr=lb:500514259worker2 # [ 23.234225] nix-grpc-daemon[786]: ts=2026-09-21T09:27:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4260worker1 # [ 23.323758] nix-grpc-daemon[781]: ts=2026-09-21T09:27:08Z level=info event=scheduler_connected addr=lb:500514261worker1 # [ 23.327622] nix-grpc-daemon[781]: ts=2026-09-21T09:27:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4262worker2 # [ 23.435257] nix-grpc-daemon[786]: ts=2026-09-21T09:27:07Z level=info event=scheduler_connected addr=lb:500514263worker2 # [ 23.439276] nix-grpc-daemon[786]: ts=2026-09-21T09:27:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4264worker1 # [ 23.528509] nix-grpc-daemon[781]: ts=2026-09-21T09:27:08Z level=info event=scheduler_connected addr=lb:500514265worker1 # [ 23.532403] nix-grpc-daemon[781]: ts=2026-09-21T09:27:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4266worker2 # [ 23.640077] nix-grpc-daemon[786]: ts=2026-09-21T09:27:07Z level=info event=scheduler_connected addr=lb:500514267worker2 # [ 23.643870] nix-grpc-daemon[786]: ts=2026-09-21T09:27:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4268worker1 # [ 23.733304] nix-grpc-daemon[781]: ts=2026-09-21T09:27:08Z level=info event=scheduler_connected addr=lb:500514269worker1 # [ 23.737323] nix-grpc-daemon[781]: ts=2026-09-21T09:27:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4270lb # [ 23.796943] rustfs-bucket-start[994]: mb s3://farm4271lb # [ 23.802334] systemd[1]: Finished rustfs-bucket.service.4272lb # [ 23.808347] systemd[1]: Starting niks3 server...4273worker2 # [ 23.845649] nix-grpc-daemon[786]: ts=2026-09-21T09:27:08Z level=info event=scheduler_connected addr=lb:500514274worker2 # [ 23.849374] nix-grpc-daemon[786]: ts=2026-09-21T09:27:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4275worker1 # [ 23.938119] nix-grpc-daemon[781]: ts=2026-09-21T09:27:09Z level=info event=scheduler_connected addr=lb:500514276worker1 # [ 23.942214] nix-grpc-daemon[781]: ts=2026-09-21T09:27:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4277worker2 # [ 24.050363] nix-grpc-daemon[786]: ts=2026-09-21T09:27:08Z level=info event=scheduler_connected addr=lb:500514278worker2 # [ 24.054144] nix-grpc-daemon[786]: ts=2026-09-21T09:27:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4279lb # [ 24.064884] postgres[1011]: [1011] ERROR: relation "goose_db_version" does not exist at character 364280lb # [ 24.066917] postgres[1011]: [1011] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4281lb # [ 24.111893] niks3-server[1002]: 2026/09/21 09:27:09 OK 20241026095416_initial_model.sql (21.3ms)4282lb # [ 24.115457] niks3-server[1002]: 2026/09/21 09:27:09 OK 20251210153512_drop_unused_gin_index.sql (3.6ms)4283lb # [ 24.120292] niks3-server[1002]: 2026/09/21 09:27:09 OK 20251218171726_add_pins.sql (4.81ms)4284lb # [ 24.125555] niks3-server[1002]: 2026/09/21 09:27:09 OK 20260628120000_add_object_size_and_stats.sql (5.23ms)4285lb # [ 24.130113] niks3-server[1002]: 2026/09/21 09:27:09 OK 20260905000000_add_claims.sql (4.61ms)4286lb # [ 24.134602] niks3-server[1002]: 2026/09/21 09:27:09 OK 20260920000000_drop_claims.sql (3.25ms)4287lb # [ 24.136080] niks3-server[1002]: 2026/09/21 09:27:09 goose: successfully migrated database to version: 202609200000004288lb # [ 24.142181] niks3-server[1002]: 2026/09/21 09:27:09 OK 1_commit_pending_closure.sql (6.1ms)4289lb # [ 24.144721] niks3-server[1002]: 2026/09/21 09:27:09 OK 2_object_stats_trigger.sql (2.57ms)4290lb # [ 24.145907] niks3-server[1002]: 2026/09/21 09:27:09 goose: up to current file version: 24291lb # [ 24.152834] niks3-server[1002]: 2026/09/21 09:27:09 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key4292worker1 # [ 24.144898] nix-grpc-daemon[781]: ts=2026-09-21T09:27:09Z level=info event=scheduler_connected addr=lb:500514293worker1 # [ 24.148118] nix-grpc-daemon[781]: ts=2026-09-21T09:27:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4294worker2 # [ 24.255202] nix-grpc-daemon[786]: ts=2026-09-21T09:27:08Z level=info event=scheduler_connected addr=lb:500514295worker2 # [ 24.258989] nix-grpc-daemon[786]: ts=2026-09-21T09:27:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4296lb # [ 24.245582] niks3-server[1002]: 2026/09/21 09:27:09 INFO Read proxy enabled — serving cache objects from S34297lb # [ 24.247638] niks3-server[1002]: 2026/09/21 09:27:09 INFO Using socket-activated listener address=0.0.0.0:57514298lb # [ 24.254097] niks3-server[1002]: 2026/09/21 09:27:09 INFO systemd watchdog enabled interval=15s4299lb # [ 24.255265] systemd[1]: Started niks3 server.4300lb # [ 24.255885] systemd[1]: Reached target Multi-User System.4301lb # [ 24.258181] systemd[1]: Startup finished in 867ms (kernel) + 4.050s (initrd) + 19.340s (userspace) = 24.258s.4302lb # [ 24.260137] niks3-server[1002]: 2026/09/21 09:27:09 INFO Starting HTTP server address=0.0.0.0:57514303lb # [ 24.266317] niks3-server[1002]: 2026/09/21 09:27:09 INFO lead: acquired remote=192.168.1.4:339724304worker2 # [ 24.303142] nix-grpc-daemon[786]: ts=2026-09-21T09:27:08Z level=info event=scheduler_take_over4305worker1 # [ 24.345544] nix-grpc-daemon[781]: ts=2026-09-21T09:27:09Z level=info event=scheduler_connected addr=lb:500514306worker1 # [ 24.349514] nix-grpc-daemon[781]: ts=2026-09-21T09:27:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4307worker2 # [ 24.460074] nix-grpc-daemon[786]: ts=2026-09-21T09:27:08Z level=info event=scheduler_connected addr=lb:500514308worker2 # [ 24.463886] nix-grpc-daemon[786]: ts=2026-09-21T09:27:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4309worker1 # [ 24.550625] nix-grpc-daemon[781]: ts=2026-09-21T09:27:09Z level=info event=scheduler_connected addr=lb:500514310worker1 # [ 24.554567] nix-grpc-daemon[781]: ts=2026-09-21T09:27:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4311worker2 # [ 24.664590] nix-grpc-daemon[786]: ts=2026-09-21T09:27:09Z level=info event=scheduler_connected addr=lb:500514312worker2 # [ 24.699394] nix-grpc-daemon[786]: ts=2026-09-21T09:27:09Z level=info event=oidc_jwks provider=mock keys=14313worker2 # [ 24.704149] nix-grpc-daemon[786]: ts=2026-09-21T09:27:09Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:476104314worker2 # [ 24.709827] nix-grpc-daemon[786]: ts=2026-09-21T09:27:09Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04315worker1 # [ 24.755408] nix-grpc-daemon[781]: ts=2026-09-21T09:27:09Z level=info event=scheduler_connected addr=lb:500514316worker2 # [ 24.795145] nix-grpc-daemon[786]: ts=2026-09-21T09:27:09Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:476104317worker2 # [ 24.799943] nix-grpc-daemon[786]: ts=2026-09-21T09:27:09Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04318lb: (finished: waiting for unit niks3.service, in 14.18 seconds)4319lb: waiting for TCP port 5751 on localhost4320lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!4321lb: (finished: waiting for TCP port 5751 on localhost, in 0.08 seconds)4322lb: waiting for unit envoy.service4323lb: (finished: waiting for unit envoy.service, in 0.06 seconds)4324lb: waiting for TCP port 50051 on localhost4325lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!4326lb # [ 25.373481] envoy[902]: conn peer=[::1]:49624 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=34327lb: (finished: waiting for TCP port 50051 on localhost, in 0.04 seconds)4328worker2 # [ 25.547634] systemd[1]: Starting scheduler-token.service...4329worker2 # [ 25.624915] systemd[1]: scheduler-token.service: Deactivated successfully.4330worker2 # [ 25.627206] systemd[1]: Finished scheduler-token.service.4331worker2 # [ 25.627947] systemd[1]: scheduler-token.service: Consumed 20ms CPU time over 79ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4332lb: waiting until exactly no endpoints of x86_64-linux are unhealthy4333lb: must succeed: curl -sf localhost:9901/clusters4334lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)4335lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.10 seconds)4336subtest: one node schedules and both builders hold a WorkerSession on it4337lb: must succeed: curl -sf localhost:9901/clusters4338lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4339worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4340worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4341lb: must succeed: curl -sf localhost:9901/clusters4342lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4343lb: must succeed: curl -sf localhost:9901/clusters4344lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4345lb: must succeed: curl -sf localhost:9901/clusters4346lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4347lb: must succeed: curl -sf localhost:9901/clusters4348lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4349worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4350worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.05 seconds)4351worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4352worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4353lb: must succeed: curl -sf localhost:9901/clusters4354lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4355lb: must succeed: curl -sf localhost:9901/clusters4356lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4357worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4358worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)4359worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4360worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4361worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4362worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4363worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4364worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4365(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.76 seconds)4366subtest: balancer auth: client cert, bearer token, nothing, unknown CN4367worker2 # [ 26.899460] nix-grpc-daemon[786]: ts=2026-09-21T09:27:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:476224368worker2 # [ 26.913868] systemd[1]: Started Nix Daemon.4369worker2 # [ 27.146857] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4370worker2 # [ 27.173212] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 18 ms (attempt 1/5)4371worker2 # [ 27.193677] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 87 ms (attempt 2/5)4372worker2 # [ 27.280841] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 384 ms (attempt 3/5)4373worker2 # [ 27.670162] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 93 ms (attempt 4/5)4374worker2 # [ 27.772966] nix-grpc-daemon[786]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4375worker2 # [ 27.777446] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4376worker1 # [ 27.789225] nix-grpc-daemon[781]: ts=2026-09-21T09:27:12Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:569564377worker1 # [ 27.803284] systemd[1]: Started Nix Daemon.4378worker1 # [ 28.058041] nix-daemon[845]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4379worker1 # [ 28.082308] nix-grpc-daemon[781]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 32 ms (attempt 1/5)4380worker1 # [ 28.116860] nix-grpc-daemon[781]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 114 ms (attempt 2/5)4381worker1 # [ 28.231441] nix-grpc-daemon[781]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 286 ms (attempt 3/5)4382worker1 # [ 28.523509] nix-grpc-daemon[781]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 416 ms (attempt 4/5)4383worker1 # [ 28.949997] nix-grpc-daemon[781]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4384worker1 # [ 28.954526] nix-grpc-daemon[781]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4385worker1 # [ 28.974979] nix-grpc-daemon[781]: ts=2026-09-21T09:27:14Z level=debug event=path_info_miss path=0v4f337qkyj5dmshnqw4pzwr5c8f8pgy-nixos-system-client-test error="error: path '/nix/store/0v4f337qkyj5dmshnqw4pzwr5c8f8pgy-nixos-system-client-test' is required, but there is no substituter that can build it"4386worker1 # [ 28.988771] nix-grpc-daemon[781]: ts=2026-09-21T09:27:14Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:569564387worker1 # [ 28.998250] nix-grpc-daemon[781]: ts=2026-09-21T09:27:14Z level=debug event=path_info_miss path=0v4f337qkyj5dmshnqw4pzwr5c8f8pgy-nixos-system-client-test error="error: path '/nix/store/0v4f337qkyj5dmshnqw4pzwr5c8f8pgy-nixos-system-client-test' is required, but there is no substituter that can build it"4388client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4389client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.10 seconds)4390client: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4391client: (finished: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4392worker2 # [ 29.476413] nix-grpc-daemon[786]: ts=2026-09-21T09:27:13Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:476224393worker1 # [ 29.466840] nix-grpc-daemon[781]: ts=2026-09-21T09:27:14Z level=info event=oidc_jwks provider=mock keys=14394worker1 # [ 29.471596] nix-grpc-daemon[781]: ts=2026-09-21T09:27:14Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:569564395worker1 # [ 29.480743] nix-grpc-daemon[781]: ts=2026-09-21T09:27:14Z level=debug event=path_info_miss path=0v4f337qkyj5dmshnqw4pzwr5c8f8pgy-nixos-system-client-test error="error: path '/nix/store/0v4f337qkyj5dmshnqw4pzwr5c8f8pgy-nixos-system-client-test' is required, but there is no substituter that can build it"4396worker2 # [ 29.524843] nix-grpc-daemon[786]: ts=2026-09-21T09:27:13Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:476224397worker2 # [ 29.543121] nix-grpc-daemon[786]: ts=2026-09-21T09:27:13Z level=debug event=path_info_miss path=0v4f337qkyj5dmshnqw4pzwr5c8f8pgy-nixos-system-client-test error="error: path '/nix/store/0v4f337qkyj5dmshnqw4pzwr5c8f8pgy-nixos-system-client-test' is required, but there is no substituter that can build it"4398client: must succeed: rm -r /var/lib/nix-grpc-store4399client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds)4400worker1 # [ 29.808735] nix-grpc-daemon[781]: ts=2026-09-21T09:27:14Z level=info event=denied method=QueryMissing cn=- role=none4401worker2 # [ 30.112752] nix-grpc-daemon[786]: ts=2026-09-21T09:27:14Z level=info event=denied method=QueryMissing cn=stranger role=none4402(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.65 seconds)4403subtest: DAG build is scheduled across workers and lands in the cache4404client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24405worker2 # [ 30.385969] nix-grpc-daemon[786]: ts=2026-09-21T09:27:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:476224406client # don't know how to build these paths:4407client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4408worker1 # [ 30.363987] nix-grpc-daemon[781]: ts=2026-09-21T09:27:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:569564409worker2 # [ 30.439911] nix-grpc-daemon[786]: ts=2026-09-21T09:27:14Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04410worker2 # [ 30.444646] nix-grpc-daemon[786]: ts=2026-09-21T09:27:14Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.4:50051 clients=14411worker2 # [ 30.508759] nix-grpc-daemon[786]: ts=2026-09-21T09:27:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224412worker2 # [ 30.529314] nix-grpc-daemon[786]: ts=2026-09-21T09:27:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:476224413client # copying 1 paths to 'lb:50051'...4414worker2 # [ 30.542732] nix-grpc-daemon[786]: ts=2026-09-21T09:27:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:476224415client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4416worker2 # [ 30.567453] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4417worker2 # [ 30.595697] nix-grpc-daemon[786]: ts=2026-09-21T09:27:14Z level=info event=niks3_push_started pid=8844418worker2 # [ 30.907686] nix-grpc-daemon[884]: time=2026-09-21T09:27:15.263Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""4419worker2 # [ 31.126074] nix-daemon[869]: accepted connection from pid 897, user nix-grpc-daemon (trusted)4420worker2 # [ 31.140593] nix-daemon[869]: reaped child process 900, status = succeeded4421worker2 # [ 31.297629] nix-daemon[869]: accepted connection from pid 902, user nix-grpc-daemon (trusted)4422worker2 # [ 31.309660] nix-daemon[869]: reaped child process 905, status = succeeded4423lb # [ 31.292293] niks3-server[1002]: 2026/09/21 09:27:16 INFO Received uploads request method=POST path=/api/pending_closures4424worker2 # [ 31.350642] nix-grpc-daemon[884]: time=2026-09-21T09:27:15.709Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4425worker2 # [ 31.353343] nix-grpc-daemon[884]: time=2026-09-21T09:27:15.712Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4426lb # [ 31.371854] niks3-server[1002]: 2026/09/21 09:27:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4427lb # [ 31.387248] niks3-server[1002]: 2026/09/21 09:27:16 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4428lb # [ 31.395630] niks3-server[1002]: 2026/09/21 09:27:16 INFO Signed narinfos id=1 count=14429worker2 # [ 31.428185] nix-grpc-daemon[884]: time=2026-09-21T09:27:15.787Z level=INFO msg="Uploading 1 narinfos"4430lb # [ 31.404183] niks3-server[1002]: 2026/09/21 09:27:16 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4431lb # [ 31.410883] niks3-server[1002]: 2026/09/21 09:27:16 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4432lb # [ 31.421713] niks3-server[1002]: 2026/09/21 09:27:16 INFO Completed upload id=14433worker2 # [ 31.452685] nix-grpc-daemon[884]: time=2026-09-21T09:27:15.811Z level=INFO msg="Upload complete. (518ms)"4434lb # [ 31.426482] niks3-server[1002]: 2026/09/21 09:27:16 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4435worker2 # [ 31.457794] nix-grpc-daemon[786]: ts=2026-09-21T09:27:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=0 paths=1 nar_bytes_in=5044436worker2 # [ 31.464703] nix-grpc-daemon[786]: ts=2026-09-21T09:27:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224437worker2 # [ 31.478913] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4438worker2 # [ 31.483378] nix-daemon[869]: reaped child process 882, status = succeeded4439client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4440worker2 # [ 31.492904] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4441client # farm-a> worker2: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4442client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4443client # farm-a> LOG-farm-a-t14444worker2 # [ 32.307435] nix-daemon[869]: reaped child process 909, status = succeeded4445worker2 # [ 32.467088] nix-daemon[869]: accepted connection from pid 926, user nix-grpc-daemon (trusted)4446worker2 # [ 32.477631] nix-daemon[869]: reaped child process 929, status = succeeded4447lb # [ 32.464900] niks3-server[1002]: 2026/09/21 09:27:17 INFO Received uploads request method=POST path=/api/pending_closures4448worker2 # [ 32.508173] nix-grpc-daemon[884]: time=2026-09-21T09:27:16.867Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4449worker2 # [ 32.509874] nix-grpc-daemon[884]: time=2026-09-21T09:27:16.868Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4450lb # [ 32.513294] niks3-server[1002]: 2026/09/21 09:27:17 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4451lb # [ 32.520447] niks3-server[1002]: 2026/09/21 09:27:17 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4452lb # [ 32.529052] niks3-server[1002]: 2026/09/21 09:27:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4453lb # [ 32.534051] niks3-server[1002]: 2026/09/21 09:27:17 INFO Signed narinfos id=2 count=14454worker2 # [ 32.565139] nix-grpc-daemon[884]: time=2026-09-21T09:27:16.923Z level=INFO msg="Uploading 1 narinfos"4455lb # [ 32.537264] niks3-server[1002]: 2026/09/21 09:27:17 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4456lb # [ 32.543346] niks3-server[1002]: 2026/09/21 09:27:17 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4457lb # [ 32.546682] niks3-server[1002]: 2026/09/21 09:27:17 INFO Completed upload id=24458worker2 # [ 32.577393] nix-grpc-daemon[884]: time=2026-09-21T09:27:16.936Z level=INFO msg="Upload complete. (271ms)"4459lb # [ 32.548409] niks3-server[1002]: 2026/09/21 09:27:17 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4460worker2 # [ 32.583755] nix-grpc-daemon[786]: ts=2026-09-21T09:27:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14461worker2 # [ 32.597745] nix-daemon[869]: reaped child process 907, status = succeeded4462worker2 # [ 32.602361] nix-grpc-daemon[786]: ts=2026-09-21T09:27:16Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04463worker2 # [ 32.604347] nix-grpc-daemon[786]: ts=2026-09-21T09:27:16Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.4:50051 clients=14464worker2 # [ 32.649207] nix-grpc-daemon[786]: ts=2026-09-21T09:27:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224465worker2 # [ 32.661286] nix-grpc-daemon[786]: ts=2026-09-21T09:27:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:476224466worker2 # [ 32.674905] nix-grpc-daemon[884]: time=2026-09-21T09:27:17.033Z level=INFO msg="All 1 paths already cached"4467client # copying 1 paths to 'lb:50051'...4468worker2 # [ 32.682944] nix-grpc-daemon[786]: ts=2026-09-21T09:27:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:476224469client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4470worker2 # [ 32.701816] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4471worker2 # [ 32.887315] nix-daemon[869]: accepted connection from pid 935, user nix-grpc-daemon (trusted)4472worker2 # [ 32.899735] nix-daemon[869]: reaped child process 938, status = succeeded4473worker2 # [ 33.062223] nix-daemon[869]: accepted connection from pid 940, user nix-grpc-daemon (trusted)4474worker2 # [ 33.073525] nix-daemon[869]: reaped child process 943, status = succeeded4475lb # [ 33.060556] niks3-server[1002]: 2026/09/21 09:27:18 INFO Received uploads request method=POST path=/api/pending_closures4476worker2 # [ 33.101073] nix-grpc-daemon[884]: time=2026-09-21T09:27:17.460Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4477worker2 # [ 33.102695] nix-grpc-daemon[884]: time=2026-09-21T09:27:17.461Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4478lb # [ 33.094717] niks3-server[1002]: 2026/09/21 09:27:18 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4479lb # [ 33.102894] niks3-server[1002]: 2026/09/21 09:27:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4480worker2 # [ 33.138439] nix-grpc-daemon[884]: time=2026-09-21T09:27:17.497Z level=INFO msg="Uploading 1 narinfos"4481lb # [ 33.109356] niks3-server[1002]: 2026/09/21 09:27:18 INFO Signed narinfos id=3 count=14482lb # [ 33.114334] niks3-server[1002]: 2026/09/21 09:27:18 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4483lb # [ 33.124131] niks3-server[1002]: 2026/09/21 09:27:18 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4484lb # [ 33.131117] niks3-server[1002]: 2026/09/21 09:27:18 INFO Completed upload id=34485worker2 # [ 33.161726] nix-grpc-daemon[884]: time=2026-09-21T09:27:17.520Z level=INFO msg="Upload complete. (445ms)"4486lb # [ 33.133572] niks3-server[1002]: 2026/09/21 09:27:18 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4487worker2 # [ 33.165484] nix-grpc-daemon[786]: ts=2026-09-21T09:27:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=0 paths=1 nar_bytes_in=6244488worker2 # [ 33.170785] nix-grpc-daemon[786]: ts=2026-09-21T09:27:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224489worker2 # [ 33.181895] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4490worker2 # [ 33.184668] nix-daemon[869]: reaped child process 933, status = succeeded4491worker2 # [ 33.190150] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4492client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4493client # farm-b> worker2: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4494client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4495client # farm-b> LOG-farm-b-t14496worker2 # [ 34.404395] nix-daemon[869]: reaped child process 947, status = succeeded4497worker2 # [ 34.556612] nix-daemon[869]: accepted connection from pid 963, user nix-grpc-daemon (trusted)4498worker2 # [ 34.566870] nix-daemon[869]: reaped child process 966, status = succeeded4499lb # [ 34.554831] niks3-server[1002]: 2026/09/21 09:27:19 INFO Received uploads request method=POST path=/api/pending_closures4500worker2 # [ 34.594029] nix-grpc-daemon[884]: time=2026-09-21T09:27:18.952Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4501worker2 # [ 34.595489] nix-grpc-daemon[884]: time=2026-09-21T09:27:18.954Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4502lb # [ 34.586844] niks3-server[1002]: 2026/09/21 09:27:19 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4503lb # [ 34.598721] niks3-server[1002]: 2026/09/21 09:27:19 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4504lb # [ 34.611995] niks3-server[1002]: 2026/09/21 09:27:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4505lb # [ 34.615334] niks3-server[1002]: 2026/09/21 09:27:19 INFO Signed narinfos id=4 count=14506worker2 # [ 34.648884] nix-grpc-daemon[884]: time=2026-09-21T09:27:19.007Z level=INFO msg="Uploading 1 narinfos"4507lb # [ 34.620858] niks3-server[1002]: 2026/09/21 09:27:19 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4508lb # [ 34.629113] niks3-server[1002]: 2026/09/21 09:27:19 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4509lb # [ 34.634993] niks3-server[1002]: 2026/09/21 09:27:19 INFO Completed upload id=44510worker2 # [ 34.665829] nix-grpc-daemon[884]: time=2026-09-21T09:27:19.024Z level=INFO msg="Upload complete. (268ms)"4511lb # [ 34.636718] niks3-server[1002]: 2026/09/21 09:27:19 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4512worker2 # [ 34.672882] nix-grpc-daemon[786]: ts=2026-09-21T09:27:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14513worker2 # [ 34.683873] nix-grpc-daemon[786]: ts=2026-09-21T09:27:19Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04514worker2 # [ 34.689674] nix-grpc-daemon[786]: ts=2026-09-21T09:27:19Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.4:50051 clients=14515worker2 # [ 34.697337] nix-daemon[869]: reaped child process 945, status = succeeded4516worker2 # [ 34.728350] nix-grpc-daemon[786]: ts=2026-09-21T09:27:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224517worker2 # [ 34.739921] nix-grpc-daemon[786]: ts=2026-09-21T09:27:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:476224518worker2 # [ 34.753453] nix-grpc-daemon[884]: time=2026-09-21T09:27:19.112Z level=INFO msg="All 2 paths already cached"4519client # copying 1 paths to 'lb:50051'...4520client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4521worker2 # [ 34.761802] nix-grpc-daemon[786]: ts=2026-09-21T09:27:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:476224522worker2 # [ 34.776305] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4523worker2 # [ 34.955372] nix-daemon[869]: accepted connection from pid 972, user nix-grpc-daemon (trusted)4524worker2 # [ 34.967411] nix-daemon[869]: reaped child process 975, status = succeeded4525worker2 # [ 35.130396] nix-daemon[869]: accepted connection from pid 977, user nix-grpc-daemon (trusted)4526worker2 # [ 35.142349] nix-daemon[869]: reaped child process 980, status = succeeded4527lb # [ 35.128735] niks3-server[1002]: 2026/09/21 09:27:20 INFO Received uploads request method=POST path=/api/pending_closures4528worker2 # [ 35.169815] nix-grpc-daemon[884]: time=2026-09-21T09:27:19.528Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4529worker2 # [ 35.171331] nix-grpc-daemon[884]: time=2026-09-21T09:27:19.530Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4530lb # [ 35.161784] niks3-server[1002]: 2026/09/21 09:27:20 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4531lb # [ 35.172534] niks3-server[1002]: 2026/09/21 09:27:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4532worker2 # [ 35.207948] nix-grpc-daemon[884]: time=2026-09-21T09:27:19.566Z level=INFO msg="Uploading 1 narinfos"4533lb # [ 35.177519] niks3-server[1002]: 2026/09/21 09:27:20 INFO Signed narinfos id=5 count=14534lb # [ 35.184543] niks3-server[1002]: 2026/09/21 09:27:20 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4535lb # [ 35.195104] niks3-server[1002]: 2026/09/21 09:27:20 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4536lb # [ 35.201619] niks3-server[1002]: 2026/09/21 09:27:20 INFO Completed upload id=54537worker2 # [ 35.234457] nix-grpc-daemon[884]: time=2026-09-21T09:27:19.593Z level=INFO msg="Upload complete. (446ms)"4538lb # [ 35.205161] niks3-server[1002]: 2026/09/21 09:27:20 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4539worker2 # [ 35.238106] nix-grpc-daemon[786]: ts=2026-09-21T09:27:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=0 paths=1 nar_bytes_in=7524540worker2 # [ 35.244288] nix-grpc-daemon[786]: ts=2026-09-21T09:27:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224541worker2 # [ 35.253136] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4542worker2 # [ 35.262664] nix-daemon[869]: reaped child process 970, status = succeeded4543worker2 # [ 35.265928] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4544client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4545client # farm-top> worker2: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4546lb # [ 36.196715] envoy[902]: conn peer=[2001:db8:1::1]:33992 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=407 ms=21894547lb # [ 36.201356] envoy[902]: conn peer=[2001:db8:1::1]:33998 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=804548lb # [ 36.204687] envoy[902]: conn peer=[2001:db8:1::1]:34000 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=154549lb # [ 36.208304] envoy[902]: conn peer=[2001:db8:1::1]:34010 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=94550client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4551client # farm-top> LOG-farm-top-t14552worker2 # [ 36.890727] nix-daemon[869]: reaped child process 984, status = succeeded4553worker2 # [ 37.072300] nix-daemon[869]: accepted connection from pid 1001, user nix-grpc-daemon (trusted)4554worker2 # [ 37.082879] nix-daemon[869]: reaped child process 1004, status = succeeded4555lb # [ 37.070788] niks3-server[1002]: 2026/09/21 09:27:22 INFO Received uploads request method=POST path=/api/pending_closures4556worker2 # [ 37.111718] nix-grpc-daemon[884]: time=2026-09-21T09:27:21.470Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4557worker2 # [ 37.114565] nix-grpc-daemon[884]: time=2026-09-21T09:27:21.472Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4558lb # [ 37.111928] niks3-server[1002]: 2026/09/21 09:27:22 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4559lb # [ 37.123372] niks3-server[1002]: 2026/09/21 09:27:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4560worker2 # [ 37.162852] nix-grpc-daemon[884]: time=2026-09-21T09:27:21.521Z level=INFO msg="Uploading 1 narinfos"4561lb # [ 37.131995] niks3-server[1002]: 2026/09/21 09:27:22 INFO Signed narinfos id=6 count=14562lb # [ 37.136457] niks3-server[1002]: 2026/09/21 09:27:22 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4563lb # [ 37.141965] niks3-server[1002]: 2026/09/21 09:27:22 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4564lb # [ 37.148764] niks3-server[1002]: 2026/09/21 09:27:22 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4565lb # [ 37.152917] niks3-server[1002]: 2026/09/21 09:27:22 INFO Completed upload id=64566lb # [ 37.155056] niks3-server[1002]: 2026/09/21 09:27:22 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4567worker2 # [ 37.184974] nix-grpc-daemon[884]: time=2026-09-21T09:27:21.543Z level=INFO msg="Upload complete. (308ms)"4568worker2 # [ 37.193963] nix-grpc-daemon[786]: ts=2026-09-21T09:27:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14569worker2 # [ 37.206485] nix-daemon[869]: reaped child process 982, status = succeeded4570client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 7.10 seconds)4571client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4572client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.22 seconds)4573client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4574client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.02 seconds)4575client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4576client # copying 3 paths...4577client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4578client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4579client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4580client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.30 seconds)4581(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.74 seconds)4582subtest: repeat is answered Cached by the scheduler without building4583worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4584worker1 # finding garbage collector roots...4585worker1 # deleting specified paths...4586worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4587worker1 # deleting unused links...4588worker1 # note: hard linking is currently saving 0.0 KiB4589worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.32 seconds)4590worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4591worker2 # finding garbage collector roots...4592worker2 # deleting specified paths...4593worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4594worker2 # deleting unused links...4595worker2 # note: hard linking is currently saving 0.0 KiB4596worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.27 seconds)4597client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24598worker2 # [ 38.747363] nix-grpc-daemon[786]: ts=2026-09-21T09:27:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:476224599client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4600client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4601worker2 # [ 38.767182] nix-grpc-daemon[786]: ts=2026-09-21T09:27:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:476224602worker2 # [ 38.779964] nix-grpc-daemon[884]: time=2026-09-21T09:27:23.138Z level=INFO msg="All 2 paths already cached"4603worker2 # [ 38.795729] nix-grpc-daemon[786]: ts=2026-09-21T09:27:23Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14604client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.36 seconds)4605client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4606client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.21 seconds)4607lb: must succeed: curl -sf localhost:9901/clusters4608lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4609lb: must succeed: curl -sf localhost:9901/clusters4610lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4611worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4612worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4613(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.44 seconds)4614subtest: inputs already in the cache are not sent to the scheduler4615lb: must succeed: curl -sf localhost:9901/clusters4616lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4617lb: must succeed: curl -sf localhost:9901/clusters4618lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4619worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4620worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4621client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24622worker2 # [ 39.735805] nix-grpc-daemon[786]: ts=2026-09-21T09:27:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:476224623client # don't know how to build these paths:4624client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4625worker2 # [ 39.744528] nix-grpc-daemon[786]: ts=2026-09-21T09:27:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:476224626worker2 # [ 39.755491] nix-grpc-daemon[884]: time=2026-09-21T09:27:24.114Z level=INFO msg="All 2 paths already cached"4627worker2 # [ 39.769649] nix-grpc-daemon[786]: ts=2026-09-21T09:27:24Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04628worker2 # [ 39.773628] nix-grpc-daemon[786]: ts=2026-09-21T09:27:24Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.4:50051 clients=14629worker2 # [ 39.812557] nix-grpc-daemon[786]: ts=2026-09-21T09:27:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224630worker2 # [ 39.825733] nix-grpc-daemon[786]: ts=2026-09-21T09:27:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:476224631worker2 # [ 39.838843] nix-grpc-daemon[884]: time=2026-09-21T09:27:24.197Z level=INFO msg="All 2 paths already cached"4632client # copying 1 paths to 'lb:50051'...4633client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4634worker2 # [ 39.847511] nix-grpc-daemon[786]: ts=2026-09-21T09:27:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:476224635worker2 # [ 39.858638] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4636worker2 # [ 40.054501] nix-daemon[869]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)4637worker2 # [ 40.067234] nix-daemon[869]: reaped child process 1037, status = succeeded4638worker2 # [ 40.234310] nix-daemon[869]: accepted connection from pid 1039, user nix-grpc-daemon (trusted)4639worker2 # [ 40.245964] nix-daemon[869]: reaped child process 1042, status = succeeded4640lb # [ 40.232855] niks3-server[1002]: 2026/09/21 09:27:25 INFO Received uploads request method=POST path=/api/pending_closures4641worker2 # [ 40.272126] nix-grpc-daemon[884]: time=2026-09-21T09:27:24.631Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4642worker2 # [ 40.273606] nix-grpc-daemon[884]: time=2026-09-21T09:27:24.632Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4643lb # [ 40.260877] niks3-server[1002]: 2026/09/21 09:27:25 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4644lb # [ 40.273064] niks3-server[1002]: 2026/09/21 09:27:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4645worker2 # [ 40.308261] nix-grpc-daemon[884]: time=2026-09-21T09:27:24.667Z level=INFO msg="Uploading 1 narinfos"4646lb # [ 40.278155] niks3-server[1002]: 2026/09/21 09:27:25 INFO Signed narinfos id=7 count=14647lb # [ 40.285109] niks3-server[1002]: 2026/09/21 09:27:25 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4648lb # [ 40.295097] niks3-server[1002]: 2026/09/21 09:27:25 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4649worker2 # [ 40.333933] nix-grpc-daemon[884]: time=2026-09-21T09:27:24.693Z level=INFO msg="Upload complete. (462ms)"4650lb # [ 40.303652] niks3-server[1002]: 2026/09/21 09:27:25 INFO Completed upload id=74651worker2 # [ 40.336753] nix-grpc-daemon[786]: ts=2026-09-21T09:27:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=0 paths=1 nar_bytes_in=7524652lb # [ 40.307161] niks3-server[1002]: 2026/09/21 09:27:25 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4653worker2 # [ 40.341891] nix-grpc-daemon[786]: ts=2026-09-21T09:27:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224654worker2 # [ 40.352271] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4655worker2 # [ 40.356128] nix-daemon[869]: reaped child process 1032, status = succeeded4656worker2 # [ 40.361362] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4657client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4658client # farm-top> worker2: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4659client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4660client # farm-top> LOG-farm-top-t1b4661worker2 # [ 41.582225] nix-daemon[869]: reaped child process 1046, status = succeeded4662worker2 # [ 41.754324] nix-daemon[869]: accepted connection from pid 1062, user nix-grpc-daemon (trusted)4663worker2 # [ 41.767531] nix-daemon[869]: reaped child process 1065, status = succeeded4664lb # [ 41.754515] niks3-server[1002]: 2026/09/21 09:27:26 INFO Received uploads request method=POST path=/api/pending_closures4665worker2 # [ 41.793603] nix-grpc-daemon[884]: time=2026-09-21T09:27:26.152Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4666worker2 # [ 41.795141] nix-grpc-daemon[884]: time=2026-09-21T09:27:26.154Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4667lb # [ 41.789157] niks3-server[1002]: 2026/09/21 09:27:26 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4668lb # [ 41.804862] niks3-server[1002]: 2026/09/21 09:27:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4669lb # [ 41.809974] niks3-server[1002]: 2026/09/21 09:27:26 INFO Signed narinfos id=8 count=14670worker2 # [ 41.843562] nix-grpc-daemon[884]: time=2026-09-21T09:27:26.202Z level=INFO msg="Uploading 1 narinfos"4671lb # [ 41.817712] niks3-server[1002]: 2026/09/21 09:27:26 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4672lb # [ 41.826650] niks3-server[1002]: 2026/09/21 09:27:26 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4673lb # [ 41.832573] niks3-server[1002]: 2026/09/21 09:27:26 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4674lb # [ 41.837775] niks3-server[1002]: 2026/09/21 09:27:26 INFO Completed upload id=84675worker2 # [ 41.868405] nix-grpc-daemon[884]: time=2026-09-21T09:27:26.227Z level=INFO msg="Upload complete. (300ms)"4676lb # [ 41.840527] niks3-server[1002]: 2026/09/21 09:27:26 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4677worker2 # [ 41.872121] nix-grpc-daemon[786]: ts=2026-09-21T09:27:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14678worker2 # [ 41.877759] nix-daemon[869]: reaped child process 1044, status = succeeded4679client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.43 seconds)4680client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4681client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.22 seconds)4682lb: must succeed: curl -sf localhost:9901/clusters4683lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4684lb: must succeed: curl -sf localhost:9901/clusters4685lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4686worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4687worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4688(finished: subtest: inputs already in the cache are not sent to the scheduler, in 3.01 seconds)4689subtest: two clients wanting the same drv build it once4690client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4691client # [ 42.382633] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4692client # Running as unit: dedup1.service; invocation ID: 1aa130526d784e2a8508f7db28e7448f4693client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.08 seconds)4694client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4695worker2 # [ 42.632556] nix-grpc-daemon[786]: ts=2026-09-21T09:27:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:476224696client # [ 42.629708] nix[892]: don't know how to build these paths:4697client # [ 42.630634] nix[892]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4698worker2 # [ 42.646144] nix-grpc-daemon[786]: ts=2026-09-21T09:27:27Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04699worker2 # [ 42.650986] nix-grpc-daemon[786]: ts=2026-09-21T09:27:27Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.3:50051 clients=14700worker1 # [ 42.617901] nix-grpc-daemon[781]: ts=2026-09-21T09:27:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:569564701worker1 # [ 42.631301] nix-grpc-daemon[781]: ts=2026-09-21T09:27:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:569564702client # [ 42.669057] nix[892]: copying 1 paths to 'lb:50051'...4703client # [ 42.669848] nix[892]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4704worker1 # [ 42.641761] nix-grpc-daemon[781]: ts=2026-09-21T09:27:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:569564705worker1 # [ 42.653305] nix-daemon[845]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4706worker1 # [ 42.676966] nix-grpc-daemon[781]: ts=2026-09-21T09:27:27Z level=info event=niks3_push_started pid=8754707worker1 # [ 42.992839] nix-grpc-daemon[875]: time=2026-09-21T09:27:28.151Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""4708worker1 # [ 43.229478] nix-daemon[845]: accepted connection from pid 888, user nix-grpc-daemon (trusted)4709worker1 # [ 43.245431] nix-daemon[845]: reaped child process 891, status = succeeded4710worker1 # [ 43.428143] nix-daemon[845]: accepted connection from pid 893, user nix-grpc-daemon (trusted)4711worker1 # [ 43.439951] nix-daemon[845]: reaped child process 896, status = succeeded4712client # [ 43.468971] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4713client # Running as unit: dedup2.service; invocation ID: e51d3e17fea64a038cac1ffac94586ad4714client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.09 seconds)4715lb # [ 43.465880] niks3-server[1002]: 2026/09/21 09:27:28 INFO Received uploads request method=POST path=/api/pending_closures4716worker1 # [ 43.468907] nix-grpc-daemon[875]: time=2026-09-21T09:27:28.631Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4717worker1 # [ 43.471515] nix-grpc-daemon[875]: time=2026-09-21T09:27:28.634Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4718lb # [ 43.505600] niks3-server[1002]: 2026/09/21 09:27:28 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4719lb # [ 43.514434] niks3-server[1002]: 2026/09/21 09:27:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4720lb # [ 43.520533] niks3-server[1002]: 2026/09/21 09:27:28 INFO Signed narinfos id=9 count=14721worker1 # [ 43.518336] nix-grpc-daemon[875]: time=2026-09-21T09:27:28.680Z level=INFO msg="Uploading 1 narinfos"4722lb # [ 43.530888] niks3-server[1002]: 2026/09/21 09:27:28 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4723lb # [ 43.541485] niks3-server[1002]: 2026/09/21 09:27:28 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4724worker1 # [ 43.540415] nix-grpc-daemon[875]: time=2026-09-21T09:27:28.702Z level=INFO msg="Upload complete. (523ms)"4725lb # [ 43.547397] niks3-server[1002]: 2026/09/21 09:27:28 INFO Completed upload id=94726worker1 # [ 43.542078] nix-grpc-daemon[781]: ts=2026-09-21T09:27:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56956 duration_s=0 paths=1 nar_bytes_in=4324727lb # [ 43.550211] niks3-server[1002]: 2026/09/21 09:27:28 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4728worker1 # [ 43.549102] nix-grpc-daemon[781]: ts=2026-09-21T09:27:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:569564729worker1 # [ 43.550778] nix-daemon[845]: reaped child process 873, status = succeeded4730worker1 # [ 43.553731] nix-daemon[845]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4731client # [ 43.588354] nix[892]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4732worker1 # [ 43.560674] nix-daemon[845]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4733client # [ 43.774208] nix[908]: don't know how to build these paths:4734client # [ 43.775674] nix[908]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4735worker2 # [ 43.783383] nix-grpc-daemon[786]: ts=2026-09-21T09:27:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:476224736worker2 # [ 43.796115] nix-grpc-daemon[786]: ts=2026-09-21T09:27:28Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04737client # [ 43.794759] nix[908]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4738worker1 # [ 43.766244] nix-grpc-daemon[781]: ts=2026-09-21T09:27:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:569564739client: must succeed: sleep 24740lb # [ 46.197740] envoy[902]: conn peer=[2001:db8:1::1]:34026 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9537 tx=4073 ms=68494741lb # [ 46.201701] envoy[902]: conn peer=[2001:db8:1::1]:59220 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=674742lb # [ 46.205143] envoy[902]: conn peer=[2001:db8:1::1]:59230 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4169 tx=1809 ms=21514743worker1 # [ 46.544763] nix-grpc-daemon[781]: E0000 00:00:1789982851.706949 821 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4744worker1 # [ 46.564061] nix-grpc-daemon[781]: E0000 00:00:1789982851.721392 821 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4745worker1 # [ 46.566204] nix-grpc-daemon[781]: E0000 00:00:1789982851.721861 821 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4746worker1 # [ 46.572713] nix-grpc-daemon[781]: E0000 00:00:1789982851.735246 821 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4747client: (finished: must succeed: sleep 2, in 2.07 seconds)4748client: waiting for success: ! systemctl is-active dedup1 dedup24749worker1 # [ 46.984922] nix-daemon[845]: reaped child process 904, status = succeeded4750worker1 # [ 47.165234] nix-daemon[845]: accepted connection from pid 930, user nix-grpc-daemon (trusted)4751worker1 # [ 47.177951] nix-daemon[845]: reaped child process 933, status = succeeded4752lb # [ 47.206249] niks3-server[1002]: 2026/09/21 09:27:32 INFO Received uploads request method=POST path=/api/pending_closures4753worker1 # [ 47.209050] nix-grpc-daemon[875]: time=2026-09-21T09:27:32.371Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4754worker1 # [ 47.210699] nix-grpc-daemon[875]: time=2026-09-21T09:27:32.373Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4755lb # [ 47.246991] niks3-server[1002]: 2026/09/21 09:27:32 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4756lb # [ 47.260727] niks3-server[1002]: 2026/09/21 09:27:32 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4757lb # [ 47.269607] niks3-server[1002]: 2026/09/21 09:27:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4758worker1 # [ 47.268236] nix-grpc-daemon[875]: time=2026-09-21T09:27:32.430Z level=INFO msg="Uploading 1 narinfos"4759lb # [ 47.274450] niks3-server[1002]: 2026/09/21 09:27:32 INFO Signed narinfos id=10 count=14760lb # [ 47.279451] niks3-server[1002]: 2026/09/21 09:27:32 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4761lb # [ 47.288549] niks3-server[1002]: 2026/09/21 09:27:32 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4762lb # [ 47.294254] niks3-server[1002]: 2026/09/21 09:27:32 INFO Completed upload id=104763worker1 # [ 47.287773] nix-grpc-daemon[875]: time=2026-09-21T09:27:32.450Z level=INFO msg="Upload complete. (315ms)"4764lb # [ 47.296505] niks3-server[1002]: 2026/09/21 09:27:32 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo4765worker1 # [ 47.294712] nix-grpc-daemon[781]: ts=2026-09-21T09:27:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56956 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14766worker1 # [ 47.301237] nix-grpc-daemon[781]: ts=2026-09-21T09:27:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56956 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14767worker1 # [ 47.311395] nix-daemon[845]: reaped child process 902, status = succeeded4768client # [ 47.371585] systemd[1]: dedup1.service: Deactivated successfully.4769client # [ 47.376293] systemd[1]: dedup1.service: Consumed 126ms CPU time over 4.991s wall clock time, 41.5M memory peak, 7.2K incoming IP traffic, 8.5K outgoing IP traffic.4770client # [ 47.413307] systemd[1]: dedup2.service: Deactivated successfully.4771client # [ 47.415630] systemd[1]: dedup2.service: Consumed 149ms CPU time over 3.946s wall clock time, 35.3M memory peak, 6K incoming IP traffic, 5.7K outgoing IP traffic.4772client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.13 seconds)4773client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4774client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.05 seconds)4775worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4776worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)4777worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4778worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)4779worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4780worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4781worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4782worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4783(finished: subtest: two clients wanting the same drv build it once, in 6.13 seconds)4784subtest: build hook: nix-daemon with builders = grpc://lb4785client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log4786client # [ 48.703528] systemd[1]: Started Nix Daemon.4787client # [ 48.931972] nix-daemon[950]: accepted connection from pid 947, user root (trusted)4788worker1 # [ 49.106748] nix-grpc-daemon[781]: ts=2026-09-21T09:27:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:569564789worker2 # [ 49.149112] nix-grpc-daemon[786]: ts=2026-09-21T09:27:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:476224790worker2 # [ 49.162982] nix-grpc-daemon[786]: ts=2026-09-21T09:27:33Z level=debug event=want client=6 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=04791worker2 # [ 49.168189] nix-grpc-daemon[786]: ts=2026-09-21T09:27:33Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.4:50051 clients=14792worker2 # [ 49.207822] nix-grpc-daemon[786]: ts=2026-09-21T09:27:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224793worker2 # [ 49.220632] nix-grpc-daemon[786]: ts=2026-09-21T09:27:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:476224794worker2 # [ 49.232707] nix-grpc-daemon[786]: ts=2026-09-21T09:27:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:476224795worker2 # [ 49.241810] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4796worker2 # [ 49.442630] nix-daemon[869]: accepted connection from pid 1102, user nix-grpc-daemon (trusted)4797worker2 # [ 49.455176] nix-daemon[869]: reaped child process 1105, status = succeeded4798worker2 # [ 49.632698] nix-daemon[869]: accepted connection from pid 1107, user nix-grpc-daemon (trusted)4799worker2 # [ 49.644121] nix-daemon[869]: reaped child process 1110, status = succeeded4800lb # [ 49.634370] niks3-server[1002]: 2026/09/21 09:27:34 INFO Received uploads request method=POST path=/api/pending_closures4801worker2 # [ 49.673870] nix-grpc-daemon[884]: time=2026-09-21T09:27:34.032Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4802worker2 # [ 49.675208] nix-grpc-daemon[884]: time=2026-09-21T09:27:34.032Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"4803lb # [ 49.662076] niks3-server[1002]: 2026/09/21 09:27:34 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst4804lb # [ 49.674842] niks3-server[1002]: 2026/09/21 09:27:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4805worker2 # [ 49.710673] nix-grpc-daemon[884]: time=2026-09-21T09:27:34.069Z level=INFO msg="Uploading 1 narinfos"4806lb # [ 49.679982] niks3-server[1002]: 2026/09/21 09:27:34 INFO Signed narinfos id=11 count=14807lb # [ 49.687133] niks3-server[1002]: 2026/09/21 09:27:34 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls4808lb # [ 49.696723] niks3-server[1002]: 2026/09/21 09:27:34 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4809lb # [ 49.702112] niks3-server[1002]: 2026/09/21 09:27:34 INFO Completed upload id=114810worker2 # [ 49.735105] nix-grpc-daemon[884]: time=2026-09-21T09:27:34.094Z level=INFO msg="Upload complete. (479ms)"4811lb # [ 49.706675] niks3-server[1002]: 2026/09/21 09:27:34 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo4812worker2 # [ 49.738755] nix-grpc-daemon[786]: ts=2026-09-21T09:27:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=0 paths=1 nar_bytes_in=5124813worker2 # [ 49.743429] nix-grpc-daemon[786]: ts=2026-09-21T09:27:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224814worker2 # [ 49.752470] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4815worker2 # [ 49.759524] nix-daemon[869]: reaped child process 1100, status = succeeded4816worker2 # [ 49.764484] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4817worker2 # [ 50.989594] nix-daemon[869]: reaped child process 1114, status = succeeded4818worker2 # [ 51.187906] nix-daemon[869]: accepted connection from pid 1130, user nix-grpc-daemon (trusted)4819worker2 # [ 51.199696] nix-daemon[869]: reaped child process 1133, status = succeeded4820lb # [ 51.189567] niks3-server[1002]: 2026/09/21 09:27:36 INFO Received uploads request method=POST path=/api/pending_closures4821worker2 # [ 51.228896] nix-grpc-daemon[884]: time=2026-09-21T09:27:35.587Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4822worker2 # [ 51.231521] nix-grpc-daemon[884]: time=2026-09-21T09:27:35.589Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"4823lb # [ 51.229597] niks3-server[1002]: 2026/09/21 09:27:36 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4824lb # [ 51.237750] niks3-server[1002]: 2026/09/21 09:27:36 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv4825lb # [ 51.248771] niks3-server[1002]: 2026/09/21 09:27:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4826lb # [ 51.254255] niks3-server[1002]: 2026/09/21 09:27:36 INFO Signed narinfos id=12 count=14827worker2 # [ 51.287383] nix-grpc-daemon[884]: time=2026-09-21T09:27:35.646Z level=INFO msg="Uploading 1 narinfos"4828lb # [ 51.259149] niks3-server[1002]: 2026/09/21 09:27:36 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls4829lb # [ 51.268821] niks3-server[1002]: 2026/09/21 09:27:36 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4830lb # [ 51.275309] niks3-server[1002]: 2026/09/21 09:27:36 INFO Completed upload id=124831worker2 # [ 51.306208] nix-grpc-daemon[884]: time=2026-09-21T09:27:35.665Z level=INFO msg="Upload complete. (334ms)"4832lb # [ 51.277435] niks3-server[1002]: 2026/09/21 09:27:36 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo4833worker2 # [ 51.313593] nix-grpc-daemon[786]: ts=2026-09-21T09:27:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=6 outputs=14834worker2 # [ 51.322112] nix-grpc-daemon[786]: ts=2026-09-21T09:27:35Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:476224835worker2 # [ 51.330877] nix-daemon[869]: reaped child process 1112, status = succeeded4836worker1 # [ 51.305491] nix-grpc-daemon[781]: ts=2026-09-21T09:27:36Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:569564837worker1 # [ 51.323223] nix-grpc-daemon[781]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...4838worker1 # [ 51.342374] nix-grpc-daemon[781]: ts=2026-09-21T09:27:36Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56956 duration_s=0 paths=1 nar_bytes_out=1284839worker2 # [ 51.680687] nix-grpc-daemon[786]: ts=2026-09-21T09:27:36Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:476224840worker2 # [ 51.687174] nix-grpc-daemon[786]: ts=2026-09-21T09:27:36Z level=debug event=rpc_start method=Connect cn=ci-1 peer=ipv4:192.168.1.2:476224841worker2 # [ 51.695165] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4842worker1 # [ 51.719798] nix-grpc-daemon[781]: ts=2026-09-21T09:27:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:569564843worker1 # [ 51.729473] nix-grpc-daemon[875]: time=2026-09-21T09:27:36.891Z level=INFO msg="All 1 paths already cached"4844worker2 # [ 51.784649] nix-grpc-daemon[786]: ts=2026-09-21T09:27:36Z level=debug event=want client=7 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=04845worker2 # [ 51.789155] nix-grpc-daemon[786]: ts=2026-09-21T09:27:36Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.4:50051 clients=14846worker2 # [ 51.797734] nix-grpc-daemon[786]: ts=2026-09-21T09:27:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224847worker2 # [ 51.810745] nix-grpc-daemon[786]: ts=2026-09-21T09:27:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:476224848worker2 # [ 51.822621] nix-grpc-daemon[884]: time=2026-09-21T09:27:36.181Z level=INFO msg="All 1 paths already cached"4849worker2 # [ 51.829887] nix-grpc-daemon[786]: ts=2026-09-21T09:27:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:476224850worker2 # [ 51.843280] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4851worker2 # [ 52.050949] nix-daemon[869]: accepted connection from pid 1143, user nix-grpc-daemon (trusted)4852worker2 # [ 52.063617] nix-daemon[869]: reaped child process 1146, status = succeeded4853worker2 # [ 52.237763] nix-daemon[869]: accepted connection from pid 1148, user nix-grpc-daemon (trusted)4854worker2 # [ 52.249673] nix-daemon[869]: reaped child process 1151, status = succeeded4855lb # [ 52.238437] niks3-server[1002]: 2026/09/21 09:27:37 INFO Received uploads request method=POST path=/api/pending_closures4856worker2 # [ 52.277535] nix-grpc-daemon[884]: time=2026-09-21T09:27:36.636Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4857worker2 # [ 52.278880] nix-grpc-daemon[884]: time=2026-09-21T09:27:36.636Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"4858lb # [ 52.263853] niks3-server[1002]: 2026/09/21 09:27:37 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst4859lb # [ 52.273578] niks3-server[1002]: 2026/09/21 09:27:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4860lb # [ 52.277786] niks3-server[1002]: 2026/09/21 09:27:37 INFO Signed narinfos id=13 count=14861worker2 # [ 52.311459] nix-grpc-daemon[884]: time=2026-09-21T09:27:36.670Z level=INFO msg="Uploading 1 narinfos"4862lb # [ 52.288719] niks3-server[1002]: 2026/09/21 09:27:37 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls4863lb # [ 52.295769] niks3-server[1002]: 2026/09/21 09:27:37 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4864worker2 # [ 52.332529] nix-grpc-daemon[884]: time=2026-09-21T09:27:36.691Z level=INFO msg="Upload complete. (476ms)"4865lb # [ 52.302234] niks3-server[1002]: 2026/09/21 09:27:37 INFO Completed upload id=134866lb # [ 52.305866] niks3-server[1002]: 2026/09/21 09:27:37 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo4867worker2 # [ 52.335866] nix-grpc-daemon[786]: ts=2026-09-21T09:27:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=0 paths=1 nar_bytes_in=6244868worker2 # [ 52.341261] nix-grpc-daemon[786]: ts=2026-09-21T09:27:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224869worker2 # [ 52.349952] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4870worker2 # [ 52.354556] nix-daemon[869]: reaped child process 1141, status = succeeded4871worker2 # [ 52.359761] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4872worker2 # [ 53.172362] nix-daemon[869]: reaped child process 1155, status = succeeded4873worker2 # [ 53.346413] nix-daemon[869]: accepted connection from pid 1171, user nix-grpc-daemon (trusted)4874worker2 # [ 53.359775] nix-daemon[869]: reaped child process 1174, status = succeeded4875lb # [ 53.347646] niks3-server[1002]: 2026/09/21 09:27:38 INFO Received uploads request method=POST path=/api/pending_closures4876worker2 # [ 53.387443] nix-grpc-daemon[884]: time=2026-09-21T09:27:37.746Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4877worker2 # [ 53.388935] nix-grpc-daemon[884]: time=2026-09-21T09:27:37.748Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"4878lb # [ 53.387412] niks3-server[1002]: 2026/09/21 09:27:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4879worker2 # [ 53.424050] nix-grpc-daemon[884]: time=2026-09-21T09:27:37.782Z level=INFO msg="Uploading 1 narinfos"4880lb # [ 53.393285] niks3-server[1002]: 2026/09/21 09:27:38 INFO Signed narinfos id=14 count=14881lb # [ 53.403065] niks3-server[1002]: 2026/09/21 09:27:38 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst4882lb # [ 53.407615] niks3-server[1002]: 2026/09/21 09:27:38 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv4883lb # [ 53.417536] niks3-server[1002]: 2026/09/21 09:27:38 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls4884lb # [ 53.423837] niks3-server[1002]: 2026/09/21 09:27:38 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4885lb # [ 53.429342] niks3-server[1002]: 2026/09/21 09:27:38 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo4886lb # [ 53.431751] niks3-server[1002]: 2026/09/21 09:27:38 INFO Completed upload id=144887worker2 # [ 53.462553] nix-grpc-daemon[884]: time=2026-09-21T09:27:37.821Z level=INFO msg="Upload complete. (297ms)"4888worker2 # [ 53.469434] nix-grpc-daemon[786]: ts=2026-09-21T09:27:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=7 outputs=14889worker2 # [ 53.481260] nix-daemon[869]: reaped child process 1153, status = succeeded4890worker2 # [ 53.515231] nix-grpc-daemon[786]: ts=2026-09-21T09:27:37Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:476224891worker2 # [ 53.538147] nix-grpc-daemon[786]: ts=2026-09-21T09:27:37Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:476224892worker2 # [ 53.542584] nix-grpc-daemon[786]: ts=2026-09-21T09:27:37Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=0 paths=1 nar_bytes_out=1764893worker2 # [ 53.601416] nix-grpc-daemon[786]: ts=2026-09-21T09:27:37Z level=info event=rpc method=Connect cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=1 bytes_in=312 bytes_out=2004894worker2 # [ 53.606105] nix-daemon[869]: reaped child process 1136, status = succeeded4895worker1 # [ 53.793656] nix-grpc-daemon[781]: ts=2026-09-21T09:27:38Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:569564896worker2 # [ 53.833992] nix-grpc-daemon[786]: ts=2026-09-21T09:27:38Z level=debug event=rpc_start method=Connect cn=ci-1 peer=ipv4:192.168.1.2:476224897worker2 # [ 53.843161] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4898worker1 # [ 53.867503] nix-grpc-daemon[781]: ts=2026-09-21T09:27:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:569564899worker1 # [ 53.884603] nix-grpc-daemon[875]: time=2026-09-21T09:27:39.046Z level=INFO msg="All 1 paths already cached"4900worker2 # [ 53.940609] nix-grpc-daemon[786]: ts=2026-09-21T09:27:38Z level=debug event=want client=8 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=04901worker2 # [ 53.945535] nix-grpc-daemon[786]: ts=2026-09-21T09:27:38Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.4:50051 clients=14902worker2 # [ 53.990112] nix-grpc-daemon[786]: ts=2026-09-21T09:27:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224903worker2 # [ 54.003432] nix-grpc-daemon[786]: ts=2026-09-21T09:27:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:476224904worker2 # [ 54.016334] nix-grpc-daemon[884]: time=2026-09-21T09:27:38.375Z level=INFO msg="All 2 paths already cached"4905worker2 # [ 54.024442] nix-grpc-daemon[786]: ts=2026-09-21T09:27:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:476224906worker2 # [ 54.033381] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4907worker2 # [ 54.226433] nix-daemon[869]: accepted connection from pid 1184, user nix-grpc-daemon (trusted)4908worker2 # [ 54.238689] nix-daemon[869]: reaped child process 1187, status = succeeded4909worker2 # [ 54.400754] nix-daemon[869]: accepted connection from pid 1189, user nix-grpc-daemon (trusted)4910worker2 # [ 54.412541] nix-daemon[869]: reaped child process 1192, status = succeeded4911lb # [ 54.400101] niks3-server[1002]: 2026/09/21 09:27:39 INFO Received uploads request method=POST path=/api/pending_closures4912worker2 # [ 54.438302] nix-grpc-daemon[884]: time=2026-09-21T09:27:38.797Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4913worker2 # [ 54.440082] nix-grpc-daemon[884]: time=2026-09-21T09:27:38.798Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"4914lb # [ 54.425676] niks3-server[1002]: 2026/09/21 09:27:39 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst4915lb # [ 54.437187] niks3-server[1002]: 2026/09/21 09:27:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4916worker2 # [ 54.473542] nix-grpc-daemon[884]: time=2026-09-21T09:27:38.832Z level=INFO msg="Uploading 1 narinfos"4917lb # [ 54.443104] niks3-server[1002]: 2026/09/21 09:27:39 INFO Signed narinfos id=15 count=14918lb # [ 54.449095] niks3-server[1002]: 2026/09/21 09:27:39 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls4919lb # [ 54.461649] niks3-server[1002]: 2026/09/21 09:27:39 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4920lb # [ 54.466449] niks3-server[1002]: 2026/09/21 09:27:39 INFO Completed upload id=154921worker2 # [ 54.499560] nix-grpc-daemon[884]: time=2026-09-21T09:27:38.858Z level=INFO msg="Upload complete. (451ms)"4922lb # [ 54.470331] niks3-server[1002]: 2026/09/21 09:27:39 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo4923worker2 # [ 54.502569] nix-grpc-daemon[786]: ts=2026-09-21T09:27:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=0 paths=1 nar_bytes_in=7524924worker2 # [ 54.507823] nix-grpc-daemon[786]: ts=2026-09-21T09:27:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476224925worker2 # [ 54.516488] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4926worker2 # [ 54.520911] nix-daemon[869]: reaped child process 1182, status = succeeded4927worker2 # [ 54.526223] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4928worker2 # [ 55.951805] nix-daemon[869]: reaped child process 1196, status = succeeded4929worker2 # [ 56.148339] nix-daemon[869]: accepted connection from pid 1212, user nix-grpc-daemon (trusted)4930worker2 # [ 56.158973] nix-daemon[869]: reaped child process 1215, status = succeeded4931lb # [ 56.144963] niks3-server[1002]: 2026/09/21 09:27:41 INFO Received uploads request method=POST path=/api/pending_closures4932worker2 # [ 56.184603] nix-grpc-daemon[884]: time=2026-09-21T09:27:40.543Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4933worker2 # [ 56.189057] nix-grpc-daemon[884]: time=2026-09-21T09:27:40.545Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"4934lb # [ 56.187144] niks3-server[1002]: 2026/09/21 09:27:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4935lb # [ 56.195855] niks3-server[1002]: 2026/09/21 09:27:41 INFO Signed narinfos id=16 count=14936lb # [ 56.199276] envoy[902]: conn peer=[2001:db8:1::1]:59234 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3011 tx=1448 ms=47164937worker2 # [ 56.233955] nix-grpc-daemon[884]: time=2026-09-21T09:27:40.592Z level=INFO msg="Uploading 1 narinfos"4938lb # [ 56.205232] envoy[902]: conn peer=[2001:db8:1::1]:39300 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1427 tx=1085 ms=36054939lb # [ 56.208705] envoy[902]: conn peer=[2001:db8:1::1]:39324 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=334 ms=964940lb # [ 56.212050] envoy[902]: conn peer=[2001:db8:1::1]:39310 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3429 tx=1737 ms=22914941lb # [ 56.215960] envoy[902]: conn peer=[2001:db8:1::1]:35142 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=378 ms=604942lb # [ 56.218730] envoy[902]: conn peer=[2001:db8:1::1]:39334 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4414 tx=2235 ms=19174943lb # [ 56.221580] niks3-server[1002]: 2026/09/21 09:27:41 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst4944lb # [ 56.228454] niks3-server[1002]: 2026/09/21 09:27:41 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv4945lb # [ 56.234211] niks3-server[1002]: 2026/09/21 09:27:41 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4946lb # [ 56.237146] niks3-server[1002]: 2026/09/21 09:27:41 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls4947lb # [ 56.239825] niks3-server[1002]: 2026/09/21 09:27:41 INFO Completed upload id=164948lb # [ 56.241519] niks3-server[1002]: 2026/09/21 09:27:41 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo4949worker2 # [ 56.271799] nix-grpc-daemon[884]: time=2026-09-21T09:27:40.630Z level=INFO msg="Upload complete. (337ms)"4950worker2 # [ 56.277984] nix-grpc-daemon[786]: ts=2026-09-21T09:27:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=8 outputs=14951worker1 # [ 56.246622] nix-grpc-daemon[781]: ts=2026-09-21T09:27:41Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:569564952worker2 # [ 56.292782] nix-daemon[869]: reaped child process 1194, status = succeeded4953worker1 # [ 56.256182] nix-grpc-daemon[781]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...4954worker1 # [ 56.266464] nix-grpc-daemon[781]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...4955worker2 # [ 56.329152] nix-grpc-daemon[786]: ts=2026-09-21T09:27:40Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:476224956worker2 # [ 56.333592] nix-grpc-daemon[786]: ts=2026-09-21T09:27:40Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=0 paths=1 nar_bytes_out=2324957worker2 # [ 56.387556] nix-grpc-daemon[786]: ts=2026-09-21T09:27:40Z level=info event=rpc method=Connect cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=2 bytes_in=376 bytes_out=2004958worker2 # [ 56.392298] nix-daemon[869]: reaped child process 1177, status = succeeded4959client # [ 56.434918] nix-daemon[950]: reaped child process 952, status = succeeded4960client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.99 seconds)4961client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top4962client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.03 seconds)4963client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4964client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.04 seconds)4965client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4966client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4967(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 8.10 seconds)4968subtest: an input only one worker has still reaches the builder4969client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only4970client # this derivation will be built:4971client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv4972client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...4973client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.49 seconds)4974client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure4975client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.23 seconds)4976worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4977worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.26 seconds)4978worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only4979worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.03 seconds)4980client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&24981client # [ 57.799685] nix-daemon[950]: accepted connection from pid 1037, user root (trusted)4982client # this derivation will be built:4983client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv4984worker1 # [ 58.000354] nix-grpc-daemon[781]: ts=2026-09-21T09:27:43Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:569564985client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...4986worker2 # [ 58.041245] nix-grpc-daemon[786]: ts=2026-09-21T09:27:42Z level=debug event=rpc_start method=Connect cn=ci-1 peer=ipv4:192.168.1.2:476224987worker2 # [ 58.049774] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4988worker1 # [ 58.074134] nix-grpc-daemon[781]: ts=2026-09-21T09:27:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:569564989worker1 # [ 58.285339] nix-daemon[845]: accepted connection from pid 952, user nix-grpc-daemon (trusted)4990worker1 # [ 58.295781] nix-daemon[845]: reaped child process 955, status = succeeded4991lb # [ 58.322449] niks3-server[1002]: 2026/09/21 09:27:43 INFO Received uploads request method=POST path=/api/pending_closures4992worker1 # [ 58.324497] nix-grpc-daemon[875]: time=2026-09-21T09:27:43.486Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4993worker1 # [ 58.325964] nix-grpc-daemon[875]: time=2026-09-21T09:27:43.486Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"4994lb # [ 58.346828] niks3-server[1002]: 2026/09/21 09:27:43 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4995lb # [ 58.359131] niks3-server[1002]: 2026/09/21 09:27:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4996lb # [ 58.363668] niks3-server[1002]: 2026/09/21 09:27:43 INFO Signed narinfos id=17 count=14997worker1 # [ 58.360157] nix-grpc-daemon[875]: time=2026-09-21T09:27:43.522Z level=INFO msg="Uploading 1 narinfos"4998lb # [ 58.370780] niks3-server[1002]: 2026/09/21 09:27:43 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls4999lb # [ 58.382388] niks3-server[1002]: 2026/09/21 09:27:43 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5000lb # [ 58.388105] niks3-server[1002]: 2026/09/21 09:27:43 INFO Completed upload id=175001worker1 # [ 58.381474] nix-grpc-daemon[875]: time=2026-09-21T09:27:43.543Z level=INFO msg="Upload complete. (302ms)"5002lb # [ 58.392239] niks3-server[1002]: 2026/09/21 09:27:43 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5003worker2 # [ 58.438201] nix-grpc-daemon[786]: ts=2026-09-21T09:27:42Z level=debug event=want client=9 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05004worker2 # [ 58.442795] nix-grpc-daemon[786]: ts=2026-09-21T09:27:42Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=15005worker1 # [ 58.445673] nix-grpc-daemon[781]: ts=2026-09-21T09:27:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:569565006worker1 # [ 58.458878] nix-grpc-daemon[781]: ts=2026-09-21T09:27:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:569565007client # copying 2 paths to 'lb:50051'...5008client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5009worker1 # [ 58.475381] nix-grpc-daemon[781]: ts=2026-09-21T09:27:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:569565010client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5011worker1 # [ 58.488171] nix-daemon[845]: accepted connection from pid 781, user nix-grpc-daemon (trusted)5012worker1 # [ 58.696758] nix-daemon[845]: accepted connection from pid 959, user nix-grpc-daemon (trusted)5013worker1 # [ 58.707803] nix-daemon[845]: reaped child process 962, status = succeeded5014worker1 # [ 58.897607] nix-daemon[845]: accepted connection from pid 964, user nix-grpc-daemon (trusted)5015worker1 # [ 58.909753] nix-daemon[845]: reaped child process 967, status = succeeded5016lb # [ 58.935774] niks3-server[1002]: 2026/09/21 09:27:44 INFO Received uploads request method=POST path=/api/pending_closures5017lb # [ 58.944944] niks3-server[1002]: 2026/09/21 09:27:44 INFO Received uploads request method=POST path=/api/pending_closures5018worker1 # [ 58.946028] nix-grpc-daemon[875]: time=2026-09-21T09:27:44.108Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5019worker1 # [ 58.948274] nix-grpc-daemon[875]: time=2026-09-21T09:27:44.110Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5020worker1 # [ 58.949889] nix-grpc-daemon[875]: time=2026-09-21T09:27:44.110Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5021lb # [ 58.982489] niks3-server[1002]: 2026/09/21 09:27:44 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5022lb # [ 58.987965] niks3-server[1002]: 2026/09/21 09:27:44 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5023lb # [ 59.000657] niks3-server[1002]: 2026/09/21 09:27:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5024lb # [ 59.004215] niks3-server[1002]: 2026/09/21 09:27:44 INFO Signed narinfos id=18 count=15025lb # [ 59.007679] niks3-server[1002]: 2026/09/21 09:27:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5026lb # [ 59.013700] niks3-server[1002]: 2026/09/21 09:27:44 INFO Signed narinfos id=19 count=25027lb # [ 59.014831] niks3-server[1002]: 2026/09/21 09:27:44 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5028worker1 # [ 59.007742] nix-grpc-daemon[875]: time=2026-09-21T09:27:44.169Z level=INFO msg="Uploading 3 narinfos"5029lb # [ 59.018873] niks3-server[1002]: 2026/09/21 09:27:44 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5030lb # [ 59.030969] niks3-server[1002]: 2026/09/21 09:27:44 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5031lb # [ 59.033556] niks3-server[1002]: 2026/09/21 09:27:44 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5032lb # [ 59.036802] niks3-server[1002]: 2026/09/21 09:27:44 INFO Completed upload id=195033lb # [ 59.038680] niks3-server[1002]: 2026/09/21 09:27:44 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5034lb # [ 59.040809] niks3-server[1002]: 2026/09/21 09:27:44 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5035lb # [ 59.042649] niks3-server[1002]: 2026/09/21 09:27:44 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5036lb # [ 59.044834] niks3-server[1002]: 2026/09/21 09:27:44 INFO Completed upload id=185037worker1 # [ 59.039362] nix-grpc-daemon[875]: time=2026-09-21T09:27:44.201Z level=INFO msg="Upload complete. (535ms)"5038worker1 # [ 59.045618] nix-grpc-daemon[781]: ts=2026-09-21T09:27:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56956 duration_s=0 paths=2 nar_bytes_in=9765039worker1 # [ 59.051177] nix-grpc-daemon[781]: ts=2026-09-21T09:27:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:569565040worker1 # [ 59.061109] nix-daemon[845]: accepted connection from pid 781, user nix-grpc-daemon (trusted)5041worker1 # [ 59.068619] nix-daemon[845]: reaped child process 957, status = succeeded5042worker1 # [ 59.074291] nix-daemon[845]: accepted connection from pid 781, user nix-grpc-daemon (trusted)5043client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5044client # uses-input-w1onlya> node-a: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5045client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5046worker1 # [ 59.886739] nix-daemon[845]: reaped child process 971, status = succeeded5047worker1 # [ 60.057646] nix-daemon[845]: accepted connection from pid 987, user nix-grpc-daemon (trusted)5048worker1 # [ 60.069753] nix-daemon[845]: reaped child process 990, status = succeeded5049lb # [ 60.095606] niks3-server[1002]: 2026/09/21 09:27:45 INFO Received uploads request method=POST path=/api/pending_closures5050worker1 # [ 60.097524] nix-grpc-daemon[875]: time=2026-09-21T09:27:45.259Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5051lb # [ 60.126274] niks3-server[1002]: 2026/09/21 09:27:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5052lb # [ 60.133114] niks3-server[1002]: 2026/09/21 09:27:45 INFO Signed narinfos id=20 count=15053worker1 # [ 60.129523] nix-grpc-daemon[875]: time=2026-09-21T09:27:45.292Z level=INFO msg="Uploading 1 narinfos"5054lb # [ 60.138577] niks3-server[1002]: 2026/09/21 09:27:45 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5055lb # [ 60.146345] niks3-server[1002]: 2026/09/21 09:27:45 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5056lb # [ 60.159129] niks3-server[1002]: 2026/09/21 09:27:45 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5057lb # [ 60.163642] niks3-server[1002]: 2026/09/21 09:27:45 INFO Completed upload id=205058worker1 # [ 60.159483] nix-grpc-daemon[875]: time=2026-09-21T09:27:45.321Z level=INFO msg="Upload complete. (279ms)"5059lb # [ 60.166416] niks3-server[1002]: 2026/09/21 09:27:45 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5060worker1 # [ 60.163188] nix-grpc-daemon[781]: ts=2026-09-21T09:27:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56956 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=9 outputs=15061client # copying 1 paths...5062worker1 # [ 60.168485] nix-grpc-daemon[781]: ts=2026-09-21T09:27:45Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:569565063worker1 # [ 60.171326] nix-daemon[845]: reaped child process 969, status = succeeded5064client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5065worker1 # [ 60.188588] nix-grpc-daemon[781]: ts=2026-09-21T09:27:45Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:569565066worker1 # [ 60.191124] nix-grpc-daemon[781]: ts=2026-09-21T09:27:45Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56956 duration_s=0 paths=1 nar_bytes_out=1205067worker2 # [ 60.283654] nix-grpc-daemon[786]: ts=2026-09-21T09:27:44Z level=info event=rpc method=Connect cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=2 bytes_in=320 bytes_out=2005068worker2 # [ 60.288983] nix-daemon[869]: reaped child process 1221, status = succeeded5069client # [ 60.331851] nix-daemon[950]: reaped child process 1040, status = succeeded5070client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.78 seconds)5071client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25072client # [ 60.558733] nix-daemon[950]: accepted connection from pid 1061, user root (trusted)5073client # this derivation will be built:5074client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5075worker2 # [ 60.803838] nix-grpc-daemon[786]: ts=2026-09-21T09:27:45Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:476225076client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5077worker2 # [ 60.809303] nix-grpc-daemon[786]: ts=2026-09-21T09:27:45Z level=debug event=rpc_start method=Connect cn=ci-1 peer=ipv4:192.168.1.2:476225078worker2 # [ 60.818127] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5079worker1 # [ 60.842144] nix-grpc-daemon[781]: ts=2026-09-21T09:27:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:569565080worker1 # [ 60.851100] nix-grpc-daemon[875]: time=2026-09-21T09:27:46.013Z level=INFO msg="All 1 paths already cached"5081worker2 # [ 60.906615] nix-grpc-daemon[786]: ts=2026-09-21T09:27:45Z level=debug event=want client=10 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05082worker2 # [ 60.911755] nix-grpc-daemon[786]: ts=2026-09-21T09:27:45Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=15083worker2 # [ 60.918979] nix-grpc-daemon[786]: ts=2026-09-21T09:27:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476225084worker2 # [ 60.931161] nix-grpc-daemon[786]: ts=2026-09-21T09:27:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:476225085client # copying 1 paths to 'lb:50051'...5086client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5087worker2 # [ 60.950346] nix-grpc-daemon[786]: ts=2026-09-21T09:27:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:476225088worker2 # [ 60.958716] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5089worker2 # [ 60.976682] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 37 ms (attempt 1/5)5090worker2 # [ 61.014048] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 145 ms (attempt 2/5)5091worker2 # [ 61.164938] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 11 ms (attempt 3/5)5092worker2 # [ 61.181731] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 592 ms (attempt 4/5)5093worker2 # [ 61.780446] nix-grpc-daemon[786]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5094worker2 # [ 61.784614] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5095worker2 # [ 61.789767] nix-grpc-daemon[786]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5096worker2 # [ 62.039530] nix-daemon[869]: accepted connection from pid 1238, user nix-grpc-daemon (trusted)5097worker2 # [ 62.050926] nix-daemon[869]: reaped child process 1241, status = succeeded5098worker2 # [ 62.231047] nix-daemon[869]: accepted connection from pid 1243, user nix-grpc-daemon (trusted)5099worker2 # [ 62.242601] nix-daemon[869]: reaped child process 1246, status = succeeded5100lb # [ 62.233917] niks3-server[1002]: 2026/09/21 09:27:47 INFO Received uploads request method=POST path=/api/pending_closures5101worker2 # [ 62.273033] nix-grpc-daemon[884]: time=2026-09-21T09:27:46.631Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5102worker2 # [ 62.274440] nix-grpc-daemon[884]: time=2026-09-21T09:27:46.631Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5103lb # [ 62.265283] niks3-server[1002]: 2026/09/21 09:27:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5104worker2 # [ 62.304134] nix-grpc-daemon[884]: time=2026-09-21T09:27:46.663Z level=INFO msg="Uploading 1 narinfos"5105lb # [ 62.273436] niks3-server[1002]: 2026/09/21 09:27:47 INFO Signed narinfos id=21 count=15106lb # [ 62.277651] niks3-server[1002]: 2026/09/21 09:27:47 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5107lb # [ 62.288048] niks3-server[1002]: 2026/09/21 09:27:47 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5108lb # [ 62.296830] niks3-server[1002]: 2026/09/21 09:27:47 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5109lb # [ 62.300052] niks3-server[1002]: 2026/09/21 09:27:47 INFO Completed upload id=215110worker2 # [ 62.330681] nix-grpc-daemon[884]: time=2026-09-21T09:27:46.689Z level=INFO msg="Upload complete. (517ms)"5111lb # [ 62.303096] niks3-server[1002]: 2026/09/21 09:27:47 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5112worker2 # [ 62.332981] nix-grpc-daemon[786]: ts=2026-09-21T09:27:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=1 paths=1 nar_bytes_in=5605113worker2 # [ 62.338276] nix-grpc-daemon[786]: ts=2026-09-21T09:27:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:476225114worker2 # [ 62.348545] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5115worker2 # [ 62.352218] nix-daemon[869]: reaped child process 1231, status = succeeded5116worker2 # [ 62.361203] nix-grpc-daemon[786]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5117worker2 # [ 62.370628] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5118client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5119client # uses-input-w1onlyb> worker2: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5120client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5121worker2 # [ 63.795257] nix-daemon[869]: reaped child process 1251, status = succeeded5122worker2 # [ 63.985363] nix-daemon[869]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5123worker2 # [ 63.997122] nix-daemon[869]: reaped child process 1270, status = succeeded5124lb # [ 63.986874] niks3-server[1002]: 2026/09/21 09:27:49 INFO Received uploads request method=POST path=/api/pending_closures5125worker2 # [ 64.025324] nix-grpc-daemon[884]: time=2026-09-21T09:27:48.384Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5126lb # [ 64.017649] niks3-server[1002]: 2026/09/21 09:27:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5127lb # [ 64.022242] niks3-server[1002]: 2026/09/21 09:27:49 INFO Signed narinfos id=22 count=15128worker2 # [ 64.055690] nix-grpc-daemon[884]: time=2026-09-21T09:27:48.414Z level=INFO msg="Uploading 1 narinfos"5129lb # [ 64.029299] niks3-server[1002]: 2026/09/21 09:27:49 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5130lb # [ 64.042526] niks3-server[1002]: 2026/09/21 09:27:49 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5131lb # [ 64.046584] niks3-server[1002]: 2026/09/21 09:27:49 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5132lb # [ 64.052504] niks3-server[1002]: 2026/09/21 09:27:49 INFO Completed upload id=225133worker2 # [ 64.082543] nix-grpc-daemon[884]: time=2026-09-21T09:27:48.441Z level=INFO msg="Upload complete. (304ms)"5134lb # [ 64.057296] niks3-server[1002]: 2026/09/21 09:27:49 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5135worker2 # [ 64.087891] nix-grpc-daemon[786]: ts=2026-09-21T09:27:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=10 outputs=15136client # copying 1 paths...5137worker2 # [ 64.100169] nix-daemon[869]: reaped child process 1248, status = succeeded5138worker1 # [ 64.060738] nix-grpc-daemon[781]: ts=2026-09-21T09:27:49Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:569565139worker1 # [ 64.072696] nix-grpc-daemon[781]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5140client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5141worker1 # [ 64.105359] nix-grpc-daemon[781]: ts=2026-09-21T09:27:49Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:569565142worker1 # [ 64.110190] nix-grpc-daemon[781]: ts=2026-09-21T09:27:49Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56956 duration_s=0 paths=1 nar_bytes_out=1205143worker2 # [ 64.201714] nix-grpc-daemon[786]: ts=2026-09-21T09:27:48Z level=info event=rpc method=Connect cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=3 bytes_in=320 bytes_out=2005144worker2 # [ 64.206566] nix-daemon[869]: reaped child process 1226, status = succeeded5145client # [ 64.251531] nix-daemon[950]: reaped child process 1064, status = succeeded5146client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.92 seconds)5147client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25148client # [ 64.456990] nix-daemon[950]: accepted connection from pid 1084, user root (trusted)5149client # this derivation will be built:5150client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5151worker1 # [ 64.665737] nix-grpc-daemon[781]: ts=2026-09-21T09:27:49Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:569565152client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5153worker2 # [ 64.706303] nix-grpc-daemon[786]: ts=2026-09-21T09:27:49Z level=debug event=rpc_start method=Connect cn=ci-1 peer=ipv4:192.168.1.2:476225154worker2 # [ 64.714402] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5155worker1 # [ 64.738789] nix-grpc-daemon[781]: ts=2026-09-21T09:27:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:569565156worker1 # [ 64.748540] nix-grpc-daemon[875]: time=2026-09-21T09:27:49.910Z level=INFO msg="All 1 paths already cached"5157worker2 # [ 64.803469] nix-grpc-daemon[786]: ts=2026-09-21T09:27:49Z level=debug event=want client=11 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05158worker2 # [ 64.808637] nix-grpc-daemon[786]: ts=2026-09-21T09:27:49Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=15159worker1 # [ 64.775154] nix-grpc-daemon[781]: ts=2026-09-21T09:27:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:569565160worker1 # [ 64.787493] nix-grpc-daemon[781]: ts=2026-09-21T09:27:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:569565161worker1 # [ 64.800180] nix-grpc-daemon[875]: time=2026-09-21T09:27:49.962Z level=INFO msg="All 1 paths already cached"5162client # copying 1 paths to 'lb:50051'...5163client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5164worker1 # [ 64.808678] nix-grpc-daemon[781]: ts=2026-09-21T09:27:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:569565165worker1 # [ 64.817850] nix-daemon[845]: accepted connection from pid 781, user nix-grpc-daemon (trusted)5166worker1 # [ 65.020186] nix-daemon[845]: accepted connection from pid 995, user nix-grpc-daemon (trusted)5167worker1 # [ 65.032675] nix-daemon[845]: reaped child process 998, status = succeeded5168worker1 # [ 65.215178] nix-daemon[845]: accepted connection from pid 1000, user nix-grpc-daemon (trusted)5169worker1 # [ 65.227040] nix-daemon[845]: reaped child process 1003, status = succeeded5170lb # [ 65.252949] niks3-server[1002]: 2026/09/21 09:27:50 INFO Received uploads request method=POST path=/api/pending_closures5171worker1 # [ 65.255709] nix-grpc-daemon[875]: time=2026-09-21T09:27:50.417Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5172worker1 # [ 65.257110] nix-grpc-daemon[875]: time=2026-09-21T09:27:50.418Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5173lb # [ 65.278884] niks3-server[1002]: 2026/09/21 09:27:50 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5174lb # [ 65.289580] niks3-server[1002]: 2026/09/21 09:27:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5175lb # [ 65.295504] niks3-server[1002]: 2026/09/21 09:27:50 INFO Signed narinfos id=23 count=15176worker1 # [ 65.291875] nix-grpc-daemon[875]: time=2026-09-21T09:27:50.454Z level=INFO msg="Uploading 1 narinfos"5177lb # [ 65.302452] niks3-server[1002]: 2026/09/21 09:27:50 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5178lb # [ 65.313566] niks3-server[1002]: 2026/09/21 09:27:50 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5179lb # [ 65.318922] niks3-server[1002]: 2026/09/21 09:27:50 INFO Completed upload id=235180worker1 # [ 65.312232] nix-grpc-daemon[875]: time=2026-09-21T09:27:50.474Z level=INFO msg="Upload complete. (480ms)"5181lb # [ 65.323832] niks3-server[1002]: 2026/09/21 09:27:50 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5182worker1 # [ 65.317243] nix-grpc-daemon[781]: ts=2026-09-21T09:27:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56956 duration_s=0 paths=1 nar_bytes_in=5605183worker1 # [ 65.321636] nix-grpc-daemon[781]: ts=2026-09-21T09:27:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:569565184worker1 # [ 65.331634] nix-daemon[845]: accepted connection from pid 781, user nix-grpc-daemon (trusted)5185worker1 # [ 65.335155] nix-daemon[845]: reaped child process 993, status = succeeded5186client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5187worker1 # [ 65.340196] nix-daemon[845]: accepted connection from pid 781, user nix-grpc-daemon (trusted)5188client # uses-input-w1onlyc> node-a: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5189client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5190worker1 # [ 65.953917] nix-daemon[845]: reaped child process 1007, status = succeeded5191worker1 # [ 66.119967] nix-daemon[845]: accepted connection from pid 1023, user nix-grpc-daemon (trusted)5192worker1 # [ 66.131828] nix-daemon[845]: reaped child process 1026, status = succeeded5193lb # [ 66.157326] niks3-server[1002]: 2026/09/21 09:27:51 INFO Received uploads request method=POST path=/api/pending_closures5194worker1 # [ 66.160251] nix-grpc-daemon[875]: time=2026-09-21T09:27:51.322Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5195lb # [ 66.188063] niks3-server[1002]: 2026/09/21 09:27:51 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5196lb # [ 66.193658] niks3-server[1002]: 2026/09/21 09:27:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5197lb # [ 66.197979] niks3-server[1002]: 2026/09/21 09:27:51 INFO Signed narinfos id=24 count=15198worker1 # [ 66.194871] nix-grpc-daemon[875]: time=2026-09-21T09:27:51.357Z level=INFO msg="Uploading 1 narinfos"5199lb # [ 66.203867] envoy[902]: conn peer=[2001:db8:1::1]:35154 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4987 tx=2598 ms=25585200lb # [ 66.207651] envoy[902]: conn peer=[2001:db8:1::1]:35168 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=585201lb # [ 66.211109] envoy[902]: conn peer=[2001:db8:1::1]:35176 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=555202lb # [ 66.215961] envoy[902]: conn peer=[2001:db8:1::1]:35174 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4430 tx=2136 ms=22455203lb # [ 66.218898] envoy[902]: conn peer=[2001:db8:1::1]:39772 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=585204lb # [ 66.221535] envoy[902]: conn peer=[2001:db8:1::1]:35188 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4314 tx=2307 ms=33975205lb # [ 66.227447] niks3-server[1002]: 2026/09/21 09:27:51 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5206lb # [ 66.235070] niks3-server[1002]: 2026/09/21 09:27:51 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5207lb # [ 66.238226] niks3-server[1002]: 2026/09/21 09:27:51 INFO Completed upload id=245208worker1 # [ 66.231948] nix-grpc-daemon[875]: time=2026-09-21T09:27:51.394Z level=INFO msg="Upload complete. (284ms)"5209lb # [ 66.240875] niks3-server[1002]: 2026/09/21 09:27:51 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5210client # copying 1 paths...5211worker1 # [ 66.238174] nix-grpc-daemon[781]: ts=2026-09-21T09:27:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56956 duration_s=0 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=11 outputs=15212worker1 # [ 66.245176] nix-grpc-daemon[781]: ts=2026-09-21T09:27:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:569565213client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5214worker1 # [ 66.255273] nix-daemon[845]: reaped child process 1005, status = succeeded5215worker1 # [ 66.261117] nix-grpc-daemon[781]: ts=2026-09-21T09:27:51Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:569565216worker1 # [ 66.264281] nix-grpc-daemon[781]: ts=2026-09-21T09:27:51Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56956 duration_s=0 paths=1 nar_bytes_out=1205217worker2 # [ 66.352842] nix-grpc-daemon[786]: ts=2026-09-21T09:27:50Z level=info event=rpc method=Connect cn=ci-1 peer=ipv4:192.168.1.2:47622 duration_s=1 bytes_in=320 bytes_out=2005218worker2 # [ 66.358216] nix-daemon[869]: reaped child process 1273, status = succeeded5219client # [ 66.395437] nix-daemon[950]: reaped child process 1087, status = succeeded5220client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.14 seconds)5221(finished: subtest: an input only one worker has still reaches the builder, in 9.85 seconds)5222subtest: upload whose reference the worker lacks is completed from the cache5223client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5224client # this derivation will be built:5225client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5226client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5227client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.47 seconds)5228client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5229client # this derivation will be built:5230client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5231client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5232client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.48 seconds)5233client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5234client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.21 seconds)5235worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5236worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.24 seconds)5237worker2 # [ 67.888803] systemd[1]: Stopping Nix worker-protocol over gRPC...5238worker2 # [ 68.607357] nix-grpc-daemon[786]: ts=2026-09-21T09:27:52Z level=info event=signal_exit5239worker2 # [ 68.855375] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=scheduler_yield5240worker1 # [ 68.822710] nix-grpc-daemon[781]: ts=2026-09-21T09:27:53Z level=info event=scheduler_restarting addr=lb:500515241lb # [ 68.830720] niks3-server[1002]: 2026/09/21 09:27:53 INFO lead: released remote=192.168.1.4:339725242worker2 # [ 68.861469] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:476105243worker2 # [ 68.866166] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=worker_gone addr=192.168.1.3:500515244worker2 # [ 68.869329] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=scheduler_restarting addr=lb:500515245worker2 # [ 68.872480] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:476105246worker2 # [ 68.875828] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=worker_gone addr=192.168.1.4:500515247worker1 # [ 69.022647] nix-grpc-daemon[781]: ts=2026-09-21T09:27:54Z level=info event=scheduler_connected addr=lb:500515248worker2 # [ 69.062711] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=scheduler_connected addr=lb:500515249worker1 # [ 69.026747] nix-grpc-daemon[781]: ts=2026-09-21T09:27:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5250worker2 # [ 69.066609] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5251worker1 # [ 69.227674] nix-grpc-daemon[781]: ts=2026-09-21T09:27:54Z level=info event=scheduler_connected addr=lb:500515252worker2 # [ 69.267883] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=scheduler_connected addr=lb:500515253worker1 # [ 69.231689] nix-grpc-daemon[781]: ts=2026-09-21T09:27:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5254worker2 # [ 69.271668] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5255lb # [ 69.264982] niks3-server[1002]: 2026/09/21 09:27:54 INFO lead: acquired remote=192.168.1.3:561165256worker1 # [ 69.261815] nix-grpc-daemon[781]: ts=2026-09-21T09:27:54Z level=info event=scheduler_take_over5257worker1 # [ 69.432574] nix-grpc-daemon[781]: ts=2026-09-21T09:27:54Z level=info event=scheduler_connected addr=lb:500515258worker2 # [ 69.472734] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=scheduler_connected addr=lb:500515259worker1 # [ 69.436542] nix-grpc-daemon[781]: ts=2026-09-21T09:27:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5260worker2 # [ 69.476578] nix-grpc-daemon[786]: ts=2026-09-21T09:27:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5261worker1 # [ 69.637509] nix-grpc-daemon[781]: ts=2026-09-21T09:27:54Z level=info event=scheduler_connected addr=lb:500515262worker2 # [ 69.678899] nix-grpc-daemon[786]: ts=2026-09-21T09:27:54Z level=info event=scheduler_connected addr=lb:500515263worker1 # [ 69.641467] nix-grpc-daemon[781]: ts=2026-09-21T09:27:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5264worker2 # [ 69.682297] nix-grpc-daemon[786]: ts=2026-09-21T09:27:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5265worker2 # [ 69.859340] nix-grpc-daemon[786]: ts=2026-09-21T09:27:54Z level=info event=scheduler_restarting5266worker1 # [ 69.842910] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=info event=scheduler_connected addr=lb:500515267worker1 # [ 69.846899] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5268worker2 # [ 69.898619] nix-grpc-daemon[786]: ts=2026-09-21T09:27:54Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5269worker2 # [ 69.902970] nix-daemon[869]: reaped child process 871, status = succeeded5270worker1 # [ 70.047694] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=info event=scheduler_connected addr=lb:500515271worker1 # [ 70.051764] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5272worker1 # [ 70.252673] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=info event=scheduler_connected addr=lb:500515273worker1 # [ 70.256816] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5274worker1 # [ 70.457654] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=info event=scheduler_connected addr=lb:500515275worker1 # [ 70.461519] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5276worker2 # [ 70.522858] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5277worker2 # [ 70.526713] systemd[1]: Stopped Nix worker-protocol over gRPC.5278worker2 # [ 70.530651] systemd[1]: nix-grpc-daemon.service: Consumed 2.613s CPU time over 54.198s wall clock time, 65.2M memory peak, 90.3K incoming IP traffic, 231.5K outgoing IP traffic.5279worker2 # [ 70.541607] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5280worker2 # [ 70.545102] systemd[1]: Closed Nix worker-protocol over gRPC.5281worker2 # [ 70.547910] systemd[1]: nix-grpc-daemon.socket: Consumed 299us CPU time over 1min 1.474s wall clock time, 256K memory peak, 92.6K incoming IP traffic, 73.3K outgoing IP traffic.5282lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5283lb: must succeed: curl -sf localhost:9901/clusters5284lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5285lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.08 seconds)5286client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25287worker1 # [ 70.662505] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=info event=scheduler_connected addr=lb:500515288worker1 # [ 70.667261] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5289worker1 # [ 70.806255] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:569565290client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5291worker1 # [ 70.811477] nix-grpc-daemon[781]: ts=2026-09-21T09:27:55Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:569565292client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.24 seconds)5293worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5294worker1 # [ 70.867846] nix-grpc-daemon[781]: ts=2026-09-21T09:27:56Z level=info event=scheduler_connected addr=lb:500515295worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.03 seconds)5296worker1 # [ 70.879359] nix-grpc-daemon[781]: ts=2026-09-21T09:27:56Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:364345297worker1 # [ 70.884126] nix-grpc-daemon[781]: ts=2026-09-21T09:27:56Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05298worker2 # [ 70.966986] systemd[1]: Listening on Nix worker-protocol over gRPC.5299worker1 # [ 70.995518] systemd[1]: Stopping Nix worker-protocol over gRPC...5300worker1 # [ 71.533444] nix-grpc-daemon[781]: ts=2026-09-21T09:27:56Z level=info event=signal_exit5301worker2 # [ 71.764812] systemd[1]: Starting scheduler-token.service...5302worker1 # [ 71.788877] nix-grpc-daemon[781]: ts=2026-09-21T09:27:56Z level=info event=scheduler_yield5303lb # [ 71.798825] niks3-server[1002]: 2026/09/21 09:27:56 INFO lead: released remote=192.168.1.3:561165304worker1 # [ 71.796475] nix-grpc-daemon[781]: ts=2026-09-21T09:27:56Z level=info event=scheduler_restarting addr=lb:500515305worker1 # [ 71.801364] nix-grpc-daemon[781]: ts=2026-09-21T09:27:56Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:364345306worker1 # [ 71.804638] nix-grpc-daemon[781]: ts=2026-09-21T09:27:56Z level=info event=worker_gone addr=192.168.1.3:500515307worker2 # [ 71.868561] systemd[1]: scheduler-token.service: Deactivated successfully.5308worker2 # [ 71.870942] systemd[1]: Finished scheduler-token.service.5309worker2 # [ 71.873576] systemd[1]: scheduler-token.service: Consumed 36ms CPU time over 104ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5310worker2 # [ 71.879324] systemd[1]: Starting Nix worker-protocol over gRPC...5311worker1 # [ 71.998477] nix-grpc-daemon[781]: ts=2026-09-21T09:27:57Z level=info event=scheduler_connected addr=lb:500515312worker1 # [ 72.002287] nix-grpc-daemon[781]: ts=2026-09-21T09:27:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5313worker2 # [ 72.067496] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:56Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5314worker2 # [ 72.069366] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:56Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5315worker2 # [ 72.073068] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:56Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15316worker2 # [ 72.081071] nix-grpc-daemon[1299]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5317worker2 # [ 72.082442] nix-grpc-daemon[1299]: E0000 00:00:1789982876.439859 1299 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5318worker2 # [ 72.087844] nix-grpc-daemon[1299]: E0000 00:00:1789982876.444995 1299 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5319worker2 # [ 72.092170] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:56Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5320worker2 # [ 72.094992] systemd[1]: Started Nix worker-protocol over gRPC.5321worker2 # [ 72.101536] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:56Z level=info event=scheduler_take_over5322lb # [ 72.071291] niks3-server[1002]: 2026/09/21 09:27:57 INFO lead: acquired remote=192.168.1.4:378125323worker2 # [ 72.105061] nix-grpc-daemon[1299]: E0000 00:00:1789982876.462843 1340 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5324worker2 # [ 72.117061] nix-grpc-daemon[1299]: E0000 00:00:1789982876.476059 1340 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5325worker2 # [ 72.121145] nix-grpc-daemon[1299]: E0000 00:00:1789982876.478355 1340 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5326worker2 # [ 72.130879] nix-grpc-daemon[1299]: E0000 00:00:1789982876.489917 1340 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5327worker2 # [ 72.140184] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:56Z level=info event=scheduler_connected addr=lb:500515328worker2 # [ 72.141681] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5329worker1 # [ 72.203124] nix-grpc-daemon[781]: ts=2026-09-21T09:27:57Z level=info event=scheduler_connected addr=lb:500515330worker1 # [ 72.207239] nix-grpc-daemon[781]: ts=2026-09-21T09:27:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5331worker2 # [ 72.342888] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:56Z level=info event=scheduler_connected addr=lb:500515332worker2 # [ 72.346862] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5333worker1 # [ 72.408130] nix-grpc-daemon[781]: ts=2026-09-21T09:27:57Z level=info event=scheduler_connected addr=lb:500515334worker1 # [ 72.412129] nix-grpc-daemon[781]: ts=2026-09-21T09:27:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5335worker2 # [ 72.547938] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:56Z level=info event=scheduler_connected addr=lb:500515336worker2 # [ 72.552006] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5337worker1 # [ 72.613131] nix-grpc-daemon[781]: ts=2026-09-21T09:27:57Z level=info event=scheduler_connected addr=lb:500515338worker1 # [ 72.616975] nix-grpc-daemon[781]: ts=2026-09-21T09:27:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5339worker2 # [ 72.753233] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:57Z level=info event=scheduler_connected addr=lb:500515340worker2 # [ 72.757133] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5341worker1 # [ 72.790558] nix-grpc-daemon[781]: ts=2026-09-21T09:27:57Z level=info event=scheduler_restarting5342worker1 # [ 72.827862] nix-daemon[845]: reaped child process 847, status = succeeded5343worker1 # [ 72.833170] nix-grpc-daemon[781]: ts=2026-09-21T09:27:57Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5344worker2 # [ 72.958228] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:57Z level=info event=scheduler_connected addr=lb:500515345worker2 # [ 72.962129] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5346worker2 # [ 73.163859] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:57Z level=info event=scheduler_connected addr=lb:500515347worker2 # [ 73.176142] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:57Z level=info event=oidc_jwks provider=mock keys=15348worker2 # [ 73.179938] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:57Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:459805349worker2 # [ 73.183833] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:57Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05350worker1 # [ 73.673634] nix-grpc-daemon[781]: E0000 00:00:1789982878.835291 827 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5351worker1 # [ 73.679239] nix-grpc-daemon[781]: E0000 00:00:1789982878.841267 827 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5352worker1 # [ 74.408723] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5353worker1 # [ 74.411786] systemd[1]: Stopped Nix worker-protocol over gRPC.5354worker1 # [ 74.416507] systemd[1]: nix-grpc-daemon.service: Consumed 1.498s CPU time over 58.225s wall clock time, 51.1M memory peak, 52.5K incoming IP traffic, 94.4K outgoing IP traffic.5355worker1 # [ 74.426868] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5356worker1 # [ 74.430472] systemd[1]: Closed Nix worker-protocol over gRPC.5357worker1 # [ 74.433274] systemd[1]: nix-grpc-daemon.socket: Consumed 299us CPU time over 1min 5.141s wall clock time, 256K memory peak, 63.6K incoming IP traffic, 58K outgoing IP traffic.5358lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5359lb: must succeed: curl -sf localhost:9901/clusters5360lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5361lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.08 seconds)5362client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25363worker2 # [ 74.835810] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:59Z level=debug event=rpc_start method=Connect cn=ci-1 peer=ipv4:192.168.1.2:459905364worker2 # [ 74.848408] nix-daemon[869]: accepted connection from pid 1299, user nix-grpc-daemon (trusted)5365worker2 # [ 74.911324] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:459905366worker2 # [ 74.921355] nix-daemon[869]: accepted connection from pid 1299, user nix-grpc-daemon (trusted)5367worker2 # [ 74.940583] nix-grpc-daemon[1299]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 15 ms (attempt 1/5)5368worker2 # [ 74.961795] nix-grpc-daemon[1299]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 92 ms (attempt 2/5)5369worker2 # [ 75.059603] nix-grpc-daemon[1299]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 94 ms (attempt 3/5)5370worker2 # [ 75.159323] nix-grpc-daemon[1299]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 179 ms (attempt 4/5)5371worker2 # [ 75.344541] nix-grpc-daemon[1299]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5372worker2 # [ 75.348518] nix-grpc-daemon[1299]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5373client # copying 2 paths...5374client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5375worker2 # [ 75.371141] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:459905376client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5377worker2 # [ 75.384177] nix-daemon[869]: accepted connection from pid 1299, user nix-grpc-daemon (trusted)5378worker2 # [ 75.406521] nix-grpc-daemon[1299]: ts=2026-09-21T09:27:59Z level=info event=niks3_push_started pid=13635379worker2 # [ 75.636056] nix-grpc-daemon[1363]: time=2026-09-21T09:27:59.994Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5380worker2 # [ 75.811373] nix-daemon[869]: accepted connection from pid 1376, user nix-grpc-daemon (trusted)5381worker2 # [ 75.824308] nix-daemon[869]: reaped child process 1379, status = succeeded5382lb # [ 75.814371] niks3-server[1002]: 2026/09/21 09:28:00 INFO Received uploads request method=POST path=/api/pending_closures5383lb # [ 75.823995] niks3-server[1002]: 2026/09/21 09:28:00 INFO Received uploads request method=POST path=/api/pending_closures5384worker2 # [ 75.861910] nix-grpc-daemon[1363]: time=2026-09-21T09:28:00.220Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5385worker2 # [ 75.863470] nix-grpc-daemon[1363]: time=2026-09-21T09:28:00.222Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5386worker2 # [ 75.865459] nix-grpc-daemon[1363]: time=2026-09-21T09:28:00.224Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5387lb # [ 75.876745] niks3-server[1002]: 2026/09/21 09:28:00 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5388lb # [ 75.893559] niks3-server[1002]: 2026/09/21 09:28:00 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5389lb # [ 75.902491] niks3-server[1002]: 2026/09/21 09:28:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5390lb # [ 75.904959] niks3-server[1002]: 2026/09/21 09:28:01 INFO Signed narinfos id=25 count=15391lb # [ 75.907425] niks3-server[1002]: 2026/09/21 09:28:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5392lb # [ 75.910981] niks3-server[1002]: 2026/09/21 09:28:01 INFO Signed narinfos id=26 count=25393worker2 # [ 75.942782] nix-grpc-daemon[1363]: time=2026-09-21T09:28:00.300Z level=INFO msg="Uploading 3 narinfos"5394lb # [ 75.916956] niks3-server[1002]: 2026/09/21 09:28:01 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5395lb # [ 75.928060] niks3-server[1002]: 2026/09/21 09:28:01 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5396lb # [ 75.934622] niks3-server[1002]: 2026/09/21 09:28:01 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5397lb # [ 75.939163] niks3-server[1002]: 2026/09/21 09:28:01 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5398lb # [ 75.942618] niks3-server[1002]: 2026/09/21 09:28:01 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5399lb # [ 75.946713] niks3-server[1002]: 2026/09/21 09:28:01 INFO Completed upload id=255400lb # [ 75.948472] niks3-server[1002]: 2026/09/21 09:28:01 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5401lb # [ 75.950718] niks3-server[1002]: 2026/09/21 09:28:01 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5402lb # [ 75.952893] niks3-server[1002]: 2026/09/21 09:28:01 INFO Completed upload id=265403worker2 # [ 75.983862] nix-grpc-daemon[1363]: time=2026-09-21T09:28:00.342Z level=INFO msg="Upload complete. (344ms)"5404worker2 # [ 75.989639] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45990 duration_s=0 paths=2 nar_bytes_in=3125405worker2 # [ 76.010538] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:00Z level=info event=rpc method=Connect cn=ci-1 peer=ipv4:192.168.1.2:45990 duration_s=1 bytes_in=400 bytes_out=2005406worker2 # [ 76.014791] nix-daemon[869]: reaped child process 1361, status = succeeded5407worker2 # [ 76.016611] nix-daemon[869]: reaped child process 1352, status = succeeded5408client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.47 seconds)5409worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5410worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.02 seconds)5411worker1 # [ 76.087723] systemd[1]: Listening on Nix worker-protocol over gRPC.5412worker1 # [ 76.096132] systemd[1]: Starting Nix worker-protocol over gRPC...5413lb # [ 76.202753] envoy[902]: conn peer=[2001:db8:1::1]:39788 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=515414lb # [ 76.207077] envoy[902]: conn peer=[2001:db8:1::1]:39782 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4274 tx=2178 ms=16515415lb # [ 76.210571] envoy[902]: conn peer=[2001:db8:1::4]:34376 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=38217 tx=2345 ms=532255416lb # [ 76.214537] envoy[902]: conn peer=[2001:db8:1::1]:39796 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=165417lb # [ 76.218005] envoy[902]: conn peer=[2001:db8:1::3]:40676 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=10589 tx=2124 ms=553755418lb # [ 76.221618] envoy[902]: conn peer=[2001:db8:1::1]:55550 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1616 tx=616 ms=11705419worker1 # [ 76.292381] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:01Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5420worker1 # [ 76.295056] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:01Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5421worker1 # [ 76.298683] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:01Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15422worker1 # [ 76.306063] nix-grpc-daemon[1049]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5423worker1 # [ 76.307454] nix-grpc-daemon[1049]: E0000 00:00:1789982881.468501 1049 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5424worker1 # [ 76.313155] nix-grpc-daemon[1049]: E0000 00:00:1789982881.473760 1049 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5425worker1 # [ 76.317726] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:01Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5426worker1 # [ 76.320571] systemd[1]: Started Nix worker-protocol over gRPC.5427lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5428lb: must succeed: curl -sf localhost:9901/clusters5429worker1 # [ 76.326086] nix-grpc-daemon[1049]: E0000 00:00:1789982881.488107 1090 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5430worker1 # [ 76.337605] nix-grpc-daemon[1049]: E0000 00:00:1789982881.500042 1090 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5431worker1 # [ 76.343056] nix-grpc-daemon[1049]: E0000 00:00:1789982881.502323 1090 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5432worker1 # [ 76.351563] nix-grpc-daemon[1049]: E0000 00:00:1789982881.514052 1090 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5433worker1 # [ 76.361514] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:01Z level=info event=scheduler_connected addr=lb:500515434worker2 # [ 76.401895] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:00Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:459805435worker2 # [ 76.405660] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:00Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05436lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5437lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.09 seconds)5438(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.03 seconds)5439subtest: scheduler down: the other node takes the lock and keeps it5440lb: must succeed: curl -sf localhost:9901/clusters5441lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5442lb: must succeed: curl -sf localhost:9901/clusters5443lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5444lb: must succeed: curl -sf localhost:9901/clusters5445lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5446lb: must succeed: curl -sf localhost:9901/clusters5447lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5448worker2 # [ 76.735550] systemd[1]: Stopping Nix worker-protocol over gRPC...5449worker2 # [ 77.094446] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:01Z level=info event=signal_exit5450worker2 # [ 77.101373] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:01Z level=info event=scheduler_yield5451lb # [ 77.072861] niks3-server[1002]: 2026/09/21 09:28:02 INFO lead: released remote=192.168.1.4:378125452worker1 # [ 77.070725] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:02Z level=info event=scheduler_restarting addr=lb:500515453worker2 # [ 77.109567] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:01Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:459805454worker2 # [ 77.115672] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:01Z level=info event=worker_gone addr=192.168.1.3:500515455worker2 # [ 77.121318] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:01Z level=info event=scheduler_restarting addr=lb:500515456worker2 # [ 77.124379] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:01Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:459805457worker2 # [ 77.127637] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:01Z level=info event=worker_gone addr=192.168.1.4:500515458worker1 # [ 77.271901] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:02Z level=info event=scheduler_connected addr=lb:500515459worker1 # [ 77.276160] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5460worker2 # [ 77.315932] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:01Z level=info event=scheduler_connected addr=lb:500515461worker2 # [ 77.319696] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5462worker1 # [ 77.477672] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:02Z level=info event=scheduler_connected addr=lb:500515463worker1 # [ 77.481468] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5464worker2 # [ 77.520758] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:01Z level=info event=scheduler_connected addr=lb:500515465worker2 # [ 77.524584] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5466worker1 # [ 77.682300] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:02Z level=info event=scheduler_connected addr=lb:500515467worker1 # [ 77.686160] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5468worker2 # [ 77.725731] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:02Z level=info event=scheduler_connected addr=lb:500515469worker2 # [ 77.729649] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5470worker1 # [ 77.886881] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:03Z level=info event=scheduler_connected addr=lb:500515471worker1 # [ 77.890844] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5472worker2 # [ 77.930746] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:02Z level=info event=scheduler_connected addr=lb:500515473worker2 # [ 77.934576] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5474worker2 # [ 78.109798] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:02Z level=info event=scheduler_restarting5475worker2 # [ 78.119090] nix-grpc-daemon[1299]: E0000 00:00:1789982882.477012 1344 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5476worker2 # [ 78.124477] nix-grpc-daemon[1299]: E0000 00:00:1789982882.483287 1344 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true5477worker1 # [ 78.091766] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:03Z level=info event=scheduler_connected addr=lb:500515478worker1 # [ 78.095841] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5479worker2 # [ 78.146537] nix-grpc-daemon[1299]: ts=2026-09-21T09:28:02Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5480worker2 # [ 78.152682] nix-daemon[869]: reaped child process 1355, status = succeeded5481worker1 # [ 78.297208] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:03Z level=info event=scheduler_connected addr=lb:500515482worker1 # [ 78.301255] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5483worker1 # [ 78.502928] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:03Z level=info event=scheduler_connected addr=lb:500515484worker1 # [ 78.506734] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5485worker1 # [ 78.707883] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:03Z level=info event=scheduler_connected addr=lb:500515486worker1 # [ 78.711883] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5487worker1 # [ 78.912745] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:04Z level=info event=scheduler_connected addr=lb:500515488worker1 # [ 78.916659] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5489worker1 # [ 79.117771] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:04Z level=info event=scheduler_connected addr=lb:500515490worker1 # [ 79.121749] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5491worker1 # [ 79.322578] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:04Z level=info event=scheduler_connected addr=lb:500515492worker1 # [ 79.326484] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5493worker1 # [ 79.528529] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:04Z level=info event=scheduler_connected addr=lb:500515494worker1 # [ 79.532115] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5495worker1 # [ 79.732983] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:04Z level=info event=scheduler_connected addr=lb:500515496worker1 # [ 79.737209] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5497worker1 # [ 79.938143] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:05Z level=info event=scheduler_connected addr=lb:500515498worker1 # [ 79.942284] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5499worker2 # [ 80.110613] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5500worker2 # [ 80.116262] systemd[1]: Stopped Nix worker-protocol over gRPC.5501worker2 # [ 80.118680] systemd[1]: nix-grpc-daemon.service: Consumed 366ms CPU time over 8.230s wall clock time, 34M memory peak, 20.1K incoming IP traffic, 31.1K outgoing IP traffic.5502worker2 # [ 80.129856] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5503worker2 # [ 80.133959] systemd[1]: Closed Nix worker-protocol over gRPC.5504worker2 # [ 80.136412] systemd[1]: nix-grpc-daemon.socket: Consumed 294us CPU time over 9.171s wall clock time, 256K memory peak, 20.2K incoming IP traffic, 15K outgoing IP traffic.5505worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5506worker1 # [ 80.142920] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:05Z level=info event=scheduler_connected addr=lb:500515507worker1 # [ 80.148146] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5508worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds)5509worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5510worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5511worker1 # [ 80.347551] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:05Z level=info event=scheduler_connected addr=lb:500515512worker1 # [ 80.351539] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5513worker1 # [ 80.553626] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:05Z level=info event=scheduler_connected addr=lb:500515514worker1 # [ 80.557172] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5515worker1 # [ 80.758074] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:05Z level=info event=scheduler_connected addr=lb:500515516worker1 # [ 80.762132] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5517worker1 # [ 80.962773] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:06Z level=info event=scheduler_connected addr=lb:500515518worker1 # [ 80.966714] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5519worker1 # [ 81.167587] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:06Z level=info event=scheduler_connected addr=lb:500515520worker1 # [ 81.171527] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5521worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5522worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5523lb # [ 81.332207] niks3-server[1002]: 2026/09/21 09:28:06 INFO lead: acquired remote=192.168.1.3:332325524worker1 # [ 81.328612] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:06Z level=info event=scheduler_take_over5525worker1 # [ 81.372392] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:06Z level=info event=scheduler_connected addr=lb:500515526worker1 # [ 81.376225] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5527worker1 # [ 81.577851] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:06Z level=info event=scheduler_connected addr=lb:500515528worker1 # [ 81.585129] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:06Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:374245529worker1 # [ 81.588694] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:06Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05530worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5531worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5532client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&25533worker1 # [ 82.660323] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:374405534worker1 # [ 82.671926] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5535client # don't know how to build these paths:5536client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5537worker1 # [ 82.686393] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374405538worker1 # [ 82.698477] nix-grpc-daemon[1049]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 59 ms (attempt 1/5)5539worker1 # [ 82.763376] nix-grpc-daemon[1049]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 56 ms (attempt 2/5)5540worker1 # [ 82.819718] nix-grpc-daemon[1049]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 269 ms (attempt 3/5)5541worker1 # [ 83.094784] nix-grpc-daemon[1049]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 511 ms (attempt 4/5)5542worker1 # [ 83.612076] nix-grpc-daemon[1049]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5543worker1 # [ 83.615841] nix-grpc-daemon[1049]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5544worker1 # [ 83.646331] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:08Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=05545worker1 # [ 83.651794] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:08Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=15546worker1 # [ 83.697548] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374405547worker1 # [ 83.710617] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374405548client # copying 1 paths to 'lb:50051'...5549client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...5550worker1 # [ 83.723356] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374405551worker1 # [ 83.735380] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5552worker1 # [ 83.754169] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:08Z level=info event=niks3_push_started pid=11265553worker1 # [ 83.987757] nix-grpc-daemon[1126]: time=2026-09-21T09:28:09.149Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""5554worker1 # [ 84.181094] nix-daemon[845]: accepted connection from pid 1139, user nix-grpc-daemon (trusted)5555worker1 # [ 84.192301] nix-daemon[845]: reaped child process 1142, status = succeeded5556worker1 # [ 84.391354] nix-daemon[845]: accepted connection from pid 1144, user nix-grpc-daemon (trusted)5557worker1 # [ 84.403460] nix-daemon[845]: reaped child process 1147, status = succeeded5558lb # [ 84.432499] niks3-server[1002]: 2026/09/21 09:28:09 INFO Received uploads request method=POST path=/api/pending_closures5559worker1 # [ 84.435253] nix-grpc-daemon[1126]: time=2026-09-21T09:28:09.597Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5560worker1 # [ 84.438547] nix-grpc-daemon[1126]: time=2026-09-21T09:28:09.599Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"5561lb # [ 84.473600] niks3-server[1002]: 2026/09/21 09:28:09 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst5562lb # [ 84.480181] niks3-server[1002]: 2026/09/21 09:28:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5563lb # [ 84.486515] niks3-server[1002]: 2026/09/21 09:28:09 INFO Signed narinfos id=27 count=15564worker1 # [ 84.483079] nix-grpc-daemon[1126]: time=2026-09-21T09:28:09.645Z level=INFO msg="Uploading 1 narinfos"5565lb # [ 84.495150] niks3-server[1002]: 2026/09/21 09:28:09 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls5566lb # [ 84.507099] niks3-server[1002]: 2026/09/21 09:28:09 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5567lb # [ 84.511955] niks3-server[1002]: 2026/09/21 09:28:09 INFO Completed upload id=275568worker1 # [ 84.505776] nix-grpc-daemon[1126]: time=2026-09-21T09:28:09.668Z level=INFO msg="Upload complete. (514ms)"5569lb # [ 84.514306] niks3-server[1002]: 2026/09/21 09:28:09 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo5570worker1 # [ 84.509633] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37440 duration_s=0 paths=1 nar_bytes_in=5205571worker1 # [ 84.517109] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374405572worker1 # [ 84.525617] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5573worker1 # [ 84.533693] nix-daemon[845]: reaped child process 1124, status = succeeded5574worker1 # [ 84.538658] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5575client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...5576client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5577client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5578client # farm-a> LOG-farm-a-on-standby5579worker1 # [ 85.564992] nix-daemon[845]: reaped child process 1151, status = succeeded5580worker1 # [ 85.764614] nix-daemon[845]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5581worker1 # [ 85.776414] nix-daemon[845]: reaped child process 1170, status = succeeded5582lb # [ 85.805362] niks3-server[1002]: 2026/09/21 09:28:10 INFO Received uploads request method=POST path=/api/pending_closures5583worker1 # [ 85.807781] nix-grpc-daemon[1126]: time=2026-09-21T09:28:10.970Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5584worker1 # [ 85.809385] nix-grpc-daemon[1126]: time=2026-09-21T09:28:10.971Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"5585lb # [ 85.841285] niks3-server[1002]: 2026/09/21 09:28:10 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5586lb # [ 85.855467] niks3-server[1002]: 2026/09/21 09:28:10 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5587lb # [ 85.865594] niks3-server[1002]: 2026/09/21 09:28:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5588lb # [ 85.870078] niks3-server[1002]: 2026/09/21 09:28:10 INFO Signed narinfos id=28 count=15589worker1 # [ 85.866240] nix-grpc-daemon[1126]: time=2026-09-21T09:28:11.028Z level=INFO msg="Uploading 1 narinfos"5590lb # [ 85.874111] niks3-server[1002]: 2026/09/21 09:28:10 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls5591lb # [ 85.884229] niks3-server[1002]: 2026/09/21 09:28:10 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5592lb # [ 85.889131] niks3-server[1002]: 2026/09/21 09:28:10 INFO Completed upload id=285593worker1 # [ 85.882616] nix-grpc-daemon[1126]: time=2026-09-21T09:28:11.045Z level=INFO msg="Upload complete. (335ms)"5594lb # [ 85.890963] niks3-server[1002]: 2026/09/21 09:28:10 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo5595worker1 # [ 85.894239] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37440 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=15596worker1 # [ 85.907457] nix-daemon[845]: reaped child process 1149, status = succeeded5597worker1 # [ 85.911605] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:11Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=05598worker1 # [ 85.913609] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:11Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=15599worker1 # [ 85.957221] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374405600worker1 # [ 85.968580] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374405601worker1 # [ 85.982554] nix-grpc-daemon[1126]: time=2026-09-21T09:28:11.145Z level=INFO msg="All 1 paths already cached"5602client # copying 1 paths to 'lb:50051'...5603client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...5604worker1 # [ 85.990613] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374405605worker1 # [ 85.999345] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5606lb # [ 86.202116] envoy[902]: conn peer=[2001:db8:1::4]:60178 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=8603 tx=535 ms=60015607worker1 # [ 86.209920] nix-daemon[845]: accepted connection from pid 1176, user nix-grpc-daemon (trusted)5608worker1 # [ 86.222213] nix-daemon[845]: reaped child process 1179, status = succeeded5609worker1 # [ 86.435404] nix-daemon[845]: accepted connection from pid 1181, user nix-grpc-daemon (trusted)5610worker1 # [ 86.447226] nix-daemon[845]: reaped child process 1184, status = succeeded5611lb # [ 86.475835] niks3-server[1002]: 2026/09/21 09:28:11 INFO Received uploads request method=POST path=/api/pending_closures5612worker1 # [ 86.478262] nix-grpc-daemon[1126]: time=2026-09-21T09:28:11.640Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5613worker1 # [ 86.479691] nix-grpc-daemon[1126]: time=2026-09-21T09:28:11.640Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"5614lb # [ 86.503924] niks3-server[1002]: 2026/09/21 09:28:11 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst5615lb # [ 86.513564] niks3-server[1002]: 2026/09/21 09:28:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5616worker1 # [ 86.512427] nix-grpc-daemon[1126]: time=2026-09-21T09:28:11.674Z level=INFO msg="Uploading 1 narinfos"5617lb # [ 86.519577] niks3-server[1002]: 2026/09/21 09:28:11 INFO Signed narinfos id=29 count=15618lb # [ 86.524563] niks3-server[1002]: 2026/09/21 09:28:11 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls5619lb # [ 86.534914] niks3-server[1002]: 2026/09/21 09:28:11 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5620lb # [ 86.543790] niks3-server[1002]: 2026/09/21 09:28:11 INFO Completed upload id=295621worker1 # [ 86.537114] nix-grpc-daemon[1126]: time=2026-09-21T09:28:11.699Z level=INFO msg="Upload complete. (524ms)"5622lb # [ 86.548086] niks3-server[1002]: 2026/09/21 09:28:11 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo5623worker1 # [ 86.540873] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37440 duration_s=0 paths=1 nar_bytes_in=6405624worker1 # [ 86.545568] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374405625worker1 # [ 86.555767] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5626worker1 # [ 86.559927] nix-daemon[845]: reaped child process 1174, status = succeeded5627worker1 # [ 86.565135] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5628client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...5629client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5630client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5631client # farm-b> LOG-farm-b-on-standby5632worker1 # [ 87.590906] nix-daemon[845]: reaped child process 1188, status = succeeded5633worker1 # [ 87.793629] nix-daemon[845]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5634worker1 # [ 87.805733] nix-daemon[845]: reaped child process 1207, status = succeeded5635lb # [ 87.834735] niks3-server[1002]: 2026/09/21 09:28:12 INFO Received uploads request method=POST path=/api/pending_closures5636worker1 # [ 87.838332] nix-grpc-daemon[1126]: time=2026-09-21T09:28:13.000Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5637worker1 # [ 87.839935] nix-grpc-daemon[1126]: time=2026-09-21T09:28:13.002Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"5638lb # [ 87.866595] niks3-server[1002]: 2026/09/21 09:28:12 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5639lb # [ 87.883729] niks3-server[1002]: 2026/09/21 09:28:12 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst5640lb # [ 87.891365] niks3-server[1002]: 2026/09/21 09:28:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5641lb # [ 87.896360] niks3-server[1002]: 2026/09/21 09:28:12 INFO Signed narinfos id=30 count=15642worker1 # [ 87.892481] nix-grpc-daemon[1126]: time=2026-09-21T09:28:13.054Z level=INFO msg="Uploading 1 narinfos"5643lb # [ 87.904576] niks3-server[1002]: 2026/09/21 09:28:13 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls5644lb # [ 87.912341] niks3-server[1002]: 2026/09/21 09:28:13 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5645lb # [ 87.917305] niks3-server[1002]: 2026/09/21 09:28:13 INFO Completed upload id=305646lb # [ 87.920206] niks3-server[1002]: 2026/09/21 09:28:13 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo5647worker1 # [ 87.912109] nix-grpc-daemon[1126]: time=2026-09-21T09:28:13.074Z level=INFO msg="Upload complete. (339ms)"5648worker1 # [ 87.921255] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37440 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=15649worker1 # [ 87.931631] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:13Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=05650worker1 # [ 87.935214] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:13Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=15651worker1 # [ 87.943946] nix-daemon[845]: reaped child process 1186, status = succeeded5652worker1 # [ 87.976111] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374405653worker1 # [ 87.988647] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374405654worker1 # [ 88.001282] nix-grpc-daemon[1126]: time=2026-09-21T09:28:13.163Z level=INFO msg="All 2 paths already cached"5655client # copying 1 paths to 'lb:50051'...5656client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...5657worker1 # [ 88.010742] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374405658worker1 # [ 88.019358] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5659worker1 # [ 88.231072] nix-daemon[845]: accepted connection from pid 1213, user nix-grpc-daemon (trusted)5660worker1 # [ 88.243698] nix-daemon[845]: reaped child process 1216, status = succeeded5661worker1 # [ 88.449312] nix-daemon[845]: accepted connection from pid 1218, user nix-grpc-daemon (trusted)5662worker1 # [ 88.461613] nix-daemon[845]: reaped child process 1221, status = succeeded5663lb # [ 88.487902] niks3-server[1002]: 2026/09/21 09:28:13 INFO Received uploads request method=POST path=/api/pending_closures5664worker1 # [ 88.489946] nix-grpc-daemon[1126]: time=2026-09-21T09:28:13.652Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5665worker1 # [ 88.491856] nix-grpc-daemon[1126]: time=2026-09-21T09:28:13.654Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"5666lb # [ 88.514597] niks3-server[1002]: 2026/09/21 09:28:13 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst5667lb # [ 88.527221] niks3-server[1002]: 2026/09/21 09:28:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5668worker1 # [ 88.525055] nix-grpc-daemon[1126]: time=2026-09-21T09:28:13.687Z level=INFO msg="Uploading 1 narinfos"5669lb # [ 88.532226] niks3-server[1002]: 2026/09/21 09:28:13 INFO Signed narinfos id=31 count=15670lb # [ 88.538391] niks3-server[1002]: 2026/09/21 09:28:13 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls5671lb # [ 88.549052] niks3-server[1002]: 2026/09/21 09:28:13 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5672worker1 # [ 88.548663] nix-grpc-daemon[1126]: time=2026-09-21T09:28:13.711Z level=INFO msg="Upload complete. (516ms)"5673lb # [ 88.555394] niks3-server[1002]: 2026/09/21 09:28:13 INFO Completed upload id=315674lb # [ 88.557853] niks3-server[1002]: 2026/09/21 09:28:13 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo5675worker1 # [ 88.551382] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37440 duration_s=0 paths=1 nar_bytes_in=7685676worker1 # [ 88.557714] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374405677worker1 # [ 88.568724] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5678worker1 # [ 88.570872] nix-daemon[845]: reaped child process 1211, status = succeeded5679worker1 # [ 88.576121] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5680client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...5681client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5682client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5683client # farm-top> LOG-farm-top-on-standby5684worker1 # [ 89.389800] nix-daemon[845]: reaped child process 1225, status = succeeded5685worker1 # [ 89.581921] nix-daemon[845]: accepted connection from pid 1241, user nix-grpc-daemon (trusted)5686worker1 # [ 89.594218] nix-daemon[845]: reaped child process 1244, status = succeeded5687lb # [ 89.622830] niks3-server[1002]: 2026/09/21 09:28:14 INFO Received uploads request method=POST path=/api/pending_closures5688worker1 # [ 89.625799] nix-grpc-daemon[1126]: time=2026-09-21T09:28:14.788Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5689worker1 # [ 89.627394] nix-grpc-daemon[1126]: time=2026-09-21T09:28:14.789Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"5690lb # [ 89.657149] niks3-server[1002]: 2026/09/21 09:28:14 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst5691lb # [ 89.670943] niks3-server[1002]: 2026/09/21 09:28:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5692lb # [ 89.676480] niks3-server[1002]: 2026/09/21 09:28:14 INFO Signed narinfos id=32 count=15693worker1 # [ 89.672943] nix-grpc-daemon[1126]: time=2026-09-21T09:28:14.835Z level=INFO msg="Uploading 1 narinfos"5694lb # [ 89.684915] niks3-server[1002]: 2026/09/21 09:28:14 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5695lb # [ 89.693463] niks3-server[1002]: 2026/09/21 09:28:14 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls5696lb # [ 89.700578] niks3-server[1002]: 2026/09/21 09:28:14 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5697lb # [ 89.706059] niks3-server[1002]: 2026/09/21 09:28:14 INFO Completed upload id=325698worker1 # [ 89.699514] nix-grpc-daemon[1126]: time=2026-09-21T09:28:14.861Z level=INFO msg="Upload complete. (317ms)"5699lb # [ 89.708870] niks3-server[1002]: 2026/09/21 09:28:14 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo5700worker1 # [ 89.705747] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37440 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=15701worker1 # [ 89.717398] nix-daemon[845]: reaped child process 1223, status = succeeded5702client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 7.36 seconds)5703client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath5704client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.23 seconds)5705worker2 # [ 90.092597] systemd[1]: Listening on Nix worker-protocol over gRPC.5706worker2 # [ 90.100382] systemd[1]: Starting scheduler-token.service...5707worker2 # [ 90.178577] systemd[1]: scheduler-token.service: Deactivated successfully.5708worker2 # [ 90.179669] systemd[1]: Finished scheduler-token.service.5709worker2 # [ 90.182235] systemd[1]: scheduler-token.service: Consumed 18ms CPU time over 79ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5710worker2 # [ 90.186768] systemd[1]: Starting Nix worker-protocol over gRPC...5711worker2 # [ 90.370808] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:14Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5712worker2 # [ 90.372605] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:14Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5713worker2 # [ 90.376473] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:14Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15714worker2 # [ 90.384068] nix-grpc-daemon[1403]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5715worker2 # [ 90.385441] nix-grpc-daemon[1403]: E0000 00:00:1789982894.742190 1403 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5716worker2 # [ 90.391068] nix-grpc-daemon[1403]: E0000 00:00:1789982894.747450 1403 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5717worker2 # [ 90.394248] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:14Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5718worker2 # [ 90.399088] systemd[1]: Started Nix worker-protocol over gRPC.5719lb: must succeed: curl -sf localhost:9901/clusters5720worker2 # [ 90.407116] nix-grpc-daemon[1403]: E0000 00:00:1789982894.765967 1445 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5721worker2 # [ 90.419130] nix-grpc-daemon[1403]: E0000 00:00:1789982894.778132 1445 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5722worker2 # [ 90.424054] nix-grpc-daemon[1403]: E0000 00:00:1789982894.780475 1445 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5723worker2 # [ 90.433284] nix-grpc-daemon[1403]: E0000 00:00:1789982894.792321 1445 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5724worker2 # [ 90.443216] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:14Z level=info event=scheduler_connected addr=lb:500515725worker1 # [ 90.414163] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:15Z level=info event=oidc_jwks provider=mock keys=15726worker1 # [ 90.418079] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:15Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:374245727worker1 # [ 90.423459] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:15Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05728lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5729worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5730worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5731lb: must succeed: curl -sf localhost:9901/clusters5732lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5733lb: must succeed: curl -sf localhost:9901/clusters5734lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5735worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5736worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5737(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 14.31 seconds)5738subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5739client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs5740client # [ 90.802456] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.5741client # Running as unit: rs.service; invocation ID: f98030823ee84b36a8d8ec1f6fe3ca825742client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.07 seconds)5743worker1 # [ 91.003705] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:374405744client # [ 91.037522] nix[1192]: don't know how to build these paths:5745client # [ 91.039092] nix[1192]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5746worker1 # [ 91.018622] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:16Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=05747worker1 # [ 91.023585] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:16Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=15748worker1 # [ 91.063544] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374405749worker1 # [ 91.076175] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374405750client # [ 91.114335] nix[1192]: copying 1 paths to 'lb:50051'...5751worker1 # [ 91.087559] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374405752client # [ 91.117159] nix[1192]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...5753worker1 # [ 91.100801] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5754worker1 # [ 91.308867] nix-daemon[845]: accepted connection from pid 1257, user nix-grpc-daemon (trusted)5755worker1 # [ 91.321471] nix-daemon[845]: reaped child process 1260, status = succeeded5756worker1 # [ 91.520478] nix-daemon[845]: accepted connection from pid 1262, user nix-grpc-daemon (trusted)5757worker1 # [ 91.531321] nix-daemon[845]: reaped child process 1265, status = succeeded5758lb # [ 91.558841] niks3-server[1002]: 2026/09/21 09:28:16 INFO Received uploads request method=POST path=/api/pending_closures5759worker1 # [ 91.560601] nix-grpc-daemon[1126]: time=2026-09-21T09:28:16.722Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5760worker1 # [ 91.562452] nix-grpc-daemon[1126]: time=2026-09-21T09:28:16.724Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"5761lb # [ 91.587733] niks3-server[1002]: 2026/09/21 09:28:16 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst5762lb # [ 91.602769] niks3-server[1002]: 2026/09/21 09:28:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5763worker1 # [ 91.600905] nix-grpc-daemon[1126]: time=2026-09-21T09:28:16.763Z level=INFO msg="Uploading 1 narinfos"5764lb # [ 91.608179] niks3-server[1002]: 2026/09/21 09:28:16 INFO Signed narinfos id=33 count=15765lb # [ 91.614553] niks3-server[1002]: 2026/09/21 09:28:16 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls5766lb # [ 91.624455] niks3-server[1002]: 2026/09/21 09:28:16 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5767worker1 # [ 91.624230] nix-grpc-daemon[1126]: time=2026-09-21T09:28:16.786Z level=INFO msg="Upload complete. (511ms)"5768lb # [ 91.630913] niks3-server[1002]: 2026/09/21 09:28:16 INFO Completed upload id=335769lb # [ 91.633698] niks3-server[1002]: 2026/09/21 09:28:16 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo5770worker1 # [ 91.626279] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37440 duration_s=0 paths=1 nar_bytes_in=4165771worker1 # [ 91.631315] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374405772worker1 # [ 91.640484] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5773worker1 # [ 91.645731] nix-daemon[845]: reaped child process 1255, status = succeeded5774worker1 # [ 91.651085] nix-daemon[845]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)5775client # [ 91.678528] nix[1192]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...5776lb: must succeed: curl -sf localhost:9901/clusters5777lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5778lb: must succeed: curl -sf localhost:9901/clusters5779lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5780lb: must succeed: curl -sf localhost:9901/clusters5781lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5782lb: must succeed: curl -sf localhost:9901/clusters5783lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5784worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5785worker2 # [ 93.788043] systemd-journald[287]: Received client request to rotate journal, rotating.5786worker2 # [ 93.807987] systemd-journald[287]: Vacuuming done, freed 0B of archived journals from /var/log/journal/d5b80479449b4deb89390529ab44d64c.5787worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.07 seconds)5788worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5789worker1 # [ 93.689523] systemd[1]: Stopping Nix worker-protocol over gRPC...5790worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)5791client: must succeed: sleep 5.05792worker1 # [ 94.320166] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:19Z level=info event=draining5793lb # [ 96.202670] envoy[902]: conn peer=[2001:db8:1::1]:55558 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9633 tx=4143 ms=70675794client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5795worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5796worker1 # [ 99.078649] nix-daemon[845]: reaped child process 1269, status = succeeded5797worker1 # [ 99.277602] nix-daemon[845]: accepted connection from pid 1310, user nix-grpc-daemon (trusted)5798worker1 # [ 99.288635] nix-daemon[845]: reaped child process 1313, status = succeeded5799lb # [ 99.318839] niks3-server[1002]: 2026/09/21 09:28:24 INFO Received uploads request method=POST path=/api/pending_closures5800worker1 # [ 99.320919] nix-grpc-daemon[1126]: time=2026-09-21T09:28:24.483Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5801worker1 # [ 99.322740] nix-grpc-daemon[1126]: time=2026-09-21T09:28:24.485Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"5802lb # [ 99.365973] niks3-server[1002]: 2026/09/21 09:28:24 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5803lb # [ 99.372551] niks3-server[1002]: 2026/09/21 09:28:24 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5804lb # [ 99.379810] niks3-server[1002]: 2026/09/21 09:28:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5805lb # [ 99.383723] niks3-server[1002]: 2026/09/21 09:28:24 INFO Signed narinfos id=34 count=15806worker1 # [ 99.379763] nix-grpc-daemon[1126]: time=2026-09-21T09:28:24.542Z level=INFO msg="Uploading 1 narinfos"5807lb # [ 99.393565] niks3-server[1002]: 2026/09/21 09:28:24 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls5808lb # [ 99.397866] niks3-server[1002]: 2026/09/21 09:28:24 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5809lb # [ 99.401749] niks3-server[1002]: 2026/09/21 09:28:24 INFO Completed upload id=345810lb # [ 99.403729] niks3-server[1002]: 2026/09/21 09:28:24 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo5811worker1 # [ 99.396631] nix-grpc-daemon[1126]: time=2026-09-21T09:28:24.559Z level=INFO msg="Upload complete. (332ms)"5812worker1 # [ 99.404581] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37440 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=15813worker1 # [ 99.417882] nix-daemon[845]: reaped child process 1267, status = succeeded5814client # [ 99.474827] systemd[1]: rs.service: Deactivated successfully.5815client # [ 99.479204] systemd[1]: rs.service: Consumed 131ms CPU time over 8.675s wall clock time, 43.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.5816worker1 # [ 100.323293] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:25Z level=info event=signal_exit5817worker1 # [ 100.546849] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:25Z level=info event=scheduler_yield5818lb # [ 100.554946] niks3-server[1002]: 2026/09/21 09:28:25 INFO lead: released remote=192.168.1.3:332325819worker1 # [ 100.552272] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:25Z level=info event=scheduler_restarting addr=lb:500515820worker2 # [ 100.590873] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:24Z level=info event=scheduler_restarting addr=lb:500515821worker1 # [ 100.557147] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:25Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:374245822worker1 # [ 100.561405] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:25Z level=info event=worker_gone addr=192.168.1.3:500515823worker1 # [ 100.564468] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:25Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:374245824worker1 # [ 100.567741] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:25Z level=info event=worker_gone addr=192.168.1.4:500515825worker1 # [ 100.753469] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:25Z level=info event=scheduler_connected addr=lb:500515826worker2 # [ 100.791295] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:25Z level=info event=scheduler_connected addr=lb:500515827worker1 # [ 100.757681] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5828worker2 # [ 100.795251] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5829worker1 # [ 100.958569] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:26Z level=info event=scheduler_connected addr=lb:500515830worker2 # [ 100.996203] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:25Z level=info event=scheduler_connected addr=lb:500515831worker2 # [ 100.999902] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5832worker1 # [ 100.962651] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5833worker2 # [ 101.200767] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:25Z level=info event=scheduler_connected addr=lb:500515834worker1 # [ 101.163592] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:26Z level=info event=scheduler_connected addr=lb:500515835worker2 # [ 101.204852] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5836worker1 # [ 101.167645] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5837worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.18 seconds)5838client: waiting for success: ! systemctl is-active rs5839client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)5840client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5841client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.04 seconds)5842client: must fail: journalctl -u rs | grep -q 'reconnecting'5843client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5844lb: must succeed: curl -sf localhost:9901/clusters5845worker2 # [ 101.405490] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:25Z level=info event=scheduler_connected addr=lb:500515846worker1 # [ 101.368566] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:26Z level=info event=scheduler_connected addr=lb:500515847worker2 # [ 101.409201] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5848worker1 # [ 101.372568] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5849lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5850worker1 # [ 101.551473] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:26Z level=info event=scheduler_restarting5851worker2 # [ 101.611098] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:25Z level=info event=scheduler_connected addr=lb:500515852worker2 # [ 101.614870] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5853worker1 # [ 101.585422] nix-daemon[845]: reaped child process 1116, status = succeeded5854worker1 # [ 101.588795] nix-grpc-daemon[1049]: ts=2026-09-21T09:28:26Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5855worker2 # [ 101.816208] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:26Z level=info event=scheduler_connected addr=lb:500515856worker2 # [ 101.820067] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5857worker2 # [ 102.021252] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:26Z level=info event=scheduler_connected addr=lb:500515858worker2 # [ 102.025165] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5859worker2 # [ 102.226214] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:26Z level=info event=scheduler_connected addr=lb:500515860worker2 # [ 102.230131] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5861worker2 # [ 102.432716] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:26Z level=info event=scheduler_connected addr=lb:500515862worker2 # [ 102.436853] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5863lb: must succeed: curl -sf localhost:9901/clusters5864worker1 # [ 102.440427] nix-grpc-daemon[1049]: E0000 00:00:1789982907.602783 1094 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5865worker1 # [ 102.445788] nix-grpc-daemon[1049]: E0000 00:00:1789982907.608208 1094 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5866lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5867worker1 # [ 102.568524] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5868worker1 # [ 102.570594] systemd[1]: Stopped Nix worker-protocol over gRPC.5869worker1 # [ 102.572280] systemd[1]: nix-grpc-daemon.service: Consumed 1.339s CPU time over 26.471s wall clock time, 35.9M memory peak, 50.4K incoming IP traffic, 100.1K outgoing IP traffic.5870worker1 # [ 102.587339] systemd[1]: Starting Nix worker-protocol over gRPC...5871worker2 # [ 102.638346] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:26Z level=info event=scheduler_connected addr=lb:500515872worker2 # [ 102.641953] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5873worker1 # [ 102.798577] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:27Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5874worker1 # [ 102.800415] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:27Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5875worker1 # [ 102.803879] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:27Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15876worker2 # [ 102.843163] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:27Z level=info event=scheduler_connected addr=lb:500515877worker2 # [ 102.846994] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5878worker1 # [ 102.812067] nix-grpc-daemon[1320]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5879worker1 # [ 102.813365] nix-grpc-daemon[1320]: E0000 00:00:1789982907.974202 1320 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5880worker1 # [ 102.818448] nix-grpc-daemon[1320]: E0000 00:00:1789982907.979356 1320 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5881worker1 # [ 102.821115] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:27Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5882worker1 # [ 102.822860] systemd[1]: Started Nix worker-protocol over gRPC.5883worker1 # [ 102.828725] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:27Z level=info event=scheduler_take_over5884lb # [ 102.836249] niks3-server[1002]: 2026/09/21 09:28:27 INFO lead: acquired remote=192.168.1.3:428725885worker1 # [ 102.833064] nix-grpc-daemon[1320]: E0000 00:00:1789982907.994135 1362 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5886worker1 # [ 102.843600] nix-grpc-daemon[1320]: E0000 00:00:1789982908.006074 1362 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5887worker1 # [ 102.848053] nix-grpc-daemon[1320]: E0000 00:00:1789982908.008364 1362 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5888worker1 # [ 102.857715] nix-grpc-daemon[1320]: E0000 00:00:1789982908.020198 1362 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5889worker1 # [ 102.867129] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=scheduler_connected addr=lb:500515890worker1 # [ 102.868663] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5891worker2 # [ 103.048442] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:27Z level=info event=scheduler_connected addr=lb:500515892worker2 # [ 103.052247] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5893worker1 # [ 103.069986] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=scheduler_connected addr=lb:500515894worker1 # [ 103.074179] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5895worker2 # [ 103.253403] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:27Z level=info event=scheduler_connected addr=lb:500515896worker2 # [ 103.257165] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5897worker1 # [ 103.274980] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=scheduler_connected addr=lb:500515898worker1 # [ 103.279181] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5899worker2 # [ 103.457903] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:27Z level=info event=scheduler_connected addr=lb:500515900worker2 # [ 103.462166] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5901worker1 # [ 103.480140] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=scheduler_connected addr=lb:500515902worker1 # [ 103.484227] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5903lb: must succeed: curl -sf localhost:9901/clusters5904lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5905worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5906worker2 # [ 103.664075] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:28Z level=info event=scheduler_connected addr=lb:500515907worker1 # [ 103.639331] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=oidc_jwks provider=mock keys=15908worker1 # [ 103.642431] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:400965909worker1 # [ 103.646418] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05910worker1 # [ 103.684386] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=scheduler_connected addr=lb:500515911worker1 # [ 103.686306] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:400965912worker1 # [ 103.687976] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:28Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05913worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds)5914(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.98 seconds)5915subtest: scheduler restart mid-build: build finishes, no double build5916client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid5917client # [ 103.782914] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.5918client # Running as unit: mid.service; invocation ID: a51ac793720943ad8142cc070ca1d2755919client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)5920worker1 # [ 103.999658] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:29Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:401125921worker1 # [ 104.007638] nix-daemon[845]: accepted connection from pid 1320, user nix-grpc-daemon (trusted)5922client # [ 104.050520] nix[1228]: don't know how to build these paths:5923client # [ 104.051510] nix[1228]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv5924worker1 # [ 104.030575] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:29Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=05925worker1 # [ 104.034886] nix-grpc-daemon[1320]: ts=2026-09-21T09:28:29Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=15926worker2 # [ 104.130566] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374205927worker2 # [ 104.141743] nix-daemon[869]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)5928worker2 # [ 104.160106] nix-grpc-daemon[1403]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 43 ms (attempt 1/5)5929worker2 # [ 104.209106] nix-grpc-daemon[1403]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 165 ms (attempt 2/5)5930worker2 # [ 104.379997] nix-grpc-daemon[1403]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 182 ms (attempt 3/5)5931worker2 # [ 104.568149] nix-grpc-daemon[1403]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 346 ms (attempt 4/5)5932worker2 # [ 104.920887] nix-grpc-daemon[1403]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5933worker2 # [ 104.924896] nix-grpc-daemon[1403]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5934worker2 # [ 104.941656] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374205935client # [ 104.943189] nix[1228]: copying 1 paths to 'lb:50051'...5936client # [ 104.947173] nix[1228]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...5937worker2 # [ 104.956840] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374205938worker2 # [ 104.967270] nix-daemon[869]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)5939worker2 # [ 104.988144] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:29Z level=info event=niks3_push_started pid=15175940worker2 # [ 105.219637] nix-grpc-daemon[1517]: time=2026-09-21T09:28:29.578Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5941worker2 # [ 105.432699] nix-daemon[869]: accepted connection from pid 1534, user nix-grpc-daemon (trusted)5942worker2 # [ 105.444268] nix-daemon[869]: reaped child process 1537, status = succeeded5943worker2 # [ 105.670450] nix-daemon[869]: accepted connection from pid 1539, user nix-grpc-daemon (trusted)5944worker2 # [ 105.680701] nix-daemon[869]: reaped child process 1542, status = succeeded5945lb # [ 105.673438] niks3-server[1002]: 2026/09/21 09:28:30 INFO Received uploads request method=POST path=/api/pending_closures5946worker2 # [ 105.715417] nix-grpc-daemon[1517]: time=2026-09-21T09:28:30.074Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5947worker2 # [ 105.716941] nix-grpc-daemon[1517]: time=2026-09-21T09:28:30.076Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"5948lb # [ 105.708988] niks3-server[1002]: 2026/09/21 09:28:30 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst5949lb # [ 105.719776] niks3-server[1002]: 2026/09/21 09:28:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5950lb # [ 105.726135] niks3-server[1002]: 2026/09/21 09:28:30 INFO Signed narinfos id=35 count=15951worker2 # [ 105.757265] nix-grpc-daemon[1517]: time=2026-09-21T09:28:30.116Z level=INFO msg="Uploading 1 narinfos"5952lb # [ 105.736061] niks3-server[1002]: 2026/09/21 09:28:30 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5953lb # [ 105.746786] niks3-server[1002]: 2026/09/21 09:28:30 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo5954lb # [ 105.751318] niks3-server[1002]: 2026/09/21 09:28:30 INFO Completed upload id=355955worker2 # [ 105.781933] nix-grpc-daemon[1517]: time=2026-09-21T09:28:30.140Z level=INFO msg="Upload complete. (557ms)"5956worker2 # [ 105.784970] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37420 duration_s=0 paths=1 nar_bytes_in=4245957lb # [ 105.759279] niks3-server[1002]: 2026/09/21 09:28:30 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls5958worker2 # [ 105.791587] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374205959worker2 # [ 105.798406] nix-daemon[869]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)5960worker2 # [ 105.804136] nix-daemon[869]: reaped child process 1515, status = succeeded5961worker2 # [ 105.808357] nix-daemon[869]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)5962client # [ 105.798994] nix[1228]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...5963lb # [ 106.203576] envoy[902]: conn peer=[2001:db8:1::1]:41512 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2960 tx=1427 ms=84095964lb # [ 106.207262] envoy[902]: conn peer=[2001:db8:1::3]:44496 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=6563 tx=1478 ms=252195965lb: must succeed: curl -sf localhost:9901/clusters5966lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5967lb: must succeed: curl -sf localhost:9901/clusters5968lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5969worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5970worker1 # [ 107.677655] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1320 (nix-grpc-daemon) on client request.5971lb # [ 107.701424] niks3-server[1002]: 2026/09/21 09:28:32 INFO lead: released remote=192.168.1.3:428725972worker1 # [ 107.692140] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5973worker2 # [ 107.737549] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:32Z level=info event=scheduler_disconnected addr=lb:50051 error="Stream removed (RST_STREAM (Received RST_STREAM with error code 2))"5974client # [ 107.730234] nix[1228]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5975worker1 # [ 107.707478] nix-daemon[845]: reaped child process 1379, status = succeeded5976worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.08 seconds)5977client: must succeed: sleep 5.05978worker1 # [ 107.723813] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5979worker1 # [ 107.725552] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5980worker1 # [ 107.727179] systemd[1]: nix-grpc-daemon.service: Consumed 161ms CPU time over 5.135s wall clock time, 13.1M memory peak, 10K incoming IP traffic, 13.7K outgoing IP traffic.5981worker1 # [ 107.869240] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1.5982worker1 # [ 107.878789] systemd[1]: Starting Nix worker-protocol over gRPC...5983worker2 # [ 107.937915] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:32Z level=info event=scheduler_connected addr=lb:500515984worker1 # [ 108.109426] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:33Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5985worker1 # [ 108.111333] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:33Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5986worker1 # [ 108.115370] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:33Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15987worker1 # [ 108.123076] nix-grpc-daemon[1407]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5988worker1 # [ 108.125945] nix-grpc-daemon[1407]: E0000 00:00:1789982913.285311 1407 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5989worker1 # [ 108.130132] nix-grpc-daemon[1407]: E0000 00:00:1789982913.288302 1407 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5990worker1 # [ 108.133697] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:33Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5991worker1 # [ 108.136186] systemd[1]: Started Nix worker-protocol over gRPC.5992worker2 # [ 108.177230] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:32Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5993worker1 # [ 108.143686] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:33Z level=info event=scheduler_take_over5994lb # [ 108.150357] niks3-server[1002]: 2026/09/21 09:28:33 INFO lead: acquired remote=192.168.1.3:428925995worker1 # [ 108.146139] nix-grpc-daemon[1407]: E0000 00:00:1789982913.306469 1448 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5996worker1 # [ 108.156910] nix-grpc-daemon[1407]: E0000 00:00:1789982913.319390 1448 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5997worker1 # [ 108.160427] nix-grpc-daemon[1407]: E0000 00:00:1789982913.321664 1448 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5998worker1 # [ 108.171053] nix-grpc-daemon[1407]: E0000 00:00:1789982913.333556 1448 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5999worker1 # [ 108.180118] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:33Z level=info event=scheduler_connected addr=lb:500516000worker1 # [ 108.181823] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:33Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:401226001worker1 # [ 108.183326] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:33Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06002worker2 # [ 108.379835] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:32Z level=info event=scheduler_connected addr=lb:500516003worker1 # [ 108.351168] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:33Z level=info event=oidc_jwks provider=mock keys=16004worker1 # [ 108.354319] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:33Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:401226005worker1 # [ 108.358637] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:33Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=16006client: (finished: must succeed: sleep 5.0, in 5.06 seconds)6007client: waiting for success: ! systemctl is-active mid6008worker2 # [ 113.237837] nix-daemon[869]: reaped child process 1546, status = succeeded6009worker2 # [ 113.429725] nix-daemon[869]: accepted connection from pid 1583, user nix-grpc-daemon (trusted)6010worker2 # [ 113.441352] nix-daemon[869]: reaped child process 1586, status = succeeded6011lb # [ 113.432495] niks3-server[1002]: 2026/09/21 09:28:38 INFO Received uploads request method=POST path=/api/pending_closures6012worker2 # [ 113.473361] nix-grpc-daemon[1517]: time=2026-09-21T09:28:37.832Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6013worker2 # [ 113.474901] nix-grpc-daemon[1517]: time=2026-09-21T09:28:37.833Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6014lb # [ 113.470677] niks3-server[1002]: 2026/09/21 09:28:38 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6015lb # [ 113.482148] niks3-server[1002]: 2026/09/21 09:28:38 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6016lb # [ 113.491636] niks3-server[1002]: 2026/09/21 09:28:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign6017lb # [ 113.495436] niks3-server[1002]: 2026/09/21 09:28:38 INFO Signed narinfos id=36 count=16018worker2 # [ 113.528879] nix-grpc-daemon[1517]: time=2026-09-21T09:28:37.887Z level=INFO msg="Uploading 1 narinfos"6019lb # [ 113.501300] niks3-server[1002]: 2026/09/21 09:28:38 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6020lb # [ 113.509528] niks3-server[1002]: 2026/09/21 09:28:38 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete6021lb # [ 113.512569] niks3-server[1002]: 2026/09/21 09:28:38 INFO Completed upload id=366022lb # [ 113.514468] niks3-server[1002]: 2026/09/21 09:28:38 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6023worker2 # [ 113.544499] nix-grpc-daemon[1517]: time=2026-09-21T09:28:37.903Z level=INFO msg="Upload complete. (323ms)"6024worker2 # [ 113.549442] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37420 duration_s=7 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16025worker2 # [ 113.561879] nix-daemon[869]: reaped child process 1544, status = succeeded6026client # [ 113.581783] systemd[1]: mid.service: Deactivated successfully.6027client # [ 113.586225] systemd[1]: mid.service: Consumed 140ms CPU time over 9.802s wall clock time, 41.4M memory peak, 7.7K incoming IP traffic, 8.8K outgoing IP traffic.6028client: (finished: waiting for success: ! systemctl is-active mid, in 1.12 seconds)6029client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6030client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds)6031(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.51 seconds)6032subtest: interrupting the client stops the build on the worker6033client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6034client # [ 114.262855] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6035client # Running as unit: intr.service; invocation ID: 2bc995535acc4c9e95c41b106943f6206036client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.04 seconds)6037worker2 # [ 114.477890] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:374206038client # [ 114.474890] nix[1264]: don't know how to build these paths:6039client # [ 114.475925] nix[1264]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6040worker1 # [ 114.455856] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:39Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06041worker1 # [ 114.461629] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:39Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.3:50051 clients=16042worker1 # [ 114.507978] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106043worker1 # [ 114.519758] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6044worker1 # [ 114.537788] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 17 ms (attempt 1/5)6045worker1 # [ 114.560720] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 140 ms (attempt 2/5)6046worker1 # [ 114.701219] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 361 ms (attempt 3/5)6047worker1 # [ 115.068074] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 571 ms (attempt 4/5)6048worker1 # [ 115.645797] nix-grpc-daemon[1407]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6049worker1 # [ 115.650336] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6050worker1 # [ 115.668635] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106051client # [ 115.708827] nix[1264]: copying 1 paths to 'lb:50051'...6052client # [ 115.712872] nix[1264]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6053worker1 # [ 115.685110] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106054worker1 # [ 115.697137] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6055worker1 # [ 115.718369] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:40Z level=info event=niks3_push_started pid=14856056worker1 # [ 115.967252] nix-grpc-daemon[1485]: time=2026-09-21T09:28:41.129Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6057worker1 # [ 116.178587] nix-daemon[845]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6058worker1 # [ 116.189397] nix-daemon[845]: reaped child process 1501, status = succeeded6059lb # [ 116.204171] envoy[902]: conn peer=[2001:db8:1::3]:59692 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1292 tx=385 ms=48376060lb # [ 116.208454] envoy[902]: conn peer=[2001:db8:1::1]:53140 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3061 tx=1521 ms=95306061worker1 # [ 116.411460] nix-daemon[845]: accepted connection from pid 1503, user nix-grpc-daemon (trusted)6062worker1 # [ 116.421948] nix-daemon[845]: reaped child process 1506, status = succeeded6063lb # [ 116.452392] niks3-server[1002]: 2026/09/21 09:28:41 INFO Received uploads request method=POST path=/api/pending_closures6064worker1 # [ 116.454555] nix-grpc-daemon[1485]: time=2026-09-21T09:28:41.617Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6065worker1 # [ 116.456121] nix-grpc-daemon[1485]: time=2026-09-21T09:28:41.618Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6066lb # [ 116.489213] niks3-server[1002]: 2026/09/21 09:28:41 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6067lb # [ 116.495271] niks3-server[1002]: 2026/09/21 09:28:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6068lb # [ 116.499950] niks3-server[1002]: 2026/09/21 09:28:41 INFO Signed narinfos id=37 count=16069worker1 # [ 116.495382] nix-grpc-daemon[1485]: time=2026-09-21T09:28:41.657Z level=INFO msg="Uploading 1 narinfos"6070lb # [ 116.508188] niks3-server[1002]: 2026/09/21 09:28:41 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6071lb # [ 116.516411] niks3-server[1002]: 2026/09/21 09:28:41 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6072lb # [ 116.520470] niks3-server[1002]: 2026/09/21 09:28:41 INFO Completed upload id=376073worker1 # [ 116.513929] nix-grpc-daemon[1485]: time=2026-09-21T09:28:41.676Z level=INFO msg="Upload complete. (542ms)"6074lb # [ 116.522425] niks3-server[1002]: 2026/09/21 09:28:41 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6075worker1 # [ 116.516568] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=4246076worker1 # [ 116.520516] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106077worker1 # [ 116.526813] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6078worker1 # [ 116.529227] nix-daemon[845]: reaped child process 1483, status = succeeded6079worker1 # [ 116.536142] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6080client # [ 116.563260] nix[1264]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6081client: must succeed: systemctl kill -s INT intr6082worker1 # [ 117.844235] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:43Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6083client # [ 117.869933] systemd[1]: intr.service: Sent signal SIGINT to main process 1264 (nix) on client request.6084client # [ 117.878546] nix[1264]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6085client # [ 117.885176] nix[1264]: error: interrupted by the user6086client: (finished: must succeed: systemctl kill -s INT intr, in 0.06 seconds)6087client # [ 117.916950] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6088client # [ 117.918640] systemd[1]: intr.service: Failed with result 'exit-code'.6089client # [ 117.920802] systemd[1]: intr.service: Consumed 120ms CPU time over 3.657s wall clock time, 41.4M memory peak, 6.5K incoming IP traffic, 8.3K outgoing IP traffic.6090worker1 # [ 117.954359] nix-daemon[845]: reaped child process 1508, status = succeeded6091worker1 # [ 117.958243] nix-daemon[1513]: unexpected Nix daemon error: error: interrupted by the user6092worker1 # [ 117.961918] nix-daemon[845]: reaped child process 1513, status = failed with exit code 16093(finished: subtest: interrupting the client stops the build on the worker, in 4.93 seconds)6094subtest: a deploy mid-build drains: the build finishes, the new generation takes over6095client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6096client # [ 119.225868] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6097client # Running as unit: sw.service; invocation ID: d9d0660fda9e4b0c9e1534cc0307d5916098client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.07 seconds)6099worker2 # [ 119.485889] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:374206100client # [ 119.483077] nix[1285]: don't know how to build these paths:6101client # [ 119.484126] nix[1285]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6102worker1 # [ 119.467816] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:44Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06103worker1 # [ 119.472604] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:44Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.4:50051 clients=16104worker2 # [ 119.553971] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374206105worker2 # [ 119.567624] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374206106client # [ 119.568181] nix[1285]: copying 1 paths to 'lb:50051'...6107client # [ 119.571678] nix[1285]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6108worker2 # [ 119.581833] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374206109worker2 # [ 119.593287] nix-daemon[869]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)6110worker2 # [ 119.808346] nix-daemon[869]: accepted connection from pid 1618, user nix-grpc-daemon (trusted)6111worker2 # [ 119.819925] nix-daemon[869]: reaped child process 1621, status = succeeded6112worker2 # [ 120.015293] nix-daemon[869]: accepted connection from pid 1623, user nix-grpc-daemon (trusted)6113worker2 # [ 120.026696] nix-daemon[869]: reaped child process 1626, status = succeeded6114lb # [ 120.018286] niks3-server[1002]: 2026/09/21 09:28:45 INFO Received uploads request method=POST path=/api/pending_closures6115worker2 # [ 120.058142] nix-grpc-daemon[1517]: time=2026-09-21T09:28:44.416Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6116worker2 # [ 120.059609] nix-grpc-daemon[1517]: time=2026-09-21T09:28:44.416Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6117lb # [ 120.044232] niks3-server[1002]: 2026/09/21 09:28:45 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6118lb # [ 120.055604] niks3-server[1002]: 2026/09/21 09:28:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6119worker2 # [ 120.091344] nix-grpc-daemon[1517]: time=2026-09-21T09:28:44.450Z level=INFO msg="Uploading 1 narinfos"6120lb # [ 120.061572] niks3-server[1002]: 2026/09/21 09:28:45 INFO Signed narinfos id=38 count=16121lb # [ 120.066680] niks3-server[1002]: 2026/09/21 09:28:45 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6122lb # [ 120.078178] niks3-server[1002]: 2026/09/21 09:28:45 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6123worker2 # [ 120.114310] nix-grpc-daemon[1517]: time=2026-09-21T09:28:44.473Z level=INFO msg="Upload complete. (509ms)"6124lb # [ 120.083931] niks3-server[1002]: 2026/09/21 09:28:45 INFO Completed upload id=386125lb # [ 120.087676] niks3-server[1002]: 2026/09/21 09:28:45 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6126worker2 # [ 120.117329] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37420 duration_s=0 paths=1 nar_bytes_in=4166127worker2 # [ 120.122559] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374206128worker2 # [ 120.133183] nix-daemon[869]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)6129worker2 # [ 120.135136] nix-daemon[869]: reaped child process 1616, status = succeeded6130worker2 # [ 120.140360] nix-daemon[869]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)6131client # [ 120.130748] nix[1285]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6132worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&26133worker2 # Checking switch inhibitors... done6134worker2 # [ 122.691965] nixos[1662]: switching to system configuration /nix/store/32111b3ip9k1svh21sms1psjc3nd6q0y-nixos-system-worker2-test6135worker2 # [ 122.694713] systemd[1]: Stopped target Local File Systems.6136worker2 # activating the configuration...6137worker2 # [ 122.696668] systemd[1]: Stopped target Remote File Systems.6138worker2 # setting up /etc...6139worker2 # [ 123.470523] systemd[1]: Reload requested from client PID 1662 ('.switch-to-conf') (unit backdoor.service)...6140worker2 # [ 123.472201] systemd[1]: Reloading...6141worker2 # [ 124.271914] systemd[1]: Reloading finished in 797 ms.6142worker2 # restarting sysinit-reactivation.target6143worker2 # [ 124.328715] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....6144worker2 # [ 124.466441] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..6145worker2 # [ 124.467623] systemd[1]: Reached target Reactivate sysinit units.6146worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service6147worker2 # [ 124.473066] systemd[1]: Reloading D-Bus System Message Bus...6148worker2 # [ 124.475169] dbus-broker-launch[533]: Caught SIGHUP, trigger reload.6149worker2 # [ 124.479272] systemd[1]: Reloading Nix worker-protocol over gRPC...6150worker2 # [ 124.503357] dbus-broker-launch[533]: Looking up NSS user entry for 'systemd-timesync'...6151worker2 # [ 124.509067] dbus-broker-launch[533]: NSS returned no entry for 'systemd-timesync'6152worker2 # [ 124.510070] dbus-broker-launch[533]: Invalid user-name in /nix/store/s93qka0z548yjbxqnmj4c4b32apww6jv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"6153worker2 # [ 124.525120] systemd[1]: Reloaded Nix worker-protocol over gRPC.6154worker2 # [ 124.542353] systemd[1]: Reloaded D-Bus System Message Bus.6155worker2 # [ 124.544436] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os6156worker2 # [ 124.546179] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki6157worker2 # [ 124.550689] systemd[1]: Reached target Local File Systems.6158worker2 # [ 124.553179] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore6159worker2 # [ 124.558849] systemd[1]: Starting Create SUID/SGID Wrappers...6160worker2 # [ 124.559696] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.6161worker2 # [ 124.560907] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc676162worker2 # [ 124.562810] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore6163worker2 # [ 124.564212] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6164worker2 # [ 124.565379] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6165worker2 # [ 124.577724] systemd[1]: Reached target Remote File Systems.6166worker2 # [ 125.014894] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.6167worker2 # [ 125.018224] systemd[1]: Finished Create SUID/SGID Wrappers.6168worker2 # [ 125.022064] systemd[1]: Starting linger-users.service...6169worker2 # [ 125.065771] systemd[1]: linger-users.service: Deactivated successfully.6170worker2 # [ 125.067205] systemd[1]: Finished linger-users.service.6171worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service6172worker2 # [ 125.323176] nixos[1662]: finished switching to system configuration /nix/store/32111b3ip9k1svh21sms1psjc3nd6q0y-nixos-system-worker2-test6173worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 3.53 seconds)6174worker2: must succeed: pgrep -f 'read -t [9]0 x'6175worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)6176client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26177worker2 # [ 125.400866] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:49Z level=info event=draining6178worker1 # [ 125.610404] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:495106179client # don't know how to build these paths:6180client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6181worker1 # [ 125.620211] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106182worker1 # [ 125.642849] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:50Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06183worker1 # [ 125.648138] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:50Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.3:50051 clients=16184worker1 # [ 125.687792] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106185worker1 # [ 125.700511] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106186client # copying 1 paths to 'lb:50051'...6187client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6188worker1 # [ 125.712910] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106189worker1 # [ 125.729366] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6190worker1 # [ 125.948088] nix-daemon[845]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6191worker1 # [ 125.958839] nix-daemon[845]: reaped child process 1568, status = succeeded6192worker1 # [ 126.171752] nix-daemon[845]: accepted connection from pid 1570, user nix-grpc-daemon (trusted)6193worker1 # [ 126.183636] nix-daemon[845]: reaped child process 1573, status = succeeded6194lb # [ 126.205236] envoy[902]: conn peer=[2001:db8:1::1]:55426 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2993 tx=851 ms=34276195lb # [ 126.212580] niks3-server[1002]: 2026/09/21 09:28:51 INFO Received uploads request method=POST path=/api/pending_closures6196worker1 # [ 126.215716] nix-grpc-daemon[1485]: time=2026-09-21T09:28:51.377Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6197worker1 # [ 126.217636] nix-grpc-daemon[1485]: time=2026-09-21T09:28:51.379Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6198lb # [ 126.243157] niks3-server[1002]: 2026/09/21 09:28:51 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6199lb # [ 126.252912] niks3-server[1002]: 2026/09/21 09:28:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6200lb # [ 126.257681] niks3-server[1002]: 2026/09/21 09:28:51 INFO Signed narinfos id=39 count=16201worker1 # [ 126.254149] nix-grpc-daemon[1485]: time=2026-09-21T09:28:51.416Z level=INFO msg="Uploading 1 narinfos"6202lb # [ 126.267451] niks3-server[1002]: 2026/09/21 09:28:51 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6203lb # [ 126.275696] niks3-server[1002]: 2026/09/21 09:28:51 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6204worker1 # [ 126.275303] nix-grpc-daemon[1485]: time=2026-09-21T09:28:51.437Z level=INFO msg="Upload complete. (533ms)"6205lb # [ 126.281974] niks3-server[1002]: 2026/09/21 09:28:51 INFO Completed upload id=396206lb # [ 126.285849] niks3-server[1002]: 2026/09/21 09:28:51 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6207worker1 # [ 126.277778] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=5286208worker1 # [ 126.283290] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106209worker1 # [ 126.291249] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6210worker1 # [ 126.298319] nix-daemon[845]: reaped child process 1563, status = succeeded6211worker1 # [ 126.303371] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6212client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6213client # farm-a> node-a: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6214client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6215client # farm-a> LOG-farm-a-during-drain6216worker1 # [ 127.328474] nix-daemon[845]: reaped child process 1577, status = succeeded6217worker1 # [ 127.535317] nix-daemon[845]: accepted connection from pid 1593, user nix-grpc-daemon (trusted)6218worker1 # [ 127.547393] nix-daemon[845]: reaped child process 1596, status = succeeded6219lb # [ 127.577586] niks3-server[1002]: 2026/09/21 09:28:52 INFO Received uploads request method=POST path=/api/pending_closures6220worker1 # [ 127.579319] nix-grpc-daemon[1485]: time=2026-09-21T09:28:52.741Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6221worker1 # [ 127.581245] nix-grpc-daemon[1485]: time=2026-09-21T09:28:52.743Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6222lb # [ 127.615789] niks3-server[1002]: 2026/09/21 09:28:52 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6223lb # [ 127.623189] niks3-server[1002]: 2026/09/21 09:28:52 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6224lb # [ 127.635453] niks3-server[1002]: 2026/09/21 09:28:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6225worker1 # [ 127.633760] nix-grpc-daemon[1485]: time=2026-09-21T09:28:52.795Z level=INFO msg="Uploading 1 narinfos"6226lb # [ 127.640118] niks3-server[1002]: 2026/09/21 09:28:52 INFO Signed narinfos id=40 count=16227lb # [ 127.647630] niks3-server[1002]: 2026/09/21 09:28:52 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6228lb # [ 127.654735] niks3-server[1002]: 2026/09/21 09:28:52 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6229lb # [ 127.659662] niks3-server[1002]: 2026/09/21 09:28:52 INFO Completed upload id=406230worker1 # [ 127.653246] nix-grpc-daemon[1485]: time=2026-09-21T09:28:52.815Z level=INFO msg="Upload complete. (342ms)"6231lb # [ 127.662278] niks3-server[1002]: 2026/09/21 09:28:52 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6232worker1 # [ 127.661490] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=3 outputs=16233worker1 # [ 127.669797] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:52Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06234worker1 # [ 127.673952] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:52Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.3:50051 clients=16235worker1 # [ 127.681426] nix-daemon[845]: reaped child process 1575, status = succeeded6236worker1 # [ 127.714447] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106237worker1 # [ 127.727639] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106238worker1 # [ 127.740727] nix-grpc-daemon[1485]: time=2026-09-21T09:28:52.903Z level=INFO msg="All 1 paths already cached"6239client # copying 1 paths to 'lb:50051'...6240worker1 # [ 127.749368] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106241client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6242worker1 # [ 127.768174] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6243worker1 # [ 127.991323] nix-daemon[845]: accepted connection from pid 1602, user nix-grpc-daemon (trusted)6244worker1 # [ 128.004062] nix-daemon[845]: reaped child process 1605, status = succeeded6245worker1 # [ 128.224496] nix-daemon[845]: accepted connection from pid 1607, user nix-grpc-daemon (trusted)6246worker1 # [ 128.236694] nix-daemon[845]: reaped child process 1610, status = succeeded6247lb # [ 128.264751] niks3-server[1002]: 2026/09/21 09:28:53 INFO Received uploads request method=POST path=/api/pending_closures6248worker1 # [ 128.266047] nix-grpc-daemon[1485]: time=2026-09-21T09:28:53.428Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6249worker1 # [ 128.267597] nix-grpc-daemon[1485]: time=2026-09-21T09:28:53.430Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6250lb # [ 128.291860] niks3-server[1002]: 2026/09/21 09:28:53 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6251lb # [ 128.304489] niks3-server[1002]: 2026/09/21 09:28:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6252lb # [ 128.312097] niks3-server[1002]: 2026/09/21 09:28:53 INFO Signed narinfos id=41 count=16253worker1 # [ 128.308691] nix-grpc-daemon[1485]: time=2026-09-21T09:28:53.471Z level=INFO msg="Uploading 1 narinfos"6254lb # [ 128.317535] niks3-server[1002]: 2026/09/21 09:28:53 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6255lb # [ 128.335421] niks3-server[1002]: 2026/09/21 09:28:53 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6256lb # [ 128.338903] niks3-server[1002]: 2026/09/21 09:28:53 INFO Completed upload id=416257lb # [ 128.340799] niks3-server[1002]: 2026/09/21 09:28:53 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6258worker1 # [ 128.333664] nix-grpc-daemon[1485]: time=2026-09-21T09:28:53.496Z level=INFO msg="Upload complete. (551ms)"6259worker1 # [ 128.337950] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=6406260worker1 # [ 128.344721] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106261worker1 # [ 128.354419] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6262worker1 # [ 128.362485] nix-daemon[845]: reaped child process 1600, status = succeeded6263client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6264worker1 # [ 128.368835] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6265client # farm-b> node-a: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6266client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6267client # farm-b> LOG-farm-b-during-drain6268worker1 # [ 129.388761] nix-daemon[845]: reaped child process 1614, status = succeeded6269worker1 # [ 129.593142] nix-daemon[845]: accepted connection from pid 1630, user nix-grpc-daemon (trusted)6270worker1 # [ 129.605228] nix-daemon[845]: reaped child process 1633, status = succeeded6271lb # [ 129.636786] niks3-server[1002]: 2026/09/21 09:28:54 INFO Received uploads request method=POST path=/api/pending_closures6272worker1 # [ 129.638575] nix-grpc-daemon[1485]: time=2026-09-21T09:28:54.800Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6273worker1 # [ 129.640343] nix-grpc-daemon[1485]: time=2026-09-21T09:28:54.802Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6274lb # [ 129.674332] niks3-server[1002]: 2026/09/21 09:28:54 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6275lb # [ 129.682928] niks3-server[1002]: 2026/09/21 09:28:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6276lb # [ 129.688832] niks3-server[1002]: 2026/09/21 09:28:54 INFO Signed narinfos id=42 count=16277worker1 # [ 129.685196] nix-grpc-daemon[1485]: time=2026-09-21T09:28:54.847Z level=INFO msg="Uploading 1 narinfos"6278lb # [ 129.698050] niks3-server[1002]: 2026/09/21 09:28:54 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6279lb # [ 129.706166] niks3-server[1002]: 2026/09/21 09:28:54 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6280lb # [ 129.712440] niks3-server[1002]: 2026/09/21 09:28:54 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6281lb # [ 129.717995] niks3-server[1002]: 2026/09/21 09:28:54 INFO Completed upload id=426282worker1 # [ 129.711571] nix-grpc-daemon[1485]: time=2026-09-21T09:28:54.874Z level=INFO msg="Upload complete. (336ms)"6283lb # [ 129.720445] niks3-server[1002]: 2026/09/21 09:28:54 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6284worker1 # [ 129.719291] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=4 outputs=16285worker1 # [ 129.727495] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:54Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06286worker1 # [ 129.731500] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:54Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.3:50051 clients=16287worker1 # [ 129.738497] nix-daemon[845]: reaped child process 1612, status = succeeded6288worker1 # [ 129.773121] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106289worker1 # [ 129.785682] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106290worker1 # [ 129.799419] nix-grpc-daemon[1485]: time=2026-09-21T09:28:54.961Z level=INFO msg="All 2 paths already cached"6291client # copying 1 paths to 'lb:50051'...6292client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6293worker1 # [ 129.807874] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106294worker1 # [ 129.816484] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6295worker1 # [ 130.041166] nix-daemon[845]: accepted connection from pid 1639, user nix-grpc-daemon (trusted)6296worker1 # [ 130.053543] nix-daemon[845]: reaped child process 1642, status = succeeded6297worker1 # [ 130.276513] nix-daemon[845]: accepted connection from pid 1644, user nix-grpc-daemon (trusted)6298worker1 # [ 130.288679] nix-daemon[845]: reaped child process 1647, status = succeeded6299lb # [ 130.317846] niks3-server[1002]: 2026/09/21 09:28:55 INFO Received uploads request method=POST path=/api/pending_closures6300worker1 # [ 130.319808] nix-grpc-daemon[1485]: time=2026-09-21T09:28:55.482Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6301worker1 # [ 130.321318] nix-grpc-daemon[1485]: time=2026-09-21T09:28:55.483Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6302lb # [ 130.346984] niks3-server[1002]: 2026/09/21 09:28:55 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6303lb # [ 130.358947] niks3-server[1002]: 2026/09/21 09:28:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6304lb # [ 130.365538] niks3-server[1002]: 2026/09/21 09:28:55 INFO Signed narinfos id=43 count=16305worker1 # [ 130.361993] nix-grpc-daemon[1485]: time=2026-09-21T09:28:55.524Z level=INFO msg="Uploading 1 narinfos"6306lb # [ 130.370291] niks3-server[1002]: 2026/09/21 09:28:55 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6307lb # [ 130.382776] niks3-server[1002]: 2026/09/21 09:28:55 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6308lb # [ 130.389374] niks3-server[1002]: 2026/09/21 09:28:55 INFO Completed upload id=436309worker1 # [ 130.382647] nix-grpc-daemon[1485]: time=2026-09-21T09:28:55.545Z level=INFO msg="Upload complete. (552ms)"6310lb # [ 130.392383] niks3-server[1002]: 2026/09/21 09:28:55 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6311worker1 # [ 130.386309] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=7686312worker1 # [ 130.391843] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106313worker1 # [ 130.401615] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6314worker1 # [ 130.405977] nix-daemon[845]: reaped child process 1637, status = succeeded6315worker1 # [ 130.411456] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6316client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6317client # farm-top> node-a: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6318client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6319client # farm-top> LOG-farm-top-during-drain6320worker1 # [ 131.034467] nix-daemon[845]: reaped child process 1651, status = succeeded6321worker1 # [ 131.239995] nix-daemon[845]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6322worker1 # [ 131.250923] nix-daemon[845]: reaped child process 1670, status = succeeded6323lb # [ 131.280475] niks3-server[1002]: 2026/09/21 09:28:56 INFO Received uploads request method=POST path=/api/pending_closures6324worker1 # [ 131.282795] nix-grpc-daemon[1485]: time=2026-09-21T09:28:56.445Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6325worker1 # [ 131.284529] nix-grpc-daemon[1485]: time=2026-09-21T09:28:56.446Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6326lb # [ 131.317624] niks3-server[1002]: 2026/09/21 09:28:56 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6327lb # [ 131.323639] niks3-server[1002]: 2026/09/21 09:28:56 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6328lb # [ 131.336384] niks3-server[1002]: 2026/09/21 09:28:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6329lb # [ 131.341629] niks3-server[1002]: 2026/09/21 09:28:56 INFO Signed narinfos id=44 count=16330worker1 # [ 131.338520] nix-grpc-daemon[1485]: time=2026-09-21T09:28:56.500Z level=INFO msg="Uploading 1 narinfos"6331lb # [ 131.346744] niks3-server[1002]: 2026/09/21 09:28:56 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6332lb # [ 131.356840] niks3-server[1002]: 2026/09/21 09:28:56 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6333lb # [ 131.362444] niks3-server[1002]: 2026/09/21 09:28:56 INFO Completed upload id=446334lb # [ 131.364306] niks3-server[1002]: 2026/09/21 09:28:56 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6335worker1 # [ 131.357192] nix-grpc-daemon[1485]: time=2026-09-21T09:28:56.519Z level=INFO msg="Upload complete. (339ms)"6336worker1 # [ 131.362740] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=5 outputs=16337worker1 # [ 131.376630] nix-daemon[845]: reaped child process 1649, status = succeeded6338client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 6.07 seconds)6339client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6340client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.24 seconds)6341worker2: must succeed: pkill -f 'read -t [9]0 x'6342worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6343client: waiting for success: ! systemctl is-active sw6344worker2 # [ 131.766350] nix-daemon[869]: reaped child process 1630, status = succeeded6345worker2 # [ 131.928430] nix-daemon[869]: accepted connection from pid 1802, user nix-grpc-daemon (trusted)6346worker2 # [ 131.940135] nix-daemon[869]: reaped child process 1805, status = succeeded6347lb # [ 131.928349] niks3-server[1002]: 2026/09/21 09:28:57 INFO Received uploads request method=POST path=/api/pending_closures6348worker2 # [ 131.967797] nix-grpc-daemon[1517]: time=2026-09-21T09:28:56.326Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6349worker2 # [ 131.969252] nix-grpc-daemon[1517]: time=2026-09-21T09:28:56.326Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6350lb # [ 131.961722] niks3-server[1002]: 2026/09/21 09:28:57 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6351lb # [ 131.967789] niks3-server[1002]: 2026/09/21 09:28:57 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6352lb # [ 131.980612] niks3-server[1002]: 2026/09/21 09:28:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6353lb # [ 131.985588] niks3-server[1002]: 2026/09/21 09:28:57 INFO Signed narinfos id=45 count=16354worker2 # [ 132.019222] nix-grpc-daemon[1517]: time=2026-09-21T09:28:56.377Z level=INFO msg="Uploading 1 narinfos"6355lb # [ 131.990836] niks3-server[1002]: 2026/09/21 09:28:57 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6356lb # [ 131.999981] niks3-server[1002]: 2026/09/21 09:28:57 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6357lb # [ 132.005834] niks3-server[1002]: 2026/09/21 09:28:57 INFO Completed upload id=456358worker2 # [ 132.036274] nix-grpc-daemon[1517]: time=2026-09-21T09:28:56.395Z level=INFO msg="Upload complete. (276ms)"6359lb # [ 132.007868] niks3-server[1002]: 2026/09/21 09:28:57 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6360worker2 # [ 132.043317] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37420 duration_s=11 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=2 outputs=16361worker2 # [ 132.054840] nix-daemon[869]: reaped child process 1628, status = succeeded6362client # [ 132.074549] systemd[1]: sw.service: Deactivated successfully.6363client # [ 132.078167] systemd[1]: sw.service: Consumed 133ms CPU time over 12.851s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6364worker2 # [ 132.403413] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:56Z level=info event=signal_exit6365client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)6366client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6367client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds)6368worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6369worker2 # [ 132.961930] systemd[1]: Starting scheduler-token.service...6370worker2 # [ 133.031935] systemd[1]: scheduler-token.service: Deactivated successfully.6371worker2 # [ 133.033643] systemd[1]: Finished scheduler-token.service.6372worker2 # [ 133.035560] systemd[1]: scheduler-token.service: Consumed 16ms CPU time over 73ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6373worker1 # [ 133.612558] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:58Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:401226374worker1 # [ 133.616210] nix-grpc-daemon[1407]: ts=2026-09-21T09:28:58Z level=info event=worker_gone addr=192.168.1.4:500516375worker2 # [ 133.654312] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:58Z level=info event=scheduler_restarting6376worker2 # [ 133.663339] nix-grpc-daemon[1403]: ts=2026-09-21T09:28:58Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6377worker2 # [ 133.681526] nix-daemon[869]: reaped child process 1509, status = succeeded6378worker2 # [ 134.487250] nix-grpc-daemon[1403]: E0000 00:00:1789982938.846014 1448 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true6379worker2 # [ 134.492168] nix-grpc-daemon[1403]: E0000 00:00:1789982938.846226 1448 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6380worker2 # [ 134.834688] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6381worker2 # [ 134.838180] systemd[1]: nix-grpc-daemon.service: Consumed 879ms CPU time over 44.647s wall clock time, 33.5M memory peak, 27.9K incoming IP traffic, 53.3K outgoing IP traffic.6382worker2 # [ 134.854650] systemd[1]: Starting Nix worker-protocol over gRPC...6383worker2 # [ 135.036462] nix-grpc-daemon[1822]: ts=2026-09-21T09:28:59Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6384worker2 # [ 135.038256] nix-grpc-daemon[1822]: ts=2026-09-21T09:28:59Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6385worker2 # [ 135.041642] nix-grpc-daemon[1822]: ts=2026-09-21T09:28:59Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16386worker2 # [ 135.048062] nix-grpc-daemon[1822]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6387worker2 # [ 135.049355] nix-grpc-daemon[1822]: E0000 00:00:1789982939.407111 1822 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6388worker2 # [ 135.055577] nix-grpc-daemon[1822]: E0000 00:00:1789982939.412177 1822 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6389worker2 # [ 135.058902] nix-grpc-daemon[1822]: ts=2026-09-21T09:28:59Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6390worker2 # [ 135.061461] systemd[1]: Started Nix worker-protocol over gRPC.6391worker2 # [ 135.067087] nix-grpc-daemon[1822]: E0000 00:00:1789982939.426079 1864 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6392worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.19 seconds)6393lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6394lb: must succeed: curl -sf localhost:9901/clusters6395worker2 # [ 135.079732] nix-grpc-daemon[1822]: E0000 00:00:1789982939.438767 1864 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6396worker2 # [ 135.083120] nix-grpc-daemon[1822]: E0000 00:00:1789982939.440883 1864 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6397worker2 # [ 135.093592] nix-grpc-daemon[1822]: E0000 00:00:1789982939.452596 1864 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6398worker2 # [ 135.105642] nix-grpc-daemon[1822]: ts=2026-09-21T09:28:59Z level=info event=scheduler_connected addr=lb:500516399worker1 # [ 135.072426] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:00Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:401226400worker1 # [ 135.076671] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:00Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06401lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6402lb: must succeed: curl -sf localhost:9901/clusters6403lb # [ 136.206073] envoy[902]: conn peer=[2001:db8:1::1]:38438 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9598 tx=4107 ms=57726404lb # [ 136.209319] envoy[902]: conn peer=[2001:db8:1::1]:55436 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2962 tx=1406 ms=125706405lb # [ 136.212504] envoy[902]: conn peer=[2001:db8:1::4]:55030 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=15724 tx=986 ms=432126406lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6407lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.18 seconds)6408lb: must succeed: curl -sf localhost:9901/clusters6409lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6410worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6411worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6412(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 17.22 seconds)6413subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6414client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16415worker2 # [ 136.670644] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:462626416worker2 # [ 136.681299] nix-daemon[869]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)6417worker2 # [ 136.694953] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462626418worker2 # [ 136.706610] nix-grpc-daemon[1822]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 67 ms (attempt 1/5)6419worker2 # [ 136.779265] nix-grpc-daemon[1822]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 34 ms (attempt 2/5)6420worker2 # [ 136.819130] nix-grpc-daemon[1822]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 190 ms (attempt 3/5)6421worker2 # [ 137.015127] nix-grpc-daemon[1822]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 347 ms (attempt 4/5)6422worker2 # [ 137.368282] nix-grpc-daemon[1822]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6423worker2 # [ 137.372137] nix-grpc-daemon[1822]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6424worker1 # [ 137.386863] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:02Z level=debug event=want client=5 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06425worker1 # [ 137.393074] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:02Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16426worker1 # [ 137.397408] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:02Z level=debug event=want client=5 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06427worker1 # [ 137.401264] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:02Z level=debug event=want client=5 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06428worker1 # [ 137.436396] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106429worker1 # [ 137.449443] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106430worker1 # [ 137.461807] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106431worker1 # [ 137.473341] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6432worker1 # [ 137.693257] nix-daemon[845]: accepted connection from pid 1679, user nix-grpc-daemon (trusted)6433worker1 # [ 137.705471] nix-daemon[845]: reaped child process 1682, status = succeeded6434worker1 # [ 137.914855] nix-daemon[845]: accepted connection from pid 1684, user nix-grpc-daemon (trusted)6435worker1 # [ 137.926528] nix-daemon[845]: reaped child process 1687, status = succeeded6436lb # [ 137.954652] niks3-server[1002]: 2026/09/21 09:29:03 INFO Received uploads request method=POST path=/api/pending_closures6437worker1 # [ 137.956935] nix-grpc-daemon[1485]: time=2026-09-21T09:29:03.118Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6438worker1 # [ 137.958388] nix-grpc-daemon[1485]: time=2026-09-21T09:29:03.118Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6439lb # [ 137.988636] niks3-server[1002]: 2026/09/21 09:29:03 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6440lb # [ 137.994929] niks3-server[1002]: 2026/09/21 09:29:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6441lb # [ 138.002410] niks3-server[1002]: 2026/09/21 09:29:03 INFO Signed narinfos id=46 count=16442worker1 # [ 137.999140] nix-grpc-daemon[1485]: time=2026-09-21T09:29:03.161Z level=INFO msg="Uploading 1 narinfos"6443lb # [ 138.010451] niks3-server[1002]: 2026/09/21 09:29:03 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6444lb # [ 138.021776] niks3-server[1002]: 2026/09/21 09:29:03 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6445lb # [ 138.028369] niks3-server[1002]: 2026/09/21 09:29:03 INFO Completed upload id=466446worker1 # [ 138.024238] nix-grpc-daemon[1485]: time=2026-09-21T09:29:03.186Z level=INFO msg="Upload complete. (539ms)"6447lb # [ 138.031638] niks3-server[1002]: 2026/09/21 09:29:03 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6448worker1 # [ 138.029149] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=5046449worker1 # [ 138.034650] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106450worker1 # [ 138.044807] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6451worker1 # [ 138.053199] nix-daemon[845]: reaped child process 1677, status = succeeded6452worker1 # [ 138.060783] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6453worker1 # [ 138.873702] nix-daemon[845]: reaped child process 1691, status = succeeded6454worker1 # [ 139.068318] nix-daemon[845]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6455worker1 # [ 139.080096] nix-daemon[845]: reaped child process 1710, status = succeeded6456lb # [ 139.107741] niks3-server[1002]: 2026/09/21 09:29:04 INFO Received uploads request method=POST path=/api/pending_closures6457worker1 # [ 139.109252] nix-grpc-daemon[1485]: time=2026-09-21T09:29:04.271Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6458worker1 # [ 139.110964] nix-grpc-daemon[1485]: time=2026-09-21T09:29:04.273Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6459lb # [ 139.145363] niks3-server[1002]: 2026/09/21 09:29:04 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6460lb # [ 139.152118] niks3-server[1002]: 2026/09/21 09:29:04 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6461lb # [ 139.163264] niks3-server[1002]: 2026/09/21 09:29:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6462worker1 # [ 139.160586] nix-grpc-daemon[1485]: time=2026-09-21T09:29:04.322Z level=INFO msg="Uploading 1 narinfos"6463lb # [ 139.167529] niks3-server[1002]: 2026/09/21 09:29:04 INFO Signed narinfos id=47 count=16464lb # [ 139.172116] niks3-server[1002]: 2026/09/21 09:29:04 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6465lb # [ 139.180552] niks3-server[1002]: 2026/09/21 09:29:04 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6466lb # [ 139.183357] niks3-server[1002]: 2026/09/21 09:29:04 INFO Completed upload id=476467worker1 # [ 139.176853] nix-grpc-daemon[1485]: time=2026-09-21T09:29:04.339Z level=INFO msg="Upload complete. (309ms)"6468lb # [ 139.185176] niks3-server[1002]: 2026/09/21 09:29:04 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6469worker1 # [ 139.183989] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=6 outputs=16470worker1 # [ 139.191159] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:04Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16471worker1 # [ 139.197160] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106472worker1 # [ 139.199409] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:04Z level=debug event=want client=5 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06473worker1 # [ 139.205310] nix-daemon[845]: reaped child process 1689, status = succeeded6474worker1 # [ 139.206513] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106475worker1 # [ 139.213638] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106476worker1 # [ 139.217687] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6477worker1 # [ 139.426943] nix-daemon[845]: accepted connection from pid 1716, user nix-grpc-daemon (trusted)6478worker1 # [ 139.439809] nix-daemon[845]: reaped child process 1719, status = succeeded6479worker1 # [ 139.658403] nix-daemon[845]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6480worker1 # [ 139.671053] nix-daemon[845]: reaped child process 1724, status = succeeded6481lb # [ 139.697733] niks3-server[1002]: 2026/09/21 09:29:04 INFO Received uploads request method=POST path=/api/pending_closures6482worker1 # [ 139.699446] nix-grpc-daemon[1485]: time=2026-09-21T09:29:04.861Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6483worker1 # [ 139.701178] nix-grpc-daemon[1485]: time=2026-09-21T09:29:04.863Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6484lb # [ 139.726917] niks3-server[1002]: 2026/09/21 09:29:04 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6485lb # [ 139.737702] niks3-server[1002]: 2026/09/21 09:29:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6486worker1 # [ 139.735667] nix-grpc-daemon[1485]: time=2026-09-21T09:29:04.898Z level=INFO msg="Uploading 1 narinfos"6487lb # [ 139.742873] niks3-server[1002]: 2026/09/21 09:29:04 INFO Signed narinfos id=48 count=16488lb # [ 139.749218] niks3-server[1002]: 2026/09/21 09:29:04 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6489lb # [ 139.759136] niks3-server[1002]: 2026/09/21 09:29:04 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6490worker1 # [ 139.758986] nix-grpc-daemon[1485]: time=2026-09-21T09:29:04.921Z level=INFO msg="Upload complete. (535ms)"6491lb # [ 139.765856] niks3-server[1002]: 2026/09/21 09:29:04 INFO Completed upload id=486492lb # [ 139.768350] niks3-server[1002]: 2026/09/21 09:29:04 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6493worker1 # [ 139.762140] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=5046494worker1 # [ 139.768473] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106495worker1 # [ 139.777559] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6496worker1 # [ 139.780527] nix-daemon[845]: reaped child process 1714, status = succeeded6497worker1 # [ 139.786493] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6498worker1 # [ 140.799476] nix-daemon[845]: reaped child process 1728, status = succeeded6499worker1 # [ 140.995498] nix-daemon[845]: accepted connection from pid 1744, user nix-grpc-daemon (trusted)6500worker1 # [ 141.007692] nix-daemon[845]: reaped child process 1747, status = succeeded6501lb # [ 141.034695] niks3-server[1002]: 2026/09/21 09:29:06 INFO Received uploads request method=POST path=/api/pending_closures6502worker1 # [ 141.036046] nix-grpc-daemon[1485]: time=2026-09-21T09:29:06.198Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6503worker1 # [ 141.037815] nix-grpc-daemon[1485]: time=2026-09-21T09:29:06.200Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6504lb # [ 141.073787] niks3-server[1002]: 2026/09/21 09:29:06 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6505lb # [ 141.081496] niks3-server[1002]: 2026/09/21 09:29:06 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6506lb # [ 141.094259] niks3-server[1002]: 2026/09/21 09:29:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6507lb # [ 141.100455] niks3-server[1002]: 2026/09/21 09:29:06 INFO Signed narinfos id=49 count=16508worker1 # [ 141.096875] nix-grpc-daemon[1485]: time=2026-09-21T09:29:06.258Z level=INFO msg="Uploading 1 narinfos"6509lb # [ 141.103302] niks3-server[1002]: 2026/09/21 09:29:06 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6510lb # [ 141.115514] niks3-server[1002]: 2026/09/21 09:29:06 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6511lb # [ 141.119807] niks3-server[1002]: 2026/09/21 09:29:06 INFO Completed upload id=496512lb # [ 141.121604] niks3-server[1002]: 2026/09/21 09:29:06 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6513worker1 # [ 141.114599] nix-grpc-daemon[1485]: time=2026-09-21T09:29:06.277Z level=INFO msg="Upload complete. (322ms)"6514worker1 # [ 141.124851] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=7 outputs=16515worker1 # [ 141.131904] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:06Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16516worker1 # [ 141.136694] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106517worker1 # [ 141.140502] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:06Z level=debug event=want client=5 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06518worker1 # [ 141.150979] nix-daemon[845]: reaped child process 1726, status = succeeded6519worker1 # [ 141.166828] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106520worker1 # [ 141.176582] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106521worker1 # [ 141.183653] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6522worker1 # [ 141.396910] nix-daemon[845]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)6523worker1 # [ 141.409849] nix-daemon[845]: reaped child process 1756, status = succeeded6524worker1 # [ 141.627643] nix-daemon[845]: accepted connection from pid 1758, user nix-grpc-daemon (trusted)6525worker1 # [ 141.639612] nix-daemon[845]: reaped child process 1761, status = succeeded6526lb # [ 141.666819] niks3-server[1002]: 2026/09/21 09:29:06 INFO Received uploads request method=POST path=/api/pending_closures6527worker1 # [ 141.668247] nix-grpc-daemon[1485]: time=2026-09-21T09:29:06.830Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6528worker1 # [ 141.669625] nix-grpc-daemon[1485]: time=2026-09-21T09:29:06.830Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6529lb # [ 141.690236] niks3-server[1002]: 2026/09/21 09:29:06 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6530lb # [ 141.702470] niks3-server[1002]: 2026/09/21 09:29:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6531lb # [ 141.707624] niks3-server[1002]: 2026/09/21 09:29:06 INFO Signed narinfos id=50 count=16532worker1 # [ 141.704206] nix-grpc-daemon[1485]: time=2026-09-21T09:29:06.866Z level=INFO msg="Uploading 1 narinfos"6533lb # [ 141.714650] niks3-server[1002]: 2026/09/21 09:29:06 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6534lb # [ 141.725218] niks3-server[1002]: 2026/09/21 09:29:06 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6535worker1 # [ 141.724755] nix-grpc-daemon[1485]: time=2026-09-21T09:29:06.887Z level=INFO msg="Upload complete. (531ms)"6536lb # [ 141.731648] niks3-server[1002]: 2026/09/21 09:29:06 INFO Completed upload id=506537lb # [ 141.734827] niks3-server[1002]: 2026/09/21 09:29:06 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6538worker1 # [ 141.727498] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=5046539worker1 # [ 141.732351] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106540worker1 # [ 141.741229] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6541worker1 # [ 141.746477] nix-daemon[845]: reaped child process 1751, status = succeeded6542worker1 # [ 141.752461] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6543worker1 # [ 142.964661] nix-daemon[845]: reaped child process 1765, status = succeeded6544worker1 # [ 143.160914] nix-daemon[845]: accepted connection from pid 1781, user nix-grpc-daemon (trusted)6545worker1 # [ 143.171723] nix-daemon[845]: reaped child process 1784, status = succeeded6546lb # [ 143.200518] niks3-server[1002]: 2026/09/21 09:29:08 INFO Received uploads request method=POST path=/api/pending_closures6547worker1 # [ 143.202240] nix-grpc-daemon[1485]: time=2026-09-21T09:29:08.364Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6548worker1 # [ 143.203803] nix-grpc-daemon[1485]: time=2026-09-21T09:29:08.366Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"6549lb # [ 143.238461] niks3-server[1002]: 2026/09/21 09:29:08 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6550lb # [ 143.249785] niks3-server[1002]: 2026/09/21 09:29:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6551worker1 # [ 143.252603] nix-grpc-daemon[1485]: time=2026-09-21T09:29:08.414Z level=INFO msg="Uploading 1 narinfos"6552lb # [ 143.258922] niks3-server[1002]: 2026/09/21 09:29:08 INFO Signed narinfos id=51 count=16553lb # [ 143.265058] niks3-server[1002]: 2026/09/21 09:29:08 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv6554lb # [ 143.280103] niks3-server[1002]: 2026/09/21 09:29:08 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6555lb # [ 143.283270] niks3-server[1002]: 2026/09/21 09:29:08 INFO Completed upload id=516556worker1 # [ 143.276806] nix-grpc-daemon[1485]: time=2026-09-21T09:29:08.439Z level=INFO msg="Upload complete. (318ms)"6557lb # [ 143.285259] niks3-server[1002]: 2026/09/21 09:29:08 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo6558lb # [ 143.291603] niks3-server[1002]: 2026/09/21 09:29:08 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls6559worker1 # [ 143.287517] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=8 outputs=16560worker1 # [ 143.294386] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:08Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=16561worker1 # [ 143.299705] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106562worker1 # [ 143.303425] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:08Z level=debug event=want client=5 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=06563worker1 # [ 143.312312] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106564worker1 # [ 143.315561] nix-daemon[845]: reaped child process 1763, status = succeeded6565worker1 # [ 143.320593] nix-grpc-daemon[1485]: time=2026-09-21T09:29:08.483Z level=INFO msg="All 1 paths already cached"6566worker1 # [ 143.324578] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106567worker1 # [ 143.328702] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6568worker1 # [ 143.533632] nix-daemon[845]: accepted connection from pid 1790, user nix-grpc-daemon (trusted)6569worker1 # [ 143.545628] nix-daemon[845]: reaped child process 1793, status = succeeded6570worker1 # [ 143.764582] nix-daemon[845]: accepted connection from pid 1795, user nix-grpc-daemon (trusted)6571worker1 # [ 143.776662] nix-daemon[845]: reaped child process 1798, status = succeeded6572lb # [ 143.803942] niks3-server[1002]: 2026/09/21 09:29:08 INFO Received uploads request method=POST path=/api/pending_closures6573worker1 # [ 143.811340] nix-grpc-daemon[1485]: time=2026-09-21T09:29:08.973Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6574worker1 # [ 143.812756] nix-grpc-daemon[1485]: time=2026-09-21T09:29:08.973Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"6575lb # [ 143.835975] niks3-server[1002]: 2026/09/21 09:29:08 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst6576lb # [ 143.847828] niks3-server[1002]: 2026/09/21 09:29:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6577worker1 # [ 143.848842] nix-grpc-daemon[1485]: time=2026-09-21T09:29:09.011Z level=INFO msg="Uploading 1 narinfos"6578lb # [ 143.855474] niks3-server[1002]: 2026/09/21 09:29:08 INFO Signed narinfos id=52 count=16579lb # [ 143.859577] niks3-server[1002]: 2026/09/21 09:29:08 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls6580lb # [ 143.869880] niks3-server[1002]: 2026/09/21 09:29:08 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6581lb # [ 143.876304] niks3-server[1002]: 2026/09/21 09:29:08 INFO Completed upload id=526582worker1 # [ 143.869529] nix-grpc-daemon[1485]: time=2026-09-21T09:29:09.031Z level=INFO msg="Upload complete. (534ms)"6583lb # [ 143.879429] niks3-server[1002]: 2026/09/21 09:29:08 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo6584worker1 # [ 143.873257] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=6246585worker1 # [ 143.878811] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106586worker1 # [ 143.887534] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6587worker1 # [ 143.894893] nix-daemon[845]: reaped child process 1788, status = succeeded6588worker1 # [ 143.900095] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6589worker1 # [ 144.724647] nix-daemon[845]: reaped child process 1802, status = succeeded6590worker1 # [ 144.938663] nix-daemon[845]: accepted connection from pid 1818, user nix-grpc-daemon (trusted)6591worker1 # [ 144.949719] nix-daemon[845]: reaped child process 1821, status = succeeded6592lb # [ 144.980606] niks3-server[1002]: 2026/09/21 09:29:10 INFO Received uploads request method=POST path=/api/pending_closures6593worker1 # [ 144.983772] nix-grpc-daemon[1485]: time=2026-09-21T09:29:10.146Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6594worker1 # [ 144.986059] nix-grpc-daemon[1485]: time=2026-09-21T09:29:10.147Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"6595lb # [ 145.023161] niks3-server[1002]: 2026/09/21 09:29:10 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv6596lb # [ 145.028692] niks3-server[1002]: 2026/09/21 09:29:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6597lb # [ 145.033275] niks3-server[1002]: 2026/09/21 09:29:10 INFO Signed narinfos id=53 count=16598worker1 # [ 145.029099] nix-grpc-daemon[1485]: time=2026-09-21T09:29:10.191Z level=INFO msg="Uploading 1 narinfos"6599lb # [ 145.042106] niks3-server[1002]: 2026/09/21 09:29:10 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst6600lb # [ 145.047583] niks3-server[1002]: 2026/09/21 09:29:10 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls6601lb # [ 145.050393] niks3-server[1002]: 2026/09/21 09:29:10 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6602lb # [ 145.053302] niks3-server[1002]: 2026/09/21 09:29:10 INFO Completed upload id=536603worker1 # [ 145.048086] nix-grpc-daemon[1485]: time=2026-09-21T09:29:10.210Z level=INFO msg="Upload complete. (340ms)"6604lb # [ 145.055831] niks3-server[1002]: 2026/09/21 09:29:10 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo6605worker1 # [ 145.053435] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=9 outputs=16606worker1 # [ 145.061602] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:10Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=16607worker1 # [ 145.065799] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106608worker1 # [ 145.070943] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:10Z level=debug event=want client=5 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=06609worker1 # [ 145.076898] nix-daemon[845]: reaped child process 1800, status = succeeded6610worker1 # [ 145.101670] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106611worker1 # [ 145.110444] nix-grpc-daemon[1485]: time=2026-09-21T09:29:10.272Z level=INFO msg="All 1 paths already cached"6612worker1 # [ 145.116097] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106613worker1 # [ 145.122954] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6614worker1 # [ 145.337748] nix-daemon[845]: accepted connection from pid 1827, user nix-grpc-daemon (trusted)6615worker1 # [ 145.350817] nix-daemon[845]: reaped child process 1830, status = succeeded6616worker1 # [ 145.573449] nix-daemon[845]: accepted connection from pid 1832, user nix-grpc-daemon (trusted)6617worker1 # [ 145.585483] nix-daemon[845]: reaped child process 1835, status = succeeded6618lb # [ 145.614451] niks3-server[1002]: 2026/09/21 09:29:10 INFO Received uploads request method=POST path=/api/pending_closures6619worker1 # [ 145.616636] nix-grpc-daemon[1485]: time=2026-09-21T09:29:10.778Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6620worker1 # [ 145.618070] nix-grpc-daemon[1485]: time=2026-09-21T09:29:10.778Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"6621lb # [ 145.639413] niks3-server[1002]: 2026/09/21 09:29:10 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst6622lb # [ 145.653076] niks3-server[1002]: 2026/09/21 09:29:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6623worker1 # [ 145.652243] nix-grpc-daemon[1485]: time=2026-09-21T09:29:10.814Z level=INFO msg="Uploading 1 narinfos"6624lb # [ 145.658778] niks3-server[1002]: 2026/09/21 09:29:10 INFO Signed narinfos id=54 count=16625lb # [ 145.663609] niks3-server[1002]: 2026/09/21 09:29:10 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls6626lb # [ 145.674160] niks3-server[1002]: 2026/09/21 09:29:10 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6627lb # [ 145.684330] niks3-server[1002]: 2026/09/21 09:29:10 INFO Completed upload id=546628worker1 # [ 145.677668] nix-grpc-daemon[1485]: time=2026-09-21T09:29:10.840Z level=INFO msg="Upload complete. (545ms)"6629lb # [ 145.687497] niks3-server[1002]: 2026/09/21 09:29:10 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo6630worker1 # [ 145.682135] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=6246631worker1 # [ 145.687283] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106632worker1 # [ 145.696111] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6633worker1 # [ 145.701240] nix-daemon[845]: reaped child process 1825, status = succeeded6634worker1 # [ 145.705966] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6635worker1 # [ 146.726378] nix-daemon[845]: reaped child process 1839, status = succeeded6636worker1 # [ 146.926177] nix-daemon[845]: accepted connection from pid 1855, user nix-grpc-daemon (trusted)6637worker1 # [ 146.938120] nix-daemon[845]: reaped child process 1858, status = succeeded6638lb # [ 146.968578] niks3-server[1002]: 2026/09/21 09:29:12 INFO Received uploads request method=POST path=/api/pending_closures6639worker1 # [ 146.970644] nix-grpc-daemon[1485]: time=2026-09-21T09:29:12.132Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6640worker1 # [ 146.972186] nix-grpc-daemon[1485]: time=2026-09-21T09:29:12.134Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"6641lb # [ 146.998122] niks3-server[1002]: 2026/09/21 09:29:12 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv6642lb # [ 147.013988] niks3-server[1002]: 2026/09/21 09:29:12 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst6643lb # [ 147.026341] niks3-server[1002]: 2026/09/21 09:29:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6644lb # [ 147.032093] niks3-server[1002]: 2026/09/21 09:29:12 INFO Signed narinfos id=55 count=16645worker1 # [ 147.028080] nix-grpc-daemon[1485]: time=2026-09-21T09:29:12.190Z level=INFO msg="Uploading 1 narinfos"6646lb # [ 147.041453] niks3-server[1002]: 2026/09/21 09:29:12 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls6647lb # [ 147.050766] niks3-server[1002]: 2026/09/21 09:29:12 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6648lb # [ 147.053612] niks3-server[1002]: 2026/09/21 09:29:12 INFO Completed upload id=556649worker1 # [ 147.049114] nix-grpc-daemon[1485]: time=2026-09-21T09:29:12.210Z level=INFO msg="Upload complete. (335ms)"6650worker1 # [ 147.055259] nix-grpc-daemon[1485]: time=2026-09-21T09:29:12.214Z level=WARN msg="Failed to register uploaded object" key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo error="server returned 404: object is not pending upload\n"6651worker1 # [ 147.061667] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:12Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=16652worker1 # [ 147.065452] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=10 outputs=16653worker1 # [ 147.072116] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106654worker1 # [ 147.075936] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:12Z level=debug event=want client=5 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=06655worker1 # [ 147.084553] nix-daemon[845]: reaped child process 1837, status = succeeded6656worker1 # [ 147.086556] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106657worker1 # [ 147.094212] nix-grpc-daemon[1485]: time=2026-09-21T09:29:12.256Z level=INFO msg="All 1 paths already cached"6658worker1 # [ 147.098576] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106659worker1 # [ 147.103412] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6660worker1 # [ 147.306031] nix-daemon[845]: accepted connection from pid 1864, user nix-grpc-daemon (trusted)6661worker1 # [ 147.318151] nix-daemon[845]: reaped child process 1867, status = succeeded6662worker1 # [ 147.537039] nix-daemon[845]: accepted connection from pid 1869, user nix-grpc-daemon (trusted)6663worker1 # [ 147.548788] nix-daemon[845]: reaped child process 1872, status = succeeded6664lb # [ 147.578767] niks3-server[1002]: 2026/09/21 09:29:12 INFO Received uploads request method=POST path=/api/pending_closures6665worker1 # [ 147.580192] nix-grpc-daemon[1485]: time=2026-09-21T09:29:12.742Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6666worker1 # [ 147.581728] nix-grpc-daemon[1485]: time=2026-09-21T09:29:12.744Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"6667lb # [ 147.607698] niks3-server[1002]: 2026/09/21 09:29:12 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst6668lb # [ 147.619273] niks3-server[1002]: 2026/09/21 09:29:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6669worker1 # [ 147.617400] nix-grpc-daemon[1485]: time=2026-09-21T09:29:12.779Z level=INFO msg="Uploading 1 narinfos"6670lb # [ 147.624120] niks3-server[1002]: 2026/09/21 09:29:12 INFO Signed narinfos id=56 count=16671lb # [ 147.630682] niks3-server[1002]: 2026/09/21 09:29:12 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls6672lb # [ 147.641822] niks3-server[1002]: 2026/09/21 09:29:12 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6673lb # [ 147.647661] niks3-server[1002]: 2026/09/21 09:29:12 INFO Completed upload id=566674worker1 # [ 147.641201] nix-grpc-daemon[1485]: time=2026-09-21T09:29:12.803Z level=INFO msg="Upload complete. (532ms)"6675lb # [ 147.650608] niks3-server[1002]: 2026/09/21 09:29:12 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo6676worker1 # [ 147.644851] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=6246677worker1 # [ 147.650565] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106678worker1 # [ 147.659883] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6679worker1 # [ 147.664577] nix-daemon[845]: reaped child process 1862, status = succeeded6680worker1 # [ 147.669528] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6681worker1 # [ 148.895577] nix-daemon[845]: reaped child process 1876, status = succeeded6682worker1 # [ 149.107495] nix-daemon[845]: accepted connection from pid 1892, user nix-grpc-daemon (trusted)6683worker1 # [ 149.119529] nix-daemon[845]: reaped child process 1895, status = succeeded6684lb # [ 149.149940] niks3-server[1002]: 2026/09/21 09:29:14 INFO Received uploads request method=POST path=/api/pending_closures6685worker1 # [ 149.154248] nix-grpc-daemon[1485]: time=2026-09-21T09:29:14.316Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6686worker1 # [ 149.157062] nix-grpc-daemon[1485]: time=2026-09-21T09:29:14.318Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"6687lb # [ 149.193944] niks3-server[1002]: 2026/09/21 09:29:14 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst6688lb # [ 149.200477] niks3-server[1002]: 2026/09/21 09:29:14 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv6689lb # [ 149.206498] niks3-server[1002]: 2026/09/21 09:29:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6690lb # [ 149.211420] niks3-server[1002]: 2026/09/21 09:29:14 INFO Signed narinfos id=57 count=16691worker1 # [ 149.207803] nix-grpc-daemon[1485]: time=2026-09-21T09:29:14.370Z level=INFO msg="Uploading 1 narinfos"6692lb # [ 149.217211] niks3-server[1002]: 2026/09/21 09:29:14 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls6693lb # [ 149.226564] niks3-server[1002]: 2026/09/21 09:29:14 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6694lb # [ 149.232452] niks3-server[1002]: 2026/09/21 09:29:14 INFO Completed upload id=576695worker1 # [ 149.225790] nix-grpc-daemon[1485]: time=2026-09-21T09:29:14.388Z level=INFO msg="Upload complete. (348ms)"6696lb # [ 149.234343] niks3-server[1002]: 2026/09/21 09:29:14 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo6697worker1 # [ 149.233989] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=11 outputs=16698worker1 # [ 149.240648] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:14Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=16699worker1 # [ 149.244855] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106700worker1 # [ 149.248456] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:14Z level=debug event=want client=5 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=06701worker1 # [ 149.257115] nix-daemon[845]: reaped child process 1874, status = succeeded6702worker1 # [ 149.259498] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106703worker1 # [ 149.267265] nix-grpc-daemon[1485]: time=2026-09-21T09:29:14.429Z level=INFO msg="All 2 paths already cached"6704worker1 # [ 149.271855] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106705worker1 # [ 149.277058] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6706worker1 # [ 149.482910] nix-daemon[845]: accepted connection from pid 1901, user nix-grpc-daemon (trusted)6707worker1 # [ 149.495095] nix-daemon[845]: reaped child process 1904, status = succeeded6708worker1 # [ 149.717398] nix-daemon[845]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6709worker1 # [ 149.730324] nix-daemon[845]: reaped child process 1909, status = succeeded6710lb # [ 149.759550] niks3-server[1002]: 2026/09/21 09:29:14 INFO Received uploads request method=POST path=/api/pending_closures6711worker1 # [ 149.761502] nix-grpc-daemon[1485]: time=2026-09-21T09:29:14.923Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6712worker1 # [ 149.762902] nix-grpc-daemon[1485]: time=2026-09-21T09:29:14.923Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"6713lb # [ 149.787062] niks3-server[1002]: 2026/09/21 09:29:14 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst6714lb # [ 149.796510] niks3-server[1002]: 2026/09/21 09:29:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6715worker1 # [ 149.796457] nix-grpc-daemon[1485]: time=2026-09-21T09:29:14.958Z level=INFO msg="Uploading 1 narinfos"6716lb # [ 149.803106] niks3-server[1002]: 2026/09/21 09:29:14 INFO Signed narinfos id=58 count=16717lb # [ 149.807317] niks3-server[1002]: 2026/09/21 09:29:14 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls6718lb # [ 149.819873] niks3-server[1002]: 2026/09/21 09:29:14 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6719lb # [ 149.827437] niks3-server[1002]: 2026/09/21 09:29:14 INFO Completed upload id=586720worker1 # [ 149.820826] nix-grpc-daemon[1485]: time=2026-09-21T09:29:14.983Z level=INFO msg="Upload complete. (538ms)"6721lb # [ 149.830475] niks3-server[1002]: 2026/09/21 09:29:14 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo6722worker1 # [ 149.825152] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=7526723worker1 # [ 149.830373] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106724worker1 # [ 149.840148] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6725worker1 # [ 149.843787] nix-daemon[845]: reaped child process 1899, status = succeeded6726worker1 # [ 149.849357] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6727worker1 # [ 150.676202] nix-daemon[845]: reaped child process 1913, status = succeeded6728worker1 # [ 150.882555] nix-daemon[845]: accepted connection from pid 1929, user nix-grpc-daemon (trusted)6729worker1 # [ 150.894707] nix-daemon[845]: reaped child process 1932, status = succeeded6730lb # [ 150.925754] niks3-server[1002]: 2026/09/21 09:29:16 INFO Received uploads request method=POST path=/api/pending_closures6731worker1 # [ 150.931348] nix-grpc-daemon[1485]: time=2026-09-21T09:29:16.092Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6732worker1 # [ 150.932831] nix-grpc-daemon[1485]: time=2026-09-21T09:29:16.092Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"6733lb # [ 150.957626] niks3-server[1002]: 2026/09/21 09:29:16 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv6734lb # [ 150.970301] niks3-server[1002]: 2026/09/21 09:29:16 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst6735lb # [ 150.980513] niks3-server[1002]: 2026/09/21 09:29:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6736lb # [ 150.985738] niks3-server[1002]: 2026/09/21 09:29:16 INFO Signed narinfos id=59 count=16737worker1 # [ 150.981536] nix-grpc-daemon[1485]: time=2026-09-21T09:29:16.143Z level=INFO msg="Uploading 1 narinfos"6738lb # [ 150.991037] niks3-server[1002]: 2026/09/21 09:29:16 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls6739lb # [ 151.000838] niks3-server[1002]: 2026/09/21 09:29:16 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6740lb # [ 151.006422] niks3-server[1002]: 2026/09/21 09:29:16 INFO Completed upload id=596741lb # [ 151.008494] niks3-server[1002]: 2026/09/21 09:29:16 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo6742worker1 # [ 150.999993] nix-grpc-daemon[1485]: time=2026-09-21T09:29:16.162Z level=INFO msg="Upload complete. (343ms)"6743worker1 # [ 151.006803] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=12 outputs=16744worker1 # [ 151.013280] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:16Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=16745worker1 # [ 151.023914] nix-daemon[845]: reaped child process 1911, status = succeeded6746worker1 # [ 151.047235] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106747worker1 # [ 151.059719] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106748worker1 # [ 151.073292] nix-grpc-daemon[1485]: time=2026-09-21T09:29:16.235Z level=INFO msg="All 2 paths already cached"6749worker1 # [ 151.080727] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106750worker1 # [ 151.089352] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6751worker1 # [ 151.308348] nix-daemon[845]: accepted connection from pid 1936, user nix-grpc-daemon (trusted)6752worker1 # [ 151.321133] nix-daemon[845]: reaped child process 1939, status = succeeded6753worker1 # [ 151.540863] nix-daemon[845]: accepted connection from pid 1941, user nix-grpc-daemon (trusted)6754worker1 # [ 151.552724] nix-daemon[845]: reaped child process 1944, status = succeeded6755lb # [ 151.583260] niks3-server[1002]: 2026/09/21 09:29:16 INFO Received uploads request method=POST path=/api/pending_closures6756worker1 # [ 151.585496] nix-grpc-daemon[1485]: time=2026-09-21T09:29:16.747Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6757worker1 # [ 151.587042] nix-grpc-daemon[1485]: time=2026-09-21T09:29:16.749Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"6758lb # [ 151.613148] niks3-server[1002]: 2026/09/21 09:29:16 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst6759lb # [ 151.625933] niks3-server[1002]: 2026/09/21 09:29:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6760worker1 # [ 151.626067] nix-grpc-daemon[1485]: time=2026-09-21T09:29:16.788Z level=INFO msg="Uploading 1 narinfos"6761lb # [ 151.633350] niks3-server[1002]: 2026/09/21 09:29:16 INFO Signed narinfos id=60 count=16762lb # [ 151.638777] niks3-server[1002]: 2026/09/21 09:29:16 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls6763lb # [ 151.652094] niks3-server[1002]: 2026/09/21 09:29:16 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6764lb # [ 151.658174] niks3-server[1002]: 2026/09/21 09:29:16 INFO Completed upload id=606765worker1 # [ 151.651784] nix-grpc-daemon[1485]: time=2026-09-21T09:29:16.814Z level=INFO msg="Upload complete. (549ms)"6766lb # [ 151.661209] niks3-server[1002]: 2026/09/21 09:29:16 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo6767worker1 # [ 151.656197] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=7526768worker1 # [ 151.663229] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106769worker1 # [ 151.672662] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6770worker1 # [ 151.682459] nix-daemon[845]: reaped child process 1934, status = succeeded6771worker1 # [ 151.687823] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6772worker1 # [ 152.708244] nix-daemon[845]: reaped child process 1948, status = succeeded6773worker1 # [ 152.910403] nix-daemon[845]: accepted connection from pid 1964, user nix-grpc-daemon (trusted)6774worker1 # [ 152.921308] nix-daemon[845]: reaped child process 1967, status = succeeded6775lb # [ 152.952997] niks3-server[1002]: 2026/09/21 09:29:18 INFO Received uploads request method=POST path=/api/pending_closures6776worker1 # [ 152.955517] nix-grpc-daemon[1485]: time=2026-09-21T09:29:18.117Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6777worker1 # [ 152.957139] nix-grpc-daemon[1485]: time=2026-09-21T09:29:18.119Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"6778lb # [ 152.993767] niks3-server[1002]: 2026/09/21 09:29:18 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv6779lb # [ 153.001477] niks3-server[1002]: 2026/09/21 09:29:18 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst6780lb # [ 153.014613] niks3-server[1002]: 2026/09/21 09:29:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6781worker1 # [ 153.014602] nix-grpc-daemon[1485]: time=2026-09-21T09:29:18.176Z level=INFO msg="Uploading 1 narinfos"6782lb # [ 153.021153] niks3-server[1002]: 2026/09/21 09:29:18 INFO Signed narinfos id=61 count=16783lb # [ 153.026587] niks3-server[1002]: 2026/09/21 09:29:18 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls6784lb # [ 153.037616] niks3-server[1002]: 2026/09/21 09:29:18 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6785lb # [ 153.044301] niks3-server[1002]: 2026/09/21 09:29:18 INFO Completed upload id=616786worker1 # [ 153.037785] nix-grpc-daemon[1485]: time=2026-09-21T09:29:18.200Z level=INFO msg="Upload complete. (343ms)"6787lb # [ 153.047256] niks3-server[1002]: 2026/09/21 09:29:18 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo6788worker1 # [ 153.042933] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=13 outputs=16789worker1 # [ 153.052476] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:18Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=16790worker1 # [ 153.063588] nix-daemon[845]: reaped child process 1946, status = succeeded6791worker1 # [ 153.094262] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106792worker1 # [ 153.107539] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106793worker1 # [ 153.120730] nix-grpc-daemon[1485]: time=2026-09-21T09:29:18.283Z level=INFO msg="All 2 paths already cached"6794worker1 # [ 153.128687] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495106795worker1 # [ 153.138257] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6796worker1 # [ 153.358611] nix-daemon[845]: accepted connection from pid 1971, user nix-grpc-daemon (trusted)6797worker1 # [ 153.370733] nix-daemon[845]: reaped child process 1974, status = succeeded6798worker1 # [ 153.594450] nix-daemon[845]: accepted connection from pid 1976, user nix-grpc-daemon (trusted)6799worker1 # [ 153.606812] nix-daemon[845]: reaped child process 1979, status = succeeded6800lb # [ 153.637069] niks3-server[1002]: 2026/09/21 09:29:18 INFO Received uploads request method=POST path=/api/pending_closures6801worker1 # [ 153.640168] nix-grpc-daemon[1485]: time=2026-09-21T09:29:18.802Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6802worker1 # [ 153.642042] nix-grpc-daemon[1485]: time=2026-09-21T09:29:18.804Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"6803lb # [ 153.671314] niks3-server[1002]: 2026/09/21 09:29:18 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst6804lb # [ 153.675873] niks3-server[1002]: 2026/09/21 09:29:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6805worker1 # [ 153.676416] nix-grpc-daemon[1485]: time=2026-09-21T09:29:18.838Z level=INFO msg="Uploading 1 narinfos"6806lb # [ 153.682990] niks3-server[1002]: 2026/09/21 09:29:18 INFO Signed narinfos id=62 count=16807lb # [ 153.692708] niks3-server[1002]: 2026/09/21 09:29:18 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls6808lb # [ 153.700173] niks3-server[1002]: 2026/09/21 09:29:18 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6809lb # [ 153.707527] niks3-server[1002]: 2026/09/21 09:29:18 INFO Completed upload id=626810worker1 # [ 153.700982] nix-grpc-daemon[1485]: time=2026-09-21T09:29:18.863Z level=INFO msg="Upload complete. (550ms)"6811lb # [ 153.709950] niks3-server[1002]: 2026/09/21 09:29:18 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo6812worker1 # [ 153.704608] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=0 paths=1 nar_bytes_in=7526813worker1 # [ 153.709880] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495106814worker1 # [ 153.721294] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6815worker1 # [ 153.726669] nix-daemon[845]: reaped child process 1969, status = succeeded6816worker1 # [ 153.730171] nix-daemon[845]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6817worker1 # [ 154.952181] nix-daemon[845]: reaped child process 1983, status = succeeded6818worker1 # [ 155.152457] nix-daemon[845]: accepted connection from pid 1999, user nix-grpc-daemon (trusted)6819worker1 # [ 155.163225] nix-daemon[845]: reaped child process 2002, status = succeeded6820lb # [ 155.194134] niks3-server[1002]: 2026/09/21 09:29:20 INFO Received uploads request method=POST path=/api/pending_closures6821worker1 # [ 155.198397] nix-grpc-daemon[1485]: time=2026-09-21T09:29:20.360Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6822worker1 # [ 155.199897] nix-grpc-daemon[1485]: time=2026-09-21T09:29:20.362Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"6823lb # [ 155.232059] niks3-server[1002]: 2026/09/21 09:29:20 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv6824lb # [ 155.242110] niks3-server[1002]: 2026/09/21 09:29:20 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst6825lb # [ 155.245323] niks3-server[1002]: 2026/09/21 09:29:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6826lb # [ 155.249810] niks3-server[1002]: 2026/09/21 09:29:20 INFO Signed narinfos id=63 count=16827worker1 # [ 155.245629] nix-grpc-daemon[1485]: time=2026-09-21T09:29:20.407Z level=INFO msg="Uploading 1 narinfos"6828lb # [ 155.258833] niks3-server[1002]: 2026/09/21 09:29:20 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6829lb # [ 155.261883] niks3-server[1002]: 2026/09/21 09:29:20 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls6830lb # [ 155.264718] niks3-server[1002]: 2026/09/21 09:29:20 INFO Completed upload id=636831worker1 # [ 155.258328] nix-grpc-daemon[1485]: time=2026-09-21T09:29:20.420Z level=INFO msg="Upload complete. (320ms)"6832lb # [ 155.267102] niks3-server[1002]: 2026/09/21 09:29:20 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo6833worker1 # [ 155.263780] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49510 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=14 outputs=16834worker1 # [ 155.275073] nix-daemon[845]: reaped child process 1981, status = succeeded6835client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 19.01 seconds)6836worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6837worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6838client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'6839client # [ 155.528741] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6840client # Running as unit: gpu.service; invocation ID: 42ac96be0b8c4a4d8bdab2f90b2fc1606841client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.07 seconds)6842lb: must succeed: curl -sf localhost:9901/clusters6843lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6844lb: must succeed: curl -sf localhost:9901/clusters6845lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6846worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6847worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6848worker2 # [ 155.791933] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:20Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:462626849client # [ 155.789370] nix[1364]: don't know how to build these paths:6850client # [ 155.790828] nix[1364]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv6851worker1 # [ 155.764744] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495106852client # [ 155.813902] nix[1364]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker6853worker1 # [ 155.784861] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:20Z level=debug event=want client=6 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=06854lb # [ 156.207696] envoy[902]: conn peer=[2001:db8:1::1]:40702 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27141 tx=11545 ms=186946855lb: must succeed: curl -sf localhost:9901/clusters6856lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6857lb: must succeed: curl -sf localhost:9901/clusters6858lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6859worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6860worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6861client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6862client # [ 185.816306] nix[1364]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}6863client # [ 185.824189] nix[1364]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed6864client # [ 185.873122] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6865client # [ 185.874382] systemd[1]: gpu.service: Failed with result 'exit-code'.6866client # [ 185.875921] systemd[1]: gpu.service: Consumed 145ms CPU time over 30.344s wall clock time, 35.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.6867client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 28.91 seconds)6868client: must succeed: journalctl -u gpu -o cat6869client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)6870lb: must succeed: curl -sf localhost:9901/clusters6871lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6872lb: must succeed: curl -sf localhost:9901/clusters6873lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6874worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6875worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6876(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 49.73 seconds)6877subtest: low disk drains a worker and builds go to the other6878worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6879lb # [ 186.209702] envoy[902]: conn peer=[2001:db8:1::1]:41500 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300586880worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.00 seconds)6881worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6882worker1 # [ 187.163110] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:52Z level=info event=unhealthy reason=min_free6883worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.13 seconds)6884client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&26885client # don't know how to build these paths:6886client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv6887worker1 # [ 188.476694] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:495106888worker2 # [ 188.519726] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462626889worker1 # [ 188.504075] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:53Z level=debug event=want client=7 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=06890worker1 # [ 188.508393] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:53Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=16891worker2 # [ 188.585392] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462626892worker2 # [ 188.598612] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462626893client # copying 1 paths to 'lb:50051'...6894client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...6895worker2 # [ 188.611823] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462626896worker2 # [ 188.621923] nix-daemon[869]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)6897worker2 # [ 188.645306] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:53Z level=info event=niks3_push_started pid=18826898worker2 # [ 188.864256] nix-grpc-daemon[1882]: time=2026-09-21T09:29:53.222Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6899worker2 # [ 189.050086] nix-daemon[869]: accepted connection from pid 1895, user nix-grpc-daemon (trusted)6900worker2 # [ 189.060866] nix-daemon[869]: reaped child process 1898, status = succeeded6901worker2 # [ 189.253139] nix-daemon[869]: accepted connection from pid 1900, user nix-grpc-daemon (trusted)6902worker2 # [ 189.264665] nix-daemon[869]: reaped child process 1903, status = succeeded6903lb # [ 189.255553] niks3-server[1002]: 2026/09/21 09:29:54 INFO Received uploads request method=POST path=/api/pending_closures6904worker2 # [ 189.296885] nix-grpc-daemon[1882]: time=2026-09-21T09:29:53.655Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6905worker2 # [ 189.300270] nix-grpc-daemon[1882]: time=2026-09-21T09:29:53.657Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"6906lb # [ 189.291751] niks3-server[1002]: 2026/09/21 09:29:54 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst6907lb # [ 189.301773] niks3-server[1002]: 2026/09/21 09:29:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6908lb # [ 189.307608] niks3-server[1002]: 2026/09/21 09:29:54 INFO Signed narinfos id=64 count=16909worker2 # [ 189.341252] nix-grpc-daemon[1882]: time=2026-09-21T09:29:53.700Z level=INFO msg="Uploading 1 narinfos"6910lb # [ 189.317874] niks3-server[1002]: 2026/09/21 09:29:54 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls6911lb # [ 189.327862] niks3-server[1002]: 2026/09/21 09:29:54 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6912lb # [ 189.334944] niks3-server[1002]: 2026/09/21 09:29:54 INFO Completed upload id=646913worker2 # [ 189.365346] nix-grpc-daemon[1882]: time=2026-09-21T09:29:53.724Z level=INFO msg="Upload complete. (497ms)"6914lb # [ 189.338380] niks3-server[1002]: 2026/09/21 09:29:54 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo6915worker2 # [ 189.370119] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46262 duration_s=0 paths=1 nar_bytes_in=5126916worker2 # [ 189.377148] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462626917worker2 # [ 189.385119] nix-daemon[869]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)6918worker2 # [ 189.390538] nix-daemon[869]: reaped child process 1880, status = succeeded6919worker2 # [ 189.395702] nix-daemon[869]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)6920client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...6921client # farm-a> worker2-next: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv6922client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6923client # farm-a> LOG-farm-a-drain6924worker2 # [ 190.007288] nix-daemon[869]: reaped child process 1907, status = succeeded6925worker2 # [ 190.180846] nix-daemon[869]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6926worker2 # [ 190.191654] nix-daemon[869]: reaped child process 1926, status = succeeded6927lb # [ 190.179581] niks3-server[1002]: 2026/09/21 09:29:55 INFO Received uploads request method=POST path=/api/pending_closures6928worker2 # [ 190.219766] nix-grpc-daemon[1882]: time=2026-09-21T09:29:54.578Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6929worker2 # [ 190.221497] nix-grpc-daemon[1882]: time=2026-09-21T09:29:54.580Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"6930lb # [ 190.218251] niks3-server[1002]: 2026/09/21 09:29:55 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv6931lb # [ 190.222508] niks3-server[1002]: 2026/09/21 09:29:55 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6932lb # [ 190.232047] niks3-server[1002]: 2026/09/21 09:29:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6933lb # [ 190.235775] niks3-server[1002]: 2026/09/21 09:29:55 INFO Signed narinfos id=65 count=16934worker2 # [ 190.270302] nix-grpc-daemon[1882]: time=2026-09-21T09:29:54.628Z level=INFO msg="Uploading 1 narinfos"6935lb # [ 190.242208] niks3-server[1002]: 2026/09/21 09:29:55 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls6936lb # [ 190.247163] niks3-server[1002]: 2026/09/21 09:29:55 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6937lb # [ 190.251039] niks3-server[1002]: 2026/09/21 09:29:55 INFO Completed upload id=656938worker2 # [ 190.281493] nix-grpc-daemon[1882]: time=2026-09-21T09:29:54.640Z level=INFO msg="Upload complete. (280ms)"6939lb # [ 190.252653] niks3-server[1002]: 2026/09/21 09:29:55 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo6940worker2 # [ 190.285969] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46262 duration_s=0 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=15 outputs=16941worker2 # [ 190.295569] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462626942worker1 # [ 190.256865] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:55Z level=debug event=want client=7 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=06943worker2 # [ 190.299961] nix-daemon[869]: reaped child process 1905, status = succeeded6944worker1 # [ 190.261077] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:55Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=16945worker2 # [ 190.304384] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462626946worker2 # [ 190.310387] nix-grpc-daemon[1882]: time=2026-09-21T09:29:54.669Z level=INFO msg="All 1 paths already cached"6947client # copying 1 paths to 'lb:50051'...6948worker2 # [ 190.315758] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462626949client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...6950worker2 # [ 190.321464] nix-daemon[869]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)6951worker2 # [ 190.496067] nix-daemon[869]: accepted connection from pid 1930, user nix-grpc-daemon (trusted)6952worker2 # [ 190.507921] nix-daemon[869]: reaped child process 1933, status = succeeded6953worker2 # [ 190.696444] nix-daemon[869]: accepted connection from pid 1935, user nix-grpc-daemon (trusted)6954worker2 # [ 190.706669] nix-daemon[869]: reaped child process 1938, status = succeeded6955lb # [ 190.695534] niks3-server[1002]: 2026/09/21 09:29:55 INFO Received uploads request method=POST path=/api/pending_closures6956worker2 # [ 190.734180] nix-grpc-daemon[1882]: time=2026-09-21T09:29:55.093Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6957worker2 # [ 190.735710] nix-grpc-daemon[1882]: time=2026-09-21T09:29:55.094Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"6958lb # [ 190.728922] niks3-server[1002]: 2026/09/21 09:29:55 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst6959lb # [ 190.740444] niks3-server[1002]: 2026/09/21 09:29:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6960lb # [ 190.745435] niks3-server[1002]: 2026/09/21 09:29:55 INFO Signed narinfos id=66 count=16961worker2 # [ 190.778481] nix-grpc-daemon[1882]: time=2026-09-21T09:29:55.137Z level=INFO msg="Uploading 1 narinfos"6962lb # [ 190.751152] niks3-server[1002]: 2026/09/21 09:29:55 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls6963lb # [ 190.760057] niks3-server[1002]: 2026/09/21 09:29:55 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6964lb # [ 190.763678] niks3-server[1002]: 2026/09/21 09:29:55 INFO Completed upload id=666965worker2 # [ 190.795732] nix-grpc-daemon[1882]: time=2026-09-21T09:29:55.154Z level=INFO msg="Upload complete. (467ms)"6966lb # [ 190.766293] niks3-server[1002]: 2026/09/21 09:29:55 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo6967worker2 # [ 190.800208] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46262 duration_s=0 paths=1 nar_bytes_in=6246968worker2 # [ 190.805261] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462626969worker2 # [ 190.813885] nix-daemon[869]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)6970worker2 # [ 190.820399] nix-daemon[869]: reaped child process 1928, status = succeeded6971worker2 # [ 190.825420] nix-daemon[869]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)6972client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...6973client # farm-b> worker2-next: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv6974client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6975client # farm-b> LOG-farm-b-drain6976worker2 # [ 191.449562] nix-daemon[869]: reaped child process 1942, status = succeeded6977worker2 # [ 191.632284] nix-daemon[869]: accepted connection from pid 1958, user nix-grpc-daemon (trusted)6978worker2 # [ 191.642675] nix-daemon[869]: reaped child process 1961, status = succeeded6979lb # [ 191.630651] niks3-server[1002]: 2026/09/21 09:29:56 INFO Received uploads request method=POST path=/api/pending_closures6980worker2 # [ 191.669678] nix-grpc-daemon[1882]: time=2026-09-21T09:29:56.028Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6981worker2 # [ 191.671222] nix-grpc-daemon[1882]: time=2026-09-21T09:29:56.030Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"6982lb # [ 191.671273] niks3-server[1002]: 2026/09/21 09:29:56 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst6983lb # [ 191.679517] niks3-server[1002]: 2026/09/21 09:29:56 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv6984lb # [ 191.683574] niks3-server[1002]: 2026/09/21 09:29:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6985lb # [ 191.690577] niks3-server[1002]: 2026/09/21 09:29:56 INFO Signed narinfos id=67 count=16986worker2 # [ 191.724326] nix-grpc-daemon[1882]: time=2026-09-21T09:29:56.083Z level=INFO msg="Uploading 1 narinfos"6987lb # [ 191.698643] niks3-server[1002]: 2026/09/21 09:29:56 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls6988lb # [ 191.706105] niks3-server[1002]: 2026/09/21 09:29:56 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6989lb # [ 191.712963] niks3-server[1002]: 2026/09/21 09:29:56 INFO Completed upload id=676990worker2 # [ 191.743426] nix-grpc-daemon[1882]: time=2026-09-21T09:29:56.102Z level=INFO msg="Upload complete. (311ms)"6991lb # [ 191.716782] niks3-server[1002]: 2026/09/21 09:29:56 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo6992worker2 # [ 191.748937] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46262 duration_s=0 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=16 outputs=16993worker2 # [ 191.761759] nix-daemon[869]: reaped child process 1940, status = succeeded6994worker1 # [ 191.762779] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:56Z level=debug event=want client=7 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=06995worker1 # [ 191.766866] nix-grpc-daemon[1407]: ts=2026-09-21T09:29:56Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=16996worker2 # [ 191.843355] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462626997worker2 # [ 191.855686] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462626998worker2 # [ 191.868398] nix-grpc-daemon[1882]: time=2026-09-21T09:29:56.227Z level=INFO msg="All 2 paths already cached"6999client # copying 1 paths to 'lb:50051'...7000client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7001worker2 # [ 191.877839] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462627002worker2 # [ 191.886168] nix-daemon[869]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)7003worker2 # [ 192.084391] nix-daemon[869]: accepted connection from pid 1965, user nix-grpc-daemon (trusted)7004worker2 # [ 192.096517] nix-daemon[869]: reaped child process 1968, status = succeeded7005worker2 # [ 192.277575] nix-daemon[869]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7006worker2 # [ 192.289050] nix-daemon[869]: reaped child process 1973, status = succeeded7007lb # [ 192.277668] niks3-server[1002]: 2026/09/21 09:29:57 INFO Received uploads request method=POST path=/api/pending_closures7008worker2 # [ 192.315746] nix-grpc-daemon[1882]: time=2026-09-21T09:29:56.674Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7009worker2 # [ 192.317256] nix-grpc-daemon[1882]: time=2026-09-21T09:29:56.676Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7010lb # [ 192.304178] niks3-server[1002]: 2026/09/21 09:29:57 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7011lb # [ 192.314634] niks3-server[1002]: 2026/09/21 09:29:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7012lb # [ 192.318764] niks3-server[1002]: 2026/09/21 09:29:57 INFO Signed narinfos id=68 count=17013worker2 # [ 192.352397] nix-grpc-daemon[1882]: time=2026-09-21T09:29:56.711Z level=INFO msg="Uploading 1 narinfos"7014lb # [ 192.331648] niks3-server[1002]: 2026/09/21 09:29:57 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7015lb # [ 192.339103] niks3-server[1002]: 2026/09/21 09:29:57 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7016lb # [ 192.344249] niks3-server[1002]: 2026/09/21 09:29:57 INFO Completed upload id=687017worker2 # [ 192.374805] nix-grpc-daemon[1882]: time=2026-09-21T09:29:56.733Z level=INFO msg="Upload complete. (475ms)"7018lb # [ 192.346973] niks3-server[1002]: 2026/09/21 09:29:57 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7019worker2 # [ 192.378299] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46262 duration_s=0 paths=1 nar_bytes_in=7527020worker2 # [ 192.384350] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462627021worker2 # [ 192.393129] nix-daemon[869]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)7022worker2 # [ 192.397955] nix-daemon[869]: reaped child process 1963, status = succeeded7023worker2 # [ 192.403137] nix-daemon[869]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)7024client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7025client # farm-top> worker2-next: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7026client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7027client # farm-top> LOG-farm-top-drain7028worker2 # [ 193.427814] nix-daemon[869]: reaped child process 1977, status = succeeded7029worker2 # [ 193.608923] nix-daemon[869]: accepted connection from pid 1993, user nix-grpc-daemon (trusted)7030worker2 # [ 193.620648] nix-daemon[869]: reaped child process 1996, status = succeeded7031lb # [ 193.610524] niks3-server[1002]: 2026/09/21 09:29:58 INFO Received uploads request method=POST path=/api/pending_closures7032worker2 # [ 193.649739] nix-grpc-daemon[1882]: time=2026-09-21T09:29:58.008Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7033worker2 # [ 193.651298] nix-grpc-daemon[1882]: time=2026-09-21T09:29:58.010Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7034lb # [ 193.650886] niks3-server[1002]: 2026/09/21 09:29:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7035lb # [ 193.660509] niks3-server[1002]: 2026/09/21 09:29:58 INFO Signed narinfos id=69 count=17036worker2 # [ 193.694249] nix-grpc-daemon[1882]: time=2026-09-21T09:29:58.053Z level=INFO msg="Uploading 1 narinfos"7037lb # [ 193.672894] niks3-server[1002]: 2026/09/21 09:29:58 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7038lb # [ 193.676675] niks3-server[1002]: 2026/09/21 09:29:58 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7039lb # [ 193.686286] niks3-server[1002]: 2026/09/21 09:29:58 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7040lb # [ 193.689226] niks3-server[1002]: 2026/09/21 09:29:58 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7041lb # [ 193.693423] niks3-server[1002]: 2026/09/21 09:29:58 INFO Completed upload id=697042worker2 # [ 193.724203] nix-grpc-daemon[1882]: time=2026-09-21T09:29:58.083Z level=INFO msg="Upload complete. (314ms)"7043lb # [ 193.695219] niks3-server[1002]: 2026/09/21 09:29:58 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7044worker2 # [ 193.728667] nix-grpc-daemon[1822]: ts=2026-09-21T09:29:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46262 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=17 outputs=17045worker2 # [ 193.739984] nix-daemon[869]: reaped child process 1975, status = succeeded7046client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 5.52 seconds)7047client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7048client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.24 seconds)7049worker1: must succeed: rm /nix/.rw-store/fill7050worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.14 seconds)7051worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7052worker1 # [ 195.167697] nix-grpc-daemon[1407]: ts=2026-09-21T09:30:00Z level=info event=healthy reason=min_free7053worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.13 seconds)7054(finished: subtest: low disk drains a worker and builds go to the other, in 9.25 seconds)7055(finished: run the VM test script, in 196.25 seconds)7056test script finished in 196.53s7057cleanup7058kill QemuMachine (pid 49)7059client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7060kill QemuMachine (pid 50)7061lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7062kill QemuMachine (pid 51)7063worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7064kill QemuMachine (pid 52)7065worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7066(finished: cleanup, in 1.00 seconds)7067additionally exposed symbols:7068 client, lb, worker1, worker2,7069 vlan1,7070 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh