vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #463
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12client: QEMU running (pid 49)13client # Disk image does not exist, creating the virtualisation disk image...14worker1 # Disk image does not exist, creating the virtualisation disk image...15worker1: QEMU running (pid 52)16worker1 # Formatting '/build/vm-state-worker1/tmp.tLymje88hO', fmt=raw size=107374182417lb: QEMU running (pid 50)18lb # Disk image does not exist, creating the virtualisation disk image...19client # Formatting '/build/vm-state-client/tmp.9rYk4Mqobb', fmt=raw size=107374182420lb # Formatting '/build/vm-state-lb/tmp.tr1awW8C9G', fmt=raw size=107374182421client # mke2fs 1.47.4 (6-Mar-2025)22lb # mke2fs 1.47.4 (6-Mar-2025)23client # Discarding device blocks: 0/262144 done24lb # Discarding device blocks: 0/262144 done25worker1 # mke2fs 1.47.4 (6-Mar-2025)26client # Creating filesystem with 262144 4k blocks and 65536 inodes27lb # Creating filesystem with 262144 4k blocks and 65536 inodes28worker2: QEMU running (pid 51)29client # Filesystem UUID: c63df857-06de-4972-a07a-1b5ddc0a439a30lb # Filesystem UUID: 34b3303e-9a79-44d3-8982-e577401d3b5d31worker1 # Discarding device blocks: 0/262144 done32worker2 # Disk image does not exist, creating the virtualisation disk image...33client # Superblock backups stored on blocks:34worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes35worker2 # Formatting '/build/vm-state-worker2/tmp.HtluNLNt64', fmt=raw size=107374182436client # 32768, 98304, 163840, 22937637worker1 # Filesystem UUID: b3d738d0-fc7e-49bf-a8c3-439e023b0a4e38worker2 # mke2fs 1.47.4 (6-Mar-2025)39client # 40worker1 # Superblock backups stored on blocks:41worker2 # Discarding device blocks: 0/262144 done42client # Allocating group tables: 0/8 done43worker1 # 32768, 98304, 163840, 22937644worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes45client # Writing inode tables: 0/8 done46worker1 # 47worker2 # Filesystem UUID: 654fcd5d-2247-4d1a-beb6-a7f140a0d70748client # Creating journal (8192 blocks): done49client # Writing superblocks and filesystem accounting information: 0/8 done50worker1 # Allocating group tables: 0/8 done51(finished: start all VMs, in 0.21 seconds)52worker2 # Superblock backups stored on blocks:53client: waiting for unit network-addresses-eth1.service54worker1 # Writing inode tables: 0/8 done55worker2 # 32768, 98304, 163840, 22937656client: waiting for the VM to finish booting57worker1 # Creating journal (8192 blocks): done58worker2 # 59client # 60lb # Superblock backups stored on blocks:61worker1 # Writing superblocks and filesystem accounting information: 0/8 done62worker2 # Allocating group tables: 0/8 done63client # Virtualisation disk image created.64lb # 32768, 98304, 163840, 22937665worker1 # 66lb # 67worker1 # Virtualisation disk image created.68lb # Allocating group tables: 0/8 done69worker2 # Writing inode tables: 0/8 done70lb # Writing inode tables: 0/8 done71worker2 # Creating journal (8192 blocks): done72lb # Creating journal (8192 blocks): done73worker2 # Writing superblocks and filesystem accounting information: 0/8 done74lb # Writing superblocks and filesystem accounting information: 0/8 done75worker2 # 76lb # 77worker2 # Virtualisation disk image created.78lb # Virtualisation disk image created.79worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83worker1 # 84worker1 # 85worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0086worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...87worker1 # 88worker1 # 89worker1 # 90worker1 # 91worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB0092worker2 # 93worker2 # 94worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...95worker1 # 96worker1 # 97worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0098worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...99worker2 # 100worker2 # 101worker2 # 102worker2 # 103worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00104client # 105client # 106lb # 107lb # 108worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...109worker2 # 110worker2 # 111client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00112lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00113client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...114client # 115client # 116client # 117client # 118lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...119lb # 120lb # 121lb # 122lb # 123client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00124lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00125lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...126client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...127lb # 128client # 129lb # 130client # 131worker1 # Booting from ROM...132worker2 # Booting from ROM...133lb # Booting from ROM...134client # Booting from ROM...135worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/dfinz1jb80f7kddpy2i5dm45jl5sdafn-nixos-system-worker1-test/init regInfo=/nix/store/vh6vnxsr6kpl55n796qpvfi7vmfgnn54-closure-info/registration console=ttyS0,115200n8 console=tty0137worker1 # [ 0.000000] BIOS-provided physical RAM map:138worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved141worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable142worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved143worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved144worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved145worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved146worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved147worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved148worker1 # [ 0.000000] NX (Execute Disable) protection: active149worker1 # [ 0.000000] APIC: Static calls initialized150worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026151worker1 # [ 0.000000] SMBIOS 2.8 present.152worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014153worker1 # [ 0.000000] DMI: Memory slots populated: 1/1154worker1 # [ 0.000000] Hypervisor detected: KVM155worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000156worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00157worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ajnaamjsnp85c2azh20np9g3bhal9f4k-nixos-system-worker2-test/init regInfo=/nix/store/lrq2vv6va7xb5ai7lrqfa5bclv9db1av-closure-info/registration console=ttyS0,115200n8 console=tty0158worker1 # [ 0.000000] kvm-clock: using sched offset of 471470683 cycles159worker2 # [ 0.000000] BIOS-provided physical RAM map:160worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable161worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns162worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved163worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor164worker1 # [ 0.000805] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000165worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved166worker1 # [ 0.000832] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs167worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable168worker1 # [ 0.000835] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT169worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved170worker1 # [ 0.002725] found SMP MP-table at [mem 0x000f5450-0x000f545f]171worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved172worker1 # [ 0.002737] Using GB pages for direct mapping173worker1 # [ 0.002874] RAMDISK: [mem 0x5e367000-0x5ffcffff]174worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved175worker1 # [ 0.002881] ACPI: Early table checksum verification disabled176worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved177lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026178worker1 # [ 0.002884] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )179worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved180worker1 # [ 0.002888] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)181client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026182worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved183worker2 # [ 0.000000] NX (Execute Disable) protection: active184worker1 # [ 0.002892] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)185worker2 # [ 0.000000] APIC: Static calls initialized186worker2 # [ 0.000000] SMBIOS 2.8 present.187worker1 # [ 0.002899] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)188worker1 # [ 0.002901] ACPI: FACS 0x000000005FFE0000 000040189lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/m097agrx0jinhy67cn4afbshq893g59l-nixos-system-lb-test/init regInfo=/nix/store/6ryicbywj6r050v0hx7xqbv6gfi18m0f-closure-info/registration console=ttyS0,115200n8 console=tty0190worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014191lb # [ 0.000000] BIOS-provided physical RAM map:192worker2 # [ 0.000000] DMI: Memory slots populated: 1/1193worker1 # [ 0.002903] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)194worker2 # [ 0.000000] Hypervisor detected: KVM195client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test/init regInfo=/nix/store/5vv6z5wsbvfjs1wlzdz267if252sv9j6-closure-info/registration console=ttyS0,115200n8 console=tty0196lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable197client # [ 0.000000] BIOS-provided physical RAM map:198worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000199lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved200worker1 # [ 0.002904] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)201worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00202client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable203lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved204worker1 # [ 0.002906] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)205worker2 # [ 0.000000] kvm-clock: using sched offset of 458288318 cycles206client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved207lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable208worker1 # [ 0.002907] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)209client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved210worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns211lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved212worker1 # [ 0.002908] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]213client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable214worker2 # [ 0.000004] tsc: Detected 2400.012 MHz processor215lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved216worker2 # [ 0.000811] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000217worker1 # [ 0.002910] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]218client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved219lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved220worker1 # [ 0.002910] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]221worker2 # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs222client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved223lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved224worker2 # [ 0.000840] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT225client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved226worker1 # [ 0.002911] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]227lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved228worker1 # [ 0.002911] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]229client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved230worker2 # [ 0.002735] found SMP MP-table at [mem 0x000f5450-0x000f545f]231worker2 # [ 0.002745] Using GB pages for direct mapping232lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved233worker1 # [ 0.002912] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]234client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved235lb # [ 0.000000] NX (Execute Disable) protection: active236worker2 # [ 0.002817] RAMDISK: [mem 0x5e367000-0x5ffcffff]237lb # [ 0.000000] APIC: Static calls initialized238worker1 # [ 0.002912] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]239worker2 # [ 0.002824] ACPI: Early table checksum verification disabled240client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved241lb # [ 0.000000] SMBIOS 2.8 present.242worker1 # [ 0.003133] No NUMA configuration found243worker2 # [ 0.002827] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )244client # [ 0.000000] NX (Execute Disable) protection: active245worker1 # [ 0.003134] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]246client # [ 0.000000] APIC: Static calls initialized247lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014248client # [ 0.000000] SMBIOS 2.8 present.249worker2 # [ 0.002831] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)250worker1 # [ 0.003136] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]251lb # [ 0.000000] DMI: Memory slots populated: 1/1252worker1 # [ 0.005774] Zone ranges:253lb # [ 0.000000] Hypervisor detected: KVM254worker2 # [ 0.002835] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)255client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014256worker1 # [ 0.005775] DMA [mem 0x0000000000001000-0x0000000000ffffff]257lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000258client # [ 0.000000] DMI: Memory slots populated: 1/1259worker1 # [ 0.005777] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]260worker2 # [ 0.002842] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)261lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00262client # [ 0.000000] Hypervisor detected: KVM263worker1 # [ 0.005778] Normal empty264worker2 # [ 0.002844] ACPI: FACS 0x000000005FFE0000 000040265worker1 # [ 0.005779] Device empty266lb # [ 0.000000] kvm-clock: using sched offset of 485065065 cycles267client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000268worker1 # [ 0.005779] Movable zone start for each node269client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00270worker1 # [ 0.005780] Early memory node ranges271worker2 # [ 0.002845] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)272lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns273client # [ 0.000000] kvm-clock: using sched offset of 494069230 cycles274worker1 # [ 0.005780] node 0: [mem 0x0000000000001000-0x000000000009efff]275lb # [ 0.000004] tsc: Detected 2400.012 MHz processor276worker2 # [ 0.002847] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)277worker1 # [ 0.005781] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]278lb # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000279client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns280worker2 # [ 0.002849] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)281client # [ 0.000005] tsc: Detected 2400.012 MHz processor282worker1 # [ 0.005782] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]283lb # [ 0.000839] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs284client # [ 0.000804] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000285worker2 # [ 0.002850] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)286worker1 # [ 0.005802] On node 0, zone DMA: 1 pages in unavailable ranges287lb # [ 0.000842] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT288worker1 # [ 0.006072] On node 0, zone DMA: 97 pages in unavailable ranges289worker2 # [ 0.002851] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]290client # [ 0.000832] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs291lb # [ 0.002741] found SMP MP-table at [mem 0x000f5450-0x000f545f]292worker1 # [ 0.033215] On node 0, zone DMA32: 40 pages in unavailable ranges293lb # [ 0.002752] Using GB pages for direct mapping294worker2 # [ 0.002852] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]295client # [ 0.000834] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT296worker1 # [ 0.033664] ACPI: PM-Timer IO Port: 0x608297lb # [ 0.002883] RAMDISK: [mem 0x5e367000-0x5ffcffff]298client # [ 0.002723] found SMP MP-table at [mem 0x000f5450-0x000f545f]299worker1 # [ 0.033675] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])300worker2 # [ 0.002853] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]301client # [ 0.002734] Using GB pages for direct mapping302lb # [ 0.002892] ACPI: Early table checksum verification disabled303worker2 # [ 0.002853] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]304worker1 # [ 0.033701] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23305client # [ 0.002802] RAMDISK: [mem 0x5e366000-0x5ffcffff]306lb # [ 0.002894] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )307worker1 # [ 0.033703] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)308client # [ 0.002810] ACPI: Early table checksum verification disabled309worker2 # [ 0.002854] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]310lb # [ 0.002898] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)311worker1 # [ 0.033705] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)312client # [ 0.002812] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )313worker2 # [ 0.002854] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]314lb # [ 0.002902] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)315worker1 # [ 0.033706] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)316client # [ 0.002816] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)317worker2 # [ 0.002855] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]318worker2 # [ 0.003082] No NUMA configuration found319worker1 # [ 0.033707] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)320lb # [ 0.002910] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)321client # [ 0.002820] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)322worker2 # [ 0.003083] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]323lb # [ 0.002913] ACPI: FACS 0x000000005FFE0000 000040324worker1 # [ 0.033708] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)325worker2 # [ 0.003085] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]326client # [ 0.002827] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)327worker1 # [ 0.033710] ACPI: Using ACPI (MADT) for SMP configuration information328worker2 # [ 0.005429] Zone ranges:329lb # [ 0.002914] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)330client # [ 0.002829] ACPI: FACS 0x000000005FFE0000 000040331worker1 # [ 0.033711] ACPI: HPET id: 0x8086a201 base: 0xfed00000332worker2 # [ 0.005430] DMA [mem 0x0000000000001000-0x0000000000ffffff]333worker1 # [ 0.033714] TSC deadline timer available334lb # [ 0.002916] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)335client # [ 0.002830] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)336worker2 # [ 0.005431] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]337worker1 # [ 0.033718] CPU topo: Max. logical packages: 1338worker2 # [ 0.005432] Normal empty339worker1 # [ 0.033719] CPU topo: Max. logical dies: 1340lb # [ 0.002917] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)341worker2 # [ 0.005433] Device empty342client # [ 0.002832] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)343worker1 # [ 0.033719] CPU topo: Max. dies per package: 1344worker2 # [ 0.005433] Movable zone start for each node345worker2 # [ 0.005434] Early memory node ranges346lb # [ 0.002919] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)347worker1 # [ 0.033724] CPU topo: Max. threads per core: 1348client # [ 0.002833] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)349worker1 # [ 0.033725] CPU topo: Num. cores per package: 1350worker2 # [ 0.005435] node 0: [mem 0x0000000000001000-0x000000000009efff]351lb # [ 0.002920] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]352worker1 # [ 0.033725] CPU topo: Num. threads per package: 1353client # [ 0.002835] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)354worker2 # [ 0.005436] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]355lb # [ 0.002921] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]356worker1 # [ 0.033726] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs357client # [ 0.002836] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]358worker2 # [ 0.005437] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]359lb # [ 0.002922] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]360worker1 # [ 0.033744] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()361client # [ 0.002837] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]362worker2 # [ 0.005460] On node 0, zone DMA: 1 pages in unavailable ranges363lb # [ 0.002922] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]364worker1 # [ 0.033773] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]365worker2 # [ 0.005735] On node 0, zone DMA: 97 pages in unavailable ranges366client # [ 0.002838] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]367lb # [ 0.002922] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]368worker2 # [ 0.033143] On node 0, zone DMA32: 40 pages in unavailable ranges369worker1 # [ 0.033775] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]370client # [ 0.002838] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]371worker2 # [ 0.033606] ACPI: PM-Timer IO Port: 0x608372lb # [ 0.002923] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]373worker1 # [ 0.033776] [mem 0x60000000-0xafffffff] available for PCI devices374client # [ 0.002839] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]375worker2 # [ 0.033618] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])376worker1 # [ 0.033778] Booting paravirtualized kernel on KVM377lb # [ 0.002923] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]378client # [ 0.002839] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]379lb # [ 0.003156] No NUMA configuration found380worker2 # [ 0.033645] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23381worker1 # [ 0.033781] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns382lb # [ 0.003156] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]383client # [ 0.002840] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]384worker2 # [ 0.033648] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)385client # [ 0.003064] No NUMA configuration found386lb # [ 0.003159] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]387worker1 # [ 0.038223] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1388worker2 # [ 0.033649] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)389lb # [ 0.005480] Zone ranges:390client # [ 0.003065] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]391worker2 # [ 0.033650] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)392worker1 # [ 0.040421] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152393lb # [ 0.005481] DMA [mem 0x0000000000001000-0x0000000000ffffff]394client # [ 0.003067] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]395worker1 # [ 0.040464] kvm-guest: PV spinlocks disabled, single CPU396client # [ 0.005903] Zone ranges:397worker2 # [ 0.033651] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)398lb # [ 0.005482] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]399lb # [ 0.005483] Normal empty400client # [ 0.005904] DMA [mem 0x0000000000001000-0x0000000000ffffff]401lb # [ 0.005484] Device empty402worker2 # [ 0.033652] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)403client # [ 0.005906] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]404lb # [ 0.005484] Movable zone start for each node405client # [ 0.005907] Normal empty406lb # [ 0.005485] Early memory node ranges407worker2 # [ 0.033654] ACPI: Using ACPI (MADT) for SMP configuration information408client # [ 0.005908] Device empty409worker2 # [ 0.033655] ACPI: HPET id: 0x8086a201 base: 0xfed00000410lb # [ 0.005486] node 0: [mem 0x0000000000001000-0x000000000009efff]411client # [ 0.005908] Movable zone start for each node412worker2 # [ 0.033659] TSC deadline timer available413client # [ 0.005909] Early memory node ranges414lb # [ 0.005487] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]415worker2 # [ 0.033663] CPU topo: Max. logical packages: 1416worker1 # [ 0.040466] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/dfinz1jb80f7kddpy2i5dm45jl5sdafn-nixos-system-worker1-test/init regInfo=/nix/store/vh6vnxsr6kpl55n796qpvfi7vmfgnn54-closure-info/registration console=ttyS0,115200n8 console=tty0417client # [ 0.005909] node 0: [mem 0x0000000000001000-0x000000000009efff]418worker2 # [ 0.033664] CPU topo: Max. logical dies: 1419lb # [ 0.005488] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]420worker2 # [ 0.033664] CPU topo: Max. dies per package: 1421client # [ 0.005910] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]422lb # [ 0.005508] On node 0, zone DMA: 1 pages in unavailable ranges423worker2 # [ 0.033668] CPU topo: Max. threads per core: 1424worker1 # [ 0.040557] Unknown kernel command line parameters "regInfo=/nix/store/vh6vnxsr6kpl55n796qpvfi7vmfgnn54-closure-info/registration", will be passed to user space.425client # [ 0.005911] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]426lb # [ 0.005774] On node 0, zone DMA: 97 pages in unavailable ranges427worker2 # [ 0.033668] CPU topo: Num. cores per package: 1428worker1 # [ 0.040570] random: crng init done429client # [ 0.005935] On node 0, zone DMA: 1 pages in unavailable ranges430worker2 # [ 0.033669] CPU topo: Num. threads per package: 1431lb # [ 0.032285] On node 0, zone DMA32: 40 pages in unavailable ranges432worker1 # [ 0.040570] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes433client # [ 0.006294] On node 0, zone DMA: 97 pages in unavailable ranges434lb # [ 0.032758] ACPI: PM-Timer IO Port: 0x608435worker2 # [ 0.033669] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs436worker1 # [ 0.042733] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)437lb # [ 0.032768] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])438client # [ 0.033355] On node 0, zone DMA32: 40 pages in unavailable ranges439worker2 # [ 0.033687] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()440client # [ 0.033807] ACPI: PM-Timer IO Port: 0x608441lb # [ 0.032798] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23442worker1 # [ 0.043817] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)443worker2 # [ 0.033718] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]444client # [ 0.033818] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])445worker1 # [ 0.043853] Fallback order for Node 0: 0446lb # [ 0.032801] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)447worker2 # [ 0.033720] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]448client # [ 0.033843] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23449worker1 # [ 0.043856] Built 1 zonelists, mobility grouping on. Total pages: 393078450lb # [ 0.032802] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)451worker1 # [ 0.043857] Policy zone: DMA32452worker2 # [ 0.033721] [mem 0x60000000-0xafffffff] available for PCI devices453client # [ 0.033846] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)454lb # [ 0.032804] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)455worker2 # [ 0.033722] Booting paravirtualized kernel on KVM456worker1 # [ 0.046645] mem auto-init: stack:all(zero), heap alloc:on, heap free:off457client # [ 0.033848] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)458lb # [ 0.032805] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)459worker1 # [ 0.049651] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1460client # [ 0.033849] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)461worker2 # [ 0.033725] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns462worker1 # [ 0.053013] allocated 3145728 bytes of page_ext463lb # [ 0.032805] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)464client # [ 0.033850] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)465worker1 # [ 0.062958] ftrace: allocating 48731 entries in 192 pages466worker2 # [ 0.038173] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1467lb # [ 0.032808] ACPI: Using ACPI (MADT) for SMP configuration information468worker1 # [ 0.062961] ftrace: allocated 192 pages with 2 groups469client # [ 0.033851] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)470lb # [ 0.032809] ACPI: HPET id: 0x8086a201 base: 0xfed00000471worker2 # [ 0.040661] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152472worker1 # [ 0.063824] Dynamic Preempt: lazy473lb # [ 0.032813] TSC deadline timer available474client # [ 0.033853] ACPI: Using ACPI (MADT) for SMP configuration information475worker2 # [ 0.040703] kvm-guest: PV spinlocks disabled, single CPU476worker1 # [ 0.063987] rcu: Preemptible hierarchical RCU implementation.477lb # [ 0.032817] CPU topo: Max. logical packages: 1478client # [ 0.033854] ACPI: HPET id: 0x8086a201 base: 0xfed00000479worker1 # [ 0.063987] rcu: RCU event tracing is enabled.480lb # [ 0.032818] CPU topo: Max. logical dies: 1481client # [ 0.033858] TSC deadline timer available482lb # [ 0.032818] CPU topo: Max. dies per package: 1483client # [ 0.033862] CPU topo: Max. logical packages: 1484worker1 # [ 0.063988] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.485lb # [ 0.032822] CPU topo: Max. threads per core: 1486client # [ 0.033863] CPU topo: Max. logical dies: 1487worker1 # [ 0.063990] Trampoline variant of Tasks RCU enabled.488lb # [ 0.032822] CPU topo: Num. cores per package: 1489client # [ 0.033863] CPU topo: Max. dies per package: 1490worker1 # [ 0.063990] Rude variant of Tasks RCU enabled.491client # [ 0.033867] CPU topo: Max. threads per core: 1492worker1 # [ 0.063990] Tracing variant of Tasks RCU enabled.493lb # [ 0.032823] CPU topo: Num. threads per package: 1494worker2 # [ 0.040704] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ajnaamjsnp85c2azh20np9g3bhal9f4k-nixos-system-worker2-test/init regInfo=/nix/store/lrq2vv6va7xb5ai7lrqfa5bclv9db1av-closure-info/registration console=ttyS0,115200n8 console=tty0495client # [ 0.033868] CPU topo: Num. cores per package: 1496lb # [ 0.032823] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs497worker1 # [ 0.063991] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.498client # [ 0.033868] CPU topo: Num. threads per package: 1499lb # [ 0.032843] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()500worker1 # [ 0.063992] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1501client # [ 0.033868] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs502worker2 # [ 0.040795] Unknown kernel command line parameters "regInfo=/nix/store/lrq2vv6va7xb5ai7lrqfa5bclv9db1av-closure-info/registration", will be passed to user space.503lb # [ 0.032877] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]504worker2 # [ 0.040808] random: crng init done505client # [ 0.033886] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()506worker1 # [ 0.064018] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.507lb # [ 0.032879] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]508worker2 # [ 0.040808] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes509client # [ 0.033917] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]510worker1 # [ 0.064020] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.511lb # [ 0.032880] [mem 0x60000000-0xafffffff] available for PCI devices512worker2 # [ 0.042989] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)513client # [ 0.033918] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]514lb # [ 0.032881] Booting paravirtualized kernel on KVM515worker1 # [ 0.064021] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.516client # [ 0.033920] [mem 0x60000000-0xafffffff] available for PCI devices517worker2 # [ 0.044079] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)518worker1 # [ 0.068435] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16519client # [ 0.033921] Booting paravirtualized kernel on KVM520worker2 # [ 0.044111] Fallback order for Node 0: 0521lb # [ 0.032885] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns522worker1 # [ 0.068706] rcu: srcu_init: Setting srcu_struct sizes based on contention.523worker2 # [ 0.044114] Built 1 zonelists, mobility grouping on. Total pages: 393078524lb # [ 0.037349] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1525client # [ 0.033924] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns526worker2 # [ 0.044115] Policy zone: DMA32527lb # [ 0.039513] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152528worker1 # [ 0.068712] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns529worker2 # [ 0.046657] mem auto-init: stack:all(zero), heap alloc:on, heap free:off530client # [ 0.038375] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1531lb # [ 0.039557] kvm-guest: PV spinlocks disabled, single CPU532worker2 # [ 0.049754] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1533worker1 # [ 0.068815] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)534client # [ 0.040571] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152535worker2 # [ 0.053448] allocated 3145728 bytes of page_ext536worker1 # [ 0.072331] Console: colour VGA+ 80x25537client # [ 0.040610] kvm-guest: PV spinlocks disabled, single CPU538worker1 # [ 0.072334] printk: legacy console [tty0] enabled539worker2 # [ 0.063303] ftrace: allocating 48731 entries in 192 pages540worker1 # [ 0.101787] printk: legacy console [ttyS0] enabled541worker2 # [ 0.063305] ftrace: allocated 192 pages with 2 groups542worker2 # [ 0.064146] Dynamic Preempt: lazy543worker1 # [ 0.196870] ACPI: Core revision 20250807544worker2 # [ 0.064318] rcu: Preemptible hierarchical RCU implementation.545lb # [ 0.039559] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/m097agrx0jinhy67cn4afbshq893g59l-nixos-system-lb-test/init regInfo=/nix/store/6ryicbywj6r050v0hx7xqbv6gfi18m0f-closure-info/registration console=ttyS0,115200n8 console=tty0546worker2 # [ 0.064318] rcu: RCU event tracing is enabled.547worker1 # [ 0.197762] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns548worker2 # [ 0.064319] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.549client # [ 0.040612] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test/init regInfo=/nix/store/5vv6z5wsbvfjs1wlzdz267if252sv9j6-closure-info/registration console=ttyS0,115200n8 console=tty0550worker2 # [ 0.064321] Trampoline variant of Tasks RCU enabled.551worker1 # [ 0.199242] APIC: Switch to symmetric I/O mode setup552lb # [ 0.039652] Unknown kernel command line parameters "regInfo=/nix/store/6ryicbywj6r050v0hx7xqbv6gfi18m0f-closure-info/registration", will be passed to user space.553worker2 # [ 0.064321] Rude variant of Tasks RCU enabled.554lb # [ 0.039664] random: crng init done555worker1 # [ 0.200168] x2apic enabled556worker2 # [ 0.064322] Tracing variant of Tasks RCU enabled.557client # [ 0.040705] Unknown kernel command line parameters "regInfo=/nix/store/5vv6z5wsbvfjs1wlzdz267if252sv9j6-closure-info/registration", will be passed to user space.558lb # [ 0.039665] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes559client # [ 0.040718] random: crng init done560worker1 # [ 0.200899] APIC: Switched APIC routing to: physical x2apic561worker2 # [ 0.064322] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.562lb # [ 0.041785] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)563client # [ 0.040718] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes564worker2 # [ 0.064323] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1565lb # [ 0.043058] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)566client # [ 0.042872] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)567worker2 # [ 0.064338] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.568lb # [ 0.043088] Fallback order for Node 0: 0569worker1 # [ 0.202664] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1570client # [ 0.043946] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)571lb # [ 0.043091] Built 1 zonelists, mobility grouping on. Total pages: 393078572worker2 # [ 0.064340] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.573client # [ 0.043977] Fallback order for Node 0: 0574lb # [ 0.043092] Policy zone: DMA32575worker1 # [ 0.203645] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns576client # [ 0.043979] Built 1 zonelists, mobility grouping on. Total pages: 393078577lb # [ 0.046018] mem auto-init: stack:all(zero), heap alloc:on, heap free:off578worker2 # [ 0.064341] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.579client # [ 0.043980] Policy zone: DMA32580lb # [ 0.049100] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1581worker2 # [ 0.068715] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16582worker1 # [ 0.205232] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)583client # [ 0.046765] mem auto-init: stack:all(zero), heap alloc:on, heap free:off584lb # [ 0.052419] allocated 3145728 bytes of page_ext585worker2 # [ 0.068988] rcu: srcu_init: Setting srcu_struct sizes based on contention.586lb # [ 0.062108] ftrace: allocating 48731 entries in 192 pages587client # [ 0.049877] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1588worker1 # [ 0.207314] x86/cpu: User Mode Instruction Prevention (UMIP) activated589client # [ 0.053215] allocated 3145728 bytes of page_ext590lb # [ 0.062110] ftrace: allocated 192 pages with 2 groups591worker2 # [ 0.068994] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns592lb # [ 0.062973] Dynamic Preempt: lazy593client # [ 0.063111] ftrace: allocating 48731 entries in 192 pages594worker1 # [ 0.208355] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127595lb # [ 0.063145] rcu: Preemptible hierarchical RCU implementation.596client # [ 0.063113] ftrace: allocated 192 pages with 2 groups597worker2 # [ 0.069094] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)598client # [ 0.063956] Dynamic Preempt: lazy599lb # [ 0.063146] rcu: RCU event tracing is enabled.600worker1 # [ 0.209230] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0601worker2 # [ 0.072602] Console: colour VGA+ 80x25602client # [ 0.064128] rcu: Preemptible hierarchical RCU implementation.603lb # [ 0.063146] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.604worker2 # [ 0.072605] printk: legacy console [tty0] enabled605client # [ 0.064129] rcu: RCU event tracing is enabled.606lb # [ 0.063148] Trampoline variant of Tasks RCU enabled.607worker2 # [ 0.101972] printk: legacy console [ttyS0] enabled608worker1 # [ 0.210233] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto609lb # [ 0.063149] Rude variant of Tasks RCU enabled.610client # [ 0.064129] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.611worker2 # [ 0.197762] ACPI: Core revision 20250807612lb # [ 0.063149] Tracing variant of Tasks RCU enabled.613client # [ 0.064131] Trampoline variant of Tasks RCU enabled.614worker1 # [ 0.211230] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl615client # [ 0.064131] Rude variant of Tasks RCU enabled.616lb # [ 0.063150] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.617client # [ 0.064132] Tracing variant of Tasks RCU enabled.618worker2 # [ 0.198623] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns619worker1 # [ 0.213230] Transient Scheduler Attacks: Vulnerable: No microcode620lb # [ 0.063150] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1621client # [ 0.064132] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.622worker2 # [ 0.200182] APIC: Switch to symmetric I/O mode setup623worker1 # [ 0.214229] Spectre V2 : Mitigation: Enhanced / Automatic IBRS624lb # [ 0.063166] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.625worker2 # [ 0.201112] x2apic enabled626client # [ 0.064133] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1627worker1 # [ 0.215229] Speculative Return Stack Overflow: Mitigation: Safe RET628lb # [ 0.063168] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.629client # [ 0.064149] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.630worker2 # [ 0.201849] APIC: Switched APIC routing to: physical x2apic631lb # [ 0.063169] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.632client # [ 0.064150] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.633lb # [ 0.067814] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16634worker2 # [ 0.203627] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1635client # [ 0.064151] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.636lb # [ 0.068090] rcu: srcu_init: Setting srcu_struct sizes based on contention.637client # [ 0.068479] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16638worker1 # [ 0.216229] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization639worker2 # [ 0.204581] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns640lb # [ 0.068098] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns641client # [ 0.068756] rcu: srcu_init: Setting srcu_struct sizes based on contention.642worker1 # [ 0.218236] Spectre V2 : Enabling IBPB for BPF643lb # [ 0.068203] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)644worker2 # [ 0.206143] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)645client # [ 0.068763] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns646lb # [ 0.071741] Console: colour VGA+ 80x25647worker1 # [ 0.219743] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier648lb # [ 0.071743] printk: legacy console [tty0] enabled649worker2 # [ 0.208227] x86/cpu: User Mode Instruction Prevention (UMIP) activated650client # [ 0.068867] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)651worker1 # [ 0.221187] active return thunk: srso_alias_return_thunk652lb # [ 0.101095] printk: legacy console [ttyS0] enabled653client # [ 0.072389] Console: colour VGA+ 80x25654client # [ 0.072392] printk: legacy console [tty0] enabled655worker2 # [ 0.209289] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127656lb # [ 0.196734] ACPI: Core revision 20250807657worker1 # [ 0.221856] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'658client # [ 0.101845] printk: legacy console [ttyS0] enabled659worker2 # [ 0.210141] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0660worker1 # [ 0.223229] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'661client # [ 0.197654] ACPI: Core revision 20250807662lb # [ 0.197622] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns663worker1 # [ 0.224166] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'664lb # [ 0.199178] APIC: Switch to symmetric I/O mode setup665worker2 # [ 0.211145] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto666client # [ 0.198574] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns667worker1 # [ 0.224955] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'668lb # [ 0.200115] x2apic enabled669client # [ 0.200156] APIC: Switch to symmetric I/O mode setup670worker2 # [ 0.212141] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl671worker1 # [ 0.225962] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'672lb # [ 0.200870] APIC: Switched APIC routing to: physical x2apic673client # [ 0.201115] x2apic enabled674worker2 # [ 0.213141] Transient Scheduler Attacks: Vulnerable: No microcode675worker1 # [ 0.226954] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'676client # [ 0.201864] APIC: Switched APIC routing to: physical x2apic677worker2 # [ 0.214141] Spectre V2 : Mitigation: Enhanced / Automatic IBRS678worker1 # [ 0.227992] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'679lb # [ 0.202665] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1680worker2 # [ 0.215141] Speculative Return Stack Overflow: Mitigation: Safe RET681worker1 # [ 0.229115] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256682client # [ 0.203667] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1683lb # [ 0.203629] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns684worker1 # [ 0.229930] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64685worker1 # [ 0.230936] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512686client # [ 0.204616] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns687lb # [ 0.205276] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)688worker1 # [ 0.231930] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024689lb # [ 0.206361] x86/cpu: User Mode Instruction Prevention (UMIP) activated690client # [ 0.206253] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)691worker2 # [ 0.216140] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization692worker1 # [ 0.232229] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8693worker2 # [ 0.218148] Spectre V2 : Enabling IBPB for BPF694lb # [ 0.207448] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127695client # [ 0.207338] x86/cpu: User Mode Instruction Prevention (UMIP) activated696worker1 # [ 0.233229] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.697lb # [ 0.208273] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0698client # [ 0.208378] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127699worker2 # [ 0.218855] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier700worker2 # [ 0.220141] active return thunk: srso_alias_return_thunk701client # [ 0.209252] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0702lb # [ 0.209277] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto703worker2 # [ 0.221162] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'704client # [ 0.210255] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto705lb # [ 0.210273] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl706worker2 # [ 0.222141] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'707lb # [ 0.211273] Transient Scheduler Attacks: Vulnerable: No microcode708client # [ 0.211251] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl709worker2 # [ 0.223141] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'710lb # [ 0.212273] Spectre V2 : Mitigation: Enhanced / Automatic IBRS711client # [ 0.212251] Transient Scheduler Attacks: Vulnerable: No microcode712worker2 # [ 0.224140] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'713lb # [ 0.213273] Speculative Return Stack Overflow: Mitigation: Safe RET714client # [ 0.213251] Spectre V2 : Mitigation: Enhanced / Automatic IBRS715worker2 # [ 0.225140] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'716client # [ 0.214251] Speculative Return Stack Overflow: Mitigation: Safe RET717worker2 # [ 0.226140] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'718worker2 # [ 0.227140] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'719worker2 # [ 0.228141] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256720lb # [ 0.214273] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization721worker2 # [ 0.229141] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64722lb # [ 0.216280] Spectre V2 : Enabling IBPB for BPF723client # [ 0.215251] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization724worker2 # [ 0.230141] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512725client # [ 0.217259] Spectre V2 : Enabling IBPB for BPF726lb # [ 0.217273] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier727worker2 # [ 0.231140] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024728lb # [ 0.218273] active return thunk: srso_alias_return_thunk729worker2 # [ 0.232140] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8730client # [ 0.218252] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier731client # [ 0.219251] active return thunk: srso_alias_return_thunk732lb # [ 0.219294] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'733worker2 # [ 0.233053] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.734lb # [ 0.220273] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'735client # [ 0.220273] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'736lb # [ 0.221273] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'737client # [ 0.221251] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'738lb # [ 0.222272] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'739client # [ 0.222251] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'740lb # [ 0.223273] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'741client # [ 0.223251] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'742lb # [ 0.224272] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'743client # [ 0.224251] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'744lb # [ 0.225272] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'745client # [ 0.225251] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'746lb # [ 0.226273] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256747client # [ 0.226251] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'748lb # [ 0.227272] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64749client # [ 0.228251] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256750lb # [ 0.228272] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512751client # [ 0.229251] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64752lb # [ 0.229273] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024753client # [ 0.230251] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512754lb # [ 0.230272] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8755client # [ 0.231251] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024756lb # [ 0.231273] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.757client # [ 0.232251] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8758client # [ 0.233218] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759worker1 # [ 0.267539] Freeing SMP alternatives memory: 44K760worker1 # [ 0.268232] pid_max: default: 32768 minimum: 301761worker1 # [ 0.269037] LSM: initializing lsm=capability,landlock,yama,bpf,ima762worker1 # [ 0.270038] landlock: Up and running.763worker1 # [ 0.270654] Yama: becoming mindful.764worker1 # [ 0.271445] LSM support for eBPF active765worker1 # [ 0.272333] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)766worker1 # [ 0.273273] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)767worker1 # [ 0.276093] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)768worker1 # [ 0.276762] Performance Events: Fam17h+ core perfctr, AMD PMU driver.769worker1 # [ 0.277234] ... version: 2770worker1 # [ 0.277918] ... bit width: 48771worker1 # [ 0.278232] ... generic counters: 6772worker1 # [ 0.278953] ... generic bitmap: 000000000000003f773worker2 # [ 0.266992] Freeing SMP alternatives memory: 44K774worker1 # [ 0.279283] ... fixed-purpose counters: 0775worker2 # [ 0.267698] pid_max: default: 32768 minimum: 301776worker1 # [ 0.279967] ... fixed-purpose bitmap: 0000000000000000777worker2 # [ 0.268243] LSM: initializing lsm=capability,landlock,yama,bpf,ima778worker1 # [ 0.280232] ... value mask: 0000ffffffffffff779worker2 # [ 0.269242] landlock: Up and running.780worker2 # [ 0.270141] Yama: becoming mindful.781worker1 # [ 0.281086] ... max period: 00007fffffffffff782worker1 # [ 0.281883] ... global_ctrl mask: 000000000000003f783worker2 # [ 0.271205] LSM support for eBPF active784worker1 # [ 0.282344] signal: max sigframe size: 3376785worker2 # [ 0.271993] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)786worker1 # [ 0.283098] rcu: Hierarchical SRCU implementation.787worker1 # [ 0.283785] rcu: Max phase no-delay instances is 400.788worker2 # [ 0.273181] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)789worker2 # [ 0.275576] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)790worker2 # [ 0.276705] Performance Events: Fam17h+ core perfctr, AMD PMU driver.791worker2 # [ 0.277146] ... version: 2792worker1 # [ 0.288720] smp: Bringing up secondary CPUs ...793worker2 # [ 0.277875] ... bit width: 48794worker1 # [ 0.289245] smp: Brought up 1 node, 1 CPU795lb # [ 0.266468] Freeing SMP alternatives memory: 44K796worker2 # [ 0.278143] ... generic counters: 6797lb # [ 0.267242] pid_max: default: 32768 minimum: 301798worker1 # [ 0.289890] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)799client # [ 0.267467] Freeing SMP alternatives memory: 44K800worker2 # [ 0.278838] ... generic bitmap: 000000000000003f801lb # [ 0.267924] LSM: initializing lsm=capability,landlock,yama,bpf,ima802client # [ 0.268222] pid_max: default: 32768 minimum: 301803worker2 # [ 0.279149] ... fixed-purpose counters: 0804lb # [ 0.269104] landlock: Up and running.805worker2 # [ 0.279819] ... fixed-purpose bitmap: 0000000000000000806worker1 # [ 0.290418] Memory: 1454572K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)807client # [ 0.268860] LSM: initializing lsm=capability,landlock,yama,bpf,ima808lb # [ 0.269699] Yama: becoming mindful.809client # [ 0.270044] landlock: Up and running.810worker2 # [ 0.280143] ... value mask: 0000ffffffffffff811worker1 # [ 0.291466] devtmpfs: initialized812lb # [ 0.270497] LSM support for eBPF active813client # [ 0.270673] Yama: becoming mindful.814worker2 # [ 0.281010] ... max period: 00007fffffffffff815worker1 # [ 0.292222] x86/mm: Memory block size: 128MB816client # [ 0.271464] LSM support for eBPF active817lb # [ 0.271346] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)818worker2 # [ 0.281798] ... global_ctrl mask: 000000000000003f819worker2 # [ 0.282240] signal: max sigframe size: 3376820client # [ 0.272279] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)821lb # [ 0.272316] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)822worker1 # [ 0.293807] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)823worker2 # [ 0.282999] rcu: Hierarchical SRCU implementation.824client # [ 0.273288] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)825worker2 # [ 0.283724] rcu: Max phase no-delay instances is 400.826worker1 # [ 0.294259] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).827worker1 # [ 0.295316] pinctrl core: initialized pinctrl subsystem828lb # [ 0.274852] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)829worker1 # [ 0.296397] PM: RTC time: 09:11:49, date: 2026-09-21830client # [ 0.275773] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)831lb # [ 0.275814] Performance Events: Fam17h+ core perfctr, AMD PMU driver.832lb # [ 0.276278] ... version: 2833client # [ 0.276833] Performance Events: Fam17h+ core perfctr, AMD PMU driver.834lb # [ 0.277035] ... bit width: 48835lb # [ 0.277275] ... generic counters: 6836client # [ 0.277255] ... version: 2837worker2 # [ 0.288679] smp: Bringing up secondary CPUs ...838client # [ 0.277953] ... bit width: 48839lb # [ 0.277959] ... generic bitmap: 000000000000003f840worker1 # [ 0.299948] NET: Registered PF_NETLINK/PF_ROUTE protocol family841worker2 # [ 0.289158] smp: Brought up 1 node, 1 CPU842client # [ 0.278254] ... generic counters: 6843lb # [ 0.278285] ... fixed-purpose counters: 0844worker2 # [ 0.289828] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)845client # [ 0.278949] ... generic bitmap: 000000000000003f846lb # [ 0.278989] ... fixed-purpose bitmap: 0000000000000000847worker1 # [ 0.300749] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations848client # [ 0.279263] ... fixed-purpose counters: 0849lb # [ 0.279275] ... value mask: 0000ffffffffffff850worker1 # [ 0.301254] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations851client # [ 0.279933] ... fixed-purpose bitmap: 0000000000000000852worker2 # [ 0.290271] Memory: 1454604K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)853lb # [ 0.280142] ... max period: 00007fffffffffff854client # [ 0.280253] ... value mask: 0000ffffffffffff855lb # [ 0.280936] ... global_ctrl mask: 000000000000003f856worker2 # [ 0.291408] devtmpfs: initialized857worker1 # [ 0.302513] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations858client # [ 0.281122] ... max period: 00007fffffffffff859lb # [ 0.281400] signal: max sigframe size: 3376860worker2 # [ 0.292227] x86/mm: Memory block size: 128MB861worker1 # [ 0.303242] audit: initializing netlink subsys (disabled)862client # [ 0.281913] ... global_ctrl mask: 000000000000003f863lb # [ 0.282182] rcu: Hierarchical SRCU implementation.864client # [ 0.282372] signal: max sigframe size: 3376865lb # [ 0.282856] rcu: Max phase no-delay instances is 400.866worker1 # [ 0.304234] audit: type=2000 audit(1789981910.489:1): state=initialized audit_enabled=0 res=1867client # [ 0.283133] rcu: Hierarchical SRCU implementation.868worker2 # [ 0.293966] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)869worker1 # [ 0.305393] thermal_sys: Registered thermal governor 'fair_share'870client # [ 0.283809] rcu: Max phase no-delay instances is 400.871worker1 # [ 0.305395] thermal_sys: Registered thermal governor 'bang_bang'872worker2 # [ 0.295028] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).873worker1 # [ 0.306233] thermal_sys: Registered thermal governor 'step_wise'874worker2 # [ 0.296240] pinctrl core: initialized pinctrl subsystem875worker1 # [ 0.307152] thermal_sys: Registered thermal governor 'user_space'876worker2 # [ 0.297434] PM: RTC time: 09:11:49, date: 2026-09-21877worker1 # [ 0.307925] thermal_sys: Registered thermal governor 'power_allocator'878lb # [ 0.287880] smp: Bringing up secondary CPUs ...879worker1 # [ 0.308245] cpuidle: using governor menu880lb # [ 0.288288] smp: Brought up 1 node, 1 CPU881client # [ 0.288784] smp: Bringing up secondary CPUs ...882lb # [ 0.288939] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)883worker1 # [ 0.310437] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5884client # [ 0.289266] smp: Brought up 1 node, 1 CPU885client # [ 0.289925] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)886worker2 # [ 0.301220] NET: Registered PF_NETLINK/PF_ROUTE protocol family887worker1 # [ 0.311496] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]888lb # [ 0.289423] Memory: 1454564K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)889worker1 # [ 0.312235] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry890worker2 # [ 0.302650] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations891lb # [ 0.290532] devtmpfs: initialized892client # [ 0.290455] Memory: 1454552K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)893worker1 # [ 0.313234] PCI: Using configuration type 1 for base access894lb # [ 0.291307] x86/mm: Memory block size: 128MB895worker2 # [ 0.303166] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations896client # [ 0.291487] devtmpfs: initialized897worker1 # [ 0.314283] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.898client # [ 0.292266] x86/mm: Memory block size: 128MB899worker2 # [ 0.304421] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations900lb # [ 0.293048] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)901worker2 # [ 0.305154] audit: initializing netlink subsys (disabled)902lb # [ 0.294158] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).903client # [ 0.294015] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)904worker2 # [ 0.306303] thermal_sys: Registered thermal governor 'fair_share'905lb # [ 0.295353] pinctrl core: initialized pinctrl subsystem906worker2 # [ 0.306305] thermal_sys: Registered thermal governor 'bang_bang'907client # [ 0.295137] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).908lb # [ 0.296452] PM: RTC time: 09:11:49, date: 2026-09-21909worker2 # [ 0.307144] thermal_sys: Registered thermal governor 'step_wise'910client # [ 0.296341] pinctrl core: initialized pinctrl subsystem911worker2 # [ 0.308092] thermal_sys: Registered thermal governor 'user_space'912client # [ 0.297464] PM: RTC time: 09:11:49, date: 2026-09-21913worker2 # [ 0.308887] audit: type=2000 audit(1789981910.503:1): state=initialized audit_enabled=0 res=1914worker1 # [ 0.321248] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages915lb # [ 0.300009] NET: Registered PF_NETLINK/PF_ROUTE protocol family916worker2 # [ 0.310147] thermal_sys: Registered thermal governor 'power_allocator'917worker1 # [ 0.322232] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page918worker2 # [ 0.310161] cpuidle: using governor menu919lb # [ 0.300785] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations920client # [ 0.301066] NET: Registered PF_NETLINK/PF_ROUTE protocol family921worker2 # [ 0.312399] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5922lb # [ 0.301300] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations923client # [ 0.302476] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations924worker2 # [ 0.313388] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]925lb # [ 0.302553] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations926client # [ 0.303275] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations927worker2 # [ 0.314146] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry928lb # [ 0.303286] audit: initializing netlink subsys (disabled)929worker1 # [ 0.327233] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages930worker2 # [ 0.315155] PCI: Using configuration type 1 for base access931client # [ 0.304531] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations932worker1 # [ 0.328232] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page933lb # [ 0.304278] audit: type=2000 audit(1789981910.514:1): state=initialized audit_enabled=0 res=1934client # [ 0.305263] audit: initializing netlink subsys (disabled)935worker2 # [ 0.316229] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.936lb # [ 0.305422] thermal_sys: Registered thermal governor 'fair_share'937client # [ 0.306444] thermal_sys: Registered thermal governor 'fair_share'938lb # [ 0.305424] thermal_sys: Registered thermal governor 'bang_bang'939client # [ 0.306446] thermal_sys: Registered thermal governor 'bang_bang'940lb # [ 0.306276] thermal_sys: Registered thermal governor 'step_wise'941client # [ 0.307255] thermal_sys: Registered thermal governor 'step_wise'942lb # [ 0.307197] thermal_sys: Registered thermal governor 'user_space'943client # [ 0.308217] thermal_sys: Registered thermal governor 'user_space'944lb # [ 0.307983] thermal_sys: Registered thermal governor 'power_allocator'945lb # [ 0.308288] cpuidle: using governor menu946client # [ 0.309006] audit: type=2000 audit(1789981910.516:1): state=initialized audit_enabled=0 res=1947client # [ 0.310256] thermal_sys: Registered thermal governor 'power_allocator'948lb # [ 0.310465] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5949client # [ 0.310279] cpuidle: using governor menu950worker2 # [ 0.323294] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages951lb # [ 0.311496] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]952worker2 # [ 0.324144] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page953client # [ 0.312483] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5954lb # [ 0.312278] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry955worker2 # [ 0.325137] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages956lb # [ 0.313288] PCI: Using configuration type 1 for base access957worker1 # [ 0.339532] ACPI: Added _OSI(Module Device)958client # [ 0.313498] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]959worker1 # [ 0.340212] ACPI: Added _OSI(Processor Device)960client # [ 0.314256] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry961lb # [ 0.314350] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.962worker1 # [ 0.340759] ACPI: Added _OSI(Processor Aggregator Device)963client # [ 0.315266] PCI: Using configuration type 1 for base access964client # [ 0.316396] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.965worker1 # [ 0.346882] ACPI: 1 ACPI AML tables successfully acquired and loaded966worker2 # [ 0.334144] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page967lb # [ 0.321546] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages968client # [ 0.321542] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages969lb # [ 0.322276] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page970client # [ 0.322255] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page971lb # [ 0.323257] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages972worker1 # [ 0.350655] ACPI: Interpreter enabled973worker1 # [ 0.351247] ACPI: PM: (supports S0 S3 S4 S5)974worker1 # [ 0.351938] ACPI: Using IOAPIC for interrupt routing975worker2 # [ 0.340587] ACPI: Added _OSI(Module Device)976client # [ 0.327254] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages977worker2 # [ 0.341145] ACPI: Added _OSI(Processor Device)978lb # [ 0.328277] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page979client # [ 0.328253] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page980worker2 # [ 0.341863] ACPI: Added _OSI(Processor Aggregator Device)981worker1 # [ 0.354297] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug982worker1 # [ 0.355233] PCI: Using E820 reservations for host bridge windows983worker1 # [ 0.358292] ACPI: Enabled 2 GPEs in block 00 to 3F984worker2 # [ 0.348724] ACPI: 1 ACPI AML tables successfully acquired and loaded985lb # [ 0.336693] ACPI: Added _OSI(Module Device)986lb # [ 0.337278] ACPI: Added _OSI(Processor Device)987lb # [ 0.337978] ACPI: Added _OSI(Processor Aggregator Device)988client # [ 0.338708] ACPI: Added _OSI(Module Device)989client # [ 0.339256] ACPI: Added _OSI(Processor Device)990worker2 # [ 0.352566] ACPI: Interpreter enabled991client # [ 0.340012] ACPI: Added _OSI(Processor Aggregator Device)992worker1 # [ 0.365532] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])993worker2 # [ 0.353158] ACPI: PM: (supports S0 S3 S4 S5)994worker2 # [ 0.353888] ACPI: Using IOAPIC for interrupt routing995worker1 # [ 0.366239] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]996worker1 # [ 0.367312] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]997worker1 # [ 0.368354] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]998lb # [ 0.344800] ACPI: 1 ACPI AML tables successfully acquired and loaded999worker2 # [ 0.356199] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1000worker1 # [ 0.369567] PCI host bridge to bus 0000:001001worker2 # [ 0.357144] PCI: Using E820 reservations for host bridge windows1002worker1 # [ 0.370236] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1003worker1 # [ 0.371232] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1004worker1 # [ 0.372232] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1005client # [ 0.347956] ACPI: 1 ACPI AML tables successfully acquired and loaded1006worker2 # [ 0.360278] ACPI: Enabled 2 GPEs in block 00 to 3F1007lb # [ 0.348680] ACPI: Interpreter enabled1008worker1 # [ 0.373232] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1009lb # [ 0.349291] ACPI: PM: (supports S0 S3 S4 S5)1010lb # [ 0.349984] ACPI: Using IOAPIC for interrupt routing1011worker1 # [ 0.374232] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1012worker1 # [ 0.375232] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1013worker1 # [ 0.376232] pci_bus 0000:00: root bus resource [bus 00-ff]1014client # [ 0.351654] ACPI: Interpreter enabled1015client # [ 0.352269] ACPI: PM: (supports S0 S3 S4 S5)1016lb # [ 0.352332] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1017worker1 # [ 0.377184] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1018client # [ 0.352960] ACPI: Using IOAPIC for interrupt routing1019lb # [ 0.353276] PCI: Using E820 reservations for host bridge windows1020worker1 # [ 0.378608] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1021worker2 # [ 0.367458] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1022worker2 # [ 0.368149] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1023lb # [ 0.356351] ACPI: Enabled 2 GPEs in block 00 to 3F1024client # [ 0.355318] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1025client # [ 0.356255] PCI: Using E820 reservations for host bridge windows1026worker2 # [ 0.369221] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1027worker2 # [ 0.370269] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1028worker2 # [ 0.371510] PCI host bridge to bus 0000:001029client # [ 0.359277] ACPI: Enabled 2 GPEs in block 00 to 3F1030worker2 # [ 0.372148] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1031worker2 # [ 0.373144] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1032worker2 # [ 0.374143] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1033worker1 # [ 0.381313] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1034worker1 # [ 0.382244] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1035worker2 # [ 0.375143] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1036lb # [ 0.364987] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1037worker1 # [ 0.383163] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1038worker2 # [ 0.376143] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1039lb # [ 0.365283] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1040worker2 # [ 0.377145] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1041worker1 # [ 0.384214] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1042worker2 # [ 0.378149] pci_bus 0000:00: root bus resource [bus 00-ff]1043lb # [ 0.366476] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1044client # [ 0.367183] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1045worker1 # [ 0.386016] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1046lb # [ 0.367398] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1047worker2 # [ 0.379209] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1048client # [ 0.368027] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1049lb # [ 0.368631] PCI host bridge to bus 0000:001050worker2 # [ 0.380589] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1051client # [ 0.369333] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1052lb # [ 0.369288] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1053worker1 # [ 0.388274] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1054lb # [ 0.370275] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1055client # [ 0.370384] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1056worker1 # [ 0.389105] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1057client # [ 0.371590] PCI host bridge to bus 0000:001058lb # [ 0.371275] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1059worker1 # [ 0.389970] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1060client # [ 0.372258] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1061lb # [ 0.372275] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1062worker1 # [ 0.390238] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1063client # [ 0.373254] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1064lb # [ 0.373276] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1065client # [ 0.374254] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1066lb # [ 0.374275] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1067worker1 # [ 0.392069] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1068client # [ 0.375254] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1069lb # [ 0.375276] pci_bus 0000:00: root bus resource [bus 00-ff]1070client # [ 0.376254] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1071lb # [ 0.376237] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1072worker1 # [ 0.394255] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1073worker2 # [ 0.383224] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1074client # [ 0.377254] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1075worker1 # [ 0.395094] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1076lb # [ 0.377696] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1077worker2 # [ 0.384156] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1078client # [ 0.378254] pci_bus 0000:00: root bus resource [bus 00-ff]1079worker1 # [ 0.395930] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1080client # [ 0.379176] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1081worker2 # [ 0.385094] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1082worker1 # [ 0.396796] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1083worker2 # [ 0.386365] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1084client # [ 0.380681] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1085worker2 # [ 0.387878] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1086worker1 # [ 0.399339] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1087worker1 # [ 0.400238] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1088worker1 # [ 0.401166] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1089lb # [ 0.380341] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1090worker2 # [ 0.391179] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1091lb # [ 0.381288] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1092worker2 # [ 0.391997] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1093worker1 # [ 0.402582] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1094lb # [ 0.382234] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1095worker2 # [ 0.392898] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1096worker2 # [ 0.393149] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1097lb # [ 0.383238] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1098worker1 # [ 0.404258] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1099client # [ 0.383336] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1100worker1 # [ 0.405064] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1101client # [ 0.384266] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1102worker2 # [ 0.394753] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1103worker1 # [ 0.405932] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1104lb # [ 0.384990] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1105client # [ 0.385237] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1106worker1 # [ 0.406809] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1107client # [ 0.386525] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1108worker2 # [ 0.397128] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1109lb # [ 0.388285] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1110worker2 # [ 0.397800] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1111worker1 # [ 0.408262] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1112client # [ 0.387951] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1113lb # [ 0.389193] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1114worker2 # [ 0.398165] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1115worker1 # [ 0.409066] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1116lb # [ 0.389990] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1117worker1 # [ 0.409937] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1118lb # [ 0.390281] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1119worker2 # [ 0.399689] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1120client # [ 0.391262] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1121worker1 # [ 0.410848] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1122client # [ 0.392131] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1123lb # [ 0.392482] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1124client # [ 0.392999] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1125worker2 # [ 0.402152] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1126client # [ 0.393260] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1127worker1 # [ 0.413263] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1128worker2 # [ 0.403045] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1129lb # [ 0.395247] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1130worker1 # [ 0.414084] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1131worker2 # [ 0.403880] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1132client # [ 0.394960] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1133lb # [ 0.395942] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1134worker1 # [ 0.415231] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1135lb # [ 0.396299] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1136worker2 # [ 0.404697] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1137worker1 # [ 0.416782] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1138client # [ 0.397261] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1139lb # [ 0.397817] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1140client # [ 0.398104] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1141worker1 # [ 0.418261] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1142client # [ 0.399022] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1143worker2 # [ 0.407151] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1144worker1 # [ 0.419079] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1145worker2 # [ 0.408023] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1146worker1 # [ 0.419938] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1147client # [ 0.400704] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1148worker1 # [ 0.420237] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1149worker2 # [ 0.408907] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1150lb # [ 0.400583] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1151lb # [ 0.401961] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1152worker1 # [ 0.421739] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1153worker2 # [ 0.409728] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1154client # [ 0.403268] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1155lb # [ 0.402297] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1156client # [ 0.404091] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1157worker1 # [ 0.423249] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1158client # [ 0.405035] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1159lb # [ 0.403835] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1160worker1 # [ 0.424242] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1161worker2 # [ 0.412176] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1162worker2 # [ 0.413034] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1163client # [ 0.406623] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1164worker1 # [ 0.425779] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1165worker2 # [ 0.413916] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1166lb # [ 0.406238] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1167lb # [ 0.406930] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1168worker1 # [ 0.427271] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1169worker2 # [ 0.414731] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1170client # [ 0.408264] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1171lb # [ 0.407297] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1172worker1 # [ 0.428082] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1173client # [ 0.409103] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1174worker1 # [ 0.428933] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1175lb # [ 0.409526] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1176client # [ 0.410034] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1177worker2 # [ 0.417161] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1178worker1 # [ 0.429798] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1179worker2 # [ 0.418002] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1180client # [ 0.411813] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1181worker2 # [ 0.418895] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1182lb # [ 0.412039] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1183worker1 # [ 0.431262] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1184lb # [ 0.412283] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1185worker1 # [ 0.432077] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1186client # [ 0.413261] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1187worker2 # [ 0.419723] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1188lb # [ 0.413253] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1189worker1 # [ 0.432931] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1190client # [ 0.414095] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1191client # [ 0.415015] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1192lb # [ 0.414718] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1193worker1 # [ 0.433803] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1194worker2 # [ 0.421152] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1195worker2 # [ 0.421994] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1196worker1 # [ 0.434837] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1197client # [ 0.415810] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1198worker1 # [ 0.435431] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1199worker2 # [ 0.422888] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1200lb # [ 0.417539] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1201worker2 # [ 0.423150] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1202worker1 # [ 0.436865] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1203lb # [ 0.418952] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1204client # [ 0.418552] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1205worker2 # [ 0.424712] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1206lb # [ 0.419297] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1207client # [ 0.419859] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1208lb # [ 0.420814] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1209client # [ 0.420275] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1210worker2 # [ 0.426156] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1211worker2 # [ 0.427113] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1212client # [ 0.421849] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1213lb # [ 0.423056] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1214worker2 # [ 0.428577] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1215lb # [ 0.423282] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1216client # [ 0.423270] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1217lb # [ 0.424297] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1218client # [ 0.424150] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1219client # [ 0.425029] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1220worker2 # [ 0.430482] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1221lb # [ 0.425281] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1222worker2 # [ 0.431808] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1223client # [ 0.426134] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1224lb # [ 0.426840] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1225worker2 # [ 0.432166] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1226worker1 # [ 0.438096] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1227client # [ 0.427803] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1228worker2 # [ 0.433685] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1229lb # [ 0.428317] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1230worker1 # [ 0.440261] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1231client # [ 0.429267] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1232lb # [ 0.429297] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1233worker1 # [ 0.441373] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1234client # [ 0.430221] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1235worker2 # [ 0.435863] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1236lb # [ 0.430811] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1237worker2 # [ 0.436153] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1238worker1 # [ 0.443077] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1239client # [ 0.431640] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1240worker2 # [ 0.437122] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1241worker1 # [ 0.444154] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1242client # [ 0.433272] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1243worker1 # [ 0.445539] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1244lb # [ 0.433013] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1245worker2 # [ 0.439260] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1246lb # [ 0.433284] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1247client # [ 0.434124] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1248worker1 # [ 0.446476] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1249lb # [ 0.434247] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1250worker2 # [ 0.440941] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1251client # [ 0.435276] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1252worker2 # [ 0.441971] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1253worker1 # [ 0.448713] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1254lb # [ 0.435655] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1255client # [ 0.436797] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1256worker1 # [ 0.449239] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1257worker2 # [ 0.443867] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1258worker1 # [ 0.450470] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1259lb # [ 0.437977] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1260client # [ 0.438263] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1261lb # [ 0.438283] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1262client # [ 0.439113] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1263worker1 # [ 0.451985] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1264lb # [ 0.439280] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1265client # [ 0.439998] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1266client # [ 0.440815] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1267lb # [ 0.440828] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1268worker1 # [ 0.455127] ACPI: PCI: Interrupt link LNKA configured for IRQ 101269worker1 # [ 0.456051] ACPI: PCI: Interrupt link LNKB configured for IRQ 101270client # [ 0.442620] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1271lb # [ 0.442110] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1272worker1 # [ 0.457169] ACPI: PCI: Interrupt link LNKC configured for IRQ 111273client # [ 0.443450] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1274lb # [ 0.443136] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1275worker1 # [ 0.458034] ACPI: PCI: Interrupt link LNKD configured for IRQ 111276client # [ 0.444852] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1277worker1 # [ 0.459033] ACPI: PCI: Interrupt link LNKE configured for IRQ 101278lb # [ 0.444972] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1279worker1 # [ 0.460043] ACPI: PCI: Interrupt link LNKF configured for IRQ 101280worker1 # [ 0.461045] ACPI: PCI: Interrupt link LNKG configured for IRQ 111281worker2 # [ 0.444347] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1282worker1 # [ 0.462037] ACPI: PCI: Interrupt link LNKH configured for IRQ 111283worker2 # [ 0.446941] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1284worker1 # [ 0.462981] ACPI: PCI: Interrupt link GSIA configured for IRQ 161285worker1 # [ 0.463247] ACPI: PCI: Interrupt link GSIB configured for IRQ 171286worker2 # [ 0.448293] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1287worker1 # [ 0.464191] ACPI: PCI: Interrupt link GSIC configured for IRQ 181288worker1 # [ 0.464955] ACPI: PCI: Interrupt link GSID configured for IRQ 191289worker2 # [ 0.449852] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1290worker1 # [ 0.465244] ACPI: PCI: Interrupt link GSIE configured for IRQ 201291worker1 # [ 0.466240] ACPI: PCI: Interrupt link GSIF configured for IRQ 211292worker2 # [ 0.450413] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1293worker1 # [ 0.467185] ACPI: PCI: Interrupt link GSIG configured for IRQ 221294worker2 # [ 0.451432] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1295worker1 # [ 0.467961] ACPI: PCI: Interrupt link GSIH configured for IRQ 231296worker2 # [ 0.452401] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1297client # [ 0.445497] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1298worker1 # [ 0.469270] iommu: Default domain type: Translated1299lb # [ 0.445473] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1300worker1 # [ 0.470051] iommu: DMA domain TLB invalidation policy: lazy mode1301client # [ 0.447955] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1302worker1 # [ 0.471199] ACPI: bus type USB registered1303lb # [ 0.448010] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1304worker2 # [ 0.454188] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1305worker1 # [ 0.471761] usbcore: registered new interface driver usbfs1306client # [ 0.448486] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1307worker2 # [ 0.455151] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1308worker1 # [ 0.472245] usbcore: registered new interface driver hub1309lb # [ 0.448614] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1310worker1 # [ 0.473073] usbcore: registered new device driver usb1311worker2 # [ 0.456390] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1312client # [ 0.450522] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1313lb # [ 0.449991] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1314worker1 # [ 0.474689] NetLabel: Initializing1315worker2 # [ 0.457959] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1316worker1 # [ 0.475233] NetLabel: domain hash size = 1281317client # [ 0.451515] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1318lb # [ 0.450501] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1319worker1 # [ 0.475915] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1320client # [ 0.452539] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1321worker1 # [ 0.476297] NetLabel: unlabeled traffic allowed by default1322lb # [ 0.451599] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1323worker1 # [ 0.477190] PCI: Using ACPI for IRQ routing1324client # [ 0.453509] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1325worker2 # [ 0.461864] ACPI: PCI: Interrupt link LNKA configured for IRQ 101326lb # [ 0.452543] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1327worker2 # [ 0.462257] ACPI: PCI: Interrupt link LNKB configured for IRQ 101328client # [ 0.455276] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1329client # [ 0.456174] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1330worker2 # [ 0.463253] ACPI: PCI: Interrupt link LNKC configured for IRQ 111331lb # [ 0.454757] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1332worker2 # [ 0.464245] ACPI: PCI: Interrupt link LNKD configured for IRQ 111333lb # [ 0.455282] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1334client # [ 0.457295] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1335worker2 # [ 0.465245] ACPI: PCI: Interrupt link LNKE configured for IRQ 101336worker2 # [ 0.466238] ACPI: PCI: Interrupt link LNKF configured for IRQ 101337lb # [ 0.456598] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1338client # [ 0.458895] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1339worker2 # [ 0.467243] ACPI: PCI: Interrupt link LNKG configured for IRQ 111340lb # [ 0.458094] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1341worker2 # [ 0.468240] ACPI: PCI: Interrupt link LNKH configured for IRQ 111342worker2 # [ 0.469182] ACPI: PCI: Interrupt link GSIA configured for IRQ 161343client # [ 0.462197] ACPI: PCI: Interrupt link LNKA configured for IRQ 101344worker2 # [ 0.470160] ACPI: PCI: Interrupt link GSIB configured for IRQ 171345worker2 # [ 0.471158] ACPI: PCI: Interrupt link GSIC configured for IRQ 181346client # [ 0.463124] ACPI: PCI: Interrupt link LNKB configured for IRQ 101347lb # [ 0.461944] ACPI: PCI: Interrupt link LNKA configured for IRQ 101348worker2 # [ 0.472155] ACPI: PCI: Interrupt link GSID configured for IRQ 191349client # [ 0.464109] ACPI: PCI: Interrupt link LNKC configured for IRQ 111350lb # [ 0.462381] ACPI: PCI: Interrupt link LNKB configured for IRQ 101351worker2 # [ 0.473121] ACPI: PCI: Interrupt link GSIE configured for IRQ 201352client # [ 0.465053] ACPI: PCI: Interrupt link LNKD configured for IRQ 111353worker2 # [ 0.473909] ACPI: PCI: Interrupt link GSIF configured for IRQ 211354client # [ 0.466093] ACPI: PCI: Interrupt link LNKE configured for IRQ 101355lb # [ 0.463376] ACPI: PCI: Interrupt link LNKC configured for IRQ 111356worker2 # [ 0.474159] ACPI: PCI: Interrupt link GSIG configured for IRQ 221357client # [ 0.467237] ACPI: PCI: Interrupt link LNKF configured for IRQ 101358lb # [ 0.464406] ACPI: PCI: Interrupt link LNKD configured for IRQ 111359worker2 # [ 0.475149] ACPI: PCI: Interrupt link GSIH configured for IRQ 231360client # [ 0.468061] ACPI: PCI: Interrupt link LNKG configured for IRQ 111361lb # [ 0.465375] ACPI: PCI: Interrupt link LNKE configured for IRQ 101362client # [ 0.469071] ACPI: PCI: Interrupt link LNKH configured for IRQ 111363lb # [ 0.466373] ACPI: PCI: Interrupt link LNKF configured for IRQ 101364worker2 # [ 0.477225] iommu: Default domain type: Translated1365client # [ 0.470040] ACPI: PCI: Interrupt link GSIA configured for IRQ 161366lb # [ 0.467379] ACPI: PCI: Interrupt link LNKG configured for IRQ 111367worker2 # [ 0.478060] iommu: DMA domain TLB invalidation policy: lazy mode1368client # [ 0.470996] ACPI: PCI: Interrupt link GSIB configured for IRQ 171369lb # [ 0.468373] ACPI: PCI: Interrupt link LNKH configured for IRQ 111370worker2 # [ 0.479169] ACPI: bus type USB registered1371client # [ 0.471268] ACPI: PCI: Interrupt link GSIC configured for IRQ 181372lb # [ 0.469313] ACPI: PCI: Interrupt link GSIA configured for IRQ 161373worker2 # [ 0.479925] usbcore: registered new interface driver usbfs1374client # [ 0.472242] ACPI: PCI: Interrupt link GSID configured for IRQ 191375lb # [ 0.470289] ACPI: PCI: Interrupt link GSIB configured for IRQ 171376worker2 # [ 0.480841] usbcore: registered new interface driver hub1377client # [ 0.472980] ACPI: PCI: Interrupt link GSIE configured for IRQ 201378worker2 # [ 0.481153] usbcore: registered new device driver usb1379lb # [ 0.471261] ACPI: PCI: Interrupt link GSIC configured for IRQ 181380client # [ 0.473265] ACPI: PCI: Interrupt link GSIF configured for IRQ 211381lb # [ 0.472027] ACPI: PCI: Interrupt link GSID configured for IRQ 191382client # [ 0.474230] ACPI: PCI: Interrupt link GSIG configured for IRQ 221383worker2 # [ 0.483199] NetLabel: Initializing1384lb # [ 0.472288] ACPI: PCI: Interrupt link GSIE configured for IRQ 201385worker2 # [ 0.483835] NetLabel: domain hash size = 1281386client # [ 0.475105] ACPI: PCI: Interrupt link GSIH configured for IRQ 231387lb # [ 0.473288] ACPI: PCI: Interrupt link GSIF configured for IRQ 211388worker2 # [ 0.484144] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1389lb # [ 0.474254] ACPI: PCI: Interrupt link GSIG configured for IRQ 221390client # [ 0.476971] iommu: Default domain type: Translated1391worker2 # [ 0.485169] NetLabel: unlabeled traffic allowed by default1392lb # [ 0.475018] ACPI: PCI: Interrupt link GSIH configured for IRQ 231393worker2 # [ 0.486077] PCI: Using ACPI for IRQ routing1394client # [ 0.477261] iommu: DMA domain TLB invalidation policy: lazy mode1395client # [ 0.478456] ACPI: bus type USB registered1396lb # [ 0.476325] iommu: Default domain type: Translated1397client # [ 0.479229] usbcore: registered new interface driver usbfs1398lb # [ 0.477164] iommu: DMA domain TLB invalidation policy: lazy mode1399client # [ 0.479948] usbcore: registered new interface driver hub1400lb # [ 0.478316] ACPI: bus type USB registered1401client # [ 0.480264] usbcore: registered new device driver usb1402lb # [ 0.479053] usbcore: registered new interface driver usbfs1403client # [ 0.482016] NetLabel: Initializing1404lb # [ 0.479294] usbcore: registered new interface driver hub1405client # [ 0.482260] NetLabel: domain hash size = 1281406lb # [ 0.480191] usbcore: registered new device driver usb1407lb # [ 0.481848] NetLabel: Initializing1408client # [ 0.482994] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1409lb # [ 0.482276] NetLabel: domain hash size = 1281410client # [ 0.483325] NetLabel: unlabeled traffic allowed by default1411lb # [ 0.483101] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1412client # [ 0.484268] PCI: Using ACPI for IRQ routing1413lb # [ 0.484020] NetLabel: unlabeled traffic allowed by default1414lb # [ 0.484288] PCI: Using ACPI for IRQ routing1415worker1 # [ 0.520456] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416worker1 # [ 0.521228] pci 0000:00:01.0: vgaarb: bridge control possible1417worker1 # [ 0.521228] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418worker1 # [ 0.521237] vgaarb: loaded1419worker1 # [ 0.521944] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01420worker1 # [ 0.522232] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1421worker1 # [ 0.527317] clocksource: Switched to clocksource kvm-clock1422worker1 # [ 0.531064] VFS: Disk quotas dquot_6.6.01423worker1 # [ 0.531767] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1424worker1 # [ 0.533093] pnp: PnP ACPI init1425worker1 # [ 0.533871] ACPI: IRQ 4 override to edge(!), high(!)1426worker1 # [ 0.534847] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1427worker1 # [ 0.536333] pnp: PnP ACPI: found 6 devices1428worker1 # [ 0.543873] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1429worker1 # [ 0.545266] clocksource: Switched to clocksource acpi_pm1430worker1 # [ 0.546195] NET: Registered PF_INET protocol family1431worker2 # [ 0.530378] pci 0000:00:01.0: vgaarb: setting as boot VGA device1432worker1 # [ 0.547339] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1433worker2 # [ 0.531139] pci 0000:00:01.0: vgaarb: bridge control possible1434worker2 # [ 0.531139] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1435worker2 # [ 0.531150] vgaarb: loaded1436worker2 # [ 0.531923] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01437worker2 # [ 0.532145] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1438client # [ 0.527463] pci 0000:00:01.0: vgaarb: setting as boot VGA device1439client # [ 0.528250] pci 0000:00:01.0: vgaarb: bridge control possible1440worker2 # [ 0.536229] clocksource: Switched to clocksource kvm-clock1441client # [ 0.528250] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1442client # [ 0.528257] vgaarb: loaded1443client # [ 0.529007] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01444lb # [ 0.529452] pci 0000:00:01.0: vgaarb: setting as boot VGA device1445client # [ 0.529255] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1446worker2 # [ 0.540107] VFS: Disk quotas dquot_6.6.01447lb # [ 0.530271] pci 0000:00:01.0: vgaarb: bridge control possible1448worker2 # [ 0.540847] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1449lb # [ 0.530271] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1450lb # [ 0.530281] vgaarb: loaded1451worker2 # [ 0.542256] pnp: PnP ACPI init1452worker2 # [ 0.543011] ACPI: IRQ 4 override to edge(!), high(!)1453lb # [ 0.531040] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01454client # [ 0.532344] clocksource: Switched to clocksource kvm-clock1455worker1 # [ 0.563745] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1456lb # [ 0.531278] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1457worker2 # [ 0.544011] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1458worker1 # [ 0.565215] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1459worker2 # [ 0.545538] pnp: PnP ACPI: found 6 devices1460worker1 # [ 0.566476] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1461client # [ 0.535766] VFS: Disk quotas dquot_6.6.01462worker1 # [ 0.567783] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1463client # [ 0.536518] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1464worker1 # [ 0.568996] TCP: Hash tables configured (established 16384 bind 16384)1465client # [ 0.537861] pnp: PnP ACPI init1466lb # [ 0.536369] clocksource: Switched to clocksource kvm-clock1467worker1 # [ 0.570091] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1468client # [ 0.538690] ACPI: IRQ 4 override to edge(!), high(!)1469worker1 # [ 0.571300] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1470client # [ 0.539667] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1471worker1 # [ 0.572322] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1472client # [ 0.541197] pnp: PnP ACPI: found 6 devices1473worker1 # [ 0.573461] NET: Registered PF_UNIX/PF_LOCAL protocol family1474lb # [ 0.539871] VFS: Disk quotas dquot_6.6.01475worker2 # [ 0.553157] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1476worker1 # [ 0.574405] NET: Registered PF_XDP protocol family1477lb # [ 0.540571] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1478worker2 # [ 0.554617] clocksource: Switched to clocksource acpi_pm1479worker1 # [ 0.575212] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1480lb # [ 0.541929] pnp: PnP ACPI init1481worker2 # [ 0.555575] NET: Registered PF_INET protocol family1482worker1 # [ 0.576185] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1483lb # [ 0.542713] ACPI: IRQ 4 override to edge(!), high(!)1484worker1 # [ 0.577158] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1485worker2 # [ 0.556753] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1486lb # [ 0.543706] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1487worker1 # [ 0.578215] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1488lb # [ 0.545259] pnp: PnP ACPI: found 6 devices1489worker1 # [ 0.579278] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1490worker1 # [ 0.580318] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1491client # [ 0.548777] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1492worker1 # [ 0.582000] ACPI: \_SB_.GSIA: Enabled at IRQ 161493client # [ 0.550269] clocksource: Switched to clocksource acpi_pm1494client # [ 0.551209] NET: Registered PF_INET protocol family1495worker1 # [ 0.584170] ACPI: \_SB_.GSIB: Enabled at IRQ 171496client # [ 0.552414] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1497worker1 # [ 0.586119] ACPI: \_SB_.GSIC: Enabled at IRQ 181498lb # [ 0.552903] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1499worker1 # [ 0.588050] ACPI: \_SB_.GSID: Enabled at IRQ 191500lb # [ 0.554449] clocksource: Switched to clocksource acpi_pm1501lb # [ 0.555461] NET: Registered PF_INET protocol family1502worker1 # [ 0.589706] PCI: CLS 0 bytes, default 641503worker1 # [ 0.590647] Trying to unpack rootfs image as initramfs...1504lb # [ 0.556687] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1505worker2 # [ 0.572905] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1506worker2 # [ 0.574476] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1507worker1 # [ 0.594635] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1508worker2 # [ 0.575801] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1509worker2 # [ 0.577189] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1510worker2 # [ 0.578477] TCP: Hash tables configured (established 16384 bind 16384)1511worker2 # [ 0.579622] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1512client # [ 0.568898] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1513worker2 # [ 0.580997] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1514worker2 # [ 0.582184] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1515client # [ 0.570468] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1516worker2 # [ 0.583403] NET: Registered PF_UNIX/PF_LOCAL protocol family1517client # [ 0.571875] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1518worker2 # [ 0.584376] NET: Registered PF_XDP protocol family1519client # [ 0.573263] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1520worker2 # [ 0.585228] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1521client # [ 0.574555] TCP: Hash tables configured (established 16384 bind 16384)1522worker2 # [ 0.586241] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1523worker2 # [ 0.587264] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1524client # [ 0.575705] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1525lb # [ 0.573380] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1526worker2 # [ 0.588321] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1527client # [ 0.576934] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1528lb # [ 0.574914] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1529worker2 # [ 0.589408] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1530client # [ 0.578003] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1531lb # [ 0.576249] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1532worker2 # [ 0.590480] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1533client # [ 0.579221] NET: Registered PF_UNIX/PF_LOCAL protocol family1534client # [ 0.580151] NET: Registered PF_XDP protocol family1535lb # [ 0.577589] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1536worker2 # [ 0.592219] ACPI: \_SB_.GSIA: Enabled at IRQ 161537client # [ 0.581019] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1538lb # [ 0.578852] TCP: Hash tables configured (established 16384 bind 16384)1539client # [ 0.582029] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1540lb # [ 0.579943] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1541worker2 # [ 0.594798] ACPI: \_SB_.GSIB: Enabled at IRQ 171542client # [ 0.583109] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1543lb # [ 0.581274] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1544client # [ 0.584199] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1545lb # [ 0.582356] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1546worker2 # [ 0.596696] ACPI: \_SB_.GSIC: Enabled at IRQ 181547client # [ 0.585305] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1548lb # [ 0.583588] NET: Registered PF_UNIX/PF_LOCAL protocol family1549lb # [ 0.584564] NET: Registered PF_XDP protocol family1550client # [ 0.586381] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1551worker2 # [ 0.598612] ACPI: \_SB_.GSID: Enabled at IRQ 191552lb # [ 0.585422] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1553client # [ 0.588105] ACPI: \_SB_.GSIA: Enabled at IRQ 161554worker2 # [ 0.600279] PCI: CLS 0 bytes, default 641555lb # [ 0.586455] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1556lb # [ 0.587531] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1557client # [ 0.590226] ACPI: \_SB_.GSIB: Enabled at IRQ 171558worker2 # [ 0.601121] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1559lb # [ 0.588658] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1560worker2 # [ 0.602823] Trying to unpack rootfs image as initramfs...1561client # [ 0.592173] ACPI: \_SB_.GSIC: Enabled at IRQ 181562lb # [ 0.589774] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1563lb # [ 0.590854] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1564client # [ 0.594176] ACPI: \_SB_.GSID: Enabled at IRQ 191565lb # [ 0.592619] ACPI: \_SB_.GSIA: Enabled at IRQ 161566client # [ 0.595965] PCI: CLS 0 bytes, default 641567lb # [ 0.595437] ACPI: \_SB_.GSIB: Enabled at IRQ 171568client # [ 0.596939] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1569client # [ 0.598743] Trying to unpack rootfs image as initramfs...1570lb # [ 0.597447] ACPI: \_SB_.GSIC: Enabled at IRQ 181571worker1 # [ 0.633153] Initialise system trusted keyrings1572lb # [ 0.599431] ACPI: \_SB_.GSID: Enabled at IRQ 191573lb # [ 0.601126] PCI: CLS 0 bytes, default 641574worker1 # [ 0.635589] workingset: timestamp_bits=40 max_order=19 bucket_order=01575lb # [ 0.602013] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1576lb # [ 0.603826] Trying to unpack rootfs image as initramfs...1577worker1 # [ 0.658621] Key type asymmetric registered1578worker1 # [ 0.659312] Asymmetric key parser 'x509' registered1579worker1 # [ 0.660139] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1580worker1 # [ 0.665703] io scheduler mq-deadline registered1581worker1 # [ 0.666434] io scheduler kyber registered1582worker2 # [ 0.646334] Initialise system trusted keyrings1583worker1 # [ 0.669083] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1584worker2 # [ 0.649215] workingset: timestamp_bits=40 max_order=19 bucket_order=01585worker1 # [ 0.670326] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1586client # [ 0.644165] Initialise system trusted keyrings1587worker1 # [ 0.676320] Linux agpgart interface v0.1031588worker1 # [ 0.677088] ACPI: bus type drm_connector registered1589client # [ 0.647567] workingset: timestamp_bits=40 max_order=19 bucket_order=01590worker1 # [ 0.679945] usbcore: registered new interface driver usbserial_generic1591lb # [ 0.646598] Initialise system trusted keyrings1592worker1 # [ 0.680996] usbserial: USB Serial support registered for generic1593lb # [ 0.650246] workingset: timestamp_bits=40 max_order=19 bucket_order=01594worker1 # [ 0.684553] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1595worker1 # [ 0.684553] Please enable it if your BIOS has the CPPC option.1596worker1 # [ 0.686712] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1597worker2 # [ 0.670380] Key type asymmetric registered1598worker1 # [ 0.690681] drop_monitor: Initializing network drop monitor service1599worker2 # [ 0.671066] Asymmetric key parser 'x509' registered1600worker1 # [ 0.691815] NET: Registered PF_INET6 protocol family1601worker2 # [ 0.671973] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1602worker1 # [ 0.694981] Segment Routing with IPv61603worker1 # [ 0.697605] In-situ OAM (IOAM) with IPv61604worker2 # [ 0.677294] io scheduler mq-deadline registered1605worker2 # [ 0.678023] io scheduler kyber registered1606worker1 # [ 0.698609] IPI shorthand broadcast: enabled1607worker2 # [ 0.680697] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1608client # [ 0.670625] Key type asymmetric registered1609worker2 # [ 0.682016] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1610client # [ 0.671320] Asymmetric key parser 'x509' registered1611client # [ 0.672160] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1612lb # [ 0.671435] Key type asymmetric registered1613lb # [ 0.672214] Asymmetric key parser 'x509' registered1614worker1 # [ 0.706837] sched_clock: Marking stable (574014634, 132516277)->(775315513, -68784602)1615client # [ 0.675629] io scheduler mq-deadline registered1616lb # [ 0.673053] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1617worker2 # [ 0.687719] Linux agpgart interface v0.1031618client # [ 0.676345] io scheduler kyber registered1619worker2 # [ 0.688517] ACPI: bus type drm_connector registered1620worker1 # [ 0.711775] registered taskstats version 11621client # [ 0.680158] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1622worker2 # [ 0.691702] usbcore: registered new interface driver usbserial_generic1623lb # [ 0.678360] io scheduler mq-deadline registered1624worker1 # [ 0.712713] Loading compiled-in X.509 certificates1625lb # [ 0.679124] io scheduler kyber registered1626worker2 # [ 0.692763] usbserial: USB Serial support registered for generic1627client # [ 0.681466] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1628lb # [ 0.681780] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1629worker2 # [ 0.696184] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1630lb # [ 0.683063] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1631worker2 # [ 0.696184] Please enable it if your BIOS has the CPPC option.1632client # [ 0.686816] Linux agpgart interface v0.1031633worker2 # [ 0.698449] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1634client # [ 0.687612] ACPI: bus type drm_connector registered1635client # [ 0.689907] usbcore: registered new interface driver usbserial_generic1636lb # [ 0.688593] Linux agpgart interface v0.1031637worker2 # [ 0.702288] drop_monitor: Initializing network drop monitor service1638client # [ 0.690970] usbserial: USB Serial support registered for generic1639lb # [ 0.689421] ACPI: bus type drm_connector registered1640worker2 # [ 0.703457] NET: Registered PF_INET6 protocol family1641client # [ 0.693534] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1642client # [ 0.693534] Please enable it if your BIOS has the CPPC option.1643worker2 # [ 0.706660] Segment Routing with IPv61644lb # [ 0.692640] usbcore: registered new interface driver usbserial_generic1645worker2 # [ 0.707338] In-situ OAM (IOAM) with IPv61646lb # [ 0.693710] usbserial: USB Serial support registered for generic1647client # [ 0.695714] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1648worker2 # [ 0.709453] IPI shorthand broadcast: enabled1649worker1 # [ 0.730545] Demotion targets for Node 0: null1650worker1 # [ 0.731379] Key type .fscrypt registered1651client # [ 0.699754] drop_monitor: Initializing network drop monitor service1652lb # [ 0.697212] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1653worker1 # [ 0.732033] Key type fscrypt-provisioning registered1654lb # [ 0.697212] Please enable it if your BIOS has the CPPC option.1655client # [ 0.700904] NET: Registered PF_INET6 protocol family1656worker1 # [ 0.732972] ima: No TPM chip found, activating TPM-bypass!1657lb # [ 0.699422] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1658worker1 # [ 0.737554] ima: Allocated hash algorithm: sha11659lb # [ 0.703378] drop_monitor: Initializing network drop monitor service1660client # [ 0.706019] Segment Routing with IPv61661worker1 # [ 0.738308] ima: No architecture policies found1662worker2 # [ 0.717789] sched_clock: Marking stable (584015160, 133146015)->(792964225, -75803050)1663lb # [ 0.704566] NET: Registered PF_INET6 protocol family1664worker1 # [ 0.739225] PM: Magic number: 10:167:1721665client # [ 0.709539] In-situ OAM (IOAM) with IPv61666lb # [ 0.707654] Segment Routing with IPv61667client # [ 0.710604] IPI shorthand broadcast: enabled1668lb # [ 0.708425] In-situ OAM (IOAM) with IPv61669worker1 # [ 0.742853] RAS: Correctable Errors collector initialized.1670worker2 # [ 0.723360] registered taskstats version 11671worker2 # [ 0.724280] Loading compiled-in X.509 certificates1672lb # [ 0.711480] IPI shorthand broadcast: enabled1673client # [ 0.718856] sched_clock: Marking stable (585014905, 133487413)->(793262994, -74760676)1674worker1 # [ 0.751457] clk: Disabling unused clocks1675worker1 # [ 0.753547] PM: genpd: Disabling unused power domains1676lb # [ 0.719725] sched_clock: Marking stable (586014619, 133173834)->(796424357, -77235904)1677client # [ 0.723730] registered taskstats version 11678client # [ 0.724731] Loading compiled-in X.509 certificates1679lb # [ 0.724402] registered taskstats version 11680lb # [ 0.725402] Loading compiled-in X.509 certificates1681worker2 # [ 0.740780] Demotion targets for Node 0: null1682worker2 # [ 0.744216] Key type .fscrypt registered1683worker2 # [ 0.744934] Key type fscrypt-provisioning registered1684worker2 # [ 0.745919] ima: No TPM chip found, activating TPM-bypass!1685worker2 # [ 0.748179] ima: Allocated hash algorithm: sha11686worker2 # [ 0.749016] ima: No architecture policies found1687worker2 # [ 0.753178] PM: Magic number: 10:167:1721688worker2 # [ 0.754799] RAS: Correctable Errors collector initialized.1689client # [ 0.743854] Demotion targets for Node 0: null1690client # [ 0.744788] Key type .fscrypt registered1691client # [ 0.745462] Key type fscrypt-provisioning registered1692lb # [ 0.744204] Demotion targets for Node 0: null1693lb # [ 0.745146] Key type .fscrypt registered1694lb # [ 0.745836] Key type fscrypt-provisioning registered1695client # [ 0.748636] ima: No TPM chip found, activating TPM-bypass!1696lb # [ 0.746793] ima: No TPM chip found, activating TPM-bypass!1697client # [ 0.749551] ima: Allocated hash algorithm: sha11698client # [ 0.750313] ima: No architecture policies found1699worker2 # [ 0.762836] clk: Disabling unused clocks1700lb # [ 0.749202] ima: Allocated hash algorithm: sha11701lb # [ 0.750014] ima: No architecture policies found1702client # [ 0.753733] PM: Magic number: 10:167:1721703worker2 # [ 0.767182] PM: genpd: Disabling unused power domains1704lb # [ 0.754217] PM: Magic number: 10:167:1721705client # [ 0.757718] RAS: Correctable Errors collector initialized.1706lb # [ 0.755901] RAS: Correctable Errors collector initialized.1707client # [ 0.766649] clk: Disabling unused clocks1708lb # [ 0.764909] clk: Disabling unused clocks1709client # [ 0.767324] PM: genpd: Disabling unused power domains1710lb # [ 0.767212] PM: genpd: Disabling unused power domains1711worker1 # [ 0.924852] Freeing initrd memory: 29092K1712worker1 # [ 0.928032] Freeing unused decrypted memory: 2028K1713worker1 # [ 0.930658] Freeing unused kernel image (initmem) memory: 3644K1714worker1 # [ 0.931716] Write protecting the kernel read-only data: 32768k1715worker1 # [ 0.933584] Freeing unused kernel image (text/rodata gap) memory: 1208K1716worker1 # [ 0.935059] Freeing unused kernel image (rodata/data gap) memory: 752K1717worker2 # [ 0.933779] Freeing initrd memory: 29092K1718worker2 # [ 0.936969] Freeing unused decrypted memory: 2028K1719worker2 # [ 0.939625] Freeing unused kernel image (initmem) memory: 3644K1720worker2 # [ 0.940765] Write protecting the kernel read-only data: 32768k1721worker2 # [ 0.942659] Freeing unused kernel image (text/rodata gap) memory: 1208K1722worker2 # [ 0.944248] Freeing unused kernel image (rodata/data gap) memory: 752K1723lb # [ 0.933020] Freeing initrd memory: 29092K1724lb # [ 0.936198] Freeing unused decrypted memory: 2028K1725lb # [ 0.938866] Freeing unused kernel image (initmem) memory: 3644K1726lb # [ 0.939978] Write protecting the kernel read-only data: 32768k1727lb # [ 0.941911] Freeing unused kernel image (text/rodata gap) memory: 1208K1728lb # [ 0.943532] Freeing unused kernel image (rodata/data gap) memory: 752K1729worker1 # [ 0.986277] x86/mm: Checked W+X mappings: passed, no W+X pages found.1730worker1 # [ 0.987302] Run /init as init process1731client # [ 0.959682] Freeing initrd memory: 29096K1732client # [ 0.963182] Freeing unused decrypted memory: 2028K1733worker1 # [ 0.997768] systemd[1]: Inserted module 'autofs4'1734client # [ 0.966196] Freeing unused kernel image (initmem) memory: 3644K1735client # [ 0.967286] Write protecting the kernel read-only data: 32768k1736client # [ 0.969271] Freeing unused kernel image (text/rodata gap) memory: 1208K1737client # [ 0.970832] Freeing unused kernel image (rodata/data gap) memory: 752K1738worker1 # [ 1.014089] fuse: init (API version 7.45)1739worker2 # [ 0.995494] x86/mm: Checked W+X mappings: passed, no W+X pages found.1740worker2 # [ 0.996597] Run /init as init process1741worker1 # [ 1.019864] ACPI: \_SB_.GSIG: Enabled at IRQ 221742worker1 # [ 1.022158] ACPI: \_SB_.GSIH: Enabled at IRQ 231743worker1 # [ 1.024915] ACPI: \_SB_.GSIE: Enabled at IRQ 201744worker1 # [ 1.027084] ACPI: \_SB_.GSIF: Enabled at IRQ 211745worker2 # [ 1.007100] systemd[1]: Inserted module 'autofs4'1746lb # [ 0.994581] x86/mm: Checked W+X mappings: passed, no W+X pages found.1747lb # [ 0.995709] Run /init as init process1748lb # [ 1.006550] systemd[1]: Inserted module 'autofs4'1749worker2 # [ 1.023413] fuse: init (API version 7.45)1750worker2 # [ 1.029185] ACPI: \_SB_.GSIG: Enabled at IRQ 221751worker2 # [ 1.031786] ACPI: \_SB_.GSIH: Enabled at IRQ 231752worker1 # [ 1.052266] systemd[1]: Successfully made /usr/ read-only.1753client # [ 1.021854] x86/mm: Checked W+X mappings: passed, no W+X pages found.1754client # [ 1.023008] Run /init as init process1755worker2 # [ 1.035019] ACPI: \_SB_.GSIE: Enabled at IRQ 201756lb # [ 1.023006] fuse: init (API version 7.45)1757worker2 # [ 1.037539] ACPI: \_SB_.GSIF: Enabled at IRQ 211758lb # [ 1.028875] ACPI: \_SB_.GSIG: Enabled at IRQ 221759client # [ 1.033865] systemd[1]: Inserted module 'autofs4'1760lb # [ 1.031711] ACPI: \_SB_.GSIH: Enabled at IRQ 231761lb # [ 1.035040] ACPI: \_SB_.GSIE: Enabled at IRQ 201762lb # [ 1.037614] ACPI: \_SB_.GSIF: Enabled at IRQ 211763client # [ 1.050813] fuse: init (API version 7.45)1764worker2 # [ 1.066943] systemd[1]: Successfully made /usr/ read-only.1765client # [ 1.056883] ACPI: \_SB_.GSIG: Enabled at IRQ 221766client # [ 1.059675] ACPI: \_SB_.GSIH: Enabled at IRQ 231767client # [ 1.062896] ACPI: \_SB_.GSIE: Enabled at IRQ 201768client # [ 1.065383] ACPI: \_SB_.GSIF: Enabled at IRQ 211769lb # [ 1.065045] systemd[1]: Successfully made /usr/ read-only.1770client # [ 1.094530] systemd[1]: Successfully made /usr/ read-only.1771worker1 # [ 1.388880] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1772worker1 # [ 1.400742] systemd[1]: Detected virtualization kvm.1773worker1 # [ 1.402824] systemd[1]: Detected architecture x86-64.1774worker1 # [ 1.404896] systemd[1]: Running in initrd.1775worker1 # [ 1.407329] systemd[1]: Initializing machine ID from random generator.1776worker1 # [ 1.410185] systemd[1]: Hostname set to <worker1>.1777worker2 # [ 1.403482] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1778worker2 # [ 1.415387] systemd[1]: Detected virtualization kvm.1779worker2 # [ 1.417492] systemd[1]: Detected architecture x86-64.1780worker2 # [ 1.419661] systemd[1]: Running in initrd.1781lb # [ 1.400646] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1782lb # [ 1.409017] systemd[1]: Detected virtualization kvm.1783worker2 # [ 1.422260] systemd[1]: Initializing machine ID from random generator.1784lb # [ 1.411154] systemd[1]: Detected architecture x86-64.1785worker2 # [ 1.425042] systemd[1]: Hostname set to <worker2>.1786lb # [ 1.413413] systemd[1]: Running in initrd.1787lb # [ 1.415942] systemd[1]: Initializing machine ID from random generator.1788lb # [ 1.418791] systemd[1]: Hostname set to <lb>.1789client # [ 1.431083] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1790client # [ 1.443018] systemd[1]: Detected virtualization kvm.1791client # [ 1.445140] systemd[1]: Detected architecture x86-64.1792client # [ 1.447272] systemd[1]: Running in initrd.1793client # [ 1.449810] systemd[1]: Initializing machine ID from random generator.1794client # [ 1.452719] systemd[1]: Hostname set to <client>.1795worker1 # [ 1.633274] systemd[1]: bpf-restrict-fs: LSM BPF program attached1796lb # [ 1.628944] systemd[1]: bpf-restrict-fs: LSM BPF program attached1797worker2 # [ 1.650906] systemd[1]: bpf-restrict-fs: LSM BPF program attached1798worker1 # [ 1.686923] systemd[1]: Queued start job for default target Initrd Default Target.1799worker1 # [ 1.690852] systemd[1]: Created slice Slice /system/modprobe.1800worker1 # [ 1.692098] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1801client # [ 1.660729] systemd[1]: bpf-restrict-fs: LSM BPF program attached1802worker1 # [ 1.693495] systemd[1]: Expecting device /dev/disk/by-label/nixos...1803worker1 # [ 1.694633] systemd[1]: Reached target Path Units.1804worker1 # [ 1.695505] systemd[1]: Reached target Slice Units.1805worker1 # [ 1.696406] systemd[1]: Reached target Swaps.1806worker1 # [ 1.697238] systemd[1]: Reached target Timer Units.1807worker1 # [ 1.698228] systemd[1]: Listening on D-Bus System Message Bus Socket.1808worker1 # [ 1.699451] systemd[1]: Listening on Journal Socket (/dev/log).1809worker1 # [ 1.700653] systemd[1]: Listening on Journal Sockets.1810worker1 # [ 1.701665] systemd[1]: Listening on udev Control Socket.1811worker1 # [ 1.702703] systemd[1]: Listening on udev Kernel Socket.1812worker1 # [ 1.703687] systemd[1]: Reached target Socket Units.1813lb # [ 1.670215] systemd[1]: Queued start job for default target Initrd Default Target.1814worker1 # [ 1.705477] systemd[1]: Starting Create List of Static Device Nodes...1815lb # [ 1.673633] systemd[1]: Created slice Slice /system/modprobe.1816lb # [ 1.674838] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1817worker1 # [ 1.709333] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1818lb # [ 1.676313] systemd[1]: Expecting device /dev/disk/by-label/nixos...1819lb # [ 1.677471] systemd[1]: Reached target Path Units.1820lb # [ 1.678379] systemd[1]: Reached target Slice Units.1821lb # [ 1.679287] systemd[1]: Reached target Swaps.1822lb # [ 1.680119] systemd[1]: Reached target Timer Units.1823lb # [ 1.681163] systemd[1]: Listening on D-Bus System Message Bus Socket.1824lb # [ 1.682488] systemd[1]: Listening on Journal Socket (/dev/log).1825worker1 # [ 1.716717] systemd[1]: Starting Load Kernel Module configfs...1826lb # [ 1.683666] systemd[1]: Listening on Journal Sockets.1827lb # [ 1.684713] systemd[1]: Listening on udev Control Socket.1828lb # [ 1.685773] systemd[1]: Listening on udev Kernel Socket.1829lb # [ 1.686748] systemd[1]: Reached target Socket Units.1830lb # [ 1.688632] systemd[1]: Starting Create List of Static Device Nodes...1831worker2 # [ 1.705411] systemd[1]: Queued start job for default target Initrd Default Target.1832lb # [ 1.693308] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1833worker1 # [ 1.729747] systemd[1]: Starting Journal Service...1834worker2 # [ 1.709475] systemd[1]: Created slice Slice /system/modprobe.1835worker2 # [ 1.710716] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1836client # [ 1.699443] systemd[1]: Queued start job for default target Initrd Default Target.1837worker2 # [ 1.712166] systemd[1]: Expecting device /dev/disk/by-label/nixos...1838worker2 # [ 1.713285] systemd[1]: Reached target Path Units.1839lb # [ 1.699881] systemd[1]: Starting Load Kernel Module configfs...1840worker2 # [ 1.714183] systemd[1]: Reached target Slice Units.1841worker2 # [ 1.715068] systemd[1]: Reached target Swaps.1842client # [ 1.703827] systemd[1]: Created slice Slice /system/modprobe.1843worker2 # [ 1.715906] systemd[1]: Reached target Timer Units.1844client # [ 1.705107] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1845worker2 # [ 1.716910] systemd[1]: Listening on D-Bus System Message Bus Socket.1846client # [ 1.706551] systemd[1]: Expecting device /dev/disk/by-label/nixos...1847worker2 # [ 1.718128] systemd[1]: Listening on Journal Socket (/dev/log).1848client # [ 1.707696] systemd[1]: Reached target Path Units.1849worker2 # [ 1.719312] systemd[1]: Listening on Journal Sockets.1850client # [ 1.708600] systemd[1]: Reached target Slice Units.1851worker2 # [ 1.720325] systemd[1]: Listening on udev Control Socket.1852worker1 # [ 1.740652] systemd[1]: Starting Load Kernel Modules...1853client # [ 1.709508] systemd[1]: Reached target Swaps.1854worker2 # [ 1.721348] systemd[1]: Listening on udev Kernel Socket.1855client # [ 1.710280] systemd[1]: Reached target Timer Units.1856worker2 # [ 1.722320] systemd[1]: Reached target Socket Units.1857client # [ 1.711261] systemd[1]: Listening on D-Bus System Message Bus Socket.1858client # [ 1.712509] systemd[1]: Listening on Journal Socket (/dev/log).1859worker2 # [ 1.724193] systemd[1]: Starting Create List of Static Device Nodes...1860client # [ 1.713697] systemd[1]: Listening on Journal Sockets.1861client # [ 1.714793] systemd[1]: Listening on udev Control Socket.1862client # [ 1.715866] systemd[1]: Listening on udev Kernel Socket.1863lb # [ 1.714286] systemd[1]: Starting Journal Service...1864worker2 # [ 1.728029] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1865client # [ 1.716860] systemd[1]: Reached target Socket Units.1866client # [ 1.719580] systemd[1]: Starting Create List of Static Device Nodes...1867worker2 # [ 1.734752] systemd[1]: Starting Load Kernel Module configfs...1868client # [ 1.723572] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1869worker1 # [ 1.748608] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1870client # [ 1.729579] systemd[1]: Starting Load Kernel Module configfs...1871lb # [ 1.729262] systemd[1]: Starting Load Kernel Modules...1872worker1 # [ 1.765932] netfs: FS-Cache loaded1873worker1 # [ 1.770660] 9pnet: Installing 9P2000 support1874worker2 # [ 1.750832] systemd[1]: Starting Journal Service...1875client # [ 1.741077] systemd[1]: Starting Journal Service...1876worker1 # [ 1.778801] systemd[1]: Starting Coldplug All udev Devices...1877lb # [ 1.734286] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1878worker2 # [ 1.763241] systemd[1]: Starting Load Kernel Modules...1879client # [ 1.753568] systemd[1]: Starting Load Kernel Modules...1880lb # [ 1.752045] netfs: FS-Cache loaded1881lb # [ 1.757016] 9pnet: Installing 9P2000 support1882worker1 # [ 1.792656] systemd[1]: Finished Create List of Static Device Nodes.1883worker1 # [ 1.798107] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1884lb # [ 1.767407] systemd[1]: Starting Coldplug All udev Devices...1885worker2 # [ 1.770234] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1886client # [ 1.760581] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1887worker1 # [ 1.805931] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1888worker2 # [ 1.786291] netfs: FS-Cache loaded1889worker2 # [ 1.791060] 9pnet: Installing 9P2000 support1890client # [ 1.779995] netfs: FS-Cache loaded1891worker1 # [ 1.811948] systemd[1]: modprobe@configfs.service: Deactivated successfully.1892worker1 # [ 1.815141] systemd-journald[67]: Collecting audit messages is disabled.1893client # [ 1.784628] 9pnet: Installing 9P2000 support1894lb # [ 1.782529] systemd[1]: Finished Create List of Static Device Nodes.1895worker1 # [ 1.817902] systemd[1]: Finished Load Kernel Module configfs.1896worker2 # [ 1.800362] systemd[1]: Starting Coldplug All udev Devices...1897lb # [ 1.786880] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1898client # [ 1.792589] systemd[1]: Starting Coldplug All udev Devices...1899worker1 # [ 1.823766] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1900worker1 # [ 1.827556] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1901lb # [ 1.795706] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1902lb # [ 1.801332] systemd-journald[67]: Collecting audit messages is disabled.1903worker2 # [ 1.816242] systemd[1]: Finished Create List of Static Device Nodes.1904worker1 # [ 1.836630] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1905lb # [ 1.802945] systemd[1]: modprobe@configfs.service: Deactivated successfully.1906worker1 # [ 1.838036] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1907client # [ 1.807629] systemd[1]: Finished Create List of Static Device Nodes.1908worker2 # [ 1.821769] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1909lb # [ 1.810564] systemd[1]: Finished Load Kernel Module configfs.1910client # [ 1.814102] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1911lb # [ 1.814680] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1912worker2 # [ 1.829594] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1913client # [ 1.821912] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1914lb # [ 1.819206] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1915worker2 # [ 1.834644] systemd[1]: modprobe@configfs.service: Deactivated successfully.1916worker2 # [ 1.836065] systemd-journald[67]: Collecting audit messages is disabled.1917client # [ 1.826128] systemd-journald[67]: Collecting audit messages is disabled.1918lb # [ 1.826226] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1919client # [ 1.828928] systemd[1]: modprobe@configfs.service: Deactivated successfully.1920worker2 # [ 1.841231] systemd[1]: Finished Load Kernel Module configfs.1921lb # [ 1.828056] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1922worker1 # [ 1.867065] systemd[1]: Finished Load Kernel Modules.1923client # [ 1.835854] systemd[1]: Finished Load Kernel Module configfs.1924worker2 # [ 1.847631] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1925client # [ 1.839951] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1926worker2 # [ 1.852339] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1927worker1 # [ 1.874740] systemd[1]: Starting Apply Kernel Variables...1928client # [ 1.842189] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1929worker2 # [ 1.860253] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1930worker2 # [ 1.861612] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1931client # [ 1.851525] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1932client # [ 1.853288] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1933worker1 # [ 1.887616] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1934lb # [ 1.857767] systemd[1]: Finished Load Kernel Modules.1935worker1 # [ 1.896120] systemd[1]: Starting Create Static Device Nodes in /dev...1936lb # [ 1.866572] systemd[1]: Starting Apply Kernel Variables...1937worker1 # [ 1.909672] systemd[1]: Finished Apply Kernel Variables.1938worker2 # [ 1.892713] systemd[1]: Finished Load Kernel Modules.1939lb # [ 1.879350] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1940client # [ 1.882056] systemd[1]: Finished Load Kernel Modules.1941worker2 # [ 1.900371] systemd[1]: Starting Apply Kernel Variables...1942client # [ 1.890580] systemd[1]: Starting Apply Kernel Variables...1943lb # [ 1.889390] systemd[1]: Starting Create Static Device Nodes in /dev...1944worker1 # [ 1.927406] systemd[1]: Started Journal Service.1945worker1 # [ 1.797199] systemd-modules-load[68]: Inserted module 'dm_mod'1946client # [ 1.899067] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1947worker1 # [ 1.799403] systemd-modules-load[68]: Inserted module 'virtio_balloon'1948worker1 # [ 1.801321] systemd-modules-load[68]: Inserted module 'virtio_gpu'1949worker2 # [ 1.913234] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1950worker1 # [ 1.802658] systemd[1]: Finished Create Static Device Nodes in /dev.1951lb # [ 1.904317] systemd[1]: Finished Apply Kernel Variables.1952worker1 # [ 1.806200] systemd[1]: Reached target Preparation for Local File Systems.1953worker1 # [ 1.807313] systemd[1]: Reached target Local File Systems.1954worker1 # [ 1.808209] systemd[1]: Starting Create System Files and Directories...1955client # [ 1.910700] systemd[1]: Starting Create Static Device Nodes in /dev...1956worker2 # [ 1.922380] systemd[1]: Starting Create Static Device Nodes in /dev...1957worker1 # [ 1.815065] systemd[1]: Starting Rule-based Manager for Device Events and Files...1958lb # [ 1.919345] systemd[1]: Started Journal Service.1959lb # [ 1.788290] systemd-modules-load[68]: Inserted module 'dm_mod'1960lb # [ 1.789676] systemd-modules-load[68]: Inserted module 'virtio_balloon'1961client # [ 1.925245] systemd[1]: Finished Apply Kernel Variables.1962worker2 # [ 1.937286] systemd[1]: Finished Apply Kernel Variables.1963lb # [ 1.792280] systemd-modules-load[68]: Inserted module 'virtio_gpu'1964lb # [ 1.798814] systemd[1]: Finished Create Static Device Nodes in /dev.1965lb # [ 1.804117] systemd[1]: Reached target Preparation for Local File Systems.1966lb # [ 1.805557] systemd[1]: Reached target Local File Systems.1967client # [ 1.807666] systemd-modules-load[68]: Inserted module 'dm_mod'1968client # [ 1.942325] systemd[1]: Started Journal Service.1969worker2 # [ 1.954313] systemd[1]: Started Journal Service.1970worker2 # [ 1.821029] systemd-modules-load[68]: Inserted module 'dm_mod'1971lb # [ 1.809138] systemd[1]: Starting Create System Files and Directories...1972client # [ 1.812230] systemd-modules-load[68]: Inserted module 'virtio_balloon'1973worker2 # [ 1.824230] systemd-modules-load[68]: Inserted module 'virtio_balloon'1974worker1 # [ 1.845384] systemd[1]: Finished Create System Files and Directories.1975client # [ 1.813300] systemd-modules-load[68]: Inserted module 'virtio_gpu'1976worker2 # [ 1.825826] systemd-modules-load[68]: Inserted module 'virtio_gpu'1977lb # [ 1.813390] systemd[1]: Starting Rule-based Manager for Device Events and Files...1978worker2 # [ 1.830081] systemd[1]: Finished Create Static Device Nodes in /dev.1979client # [ 1.818425] systemd[1]: Finished Create Static Device Nodes in /dev.1980client # [ 1.825078] systemd[1]: Reached target Preparation for Local File Systems.1981worker2 # [ 1.836878] systemd[1]: Reached target Preparation for Local File Systems.1982client # [ 1.826043] systemd[1]: Reached target Local File Systems.1983worker2 # [ 1.838204] systemd[1]: Reached target Local File Systems.1984worker2 # [ 1.840777] systemd[1]: Starting Create System Files and Directories...1985client # [ 1.829036] systemd[1]: Starting Create System Files and Directories...1986worker2 # [ 1.845945] systemd[1]: Starting Rule-based Manager for Device Events and Files...1987worker1 # [ 1.867392] systemd-udevd[76]: Using default interface naming scheme 'v261'.1988client # [ 1.836114] systemd[1]: Starting Rule-based Manager for Device Events and Files...1989lb # [ 1.841609] systemd[1]: Finished Create System Files and Directories.1990worker1 # [ 1.891134] systemd[1]: Started Rule-based Manager for Device Events and Files.1991client # [ 1.859226] systemd[1]: Finished Create System Files and Directories.1992worker2 # [ 1.875467] systemd[1]: Finished Create System Files and Directories.1993lb # [ 1.865703] systemd-udevd[83]: Using default interface naming scheme 'v261'.1994client # [ 1.881586] systemd-udevd[83]: Using default interface naming scheme 'v261'.1995worker2 # [ 1.896246] systemd-udevd[83]: Using default interface naming scheme 'v261'.1996lb # [ 1.890121] systemd[1]: Started Rule-based Manager for Device Events and Files.1997worker1 # [ 1.938113] systemd[1]: Finished Coldplug All udev Devices.1998client # [ 1.906061] systemd[1]: Started Rule-based Manager for Device Events and Files.1999worker1 # [ 1.940391] systemd[1]: Reached target System Initialization.2000worker1 # [ 1.941246] systemd[1]: Reached target Basic System.
loading 2,137 hidden lines…
4138worker2 # [ 18.583567] nix-grpc-daemon[786]: E0000 00:00:1789981927.794016 786 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4139worker2 # [ 18.587710] nix-grpc-daemon[786]: ts=2026-09-21T09:12:07Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4140worker2 # [ 18.590976] systemd[1]: Started Nix worker-protocol over gRPC.4141worker2 # [ 18.639105] nix-grpc-daemon[786]: E0000 00:00:1789981927.848700 825 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4142worker2 # [ 18.654332] nix-grpc-daemon[786]: E0000 00:00:1789981927.864760 825 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4143worker2 # [ 18.658057] nix-grpc-daemon[786]: E0000 00:00:1789981927.868406 825 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4144worker2 # [ 18.671039] nix-grpc-daemon[786]: E0000 00:00:1789981927.881450 825 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4145worker2 # [ 18.694381] nix-grpc-daemon[786]: ts=2026-09-21T09:12:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4146worker1 # [ 18.744255] nix-grpc-daemon[785]: ts=2026-09-21T09:12:07Z level=info event=scheduler_connected addr=lb:500514147worker1 # [ 18.748264] nix-grpc-daemon[785]: ts=2026-09-21T09:12:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4148worker2 # [ 18.896400] nix-grpc-daemon[786]: ts=2026-09-21T09:12:08Z level=info event=scheduler_connected addr=lb:500514149worker2 # [ 18.898227] nix-grpc-daemon[786]: ts=2026-09-21T09:12:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4150worker1 # [ 18.948458] nix-grpc-daemon[785]: ts=2026-09-21T09:12:08Z level=info event=scheduler_connected addr=lb:500514151worker1 # [ 18.950267] nix-grpc-daemon[785]: ts=2026-09-21T09:12:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4152worker2 # [ 19.099425] nix-grpc-daemon[786]: ts=2026-09-21T09:12:08Z level=info event=scheduler_connected addr=lb:500514153worker2 # [ 19.103280] nix-grpc-daemon[786]: ts=2026-09-21T09:12:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4154worker1 # [ 19.150872] nix-grpc-daemon[785]: ts=2026-09-21T09:12:08Z level=info event=scheduler_connected addr=lb:500514155worker1 # [ 19.153173] nix-grpc-daemon[785]: ts=2026-09-21T09:12:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4156worker2 # [ 19.304346] nix-grpc-daemon[786]: ts=2026-09-21T09:12:08Z level=info event=scheduler_connected addr=lb:500514157worker2 # [ 19.308121] nix-grpc-daemon[786]: ts=2026-09-21T09:12:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4158worker1 # [ 19.353964] nix-grpc-daemon[785]: ts=2026-09-21T09:12:08Z level=info event=scheduler_connected addr=lb:500514159worker1 # [ 19.357897] nix-grpc-daemon[785]: ts=2026-09-21T09:12:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4160worker2 # [ 19.509830] nix-grpc-daemon[786]: ts=2026-09-21T09:12:08Z level=info event=scheduler_connected addr=lb:500514161worker2 # [ 19.513457] nix-grpc-daemon[786]: ts=2026-09-21T09:12:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4162worker1 # [ 19.560065] nix-grpc-daemon[785]: ts=2026-09-21T09:12:08Z level=info event=scheduler_connected addr=lb:500514163worker1 # [ 19.563202] nix-grpc-daemon[785]: ts=2026-09-21T09:12:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4164worker2 # [ 19.711533] nix-grpc-daemon[786]: ts=2026-09-21T09:12:08Z level=info event=scheduler_connected addr=lb:500514165worker2 # [ 19.715383] nix-grpc-daemon[786]: ts=2026-09-21T09:12:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4166worker1 # [ 19.760972] nix-grpc-daemon[785]: ts=2026-09-21T09:12:09Z level=info event=scheduler_connected addr=lb:500514167worker1 # [ 19.764853] nix-grpc-daemon[785]: ts=2026-09-21T09:12:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4168worker2 # [ 19.916534] nix-grpc-daemon[786]: ts=2026-09-21T09:12:09Z level=info event=scheduler_connected addr=lb:500514169worker2 # [ 19.920570] nix-grpc-daemon[786]: ts=2026-09-21T09:12:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4170worker1 # [ 19.965790] nix-grpc-daemon[785]: ts=2026-09-21T09:12:09Z level=info event=scheduler_connected addr=lb:500514171worker1 # [ 19.969618] nix-grpc-daemon[785]: ts=2026-09-21T09:12:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4172worker2 # [ 20.121903] nix-grpc-daemon[786]: ts=2026-09-21T09:12:09Z level=info event=scheduler_connected addr=lb:500514173worker2 # [ 20.126379] nix-grpc-daemon[786]: ts=2026-09-21T09:12:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4174worker1 # [ 20.170522] nix-grpc-daemon[785]: ts=2026-09-21T09:12:09Z level=info event=scheduler_connected addr=lb:500514175worker1 # [ 20.174306] nix-grpc-daemon[785]: ts=2026-09-21T09:12:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4176worker2 # [ 20.326906] nix-grpc-daemon[786]: ts=2026-09-21T09:12:09Z level=info event=scheduler_connected addr=lb:500514177worker2 # [ 20.330779] nix-grpc-daemon[786]: ts=2026-09-21T09:12:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4178worker1 # [ 20.375260] nix-grpc-daemon[785]: ts=2026-09-21T09:12:09Z level=info event=scheduler_connected addr=lb:500514179worker1 # [ 20.379075] nix-grpc-daemon[785]: ts=2026-09-21T09:12:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4180worker2 # [ 20.533512] nix-grpc-daemon[786]: ts=2026-09-21T09:12:09Z level=info event=scheduler_connected addr=lb:500514181worker2 # [ 20.536653] nix-grpc-daemon[786]: ts=2026-09-21T09:12:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4182worker1 # [ 20.580498] nix-grpc-daemon[785]: ts=2026-09-21T09:12:09Z level=info event=scheduler_connected addr=lb:500514183worker1 # [ 20.583618] nix-grpc-daemon[785]: ts=2026-09-21T09:12:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4184worker2 # [ 20.734697] nix-grpc-daemon[786]: ts=2026-09-21T09:12:09Z level=info event=scheduler_connected addr=lb:500514185worker2 # [ 20.738536] nix-grpc-daemon[786]: ts=2026-09-21T09:12:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4186worker1 # [ 20.784422] nix-grpc-daemon[785]: ts=2026-09-21T09:12:10Z level=info event=scheduler_connected addr=lb:500514187worker1 # [ 20.788275] nix-grpc-daemon[785]: ts=2026-09-21T09:12:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4188worker2 # [ 20.940076] nix-grpc-daemon[786]: ts=2026-09-21T09:12:10Z level=info event=scheduler_connected addr=lb:500514189worker2 # [ 20.944276] nix-grpc-daemon[786]: ts=2026-09-21T09:12:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4190worker1 # [ 20.989130] nix-grpc-daemon[785]: ts=2026-09-21T09:12:10Z level=info event=scheduler_connected addr=lb:500514191worker1 # [ 20.992868] nix-grpc-daemon[785]: ts=2026-09-21T09:12:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4192worker2 # [ 21.145553] nix-grpc-daemon[786]: ts=2026-09-21T09:12:10Z level=info event=scheduler_connected addr=lb:500514193worker2 # [ 21.149394] nix-grpc-daemon[786]: ts=2026-09-21T09:12:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4194worker1 # [ 21.193701] nix-grpc-daemon[785]: ts=2026-09-21T09:12:10Z level=info event=scheduler_connected addr=lb:500514195worker1 # [ 21.197797] nix-grpc-daemon[785]: ts=2026-09-21T09:12:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4196worker2 # [ 21.350669] nix-grpc-daemon[786]: ts=2026-09-21T09:12:10Z level=info event=scheduler_connected addr=lb:500514197worker2 # [ 21.354502] nix-grpc-daemon[786]: ts=2026-09-21T09:12:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4198worker1 # [ 21.398528] nix-grpc-daemon[785]: ts=2026-09-21T09:12:10Z level=info event=scheduler_connected addr=lb:500514199worker1 # [ 21.402492] nix-grpc-daemon[785]: ts=2026-09-21T09:12:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4200worker2 # [ 21.556459] nix-grpc-daemon[786]: ts=2026-09-21T09:12:10Z level=info event=scheduler_connected addr=lb:500514201worker2 # [ 21.560176] nix-grpc-daemon[786]: ts=2026-09-21T09:12:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4202worker1 # [ 21.604485] nix-grpc-daemon[785]: ts=2026-09-21T09:12:10Z level=info event=scheduler_connected addr=lb:500514203worker1 # [ 21.608375] nix-grpc-daemon[785]: ts=2026-09-21T09:12:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4204worker2 # [ 21.761258] nix-grpc-daemon[786]: ts=2026-09-21T09:12:10Z level=info event=scheduler_connected addr=lb:500514205worker2 # [ 21.765165] nix-grpc-daemon[786]: ts=2026-09-21T09:12:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4206worker1 # [ 21.809208] nix-grpc-daemon[785]: ts=2026-09-21T09:12:11Z level=info event=scheduler_connected addr=lb:500514207worker1 # [ 21.813108] nix-grpc-daemon[785]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4208worker2 # [ 21.966294] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_connected addr=lb:500514209worker2 # [ 21.970186] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4210worker1 # [ 22.014061] nix-grpc-daemon[785]: ts=2026-09-21T09:12:11Z level=info event=scheduler_connected addr=lb:500514211worker1 # [ 22.017855] nix-grpc-daemon[785]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4212worker2 # [ 22.171552] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_connected addr=lb:500514213worker2 # [ 22.175808] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4214worker1 # [ 22.218707] nix-grpc-daemon[785]: ts=2026-09-21T09:12:11Z level=info event=scheduler_connected addr=lb:500514215worker1 # [ 22.222619] nix-grpc-daemon[785]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4216worker2 # [ 22.377063] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_connected addr=lb:500514217worker2 # [ 22.381273] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4218worker1 # [ 22.423656] nix-grpc-daemon[785]: ts=2026-09-21T09:12:11Z level=info event=scheduler_connected addr=lb:500514219worker1 # [ 22.427488] nix-grpc-daemon[785]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4220worker2 # [ 22.583699] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_connected addr=lb:500514221worker2 # [ 22.586756] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4222worker1 # [ 22.629310] nix-grpc-daemon[785]: ts=2026-09-21T09:12:11Z level=info event=scheduler_connected addr=lb:500514223worker1 # [ 22.632905] nix-grpc-daemon[785]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4224worker2 # [ 22.784882] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_connected addr=lb:500514225worker2 # [ 22.789367] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4226worker1 # [ 22.833878] nix-grpc-daemon[785]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514227worker1 # [ 22.837919] nix-grpc-daemon[785]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4228worker2 # [ 22.990188] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514229worker2 # [ 22.994114] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4230worker1 # [ 23.038772] nix-grpc-daemon[785]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514231worker1 # [ 23.042671] nix-grpc-daemon[785]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4232worker2 # [ 23.195319] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514233worker2 # [ 23.199196] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4234worker1 # [ 23.243719] nix-grpc-daemon[785]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514235worker1 # [ 23.247941] nix-grpc-daemon[785]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4236worker2 # [ 23.400439] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514237worker2 # [ 23.404320] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4238worker1 # [ 23.448661] nix-grpc-daemon[785]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514239worker1 # [ 23.452474] nix-grpc-daemon[785]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4240worker2 # [ 23.606360] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514241worker2 # [ 23.609429] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4242worker1 # [ 23.654779] nix-grpc-daemon[785]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514243worker1 # [ 23.657824] nix-grpc-daemon[785]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4244worker2 # [ 23.807338] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514245worker2 # [ 23.811223] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4246worker1 # [ 23.855323] nix-grpc-daemon[785]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514247worker1 # [ 23.858706] nix-grpc-daemon[785]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4248worker2 # [ 24.012375] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514249worker2 # [ 24.016245] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4250worker1 # [ 24.059546] nix-grpc-daemon[785]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514251worker1 # [ 24.063354] nix-grpc-daemon[785]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4252worker2 # [ 24.217712] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514253worker2 # [ 24.221639] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4254lb # [ 24.213556] rustfs-bucket-start[992]: mb s3://farm4255lb # [ 24.220482] systemd[1]: Finished rustfs-bucket.service.4256lb # [ 24.227186] systemd[1]: Starting niks3 server...4257worker1 # [ 24.264221] nix-grpc-daemon[785]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514258worker1 # [ 24.267973] nix-grpc-daemon[785]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4259worker2 # [ 24.422706] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514260worker2 # [ 24.426468] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4261worker1 # [ 24.469061] nix-grpc-daemon[785]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514262worker1 # [ 24.472691] nix-grpc-daemon[785]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4263lb # [ 24.528747] postgres[1009]: [1009] ERROR: relation "goose_db_version" does not exist at character 364264lb # [ 24.530020] postgres[1009]: [1009] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4265lb # [ 24.575985] niks3-server[1003]: 2026/09/21 09:12:14 OK 20241026095416_initial_model.sql (24.13ms)4266lb # [ 24.580455] niks3-server[1003]: 2026/09/21 09:12:14 OK 20251210153512_drop_unused_gin_index.sql (2.73ms)4267lb # [ 24.585764] niks3-server[1003]: 2026/09/21 09:12:14 OK 20251218171726_add_pins.sql (5.04ms)4268lb # [ 24.591439] niks3-server[1003]: 2026/09/21 09:12:14 OK 20260628120000_add_object_size_and_stats.sql (5.86ms)4269lb # [ 24.595837] niks3-server[1003]: 2026/09/21 09:12:14 OK 20260905000000_add_claims.sql (4.38ms)4270lb # [ 24.601975] niks3-server[1003]: 2026/09/21 09:12:14 OK 20260920000000_drop_claims.sql (5.84ms)4271lb # [ 24.603257] niks3-server[1003]: 2026/09/21 09:12:14 goose: successfully migrated database to version: 202609200000004272lb # [ 24.609766] niks3-server[1003]: 2026/09/21 09:12:14 OK 1_commit_pending_closure.sql (7.94ms)4273lb # [ 24.613070] niks3-server[1003]: 2026/09/21 09:12:14 OK 2_object_stats_trigger.sql (3.27ms)4274lb # [ 24.614950] niks3-server[1003]: 2026/09/21 09:12:14 goose: up to current file version: 24275worker2 # [ 24.628250] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514276worker2 # [ 24.632066] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4277lb # [ 24.624675] niks3-server[1003]: 2026/09/21 09:12:14 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key4278worker1 # [ 24.671937] nix-grpc-daemon[785]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514279worker1 # [ 24.675136] nix-grpc-daemon[785]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4280lb # [ 24.720787] niks3-server[1003]: 2026/09/21 09:12:14 INFO Read proxy enabled — serving cache objects from S34281lb # [ 24.722875] niks3-server[1003]: 2026/09/21 09:12:14 INFO Using socket-activated listener address=0.0.0.0:57514282lb # [ 24.728272] systemd[1]: Started niks3 server.4283lb # [ 24.729214] niks3-server[1003]: 2026/09/21 09:12:14 INFO systemd watchdog enabled interval=15s4284lb # [ 24.730930] systemd[1]: Reached target Multi-User System.4285lb # [ 24.732801] niks3-server[1003]: 2026/09/21 09:12:14 INFO Starting HTTP server address=0.0.0.0:57514286lb # [ 24.734406] systemd[1]: Startup finished in 866ms (kernel) + 4.969s (initrd) + 18.898s (userspace) = 24.734s.4287lb # [ 24.742222] niks3-server[1003]: 2026/09/21 09:12:14 INFO lead: acquired remote=192.168.1.4:523604288worker2 # [ 24.763446] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_take_over4289worker2 # [ 24.833087] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514290worker2 # [ 24.836811] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4291worker1 # [ 24.872653] nix-grpc-daemon[785]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514292worker1 # [ 24.876384] nix-grpc-daemon[785]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4293worker2 # [ 25.038167] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514294worker2 # [ 25.041951] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4295worker1 # [ 25.077247] nix-grpc-daemon[785]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514296worker1 # [ 25.081115] nix-grpc-daemon[785]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4297worker2 # [ 25.242982] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514298worker2 # [ 25.246859] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4299worker1 # [ 25.281874] nix-grpc-daemon[785]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514300worker1 # [ 25.285751] nix-grpc-daemon[785]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4301worker2 # [ 25.448176] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514302worker2 # [ 25.451949] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4303worker1 # [ 25.486440] nix-grpc-daemon[785]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514304worker1 # [ 25.490254] nix-grpc-daemon[785]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4305worker2 # [ 25.653224] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514306worker1 # [ 25.691939] nix-grpc-daemon[785]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514307worker2 # [ 25.671920] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:449924308worker2 # [ 25.677882] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04309worker2 # [ 25.695509] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=oidc_jwks provider=mock keys=14310worker2 # [ 25.699907] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:449924311worker2 # [ 25.704163] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04312lb: (finished: waiting for unit niks3.service, in 14.43 seconds)4313lb: waiting for TCP port 5751 on localhost4314lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!4315lb: (finished: waiting for TCP port 5751 on localhost, in 0.10 seconds)4316lb: waiting for unit envoy.service4317lb: (finished: waiting for unit envoy.service, in 0.08 seconds)4318lb: waiting for TCP port 50051 on localhost4319lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!4320lb # [ 25.969201] envoy[918]: conn peer=[::1]:49486 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=34321lb: (finished: waiting for TCP port 50051 on localhost, in 0.04 seconds)4322worker2 # [ 26.096664] systemd[1]: Starting scheduler-token.service...4323worker2 # [ 26.163582] systemd[1]: scheduler-token.service: Deactivated successfully.4324worker2 # [ 26.165834] systemd[1]: Finished scheduler-token.service.4325worker2 # [ 26.168141] systemd[1]: scheduler-token.service: Consumed 19ms CPU time over 68ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4326lb: waiting until exactly no endpoints of x86_64-linux are unhealthy4327lb: must succeed: curl -sf localhost:9901/clusters4328lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds)4329lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.11 seconds)4330subtest: one node schedules and both builders hold a WorkerSession on it4331lb: must succeed: curl -sf localhost:9901/clusters4332lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4333worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4334worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4335lb: must succeed: curl -sf localhost:9901/clusters4336lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4337lb: must succeed: curl -sf localhost:9901/clusters4338lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4339lb: must succeed: curl -sf localhost:9901/clusters4340lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4341lb: must succeed: curl -sf localhost:9901/clusters4342lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4343worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4344worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.05 seconds)4345worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4346worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4347lb: must succeed: curl -sf localhost:9901/clusters4348lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4349lb: must succeed: curl -sf localhost:9901/clusters4350lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4351worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4352worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)4353worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4354worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4355worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4356worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4357worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4358worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4359(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.77 seconds)4360subtest: balancer auth: client cert, bearer token, nothing, unknown CN4361worker1 # [ 27.456715] nix-grpc-daemon[785]: ts=2026-09-21T09:12:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:393384362worker1 # [ 27.468844] systemd[1]: Started Nix Daemon.4363worker1 # [ 27.696480] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4364worker1 # [ 27.749450] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 28 ms (attempt 1/5)4365worker1 # [ 27.779898] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 147 ms (attempt 2/5)4366worker1 # [ 27.929671] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 10 ms (attempt 3/5)4367worker1 # [ 27.945338] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 719 ms (attempt 4/5)4368worker1 # [ 28.676880] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4369worker1 # [ 28.712574] nix-grpc-daemon[785]: ts=2026-09-21T09:12:17Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:393384370worker1 # [ 28.734147] nix-grpc-daemon[785]: ts=2026-09-21T09:12:17Z level=debug event=path_info_miss path=bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test error="error: path '/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test' is required, but there is no substituter that can build it"4371worker2 # [ 28.730679] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:450084372worker2 # [ 28.738030] systemd[1]: Started Nix Daemon.4373worker2 # [ 28.970365] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4374worker2 # [ 29.003971] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 43 ms (attempt 1/5)4375worker2 # [ 29.049435] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 113 ms (attempt 2/5)4376worker2 # [ 29.166184] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 335 ms (attempt 3/5)4377worker2 # [ 29.500791] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 538 ms (attempt 4/5)4378worker2 # [ 30.051934] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4379worker2 # [ 30.073801] nix-grpc-daemon[786]: ts=2026-09-21T09:12:19Z level=debug event=path_info_miss path=bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test error="error: path '/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test' is required, but there is no substituter that can build it"4380client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4381client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.09 seconds)4382client: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4383client: (finished: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4384worker1 # [ 30.512706] nix-grpc-daemon[785]: ts=2026-09-21T09:12:19Z level=info event=oidc_jwks provider=mock keys=14385worker1 # [ 30.517761] nix-grpc-daemon[785]: ts=2026-09-21T09:12:19Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:393384386worker1 # [ 30.529376] nix-grpc-daemon[785]: ts=2026-09-21T09:12:19Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:393384387worker1 # [ 30.538736] nix-grpc-daemon[785]: ts=2026-09-21T09:12:19Z level=debug event=path_info_miss path=bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test error="error: path '/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test' is required, but there is no substituter that can build it"4388worker1 # [ 30.547543] nix-grpc-daemon[785]: ts=2026-09-21T09:12:19Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:393384389worker1 # [ 30.556359] nix-grpc-daemon[785]: ts=2026-09-21T09:12:19Z level=debug event=path_info_miss path=bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test error="error: path '/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test' is required, but there is no substituter that can build it"4390client: must succeed: rm -r /var/lib/nix-grpc-store4391client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.04 seconds)4392worker2 # [ 31.853908] nix-grpc-daemon[786]: ts=2026-09-21T09:12:21Z level=info event=denied method=QueryMissing cn=- role=none4393worker2 # [ 32.108766] nix-grpc-daemon[786]: ts=2026-09-21T09:12:21Z level=info event=denied method=QueryMissing cn=stranger role=none4394(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 5.08 seconds)4395subtest: DAG build is scheduled across workers and lands in the cache4396client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24397worker2 # [ 32.402323] nix-grpc-daemon[786]: ts=2026-09-21T09:12:21Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:450084398client # don't know how to build these paths:4399client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4400worker1 # [ 32.447449] nix-grpc-daemon[785]: ts=2026-09-21T09:12:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393384401worker2 # [ 32.464120] nix-grpc-daemon[786]: ts=2026-09-21T09:12:21Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04402worker2 # [ 32.466159] nix-grpc-daemon[786]: ts=2026-09-21T09:12:21Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14403worker1 # [ 32.545590] nix-grpc-daemon[785]: ts=2026-09-21T09:12:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384404worker1 # [ 32.572243] nix-grpc-daemon[785]: ts=2026-09-21T09:12:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393384405client # copying 1 paths to 'lb:50051'...4406worker1 # [ 32.586099] nix-grpc-daemon[785]: ts=2026-09-21T09:12:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393384407client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4408worker1 # [ 32.612919] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4409worker1 # [ 32.643815] nix-grpc-daemon[785]: ts=2026-09-21T09:12:21Z level=info event=niks3_push_started pid=8584410worker1 # [ 32.953816] nix-grpc-daemon[858]: time=2026-09-21T09:12:22.200Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""4411worker1 # [ 33.117897] nix-daemon[848]: accepted connection from pid 871, user nix-grpc-daemon (trusted)4412worker1 # [ 33.133415] nix-daemon[848]: reaped child process 874, status = succeeded4413worker1 # [ 33.270156] nix-daemon[848]: accepted connection from pid 876, user nix-grpc-daemon (trusted)4414worker1 # [ 33.281669] nix-daemon[848]: reaped child process 879, status = succeeded4415lb # [ 33.260313] niks3-server[1003]: 2026/09/21 09:12:23 INFO Received uploads request method=POST path=/api/pending_closures4416worker1 # [ 33.321323] nix-grpc-daemon[858]: time=2026-09-21T09:12:22.571Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4417worker1 # [ 33.324101] nix-grpc-daemon[858]: time=2026-09-21T09:12:22.574Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4418lb # [ 33.336682] niks3-server[1003]: 2026/09/21 09:12:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4419lb # [ 33.341829] niks3-server[1003]: 2026/09/21 09:12:23 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4420lb # [ 33.355670] niks3-server[1003]: 2026/09/21 09:12:23 INFO Signed narinfos id=1 count=14421worker1 # [ 33.392391] nix-grpc-daemon[858]: time=2026-09-21T09:12:22.642Z level=INFO msg="Uploading 1 narinfos"4422lb # [ 33.366297] niks3-server[1003]: 2026/09/21 09:12:23 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4423lb # [ 33.370123] niks3-server[1003]: 2026/09/21 09:12:23 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4424lb # [ 33.380298] niks3-server[1003]: 2026/09/21 09:12:23 INFO Completed upload id=14425worker1 # [ 33.415246] nix-grpc-daemon[858]: time=2026-09-21T09:12:22.665Z level=INFO msg="Upload complete. (450ms)"4426worker1 # [ 33.417075] nix-grpc-daemon[785]: ts=2026-09-21T09:12:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 paths=1 nar_bytes_in=5044427lb # [ 33.387741] niks3-server[1003]: 2026/09/21 09:12:23 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4428worker1 # [ 33.423196] nix-daemon[848]: reaped child process 856, status = succeeded4429worker1 # [ 33.424836] nix-grpc-daemon[785]: ts=2026-09-21T09:12:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384430worker1 # [ 33.428488] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4431worker1 # [ 33.435488] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4432client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4433client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4434client # farm-a> LOG-farm-a-t14435worker1 # [ 34.659439] nix-daemon[848]: reaped child process 884, status = succeeded4436worker1 # [ 34.839977] nix-daemon[848]: accepted connection from pid 899, user nix-grpc-daemon (trusted)4437worker1 # [ 34.850631] nix-daemon[848]: reaped child process 902, status = succeeded4438lb # [ 34.833632] niks3-server[1003]: 2026/09/21 09:12:24 INFO Received uploads request method=POST path=/api/pending_closures4439worker1 # [ 34.880794] nix-grpc-daemon[858]: time=2026-09-21T09:12:24.130Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4440worker1 # [ 34.882402] nix-grpc-daemon[858]: time=2026-09-21T09:12:24.132Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4441lb # [ 34.877261] niks3-server[1003]: 2026/09/21 09:12:24 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4442lb # [ 34.887818] niks3-server[1003]: 2026/09/21 09:12:24 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4443lb # [ 34.898302] niks3-server[1003]: 2026/09/21 09:12:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4444lb # [ 34.901412] niks3-server[1003]: 2026/09/21 09:12:24 INFO Signed narinfos id=2 count=14445worker1 # [ 34.939878] nix-grpc-daemon[858]: time=2026-09-21T09:12:24.189Z level=INFO msg="Uploading 1 narinfos"4446lb # [ 34.907960] niks3-server[1003]: 2026/09/21 09:12:24 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4447lb # [ 34.916606] niks3-server[1003]: 2026/09/21 09:12:24 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4448lb # [ 34.919344] niks3-server[1003]: 2026/09/21 09:12:24 INFO Completed upload id=24449lb # [ 34.921798] niks3-server[1003]: 2026/09/21 09:12:24 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4450worker1 # [ 34.954407] nix-grpc-daemon[858]: time=2026-09-21T09:12:24.204Z level=INFO msg="Upload complete. (307ms)"4451worker1 # [ 34.960793] nix-grpc-daemon[785]: ts=2026-09-21T09:12:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14452worker1 # [ 34.972845] nix-daemon[848]: reaped child process 882, status = succeeded4453worker2 # [ 34.959887] nix-grpc-daemon[786]: ts=2026-09-21T09:12:24Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04454worker2 # [ 34.963978] nix-grpc-daemon[786]: ts=2026-09-21T09:12:24Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14455worker1 # [ 35.025580] nix-grpc-daemon[785]: ts=2026-09-21T09:12:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384456worker1 # [ 35.038823] nix-grpc-daemon[785]: ts=2026-09-21T09:12:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393384457worker1 # [ 35.052068] nix-grpc-daemon[858]: time=2026-09-21T09:12:24.302Z level=INFO msg="All 1 paths already cached"4458client # copying 1 paths to 'lb:50051'...4459worker1 # [ 35.060309] nix-grpc-daemon[785]: ts=2026-09-21T09:12:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393384460client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4461worker1 # [ 35.079322] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4462worker1 # [ 35.262822] nix-daemon[848]: accepted connection from pid 906, user nix-grpc-daemon (trusted)4463worker1 # [ 35.275435] nix-daemon[848]: reaped child process 909, status = succeeded4464worker1 # [ 35.434384] nix-daemon[848]: accepted connection from pid 911, user nix-grpc-daemon (trusted)4465worker1 # [ 35.446230] nix-daemon[848]: reaped child process 914, status = succeeded4466lb # [ 35.425991] niks3-server[1003]: 2026/09/21 09:12:25 INFO Received uploads request method=POST path=/api/pending_closures4467worker1 # [ 35.473135] nix-grpc-daemon[858]: time=2026-09-21T09:12:24.723Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4468worker1 # [ 35.474684] nix-grpc-daemon[858]: time=2026-09-21T09:12:24.724Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4469lb # [ 35.462145] niks3-server[1003]: 2026/09/21 09:12:25 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4470lb # [ 35.469983] niks3-server[1003]: 2026/09/21 09:12:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4471worker1 # [ 35.511465] nix-grpc-daemon[858]: time=2026-09-21T09:12:24.761Z level=INFO msg="Uploading 1 narinfos"4472lb # [ 35.476840] niks3-server[1003]: 2026/09/21 09:12:25 INFO Signed narinfos id=3 count=14473lb # [ 35.483923] niks3-server[1003]: 2026/09/21 09:12:25 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4474lb # [ 35.492919] niks3-server[1003]: 2026/09/21 09:12:25 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4475lb # [ 35.501637] niks3-server[1003]: 2026/09/21 09:12:25 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4476lb # [ 35.504784] niks3-server[1003]: 2026/09/21 09:12:25 INFO Completed upload id=34477worker1 # [ 35.539878] nix-grpc-daemon[858]: time=2026-09-21T09:12:24.789Z level=INFO msg="Upload complete. (446ms)"4478worker1 # [ 35.545148] nix-grpc-daemon[785]: ts=2026-09-21T09:12:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 paths=1 nar_bytes_in=6244479worker1 # [ 35.550594] nix-grpc-daemon[785]: ts=2026-09-21T09:12:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384480worker1 # [ 35.560191] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4481worker1 # [ 35.568402] nix-daemon[848]: reaped child process 904, status = succeeded4482worker1 # [ 35.573524] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4483client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4484client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4485client # farm-b> LOG-farm-b-t14486worker1 # [ 36.999146] nix-daemon[848]: reaped child process 918, status = succeeded4487worker1 # [ 37.169960] nix-daemon[848]: accepted connection from pid 934, user nix-grpc-daemon (trusted)4488worker1 # [ 37.181699] nix-daemon[848]: reaped child process 937, status = succeeded4489lb # [ 37.165772] niks3-server[1003]: 2026/09/21 09:12:27 INFO Received uploads request method=POST path=/api/pending_closures4490worker1 # [ 37.207678] nix-grpc-daemon[858]: time=2026-09-21T09:12:26.457Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4491worker1 # [ 37.209330] nix-grpc-daemon[858]: time=2026-09-21T09:12:26.459Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4492lb # [ 37.198196] niks3-server[1003]: 2026/09/21 09:12:27 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4493lb # [ 37.213239] niks3-server[1003]: 2026/09/21 09:12:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4494lb # [ 37.218098] niks3-server[1003]: 2026/09/21 09:12:27 INFO Signed narinfos id=4 count=14495worker1 # [ 37.255965] nix-grpc-daemon[858]: time=2026-09-21T09:12:26.505Z level=INFO msg="Uploading 1 narinfos"4496lb # [ 37.226484] niks3-server[1003]: 2026/09/21 09:12:27 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4497lb # [ 37.234273] niks3-server[1003]: 2026/09/21 09:12:27 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4498lb # [ 37.241193] niks3-server[1003]: 2026/09/21 09:12:27 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4499lb # [ 37.244882] niks3-server[1003]: 2026/09/21 09:12:27 INFO Completed upload id=44500lb # [ 37.246623] niks3-server[1003]: 2026/09/21 09:12:27 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4501worker1 # [ 37.281274] nix-grpc-daemon[858]: time=2026-09-21T09:12:26.531Z level=INFO msg="Upload complete. (299ms)"4502worker1 # [ 37.286802] nix-grpc-daemon[785]: ts=2026-09-21T09:12:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14503worker2 # [ 37.273890] nix-grpc-daemon[786]: ts=2026-09-21T09:12:26Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04504worker1 # [ 37.301126] nix-daemon[848]: reaped child process 916, status = succeeded4505worker2 # [ 37.279074] nix-grpc-daemon[786]: ts=2026-09-21T09:12:26Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14506worker1 # [ 37.344948] nix-grpc-daemon[785]: ts=2026-09-21T09:12:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384507worker1 # [ 37.357248] nix-grpc-daemon[785]: ts=2026-09-21T09:12:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393384508worker1 # [ 37.369729] nix-grpc-daemon[858]: time=2026-09-21T09:12:26.619Z level=INFO msg="All 2 paths already cached"4509client # copying 1 paths to 'lb:50051'...4510worker1 # [ 37.384360] nix-grpc-daemon[785]: ts=2026-09-21T09:12:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393384511client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4512worker1 # [ 37.401576] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4513worker1 # [ 37.589539] nix-daemon[848]: accepted connection from pid 941, user nix-grpc-daemon (trusted)4514worker1 # [ 37.601435] nix-daemon[848]: reaped child process 944, status = succeeded4515worker1 # [ 37.763226] nix-daemon[848]: accepted connection from pid 946, user nix-grpc-daemon (trusted)4516worker1 # [ 37.777879] nix-daemon[848]: reaped child process 949, status = succeeded4517lb # [ 37.758094] niks3-server[1003]: 2026/09/21 09:12:27 INFO Received uploads request method=POST path=/api/pending_closures4518worker1 # [ 37.801987] nix-grpc-daemon[858]: time=2026-09-21T09:12:27.052Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4519worker1 # [ 37.803598] nix-grpc-daemon[858]: time=2026-09-21T09:12:27.053Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4520lb # [ 37.787459] niks3-server[1003]: 2026/09/21 09:12:27 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4521lb # [ 37.799349] niks3-server[1003]: 2026/09/21 09:12:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4522lb # [ 37.804352] niks3-server[1003]: 2026/09/21 09:12:27 INFO Signed narinfos id=5 count=14523worker1 # [ 37.842328] nix-grpc-daemon[858]: time=2026-09-21T09:12:27.092Z level=INFO msg="Uploading 1 narinfos"4524lb # [ 37.810574] niks3-server[1003]: 2026/09/21 09:12:27 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4525lb # [ 37.823740] niks3-server[1003]: 2026/09/21 09:12:27 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4526worker1 # [ 37.863889] nix-grpc-daemon[858]: time=2026-09-21T09:12:27.114Z level=INFO msg="Upload complete. (447ms)"4527lb # [ 37.829065] niks3-server[1003]: 2026/09/21 09:12:27 INFO Completed upload id=54528worker1 # [ 37.865983] nix-grpc-daemon[785]: ts=2026-09-21T09:12:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 paths=1 nar_bytes_in=7524529lb # [ 37.833229] niks3-server[1003]: 2026/09/21 09:12:27 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4530worker1 # [ 37.872128] nix-grpc-daemon[785]: ts=2026-09-21T09:12:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384531worker1 # [ 37.882447] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4532worker1 # [ 37.884683] nix-daemon[848]: reaped child process 939, status = succeeded4533worker1 # [ 37.890644] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4534client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4535client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4536lb # [ 38.125719] envoy[918]: conn peer=[2001:db8:1::1]:50982 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=749 tx=423 ms=26964537lb # [ 38.129395] envoy[918]: conn peer=[2001:db8:1::1]:50998 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=694538lb # [ 38.132802] envoy[918]: conn peer=[2001:db8:1::1]:51014 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=114539lb # [ 38.136106] envoy[918]: conn peer=[2001:db8:1::1]:51016 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=94540client # farm-top> LOG-farm-top-t14541worker1 # [ 38.903964] nix-daemon[848]: reaped child process 953, status = succeeded4542worker1 # [ 39.056359] nix-daemon[848]: accepted connection from pid 968, user nix-grpc-daemon (trusted)4543worker1 # [ 39.067984] nix-daemon[848]: reaped child process 971, status = succeeded4544lb # [ 39.051226] niks3-server[1003]: 2026/09/21 09:12:29 INFO Received uploads request method=POST path=/api/pending_closures4545worker1 # [ 39.094883] nix-grpc-daemon[858]: time=2026-09-21T09:12:28.344Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4546worker1 # [ 39.096527] nix-grpc-daemon[858]: time=2026-09-21T09:12:28.346Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4547lb # [ 39.092587] niks3-server[1003]: 2026/09/21 09:12:29 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4548lb # [ 39.104907] niks3-server[1003]: 2026/09/21 09:12:29 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4549lb # [ 39.111682] niks3-server[1003]: 2026/09/21 09:12:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4550lb # [ 39.116106] niks3-server[1003]: 2026/09/21 09:12:29 INFO Signed narinfos id=6 count=14551worker1 # [ 39.151576] nix-grpc-daemon[858]: time=2026-09-21T09:12:28.401Z level=INFO msg="Uploading 1 narinfos"4552lb # [ 39.123675] niks3-server[1003]: 2026/09/21 09:12:29 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4553lb # [ 39.131123] niks3-server[1003]: 2026/09/21 09:12:29 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4554lb # [ 39.137384] niks3-server[1003]: 2026/09/21 09:12:29 INFO Completed upload id=64555lb # [ 39.139303] niks3-server[1003]: 2026/09/21 09:12:29 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4556worker1 # [ 39.173868] nix-grpc-daemon[858]: time=2026-09-21T09:12:28.423Z level=INFO msg="Upload complete. (276ms)"4557worker1 # [ 39.180635] nix-grpc-daemon[785]: ts=2026-09-21T09:12:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14558worker1 # [ 39.194139] nix-daemon[848]: reaped child process 951, status = succeeded4559client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 7.07 seconds)4560client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4561client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.22 seconds)4562client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4563client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.02 seconds)4564client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4565client # copying 3 paths...4566client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4567client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4568client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4569client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.29 seconds)4570(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.67 seconds)4571subtest: repeat is answered Cached by the scheduler without building4572worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4573worker1 # finding garbage collector roots...4574worker1 # deleting specified paths...4575worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4576worker1 # deleting unused links...4577worker1 # note: hard linking is currently saving 0.0 KiB4578worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.24 seconds)4579worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4580worker2 # finding garbage collector roots...4581worker2 # deleting specified paths...4582worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4583worker2 # deleting unused links...4584worker2 # note: hard linking is currently saving 0.0 KiB4585worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.29 seconds)4586client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24587worker1 # [ 40.613805] nix-grpc-daemon[785]: ts=2026-09-21T09:12:29Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:393384588client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4589client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4590worker1 # [ 40.628903] nix-grpc-daemon[785]: ts=2026-09-21T09:12:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393384591worker1 # [ 40.639949] nix-grpc-daemon[858]: time=2026-09-21T09:12:29.889Z level=INFO msg="All 2 paths already cached"4592worker2 # [ 40.633867] nix-grpc-daemon[786]: ts=2026-09-21T09:12:29Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14593client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.33 seconds)4594client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4595client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.20 seconds)4596lb: must succeed: curl -sf localhost:9901/clusters4597lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4598lb: must succeed: curl -sf localhost:9901/clusters4599lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4600worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4601worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4602(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.35 seconds)4603subtest: inputs already in the cache are not sent to the scheduler4604lb: must succeed: curl -sf localhost:9901/clusters4605lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4606lb: must succeed: curl -sf localhost:9901/clusters4607lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4608worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4609worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4610client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24611client # don't know how to build these paths:4612client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4613worker1 # [ 41.572070] nix-grpc-daemon[785]: ts=2026-09-21T09:12:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:393384614worker2 # [ 41.556149] nix-grpc-daemon[786]: ts=2026-09-21T09:12:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:450084615worker2 # [ 41.582217] nix-grpc-daemon[786]: ts=2026-09-21T09:12:30Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04616worker2 # [ 41.586221] nix-grpc-daemon[786]: ts=2026-09-21T09:12:30Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14617worker1 # [ 41.657726] nix-grpc-daemon[785]: ts=2026-09-21T09:12:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384618worker1 # [ 41.669975] nix-grpc-daemon[785]: ts=2026-09-21T09:12:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393384619worker1 # [ 41.683780] nix-grpc-daemon[858]: time=2026-09-21T09:12:30.933Z level=INFO msg="All 2 paths already cached"4620client # copying 1 paths to 'lb:50051'...4621client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4622worker1 # [ 41.692238] nix-grpc-daemon[785]: ts=2026-09-21T09:12:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393384623worker1 # [ 41.703172] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4624worker1 # [ 41.892461] nix-daemon[848]: accepted connection from pid 989, user nix-grpc-daemon (trusted)4625worker1 # [ 41.904849] nix-daemon[848]: reaped child process 992, status = succeeded4626worker1 # [ 42.080077] nix-daemon[848]: accepted connection from pid 994, user nix-grpc-daemon (trusted)4627worker1 # [ 42.091691] nix-daemon[848]: reaped child process 997, status = succeeded4628lb # [ 42.075247] niks3-server[1003]: 2026/09/21 09:12:32 INFO Received uploads request method=POST path=/api/pending_closures4629worker1 # [ 42.118838] nix-grpc-daemon[858]: time=2026-09-21T09:12:31.368Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4630worker1 # [ 42.120251] nix-grpc-daemon[858]: time=2026-09-21T09:12:31.368Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4631lb # [ 42.100499] niks3-server[1003]: 2026/09/21 09:12:32 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4632lb # [ 42.111576] niks3-server[1003]: 2026/09/21 09:12:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4633lb # [ 42.117664] niks3-server[1003]: 2026/09/21 09:12:32 INFO Signed narinfos id=7 count=14634worker1 # [ 42.155647] nix-grpc-daemon[858]: time=2026-09-21T09:12:31.405Z level=INFO msg="Uploading 1 narinfos"4635lb # [ 42.124257] niks3-server[1003]: 2026/09/21 09:12:32 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4636lb # [ 42.134785] niks3-server[1003]: 2026/09/21 09:12:32 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4637worker1 # [ 42.174327] nix-grpc-daemon[858]: time=2026-09-21T09:12:31.424Z level=INFO msg="Upload complete. (458ms)"4638lb # [ 42.139487] niks3-server[1003]: 2026/09/21 09:12:32 INFO Completed upload id=74639worker1 # [ 42.176794] nix-grpc-daemon[785]: ts=2026-09-21T09:12:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 paths=1 nar_bytes_in=7524640lb # [ 42.145143] niks3-server[1003]: 2026/09/21 09:12:32 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4641worker1 # [ 42.182269] nix-grpc-daemon[785]: ts=2026-09-21T09:12:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384642worker1 # [ 42.190263] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4643worker1 # [ 42.195571] nix-daemon[848]: reaped child process 987, status = succeeded4644worker1 # [ 42.199808] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4645client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4646client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4647client # farm-top> LOG-farm-top-t1b4648worker1 # [ 43.413098] nix-daemon[848]: reaped child process 1001, status = succeeded4649worker1 # [ 43.581353] nix-daemon[848]: accepted connection from pid 1017, user nix-grpc-daemon (trusted)4650worker1 # [ 43.592194] nix-daemon[848]: reaped child process 1020, status = succeeded4651lb # [ 43.575464] niks3-server[1003]: 2026/09/21 09:12:33 INFO Received uploads request method=POST path=/api/pending_closures4652worker1 # [ 43.618744] nix-grpc-daemon[858]: time=2026-09-21T09:12:32.868Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4653worker1 # [ 43.620792] nix-grpc-daemon[858]: time=2026-09-21T09:12:32.870Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4654lb # [ 43.609786] niks3-server[1003]: 2026/09/21 09:12:33 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4655lb # [ 43.624132] niks3-server[1003]: 2026/09/21 09:12:33 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4656lb # [ 43.631191] niks3-server[1003]: 2026/09/21 09:12:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4657lb # [ 43.635559] niks3-server[1003]: 2026/09/21 09:12:33 INFO Signed narinfos id=8 count=14658worker1 # [ 43.674963] nix-grpc-daemon[858]: time=2026-09-21T09:12:32.924Z level=INFO msg="Uploading 1 narinfos"4659lb # [ 43.641423] niks3-server[1003]: 2026/09/21 09:12:33 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4660lb # [ 43.650109] niks3-server[1003]: 2026/09/21 09:12:33 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4661lb # [ 43.657186] niks3-server[1003]: 2026/09/21 09:12:33 INFO Completed upload id=84662worker1 # [ 43.691619] nix-grpc-daemon[858]: time=2026-09-21T09:12:32.941Z level=INFO msg="Upload complete. (285ms)"4663lb # [ 43.660948] niks3-server[1003]: 2026/09/21 09:12:33 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4664worker1 # [ 43.697522] nix-grpc-daemon[785]: ts=2026-09-21T09:12:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14665worker1 # [ 43.710173] nix-daemon[848]: reaped child process 999, status = succeeded4666client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.41 seconds)4667client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4668client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.20 seconds)4669lb: must succeed: curl -sf localhost:9901/clusters4670lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4671lb: must succeed: curl -sf localhost:9901/clusters4672lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4673worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4674worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4675(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.95 seconds)4676subtest: two clients wanting the same drv build it once4677client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4678client # [ 44.149206] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4679client # Running as unit: dedup1.service; invocation ID: d8455074ef0d42f0bdd3ef18248530534680client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.07 seconds)4681client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4682worker1 # [ 44.398402] nix-grpc-daemon[785]: ts=2026-09-21T09:12:33Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:393384683client # [ 44.371854] nix[895]: don't know how to build these paths:4684client # [ 44.372964] nix[895]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4685worker2 # [ 44.393739] nix-grpc-daemon[786]: ts=2026-09-21T09:12:33Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04686worker2 # [ 44.399418] nix-grpc-daemon[786]: ts=2026-09-21T09:12:33Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=14687worker2 # [ 44.403959] nix-grpc-daemon[786]: ts=2026-09-21T09:12:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450084688worker2 # [ 44.419837] nix-grpc-daemon[786]: ts=2026-09-21T09:12:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:450084689client # [ 44.417723] nix[895]: copying 1 paths to 'lb:50051'...4690client # [ 44.418519] nix[895]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4691worker2 # [ 44.430319] nix-grpc-daemon[786]: ts=2026-09-21T09:12:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:450084692worker2 # [ 44.441841] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4693worker2 # [ 44.464922] nix-grpc-daemon[786]: ts=2026-09-21T09:12:33Z level=info event=niks3_push_started pid=9214694worker2 # [ 44.771732] nix-grpc-daemon[921]: time=2026-09-21T09:12:33.978Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""4695worker2 # [ 44.980884] nix-daemon[869]: accepted connection from pid 933, user nix-grpc-daemon (trusted)4696worker2 # [ 44.995832] nix-daemon[869]: reaped child process 936, status = succeeded4697worker2 # [ 45.169918] nix-daemon[869]: accepted connection from pid 938, user nix-grpc-daemon (trusted)4698worker2 # [ 45.181987] nix-daemon[869]: reaped child process 941, status = succeeded4699lb # [ 45.187201] niks3-server[1003]: 2026/09/21 09:12:35 INFO Received uploads request method=POST path=/api/pending_closures4700worker2 # [ 45.212150] nix-grpc-daemon[921]: time=2026-09-21T09:12:34.422Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4701worker2 # [ 45.214694] nix-grpc-daemon[921]: time=2026-09-21T09:12:34.425Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4702lb # [ 45.227850] niks3-server[1003]: 2026/09/21 09:12:35 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4703client # [ 45.230644] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4704lb # [ 45.238326] niks3-server[1003]: 2026/09/21 09:12:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4705client # Running as unit: dedup2.service; invocation ID: d88ef0aa02d048548e617222939999204706lb # [ 45.244263] niks3-server[1003]: 2026/09/21 09:12:35 INFO Signed narinfos id=9 count=14707worker2 # [ 45.260836] nix-grpc-daemon[921]: time=2026-09-21T09:12:34.471Z level=INFO msg="Uploading 1 narinfos"4708client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.09 seconds)4709lb # [ 45.255184] niks3-server[1003]: 2026/09/21 09:12:35 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4710lb # [ 45.262365] niks3-server[1003]: 2026/09/21 09:12:35 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4711lb # [ 45.269127] niks3-server[1003]: 2026/09/21 09:12:35 INFO Completed upload id=94712worker2 # [ 45.283224] nix-grpc-daemon[921]: time=2026-09-21T09:12:34.493Z level=INFO msg="Upload complete. (494ms)"4713lb # [ 45.274075] niks3-server[1003]: 2026/09/21 09:12:35 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4714worker2 # [ 45.287183] nix-grpc-daemon[786]: ts=2026-09-21T09:12:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45008 duration_s=0 paths=1 nar_bytes_in=4324715worker2 # [ 45.292566] nix-grpc-daemon[786]: ts=2026-09-21T09:12:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450084716worker2 # [ 45.300966] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4717worker2 # [ 45.303489] nix-daemon[869]: reaped child process 919, status = succeeded4718client # [ 45.294717] nix[895]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4719worker2 # [ 45.307500] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4720client # [ 45.508404] nix[911]: don't know how to build these paths:4721worker1 # [ 45.538637] nix-grpc-daemon[785]: ts=2026-09-21T09:12:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:393384722client # [ 45.509244] nix[911]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4723worker2 # [ 45.528840] nix-grpc-daemon[786]: ts=2026-09-21T09:12:34Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04724worker2 # [ 45.533122] nix-grpc-daemon[786]: ts=2026-09-21T09:12:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450084725client # [ 45.523646] nix[911]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4726client: must succeed: sleep 24727lb # [ 48.126318] envoy[918]: conn peer=[2001:db8:1::1]:51018 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9554 tx=3855 ms=68164728lb # [ 48.130162] envoy[918]: conn peer=[2001:db8:1::1]:50652 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=574729lb # [ 48.133590] envoy[918]: conn peer=[2001:db8:1::1]:50654 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4139 tx=1707 ms=21384730client: (finished: must succeed: sleep 2, in 2.05 seconds)4731client: waiting for success: ! systemctl is-active dedup1 dedup24732worker2 # [ 48.721739] nix-daemon[869]: reaped child process 945, status = succeeded4733worker2 # [ 48.883089] nix-daemon[869]: accepted connection from pid 976, user nix-grpc-daemon (trusted)4734worker2 # [ 48.893462] nix-daemon[869]: reaped child process 979, status = succeeded4735lb # [ 48.898272] niks3-server[1003]: 2026/09/21 09:12:38 INFO Received uploads request method=POST path=/api/pending_closures4736worker2 # [ 48.921249] nix-grpc-daemon[921]: time=2026-09-21T09:12:38.131Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4737worker2 # [ 48.922889] nix-grpc-daemon[921]: time=2026-09-21T09:12:38.133Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4738lb # [ 48.939904] niks3-server[1003]: 2026/09/21 09:12:38 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4739lb # [ 48.948167] niks3-server[1003]: 2026/09/21 09:12:38 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4740lb # [ 48.958974] niks3-server[1003]: 2026/09/21 09:12:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4741lb # [ 48.964123] niks3-server[1003]: 2026/09/21 09:12:38 INFO Signed narinfos id=10 count=14742worker2 # [ 48.981347] nix-grpc-daemon[921]: time=2026-09-21T09:12:38.191Z level=INFO msg="Uploading 1 narinfos"4743lb # [ 48.969727] niks3-server[1003]: 2026/09/21 09:12:39 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4744lb # [ 48.980504] niks3-server[1003]: 2026/09/21 09:12:39 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4745lb # [ 48.985367] niks3-server[1003]: 2026/09/21 09:12:39 INFO Completed upload id=104746lb # [ 48.988259] niks3-server[1003]: 2026/09/21 09:12:39 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo4747worker2 # [ 49.001676] nix-grpc-daemon[921]: time=2026-09-21T09:12:38.212Z level=INFO msg="Upload complete. (283ms)"4748worker2 # [ 49.010259] nix-grpc-daemon[786]: ts=2026-09-21T09:12:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45008 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14749worker2 # [ 49.018140] nix-grpc-daemon[786]: ts=2026-09-21T09:12:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45008 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14750worker2 # [ 49.027643] nix-daemon[869]: reaped child process 943, status = succeeded4751client # [ 49.043108] systemd[1]: dedup1.service: Deactivated successfully.4752client # [ 49.051269] systemd[1]: dedup1.service: Consumed 124ms CPU time over 4.897s wall clock time, 41.5M memory peak, 7K incoming IP traffic, 8.2K outgoing IP traffic.4753client # [ 49.086935] systemd[1]: dedup2.service: Deactivated successfully.4754client # [ 49.088386] systemd[1]: dedup2.service: Consumed 148ms CPU time over 3.858s wall clock time, 35.3M memory peak, 6K incoming IP traffic, 5.7K outgoing IP traffic.4755client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.09 seconds)4756client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4757client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.03 seconds)4758worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4759worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)4760worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4761worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)4762worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4763worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4764worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4765worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4766(finished: subtest: two clients wanting the same drv build it once, in 5.95 seconds)4767subtest: build hook: nix-daemon with builders = grpc://lb4768client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log4769client # [ 50.253774] systemd[1]: Started Nix Daemon.4770client # [ 50.434232] nix-daemon[953]: accepted connection from pid 950, user root (trusted)4771worker1 # [ 50.681799] nix-grpc-daemon[785]: ts=2026-09-21T09:12:39Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:393384772worker2 # [ 50.665321] nix-grpc-daemon[786]: ts=2026-09-21T09:12:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:450084773worker2 # [ 50.681566] nix-grpc-daemon[786]: ts=2026-09-21T09:12:39Z level=debug event=want client=6 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=04774worker2 # [ 50.686525] nix-grpc-daemon[786]: ts=2026-09-21T09:12:39Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=14775worker1 # [ 50.746594] nix-grpc-daemon[785]: ts=2026-09-21T09:12:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384776worker1 # [ 50.758860] nix-grpc-daemon[785]: ts=2026-09-21T09:12:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393384777worker1 # [ 50.770343] nix-grpc-daemon[785]: ts=2026-09-21T09:12:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393384778worker1 # [ 50.781338] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4779worker1 # [ 50.974274] nix-daemon[848]: accepted connection from pid 1045, user nix-grpc-daemon (trusted)4780worker1 # [ 50.988211] nix-daemon[848]: reaped child process 1048, status = succeeded4781worker1 # [ 51.161536] nix-daemon[848]: accepted connection from pid 1050, user nix-grpc-daemon (trusted)4782worker1 # [ 51.173660] nix-daemon[848]: reaped child process 1053, status = succeeded4783lb # [ 51.154792] niks3-server[1003]: 2026/09/21 09:12:41 INFO Received uploads request method=POST path=/api/pending_closures4784worker1 # [ 51.197937] nix-grpc-daemon[858]: time=2026-09-21T09:12:40.447Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4785worker1 # [ 51.199254] nix-grpc-daemon[858]: time=2026-09-21T09:12:40.447Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"4786lb # [ 51.180485] niks3-server[1003]: 2026/09/21 09:12:41 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst4787lb # [ 51.194807] niks3-server[1003]: 2026/09/21 09:12:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4788worker1 # [ 51.235083] nix-grpc-daemon[858]: time=2026-09-21T09:12:40.484Z level=INFO msg="Uploading 1 narinfos"4789lb # [ 51.200124] niks3-server[1003]: 2026/09/21 09:12:41 INFO Signed narinfos id=11 count=14790lb # [ 51.206625] niks3-server[1003]: 2026/09/21 09:12:41 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls4791lb # [ 51.218139] niks3-server[1003]: 2026/09/21 09:12:41 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4792lb # [ 51.225170] niks3-server[1003]: 2026/09/21 09:12:41 INFO Completed upload id=114793worker1 # [ 51.260028] nix-grpc-daemon[858]: time=2026-09-21T09:12:40.510Z level=INFO msg="Upload complete. (467ms)"4794lb # [ 51.227925] niks3-server[1003]: 2026/09/21 09:12:41 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo4795worker1 # [ 51.263329] nix-grpc-daemon[785]: ts=2026-09-21T09:12:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 paths=1 nar_bytes_in=5124796worker1 # [ 51.268922] nix-grpc-daemon[785]: ts=2026-09-21T09:12:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384797worker1 # [ 51.279451] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4798worker1 # [ 51.282461] nix-daemon[848]: reaped child process 1043, status = succeeded4799worker1 # [ 51.288212] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4800worker1 # [ 52.114746] nix-daemon[848]: reaped child process 1057, status = succeeded4801worker1 # [ 52.250766] nix-daemon[848]: accepted connection from pid 1072, user nix-grpc-daemon (trusted)4802worker1 # [ 52.263099] nix-daemon[848]: reaped child process 1075, status = succeeded4803lb # [ 52.238422] niks3-server[1003]: 2026/09/21 09:12:42 INFO Received uploads request method=POST path=/api/pending_closures4804worker1 # [ 52.277752] nix-grpc-daemon[858]: time=2026-09-21T09:12:41.527Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4805worker1 # [ 52.279255] nix-grpc-daemon[858]: time=2026-09-21T09:12:41.529Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"4806lb # [ 52.258690] niks3-server[1003]: 2026/09/21 09:12:42 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv4807lb # [ 52.264056] niks3-server[1003]: 2026/09/21 09:12:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4808lb # [ 52.266626] niks3-server[1003]: 2026/09/21 09:12:42 INFO Signed narinfos id=12 count=14809worker1 # [ 52.301513] nix-grpc-daemon[858]: time=2026-09-21T09:12:41.551Z level=INFO msg="Uploading 1 narinfos"4810lb # [ 52.269702] niks3-server[1003]: 2026/09/21 09:12:42 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4811lb # [ 52.272451] niks3-server[1003]: 2026/09/21 09:12:42 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls4812lb # [ 52.276053] niks3-server[1003]: 2026/09/21 09:12:42 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4813lb # [ 52.279621] niks3-server[1003]: 2026/09/21 09:12:42 INFO Completed upload id=124814worker1 # [ 52.314516] nix-grpc-daemon[858]: time=2026-09-21T09:12:41.564Z level=INFO msg="Upload complete. (220ms)"4815lb # [ 52.281428] niks3-server[1003]: 2026/09/21 09:12:42 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo4816worker1 # [ 52.316908] nix-grpc-daemon[785]: ts=2026-09-21T09:12:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=6 outputs=14817worker1 # [ 52.323073] nix-daemon[848]: reaped child process 1055, status = succeeded4818worker2 # [ 52.343680] nix-grpc-daemon[786]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:450084819worker2 # [ 52.356521] nix-grpc-daemon[786]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...4820worker1 # [ 52.407885] nix-grpc-daemon[785]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:393384821worker1 # [ 52.412672] nix-grpc-daemon[785]: ts=2026-09-21T09:12:41Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 paths=1 nar_bytes_out=1284822worker1 # [ 52.664261] nix-grpc-daemon[785]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:393384823worker2 # [ 52.645972] nix-grpc-daemon[786]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:450084824worker2 # [ 52.654637] nix-grpc-daemon[921]: time=2026-09-21T09:12:41.864Z level=INFO msg="All 1 paths already cached"4825worker2 # [ 52.667183] nix-grpc-daemon[786]: ts=2026-09-21T09:12:41Z level=debug event=want client=7 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=04826worker2 # [ 52.671065] nix-grpc-daemon[786]: ts=2026-09-21T09:12:41Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=14827worker1 # [ 52.698717] nix-grpc-daemon[785]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384828worker1 # [ 52.709074] nix-grpc-daemon[785]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393384829worker1 # [ 52.718632] nix-grpc-daemon[858]: time=2026-09-21T09:12:41.968Z level=INFO msg="All 1 paths already cached"4830worker1 # [ 52.724226] nix-grpc-daemon[785]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393384831worker1 # [ 52.734178] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4832worker1 # [ 52.884613] nix-daemon[848]: accepted connection from pid 1079, user nix-grpc-daemon (trusted)4833worker1 # [ 52.898489] nix-daemon[848]: reaped child process 1082, status = succeeded4834worker1 # [ 53.027603] nix-daemon[848]: accepted connection from pid 1084, user nix-grpc-daemon (trusted)4835worker1 # [ 53.039802] nix-daemon[848]: reaped child process 1087, status = succeeded4836lb # [ 53.014174] niks3-server[1003]: 2026/09/21 09:12:43 INFO Received uploads request method=POST path=/api/pending_closures4837worker1 # [ 53.053790] nix-grpc-daemon[858]: time=2026-09-21T09:12:42.303Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4838worker1 # [ 53.055197] nix-grpc-daemon[858]: time=2026-09-21T09:12:42.303Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"4839lb # [ 53.031195] niks3-server[1003]: 2026/09/21 09:12:43 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst4840lb # [ 53.044234] niks3-server[1003]: 2026/09/21 09:12:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4841worker1 # [ 53.084158] nix-grpc-daemon[858]: time=2026-09-21T09:12:42.334Z level=INFO msg="Uploading 1 narinfos"4842lb # [ 53.050141] niks3-server[1003]: 2026/09/21 09:12:43 INFO Signed narinfos id=13 count=14843lb # [ 53.056512] niks3-server[1003]: 2026/09/21 09:12:43 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls4844lb # [ 53.066622] niks3-server[1003]: 2026/09/21 09:12:43 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4845worker1 # [ 53.108356] nix-grpc-daemon[858]: time=2026-09-21T09:12:42.358Z level=INFO msg="Upload complete. (361ms)"4846lb # [ 53.073523] niks3-server[1003]: 2026/09/21 09:12:43 INFO Completed upload id=134847worker1 # [ 53.110283] nix-grpc-daemon[785]: ts=2026-09-21T09:12:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 paths=1 nar_bytes_in=6244848lb # [ 53.077066] niks3-server[1003]: 2026/09/21 09:12:43 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo4849worker1 # [ 53.115128] nix-grpc-daemon[785]: ts=2026-09-21T09:12:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384850worker1 # [ 53.120468] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4851worker1 # [ 53.122722] nix-daemon[848]: reaped child process 1077, status = succeeded4852worker1 # [ 53.127945] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4853worker1 # [ 53.739062] nix-daemon[848]: reaped child process 1091, status = succeeded4854worker1 # [ 53.896294] nix-daemon[848]: accepted connection from pid 1106, user nix-grpc-daemon (trusted)4855worker1 # [ 53.906934] nix-daemon[848]: reaped child process 1109, status = succeeded4856lb # [ 53.891391] niks3-server[1003]: 2026/09/21 09:12:43 INFO Received uploads request method=POST path=/api/pending_closures4857worker1 # [ 53.933095] nix-grpc-daemon[858]: time=2026-09-21T09:12:43.183Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4858worker1 # [ 53.934913] nix-grpc-daemon[858]: time=2026-09-21T09:12:43.184Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"4859lb # [ 53.930140] niks3-server[1003]: 2026/09/21 09:12:43 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv4860lb # [ 53.936673] niks3-server[1003]: 2026/09/21 09:12:43 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst4861lb # [ 53.945386] niks3-server[1003]: 2026/09/21 09:12:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4862worker1 # [ 53.985214] nix-grpc-daemon[858]: time=2026-09-21T09:12:43.235Z level=INFO msg="Uploading 1 narinfos"4863lb # [ 53.950646] niks3-server[1003]: 2026/09/21 09:12:43 INFO Signed narinfos id=14 count=14864lb # [ 53.955801] niks3-server[1003]: 2026/09/21 09:12:43 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls4865lb # [ 53.964319] niks3-server[1003]: 2026/09/21 09:12:43 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4866lb # [ 53.969955] niks3-server[1003]: 2026/09/21 09:12:44 INFO Completed upload id=144867worker1 # [ 54.004898] nix-grpc-daemon[858]: time=2026-09-21T09:12:43.254Z level=INFO msg="Upload complete. (271ms)"4868lb # [ 53.973372] niks3-server[1003]: 2026/09/21 09:12:44 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo4869worker1 # [ 54.011141] nix-grpc-daemon[785]: ts=2026-09-21T09:12:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=7 outputs=14870worker1 # [ 54.018553] nix-grpc-daemon[785]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:393384871worker1 # [ 54.027611] nix-daemon[848]: reaped child process 1089, status = succeeded4872worker1 # [ 54.031752] nix-grpc-daemon[785]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:393384873worker1 # [ 54.035267] nix-grpc-daemon[785]: ts=2026-09-21T09:12:43Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 paths=1 nar_bytes_out=1764874worker1 # [ 54.331112] nix-grpc-daemon[785]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:393384875worker1 # [ 54.336984] nix-grpc-daemon[785]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393384876worker1 # [ 54.342097] nix-grpc-daemon[858]: time=2026-09-21T09:12:43.591Z level=INFO msg="All 2 paths already cached"4877worker2 # [ 54.331852] nix-grpc-daemon[786]: ts=2026-09-21T09:12:43Z level=debug event=want client=8 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=04878worker2 # [ 54.336649] nix-grpc-daemon[786]: ts=2026-09-21T09:12:43Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=14879worker1 # [ 54.359516] nix-grpc-daemon[785]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384880worker1 # [ 54.371734] nix-grpc-daemon[785]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393384881worker1 # [ 54.383232] nix-grpc-daemon[858]: time=2026-09-21T09:12:43.633Z level=INFO msg="All 2 paths already cached"4882worker1 # [ 54.389863] nix-grpc-daemon[785]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393384883worker1 # [ 54.401389] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4884worker1 # [ 54.581161] nix-daemon[848]: accepted connection from pid 1113, user nix-grpc-daemon (trusted)4885worker1 # [ 54.593409] nix-daemon[848]: reaped child process 1116, status = succeeded4886worker1 # [ 54.762625] nix-daemon[848]: accepted connection from pid 1118, user nix-grpc-daemon (trusted)4887worker1 # [ 54.774342] nix-daemon[848]: reaped child process 1121, status = succeeded4888lb # [ 54.755689] niks3-server[1003]: 2026/09/21 09:12:44 INFO Received uploads request method=POST path=/api/pending_closures4889worker1 # [ 54.798335] nix-grpc-daemon[858]: time=2026-09-21T09:12:44.048Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4890worker1 # [ 54.799685] nix-grpc-daemon[858]: time=2026-09-21T09:12:44.048Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"4891lb # [ 54.779146] niks3-server[1003]: 2026/09/21 09:12:44 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst4892lb # [ 54.792102] niks3-server[1003]: 2026/09/21 09:12:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4893lb # [ 54.796947] niks3-server[1003]: 2026/09/21 09:12:44 INFO Signed narinfos id=15 count=14894worker1 # [ 54.835056] nix-grpc-daemon[858]: time=2026-09-21T09:12:44.085Z level=INFO msg="Uploading 1 narinfos"4895lb # [ 54.803937] niks3-server[1003]: 2026/09/21 09:12:44 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls4896lb # [ 54.814299] niks3-server[1003]: 2026/09/21 09:12:44 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4897lb # [ 54.820652] niks3-server[1003]: 2026/09/21 09:12:44 INFO Completed upload id=154898worker1 # [ 54.855284] nix-grpc-daemon[858]: time=2026-09-21T09:12:44.105Z level=INFO msg="Upload complete. (442ms)"4899lb # [ 54.824742] niks3-server[1003]: 2026/09/21 09:12:44 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo4900worker1 # [ 54.858640] nix-grpc-daemon[785]: ts=2026-09-21T09:12:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 paths=1 nar_bytes_in=7524901worker1 # [ 54.863856] nix-grpc-daemon[785]: ts=2026-09-21T09:12:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393384902worker1 # [ 54.873430] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4903worker1 # [ 54.876793] nix-daemon[848]: reaped child process 1111, status = succeeded4904worker1 # [ 54.882389] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4905worker1 # [ 55.707714] nix-daemon[848]: reaped child process 1125, status = succeeded4906worker1 # [ 55.874326] nix-daemon[848]: accepted connection from pid 1140, user nix-grpc-daemon (trusted)4907worker1 # [ 55.886086] nix-daemon[848]: reaped child process 1143, status = succeeded4908lb # [ 55.868906] niks3-server[1003]: 2026/09/21 09:12:45 INFO Received uploads request method=POST path=/api/pending_closures4909worker1 # [ 55.912206] nix-grpc-daemon[858]: time=2026-09-21T09:12:45.162Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4910worker1 # [ 55.913644] nix-grpc-daemon[858]: time=2026-09-21T09:12:45.163Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"4911lb # [ 55.910096] niks3-server[1003]: 2026/09/21 09:12:45 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv4912lb # [ 55.917489] niks3-server[1003]: 2026/09/21 09:12:45 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst4913lb # [ 55.929098] niks3-server[1003]: 2026/09/21 09:12:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4914lb # [ 55.933623] niks3-server[1003]: 2026/09/21 09:12:45 INFO Signed narinfos id=16 count=14915worker1 # [ 55.971364] nix-grpc-daemon[858]: time=2026-09-21T09:12:45.221Z level=INFO msg="Uploading 1 narinfos"4916lb # [ 55.940223] niks3-server[1003]: 2026/09/21 09:12:45 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls4917lb # [ 55.948979] niks3-server[1003]: 2026/09/21 09:12:45 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4918lb # [ 55.955533] niks3-server[1003]: 2026/09/21 09:12:45 INFO Completed upload id=164919lb # [ 55.957980] niks3-server[1003]: 2026/09/21 09:12:45 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo4920worker1 # [ 55.992365] nix-grpc-daemon[858]: time=2026-09-21T09:12:45.242Z level=INFO msg="Upload complete. (303ms)"4921worker1 # [ 55.998437] nix-grpc-daemon[785]: ts=2026-09-21T09:12:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=8 outputs=14922worker2 # [ 55.982240] nix-grpc-daemon[786]: ts=2026-09-21T09:12:45Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:450084923worker1 # [ 56.012272] nix-daemon[848]: reaped child process 1123, status = succeeded4924worker2 # [ 55.996461] nix-grpc-daemon[786]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...4925worker2 # [ 56.008468] nix-grpc-daemon[786]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...4926worker2 # [ 56.036354] nix-grpc-daemon[786]: ts=2026-09-21T09:12:45Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:450084927worker2 # [ 56.041896] nix-grpc-daemon[786]: ts=2026-09-21T09:12:45Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:45008 duration_s=0 paths=1 nar_bytes_out=2324928client # [ 56.141228] nix-daemon[953]: reaped child process 955, status = succeeded4929client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.12 seconds)4930client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top4931client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.03 seconds)4932client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4933client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.04 seconds)4934client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4935client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4936(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 6.21 seconds)4937subtest: an input only one worker has still reaches the builder4938client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only4939client # this derivation will be built:4940client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv4941client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...4942client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.43 seconds)4943client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure4944client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.20 seconds)4945worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4946worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.22 seconds)4947worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only4948worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.03 seconds)4949client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&24950client # [ 57.318662] nix-daemon[953]: accepted connection from pid 1036, user root (trusted)4951client # this derivation will be built:4952client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv4953worker2 # [ 57.519297] nix-grpc-daemon[786]: ts=2026-09-21T09:12:46Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:450084954client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...4955worker1 # [ 57.544651] nix-grpc-daemon[785]: ts=2026-09-21T09:12:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393384956worker1 # [ 57.719865] nix-daemon[848]: accepted connection from pid 1150, user nix-grpc-daemon (trusted)4957worker1 # [ 57.730383] nix-daemon[848]: reaped child process 1153, status = succeeded4958lb # [ 57.713421] niks3-server[1003]: 2026/09/21 09:12:47 INFO Received uploads request method=POST path=/api/pending_closures4959worker1 # [ 57.756637] nix-grpc-daemon[858]: time=2026-09-21T09:12:47.006Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4960worker1 # [ 57.757981] nix-grpc-daemon[858]: time=2026-09-21T09:12:47.006Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"4961lb # [ 57.746140] niks3-server[1003]: 2026/09/21 09:12:47 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4962lb # [ 57.754296] niks3-server[1003]: 2026/09/21 09:12:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4963worker1 # [ 57.794905] nix-grpc-daemon[858]: time=2026-09-21T09:12:47.044Z level=INFO msg="Uploading 1 narinfos"4964lb # [ 57.759914] niks3-server[1003]: 2026/09/21 09:12:47 INFO Signed narinfos id=17 count=14965lb # [ 57.765681] niks3-server[1003]: 2026/09/21 09:12:47 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls4966lb # [ 57.778243] niks3-server[1003]: 2026/09/21 09:12:47 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4967lb # [ 57.784955] niks3-server[1003]: 2026/09/21 09:12:47 INFO Completed upload id=174968worker1 # [ 57.819878] nix-grpc-daemon[858]: time=2026-09-21T09:12:47.070Z level=INFO msg="Upload complete. (269ms)"4969lb # [ 57.789092] niks3-server[1003]: 2026/09/21 09:12:47 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo4970worker2 # [ 57.817369] nix-grpc-daemon[786]: ts=2026-09-21T09:12:47Z level=debug event=want client=9 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=04971worker2 # [ 57.822869] nix-grpc-daemon[786]: ts=2026-09-21T09:12:47Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14972worker2 # [ 57.861949] nix-grpc-daemon[786]: ts=2026-09-21T09:12:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450084973worker2 # [ 57.874118] nix-grpc-daemon[786]: ts=2026-09-21T09:12:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:450084974client # copying 2 paths to 'lb:50051'...4975client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...4976worker2 # [ 57.890969] nix-grpc-daemon[786]: ts=2026-09-21T09:12:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:450084977client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...4978worker2 # [ 57.902843] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4979worker2 # [ 58.106431] nix-daemon[869]: accepted connection from pid 1007, user nix-grpc-daemon (trusted)4980worker2 # [ 58.119555] nix-daemon[869]: reaped child process 1010, status = succeeded4981lb # [ 58.126924] envoy[918]: conn peer=[2001:db8:1::1]:49940 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2955 tx=1355 ms=46494982lb # [ 58.131677] envoy[918]: conn peer=[2001:db8:1::1]:49956 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1440 tx=1085 ms=35574983lb # [ 58.136208] envoy[918]: conn peer=[2001:db8:1::1]:49966 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=204984lb # [ 58.139638] envoy[918]: conn peer=[2001:db8:1::1]:49962 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3414 tx=1773 ms=17494985lb # [ 58.143217] envoy[918]: conn peer=[2001:db8:1::1]:59504 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=554986lb # [ 58.147094] envoy[918]: conn peer=[2001:db8:1::1]:49978 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3850 tx=1833 ms=14254987lb # [ 58.151317] envoy[918]: conn peer=[2001:db8:1::1]:59522 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=594988lb # [ 58.155184] envoy[918]: conn peer=[2001:db8:1::1]:59510 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4328 tx=2197 ms=17854989worker2 # [ 58.290284] nix-daemon[869]: accepted connection from pid 1012, user nix-grpc-daemon (trusted)4990worker2 # [ 58.302260] nix-daemon[869]: reaped child process 1015, status = succeeded4991lb # [ 58.305335] niks3-server[1003]: 2026/09/21 09:12:48 INFO Received uploads request method=POST path=/api/pending_closures4992lb # [ 58.314148] niks3-server[1003]: 2026/09/21 09:12:48 INFO Received uploads request method=POST path=/api/pending_closures4993worker2 # [ 58.336053] nix-grpc-daemon[921]: time=2026-09-21T09:12:47.546Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4994worker2 # [ 58.337537] nix-grpc-daemon[921]: time=2026-09-21T09:12:47.548Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"4995worker2 # [ 58.339944] nix-grpc-daemon[921]: time=2026-09-21T09:12:47.549Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"4996lb # [ 58.348742] niks3-server[1003]: 2026/09/21 09:12:48 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst4997lb # [ 58.360587] niks3-server[1003]: 2026/09/21 09:12:48 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst4998lb # [ 58.367913] niks3-server[1003]: 2026/09/21 09:12:48 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls4999lb # [ 58.374880] niks3-server[1003]: 2026/09/21 09:12:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5000lb # [ 58.377415] niks3-server[1003]: 2026/09/21 09:12:48 INFO Signed narinfos id=18 count=15001lb # [ 58.379044] niks3-server[1003]: 2026/09/21 09:12:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5002lb # [ 58.380879] niks3-server[1003]: 2026/09/21 09:12:48 INFO Signed narinfos id=19 count=25003worker2 # [ 58.395758] nix-grpc-daemon[921]: time=2026-09-21T09:12:47.605Z level=INFO msg="Uploading 3 narinfos"5004lb # [ 58.384237] niks3-server[1003]: 2026/09/21 09:12:48 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5005lb # [ 58.393411] niks3-server[1003]: 2026/09/21 09:12:48 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5006lb # [ 58.398682] niks3-server[1003]: 2026/09/21 09:12:48 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5007lb # [ 58.401750] niks3-server[1003]: 2026/09/21 09:12:48 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5008lb # [ 58.404755] niks3-server[1003]: 2026/09/21 09:12:48 INFO Completed upload id=185009lb # [ 58.406356] niks3-server[1003]: 2026/09/21 09:12:48 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5010lb # [ 58.408718] niks3-server[1003]: 2026/09/21 09:12:48 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5011lb # [ 58.410962] niks3-server[1003]: 2026/09/21 09:12:48 INFO Completed upload id=195012worker2 # [ 58.425128] nix-grpc-daemon[921]: time=2026-09-21T09:12:47.635Z level=INFO msg="Upload complete. (505ms)"5013worker2 # [ 58.429437] nix-grpc-daemon[786]: ts=2026-09-21T09:12:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45008 duration_s=0 paths=2 nar_bytes_in=9765014worker2 # [ 58.435529] nix-grpc-daemon[786]: ts=2026-09-21T09:12:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450085015worker2 # [ 58.447129] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5016worker2 # [ 58.456226] nix-daemon[869]: reaped child process 1004, status = succeeded5017worker2 # [ 58.466625] nix-grpc-daemon[786]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5018client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5019worker2 # [ 58.477998] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5020client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5021worker2 # [ 59.091162] nix-daemon[869]: reaped child process 1020, status = succeeded5022worker2 # [ 59.247256] nix-daemon[869]: accepted connection from pid 1035, user nix-grpc-daemon (trusted)5023worker2 # [ 59.258690] nix-daemon[869]: reaped child process 1038, status = succeeded5024lb # [ 59.262981] niks3-server[1003]: 2026/09/21 09:12:49 INFO Received uploads request method=POST path=/api/pending_closures5025worker2 # [ 59.285025] nix-grpc-daemon[921]: time=2026-09-21T09:12:48.495Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5026lb # [ 59.295057] niks3-server[1003]: 2026/09/21 09:12:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5027worker2 # [ 59.314418] nix-grpc-daemon[921]: time=2026-09-21T09:12:48.524Z level=INFO msg="Uploading 1 narinfos"5028lb # [ 59.300792] niks3-server[1003]: 2026/09/21 09:12:49 INFO Signed narinfos id=20 count=15029lb # [ 59.306595] niks3-server[1003]: 2026/09/21 09:12:49 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5030lb # [ 59.319068] niks3-server[1003]: 2026/09/21 09:12:49 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5031lb # [ 59.326384] niks3-server[1003]: 2026/09/21 09:12:49 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5032lb # [ 59.332194] niks3-server[1003]: 2026/09/21 09:12:49 INFO Completed upload id=205033worker2 # [ 59.348693] nix-grpc-daemon[921]: time=2026-09-21T09:12:48.558Z level=INFO msg="Upload complete. (263ms)"5034lb # [ 59.335082] niks3-server[1003]: 2026/09/21 09:12:49 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5035worker2 # [ 59.354282] nix-grpc-daemon[786]: ts=2026-09-21T09:12:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45008 duration_s=0 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=9 outputs=15036worker2 # [ 59.364189] nix-daemon[869]: reaped child process 1017, status = succeeded5037client # copying 1 paths...5038worker1 # [ 59.420792] nix-grpc-daemon[785]: ts=2026-09-21T09:12:48Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:393385039worker1 # [ 59.437334] nix-grpc-daemon[785]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5040client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5041worker2 # [ 59.456157] nix-grpc-daemon[786]: ts=2026-09-21T09:12:48Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:450085042worker2 # [ 59.460518] nix-grpc-daemon[786]: ts=2026-09-21T09:12:48Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:45008 duration_s=0 paths=1 nar_bytes_out=1205043client # [ 59.513351] nix-daemon[953]: reaped child process 1039, status = succeeded5044client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.39 seconds)5045client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25046client # [ 59.689090] nix-daemon[953]: accepted connection from pid 1058, user root (trusted)5047client # this derivation will be built:5048client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5049worker2 # [ 59.907810] nix-grpc-daemon[786]: ts=2026-09-21T09:12:49Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:450085050client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5051worker2 # [ 59.913501] nix-grpc-daemon[786]: ts=2026-09-21T09:12:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:450085052worker2 # [ 59.918530] nix-grpc-daemon[921]: time=2026-09-21T09:12:49.128Z level=INFO msg="All 1 paths already cached"5053worker2 # [ 59.929164] nix-grpc-daemon[786]: ts=2026-09-21T09:12:49Z level=debug event=want client=10 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05054worker2 # [ 59.933427] nix-grpc-daemon[786]: ts=2026-09-21T09:12:49Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15055worker1 # [ 59.994856] nix-grpc-daemon[785]: ts=2026-09-21T09:12:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393385056worker1 # [ 60.007185] nix-grpc-daemon[785]: ts=2026-09-21T09:12:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393385057client # copying 1 paths to 'lb:50051'...5058client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5059worker1 # [ 60.025306] nix-grpc-daemon[785]: ts=2026-09-21T09:12:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393385060worker1 # [ 60.033951] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5061worker1 # [ 60.052270] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 85 ms (attempt 1/5)5062worker1 # [ 60.143226] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 25 ms (attempt 2/5)5063worker1 # [ 60.174071] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 177 ms (attempt 3/5)5064worker1 # [ 60.357079] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 756 ms (attempt 4/5)5065worker1 # [ 61.119915] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5066worker1 # [ 61.125907] nix-grpc-daemon[785]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5067worker1 # [ 61.331387] nix-daemon[848]: accepted connection from pid 1162, user nix-grpc-daemon (trusted)5068worker1 # [ 61.344225] nix-daemon[848]: reaped child process 1165, status = succeeded5069worker1 # [ 61.511178] nix-daemon[848]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5070worker1 # [ 61.523086] nix-daemon[848]: reaped child process 1170, status = succeeded5071lb # [ 61.505980] niks3-server[1003]: 2026/09/21 09:12:51 INFO Received uploads request method=POST path=/api/pending_closures5072worker1 # [ 61.549504] nix-grpc-daemon[858]: time=2026-09-21T09:12:50.799Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5073worker1 # [ 61.551246] nix-grpc-daemon[858]: time=2026-09-21T09:12:50.801Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5074lb # [ 61.535774] niks3-server[1003]: 2026/09/21 09:12:51 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5075lb # [ 61.548170] niks3-server[1003]: 2026/09/21 09:12:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5076worker1 # [ 61.589882] nix-grpc-daemon[858]: time=2026-09-21T09:12:50.839Z level=INFO msg="Uploading 1 narinfos"5077lb # [ 61.554967] niks3-server[1003]: 2026/09/21 09:12:51 INFO Signed narinfos id=21 count=15078lb # [ 61.559373] niks3-server[1003]: 2026/09/21 09:12:51 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5079lb # [ 61.570562] niks3-server[1003]: 2026/09/21 09:12:51 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5080lb # [ 61.576711] niks3-server[1003]: 2026/09/21 09:12:51 INFO Completed upload id=215081worker1 # [ 61.613640] nix-grpc-daemon[858]: time=2026-09-21T09:12:50.863Z level=INFO msg="Upload complete. (468ms)"5082lb # [ 61.579614] niks3-server[1003]: 2026/09/21 09:12:51 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5083worker1 # [ 61.616760] nix-grpc-daemon[785]: ts=2026-09-21T09:12:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=1 paths=1 nar_bytes_in=5605084worker1 # [ 61.622268] nix-grpc-daemon[785]: ts=2026-09-21T09:12:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393385085worker1 # [ 61.631206] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5086worker1 # [ 61.640235] nix-daemon[848]: reaped child process 1156, status = succeeded5087worker1 # [ 61.644171] nix-daemon[848]: accepted connection from pid 785, user nix-grpc-daemon (trusted)5088client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5089client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5090worker1 # [ 63.068639] nix-daemon[848]: reaped child process 1174, status = succeeded5091worker1 # [ 63.226260] nix-daemon[848]: accepted connection from pid 1189, user nix-grpc-daemon (trusted)5092worker1 # [ 63.236622] nix-daemon[848]: reaped child process 1192, status = succeeded5093lb # [ 63.218208] niks3-server[1003]: 2026/09/21 09:12:53 INFO Received uploads request method=POST path=/api/pending_closures5094worker1 # [ 63.257740] nix-grpc-daemon[858]: time=2026-09-21T09:12:52.507Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5095lb # [ 63.234574] niks3-server[1003]: 2026/09/21 09:12:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5096lb # [ 63.238040] niks3-server[1003]: 2026/09/21 09:12:53 INFO Signed narinfos id=22 count=15097worker1 # [ 63.273039] nix-grpc-daemon[858]: time=2026-09-21T09:12:52.523Z level=INFO msg="Uploading 1 narinfos"5098lb # [ 63.239443] niks3-server[1003]: 2026/09/21 09:12:53 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5099lb # [ 63.246921] niks3-server[1003]: 2026/09/21 09:12:53 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5100lb # [ 63.250251] niks3-server[1003]: 2026/09/21 09:12:53 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5101lb # [ 63.253163] niks3-server[1003]: 2026/09/21 09:12:53 INFO Completed upload id=225102worker1 # [ 63.288112] nix-grpc-daemon[858]: time=2026-09-21T09:12:52.538Z level=INFO msg="Upload complete. (237ms)"5103lb # [ 63.255752] niks3-server[1003]: 2026/09/21 09:12:53 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5104worker1 # [ 63.290350] nix-grpc-daemon[785]: ts=2026-09-21T09:12:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=10 outputs=15105worker1 # [ 63.296645] nix-daemon[848]: reaped child process 1172, status = succeeded5106client # copying 1 paths...5107worker2 # [ 63.316946] nix-grpc-daemon[786]: ts=2026-09-21T09:12:52Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:450085108worker2 # [ 63.326181] nix-grpc-daemon[786]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5109client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5110worker1 # [ 63.375322] nix-grpc-daemon[785]: ts=2026-09-21T09:12:52Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:393385111worker1 # [ 63.379281] nix-grpc-daemon[785]: ts=2026-09-21T09:12:52Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 paths=1 nar_bytes_out=1205112client # [ 63.409424] nix-daemon[953]: reaped child process 1061, status = succeeded5113client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.90 seconds)5114client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25115client # [ 63.593169] nix-daemon[953]: accepted connection from pid 1080, user root (trusted)5116client # this derivation will be built:5117client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5118worker2 # [ 63.806629] nix-grpc-daemon[786]: ts=2026-09-21T09:12:53Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:450085119client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5120worker2 # [ 63.817335] nix-grpc-daemon[786]: ts=2026-09-21T09:12:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:450085121worker2 # [ 63.826767] nix-grpc-daemon[921]: time=2026-09-21T09:12:53.036Z level=INFO msg="All 1 paths already cached"5122worker2 # [ 63.841806] nix-grpc-daemon[786]: ts=2026-09-21T09:12:53Z level=debug event=want client=11 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05123worker2 # [ 63.847749] nix-grpc-daemon[786]: ts=2026-09-21T09:12:53Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15124worker2 # [ 63.886824] nix-grpc-daemon[786]: ts=2026-09-21T09:12:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450085125worker2 # [ 63.899682] nix-grpc-daemon[786]: ts=2026-09-21T09:12:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:450085126worker2 # [ 63.912145] nix-grpc-daemon[921]: time=2026-09-21T09:12:53.122Z level=INFO msg="All 1 paths already cached"5127client # copying 1 paths to 'lb:50051'...5128client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5129worker2 # [ 63.920198] nix-grpc-daemon[786]: ts=2026-09-21T09:12:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:450085130worker2 # [ 63.931286] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5131worker2 # [ 64.107036] nix-daemon[869]: accepted connection from pid 1047, user nix-grpc-daemon (trusted)5132worker2 # [ 64.119514] nix-daemon[869]: reaped child process 1050, status = succeeded5133worker2 # [ 64.283755] nix-daemon[869]: accepted connection from pid 1052, user nix-grpc-daemon (trusted)5134worker2 # [ 64.295908] nix-daemon[869]: reaped child process 1055, status = succeeded5135lb # [ 64.299624] niks3-server[1003]: 2026/09/21 09:12:54 INFO Received uploads request method=POST path=/api/pending_closures5136worker2 # [ 64.322268] nix-grpc-daemon[921]: time=2026-09-21T09:12:53.532Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5137worker2 # [ 64.323670] nix-grpc-daemon[921]: time=2026-09-21T09:12:53.532Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5138lb # [ 64.326181] niks3-server[1003]: 2026/09/21 09:12:54 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5139lb # [ 64.339296] niks3-server[1003]: 2026/09/21 09:12:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5140lb # [ 64.346240] niks3-server[1003]: 2026/09/21 09:12:54 INFO Signed narinfos id=23 count=15141worker2 # [ 64.363475] nix-grpc-daemon[921]: time=2026-09-21T09:12:53.573Z level=INFO msg="Uploading 1 narinfos"5142lb # [ 64.353984] niks3-server[1003]: 2026/09/21 09:12:54 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5143lb # [ 64.365637] niks3-server[1003]: 2026/09/21 09:12:54 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5144lb # [ 64.372733] niks3-server[1003]: 2026/09/21 09:12:54 INFO Completed upload id=235145worker2 # [ 64.386763] nix-grpc-daemon[921]: time=2026-09-21T09:12:53.597Z level=INFO msg="Upload complete. (443ms)"5146lb # [ 64.375531] niks3-server[1003]: 2026/09/21 09:12:54 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5147worker2 # [ 64.392191] nix-grpc-daemon[786]: ts=2026-09-21T09:12:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45008 duration_s=0 paths=1 nar_bytes_in=5605148worker2 # [ 64.397661] nix-grpc-daemon[786]: ts=2026-09-21T09:12:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450085149worker2 # [ 64.408103] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5150worker2 # [ 64.416412] nix-daemon[869]: reaped child process 1045, status = succeeded5151worker2 # [ 64.420827] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5152client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5153client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5154worker2 # [ 65.835159] nix-daemon[869]: reaped child process 1059, status = succeeded5155worker2 # [ 65.994470] nix-daemon[869]: accepted connection from pid 1074, user nix-grpc-daemon (trusted)5156worker2 # [ 66.004868] nix-daemon[869]: reaped child process 1077, status = succeeded5157lb # [ 66.009441] niks3-server[1003]: 2026/09/21 09:12:56 INFO Received uploads request method=POST path=/api/pending_closures5158worker2 # [ 66.032651] nix-grpc-daemon[921]: time=2026-09-21T09:12:55.242Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5159lb # [ 66.043862] niks3-server[1003]: 2026/09/21 09:12:56 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5160lb # [ 66.047837] niks3-server[1003]: 2026/09/21 09:12:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5161lb # [ 66.053234] niks3-server[1003]: 2026/09/21 09:12:56 INFO Signed narinfos id=24 count=15162worker2 # [ 66.069938] nix-grpc-daemon[921]: time=2026-09-21T09:12:55.280Z level=INFO msg="Uploading 1 narinfos"5163lb # [ 66.062391] niks3-server[1003]: 2026/09/21 09:12:56 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5164lb # [ 66.073191] niks3-server[1003]: 2026/09/21 09:12:56 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5165lb # [ 66.083888] niks3-server[1003]: 2026/09/21 09:12:56 INFO Completed upload id=245166worker2 # [ 66.097701] nix-grpc-daemon[921]: time=2026-09-21T09:12:55.307Z level=INFO msg="Upload complete. (269ms)"5167lb # [ 66.086900] niks3-server[1003]: 2026/09/21 09:12:56 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5168worker2 # [ 66.103592] nix-grpc-daemon[786]: ts=2026-09-21T09:12:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45008 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=11 outputs=15169worker2 # [ 66.117067] nix-daemon[869]: reaped child process 1057, status = succeeded5170client # copying 1 paths...5171worker1 # [ 66.172520] nix-grpc-daemon[785]: ts=2026-09-21T09:12:55Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:393385172worker1 # [ 66.184350] nix-grpc-daemon[785]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5173client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5174worker1 # [ 66.217727] nix-grpc-daemon[785]: ts=2026-09-21T09:12:55Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:393385175worker1 # [ 66.222453] nix-grpc-daemon[785]: ts=2026-09-21T09:12:55Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:39338 duration_s=0 paths=1 nar_bytes_out=1205176client # [ 66.294202] nix-daemon[953]: reaped child process 1083, status = succeeded5177client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.89 seconds)5178(finished: subtest: an input only one worker has still reaches the builder, in 10.06 seconds)5179subtest: upload whose reference the worker lacks is completed from the cache5180client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5181client # this derivation will be built:5182client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5183client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5184client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.42 seconds)5185client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5186client # this derivation will be built:5187client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5188client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5189client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.40 seconds)5190client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5191client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.19 seconds)5192worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5193worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.21 seconds)5194worker2 # [ 67.600649] systemd[1]: Stopping Nix worker-protocol over gRPC...5195worker2 # [ 67.608835] nix-grpc-daemon[786]: ts=2026-09-21T09:12:56Z level=info event=signal_exit5196worker2 # [ 67.857421] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=scheduler_yield5197lb # [ 67.847523] niks3-server[1003]: 2026/09/21 09:12:57 INFO lead: released remote=192.168.1.4:523605198worker2 # [ 67.865525] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=scheduler_restarting addr=lb:500515199worker2 # [ 67.870135] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:449925200worker1 # [ 67.892158] nix-grpc-daemon[785]: ts=2026-09-21T09:12:57Z level=info event=scheduler_restarting addr=lb:500515201worker2 # [ 67.874951] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=worker_gone addr=192.168.1.4:500515202worker2 # [ 67.878479] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:449925203worker2 # [ 67.881671] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=worker_gone addr=192.168.1.3:500515204worker2 # [ 68.067176] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=scheduler_connected addr=lb:500515205worker2 # [ 68.070884] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5206worker1 # [ 68.092487] nix-grpc-daemon[785]: ts=2026-09-21T09:12:57Z level=info event=scheduler_connected addr=lb:500515207worker1 # [ 68.096342] nix-grpc-daemon[785]: ts=2026-09-21T09:12:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5208lb # [ 68.127729] envoy[918]: conn peer=[2001:db8:1::1]:59540 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=175209lb # [ 68.131369] envoy[918]: conn peer=[2001:db8:1::1]:59536 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3854 tx=1878 ms=19535210lb # [ 68.135001] envoy[918]: conn peer=[2001:db8:1::1]:59542 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3715 tx=1916 ms=34615211lb # [ 68.138501] envoy[918]: conn peer=[2001:db8:1::1]:48016 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=145212lb # [ 68.141974] envoy[918]: conn peer=[2001:db8:1::1]:48036 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=585213lb # [ 68.145406] envoy[918]: conn peer=[2001:db8:1::1]:48022 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3717 tx=1919 ms=24475214worker2 # [ 68.272090] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=scheduler_connected addr=lb:500515215worker1 # [ 68.297262] nix-grpc-daemon[785]: ts=2026-09-21T09:12:57Z level=info event=scheduler_connected addr=lb:500515216worker2 # [ 68.275943] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5217worker1 # [ 68.300981] nix-grpc-daemon[785]: ts=2026-09-21T09:12:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5218worker2 # [ 68.477132] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=scheduler_connected addr=lb:500515219worker1 # [ 68.501757] nix-grpc-daemon[785]: ts=2026-09-21T09:12:57Z level=info event=scheduler_connected addr=lb:500515220worker2 # [ 68.480916] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5221worker1 # [ 68.505531] nix-grpc-daemon[785]: ts=2026-09-21T09:12:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5222worker2 # [ 68.682448] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=scheduler_connected addr=lb:500515223worker1 # [ 68.706282] nix-grpc-daemon[785]: ts=2026-09-21T09:12:57Z level=info event=scheduler_connected addr=lb:500515224worker2 # [ 68.686152] nix-grpc-daemon[786]: ts=2026-09-21T09:12:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5225worker1 # [ 68.710105] nix-grpc-daemon[785]: ts=2026-09-21T09:12:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5226worker2 # [ 68.861540] nix-grpc-daemon[786]: ts=2026-09-21T09:12:58Z level=info event=scheduler_restarting5227worker1 # [ 68.912202] nix-grpc-daemon[785]: ts=2026-09-21T09:12:58Z level=info event=scheduler_connected addr=lb:500515228worker2 # [ 68.895655] nix-daemon[869]: reaped child process 871, status = succeeded5229worker1 # [ 68.915336] nix-grpc-daemon[785]: ts=2026-09-21T09:12:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5230worker2 # [ 68.902483] nix-grpc-daemon[786]: ts=2026-09-21T09:12:58Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5231worker1 # [ 69.113263] nix-grpc-daemon[785]: ts=2026-09-21T09:12:58Z level=info event=scheduler_connected addr=lb:500515232worker1 # [ 69.117130] nix-grpc-daemon[785]: ts=2026-09-21T09:12:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5233worker1 # [ 69.317858] nix-grpc-daemon[785]: ts=2026-09-21T09:12:58Z level=info event=scheduler_connected addr=lb:500515234worker1 # [ 69.321782] nix-grpc-daemon[785]: ts=2026-09-21T09:12:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5235worker1 # [ 69.522534] nix-grpc-daemon[785]: ts=2026-09-21T09:12:58Z level=info event=scheduler_connected addr=lb:500515236worker1 # [ 69.526402] nix-grpc-daemon[785]: ts=2026-09-21T09:12:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5237worker1 # [ 69.727120] nix-grpc-daemon[785]: ts=2026-09-21T09:12:58Z level=info event=scheduler_connected addr=lb:500515238worker1 # [ 69.730886] nix-grpc-daemon[785]: ts=2026-09-21T09:12:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5239worker2 # [ 69.721140] nix-grpc-daemon[786]: E0000 00:00:1789981978.930651 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5240worker2 # [ 69.726077] nix-grpc-daemon[786]: E0000 00:00:1789981978.930769 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5241lb # [ 69.740518] niks3-server[1003]: 2026/09/21 09:12:59 INFO lead: acquired remote=192.168.1.3:446265242worker1 # [ 69.778752] nix-grpc-daemon[785]: ts=2026-09-21T09:12:59Z level=info event=scheduler_take_over5243worker1 # [ 69.932387] nix-grpc-daemon[785]: ts=2026-09-21T09:12:59Z level=info event=scheduler_connected addr=lb:500515244worker1 # [ 69.936206] nix-grpc-daemon[785]: ts=2026-09-21T09:12:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5245worker2 # [ 70.100524] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5246worker2 # [ 70.104285] systemd[1]: Stopped Nix worker-protocol over gRPC.5247worker2 # [ 70.108255] systemd[1]: nix-grpc-daemon.service: Consumed 1.368s CPU time over 51.819s wall clock time, 52.6M memory peak, 53.5K incoming IP traffic, 148.8K outgoing IP traffic.5248worker1 # [ 70.137182] nix-grpc-daemon[785]: ts=2026-09-21T09:12:59Z level=info event=scheduler_connected addr=lb:500515249worker2 # [ 70.119064] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5250worker1 # [ 70.140996] nix-grpc-daemon[785]: ts=2026-09-21T09:12:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5251worker2 # [ 70.123356] systemd[1]: Closed Nix worker-protocol over gRPC.5252worker2 # [ 70.126649] systemd[1]: nix-grpc-daemon.socket: Consumed 283us CPU time over 1min 871ms wall clock time, 256K memory peak, 71K incoming IP traffic, 61.1K outgoing IP traffic.5253lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5254lb: must succeed: curl -sf localhost:9901/clusters5255lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5256lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.07 seconds)5257client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25258worker1 # [ 70.341822] nix-grpc-daemon[785]: ts=2026-09-21T09:12:59Z level=info event=scheduler_connected addr=lb:500515259worker1 # [ 70.345674] nix-grpc-daemon[785]: ts=2026-09-21T09:12:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5260worker1 # [ 70.473193] nix-grpc-daemon[785]: ts=2026-09-21T09:12:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:393385261worker1 # [ 70.479961] nix-grpc-daemon[785]: ts=2026-09-21T09:12:59Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:393385262client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5263client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.28 seconds)5264worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5265worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.03 seconds)5266worker1 # [ 70.546429] nix-grpc-daemon[785]: ts=2026-09-21T09:12:59Z level=info event=scheduler_connected addr=lb:500515267worker1 # [ 70.550270] nix-grpc-daemon[785]: ts=2026-09-21T09:12:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5268worker2 # [ 70.577241] systemd[1]: Listening on Nix worker-protocol over gRPC.5269worker1 # [ 70.658742] systemd[1]: Stopping Nix worker-protocol over gRPC...5270worker1 # [ 70.750971] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=scheduler_connected addr=lb:500515271worker1 # [ 70.763164] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:435945272worker1 # [ 70.768135] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05273worker2 # [ 71.087708] systemd[1]: Starting scheduler-token.service...5274worker2 # [ 71.194662] systemd[1]: scheduler-token.service: Deactivated successfully.5275worker2 # [ 71.196817] systemd[1]: Finished scheduler-token.service.5276worker2 # [ 71.198226] systemd[1]: scheduler-token.service: Consumed 36ms CPU time over 109ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5277worker2 # [ 71.203513] systemd[1]: Starting Nix worker-protocol over gRPC...5278worker2 # [ 71.388892] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:00Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5279worker2 # [ 71.391381] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:00Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5280worker2 # [ 71.395061] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:00Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15281worker2 # [ 71.403067] nix-grpc-daemon[1101]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5282worker2 # [ 71.404582] nix-grpc-daemon[1101]: E0000 00:00:1789981980.612668 1101 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5283worker2 # [ 71.410068] nix-grpc-daemon[1101]: E0000 00:00:1789981980.617950 1101 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5284worker2 # [ 71.412222] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:00Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5285worker2 # [ 71.415976] systemd[1]: Started Nix worker-protocol over gRPC.5286worker2 # [ 71.423277] nix-grpc-daemon[1101]: E0000 00:00:1789981980.633702 1142 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5287worker1 # [ 71.450149] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=signal_exit5288worker2 # [ 71.435860] nix-grpc-daemon[1101]: E0000 00:00:1789981980.646325 1142 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5289worker2 # [ 71.440123] nix-grpc-daemon[1101]: E0000 00:00:1789981980.648659 1142 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5290worker2 # [ 71.450453] nix-grpc-daemon[1101]: E0000 00:00:1789981980.660897 1142 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5291worker2 # [ 71.459658] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:00Z level=info event=scheduler_connected addr=lb:500515292worker1 # [ 71.482589] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:435945293worker1 # [ 71.487401] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05294worker1 # [ 71.719067] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=scheduler_yield5295worker2 # [ 71.700692] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:00Z level=info event=scheduler_restarting addr=lb:500515296lb # [ 71.686977] niks3-server[1003]: 2026/09/21 09:13:01 INFO lead: released remote=192.168.1.3:446265297worker1 # [ 71.723735] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:435945298worker1 # [ 71.727952] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=worker_gone addr=192.168.1.4:500515299worker1 # [ 71.732433] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=scheduler_restarting addr=lb:500515300worker1 # [ 71.735356] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:435945301worker1 # [ 71.738606] nix-grpc-daemon[785]: ts=2026-09-21T09:13:00Z level=info event=worker_gone addr=192.168.1.3:500515302worker2 # [ 71.902170] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515303worker2 # [ 71.906327] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5304worker1 # [ 71.928885] nix-grpc-daemon[785]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515305worker1 # [ 71.932816] nix-grpc-daemon[785]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5306worker2 # [ 72.107421] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515307worker2 # [ 72.111306] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5308worker1 # [ 72.133754] nix-grpc-daemon[785]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515309worker1 # [ 72.137484] nix-grpc-daemon[785]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5310worker2 # [ 72.312441] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515311worker2 # [ 72.316318] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5312worker1 # [ 72.338409] nix-grpc-daemon[785]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515313worker1 # [ 72.342872] nix-grpc-daemon[785]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5314worker2 # [ 72.518124] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515315worker2 # [ 72.521853] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5316worker1 # [ 72.544906] nix-grpc-daemon[785]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515317worker1 # [ 72.548132] nix-grpc-daemon[785]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5318worker1 # [ 72.720630] nix-grpc-daemon[785]: ts=2026-09-21T09:13:01Z level=info event=scheduler_restarting5319worker2 # [ 72.723057] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515320worker2 # [ 72.727152] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5321worker1 # [ 72.749691] nix-daemon[848]: reaped child process 850, status = succeeded5322worker1 # [ 72.762258] nix-grpc-daemon[785]: ts=2026-09-21T09:13:02Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5323worker2 # [ 72.927971] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515324worker2 # [ 72.931914] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5325worker2 # [ 73.132932] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515326worker2 # [ 73.136860] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5327worker2 # [ 73.338163] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515328worker2 # [ 73.342119] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5329worker2 # [ 73.543981] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515330worker2 # [ 73.547787] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5331worker1 # [ 73.582685] nix-grpc-daemon[785]: E0000 00:00:1789981982.831968 830 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5332worker1 # [ 73.587524] nix-grpc-daemon[785]: E0000 00:00:1789981982.832089 830 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5333worker2 # [ 73.749209] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515334worker2 # [ 73.753468] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5335worker2 # [ 73.954688] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515336worker2 # [ 73.958766] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5337worker1 # [ 74.061519] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5338worker1 # [ 74.065422] systemd[1]: Stopped Nix worker-protocol over gRPC.5339worker1 # [ 74.068944] systemd[1]: nix-grpc-daemon.service: Consumed 2.653s CPU time over 55.907s wall clock time, 61.5M memory peak, 85.1K incoming IP traffic, 159.7K outgoing IP traffic.5340worker1 # [ 74.080858] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5341worker1 # [ 74.084540] systemd[1]: Closed Nix worker-protocol over gRPC.5342worker1 # [ 74.087245] systemd[1]: nix-grpc-daemon.socket: Consumed 288us CPU time over 1min 3.895s wall clock time, 256K memory peak, 82.1K incoming IP traffic, 67.2K outgoing IP traffic.5343lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5344lb: must succeed: curl -sf localhost:9901/clusters5345lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5346lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.08 seconds)5347client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25348worker2 # [ 74.160080] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515349worker2 # [ 74.163901] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5350worker2 # [ 74.365130] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515351worker2 # [ 74.369147] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5352worker2 # [ 74.398137] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:537125353worker2 # [ 74.407926] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5354worker2 # [ 74.428156] nix-grpc-daemon[1101]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 58 ms (attempt 1/5)5355worker2 # [ 74.494379] nix-grpc-daemon[1101]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 118 ms (attempt 2/5)5356worker2 # [ 74.570938] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515357worker2 # [ 74.574684] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5358worker2 # [ 74.612649] nix-grpc-daemon[1101]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 21 ms (attempt 3/5)5359worker2 # [ 74.639231] nix-grpc-daemon[1101]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 681 ms (attempt 4/5)5360worker2 # [ 74.776073] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515361worker2 # [ 74.780115] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5362worker2 # [ 74.981220] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:04Z level=info event=scheduler_connected addr=lb:500515363worker2 # [ 74.985214] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5364worker2 # [ 75.186450] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:04Z level=info event=scheduler_connected addr=lb:500515365worker2 # [ 75.190333] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5366worker2 # [ 75.326390] nix-grpc-daemon[1101]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5367client # copying 2 paths...5368client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5369worker2 # [ 75.349826] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:537125370client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5371worker2 # [ 75.361168] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5372worker2 # [ 75.383551] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:04Z level=info event=niks3_push_started pid=11605373worker2 # [ 75.391881] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:04Z level=info event=scheduler_connected addr=lb:500515374worker2 # [ 75.396300] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5375worker2 # [ 75.596967] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:04Z level=info event=scheduler_connected addr=lb:500515376worker2 # [ 75.598759] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5377worker2 # [ 75.611697] nix-grpc-daemon[1160]: time=2026-09-21T09:13:04.821Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5378worker2 # [ 75.795522] nix-daemon[869]: accepted connection from pid 1173, user nix-grpc-daemon (trusted)5379worker2 # [ 75.799220] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515380worker2 # [ 75.800887] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5381worker2 # [ 75.810145] nix-daemon[869]: reaped child process 1176, status = succeeded5382lb # [ 75.818265] niks3-server[1003]: 2026/09/21 09:13:05 INFO Received uploads request method=POST path=/api/pending_closures5383lb # [ 75.829380] niks3-server[1003]: 2026/09/21 09:13:05 INFO Received uploads request method=POST path=/api/pending_closures5384worker2 # [ 75.852126] nix-grpc-daemon[1160]: time=2026-09-21T09:13:05.061Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5385worker2 # [ 75.853571] nix-grpc-daemon[1160]: time=2026-09-21T09:13:05.061Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5386worker2 # [ 75.856270] nix-grpc-daemon[1160]: time=2026-09-21T09:13:05.061Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5387lb # [ 75.869990] niks3-server[1003]: 2026/09/21 09:13:05 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5388lb # [ 75.887451] niks3-server[1003]: 2026/09/21 09:13:05 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5389lb # [ 75.898393] niks3-server[1003]: 2026/09/21 09:13:05 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5390lb # [ 75.901185] niks3-server[1003]: 2026/09/21 09:13:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5391lb # [ 75.903785] niks3-server[1003]: 2026/09/21 09:13:05 INFO Signed narinfos id=26 count=25392lb # [ 75.906310] niks3-server[1003]: 2026/09/21 09:13:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5393lb # [ 75.908703] niks3-server[1003]: 2026/09/21 09:13:05 INFO Signed narinfos id=25 count=15394lb # [ 75.910147] niks3-server[1003]: 2026/09/21 09:13:05 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5395worker2 # [ 75.923905] nix-grpc-daemon[1160]: time=2026-09-21T09:13:05.133Z level=INFO msg="Uploading 3 narinfos"5396lb # [ 75.923449] niks3-server[1003]: 2026/09/21 09:13:05 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5397lb # [ 75.926491] niks3-server[1003]: 2026/09/21 09:13:05 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5398lb # [ 75.930714] niks3-server[1003]: 2026/09/21 09:13:05 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5399lb # [ 75.933702] niks3-server[1003]: 2026/09/21 09:13:05 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5400lb # [ 75.942687] niks3-server[1003]: 2026/09/21 09:13:05 INFO Completed upload id=255401lb # [ 75.943939] niks3-server[1003]: 2026/09/21 09:13:05 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5402lb # [ 75.947646] niks3-server[1003]: 2026/09/21 09:13:05 INFO Completed upload id=265403worker2 # [ 75.962122] nix-grpc-daemon[1160]: time=2026-09-21T09:13:05.172Z level=INFO msg="Upload complete. (346ms)"5404worker2 # [ 75.967155] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53712 duration_s=0 paths=2 nar_bytes_in=3125405worker2 # [ 75.979707] nix-daemon[869]: reaped child process 1158, status = succeeded5406worker2 # [ 76.003351] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515407client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.85 seconds)5408worker2 # [ 76.006788] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5409worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5410worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.03 seconds)5411worker1 # [ 76.120687] systemd[1]: Listening on Nix worker-protocol over gRPC.5412worker1 # [ 76.128146] systemd[1]: Starting Nix worker-protocol over gRPC...5413worker2 # [ 76.207873] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515414worker2 # [ 76.211800] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5415worker1 # [ 76.289771] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:05Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5416worker1 # [ 76.291524] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:05Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5417worker1 # [ 76.295772] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:05Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15418worker1 # [ 76.303067] nix-grpc-daemon[1216]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5419worker1 # [ 76.304606] nix-grpc-daemon[1216]: E0000 00:00:1789981985.552389 1216 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5420worker1 # [ 76.308542] nix-grpc-daemon[1216]: E0000 00:00:1789981985.557599 1216 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5421worker1 # [ 76.311489] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:05Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5422worker1 # [ 76.315327] systemd[1]: Started Nix worker-protocol over gRPC.5423worker1 # [ 76.319335] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:05Z level=info event=scheduler_take_over5424lb # [ 76.284314] niks3-server[1003]: 2026/09/21 09:13:06 INFO lead: acquired remote=192.168.1.3:557225425lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5426lb: must succeed: curl -sf localhost:9901/clusters5427worker1 # [ 76.322482] nix-grpc-daemon[1216]: E0000 00:00:1789981985.572657 1258 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5428worker1 # [ 76.334593] nix-grpc-daemon[1216]: E0000 00:00:1789981985.584706 1258 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5429worker1 # [ 76.339052] nix-grpc-daemon[1216]: E0000 00:00:1789981985.587048 1258 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5430worker1 # [ 76.348716] nix-grpc-daemon[1216]: E0000 00:00:1789981985.598848 1258 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5431worker1 # [ 76.358485] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515432worker1 # [ 76.359969] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5433lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5434worker2 # [ 76.413148] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515435worker2 # [ 76.416895] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5436worker1 # [ 76.560909] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515437worker1 # [ 76.564808] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5438worker2 # [ 76.618933] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515439worker2 # [ 76.622701] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5440worker1 # [ 76.765573] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:06Z level=info event=scheduler_connected addr=lb:500515441worker1 # [ 76.769491] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5442worker2 # [ 76.823671] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:06Z level=info event=scheduler_connected addr=lb:500515443worker2 # [ 76.827624] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5444worker1 # [ 76.970357] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:06Z level=info event=scheduler_connected addr=lb:500515445worker1 # [ 76.974295] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5446worker2 # [ 77.028723] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:06Z level=info event=scheduler_connected addr=lb:500515447worker2 # [ 77.032764] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5448worker1 # [ 77.174892] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:06Z level=info event=scheduler_connected addr=lb:500515449worker1 # [ 77.181694] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:06Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:333205450worker1 # [ 77.186350] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:06Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05451worker2 # [ 77.234062] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:06Z level=info event=scheduler_connected addr=lb:500515452worker1 # [ 77.262253] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:06Z level=info event=oidc_jwks provider=mock keys=15453worker1 # [ 77.266131] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:06Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:333205454worker1 # [ 77.269951] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:06Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05455lb: must succeed: curl -sf localhost:9901/clusters5456lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5457lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.15 seconds)5458(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.13 seconds)5459subtest: scheduler down: the other node takes the lock and keeps it5460lb: must succeed: curl -sf localhost:9901/clusters5461lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5462lb: must succeed: curl -sf localhost:9901/clusters5463lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5464lb: must succeed: curl -sf localhost:9901/clusters5465lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5466lb: must succeed: curl -sf localhost:9901/clusters5467lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5468worker1 # [ 77.722973] systemd[1]: Stopping Nix worker-protocol over gRPC...5469lb # [ 78.127904] envoy[918]: conn peer=[2001:db8:1::4]:46944 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=32979 tx=1782 ms=502205470lb # [ 78.131887] envoy[918]: conn peer=[2001:db8:1::1]:48050 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=205471lb # [ 78.135349] envoy[918]: conn peer=192.168.1.3:44802 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=12693 tx=2715 ms=542365472lb # [ 78.139060] envoy[918]: conn peer=[2001:db8:1::1]:49250 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1008 tx=281 ms=15845473worker1 # [ 78.309736] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:07Z level=info event=signal_exit5474worker1 # [ 78.518799] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:07Z level=info event=scheduler_yield5475lb # [ 78.487597] niks3-server[1003]: 2026/09/21 09:13:08 INFO lead: released remote=192.168.1.3:557225476worker2 # [ 78.501390] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:07Z level=info event=scheduler_restarting addr=lb:500515477worker1 # [ 78.523105] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:07Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:333205478worker1 # [ 78.527360] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:07Z level=info event=worker_gone addr=192.168.1.4:500515479worker1 # [ 78.531268] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:07Z level=info event=scheduler_restarting addr=lb:500515480worker1 # [ 78.534265] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:07Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:333205481worker1 # [ 78.537486] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:07Z level=info event=worker_gone addr=192.168.1.3:500515482worker2 # [ 78.701743] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:07Z level=info event=scheduler_connected addr=lb:500515483worker1 # [ 78.727893] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:07Z level=info event=scheduler_connected addr=lb:500515484worker2 # [ 78.705946] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:07Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5485worker1 # [ 78.732542] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:07Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5486worker2 # [ 78.907683] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:08Z level=info event=scheduler_connected addr=lb:500515487worker1 # [ 78.933290] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:08Z level=info event=scheduler_connected addr=lb:500515488worker2 # [ 78.911759] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:08Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5489worker1 # [ 78.937900] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:08Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5490worker2 # [ 79.112973] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:08Z level=info event=scheduler_connected addr=lb:500515491worker2 # [ 79.116990] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5492worker1 # [ 79.138931] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:08Z level=info event=scheduler_connected addr=lb:500515493worker1 # [ 79.142750] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5494worker2 # [ 79.318563] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:08Z level=info event=scheduler_connected addr=lb:500515495worker2 # [ 79.322762] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5496worker1 # [ 79.344529] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:08Z level=info event=scheduler_connected addr=lb:500515497worker1 # [ 79.348319] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5498worker1 # [ 79.523167] nix-grpc-daemon[1216]: ts=2026-09-21T09:13:08Z level=info event=scheduler_restarting5499worker2 # [ 79.525382] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:08Z level=info event=scheduler_connected addr=lb:500515500worker2 # [ 79.528663] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5501worker2 # [ 79.726898] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:08Z level=info event=scheduler_connected addr=lb:500515502worker2 # [ 79.730987] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5503worker2 # [ 79.932514] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:09Z level=info event=scheduler_connected addr=lb:500515504worker2 # [ 79.936668] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5505worker1 # [ 80.060316] nix-grpc-daemon[1216]: E0000 00:00:1789981989.310328 1261 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true5506worker1 # [ 80.065648] nix-grpc-daemon[1216]: E0000 00:00:1789981989.315550 1261 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5507worker2 # [ 80.138180] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:09Z level=info event=scheduler_connected addr=lb:500515508worker2 # [ 80.142248] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5509worker1 # [ 80.327805] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5510worker1 # [ 80.331779] systemd[1]: Stopped Nix worker-protocol over gRPC.5511worker1 # [ 80.335813] systemd[1]: nix-grpc-daemon.service: Consumed 162ms CPU time over 4.199s wall clock time, 12.6M memory peak, 10.2K incoming IP traffic, 10K outgoing IP traffic.5512worker1 # [ 80.345202] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5513worker1 # [ 80.349476] systemd[1]: Closed Nix worker-protocol over gRPC.5514worker1 # [ 80.352968] systemd[1]: nix-grpc-daemon.socket: Consumed 622us CPU time over 4.236s wall clock time, 256K memory peak, 16.6K incoming IP traffic, 11.8K outgoing IP traffic.5515worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5516worker2 # [ 80.343661] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:09Z level=info event=scheduler_connected addr=lb:500515517worker2 # [ 80.348675] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5518worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds)5519worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5520worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5521worker2 # [ 80.549235] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:09Z level=info event=scheduler_connected addr=lb:500515522worker2 # [ 80.552642] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5523worker2 # [ 80.754093] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:09Z level=info event=scheduler_connected addr=lb:500515524worker2 # [ 80.758146] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5525worker2 # [ 80.959541] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515526worker2 # [ 80.963723] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5527worker2 # [ 81.165189] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515528worker2 # [ 81.169323] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5529worker2 # [ 81.370741] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515530worker2 # [ 81.374807] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5531worker2 # [ 81.425522] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:10Z level=info event=scheduler_take_over5532lb # [ 81.411926] niks3-server[1003]: 2026/09/21 09:13:11 INFO lead: acquired remote=192.168.1.4:403745533worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5534worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5535worker2 # [ 81.576423] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515536worker2 # [ 81.595175] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:10Z level=info event=oidc_jwks provider=mock keys=15537worker2 # [ 81.598177] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:10Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:537225538worker2 # [ 81.602414] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:10Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05539worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5540worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5541client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&25542worker2 # [ 82.871925] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:537125543client # don't know how to build these paths:5544client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5545worker2 # [ 82.880819] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:537125546worker2 # [ 82.904698] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:12Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=05547worker2 # [ 82.909987] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:12Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.4:50051 clients=15548worker2 # [ 82.948523] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:537125549worker2 # [ 82.961533] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:537125550client # copying 1 paths to 'lb:50051'...5551client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...5552worker2 # [ 82.974443] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:537125553worker2 # [ 82.986829] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5554worker2 # [ 83.202831] nix-daemon[869]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)5555worker2 # [ 83.217961] nix-daemon[869]: reaped child process 1203, status = succeeded5556worker2 # [ 83.430440] nix-daemon[869]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5557worker2 # [ 83.444609] nix-daemon[869]: reaped child process 1208, status = succeeded5558lb # [ 83.452322] niks3-server[1003]: 2026/09/21 09:13:13 INFO Received uploads request method=POST path=/api/pending_closures5559worker2 # [ 83.474449] nix-grpc-daemon[1160]: time=2026-09-21T09:13:12.684Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5560worker2 # [ 83.476496] nix-grpc-daemon[1160]: time=2026-09-21T09:13:12.686Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"5561lb # [ 83.481907] niks3-server[1003]: 2026/09/21 09:13:13 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst5562lb # [ 83.495241] niks3-server[1003]: 2026/09/21 09:13:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5563lb # [ 83.508145] niks3-server[1003]: 2026/09/21 09:13:13 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls5564lb # [ 83.516701] niks3-server[1003]: 2026/09/21 09:13:13 INFO Signed narinfos id=27 count=15565worker2 # [ 83.530844] nix-grpc-daemon[1160]: time=2026-09-21T09:13:12.741Z level=INFO msg="Uploading 1 narinfos"5566lb # [ 83.525642] niks3-server[1003]: 2026/09/21 09:13:13 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5567lb # [ 83.530337] niks3-server[1003]: 2026/09/21 09:13:13 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo5568lb # [ 83.535653] niks3-server[1003]: 2026/09/21 09:13:13 INFO Completed upload id=275569worker2 # [ 83.549834] nix-grpc-daemon[1160]: time=2026-09-21T09:13:12.760Z level=INFO msg="Upload complete. (550ms)"5570worker2 # [ 83.554429] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53712 duration_s=0 paths=1 nar_bytes_in=5205571worker2 # [ 83.562247] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:537125572worker2 # [ 83.571957] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5573worker2 # [ 83.585509] nix-daemon[869]: reaped child process 1198, status = succeeded5574worker2 # [ 83.589225] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5575client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...5576client # farm-a> worker2: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5577client # farm-a> LOG-farm-a-on-standby5578worker2 # [ 84.605781] nix-daemon[869]: reaped child process 1212, status = succeeded5579worker2 # [ 84.790175] nix-daemon[869]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5580worker2 # [ 84.804770] nix-daemon[869]: reaped child process 1230, status = succeeded5581lb # [ 84.811152] niks3-server[1003]: 2026/09/21 09:13:14 INFO Received uploads request method=POST path=/api/pending_closures5582worker2 # [ 84.833516] nix-grpc-daemon[1160]: time=2026-09-21T09:13:14.043Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5583worker2 # [ 84.835086] nix-grpc-daemon[1160]: time=2026-09-21T09:13:14.045Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"5584lb # [ 84.844874] niks3-server[1003]: 2026/09/21 09:13:14 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5585lb # [ 84.858501] niks3-server[1003]: 2026/09/21 09:13:14 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5586lb # [ 84.867191] niks3-server[1003]: 2026/09/21 09:13:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5587lb # [ 84.873109] niks3-server[1003]: 2026/09/21 09:13:14 INFO Signed narinfos id=28 count=15588worker2 # [ 84.890888] nix-grpc-daemon[1160]: time=2026-09-21T09:13:14.100Z level=INFO msg="Uploading 1 narinfos"5589lb # [ 84.879930] niks3-server[1003]: 2026/09/21 09:13:14 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls5590lb # [ 84.889580] niks3-server[1003]: 2026/09/21 09:13:14 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5591lb # [ 84.895305] niks3-server[1003]: 2026/09/21 09:13:14 INFO Completed upload id=285592lb # [ 84.897500] niks3-server[1003]: 2026/09/21 09:13:14 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo5593worker2 # [ 84.910879] nix-grpc-daemon[1160]: time=2026-09-21T09:13:14.121Z level=INFO msg="Upload complete. (314ms)"5594worker2 # [ 84.921528] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53712 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=15595worker2 # [ 84.928806] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:14Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=05596worker2 # [ 84.932232] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:14Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.4:50051 clients=15597worker2 # [ 84.941444] nix-daemon[869]: reaped child process 1210, status = succeeded5598worker2 # [ 84.973783] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:537125599worker2 # [ 84.986352] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:537125600worker2 # [ 84.999160] nix-grpc-daemon[1160]: time=2026-09-21T09:13:14.209Z level=INFO msg="All 1 paths already cached"5601client # copying 1 paths to 'lb:50051'...5602client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...5603worker2 # [ 85.008067] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:537125604worker2 # [ 85.016690] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5605worker2 # [ 85.223967] nix-daemon[869]: accepted connection from pid 1236, user nix-grpc-daemon (trusted)5606worker2 # [ 85.236897] nix-daemon[869]: reaped child process 1239, status = succeeded5607worker2 # [ 85.439621] nix-daemon[869]: accepted connection from pid 1241, user nix-grpc-daemon (trusted)5608worker2 # [ 85.455060] nix-daemon[869]: reaped child process 1244, status = succeeded5609lb # [ 85.460982] niks3-server[1003]: 2026/09/21 09:13:15 INFO Received uploads request method=POST path=/api/pending_closures5610worker2 # [ 85.483165] nix-grpc-daemon[1160]: time=2026-09-21T09:13:14.693Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5611worker2 # [ 85.484814] nix-grpc-daemon[1160]: time=2026-09-21T09:13:14.695Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"5612lb # [ 85.492947] niks3-server[1003]: 2026/09/21 09:13:15 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst5613lb # [ 85.502237] niks3-server[1003]: 2026/09/21 09:13:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5614lb # [ 85.507400] niks3-server[1003]: 2026/09/21 09:13:15 INFO Signed narinfos id=29 count=15615worker2 # [ 85.524516] nix-grpc-daemon[1160]: time=2026-09-21T09:13:14.734Z level=INFO msg="Uploading 1 narinfos"5616lb # [ 85.514361] niks3-server[1003]: 2026/09/21 09:13:15 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls5617lb # [ 85.524626] niks3-server[1003]: 2026/09/21 09:13:15 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5618lb # [ 85.529601] niks3-server[1003]: 2026/09/21 09:13:15 INFO Completed upload id=295619worker2 # [ 85.543431] nix-grpc-daemon[1160]: time=2026-09-21T09:13:14.753Z level=INFO msg="Upload complete. (512ms)"5620worker2 # [ 85.548225] nix-grpc-daemon[1160]: time=2026-09-21T09:13:14.756Z level=WARN msg="Failed to register uploaded object" key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo error="server returned 404: object is not pending upload\n"5621worker2 # [ 85.554515] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53712 duration_s=0 paths=1 nar_bytes_in=6405622worker2 # [ 85.558308] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:537125623worker2 # [ 85.566562] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5624worker2 # [ 85.572437] nix-daemon[869]: reaped child process 1234, status = succeeded5625worker2 # [ 85.577950] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5626client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...5627client # farm-b> worker2: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5628client # farm-b> LOG-farm-b-on-standby5629worker2 # [ 86.404937] nix-daemon[869]: reaped child process 1248, status = succeeded5630worker2 # [ 86.606696] nix-daemon[869]: accepted connection from pid 1263, user nix-grpc-daemon (trusted)5631worker2 # [ 86.621253] nix-daemon[869]: reaped child process 1266, status = succeeded5632lb # [ 86.627773] niks3-server[1003]: 2026/09/21 09:13:16 INFO Received uploads request method=POST path=/api/pending_closures5633worker2 # [ 86.654666] nix-grpc-daemon[1160]: time=2026-09-21T09:13:15.864Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5634worker2 # [ 86.656138] nix-grpc-daemon[1160]: time=2026-09-21T09:13:15.864Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"5635lb # [ 86.670148] niks3-server[1003]: 2026/09/21 09:13:16 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst5636lb # [ 86.675426] niks3-server[1003]: 2026/09/21 09:13:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5637worker2 # [ 86.700745] nix-grpc-daemon[1160]: time=2026-09-21T09:13:15.910Z level=INFO msg="Uploading 1 narinfos"5638lb # [ 86.686780] niks3-server[1003]: 2026/09/21 09:13:16 INFO Signed narinfos id=30 count=15639lb # [ 86.693674] niks3-server[1003]: 2026/09/21 09:13:16 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls5640lb # [ 86.706179] niks3-server[1003]: 2026/09/21 09:13:16 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5641lb # [ 86.708577] niks3-server[1003]: 2026/09/21 09:13:16 INFO Completed upload id=305642worker2 # [ 86.722791] nix-grpc-daemon[1160]: time=2026-09-21T09:13:15.933Z level=INFO msg="Upload complete. (336ms)"5643lb # [ 86.711575] niks3-server[1003]: 2026/09/21 09:13:16 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo5644lb # [ 86.716054] niks3-server[1003]: 2026/09/21 09:13:16 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5645worker2 # [ 86.730725] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53712 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=15646worker2 # [ 86.744548] nix-daemon[869]: reaped child process 1246, status = succeeded5647worker2 # [ 86.779254] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:15Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=05648worker2 # [ 86.783279] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:15Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.4:50051 clients=15649worker2 # [ 86.834093] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:537125650worker2 # [ 86.847574] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:537125651worker2 # [ 86.861915] nix-grpc-daemon[1160]: time=2026-09-21T09:13:16.072Z level=INFO msg="All 2 paths already cached"5652client # copying 1 paths to 'lb:50051'...5653client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...5654worker2 # [ 86.870270] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:537125655worker2 # [ 86.879337] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5656worker2 # [ 87.097109] nix-daemon[869]: accepted connection from pid 1272, user nix-grpc-daemon (trusted)5657worker2 # [ 87.110579] nix-daemon[869]: reaped child process 1275, status = succeeded5658worker2 # [ 87.313823] nix-daemon[869]: accepted connection from pid 1277, user nix-grpc-daemon (trusted)5659worker2 # [ 87.328088] nix-daemon[869]: reaped child process 1280, status = succeeded5660lb # [ 87.334832] niks3-server[1003]: 2026/09/21 09:13:17 INFO Received uploads request method=POST path=/api/pending_closures5661worker2 # [ 87.361675] nix-grpc-daemon[1160]: time=2026-09-21T09:13:16.572Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5662worker2 # [ 87.363769] nix-grpc-daemon[1160]: time=2026-09-21T09:13:16.573Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"5663lb # [ 87.369803] niks3-server[1003]: 2026/09/21 09:13:17 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst5664lb # [ 87.379955] niks3-server[1003]: 2026/09/21 09:13:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5665lb # [ 87.386333] niks3-server[1003]: 2026/09/21 09:13:17 INFO Signed narinfos id=31 count=15666worker2 # [ 87.403479] nix-grpc-daemon[1160]: time=2026-09-21T09:13:16.613Z level=INFO msg="Uploading 1 narinfos"5667lb # [ 87.389619] niks3-server[1003]: 2026/09/21 09:13:17 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls5668lb # [ 87.401836] niks3-server[1003]: 2026/09/21 09:13:17 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5669lb # [ 87.407092] niks3-server[1003]: 2026/09/21 09:13:17 INFO Completed upload id=315670worker2 # [ 87.421188] nix-grpc-daemon[1160]: time=2026-09-21T09:13:16.631Z level=INFO msg="Upload complete. (525ms)"5671lb # [ 87.411843] niks3-server[1003]: 2026/09/21 09:13:17 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo5672worker2 # [ 87.425497] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53712 duration_s=0 paths=1 nar_bytes_in=7685673worker2 # [ 87.431278] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:537125674worker2 # [ 87.444463] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5675worker2 # [ 87.446807] nix-daemon[869]: reaped child process 1270, status = succeeded5676worker2 # [ 87.452423] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5677client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...5678client # farm-top> worker2: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5679lb # [ 88.127837] envoy[918]: conn peer=[2001:db8:1::3]:35788 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2144 tx=513 ms=31965680client # farm-top> LOG-farm-top-on-standby5681worker2 # [ 88.473169] nix-daemon[869]: reaped child process 1284, status = succeeded5682worker2 # [ 88.624353] nix-daemon[869]: accepted connection from pid 1299, user nix-grpc-daemon (trusted)5683worker2 # [ 88.635347] nix-daemon[869]: reaped child process 1302, status = succeeded5684lb # [ 88.634796] niks3-server[1003]: 2026/09/21 09:13:18 INFO Received uploads request method=POST path=/api/pending_closures5685worker2 # [ 88.656984] nix-grpc-daemon[1160]: time=2026-09-21T09:13:17.867Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5686worker2 # [ 88.658624] nix-grpc-daemon[1160]: time=2026-09-21T09:13:17.868Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"5687lb # [ 88.670249] niks3-server[1003]: 2026/09/21 09:13:18 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst5688lb # [ 88.680531] niks3-server[1003]: 2026/09/21 09:13:18 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5689lb # [ 88.690158] niks3-server[1003]: 2026/09/21 09:13:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5690worker2 # [ 88.708833] nix-grpc-daemon[1160]: time=2026-09-21T09:13:17.919Z level=INFO msg="Uploading 1 narinfos"5691lb # [ 88.696142] niks3-server[1003]: 2026/09/21 09:13:18 INFO Signed narinfos id=32 count=15692lb # [ 88.700920] niks3-server[1003]: 2026/09/21 09:13:18 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls5693lb # [ 88.710043] niks3-server[1003]: 2026/09/21 09:13:18 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5694lb # [ 88.716113] niks3-server[1003]: 2026/09/21 09:13:18 INFO Completed upload id=325695worker2 # [ 88.730173] nix-grpc-daemon[1160]: time=2026-09-21T09:13:17.940Z level=INFO msg="Upload complete. (270ms)"5696lb # [ 88.718324] niks3-server[1003]: 2026/09/21 09:13:18 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo5697worker2 # [ 88.737704] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53712 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=15698worker2 # [ 88.749905] nix-daemon[869]: reaped child process 1282, status = succeeded5699client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 6.16 seconds)5700client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath5701client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.21 seconds)5702worker1 # [ 89.058957] systemd[1]: Listening on Nix worker-protocol over gRPC.5703worker1 # [ 89.063588] systemd[1]: Starting Nix worker-protocol over gRPC...5704worker1 # [ 89.209695] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:18Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5705worker1 # [ 89.211961] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:18Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5706worker1 # [ 89.215533] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:18Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15707worker1 # [ 89.222064] nix-grpc-daemon[1283]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5708worker1 # [ 89.223425] nix-grpc-daemon[1283]: E0000 00:00:1789981998.471673 1283 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5709worker1 # [ 89.229159] nix-grpc-daemon[1283]: E0000 00:00:1789981998.476802 1283 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5710worker1 # [ 89.232128] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:18Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5711worker1 # [ 89.235309] systemd[1]: Started Nix worker-protocol over gRPC.5712worker1 # [ 89.242085] nix-grpc-daemon[1283]: E0000 00:00:1789981998.490009 1325 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5713lb: must succeed: curl -sf localhost:9901/clusters5714worker1 # [ 89.252295] nix-grpc-daemon[1283]: E0000 00:00:1789981998.502471 1325 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5715worker1 # [ 89.256054] nix-grpc-daemon[1283]: E0000 00:00:1789981998.504807 1325 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5716worker1 # [ 89.266899] nix-grpc-daemon[1283]: E0000 00:00:1789981998.516530 1325 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5717worker1 # [ 89.279536] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:18Z level=info event=scheduler_connected addr=lb:500515718worker2 # [ 89.261290] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:18Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:537225719worker2 # [ 89.264960] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:18Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05720lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5721worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5722worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5723lb: must succeed: curl -sf localhost:9901/clusters5724lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5725lb: must succeed: curl -sf localhost:9901/clusters5726lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5727worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5728worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5729(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.12 seconds)5730subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5731client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs5732client # [ 89.613465] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.5733client # Running as unit: rs.service; invocation ID: 17a53b96ddb54bc7b7f1ddad724cafbf5734client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.07 seconds)5735client # [ 89.845793] nix[1182]: don't know how to build these paths:5736client # [ 89.846738] nix[1182]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5737worker2 # [ 89.856413] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:537125738worker2 # [ 89.868206] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:19Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=05739worker2 # [ 89.873095] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:19Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.4:50051 clients=15740worker2 # [ 89.912915] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:537125741worker2 # [ 89.925701] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:537125742client # [ 89.924269] nix[1182]: copying 1 paths to 'lb:50051'...5743worker2 # [ 89.936888] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:537125744client # [ 89.927200] nix[1182]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...5745worker2 # [ 89.948633] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5746worker2 # [ 90.132318] nix-daemon[869]: accepted connection from pid 1315, user nix-grpc-daemon (trusted)5747worker2 # [ 90.144503] nix-daemon[869]: reaped child process 1318, status = succeeded5748worker2 # [ 90.317632] nix-daemon[869]: accepted connection from pid 1320, user nix-grpc-daemon (trusted)5749worker2 # [ 90.327913] nix-daemon[869]: reaped child process 1323, status = succeeded5750lb # [ 90.332927] niks3-server[1003]: 2026/09/21 09:13:20 INFO Received uploads request method=POST path=/api/pending_closures5751worker2 # [ 90.355860] nix-grpc-daemon[1160]: time=2026-09-21T09:13:19.566Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5752worker2 # [ 90.357649] nix-grpc-daemon[1160]: time=2026-09-21T09:13:19.567Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"5753lb # [ 90.363660] niks3-server[1003]: 2026/09/21 09:13:20 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst5754lb # [ 90.375332] niks3-server[1003]: 2026/09/21 09:13:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5755lb # [ 90.380357] niks3-server[1003]: 2026/09/21 09:13:20 INFO Signed narinfos id=33 count=15756worker2 # [ 90.397250] nix-grpc-daemon[1160]: time=2026-09-21T09:13:19.607Z level=INFO msg="Uploading 1 narinfos"5757lb # [ 90.389529] niks3-server[1003]: 2026/09/21 09:13:20 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls5758lb # [ 90.400207] niks3-server[1003]: 2026/09/21 09:13:20 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5759lb # [ 90.407638] niks3-server[1003]: 2026/09/21 09:13:20 INFO Completed upload id=335760worker2 # [ 90.421741] nix-grpc-daemon[1160]: time=2026-09-21T09:13:19.632Z level=INFO msg="Upload complete. (460ms)"5761lb # [ 90.409464] niks3-server[1003]: 2026/09/21 09:13:20 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo5762worker2 # [ 90.426069] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53712 duration_s=0 paths=1 nar_bytes_in=4165763worker2 # [ 90.432963] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:537125764worker2 # [ 90.442475] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5765worker2 # [ 90.450125] nix-daemon[869]: reaped child process 1313, status = succeeded5766worker2 # [ 90.456065] nix-daemon[869]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5767client # [ 90.443789] nix[1182]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...5768lb: must succeed: curl -sf localhost:9901/clusters5769lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5770lb: must succeed: curl -sf localhost:9901/clusters5771lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5772lb: must succeed: curl -sf localhost:9901/clusters5773lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5774lb: must succeed: curl -sf localhost:9901/clusters5775lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5776worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5777worker1 # [ 92.564731] systemd-journald[287]: Received client request to rotate journal, rotating.5778worker1 # [ 92.582393] systemd-journald[287]: Vacuuming done, freed 0B of archived journals from /var/log/journal/d6daef13a9f14448bda329d1d4712de2.5779worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.07 seconds)5780worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service5781worker2 # [ 92.484643] systemd[1]: Stopping Nix worker-protocol over gRPC...5782worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)5783client: must succeed: sleep 5.05784worker2 # [ 93.419175] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:22Z level=info event=draining5785client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5786worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5787worker2 # [ 97.874994] nix-daemon[869]: reaped child process 1327, status = succeeded5788worker2 # [ 98.026300] nix-daemon[869]: accepted connection from pid 1367, user nix-grpc-daemon (trusted)5789worker2 # [ 98.037756] nix-daemon[869]: reaped child process 1370, status = succeeded5790lb # [ 98.042849] niks3-server[1003]: 2026/09/21 09:13:28 INFO Received uploads request method=POST path=/api/pending_closures5791worker2 # [ 98.067943] nix-grpc-daemon[1160]: time=2026-09-21T09:13:27.278Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5792worker2 # [ 98.069456] nix-grpc-daemon[1160]: time=2026-09-21T09:13:27.279Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"5793lb # [ 98.096738] niks3-server[1003]: 2026/09/21 09:13:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5794lb # [ 98.103729] niks3-server[1003]: 2026/09/21 09:13:28 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5795lb # [ 98.108383] niks3-server[1003]: 2026/09/21 09:13:28 INFO Signed narinfos id=34 count=15796worker2 # [ 98.122050] nix-grpc-daemon[1160]: time=2026-09-21T09:13:27.332Z level=INFO msg="Uploading 1 narinfos"5797lb # [ 98.114627] niks3-server[1003]: 2026/09/21 09:13:28 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5798lb # [ 98.118746] niks3-server[1003]: 2026/09/21 09:13:28 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls5799lb # [ 98.125256] niks3-server[1003]: 2026/09/21 09:13:28 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5800lb # [ 98.128629] envoy[918]: conn peer=[2001:db8:1::1]:35810 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9646 tx=3891 ms=58875801lb # [ 98.131815] niks3-server[1003]: 2026/09/21 09:13:28 INFO Completed upload id=345802lb # [ 98.133678] niks3-server[1003]: 2026/09/21 09:13:28 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo5803worker2 # [ 98.146237] nix-grpc-daemon[1160]: time=2026-09-21T09:13:27.356Z level=INFO msg="Upload complete. (277ms)"5804worker2 # [ 98.155160] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53712 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=15805worker2 # [ 98.167272] nix-daemon[869]: reaped child process 1325, status = succeeded5806client # [ 98.181699] systemd[1]: rs.service: Deactivated successfully.5807client # [ 98.185801] systemd[1]: rs.service: Consumed 132ms CPU time over 8.570s wall clock time, 41.4M memory peak, 7K incoming IP traffic, 8.3K outgoing IP traffic.5808worker2 # [ 98.420553] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:27Z level=info event=signal_exit5809worker2 # [ 98.650195] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:27Z level=info event=scheduler_yield5810lb # [ 98.636773] niks3-server[1003]: 2026/09/21 09:13:28 INFO lead: released remote=192.168.1.4:403745811worker2 # [ 98.655754] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:27Z level=info event=scheduler_restarting addr=lb:500515812worker1 # [ 98.678358] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:27Z level=info event=scheduler_restarting addr=lb:500515813worker2 # [ 98.660371] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:27Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:537225814worker2 # [ 98.664135] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:27Z level=info event=worker_gone addr=192.168.1.4:500515815worker2 # [ 98.667109] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:27Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:537225816worker2 # [ 98.670314] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:27Z level=info event=worker_gone addr=192.168.1.3:500515817worker2 # [ 98.857074] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:28Z level=info event=scheduler_connected addr=lb:500515818worker1 # [ 98.878264] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:28Z level=info event=scheduler_connected addr=lb:500515819worker2 # [ 98.860978] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5820worker1 # [ 98.883287] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5821worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 1.13 seconds)5822client: waiting for success: ! systemctl is-active rs5823client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)5824client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5825client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.04 seconds)5826client: must fail: journalctl -u rs | grep -q 'reconnecting'5827client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5828lb: must succeed: curl -sf localhost:9901/clusters5829worker2 # [ 99.062202] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:28Z level=info event=scheduler_connected addr=lb:500515830worker1 # [ 99.083534] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:28Z level=info event=scheduler_connected addr=lb:500515831worker2 # [ 99.066179] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5832worker1 # [ 99.087622] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5833lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5834lb # [ 99.205715] niks3-server[1003]: 2026/09/21 09:13:29 INFO lead: acquired remote=192.168.1.3:397165835worker1 # [ 99.240556] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:28Z level=info event=scheduler_take_over5836worker1 # [ 99.288397] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:28Z level=info event=scheduler_connected addr=lb:500515837worker2 # [ 99.268148] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:28Z level=info event=scheduler_connected addr=lb:500515838worker1 # [ 99.292326] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5839worker2 # [ 99.272094] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5840worker1 # [ 99.493360] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:28Z level=info event=scheduler_connected addr=lb:500515841worker2 # [ 99.473523] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:28Z level=info event=scheduler_connected addr=lb:500515842worker1 # [ 99.497421] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5843worker2 # [ 99.477490] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5844worker2 # [ 99.654826] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:28Z level=info event=scheduler_restarting5845worker1 # [ 99.699151] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:28Z level=info event=scheduler_connected addr=lb:500515846worker1 # [ 99.702992] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5847worker2 # [ 99.692389] nix-daemon[869]: reaped child process 1153, status = succeeded5848worker2 # [ 99.694698] nix-grpc-daemon[1101]: ts=2026-09-21T09:13:28Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5849worker1 # [ 99.903881] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:29Z level=info event=scheduler_connected addr=lb:500515850worker1 # [ 99.907896] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5851worker1 # [ 100.108745] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:29Z level=info event=scheduler_connected addr=lb:500515852worker1 # [ 100.112704] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5853lb: must succeed: curl -sf localhost:9901/clusters5854lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5855worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5856worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5857worker1 # [ 100.313413] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:29Z level=info event=scheduler_connected addr=lb:500515858worker1 # [ 100.317305] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5859worker2 # [ 100.484120] nix-grpc-daemon[1101]: E0000 00:00:1789982009.693511 1146 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5860worker2 # [ 100.488975] nix-grpc-daemon[1101]: E0000 00:00:1789982009.693696 1146 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5861worker1 # [ 100.518142] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:29Z level=info event=scheduler_connected addr=lb:500515862worker1 # [ 100.525135] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:29Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:341645863worker1 # [ 100.528635] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:29Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05864lb: must succeed: curl -sf localhost:9901/clusters5865lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5866worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5867worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5868worker2 # [ 101.425536] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5869worker2 # [ 101.428241] systemd[1]: Stopped Nix worker-protocol over gRPC.5870worker2 # [ 101.430379] systemd[1]: nix-grpc-daemon.service: Consumed 1.537s CPU time over 30.219s wall clock time, 35.9M memory peak, 63.2K incoming IP traffic, 168.1K outgoing IP traffic.5871worker2 # [ 101.445833] systemd[1]: Starting scheduler-token.service...5872worker2 # [ 101.527730] systemd[1]: scheduler-token.service: Deactivated successfully.5873worker2 # [ 101.528841] systemd[1]: Finished scheduler-token.service.5874worker2 # [ 101.530500] systemd[1]: scheduler-token.service: Consumed 22ms CPU time over 81ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5875worker2 # [ 101.534920] systemd[1]: Starting Nix worker-protocol over gRPC...5876worker2 # [ 101.707217] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:30Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5877worker2 # [ 101.709417] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:30Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5878worker2 # [ 101.713733] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:30Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15879worker2 # [ 101.721063] nix-grpc-daemon[1380]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5880worker2 # [ 101.722433] nix-grpc-daemon[1380]: E0000 00:00:1789982010.931065 1380 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5881worker2 # [ 101.727430] nix-grpc-daemon[1380]: E0000 00:00:1789982010.936221 1380 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5882worker2 # [ 101.729843] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:30Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5883worker2 # [ 101.732404] systemd[1]: Started Nix worker-protocol over gRPC.5884worker2 # [ 101.739071] nix-grpc-daemon[1380]: E0000 00:00:1789982010.948947 1422 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5885worker2 # [ 101.750391] nix-grpc-daemon[1380]: E0000 00:00:1789982010.960795 1422 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5886worker2 # [ 101.755053] nix-grpc-daemon[1380]: E0000 00:00:1789982010.963163 1422 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5887worker2 # [ 101.764550] nix-grpc-daemon[1380]: E0000 00:00:1789982010.974986 1422 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5888worker2 # [ 101.773955] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:30Z level=info event=scheduler_connected addr=lb:500515889worker1 # [ 101.803811] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:31Z level=info event=oidc_jwks provider=mock keys=15890worker1 # [ 101.808074] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:31Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:341645891worker1 # [ 101.811840] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:31Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05892lb: must succeed: curl -sf localhost:9901/clusters5893lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5894worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5895worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5896(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.99 seconds)5897subtest: scheduler restart mid-build: build finishes, no double build5898client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid5899client # [ 102.607682] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.5900client # Running as unit: mid.service; invocation ID: 4a616def27a448e188a9eff1a7e4fd785901client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)5902worker1 # [ 102.886810] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:485545903worker1 # [ 102.897842] nix-daemon[848]: accepted connection from pid 1283, user nix-grpc-daemon (trusted)5904client # [ 102.877599] nix[1218]: don't know how to build these paths:5905client # [ 102.878627] nix[1218]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv5906worker1 # [ 102.920502] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:32Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=05907worker1 # [ 102.925759] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:32Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=15908worker1 # [ 102.965400] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485545909worker1 # [ 102.977319] nix-grpc-daemon[1283]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 94 ms (attempt 1/5)5910worker1 # [ 103.077147] nix-grpc-daemon[1283]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 130 ms (attempt 2/5)5911worker1 # [ 103.213083] nix-grpc-daemon[1283]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 282 ms (attempt 3/5)5912worker1 # [ 103.501151] nix-grpc-daemon[1283]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 552 ms (attempt 4/5)5913worker1 # [ 104.060174] nix-grpc-daemon[1283]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5914worker1 # [ 104.078998] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485545915client # [ 104.058911] nix[1218]: copying 1 paths to 'lb:50051'...5916worker1 # [ 104.092455] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:485545917client # [ 104.061932] nix[1218]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...5918worker1 # [ 104.106930] nix-daemon[848]: accepted connection from pid 1283, user nix-grpc-daemon (trusted)5919worker1 # [ 104.125279] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:33Z level=info event=niks3_push_started pid=14055920worker1 # [ 104.349549] nix-grpc-daemon[1405]: time=2026-09-21T09:13:33.599Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""5921worker1 # [ 104.535353] nix-daemon[848]: accepted connection from pid 1418, user nix-grpc-daemon (trusted)5922worker1 # [ 104.545631] nix-daemon[848]: reaped child process 1421, status = succeeded5923worker1 # [ 104.733204] nix-daemon[848]: accepted connection from pid 1423, user nix-grpc-daemon (trusted)5924worker1 # [ 104.744809] nix-daemon[848]: reaped child process 1426, status = succeeded5925lb # [ 104.730615] niks3-server[1003]: 2026/09/21 09:13:34 INFO Received uploads request method=POST path=/api/pending_closures5926worker1 # [ 104.773330] nix-grpc-daemon[1405]: time=2026-09-21T09:13:34.023Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5927worker1 # [ 104.776286] nix-grpc-daemon[1405]: time=2026-09-21T09:13:34.024Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"5928lb # [ 104.773836] niks3-server[1003]: 2026/09/21 09:13:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5929lb # [ 104.778727] niks3-server[1003]: 2026/09/21 09:13:34 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst5930worker1 # [ 104.821757] nix-grpc-daemon[1405]: time=2026-09-21T09:13:34.071Z level=INFO msg="Uploading 1 narinfos"5931lb # [ 104.786589] niks3-server[1003]: 2026/09/21 09:13:34 INFO Signed narinfos id=35 count=15932lb # [ 104.790708] niks3-server[1003]: 2026/09/21 09:13:34 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls5933lb # [ 104.800772] niks3-server[1003]: 2026/09/21 09:13:34 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5934lb # [ 104.807809] niks3-server[1003]: 2026/09/21 09:13:34 INFO Completed upload id=355935worker1 # [ 104.842521] nix-grpc-daemon[1405]: time=2026-09-21T09:13:34.092Z level=INFO msg="Upload complete. (488ms)"5936lb # [ 104.809817] niks3-server[1003]: 2026/09/21 09:13:34 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo5937worker1 # [ 104.846180] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48554 duration_s=0 paths=1 nar_bytes_in=4245938worker1 # [ 104.852254] nix-grpc-daemon[1283]: ts=2026-09-21T09:13:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485545939worker1 # [ 104.860244] nix-daemon[848]: accepted connection from pid 1283, user nix-grpc-daemon (trusted)5940worker1 # [ 104.868750] nix-daemon[848]: reaped child process 1403, status = succeeded5941worker1 # [ 104.874044] nix-daemon[848]: accepted connection from pid 1283, user nix-grpc-daemon (trusted)5942client # [ 104.840281] nix[1218]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...5943lb: must succeed: curl -sf localhost:9901/clusters5944lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5945lb: must succeed: curl -sf localhost:9901/clusters5946lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5947worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5948lb # [ 106.449983] niks3-server[1003]: 2026/09/21 09:13:36 INFO lead: released remote=192.168.1.3:397165949worker1 # [ 106.484352] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1283 (nix-grpc-daemon) on client request.5950worker1 # [ 106.486230] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5951client # [ 106.460580] nix[1218]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5952worker2 # [ 106.471123] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:35Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5953client # [ 106.464637] nix[1218]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5954worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.06 seconds)5955client: must succeed: sleep 5.05956worker1 # [ 106.512104] nix-daemon[848]: reaped child process 1428, status = succeeded5957worker1 # [ 106.513389] nix-daemon[848]: reaped child process 1392, status = succeeded5958worker1 # [ 106.518903] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5959worker1 # [ 106.521597] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5960worker1 # [ 106.523313] systemd[1]: nix-grpc-daemon.service: Consumed 445ms CPU time over 17.454s wall clock time, 33.2M memory peak, 17.3K incoming IP traffic, 31.9K outgoing IP traffic.5961worker1 # [ 106.532244] nix-daemon[1430]: unexpected Nix daemon error: error: interrupted by the user5962worker1 # [ 106.535360] nix-daemon[848]: reaped child process 1430, status = failed with exit code 15963worker1 # [ 106.645266] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1.5964worker1 # [ 106.649500] systemd[1]: Starting Nix worker-protocol over gRPC...5965worker2 # [ 106.671878] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:35Z level=info event=scheduler_connected addr=lb:500515966lb # [ 106.724284] niks3-server[1003]: 2026/09/21 09:13:36 INFO lead: acquired remote=192.168.1.4:498225967worker2 # [ 106.738511] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:35Z level=info event=scheduler_take_over5968worker1 # [ 106.822387] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5969worker1 # [ 106.824890] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5970worker1 # [ 106.828519] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15971worker1 # [ 106.836068] nix-grpc-daemon[1466]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5972worker1 # [ 106.837433] nix-grpc-daemon[1466]: E0000 00:00:1789982016.085296 1466 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5973worker1 # [ 106.842976] nix-grpc-daemon[1466]: E0000 00:00:1789982016.090420 1466 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5974worker1 # [ 106.845736] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5975worker1 # [ 106.848120] systemd[1]: Started Nix worker-protocol over gRPC.5976worker2 # [ 106.830380] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5977worker1 # [ 106.856067] nix-grpc-daemon[1466]: E0000 00:00:1789982016.105286 1507 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5978worker1 # [ 106.867034] nix-grpc-daemon[1466]: E0000 00:00:1789982016.117113 1507 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5979worker1 # [ 106.871052] nix-grpc-daemon[1466]: E0000 00:00:1789982016.119321 1507 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5980worker1 # [ 106.880670] nix-grpc-daemon[1466]: E0000 00:00:1789982016.130796 1507 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5981worker1 # [ 106.889613] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500515982worker1 # [ 106.891915] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5983client # [ 106.963919] nix[1218]: scheduler at lb:50051: scheduler passive, another node serves, reconnecting5984worker2 # [ 107.032139] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500515985worker2 # [ 107.036156] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5986worker1 # [ 107.092994] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500515987worker1 # [ 107.097753] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5988worker2 # [ 107.237458] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500515989worker2 # [ 107.241453] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5990worker1 # [ 107.298616] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500515991worker1 # [ 107.303537] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5992worker2 # [ 107.442241] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500515993worker2 # [ 107.445994] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5994worker1 # [ 107.504525] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500515995worker1 # [ 107.508406] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5996worker2 # [ 107.647664] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500515997worker2 # [ 107.651378] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5998worker1 # [ 107.709325] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500515999worker1 # [ 107.713219] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6000worker2 # [ 107.852810] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500516001worker2 # [ 107.856725] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6002worker1 # [ 107.914825] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500516003worker1 # [ 107.918539] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6004client # [ 107.967512] nix[1218]: scheduler at lb:50051: no healthy upstream, reconnecting6005worker2 # [ 108.057684] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500516006worker2 # [ 108.061530] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6007worker1 # [ 108.119439] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500516008worker1 # [ 108.123313] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6009lb # [ 108.130380] envoy[918]: conn peer=[2001:db8:1::1]:35824 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2934 tx=1350 ms=83106010lb # [ 108.134450] envoy[918]: conn peer=[2001:db8:1::4]:55384 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=40021 tx=2180 ms=282246011lb # [ 108.138146] envoy[918]: conn peer=[2001:db8:1::3]:36622 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2386 tx=618 ms=172206012worker2 # [ 108.262622] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500516013worker2 # [ 108.266543] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6014worker1 # [ 108.324378] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500516015worker1 # [ 108.328286] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6016worker2 # [ 108.467484] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500516017worker2 # [ 108.484264] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=oidc_jwks provider=mock keys=16018worker2 # [ 108.488651] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:416286019worker2 # [ 108.492497] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06020worker1 # [ 108.528950] nix-grpc-daemon[1466]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500516021worker2 # [ 108.509699] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:416286022worker2 # [ 108.513355] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:37Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06023worker2 # [ 109.991891] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:39Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06024worker2 # [ 109.995975] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:39Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=16025worker2 # [ 110.043767] nix-grpc-daemon[1380]: ts=2026-09-21T09:13:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416446026worker2 # [ 110.054798] nix-daemon[869]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6027worker2 # [ 110.071699] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 12 ms (attempt 1/5)6028worker2 # [ 110.089569] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 161 ms (attempt 2/5)6029worker2 # [ 110.256924] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 77 ms (attempt 3/5)6030worker2 # [ 110.340168] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 162 ms (attempt 4/5)6031worker2 # [ 110.508590] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6032worker2 # [ 110.525555] nix-grpc-daemon[1380]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' from 'http://lb:5751'...6033worker2 # [ 110.549273] nix-daemon[869]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6034worker2 # [ 110.564183] nix-daemon[869]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6035client # [ 110.552124] nix[1218]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6036client: (finished: must succeed: sleep 5.0, in 5.05 seconds)6037client: waiting for success: ! systemctl is-active mid6038worker2 # [ 201.863644] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:11Z level=info event=niks3_push_started pid=14866039worker2 # [ 201.874109] nix-daemon[869]: reaped child process 1462, status = succeeded6040worker2 # [ 202.079525] nix-grpc-daemon[1486]: time=2026-09-21T09:15:11.289Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6041worker2 # [ 202.275363] nix-daemon[869]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6042worker2 # [ 202.287274] nix-daemon[869]: reaped child process 1502, status = succeeded6043lb # [ 202.294944] niks3-server[1003]: 2026/09/21 09:15:12 INFO Received uploads request method=POST path=/api/pending_closures6044worker2 # [ 202.318183] nix-grpc-daemon[1486]: time=2026-09-21T09:15:11.528Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6045worker2 # [ 202.321436] nix-grpc-daemon[1486]: time=2026-09-21T09:15:11.530Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6046lb # [ 202.348188] niks3-server[1003]: 2026/09/21 09:15:12 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6047lb # [ 202.356113] niks3-server[1003]: 2026/09/21 09:15:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign6048lb # [ 202.359906] niks3-server[1003]: 2026/09/21 09:15:12 INFO Signed narinfos id=36 count=16049worker2 # [ 202.376903] nix-grpc-daemon[1486]: time=2026-09-21T09:15:11.586Z level=INFO msg="Uploading 1 narinfos"6050lb # [ 202.373387] niks3-server[1003]: 2026/09/21 09:15:12 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6051lb # [ 202.378394] niks3-server[1003]: 2026/09/21 09:15:12 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete6052lb # [ 202.383583] niks3-server[1003]: 2026/09/21 09:15:12 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6053lb # [ 202.389244] niks3-server[1003]: 2026/09/21 09:15:12 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6054lb # [ 202.395357] niks3-server[1003]: 2026/09/21 09:15:12 INFO Completed upload id=366055worker2 # [ 202.409620] nix-grpc-daemon[1486]: time=2026-09-21T09:15:11.619Z level=INFO msg="Upload complete. (326ms)"6056worker2 # [ 202.418151] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41644 duration_s=92 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16057worker2 # [ 202.432197] nix-daemon[869]: reaped child process 1460, status = succeeded6058client # [ 202.457484] systemd[1]: mid.service: Deactivated successfully.6059client # [ 202.460908] systemd[1]: mid.service: Consumed 254ms CPU time over 1min 39.852s wall clock time, 41.5M memory peak, 4K written to disk, 8.4K incoming IP traffic, 10.5K outgoing IP traffic.6060client: (finished: waiting for success: ! systemctl is-active mid, in 91.26 seconds)6061client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6062client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds)6063(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 100.53 seconds)6064subtest: interrupting the client stops the build on the worker6065client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6066client # [ 203.113707] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6067client # Running as unit: intr.service; invocation ID: ffa6cd6ea0754b1da56408664522b2cf6068client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.04 seconds)6069worker1 # [ 203.396380] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:503626070worker1 # [ 203.406657] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6071client # [ 203.386220] nix[1679]: don't know how to build these paths:6072client # [ 203.387453] nix[1679]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6073worker2 # [ 203.410890] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:12Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06074worker2 # [ 203.415402] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:12Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.3:50051 clients=16075worker1 # [ 203.480123] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626076worker1 # [ 203.492200] nix-grpc-daemon[1466]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 44 ms (attempt 1/5)6077worker1 # [ 203.542625] nix-grpc-daemon[1466]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 82 ms (attempt 2/5)6078worker1 # [ 203.630470] nix-grpc-daemon[1466]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 363 ms (attempt 3/5)6079worker1 # [ 203.999460] nix-grpc-daemon[1466]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 485 ms (attempt 4/5)6080worker1 # [ 204.491424] nix-grpc-daemon[1466]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6081worker1 # [ 204.510125] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626082client # [ 204.488574] nix[1679]: copying 1 paths to 'lb:50051'...6083client # [ 204.491888] nix[1679]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6084worker1 # [ 204.525307] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626085worker1 # [ 204.533881] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6086worker1 # [ 204.556483] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:13Z level=info event=niks3_push_started pid=15416087worker1 # [ 204.811055] nix-grpc-daemon[1541]: time=2026-09-21T09:15:14.060Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6088worker1 # [ 205.027655] nix-daemon[848]: accepted connection from pid 1554, user nix-grpc-daemon (trusted)6089worker1 # [ 205.039372] nix-daemon[848]: reaped child process 1557, status = succeeded6090worker1 # [ 205.268108] nix-daemon[848]: accepted connection from pid 1559, user nix-grpc-daemon (trusted)6091worker1 # [ 205.279553] nix-daemon[848]: reaped child process 1562, status = succeeded6092lb # [ 205.270136] niks3-server[1003]: 2026/09/21 09:15:15 INFO Received uploads request method=POST path=/api/pending_closures6093worker1 # [ 205.313934] nix-grpc-daemon[1541]: time=2026-09-21T09:15:14.564Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6094worker1 # [ 205.316848] nix-grpc-daemon[1541]: time=2026-09-21T09:15:14.565Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6095lb # [ 205.305073] niks3-server[1003]: 2026/09/21 09:15:15 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6096lb # [ 205.313191] niks3-server[1003]: 2026/09/21 09:15:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6097lb # [ 205.317953] niks3-server[1003]: 2026/09/21 09:15:15 INFO Signed narinfos id=37 count=16098worker1 # [ 205.356288] nix-grpc-daemon[1541]: time=2026-09-21T09:15:14.606Z level=INFO msg="Uploading 1 narinfos"6099lb # [ 205.329440] niks3-server[1003]: 2026/09/21 09:15:15 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6100lb # [ 205.337936] niks3-server[1003]: 2026/09/21 09:15:15 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6101worker1 # [ 205.379746] nix-grpc-daemon[1541]: time=2026-09-21T09:15:14.629Z level=INFO msg="Upload complete. (565ms)"6102lb # [ 205.345506] niks3-server[1003]: 2026/09/21 09:15:15 INFO Completed upload id=376103worker1 # [ 205.381720] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=4246104lb # [ 205.349226] niks3-server[1003]: 2026/09/21 09:15:15 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6105worker1 # [ 205.387211] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626106worker1 # [ 205.395409] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6107worker1 # [ 205.397665] nix-daemon[848]: reaped child process 1539, status = succeeded6108worker1 # [ 205.403082] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6109client # [ 205.369661] nix[1679]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6110client: must succeed: systemctl kill -s INT intr6111client # [ 206.719432] systemd[1]: intr.service: Sent signal SIGINT to main process 1679 (nix) on client request.6112worker1 # [ 206.759903] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:16Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6113client # [ 206.727940] nix[1679]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6114client # [ 206.735684] nix[1679]: error: interrupted by the user6115client: (finished: must succeed: systemctl kill -s INT intr, in 0.06 seconds)6116client # [ 206.768482] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6117client # [ 206.770518] systemd[1]: intr.service: Failed with result 'exit-code'.6118client # [ 206.772080] systemd[1]: intr.service: Consumed 124ms CPU time over 3.656s wall clock time, 41.4M memory peak, 6.2K incoming IP traffic, 8K outgoing IP traffic.6119worker1 # [ 206.821040] nix-daemon[848]: reaped child process 1564, status = succeeded6120worker1 # [ 206.825319] nix-daemon[1566]: unexpected Nix daemon error: error: interrupted by the user6121worker1 # [ 206.832140] nix-daemon[848]: reaped child process 1566, status = failed with exit code 16122(finished: subtest: interrupting the client stops the build on the worker, in 3.79 seconds)6123subtest: a deploy mid-build drains: the build finishes, the new generation takes over6124client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6125client # [ 206.912493] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6126client # Running as unit: sw.service; invocation ID: 9a96b1cddaac4e6ab6bb94b5daf7754f6127client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.04 seconds)6128worker2 # [ 207.171708] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:416446129client # [ 207.166412] nix[1700]: don't know how to build these paths:6130client # [ 207.167401] nix[1700]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6131worker2 # [ 207.187424] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:16Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06132worker2 # [ 207.192936] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:16Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.4:50051 clients=16133worker2 # [ 207.231920] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416446134worker2 # [ 207.248818] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416446135client # [ 207.247204] nix[1700]: copying 1 paths to 'lb:50051'...6136worker2 # [ 207.260417] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:416446137client # [ 207.250147] nix[1700]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6138worker2 # [ 207.273748] nix-daemon[869]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6139worker2 # [ 207.483700] nix-daemon[869]: accepted connection from pid 1536, user nix-grpc-daemon (trusted)6140worker2 # [ 207.495444] nix-daemon[869]: reaped child process 1539, status = succeeded6141worker2 # [ 207.692665] nix-daemon[869]: accepted connection from pid 1541, user nix-grpc-daemon (trusted)6142worker2 # [ 207.704231] nix-daemon[869]: reaped child process 1544, status = succeeded6143lb # [ 207.710366] niks3-server[1003]: 2026/09/21 09:15:17 INFO Received uploads request method=POST path=/api/pending_closures6144worker2 # [ 207.737359] nix-grpc-daemon[1486]: time=2026-09-21T09:15:16.947Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6145worker2 # [ 207.738786] nix-grpc-daemon[1486]: time=2026-09-21T09:15:16.947Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6146lb # [ 207.742665] niks3-server[1003]: 2026/09/21 09:15:17 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6147lb # [ 207.754125] niks3-server[1003]: 2026/09/21 09:15:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6148worker2 # [ 207.774741] nix-grpc-daemon[1486]: time=2026-09-21T09:15:16.985Z level=INFO msg="Uploading 1 narinfos"6149lb # [ 207.760703] niks3-server[1003]: 2026/09/21 09:15:17 INFO Signed narinfos id=38 count=16150lb # [ 207.765225] niks3-server[1003]: 2026/09/21 09:15:17 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6151lb # [ 207.777316] niks3-server[1003]: 2026/09/21 09:15:17 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6152lb # [ 207.782168] niks3-server[1003]: 2026/09/21 09:15:17 INFO Completed upload id=386153lb # [ 207.785130] niks3-server[1003]: 2026/09/21 09:15:17 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6154worker2 # [ 207.798557] nix-grpc-daemon[1486]: time=2026-09-21T09:15:17.008Z level=INFO msg="Upload complete. (513ms)"6155worker2 # [ 207.802287] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41644 duration_s=0 paths=1 nar_bytes_in=4166156worker2 # [ 207.808250] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416446157worker2 # [ 207.819107] nix-daemon[869]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6158worker2 # [ 207.825483] nix-daemon[869]: reaped child process 1534, status = succeeded6159worker2 # [ 207.830557] nix-daemon[869]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6160client # [ 207.818181] nix[1700]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6161lb # [ 208.136389] envoy[918]: conn peer=[2001:db8:1::1]:48448 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4179 tx=1810 ms=995806162lb # [ 208.140228] envoy[918]: conn peer=[2001:db8:1::1]:41978 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2924 tx=756 ms=33926163worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&26164worker2 # Checking switch inhibitors... done6165worker2 # [ 210.493801] nixos[1579]: switching to system configuration /nix/store/v11lvsn6lvm6c4sin29rwv464gq3s2s3-nixos-system-worker2-test6166worker2 # [ 210.496433] systemd[1]: Stopped target Remote File Systems.6167worker2 # activating the configuration...6168worker2 # [ 210.498161] systemd[1]: Stopped target Local File Systems.6169worker2 # setting up /etc...6170worker2 # [ 211.256727] systemd[1]: Reload requested from client PID 1579 ('.switch-to-conf') (unit backdoor.service)...6171worker2 # [ 211.258326] systemd[1]: Reloading...6172worker2 # [ 212.121907] systemd[1]: Reloading finished in 859 ms.6173worker2 # restarting sysinit-reactivation.target6174worker2 # [ 212.179393] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....6175worker2 # [ 212.329511] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..6176worker2 # [ 212.333438] systemd[1]: Reached target Reactivate sysinit units.6177worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service6178worker2 # [ 212.338736] systemd[1]: Reloading D-Bus System Message Bus...6179worker2 # [ 212.339888] dbus-broker-launch[533]: Caught SIGHUP, trigger reload.6180worker2 # [ 212.344201] systemd[1]: Reloading Nix worker-protocol over gRPC...6181worker2 # [ 212.374793] dbus-broker-launch[533]: Looking up NSS user entry for 'systemd-timesync'...6182worker2 # [ 212.377137] dbus-broker-launch[533]: NSS returned no entry for 'systemd-timesync'6183worker2 # [ 212.379788] dbus-broker-launch[533]: Invalid user-name in /nix/store/nl02z4vnl25dc1df60lqifjnw6c89zpg-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"6184worker2 # [ 212.395085] systemd[1]: Reloaded Nix worker-protocol over gRPC.6185worker2 # [ 212.410763] systemd[1]: Reloaded D-Bus System Message Bus.6186worker2 # [ 212.413696] systemd[1]: Reached target Local File Systems.6187worker2 # [ 212.416731] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore6188worker2 # [ 212.423526] systemd[1]: Starting Create SUID/SGID Wrappers...6189worker2 # [ 212.424810] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.6190worker2 # [ 212.428141] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc676191worker2 # [ 212.430209] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore6192worker2 # [ 212.431663] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6193worker2 # [ 212.433660] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6194worker2 # [ 212.442098] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os6195worker2 # [ 212.443349] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki6196worker2 # [ 212.446127] systemd[1]: Reached target Remote File Systems.6197worker2 # [ 212.761870] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:21Z level=info event=draining6198worker2 # [ 212.939494] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.6199worker2 # [ 212.942513] systemd[1]: Finished Create SUID/SGID Wrappers.6200worker2 # [ 212.948063] systemd[1]: Starting linger-users.service...6201worker2 # [ 212.995304] systemd[1]: linger-users.service: Deactivated successfully.6202worker2 # [ 212.996261] systemd[1]: Finished linger-users.service.6203worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service6204worker2 # [ 213.251672] nixos[1579]: finished switching to system configuration /nix/store/v11lvsn6lvm6c4sin29rwv464gq3s2s3-nixos-system-worker2-test6205worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 3.74 seconds)6206worker2: must succeed: pgrep -f 'read -t [9]0 x'6207worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)6208client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26209worker1 # [ 213.606425] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:503626210client # don't know how to build these paths:6211client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6212worker1 # [ 213.618256] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626213worker2 # [ 213.624052] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:22Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06214worker2 # [ 213.627139] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:22Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.3:50051 clients=16215worker1 # [ 213.690134] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626216worker1 # [ 213.703404] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626217client # copying 1 paths to 'lb:50051'...6218worker1 # [ 213.719195] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626219client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6220worker1 # [ 213.743621] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6221worker1 # [ 213.977498] nix-daemon[848]: accepted connection from pid 1612, user nix-grpc-daemon (trusted)6222worker1 # [ 213.989208] nix-daemon[848]: reaped child process 1615, status = succeeded6223worker1 # [ 214.193236] nix-daemon[848]: accepted connection from pid 1617, user nix-grpc-daemon (trusted)6224worker1 # [ 214.204826] nix-daemon[848]: reaped child process 1620, status = succeeded6225lb # [ 214.192249] niks3-server[1003]: 2026/09/21 09:15:24 INFO Received uploads request method=POST path=/api/pending_closures6226worker1 # [ 214.236901] nix-grpc-daemon[1541]: time=2026-09-21T09:15:23.486Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6227worker1 # [ 214.238542] nix-grpc-daemon[1541]: time=2026-09-21T09:15:23.488Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6228lb # [ 214.224795] niks3-server[1003]: 2026/09/21 09:15:24 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6229lb # [ 214.235782] niks3-server[1003]: 2026/09/21 09:15:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6230worker1 # [ 214.278478] nix-grpc-daemon[1541]: time=2026-09-21T09:15:23.528Z level=INFO msg="Uploading 1 narinfos"6231lb # [ 214.243709] niks3-server[1003]: 2026/09/21 09:15:24 INFO Signed narinfos id=39 count=16232lb # [ 214.250655] niks3-server[1003]: 2026/09/21 09:15:24 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6233lb # [ 214.262682] niks3-server[1003]: 2026/09/21 09:15:24 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6234lb # [ 214.267547] niks3-server[1003]: 2026/09/21 09:15:24 INFO Completed upload id=396235worker1 # [ 214.303094] nix-grpc-daemon[1541]: time=2026-09-21T09:15:23.553Z level=INFO msg="Upload complete. (546ms)"6236lb # [ 214.270669] niks3-server[1003]: 2026/09/21 09:15:24 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6237worker1 # [ 214.308162] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=5286238worker1 # [ 214.314600] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626239worker1 # [ 214.326109] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6240worker1 # [ 214.335421] nix-daemon[848]: reaped child process 1610, status = succeeded6241worker1 # [ 214.339513] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6242client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6243client # farm-a> node-a: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6244client # farm-a> LOG-farm-a-during-drain6245worker1 # [ 215.164968] nix-daemon[848]: reaped child process 1624, status = succeeded6246worker1 # [ 215.375845] nix-daemon[848]: accepted connection from pid 1639, user nix-grpc-daemon (trusted)6247worker1 # [ 215.386189] nix-daemon[848]: reaped child process 1642, status = succeeded6248lb # [ 215.373997] niks3-server[1003]: 2026/09/21 09:15:25 INFO Received uploads request method=POST path=/api/pending_closures6249worker1 # [ 215.417513] nix-grpc-daemon[1541]: time=2026-09-21T09:15:24.667Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6250worker1 # [ 215.420056] nix-grpc-daemon[1541]: time=2026-09-21T09:15:24.669Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6251lb # [ 215.412818] niks3-server[1003]: 2026/09/21 09:15:25 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6252lb # [ 215.422791] niks3-server[1003]: 2026/09/21 09:15:25 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6253lb # [ 215.431762] niks3-server[1003]: 2026/09/21 09:15:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6254lb # [ 215.436193] niks3-server[1003]: 2026/09/21 09:15:25 INFO Signed narinfos id=40 count=16255worker1 # [ 215.474695] nix-grpc-daemon[1541]: time=2026-09-21T09:15:24.724Z level=INFO msg="Uploading 1 narinfos"6256lb # [ 215.441726] niks3-server[1003]: 2026/09/21 09:15:25 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6257lb # [ 215.450378] niks3-server[1003]: 2026/09/21 09:15:25 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6258lb # [ 215.458502] niks3-server[1003]: 2026/09/21 09:15:25 INFO Completed upload id=406259worker1 # [ 215.493719] nix-grpc-daemon[1541]: time=2026-09-21T09:15:24.743Z level=INFO msg="Upload complete. (347ms)"6260lb # [ 215.460413] niks3-server[1003]: 2026/09/21 09:15:25 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6261worker1 # [ 215.499502] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16262worker2 # [ 215.481977] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:24Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06263worker2 # [ 215.484770] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:24Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.3:50051 clients=16264worker1 # [ 215.511703] nix-daemon[848]: reaped child process 1622, status = succeeded6265worker1 # [ 215.548098] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626266worker1 # [ 215.560734] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626267worker1 # [ 215.574337] nix-grpc-daemon[1541]: time=2026-09-21T09:15:24.824Z level=INFO msg="All 1 paths already cached"6268client # copying 1 paths to 'lb:50051'...6269worker1 # [ 215.582738] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626270client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6271worker1 # [ 215.600600] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6272worker1 # [ 215.820316] nix-daemon[848]: accepted connection from pid 1646, user nix-grpc-daemon (trusted)6273worker1 # [ 215.832119] nix-daemon[848]: reaped child process 1649, status = succeeded6274worker1 # [ 216.030041] nix-daemon[848]: accepted connection from pid 1651, user nix-grpc-daemon (trusted)6275worker1 # [ 216.041748] nix-daemon[848]: reaped child process 1654, status = succeeded6276lb # [ 216.028180] niks3-server[1003]: 2026/09/21 09:15:26 INFO Received uploads request method=POST path=/api/pending_closures6277worker1 # [ 216.071934] nix-grpc-daemon[1541]: time=2026-09-21T09:15:25.321Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6278worker1 # [ 216.073335] nix-grpc-daemon[1541]: time=2026-09-21T09:15:25.321Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6279lb # [ 216.056627] niks3-server[1003]: 2026/09/21 09:15:26 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6280lb # [ 216.065582] niks3-server[1003]: 2026/09/21 09:15:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6281lb # [ 216.070791] niks3-server[1003]: 2026/09/21 09:15:26 INFO Signed narinfos id=41 count=16282worker1 # [ 216.108349] nix-grpc-daemon[1541]: time=2026-09-21T09:15:25.358Z level=INFO msg="Uploading 1 narinfos"6283lb # [ 216.076510] niks3-server[1003]: 2026/09/21 09:15:26 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6284lb # [ 216.088687] niks3-server[1003]: 2026/09/21 09:15:26 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6285lb # [ 216.093520] niks3-server[1003]: 2026/09/21 09:15:26 INFO Completed upload id=416286worker1 # [ 216.130551] nix-grpc-daemon[1541]: time=2026-09-21T09:15:25.380Z level=INFO msg="Upload complete. (513ms)"6287lb # [ 216.096474] niks3-server[1003]: 2026/09/21 09:15:26 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6288worker1 # [ 216.132919] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=6406289worker1 # [ 216.137915] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626290worker1 # [ 216.149228] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6291worker1 # [ 216.152187] nix-daemon[848]: reaped child process 1644, status = succeeded6292worker1 # [ 216.156976] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6293client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6294client # farm-b> node-a: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6295client # farm-b> LOG-farm-b-during-drain6296worker1 # [ 217.182520] nix-daemon[848]: reaped child process 1658, status = succeeded6297worker1 # [ 217.384690] nix-daemon[848]: accepted connection from pid 1673, user nix-grpc-daemon (trusted)6298worker1 # [ 217.396687] nix-daemon[848]: reaped child process 1676, status = succeeded6299lb # [ 217.382591] niks3-server[1003]: 2026/09/21 09:15:27 INFO Received uploads request method=POST path=/api/pending_closures6300worker1 # [ 217.426221] nix-grpc-daemon[1541]: time=2026-09-21T09:15:26.676Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6301worker1 # [ 217.427671] nix-grpc-daemon[1541]: time=2026-09-21T09:15:26.677Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6302lb # [ 217.419621] niks3-server[1003]: 2026/09/21 09:15:27 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6303lb # [ 217.428115] niks3-server[1003]: 2026/09/21 09:15:27 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6304lb # [ 217.437475] niks3-server[1003]: 2026/09/21 09:15:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6305lb # [ 217.442904] niks3-server[1003]: 2026/09/21 09:15:27 INFO Signed narinfos id=42 count=16306worker1 # [ 217.480336] nix-grpc-daemon[1541]: time=2026-09-21T09:15:26.730Z level=INFO msg="Uploading 1 narinfos"6307lb # [ 217.448378] niks3-server[1003]: 2026/09/21 09:15:27 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6308lb # [ 217.459457] niks3-server[1003]: 2026/09/21 09:15:27 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6309lb # [ 217.464237] niks3-server[1003]: 2026/09/21 09:15:27 INFO Completed upload id=426310lb # [ 217.467495] niks3-server[1003]: 2026/09/21 09:15:27 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6311worker1 # [ 217.501769] nix-grpc-daemon[1541]: time=2026-09-21T09:15:26.751Z level=INFO msg="Upload complete. (333ms)"6312worker2 # [ 217.491183] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:26Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06313worker1 # [ 217.506963] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16314worker2 # [ 217.492988] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:26Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.3:50051 clients=16315worker1 # [ 217.521711] nix-daemon[848]: reaped child process 1656, status = succeeded6316worker1 # [ 217.556971] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626317worker1 # [ 217.569332] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626318worker1 # [ 217.582142] nix-grpc-daemon[1541]: time=2026-09-21T09:15:26.832Z level=INFO msg="All 2 paths already cached"6319client # copying 1 paths to 'lb:50051'...6320client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6321worker1 # [ 217.590642] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626322worker1 # [ 217.598791] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6323worker1 # [ 217.806436] nix-daemon[848]: accepted connection from pid 1680, user nix-grpc-daemon (trusted)6324worker1 # [ 217.818387] nix-daemon[848]: reaped child process 1683, status = succeeded6325worker1 # [ 218.023399] nix-daemon[848]: accepted connection from pid 1685, user nix-grpc-daemon (trusted)6326worker1 # [ 218.034984] nix-daemon[848]: reaped child process 1688, status = succeeded6327lb # [ 218.021981] niks3-server[1003]: 2026/09/21 09:15:28 INFO Received uploads request method=POST path=/api/pending_closures6328worker1 # [ 218.065283] nix-grpc-daemon[1541]: time=2026-09-21T09:15:27.315Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6329worker1 # [ 218.067161] nix-grpc-daemon[1541]: time=2026-09-21T09:15:27.316Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6330lb # [ 218.052146] niks3-server[1003]: 2026/09/21 09:15:28 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6331lb # [ 218.061739] niks3-server[1003]: 2026/09/21 09:15:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6332worker1 # [ 218.102182] nix-grpc-daemon[1541]: time=2026-09-21T09:15:27.352Z level=INFO msg="Uploading 1 narinfos"6333lb # [ 218.067148] niks3-server[1003]: 2026/09/21 09:15:28 INFO Signed narinfos id=43 count=16334lb # [ 218.074780] niks3-server[1003]: 2026/09/21 09:15:28 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6335lb # [ 218.084814] niks3-server[1003]: 2026/09/21 09:15:28 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6336worker1 # [ 218.124310] nix-grpc-daemon[1541]: time=2026-09-21T09:15:27.374Z level=INFO msg="Upload complete. (510ms)"6337lb # [ 218.089606] niks3-server[1003]: 2026/09/21 09:15:28 INFO Completed upload id=436338worker1 # [ 218.126316] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=7686339lb # [ 218.093484] niks3-server[1003]: 2026/09/21 09:15:28 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6340worker1 # [ 218.132224] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626341worker1 # [ 218.141329] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6342worker1 # [ 218.143540] nix-daemon[848]: reaped child process 1678, status = succeeded6343worker1 # [ 218.149054] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6344client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6345client # farm-top> node-a: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6346client # farm-top> LOG-farm-top-during-drain6347worker1 # [ 219.363086] nix-daemon[848]: reaped child process 1692, status = succeeded6348worker1 # [ 219.551371] nix-daemon[848]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6349worker1 # [ 219.562819] nix-daemon[848]: reaped child process 1710, status = succeeded6350lb # [ 219.550784] niks3-server[1003]: 2026/09/21 09:15:29 INFO Received uploads request method=POST path=/api/pending_closures6351worker1 # [ 219.594477] nix-grpc-daemon[1541]: time=2026-09-21T09:15:28.844Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6352worker1 # [ 219.596047] nix-grpc-daemon[1541]: time=2026-09-21T09:15:28.846Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6353lb # [ 219.588176] niks3-server[1003]: 2026/09/21 09:15:29 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6354lb # [ 219.596073] niks3-server[1003]: 2026/09/21 09:15:29 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6355lb # [ 219.608464] niks3-server[1003]: 2026/09/21 09:15:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6356lb # [ 219.613876] niks3-server[1003]: 2026/09/21 09:15:29 INFO Signed narinfos id=44 count=16357worker1 # [ 219.651226] nix-grpc-daemon[1541]: time=2026-09-21T09:15:28.901Z level=INFO msg="Uploading 1 narinfos"6358lb # [ 219.619215] niks3-server[1003]: 2026/09/21 09:15:29 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6359lb # [ 219.629089] niks3-server[1003]: 2026/09/21 09:15:29 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6360lb # [ 219.634661] niks3-server[1003]: 2026/09/21 09:15:29 INFO Completed upload id=446361worker1 # [ 219.669710] nix-grpc-daemon[1541]: time=2026-09-21T09:15:28.919Z level=INFO msg="Upload complete. (313ms)"6362lb # [ 219.636508] niks3-server[1003]: 2026/09/21 09:15:29 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6363worker1 # [ 219.676214] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16364worker1 # [ 219.688303] nix-daemon[848]: reaped child process 1690, status = succeeded6365client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 6.39 seconds)6366client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6367client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.24 seconds)6368worker2: must succeed: pkill -f 'read -t [9]0 x'6369worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6370client: waiting for success: ! systemctl is-active sw6371worker2 # [ 220.054833] nix-daemon[869]: reaped child process 1548, status = succeeded6372worker2 # [ 220.227298] nix-daemon[869]: accepted connection from pid 1725, user nix-grpc-daemon (trusted)6373worker2 # [ 220.238951] nix-daemon[869]: reaped child process 1728, status = succeeded6374lb # [ 220.244457] niks3-server[1003]: 2026/09/21 09:15:30 INFO Received uploads request method=POST path=/api/pending_closures6375worker2 # [ 220.267307] nix-grpc-daemon[1486]: time=2026-09-21T09:15:29.477Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6376worker2 # [ 220.268799] nix-grpc-daemon[1486]: time=2026-09-21T09:15:29.479Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6377lb # [ 220.279949] niks3-server[1003]: 2026/09/21 09:15:30 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6378lb # [ 220.292229] niks3-server[1003]: 2026/09/21 09:15:30 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6379lb # [ 220.298967] niks3-server[1003]: 2026/09/21 09:15:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6380worker2 # [ 220.317547] nix-grpc-daemon[1486]: time=2026-09-21T09:15:29.527Z level=INFO msg="Uploading 1 narinfos"6381lb # [ 220.303356] niks3-server[1003]: 2026/09/21 09:15:30 INFO Signed narinfos id=45 count=16382lb # [ 220.307352] niks3-server[1003]: 2026/09/21 09:15:30 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6383lb # [ 220.314796] niks3-server[1003]: 2026/09/21 09:15:30 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6384lb # [ 220.318126] niks3-server[1003]: 2026/09/21 09:15:30 INFO Completed upload id=456385worker2 # [ 220.332294] nix-grpc-daemon[1486]: time=2026-09-21T09:15:29.542Z level=INFO msg="Upload complete. (284ms)"6386lb # [ 220.320315] niks3-server[1003]: 2026/09/21 09:15:30 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6387worker2 # [ 220.338534] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41644 duration_s=12 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16388worker2 # [ 220.349723] nix-daemon[869]: reaped child process 1546, status = succeeded6389client # [ 220.368811] systemd[1]: sw.service: Deactivated successfully.6390client # [ 220.372192] systemd[1]: sw.service: Consumed 129ms CPU time over 13.458s wall clock time, 43.4M memory peak, 6.9K incoming IP traffic, 8.2K outgoing IP traffic.6391worker2 # [ 220.764128] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:29Z level=info event=signal_exit6392worker2 # [ 221.054467] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=scheduler_yield6393worker1 # [ 221.079178] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:30Z level=info event=scheduler_restarting addr=lb:500516394worker2 # [ 221.058861] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:416286395lb # [ 221.043061] niks3-server[1003]: 2026/09/21 09:15:31 INFO lead: released remote=192.168.1.4:498226396worker2 # [ 221.063179] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=worker_gone addr=192.168.1.3:500516397worker2 # [ 221.067298] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=scheduler_restarting addr=lb:500516398worker2 # [ 221.070361] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:416286399worker2 # [ 221.073649] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=worker_gone addr=192.168.1.4:500516400client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)6401client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6402client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds)6403worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6404worker2 # [ 221.207262] systemd[1]: Starting scheduler-token.service...6405worker1 # [ 221.279511] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:30Z level=info event=scheduler_connected addr=lb:500516406worker2 # [ 221.264277] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=scheduler_connected addr=lb:500516407worker1 # [ 221.283626] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:30Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6408worker2 # [ 221.266697] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6409worker2 # [ 221.284717] systemd[1]: scheduler-token.service: Deactivated successfully.6410worker2 # [ 221.286255] systemd[1]: Finished scheduler-token.service.6411worker2 # [ 221.287585] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 79ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6412worker1 # [ 221.484594] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:30Z level=info event=scheduler_connected addr=lb:500516413worker1 # [ 221.488673] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:30Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6414worker2 # [ 221.470172] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=scheduler_connected addr=lb:500516415worker2 # [ 221.473292] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6416worker1 # [ 221.689504] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:30Z level=info event=scheduler_connected addr=lb:500516417worker2 # [ 221.671401] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=scheduler_connected addr=lb:500516418worker1 # [ 221.693459] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6419worker2 # [ 221.675181] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6420lb # [ 221.819498] niks3-server[1003]: 2026/09/21 09:15:31 INFO lead: acquired remote=192.168.1.3:551046421worker1 # [ 221.858533] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:31Z level=info event=scheduler_take_over6422worker1 # [ 221.894192] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:31Z level=info event=scheduler_connected addr=lb:500516423worker2 # [ 221.876388] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:31Z level=info event=scheduler_connected addr=lb:500516424worker1 # [ 221.898120] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6425worker2 # [ 221.880245] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6426worker2 # [ 222.058964] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:31Z level=info event=scheduler_restarting6427worker1 # [ 222.100364] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:31Z level=info event=scheduler_connected addr=lb:500516428worker1 # [ 222.103484] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6429worker2 # [ 222.085753] nix-daemon[869]: reaped child process 1454, status = succeeded6430worker2 # [ 222.095387] nix-grpc-daemon[1380]: ts=2026-09-21T09:15:31Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6431worker1 # [ 222.301528] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:31Z level=info event=scheduler_connected addr=lb:500516432worker1 # [ 222.305513] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6433worker1 # [ 222.506444] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:31Z level=info event=scheduler_connected addr=lb:500516434worker1 # [ 222.510408] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6435worker2 # [ 222.623137] nix-grpc-daemon[1380]: E0000 00:00:1789982131.832466 1425 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true6436worker2 # [ 222.627880] nix-grpc-daemon[1380]: E0000 00:00:1789982131.832762 1425 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6437worker1 # [ 222.711381] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:31Z level=info event=scheduler_connected addr=lb:500516438worker1 # [ 222.715710] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6439worker1 # [ 222.916517] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:32Z level=info event=scheduler_connected addr=lb:500516440worker1 # [ 222.920547] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6441worker1 # [ 223.122351] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:32Z level=info event=scheduler_connected addr=lb:500516442worker1 # [ 223.126137] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6443worker1 # [ 223.327315] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:32Z level=info event=scheduler_connected addr=lb:500516444worker1 # [ 223.331216] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6445worker1 # [ 223.532144] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:32Z level=info event=scheduler_connected addr=lb:500516446worker1 # [ 223.536148] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6447worker1 # [ 223.736831] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:32Z level=info event=scheduler_connected addr=lb:500516448worker1 # [ 223.748128] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:32Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:434346449worker1 # [ 223.752266] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:33Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06450worker2 # [ 223.872495] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6451worker2 # [ 223.876471] systemd[1]: nix-grpc-daemon.service: Consumed 889ms CPU time over 2min 2.336s wall clock time, 33.1M memory peak, 29.3K incoming IP traffic, 47.8K outgoing IP traffic.6452worker2 # [ 223.892845] systemd[1]: Starting Nix worker-protocol over gRPC...6453worker2 # [ 224.082916] nix-grpc-daemon[1745]: ts=2026-09-21T09:15:33Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6454worker2 # [ 224.085473] nix-grpc-daemon[1745]: ts=2026-09-21T09:15:33Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6455worker2 # [ 224.089373] nix-grpc-daemon[1745]: ts=2026-09-21T09:15:33Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16456worker2 # [ 224.097065] nix-grpc-daemon[1745]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6457worker2 # [ 224.098675] nix-grpc-daemon[1745]: E0000 00:00:1789982133.306761 1745 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6458worker2 # [ 224.103528] nix-grpc-daemon[1745]: E0000 00:00:1789982133.312065 1745 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6459worker2 # [ 224.107287] nix-grpc-daemon[1745]: ts=2026-09-21T09:15:33Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6460worker2 # [ 224.110574] systemd[1]: Started Nix worker-protocol over gRPC.6461worker2 # [ 224.117102] nix-grpc-daemon[1745]: E0000 00:00:1789982133.326526 1787 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6462worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.99 seconds)6463lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6464lb: must succeed: curl -sf localhost:9901/clusters6465worker2 # [ 224.130052] nix-grpc-daemon[1745]: E0000 00:00:1789982133.339396 1787 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6466worker2 # [ 224.133101] nix-grpc-daemon[1745]: E0000 00:00:1789982133.339599 1787 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6467worker2 # [ 224.143127] nix-grpc-daemon[1745]: E0000 00:00:1789982133.353565 1787 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6468worker2 # [ 224.154218] nix-grpc-daemon[1745]: ts=2026-09-21T09:15:33Z level=info event=scheduler_connected addr=lb:500516469worker1 # [ 224.184866] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:33Z level=info event=oidc_jwks provider=mock keys=16470worker1 # [ 224.189522] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:33Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:434346471worker1 # [ 224.193696] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:33Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06472lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6473lb: must succeed: curl -sf localhost:9901/clusters6474lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6475lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.14 seconds)6476lb: must succeed: curl -sf localhost:9901/clusters6477lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6478worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6479worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6480(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 18.50 seconds)6481subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6482client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16483worker1 # [ 225.671471] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:503626484worker1 # [ 225.676752] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626485worker1 # [ 225.721891] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:34Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06486worker1 # [ 225.727106] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:34Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16487worker1 # [ 225.731778] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:34Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06488worker1 # [ 225.735943] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:34Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06489worker1 # [ 225.766829] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626490worker1 # [ 225.780115] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626491worker1 # [ 225.791889] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626492worker1 # [ 225.804174] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6493worker1 # [ 226.015337] nix-daemon[848]: accepted connection from pid 1719, user nix-grpc-daemon (trusted)6494worker1 # [ 226.027166] nix-daemon[848]: reaped child process 1722, status = succeeded6495worker1 # [ 226.213632] nix-daemon[848]: accepted connection from pid 1724, user nix-grpc-daemon (trusted)6496worker1 # [ 226.225219] nix-daemon[848]: reaped child process 1727, status = succeeded6497lb # [ 226.209785] niks3-server[1003]: 2026/09/21 09:15:36 INFO Received uploads request method=POST path=/api/pending_closures6498worker1 # [ 226.251242] nix-grpc-daemon[1541]: time=2026-09-21T09:15:35.500Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6499worker1 # [ 226.252570] nix-grpc-daemon[1541]: time=2026-09-21T09:15:35.500Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6500lb # [ 226.233244] niks3-server[1003]: 2026/09/21 09:15:36 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6501lb # [ 226.243541] niks3-server[1003]: 2026/09/21 09:15:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6502worker1 # [ 226.284343] nix-grpc-daemon[1541]: time=2026-09-21T09:15:35.534Z level=INFO msg="Uploading 1 narinfos"6503lb # [ 226.249323] niks3-server[1003]: 2026/09/21 09:15:36 INFO Signed narinfos id=46 count=16504lb # [ 226.255578] niks3-server[1003]: 2026/09/21 09:15:36 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6505lb # [ 226.268553] niks3-server[1003]: 2026/09/21 09:15:36 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6506worker1 # [ 226.308045] nix-grpc-daemon[1541]: time=2026-09-21T09:15:35.558Z level=INFO msg="Upload complete. (491ms)"6507lb # [ 226.273537] niks3-server[1003]: 2026/09/21 09:15:36 INFO Completed upload id=466508worker1 # [ 226.311072] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=5046509lb # [ 226.276479] niks3-server[1003]: 2026/09/21 09:15:36 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6510worker1 # [ 226.316415] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626511worker1 # [ 226.320598] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6512worker1 # [ 226.322789] nix-daemon[848]: reaped child process 1717, status = succeeded6513worker1 # [ 226.328168] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6514worker1 # [ 227.141555] nix-daemon[848]: reaped child process 1731, status = succeeded6515worker1 # [ 227.313410] nix-daemon[848]: accepted connection from pid 1746, user nix-grpc-daemon (trusted)6516worker1 # [ 227.323742] nix-daemon[848]: reaped child process 1749, status = succeeded6517lb # [ 227.307210] niks3-server[1003]: 2026/09/21 09:15:37 INFO Received uploads request method=POST path=/api/pending_closures6518worker1 # [ 227.354401] nix-grpc-daemon[1541]: time=2026-09-21T09:15:36.604Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6519worker1 # [ 227.355850] nix-grpc-daemon[1541]: time=2026-09-21T09:15:36.606Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6520lb # [ 227.352707] niks3-server[1003]: 2026/09/21 09:15:37 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6521lb # [ 227.358307] niks3-server[1003]: 2026/09/21 09:15:37 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6522lb # [ 227.370274] niks3-server[1003]: 2026/09/21 09:15:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6523lb # [ 227.374450] niks3-server[1003]: 2026/09/21 09:15:37 INFO Signed narinfos id=47 count=16524worker1 # [ 227.412144] nix-grpc-daemon[1541]: time=2026-09-21T09:15:36.661Z level=INFO msg="Uploading 1 narinfos"6525lb # [ 227.379518] niks3-server[1003]: 2026/09/21 09:15:37 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6526lb # [ 227.388830] niks3-server[1003]: 2026/09/21 09:15:37 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6527lb # [ 227.394445] niks3-server[1003]: 2026/09/21 09:15:37 INFO Completed upload id=476528worker1 # [ 227.429575] nix-grpc-daemon[1541]: time=2026-09-21T09:15:36.679Z level=INFO msg="Upload complete. (294ms)"6529lb # [ 227.397168] niks3-server[1003]: 2026/09/21 09:15:37 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6530worker1 # [ 227.437505] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=16531worker1 # [ 227.457183] nix-daemon[848]: reaped child process 1729, status = succeeded6532worker1 # [ 227.465160] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:36Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16533worker1 # [ 227.469555] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:36Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06534worker1 # [ 227.510624] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626535worker1 # [ 227.523110] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626536worker1 # [ 227.535103] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626537worker1 # [ 227.546141] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6538worker1 # [ 227.746169] nix-daemon[848]: accepted connection from pid 1755, user nix-grpc-daemon (trusted)6539worker1 # [ 227.758449] nix-daemon[848]: reaped child process 1758, status = succeeded6540worker1 # [ 227.947694] nix-daemon[848]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6541worker1 # [ 227.958969] nix-daemon[848]: reaped child process 1763, status = succeeded6542lb # [ 227.941953] niks3-server[1003]: 2026/09/21 09:15:37 INFO Received uploads request method=POST path=/api/pending_closures6543worker1 # [ 227.984825] nix-grpc-daemon[1541]: time=2026-09-21T09:15:37.234Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6544worker1 # [ 227.986442] nix-grpc-daemon[1541]: time=2026-09-21T09:15:37.236Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6545lb # [ 227.972149] niks3-server[1003]: 2026/09/21 09:15:38 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6546lb # [ 227.980987] niks3-server[1003]: 2026/09/21 09:15:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6547lb # [ 227.986551] niks3-server[1003]: 2026/09/21 09:15:38 INFO Signed narinfos id=48 count=16548worker1 # [ 228.024393] nix-grpc-daemon[1541]: time=2026-09-21T09:15:37.274Z level=INFO msg="Uploading 1 narinfos"6549lb # [ 227.992636] niks3-server[1003]: 2026/09/21 09:15:38 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6550lb # [ 228.005913] niks3-server[1003]: 2026/09/21 09:15:38 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6551worker1 # [ 228.045198] nix-grpc-daemon[1541]: time=2026-09-21T09:15:37.295Z level=INFO msg="Upload complete. (487ms)"6552lb # [ 228.010521] niks3-server[1003]: 2026/09/21 09:15:38 INFO Completed upload id=486553lb # [ 228.013657] niks3-server[1003]: 2026/09/21 09:15:38 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6554worker1 # [ 228.047925] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=5046555worker1 # [ 228.052926] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626556worker1 # [ 228.063484] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6557worker1 # [ 228.066452] nix-daemon[848]: reaped child process 1753, status = succeeded6558worker1 # [ 228.071542] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6559lb # [ 228.138478] envoy[918]: conn peer=[2001:db8:1::1]:50312 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9598 tx=3819 ms=60856560lb # [ 228.142231] envoy[918]: conn peer=[2001:db8:1::1]:41982 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2932 tx=1334 ms=131766561lb # [ 228.145718] envoy[918]: conn peer=[2001:db8:1::4]:33494 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=12352 tx=1007 ms=1203146562worker1 # [ 229.083963] nix-daemon[848]: reaped child process 1767, status = succeeded6563worker1 # [ 229.262075] nix-daemon[848]: accepted connection from pid 1782, user nix-grpc-daemon (trusted)6564worker1 # [ 229.273701] nix-daemon[848]: reaped child process 1785, status = succeeded6565lb # [ 229.256800] niks3-server[1003]: 2026/09/21 09:15:39 INFO Received uploads request method=POST path=/api/pending_closures6566worker1 # [ 229.302282] nix-grpc-daemon[1541]: time=2026-09-21T09:15:38.550Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6567worker1 # [ 229.303620] nix-grpc-daemon[1541]: time=2026-09-21T09:15:38.550Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6568lb # [ 229.300152] niks3-server[1003]: 2026/09/21 09:15:39 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6569lb # [ 229.308128] niks3-server[1003]: 2026/09/21 09:15:39 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6570lb # [ 229.321275] niks3-server[1003]: 2026/09/21 09:15:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6571lb # [ 229.327239] niks3-server[1003]: 2026/09/21 09:15:39 INFO Signed narinfos id=49 count=16572worker1 # [ 229.364517] nix-grpc-daemon[1541]: time=2026-09-21T09:15:38.614Z level=INFO msg="Uploading 1 narinfos"6573lb # [ 229.329713] niks3-server[1003]: 2026/09/21 09:15:39 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6574lb # [ 229.338549] niks3-server[1003]: 2026/09/21 09:15:39 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6575lb # [ 229.342580] niks3-server[1003]: 2026/09/21 09:15:39 INFO Completed upload id=496576worker1 # [ 229.377465] nix-grpc-daemon[1541]: time=2026-09-21T09:15:38.627Z level=INFO msg="Upload complete. (299ms)"6577lb # [ 229.344355] niks3-server[1003]: 2026/09/21 09:15:39 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6578worker1 # [ 229.390246] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=16579worker1 # [ 229.395220] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:38Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16580worker1 # [ 229.398597] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626581worker1 # [ 229.401572] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:38Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06582worker1 # [ 229.411724] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626583worker1 # [ 229.413933] nix-daemon[848]: reaped child process 1765, status = succeeded6584worker1 # [ 229.418238] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626585worker1 # [ 229.422179] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6586worker1 # [ 229.611109] nix-daemon[848]: accepted connection from pid 1791, user nix-grpc-daemon (trusted)6587worker1 # [ 229.622716] nix-daemon[848]: reaped child process 1794, status = succeeded6588worker1 # [ 229.820659] nix-daemon[848]: accepted connection from pid 1796, user nix-grpc-daemon (trusted)6589worker1 # [ 229.832230] nix-daemon[848]: reaped child process 1799, status = succeeded6590lb # [ 229.816413] niks3-server[1003]: 2026/09/21 09:15:39 INFO Received uploads request method=POST path=/api/pending_closures6591worker1 # [ 229.859968] nix-grpc-daemon[1541]: time=2026-09-21T09:15:39.109Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6592worker1 # [ 229.861384] nix-grpc-daemon[1541]: time=2026-09-21T09:15:39.109Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6593lb # [ 229.842851] niks3-server[1003]: 2026/09/21 09:15:39 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6594lb # [ 229.856278] niks3-server[1003]: 2026/09/21 09:15:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6595lb # [ 229.862126] niks3-server[1003]: 2026/09/21 09:15:39 INFO Signed narinfos id=50 count=16596worker1 # [ 229.900670] nix-grpc-daemon[1541]: time=2026-09-21T09:15:39.150Z level=INFO msg="Uploading 1 narinfos"6597lb # [ 229.870126] niks3-server[1003]: 2026/09/21 09:15:39 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6598lb # [ 229.881157] niks3-server[1003]: 2026/09/21 09:15:39 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6599worker1 # [ 229.921771] nix-grpc-daemon[1541]: time=2026-09-21T09:15:39.171Z level=INFO msg="Upload complete. (494ms)"6600lb # [ 229.887181] niks3-server[1003]: 2026/09/21 09:15:39 INFO Completed upload id=506601lb # [ 229.890592] niks3-server[1003]: 2026/09/21 09:15:39 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6602worker1 # [ 229.924899] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=5046603worker1 # [ 229.930250] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626604worker1 # [ 229.940108] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6605worker1 # [ 229.943942] nix-daemon[848]: reaped child process 1789, status = succeeded6606worker1 # [ 229.949062] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6607worker1 # [ 230.974583] nix-daemon[848]: reaped child process 1803, status = succeeded6608worker1 # [ 231.164318] nix-daemon[848]: accepted connection from pid 1818, user nix-grpc-daemon (trusted)6609worker1 # [ 231.176071] nix-daemon[848]: reaped child process 1821, status = succeeded6610lb # [ 231.161683] niks3-server[1003]: 2026/09/21 09:15:41 INFO Received uploads request method=POST path=/api/pending_closures6611worker1 # [ 231.205797] nix-grpc-daemon[1541]: time=2026-09-21T09:15:40.455Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6612worker1 # [ 231.207299] nix-grpc-daemon[1541]: time=2026-09-21T09:15:40.457Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"6613lb # [ 231.197294] niks3-server[1003]: 2026/09/21 09:15:41 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv6614lb # [ 231.211261] niks3-server[1003]: 2026/09/21 09:15:41 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6615lb # [ 231.222945] niks3-server[1003]: 2026/09/21 09:15:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6616lb # [ 231.230787] niks3-server[1003]: 2026/09/21 09:15:41 INFO Signed narinfos id=51 count=16617worker1 # [ 231.268970] nix-grpc-daemon[1541]: time=2026-09-21T09:15:40.518Z level=INFO msg="Uploading 1 narinfos"6618lb # [ 231.236457] niks3-server[1003]: 2026/09/21 09:15:41 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls6619lb # [ 231.245513] niks3-server[1003]: 2026/09/21 09:15:41 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6620lb # [ 231.251462] niks3-server[1003]: 2026/09/21 09:15:41 INFO Completed upload id=516621worker1 # [ 231.286645] nix-grpc-daemon[1541]: time=2026-09-21T09:15:40.536Z level=INFO msg="Upload complete. (330ms)"6622lb # [ 231.255598] niks3-server[1003]: 2026/09/21 09:15:41 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo6623worker1 # [ 231.295607] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=16624worker1 # [ 231.302192] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:40Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=16625worker1 # [ 231.312330] nix-daemon[848]: reaped child process 1801, status = succeeded6626worker1 # [ 231.337748] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626627worker1 # [ 231.348944] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:40Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=06628worker1 # [ 231.357655] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626629worker1 # [ 231.370557] nix-grpc-daemon[1541]: time=2026-09-21T09:15:40.620Z level=INFO msg="All 1 paths already cached"6630worker1 # [ 231.377766] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626631worker1 # [ 231.386259] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6632worker1 # [ 231.585110] nix-daemon[848]: accepted connection from pid 1827, user nix-grpc-daemon (trusted)6633worker1 # [ 231.596772] nix-daemon[848]: reaped child process 1830, status = succeeded6634worker1 # [ 231.797273] nix-daemon[848]: accepted connection from pid 1832, user nix-grpc-daemon (trusted)6635worker1 # [ 231.808711] nix-daemon[848]: reaped child process 1835, status = succeeded6636lb # [ 231.794660] niks3-server[1003]: 2026/09/21 09:15:41 INFO Received uploads request method=POST path=/api/pending_closures6637worker1 # [ 231.841284] nix-grpc-daemon[1541]: time=2026-09-21T09:15:41.091Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6638worker1 # [ 231.842648] nix-grpc-daemon[1541]: time=2026-09-21T09:15:41.091Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"6639lb # [ 231.826818] niks3-server[1003]: 2026/09/21 09:15:41 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst6640lb # [ 231.835804] niks3-server[1003]: 2026/09/21 09:15:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6641lb # [ 231.840197] niks3-server[1003]: 2026/09/21 09:15:41 INFO Signed narinfos id=52 count=16642worker1 # [ 231.879387] nix-grpc-daemon[1541]: time=2026-09-21T09:15:41.129Z level=INFO msg="Uploading 1 narinfos"6643lb # [ 231.847474] niks3-server[1003]: 2026/09/21 09:15:41 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls6644lb # [ 231.858221] niks3-server[1003]: 2026/09/21 09:15:41 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6645lb # [ 231.860929] niks3-server[1003]: 2026/09/21 09:15:41 INFO Completed upload id=526646lb # [ 231.862692] niks3-server[1003]: 2026/09/21 09:15:41 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo6647worker1 # [ 231.897147] nix-grpc-daemon[1541]: time=2026-09-21T09:15:41.147Z level=INFO msg="Upload complete. (498ms)"6648worker1 # [ 231.900748] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=6246649worker1 # [ 231.906255] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626650worker1 # [ 231.914597] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6651worker1 # [ 231.919427] nix-daemon[848]: reaped child process 1825, status = succeeded6652worker1 # [ 231.925421] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6653worker1 # [ 233.550600] nix-daemon[848]: reaped child process 1839, status = succeeded6654worker1 # [ 233.744272] nix-daemon[848]: accepted connection from pid 1854, user nix-grpc-daemon (trusted)6655worker1 # [ 233.756081] nix-daemon[848]: reaped child process 1857, status = succeeded6656lb # [ 233.740758] niks3-server[1003]: 2026/09/21 09:15:43 INFO Received uploads request method=POST path=/api/pending_closures6657worker1 # [ 233.786798] nix-grpc-daemon[1541]: time=2026-09-21T09:15:43.036Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6658worker1 # [ 233.788276] nix-grpc-daemon[1541]: time=2026-09-21T09:15:43.038Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"6659lb # [ 233.780366] niks3-server[1003]: 2026/09/21 09:15:43 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst6660lb # [ 233.792496] niks3-server[1003]: 2026/09/21 09:15:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6661lb # [ 233.799374] niks3-server[1003]: 2026/09/21 09:15:43 INFO Signed narinfos id=53 count=16662worker1 # [ 233.837186] nix-grpc-daemon[1541]: time=2026-09-21T09:15:43.087Z level=INFO msg="Uploading 1 narinfos"6663lb # [ 233.806960] niks3-server[1003]: 2026/09/21 09:15:43 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls6664lb # [ 233.816349] niks3-server[1003]: 2026/09/21 09:15:43 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6665lb # [ 233.821381] niks3-server[1003]: 2026/09/21 09:15:43 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv6666lb # [ 233.825032] niks3-server[1003]: 2026/09/21 09:15:43 INFO Completed upload id=536667worker1 # [ 233.859245] nix-grpc-daemon[1541]: time=2026-09-21T09:15:43.109Z level=INFO msg="Upload complete. (325ms)"6668worker1 # [ 233.869797] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=16669worker1 # [ 233.876256] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:43Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=16670worker1 # [ 233.879674] nix-grpc-daemon[1541]: time=2026-09-21T09:15:43.117Z level=WARN msg="Failed to register uploaded object" key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo error="server returned 404: object is not pending upload\n"6671worker1 # [ 233.884113] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626672worker1 # [ 233.886595] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:43Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=06673worker1 # [ 233.892835] nix-daemon[848]: reaped child process 1837, status = succeeded6674worker1 # [ 233.895375] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626675worker1 # [ 233.903818] nix-grpc-daemon[1541]: time=2026-09-21T09:15:43.153Z level=INFO msg="All 1 paths already cached"6676worker1 # [ 233.908143] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626677worker1 # [ 233.912201] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6678worker1 # [ 234.096609] nix-daemon[848]: accepted connection from pid 1863, user nix-grpc-daemon (trusted)6679worker1 # [ 234.108510] nix-daemon[848]: reaped child process 1866, status = succeeded6680worker1 # [ 234.310554] nix-daemon[848]: accepted connection from pid 1868, user nix-grpc-daemon (trusted)6681worker1 # [ 234.321993] nix-daemon[848]: reaped child process 1871, status = succeeded6682lb # [ 234.307219] niks3-server[1003]: 2026/09/21 09:15:44 INFO Received uploads request method=POST path=/api/pending_closures6683worker1 # [ 234.353675] nix-grpc-daemon[1541]: time=2026-09-21T09:15:43.603Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6684worker1 # [ 234.355100] nix-grpc-daemon[1541]: time=2026-09-21T09:15:43.603Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"6685lb # [ 234.342230] niks3-server[1003]: 2026/09/21 09:15:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6686worker1 # [ 234.384544] nix-grpc-daemon[1541]: time=2026-09-21T09:15:43.634Z level=INFO msg="Uploading 1 narinfos"6687lb # [ 234.349568] niks3-server[1003]: 2026/09/21 09:15:44 INFO Signed narinfos id=54 count=16688lb # [ 234.358499] niks3-server[1003]: 2026/09/21 09:15:44 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst6689lb # [ 234.364397] niks3-server[1003]: 2026/09/21 09:15:44 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls6690lb # [ 234.374110] niks3-server[1003]: 2026/09/21 09:15:44 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6691lb # [ 234.378827] niks3-server[1003]: 2026/09/21 09:15:44 INFO Completed upload id=546692worker1 # [ 234.416100] nix-grpc-daemon[1541]: time=2026-09-21T09:15:43.666Z level=INFO msg="Upload complete. (497ms)"6693lb # [ 234.382736] niks3-server[1003]: 2026/09/21 09:15:44 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo6694worker1 # [ 234.420112] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=6246695worker1 # [ 234.425364] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626696worker1 # [ 234.434826] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6697worker1 # [ 234.443461] nix-daemon[848]: reaped child process 1861, status = succeeded6698worker1 # [ 234.449106] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6699worker1 # [ 235.872454] nix-daemon[848]: reaped child process 1875, status = succeeded6700worker1 # [ 236.071079] nix-daemon[848]: accepted connection from pid 1890, user nix-grpc-daemon (trusted)6701worker1 # [ 236.082758] nix-daemon[848]: reaped child process 1893, status = succeeded6702lb # [ 236.067822] niks3-server[1003]: 2026/09/21 09:15:46 INFO Received uploads request method=POST path=/api/pending_closures6703worker1 # [ 236.115280] nix-grpc-daemon[1541]: time=2026-09-21T09:15:45.364Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6704worker1 # [ 236.116665] nix-grpc-daemon[1541]: time=2026-09-21T09:15:45.364Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"6705lb # [ 236.108732] niks3-server[1003]: 2026/09/21 09:15:46 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv6706lb # [ 236.119482] niks3-server[1003]: 2026/09/21 09:15:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6707lb # [ 236.125897] niks3-server[1003]: 2026/09/21 09:15:46 INFO Signed narinfos id=55 count=16708worker1 # [ 236.164024] nix-grpc-daemon[1541]: time=2026-09-21T09:15:45.414Z level=INFO msg="Uploading 1 narinfos"6709lb # [ 236.129648] niks3-server[1003]: 2026/09/21 09:15:46 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst6710lb # [ 236.134867] niks3-server[1003]: 2026/09/21 09:15:46 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls6711lb # [ 236.146130] niks3-server[1003]: 2026/09/21 09:15:46 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6712lb # [ 236.150917] niks3-server[1003]: 2026/09/21 09:15:46 INFO Completed upload id=556713worker1 # [ 236.185857] nix-grpc-daemon[1541]: time=2026-09-21T09:15:45.435Z level=INFO msg="Upload complete. (327ms)"6714lb # [ 236.152826] niks3-server[1003]: 2026/09/21 09:15:46 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo6715worker1 # [ 236.195596] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=16716worker1 # [ 236.203464] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:45Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=16717worker1 # [ 236.208245] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626718worker1 # [ 236.212501] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:45Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=06719worker1 # [ 236.222740] nix-daemon[848]: reaped child process 1873, status = succeeded6720worker1 # [ 236.224510] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626721worker1 # [ 236.232576] nix-grpc-daemon[1541]: time=2026-09-21T09:15:45.482Z level=INFO msg="All 1 paths already cached"6722worker1 # [ 236.236898] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626723worker1 # [ 236.240755] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6724worker1 # [ 236.427380] nix-daemon[848]: accepted connection from pid 1899, user nix-grpc-daemon (trusted)6725worker1 # [ 236.439468] nix-daemon[848]: reaped child process 1902, status = succeeded6726worker1 # [ 236.612598] nix-daemon[848]: accepted connection from pid 1904, user nix-grpc-daemon (trusted)6727worker1 # [ 236.624339] nix-daemon[848]: reaped child process 1907, status = succeeded6728lb # [ 236.606102] niks3-server[1003]: 2026/09/21 09:15:46 INFO Received uploads request method=POST path=/api/pending_closures6729worker1 # [ 236.649864] nix-grpc-daemon[1541]: time=2026-09-21T09:15:45.899Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6730worker1 # [ 236.651293] nix-grpc-daemon[1541]: time=2026-09-21T09:15:45.899Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"6731lb # [ 236.632960] niks3-server[1003]: 2026/09/21 09:15:46 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst6732lb # [ 236.641514] niks3-server[1003]: 2026/09/21 09:15:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6733lb # [ 236.646668] niks3-server[1003]: 2026/09/21 09:15:46 INFO Signed narinfos id=56 count=16734worker1 # [ 236.684533] nix-grpc-daemon[1541]: time=2026-09-21T09:15:45.934Z level=INFO msg="Uploading 1 narinfos"6735lb # [ 236.653743] niks3-server[1003]: 2026/09/21 09:15:46 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls6736lb # [ 236.666496] niks3-server[1003]: 2026/09/21 09:15:46 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6737lb # [ 236.671315] niks3-server[1003]: 2026/09/21 09:15:46 INFO Completed upload id=566738worker1 # [ 236.706131] nix-grpc-daemon[1541]: time=2026-09-21T09:15:45.956Z level=INFO msg="Upload complete. (459ms)"6739worker1 # [ 236.708147] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=6246740lb # [ 236.674435] niks3-server[1003]: 2026/09/21 09:15:46 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo6741worker1 # [ 236.713352] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626742worker1 # [ 236.718369] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6743worker1 # [ 236.720549] nix-daemon[848]: reaped child process 1897, status = succeeded6744worker1 # [ 236.726084] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6745worker1 # [ 237.750750] nix-daemon[848]: reaped child process 1911, status = succeeded6746worker1 # [ 237.938081] nix-daemon[848]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)6747worker1 # [ 237.949739] nix-daemon[848]: reaped child process 1929, status = succeeded6748lb # [ 237.934355] niks3-server[1003]: 2026/09/21 09:15:47 INFO Received uploads request method=POST path=/api/pending_closures6749worker1 # [ 237.982284] nix-grpc-daemon[1541]: time=2026-09-21T09:15:47.231Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6750worker1 # [ 237.983678] nix-grpc-daemon[1541]: time=2026-09-21T09:15:47.231Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"6751lb # [ 237.973635] niks3-server[1003]: 2026/09/21 09:15:48 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv6752lb # [ 237.982954] niks3-server[1003]: 2026/09/21 09:15:48 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst6753lb # [ 237.991961] niks3-server[1003]: 2026/09/21 09:15:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6754lb # [ 237.997467] niks3-server[1003]: 2026/09/21 09:15:48 INFO Signed narinfos id=57 count=16755worker1 # [ 238.035851] nix-grpc-daemon[1541]: time=2026-09-21T09:15:47.285Z level=INFO msg="Uploading 1 narinfos"6756lb # [ 238.002452] niks3-server[1003]: 2026/09/21 09:15:48 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls6757lb # [ 238.011615] niks3-server[1003]: 2026/09/21 09:15:48 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6758lb # [ 238.016671] niks3-server[1003]: 2026/09/21 09:15:48 INFO Completed upload id=576759worker1 # [ 238.051673] nix-grpc-daemon[1541]: time=2026-09-21T09:15:47.301Z level=INFO msg="Upload complete. (318ms)"6760lb # [ 238.019992] niks3-server[1003]: 2026/09/21 09:15:48 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo6761worker1 # [ 238.062743] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=16762worker1 # [ 238.068752] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:47Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=16763worker1 # [ 238.073313] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626764worker1 # [ 238.076535] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:47Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=06765worker1 # [ 238.086365] nix-daemon[848]: reaped child process 1909, status = succeeded6766worker1 # [ 238.088271] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626767worker1 # [ 238.096377] nix-grpc-daemon[1541]: time=2026-09-21T09:15:47.346Z level=INFO msg="All 2 paths already cached"6768worker1 # [ 238.100750] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626769worker1 # [ 238.104582] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6770worker1 # [ 238.286112] nix-daemon[848]: accepted connection from pid 1935, user nix-grpc-daemon (trusted)6771worker1 # [ 238.297955] nix-daemon[848]: reaped child process 1938, status = succeeded6772worker1 # [ 238.490634] nix-daemon[848]: accepted connection from pid 1940, user nix-grpc-daemon (trusted)6773worker1 # [ 238.502160] nix-daemon[848]: reaped child process 1943, status = succeeded6774lb # [ 238.486611] niks3-server[1003]: 2026/09/21 09:15:48 INFO Received uploads request method=POST path=/api/pending_closures6775worker1 # [ 238.529660] nix-grpc-daemon[1541]: time=2026-09-21T09:15:47.779Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6776worker1 # [ 238.531096] nix-grpc-daemon[1541]: time=2026-09-21T09:15:47.779Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"6777lb # [ 238.513209] niks3-server[1003]: 2026/09/21 09:15:48 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst6778lb # [ 238.522696] niks3-server[1003]: 2026/09/21 09:15:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6779lb # [ 238.528062] niks3-server[1003]: 2026/09/21 09:15:48 INFO Signed narinfos id=58 count=16780worker1 # [ 238.565912] nix-grpc-daemon[1541]: time=2026-09-21T09:15:47.815Z level=INFO msg="Uploading 1 narinfos"6781lb # [ 238.534925] niks3-server[1003]: 2026/09/21 09:15:48 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls6782lb # [ 238.546533] niks3-server[1003]: 2026/09/21 09:15:48 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6783worker1 # [ 238.585922] nix-grpc-daemon[1541]: time=2026-09-21T09:15:47.836Z level=INFO msg="Upload complete. (475ms)"6784lb # [ 238.551369] niks3-server[1003]: 2026/09/21 09:15:48 INFO Completed upload id=586785worker1 # [ 238.588275] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=7526786lb # [ 238.554712] niks3-server[1003]: 2026/09/21 09:15:48 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo6787worker1 # [ 238.593913] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626788worker1 # [ 238.602500] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6789worker1 # [ 238.606604] nix-daemon[848]: reaped child process 1933, status = succeeded6790worker1 # [ 238.612384] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6791worker1 # [ 239.835898] nix-daemon[848]: reaped child process 1947, status = succeeded6792worker1 # [ 240.027782] nix-daemon[848]: accepted connection from pid 1962, user nix-grpc-daemon (trusted)6793worker1 # [ 240.038677] nix-daemon[848]: reaped child process 1965, status = succeeded6794lb # [ 240.023911] niks3-server[1003]: 2026/09/21 09:15:50 INFO Received uploads request method=POST path=/api/pending_closures6795worker1 # [ 240.067495] nix-grpc-daemon[1541]: time=2026-09-21T09:15:49.317Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6796worker1 # [ 240.069140] nix-grpc-daemon[1541]: time=2026-09-21T09:15:49.319Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"6797lb # [ 240.057249] niks3-server[1003]: 2026/09/21 09:15:50 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv6798lb # [ 240.069139] niks3-server[1003]: 2026/09/21 09:15:50 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst6799lb # [ 240.081133] niks3-server[1003]: 2026/09/21 09:15:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6800lb # [ 240.085984] niks3-server[1003]: 2026/09/21 09:15:50 INFO Signed narinfos id=59 count=16801worker1 # [ 240.123419] nix-grpc-daemon[1541]: time=2026-09-21T09:15:49.373Z level=INFO msg="Uploading 1 narinfos"6802lb # [ 240.091473] niks3-server[1003]: 2026/09/21 09:15:50 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls6803lb # [ 240.101101] niks3-server[1003]: 2026/09/21 09:15:50 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6804lb # [ 240.105816] niks3-server[1003]: 2026/09/21 09:15:50 INFO Completed upload id=596805lb # [ 240.108603] niks3-server[1003]: 2026/09/21 09:15:50 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo6806worker1 # [ 240.142079] nix-grpc-daemon[1541]: time=2026-09-21T09:15:49.392Z level=INFO msg="Upload complete. (323ms)"6807worker1 # [ 240.149984] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=16808worker1 # [ 240.156590] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:49Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=16809worker1 # [ 240.167384] nix-daemon[848]: reaped child process 1945, status = succeeded6810worker1 # [ 240.192782] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626811worker1 # [ 240.204875] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626812worker1 # [ 240.217974] nix-grpc-daemon[1541]: time=2026-09-21T09:15:49.468Z level=INFO msg="All 2 paths already cached"6813worker1 # [ 240.225330] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626814worker1 # [ 240.233955] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6815worker1 # [ 240.436396] nix-daemon[848]: accepted connection from pid 1969, user nix-grpc-daemon (trusted)6816worker1 # [ 240.448346] nix-daemon[848]: reaped child process 1972, status = succeeded6817worker1 # [ 240.641370] nix-daemon[848]: accepted connection from pid 1974, user nix-grpc-daemon (trusted)6818worker1 # [ 240.652903] nix-daemon[848]: reaped child process 1977, status = succeeded6819lb # [ 240.637135] niks3-server[1003]: 2026/09/21 09:15:50 INFO Received uploads request method=POST path=/api/pending_closures6820worker1 # [ 240.679917] nix-grpc-daemon[1541]: time=2026-09-21T09:15:49.930Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6821worker1 # [ 240.681711] nix-grpc-daemon[1541]: time=2026-09-21T09:15:49.931Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"6822lb # [ 240.665740] niks3-server[1003]: 2026/09/21 09:15:50 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst6823lb # [ 240.675434] niks3-server[1003]: 2026/09/21 09:15:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6824lb # [ 240.680196] niks3-server[1003]: 2026/09/21 09:15:50 INFO Signed narinfos id=60 count=16825worker1 # [ 240.718513] nix-grpc-daemon[1541]: time=2026-09-21T09:15:49.968Z level=INFO msg="Uploading 1 narinfos"6826lb # [ 240.687149] niks3-server[1003]: 2026/09/21 09:15:50 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls6827lb # [ 240.697301] niks3-server[1003]: 2026/09/21 09:15:50 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6828worker1 # [ 240.738215] nix-grpc-daemon[1541]: time=2026-09-21T09:15:49.988Z level=INFO msg="Upload complete. (492ms)"6829lb # [ 240.703340] niks3-server[1003]: 2026/09/21 09:15:50 INFO Completed upload id=606830worker1 # [ 240.740795] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=7526831lb # [ 240.707540] niks3-server[1003]: 2026/09/21 09:15:50 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo6832worker1 # [ 240.746449] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626833worker1 # [ 240.757777] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6834worker1 # [ 240.759461] nix-daemon[848]: reaped child process 1967, status = succeeded6835worker1 # [ 240.764478] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6836worker1 # [ 241.776672] nix-daemon[848]: reaped child process 1981, status = succeeded6837worker1 # [ 241.935515] nix-daemon[848]: accepted connection from pid 1996, user nix-grpc-daemon (trusted)6838worker1 # [ 241.947298] nix-daemon[848]: reaped child process 1999, status = succeeded6839lb # [ 241.929616] niks3-server[1003]: 2026/09/21 09:15:51 INFO Received uploads request method=POST path=/api/pending_closures6840worker1 # [ 241.972939] nix-grpc-daemon[1541]: time=2026-09-21T09:15:51.222Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6841worker1 # [ 241.974418] nix-grpc-daemon[1541]: time=2026-09-21T09:15:51.224Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"6842lb # [ 241.967485] niks3-server[1003]: 2026/09/21 09:15:51 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv6843lb # [ 241.975336] niks3-server[1003]: 2026/09/21 09:15:52 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst6844lb # [ 241.985785] niks3-server[1003]: 2026/09/21 09:15:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6845worker1 # [ 242.024965] nix-grpc-daemon[1541]: time=2026-09-21T09:15:51.275Z level=INFO msg="Uploading 1 narinfos"6846lb # [ 241.990566] niks3-server[1003]: 2026/09/21 09:15:52 INFO Signed narinfos id=61 count=16847lb # [ 241.996707] niks3-server[1003]: 2026/09/21 09:15:52 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls6848lb # [ 242.005548] niks3-server[1003]: 2026/09/21 09:15:52 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6849lb # [ 242.010795] niks3-server[1003]: 2026/09/21 09:15:52 INFO Completed upload id=616850worker1 # [ 242.045921] nix-grpc-daemon[1541]: time=2026-09-21T09:15:51.295Z level=INFO msg="Upload complete. (273ms)"6851lb # [ 242.012675] niks3-server[1003]: 2026/09/21 09:15:52 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo6852worker1 # [ 242.053594] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=16853worker1 # [ 242.060263] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:51Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=16854worker1 # [ 242.069507] nix-daemon[848]: reaped child process 1979, status = succeeded6855worker1 # [ 242.096315] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626856worker1 # [ 242.108840] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626857worker1 # [ 242.121871] nix-grpc-daemon[1541]: time=2026-09-21T09:15:51.371Z level=INFO msg="All 2 paths already cached"6858worker1 # [ 242.128944] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503626859worker1 # [ 242.139161] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6860worker1 # [ 242.317117] nix-daemon[848]: accepted connection from pid 2003, user nix-grpc-daemon (trusted)6861worker1 # [ 242.328827] nix-daemon[848]: reaped child process 2006, status = succeeded6862worker1 # [ 242.498783] nix-daemon[848]: accepted connection from pid 2008, user nix-grpc-daemon (trusted)6863worker1 # [ 242.510306] nix-daemon[848]: reaped child process 2011, status = succeeded6864lb # [ 242.492397] niks3-server[1003]: 2026/09/21 09:15:52 INFO Received uploads request method=POST path=/api/pending_closures6865worker1 # [ 242.534481] nix-grpc-daemon[1541]: time=2026-09-21T09:15:51.784Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6866worker1 # [ 242.536135] nix-grpc-daemon[1541]: time=2026-09-21T09:15:51.786Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"6867lb # [ 242.518695] niks3-server[1003]: 2026/09/21 09:15:52 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst6868lb # [ 242.530574] niks3-server[1003]: 2026/09/21 09:15:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6869lb # [ 242.535118] niks3-server[1003]: 2026/09/21 09:15:52 INFO Signed narinfos id=62 count=16870worker1 # [ 242.572900] nix-grpc-daemon[1541]: time=2026-09-21T09:15:51.822Z level=INFO msg="Uploading 1 narinfos"6871lb # [ 242.542456] niks3-server[1003]: 2026/09/21 09:15:52 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls6872lb # [ 242.552336] niks3-server[1003]: 2026/09/21 09:15:52 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6873lb # [ 242.557722] niks3-server[1003]: 2026/09/21 09:15:52 INFO Completed upload id=626874worker1 # [ 242.592579] nix-grpc-daemon[1541]: time=2026-09-21T09:15:51.842Z level=INFO msg="Upload complete. (442ms)"6875worker1 # [ 242.595928] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=0 paths=1 nar_bytes_in=7526876lb # [ 242.562490] niks3-server[1003]: 2026/09/21 09:15:52 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo6877worker1 # [ 242.601601] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503626878worker1 # [ 242.611436] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6879worker1 # [ 242.613716] nix-daemon[848]: reaped child process 2001, status = succeeded6880worker1 # [ 242.619315] nix-daemon[848]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6881worker1 # [ 243.441857] nix-daemon[848]: reaped child process 2015, status = succeeded6882worker1 # [ 243.611509] nix-daemon[848]: accepted connection from pid 2030, user nix-grpc-daemon (trusted)6883worker1 # [ 243.623375] nix-daemon[848]: reaped child process 2033, status = succeeded6884lb # [ 243.605764] niks3-server[1003]: 2026/09/21 09:15:53 INFO Received uploads request method=POST path=/api/pending_closures6885worker1 # [ 243.649518] nix-grpc-daemon[1541]: time=2026-09-21T09:15:52.899Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6886worker1 # [ 243.651054] nix-grpc-daemon[1541]: time=2026-09-21T09:15:52.901Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"6887lb # [ 243.642951] niks3-server[1003]: 2026/09/21 09:15:53 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst6888lb # [ 243.653976] niks3-server[1003]: 2026/09/21 09:15:53 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv6889lb # [ 243.659606] niks3-server[1003]: 2026/09/21 09:15:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6890lb # [ 243.664758] niks3-server[1003]: 2026/09/21 09:15:53 INFO Signed narinfos id=63 count=16891worker1 # [ 243.702462] nix-grpc-daemon[1541]: time=2026-09-21T09:15:52.952Z level=INFO msg="Uploading 1 narinfos"6892lb # [ 243.673166] niks3-server[1003]: 2026/09/21 09:15:53 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls6893lb # [ 243.680929] niks3-server[1003]: 2026/09/21 09:15:53 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6894lb # [ 243.685736] niks3-server[1003]: 2026/09/21 09:15:53 INFO Completed upload id=636895worker1 # [ 243.720832] nix-grpc-daemon[1541]: time=2026-09-21T09:15:52.970Z level=INFO msg="Upload complete. (293ms)"6896lb # [ 243.688745] niks3-server[1003]: 2026/09/21 09:15:53 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo6897worker1 # [ 243.726766] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50362 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=16898worker1 # [ 243.740115] nix-daemon[848]: reaped child process 2013, status = succeeded6899client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 18.39 seconds)6900worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6901worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6902client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'6903client # [ 243.887681] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6904client # Running as unit: gpu.service; invocation ID: a28d4dd3753c4b2a897fe063f79bf7e66905client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.07 seconds)6906lb: must succeed: curl -sf localhost:9901/clusters6907lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6908lb: must succeed: curl -sf localhost:9901/clusters6909lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6910worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6911worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6912client # [ 244.154465] nix[1779]: don't know how to build these paths:6913client # [ 244.155924] nix[1779]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv6914worker1 # [ 244.186120] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:503626915worker2 # [ 244.173558] nix-grpc-daemon[1745]: ts=2026-09-21T09:15:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:601086916worker2 # [ 244.184807] nix-daemon[869]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)6917worker2 # [ 244.202141] nix-grpc-daemon[1745]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 76 ms (attempt 1/5)6918worker2 # [ 244.284080] nix-grpc-daemon[1745]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 90 ms (attempt 2/5)6919worker2 # [ 244.380072] nix-grpc-daemon[1745]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 125 ms (attempt 3/5)6920worker2 # [ 244.511144] nix-grpc-daemon[1745]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 715 ms (attempt 4/5)6921lb: must succeed: curl -sf localhost:9901/clusters6922lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6923lb: must succeed: curl -sf localhost:9901/clusters6924worker2 # [ 245.233743] nix-grpc-daemon[1745]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6925client # [ 245.246236] nix[1779]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker6926worker1 # [ 245.278914] nix-grpc-daemon[1466]: ts=2026-09-21T09:15:54Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=06927lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6928worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6929worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6930client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6931lb # [ 248.141144] envoy[918]: conn peer=[2001:db8:1::1]:59986 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27141 tx=10680 ms=180786932client # [ 275.248240] nix[1779]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}6933client # [ 275.255200] nix[1779]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed6934client # [ 275.295835] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6935client # [ 275.298506] systemd[1]: gpu.service: Failed with result 'exit-code'.6936client # [ 275.302223] systemd[1]: gpu.service: Consumed 141ms CPU time over 31.413s wall clock time, 35.4M memory peak, 4K written to disk, 5.3K incoming IP traffic, 5.1K outgoing IP traffic.6937client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 30.90 seconds)6938client: must succeed: journalctl -u gpu -o cat6939client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)6940lb: must succeed: curl -sf localhost:9901/clusters6941lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6942lb: must succeed: curl -sf localhost:9901/clusters6943lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6944worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6945worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6946(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 51.10 seconds)6947subtest: low disk drains a worker and builds go to the other6948worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6949worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.98 seconds)6950worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6951worker1 # [ 277.906870] nix-grpc-daemon[1466]: ts=2026-09-21T09:16:27Z level=info event=unhealthy reason=min_free6952lb # [ 278.142355] envoy[918]: conn peer=[2001:db8:1::1]:37894 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=912 tx=539 ms=311126953worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.12 seconds)6954client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&26955client # don't know how to build these paths:6956client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv6957worker2 # [ 278.829493] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:601086958worker1 # [ 278.854418] nix-grpc-daemon[1466]: ts=2026-09-21T09:16:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503626959worker1 # [ 278.871846] nix-grpc-daemon[1466]: ts=2026-09-21T09:16:28Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=06960worker1 # [ 278.874296] nix-grpc-daemon[1466]: ts=2026-09-21T09:16:28Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=16961worker2 # [ 278.896146] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:601086962worker2 # [ 278.909161] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:601086963client # copying 1 paths to 'lb:50051'...6964client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...6965worker2 # [ 278.924468] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:601086966worker2 # [ 278.936320] nix-daemon[869]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)6967worker2 # [ 278.956663] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:28Z level=info event=niks3_push_started pid=18046968worker2 # [ 279.182243] nix-grpc-daemon[1804]: time=2026-09-21T09:16:28.392Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6969worker2 # [ 279.371854] nix-daemon[869]: accepted connection from pid 1817, user nix-grpc-daemon (trusted)6970worker2 # [ 279.382542] nix-daemon[869]: reaped child process 1820, status = succeeded6971worker2 # [ 279.572233] nix-daemon[869]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)6972worker2 # [ 279.584299] nix-daemon[869]: reaped child process 1825, status = succeeded6973lb # [ 279.590219] niks3-server[1003]: 2026/09/21 09:16:29 INFO Received uploads request method=POST path=/api/pending_closures6974worker2 # [ 279.616337] nix-grpc-daemon[1804]: time=2026-09-21T09:16:28.826Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6975worker2 # [ 279.619507] nix-grpc-daemon[1804]: time=2026-09-21T09:16:28.828Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"6976lb # [ 279.631682] niks3-server[1003]: 2026/09/21 09:16:29 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst6977lb # [ 279.643112] niks3-server[1003]: 2026/09/21 09:16:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6978lb # [ 279.648369] niks3-server[1003]: 2026/09/21 09:16:29 INFO Signed narinfos id=64 count=16979worker2 # [ 279.665492] nix-grpc-daemon[1804]: time=2026-09-21T09:16:28.875Z level=INFO msg="Uploading 1 narinfos"6980lb # [ 279.663809] niks3-server[1003]: 2026/09/21 09:16:29 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls6981lb # [ 279.666774] niks3-server[1003]: 2026/09/21 09:16:29 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6982lb # [ 279.677062] niks3-server[1003]: 2026/09/21 09:16:29 INFO Completed upload id=646983worker2 # [ 279.691231] nix-grpc-daemon[1804]: time=2026-09-21T09:16:28.901Z level=INFO msg="Upload complete. (505ms)"6984lb # [ 279.679397] niks3-server[1003]: 2026/09/21 09:16:29 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo6985worker2 # [ 279.695221] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60108 duration_s=0 paths=1 nar_bytes_in=5126986worker2 # [ 279.700641] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:601086987worker2 # [ 279.709855] nix-daemon[869]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)6988worker2 # [ 279.717125] nix-daemon[869]: reaped child process 1802, status = succeeded6989client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...6990worker2 # [ 279.720334] nix-daemon[869]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)6991client # farm-a> worker2-next: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv6992client # farm-a> LOG-farm-a-drain6993worker2 # [ 280.745688] nix-daemon[869]: reaped child process 1829, status = succeeded6994worker2 # [ 280.894871] nix-daemon[869]: accepted connection from pid 1844, user nix-grpc-daemon (trusted)6995worker2 # [ 280.905203] nix-daemon[869]: reaped child process 1847, status = succeeded6996lb # [ 280.904973] niks3-server[1003]: 2026/09/21 09:16:30 INFO Received uploads request method=POST path=/api/pending_closures6997worker2 # [ 280.929527] nix-grpc-daemon[1804]: time=2026-09-21T09:16:30.139Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6998worker2 # [ 280.931075] nix-grpc-daemon[1804]: time=2026-09-21T09:16:30.141Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"6999lb # [ 280.945097] niks3-server[1003]: 2026/09/21 09:16:30 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7000lb # [ 280.953252] niks3-server[1003]: 2026/09/21 09:16:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7001lb # [ 280.960147] niks3-server[1003]: 2026/09/21 09:16:30 INFO Signed narinfos id=65 count=17002worker2 # [ 280.976989] nix-grpc-daemon[1804]: time=2026-09-21T09:16:30.186Z level=INFO msg="Uploading 1 narinfos"7003lb # [ 280.964635] niks3-server[1003]: 2026/09/21 09:16:30 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7004lb # [ 280.970429] niks3-server[1003]: 2026/09/21 09:16:31 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7005lb # [ 280.975968] niks3-server[1003]: 2026/09/21 09:16:31 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7006lb # [ 280.979174] niks3-server[1003]: 2026/09/21 09:16:31 INFO Completed upload id=657007worker2 # [ 280.993109] nix-grpc-daemon[1804]: time=2026-09-21T09:16:30.203Z level=INFO msg="Upload complete. (264ms)"7008lb # [ 280.982597] niks3-server[1003]: 2026/09/21 09:16:31 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7009worker2 # [ 280.995925] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60108 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=17010worker1 # [ 281.023098] nix-grpc-daemon[1466]: ts=2026-09-21T09:16:30Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07011worker1 # [ 281.024773] nix-grpc-daemon[1466]: ts=2026-09-21T09:16:30Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17012worker2 # [ 281.007511] nix-daemon[869]: reaped child process 1827, status = succeeded7013worker2 # [ 281.046530] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:601087014worker2 # [ 281.061516] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:601087015worker2 # [ 281.077695] nix-grpc-daemon[1804]: time=2026-09-21T09:16:30.288Z level=INFO msg="All 1 paths already cached"7016client # copying 1 paths to 'lb:50051'...7017client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7018worker2 # [ 281.086190] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:601087019worker2 # [ 281.094337] nix-daemon[869]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)7020worker2 # [ 281.284529] nix-daemon[869]: accepted connection from pid 1851, user nix-grpc-daemon (trusted)7021worker2 # [ 281.296291] nix-daemon[869]: reaped child process 1854, status = succeeded7022worker2 # [ 281.482570] nix-daemon[869]: accepted connection from pid 1856, user nix-grpc-daemon (trusted)7023worker2 # [ 281.494149] nix-daemon[869]: reaped child process 1859, status = succeeded7024lb # [ 281.499133] niks3-server[1003]: 2026/09/21 09:16:31 INFO Received uploads request method=POST path=/api/pending_closures7025worker2 # [ 281.521788] nix-grpc-daemon[1804]: time=2026-09-21T09:16:30.731Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7026worker2 # [ 281.523189] nix-grpc-daemon[1804]: time=2026-09-21T09:16:30.731Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7027lb # [ 281.524264] niks3-server[1003]: 2026/09/21 09:16:31 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7028lb # [ 281.534160] niks3-server[1003]: 2026/09/21 09:16:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7029lb # [ 281.539726] niks3-server[1003]: 2026/09/21 09:16:31 INFO Signed narinfos id=66 count=17030worker2 # [ 281.556858] nix-grpc-daemon[1804]: time=2026-09-21T09:16:30.767Z level=INFO msg="Uploading 1 narinfos"7031lb # [ 281.546528] niks3-server[1003]: 2026/09/21 09:16:31 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7032lb # [ 281.559291] niks3-server[1003]: 2026/09/21 09:16:31 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7033lb # [ 281.564448] niks3-server[1003]: 2026/09/21 09:16:31 INFO Completed upload id=667034worker2 # [ 281.580625] nix-grpc-daemon[1804]: time=2026-09-21T09:16:30.791Z level=INFO msg="Upload complete. (472ms)"7035lb # [ 281.567487] niks3-server[1003]: 2026/09/21 09:16:31 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7036worker2 # [ 281.584071] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60108 duration_s=0 paths=1 nar_bytes_in=6247037worker2 # [ 281.588385] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:601087038worker2 # [ 281.593408] nix-daemon[869]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)7039worker2 # [ 281.595689] nix-daemon[869]: reaped child process 1849, status = succeeded7040worker2 # [ 281.601127] nix-daemon[869]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)7041client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7042client # farm-b> worker2-next: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7043client # farm-b> LOG-farm-b-drain7044worker2 # [ 283.025659] nix-daemon[869]: reaped child process 1863, status = succeeded7045worker2 # [ 283.204281] nix-daemon[869]: accepted connection from pid 1878, user nix-grpc-daemon (trusted)7046worker2 # [ 283.215878] nix-daemon[869]: reaped child process 1881, status = succeeded7047lb # [ 283.220801] niks3-server[1003]: 2026/09/21 09:16:33 INFO Received uploads request method=POST path=/api/pending_closures7048worker2 # [ 283.243258] nix-grpc-daemon[1804]: time=2026-09-21T09:16:32.453Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7049worker2 # [ 283.244847] nix-grpc-daemon[1804]: time=2026-09-21T09:16:32.455Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7050lb # [ 283.254912] niks3-server[1003]: 2026/09/21 09:16:33 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7051lb # [ 283.264068] niks3-server[1003]: 2026/09/21 09:16:33 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7052lb # [ 283.275811] niks3-server[1003]: 2026/09/21 09:16:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7053worker2 # [ 283.295475] nix-grpc-daemon[1804]: time=2026-09-21T09:16:32.505Z level=INFO msg="Uploading 1 narinfos"7054lb # [ 283.281404] niks3-server[1003]: 2026/09/21 09:16:33 INFO Signed narinfos id=67 count=17055lb # [ 283.286368] niks3-server[1003]: 2026/09/21 09:16:33 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7056lb # [ 283.295744] niks3-server[1003]: 2026/09/21 09:16:33 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7057lb # [ 283.300792] niks3-server[1003]: 2026/09/21 09:16:33 INFO Completed upload id=677058worker2 # [ 283.314756] nix-grpc-daemon[1804]: time=2026-09-21T09:16:32.525Z level=INFO msg="Upload complete. (306ms)"7059lb # [ 283.302598] niks3-server[1003]: 2026/09/21 09:16:33 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7060worker2 # [ 283.317348] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60108 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=17061worker2 # [ 283.323835] nix-daemon[869]: reaped child process 1861, status = succeeded7062worker1 # [ 283.346099] nix-grpc-daemon[1466]: ts=2026-09-21T09:16:32Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07063worker1 # [ 283.348054] nix-grpc-daemon[1466]: ts=2026-09-21T09:16:32Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17064worker2 # [ 283.369154] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:601087065worker2 # [ 283.377790] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:601087066worker2 # [ 283.386378] nix-grpc-daemon[1804]: time=2026-09-21T09:16:32.596Z level=INFO msg="All 2 paths already cached"7067client # copying 1 paths to 'lb:50051'...7068worker2 # [ 283.392082] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:601087069client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7070worker2 # [ 283.397381] nix-daemon[869]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)7071worker2 # [ 283.565668] nix-daemon[869]: accepted connection from pid 1885, user nix-grpc-daemon (trusted)7072worker2 # [ 283.577686] nix-daemon[869]: reaped child process 1888, status = succeeded7073worker2 # [ 283.746351] nix-daemon[869]: accepted connection from pid 1890, user nix-grpc-daemon (trusted)7074worker2 # [ 283.757892] nix-daemon[869]: reaped child process 1893, status = succeeded7075lb # [ 283.761885] niks3-server[1003]: 2026/09/21 09:16:33 INFO Received uploads request method=POST path=/api/pending_closures7076worker2 # [ 283.784289] nix-grpc-daemon[1804]: time=2026-09-21T09:16:32.994Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7077worker2 # [ 283.785661] nix-grpc-daemon[1804]: time=2026-09-21T09:16:32.994Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7078lb # [ 283.789088] niks3-server[1003]: 2026/09/21 09:16:33 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7079lb # [ 283.797268] niks3-server[1003]: 2026/09/21 09:16:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7080lb # [ 283.802676] niks3-server[1003]: 2026/09/21 09:16:33 INFO Signed narinfos id=68 count=17081worker2 # [ 283.819662] nix-grpc-daemon[1804]: time=2026-09-21T09:16:33.029Z level=INFO msg="Uploading 1 narinfos"7082lb # [ 283.809666] niks3-server[1003]: 2026/09/21 09:16:33 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7083lb # [ 283.820315] niks3-server[1003]: 2026/09/21 09:16:33 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7084worker2 # [ 283.839587] nix-grpc-daemon[1804]: time=2026-09-21T09:16:33.050Z level=INFO msg="Upload complete. (435ms)"7085lb # [ 283.825709] niks3-server[1003]: 2026/09/21 09:16:33 INFO Completed upload id=687086worker2 # [ 283.841484] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60108 duration_s=0 paths=1 nar_bytes_in=7527087worker2 # [ 283.845095] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:601087088lb # [ 283.831180] niks3-server[1003]: 2026/09/21 09:16:33 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7089worker2 # [ 283.849210] nix-daemon[869]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)7090worker2 # [ 283.854555] nix-daemon[869]: reaped child process 1883, status = succeeded7091client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7092worker2 # [ 283.859713] nix-daemon[869]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)7093client # farm-top> worker2-next: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7094client # farm-top> LOG-farm-top-drain7095worker2 # [ 285.473587] nix-daemon[869]: reaped child process 1897, status = succeeded7096worker2 # [ 285.629365] nix-daemon[869]: accepted connection from pid 1912, user nix-grpc-daemon (trusted)7097worker2 # [ 285.641127] nix-daemon[869]: reaped child process 1915, status = succeeded7098lb # [ 285.645583] niks3-server[1003]: 2026/09/21 09:16:35 INFO Received uploads request method=POST path=/api/pending_closures7099worker2 # [ 285.668860] nix-grpc-daemon[1804]: time=2026-09-21T09:16:34.879Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7100worker2 # [ 285.671058] nix-grpc-daemon[1804]: time=2026-09-21T09:16:34.880Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7101lb # [ 285.679389] niks3-server[1003]: 2026/09/21 09:16:35 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7102lb # [ 285.691177] niks3-server[1003]: 2026/09/21 09:16:35 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7103lb # [ 285.702867] niks3-server[1003]: 2026/09/21 09:16:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7104worker2 # [ 285.723044] nix-grpc-daemon[1804]: time=2026-09-21T09:16:34.933Z level=INFO msg="Uploading 1 narinfos"7105lb # [ 285.709159] niks3-server[1003]: 2026/09/21 09:16:35 INFO Signed narinfos id=69 count=17106lb # [ 285.714485] niks3-server[1003]: 2026/09/21 09:16:35 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7107lb # [ 285.722620] niks3-server[1003]: 2026/09/21 09:16:35 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7108lb # [ 285.727904] niks3-server[1003]: 2026/09/21 09:16:35 INFO Completed upload id=697109worker2 # [ 285.741828] nix-grpc-daemon[1804]: time=2026-09-21T09:16:34.952Z level=INFO msg="Upload complete. (275ms)"7110lb # [ 285.730670] niks3-server[1003]: 2026/09/21 09:16:35 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7111worker2 # [ 285.744795] nix-grpc-daemon[1745]: ts=2026-09-21T09:16:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60108 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=17112worker2 # [ 285.750925] nix-daemon[869]: reaped child process 1895, status = succeeded7113client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 7.21 seconds)7114client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7115client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.20 seconds)7116worker1: must succeed: rm /nix/.rw-store/fill7117worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.13 seconds)7118worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7119worker1 # [ 286.912695] nix-grpc-daemon[1466]: ts=2026-09-21T09:16:36Z level=info event=healthy reason=min_free7120worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.10 seconds)7121(finished: subtest: low disk drains a worker and builds go to the other, in 10.81 seconds)7122(finished: run the VM test script, in 288.18 seconds)7123test script finished in 288.45s7124cleanup7125kill QemuMachine (pid 49)7126client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7127kill QemuMachine (pid 50)7128lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7129kill QemuMachine (pid 52)7130worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7131kill QemuMachine (pid 51)7132worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7133(finished: cleanup, in 1.04 seconds)7134additionally exposed symbols:7135 client, lb, worker1, worker2,7136 vlan1,7137 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh