vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #464
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker2 # Disk image does not exist, creating the virtualisation disk image...13worker2 # Formatting '/build/vm-state-worker2/tmp.iJFcKmGbzO', fmt=raw size=107374182414worker2: QEMU running (pid 52)15worker2 # mke2fs 1.47.4 (6-Mar-2025)16client # Disk image does not exist, creating the virtualisation disk image...17worker2 # Discarding device blocks: 0/262144 done18client: QEMU running (pid 49)19worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes20worker1 # Disk image does not exist, creating the virtualisation disk image...21worker2 # Filesystem UUID: 5ec7b4e4-e1b8-47e4-ad1e-33b884a89fc622worker1: QEMU running (pid 51)23client # Formatting '/build/vm-state-client/tmp.kVRJmSliXO', fmt=raw size=107374182424worker1 # Formatting '/build/vm-state-worker1/tmp.WUJthaKQwl', fmt=raw size=107374182425client # mke2fs 1.47.4 (6-Mar-2025)26worker1 # mke2fs 1.47.4 (6-Mar-2025)27client # Discarding device blocks: 0/262144 done28worker1 # Discarding device blocks: 0/262144 done29client # Creating filesystem with 262144 4k blocks and 65536 inodes30worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes31client # Filesystem UUID: 9092480e-1426-4de0-a76b-7cd2ee88b89d32worker1 # Filesystem UUID: 3f95fadd-55c7-4547-8479-e7da2437de7133client # Superblock backups stored on blocks:34worker1 # Superblock backups stored on blocks:35client # 32768, 98304, 163840, 22937636worker1 # 32768, 98304, 163840, 22937637client # 38worker1 # 39client # Allocating group tables: 0/8 done40worker2 # Superblock backups stored on blocks:41worker1 # Allocating group tables: 0/8 done42client # Writing inode tables: 0/8 done43worker2 # 32768, 98304, 163840, 22937644worker1 # Writing inode tables: 0/8 done45client # Creating journal (8192 blocks): done46worker2 # 47worker1 # Creating journal (8192 blocks): done48client # Writing superblocks and filesystem accounting information: 0/8 done49worker2 # Allocating group tables: 0/8 done50worker1 # Writing superblocks and filesystem accounting information: 0/8 done51client # 52worker2 # Writing inode tables: 0/8 done53worker1 # 54client # Virtualisation disk image created.55worker2 # Creating journal (8192 blocks): done56worker1 # Virtualisation disk image created.57lb # Disk image does not exist, creating the virtualisation disk image...58lb: QEMU running (pid 50)59lb # Formatting '/build/vm-state-lb/tmp.GGfnwpQhFc', fmt=raw size=107374182460worker2 # Writing superblocks and filesystem accounting information: 0/8 done61lb # mke2fs 1.47.4 (6-Mar-2025)62worker2 # 63lb # Discarding device blocks: 0/262144 done64lb # Creating filesystem with 262144 4k blocks and 65536 inodes65(finished: start all VMs, in 0.26 seconds)66worker2 # Virtualisation disk image created.67lb # Filesystem UUID: 783b7a03-e464-4621-9c84-ef385f2510dd68lb # Superblock backups stored on blocks:69client: waiting for unit network-addresses-eth1.service70lb # 32768, 98304, 163840, 22937671client: waiting for the VM to finish booting72lb # 73lb # Allocating group tables: 0/8 done74lb # Writing inode tables: 0/8 done75lb # Creating journal (8192 blocks): done76lb # Writing superblocks and filesystem accounting information: 0/8 done77lb # 78lb # Virtualisation disk image created.79client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83worker2 # 84worker2 # 85client # 86client # 87worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0088client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0089worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...90worker2 # 91worker2 # 92worker2 # 93worker2 # 94client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...95client # 96client # 97client # 98client # 99lb # 100lb # 101worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00102client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00103lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00104worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...105worker2 # 106worker2 # 107lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...108client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...109lb # 110client # 111lb # 112client # 113lb # 114lb # 115lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00116worker1 # 117worker1 # 118worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00119lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...120lb # 121lb # 122worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...123worker1 # 124worker1 # 125worker1 # 126worker1 # 127worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00128worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129worker1 # 130worker1 # 131worker2 # Booting from ROM...132client # Booting from ROM...133lb # Booting from ROM...134worker1 # Booting from ROM...135client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test/init regInfo=/nix/store/5vv6z5wsbvfjs1wlzdz267if252sv9j6-closure-info/registration console=ttyS0,115200n8 console=tty0137client # [ 0.000000] BIOS-provided physical RAM map:138client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved141worker2 # Probing EDD (edd=off to disable)... o[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026142client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable143client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved144client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved145client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved146worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ajnaamjsnp85c2azh20np9g3bhal9f4k-nixos-system-worker2-test/init regInfo=/nix/store/lrq2vv6va7xb5ai7lrqfa5bclv9db1av-closure-info/registration console=ttyS0,115200n8 console=tty0147worker2 # [ 0.000000] BIOS-provided physical RAM map:148client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved149worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable150client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved151worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved152client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved153worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved154client # [ 0.000000] NX (Execute Disable) protection: active155worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable156client # [ 0.000000] APIC: Static calls initialized157client # [ 0.000000] SMBIOS 2.8 present.158worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved159worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved160client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014161client # [ 0.000000] DMI: Memory slots populated: 1/1162worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved163client # [ 0.000000] Hypervisor detected: KVM164worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved165client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000166client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00167worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved168lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026169client # [ 0.000000] kvm-clock: using sched offset of 472098760 cycles170worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved171worker2 # [ 0.000000] NX (Execute Disable) protection: active172client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns173worker2 # [ 0.000000] APIC: Static calls initialized174worker2 # [ 0.000000] SMBIOS 2.8 present.175client # [ 0.000005] tsc: Detected 2400.012 MHz processor176client # [ 0.000810] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000177worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014178lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/c43v4kfblicdpldaa61n4j8w84hpmhmi-nixos-system-lb-test/init regInfo=/nix/store/jynfd7hpm82yx9w7xm3qqfqqxscm4jax-closure-info/registration console=ttyS0,115200n8 console=tty0179worker2 # [ 0.000000] DMI: Memory slots populated: 1/1180client # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs181lb # [ 0.000000] BIOS-provided physical RAM map:182worker2 # [ 0.000000] Hypervisor detected: KVM183client # [ 0.000840] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT184worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000185lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable186client # [ 0.002734] found SMP MP-table at [mem 0x000f5450-0x000f545f]187worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00188client # [ 0.002746] Using GB pages for direct mapping189lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved190worker2 # [ 0.000000] kvm-clock: using sched offset of 475869601 cycles191client # [ 0.002861] RAMDISK: [mem 0x5e366000-0x5ffcffff]192lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved193client # [ 0.002868] ACPI: Early table checksum verification disabled194worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns195client # [ 0.002871] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )196lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable197worker2 # [ 0.000006] tsc: Detected 2400.012 MHz processor198lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved199client # [ 0.002876] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)200worker2 # [ 0.000806] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000201lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved202client # [ 0.002880] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)203worker2 # [ 0.000833] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs204lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved205client # [ 0.002887] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)206worker2 # [ 0.000836] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT207lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved208client # [ 0.002889] ACPI: FACS 0x000000005FFE0000 000040209worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026210worker2 # [ 0.002730] found SMP MP-table at [mem 0x000f5450-0x000f545f]211worker2 # [ 0.002742] Using GB pages for direct mapping212lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved213client # [ 0.002891] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)214worker2 # [ 0.002854] RAMDISK: [mem 0x5e367000-0x5ffcffff]215lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved216worker2 # [ 0.002862] ACPI: Early table checksum verification disabled217client # [ 0.002893] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)218lb # [ 0.000000] NX (Execute Disable) protection: active219worker2 # [ 0.002870] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )220lb # [ 0.000000] APIC: Static calls initialized221client # [ 0.002894] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)222lb # [ 0.000000] SMBIOS 2.8 present.223worker2 # [ 0.002875] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)224worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/dfinz1jb80f7kddpy2i5dm45jl5sdafn-nixos-system-worker1-test/init regInfo=/nix/store/vh6vnxsr6kpl55n796qpvfi7vmfgnn54-closure-info/registration console=ttyS0,115200n8 console=tty0225client # [ 0.002896] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)226worker1 # [ 0.000000] BIOS-provided physical RAM map:227lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014228worker2 # [ 0.002879] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)229client # [ 0.002897] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]230lb # [ 0.000000] DMI: Memory slots populated: 1/1231worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable232lb # [ 0.000000] Hypervisor detected: KVM233worker2 # [ 0.002886] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)234worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved235client # [ 0.002898] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]236lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000237worker2 # [ 0.002888] ACPI: FACS 0x000000005FFE0000 000040238client # [ 0.002899] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]239worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved240lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00241worker2 # [ 0.002890] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)242client # [ 0.002899] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]243lb # [ 0.000000] kvm-clock: using sched offset of 452048659 cycles244worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable245client # [ 0.002900] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]246worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved247worker2 # [ 0.002892] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)248lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns249client # [ 0.002900] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]250worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved251lb # [ 0.000005] tsc: Detected 2400.012 MHz processor252worker2 # [ 0.002893] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)253client # [ 0.002901] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]254lb # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000255worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved256client # [ 0.003119] No NUMA configuration found257worker2 # [ 0.002895] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)258worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved259lb # [ 0.000840] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs260client # [ 0.003119] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]261worker2 # [ 0.002896] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]262worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved263lb # [ 0.000842] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT264client # [ 0.003122] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]265client # [ 0.005483] Zone ranges:266worker2 # [ 0.002897] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]267lb # [ 0.002735] found SMP MP-table at [mem 0x000f5450-0x000f545f]268worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved269client # [ 0.005485] DMA [mem 0x0000000000001000-0x0000000000ffffff]270worker2 # [ 0.002898] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]271lb # [ 0.002748] Using GB pages for direct mapping272worker1 # [ 0.000000] NX (Execute Disable) protection: active273client # [ 0.005487] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]274lb # [ 0.002845] RAMDISK: [mem 0x5e367000-0x5ffcffff]275worker1 # [ 0.000000] APIC: Static calls initialized276worker2 # [ 0.002898] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]277client # [ 0.005488] Normal empty278worker1 # [ 0.000000] SMBIOS 2.8 present.279client # [ 0.005489] Device empty280lb # [ 0.002854] ACPI: Early table checksum verification disabled281worker2 # [ 0.002899] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]282client # [ 0.005490] Movable zone start for each node283lb # [ 0.002857] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )284client # [ 0.005491] Early memory node ranges285worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014286worker2 # [ 0.002899] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]287worker1 # [ 0.000000] DMI: Memory slots populated: 1/1288client # [ 0.005491] node 0: [mem 0x0000000000001000-0x000000000009efff]289lb # [ 0.002861] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)290worker1 # [ 0.000000] Hypervisor detected: KVM291worker2 # [ 0.002899] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]292client # [ 0.005493] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]293worker2 # [ 0.003121] No NUMA configuration found294worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000295lb # [ 0.002865] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)296client # [ 0.005494] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]297worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00298worker2 # [ 0.003122] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]299lb # [ 0.002871] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)300client # [ 0.005500] On node 0, zone DMA: 1 pages in unavailable ranges301worker1 # [ 0.000000] kvm-clock: using sched offset of 462696146 cycles302worker2 # [ 0.003125] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]303lb # [ 0.002873] ACPI: FACS 0x000000005FFE0000 000040304worker2 # [ 0.005518] Zone ranges:305client # [ 0.005516] On node 0, zone DMA: 97 pages in unavailable ranges306worker2 # [ 0.005520] DMA [mem 0x0000000000001000-0x0000000000ffffff]307worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns308client # [ 0.024035] On node 0, zone DMA32: 40 pages in unavailable ranges309lb # [ 0.002875] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)310client # [ 0.024490] ACPI: PM-Timer IO Port: 0x608311worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor312worker2 # [ 0.005522] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]313worker2 # [ 0.005523] Normal empty314client # [ 0.024504] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])315worker1 # [ 0.000813] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000316lb # [ 0.002876] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)317worker2 # [ 0.005524] Device empty318worker2 # [ 0.005525] Movable zone start for each node319client # [ 0.024530] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23320lb # [ 0.002878] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)321worker2 # [ 0.005526] Early memory node ranges322worker1 # [ 0.000841] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs323client # [ 0.024533] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)324worker1 # [ 0.000844] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT325worker2 # [ 0.005526] node 0: [mem 0x0000000000001000-0x000000000009efff]326lb # [ 0.002880] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)327client # [ 0.024534] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)328worker1 # [ 0.002735] found SMP MP-table at [mem 0x000f5450-0x000f545f]329worker2 # [ 0.005527] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]330lb # [ 0.002881] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]331client # [ 0.024536] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)332worker1 # [ 0.002748] Using GB pages for direct mapping333worker2 # [ 0.005529] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]334worker1 # [ 0.002826] RAMDISK: [mem 0x5e367000-0x5ffcffff]335lb # [ 0.002882] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]336client # [ 0.024537] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)337worker1 # [ 0.002834] ACPI: Early table checksum verification disabled338worker2 # [ 0.005550] On node 0, zone DMA: 1 pages in unavailable ranges339lb # [ 0.002883] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]340client # [ 0.024537] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)341worker2 # [ 0.005816] On node 0, zone DMA: 97 pages in unavailable ranges342worker1 # [ 0.002838] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )343client # [ 0.024540] ACPI: Using ACPI (MADT) for SMP configuration information344lb # [ 0.002883] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]345worker2 # [ 0.031387] On node 0, zone DMA32: 40 pages in unavailable ranges346client # [ 0.024541] ACPI: HPET id: 0x8086a201 base: 0xfed00000347worker1 # [ 0.002841] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)348worker2 # [ 0.031847] ACPI: PM-Timer IO Port: 0x608349lb # [ 0.002883] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]350client # [ 0.024546] TSC deadline timer available351worker2 # [ 0.031861] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])352client # [ 0.024550] CPU topo: Max. logical packages: 1353worker1 # [ 0.002846] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)354lb # [ 0.002884] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]355worker2 # [ 0.031893] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23356client # [ 0.024551] CPU topo: Max. logical dies: 1357worker1 # [ 0.002853] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)358lb # [ 0.002884] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]359client # [ 0.024551] CPU topo: Max. dies per package: 1360worker2 # [ 0.031896] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)361lb # [ 0.003118] No NUMA configuration found362worker1 # [ 0.002855] ACPI: FACS 0x000000005FFE0000 000040363client # [ 0.024555] CPU topo: Max. threads per core: 1364worker2 # [ 0.031897] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)365client # [ 0.024556] CPU topo: Num. cores per package: 1366lb # [ 0.003119] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]367worker1 # [ 0.002857] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)368client # [ 0.024556] CPU topo: Num. threads per package: 1369worker2 # [ 0.031899] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)370lb # [ 0.003121] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]371lb # [ 0.005860] Zone ranges:372client # [ 0.024556] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs373worker1 # [ 0.002858] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)374worker2 # [ 0.031900] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)375lb # [ 0.005861] DMA [mem 0x0000000000001000-0x0000000000ffffff]376client # [ 0.024578] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()377worker2 # [ 0.031900] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)378worker1 # [ 0.002860] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)379lb # [ 0.005863] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]380client # [ 0.024613] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]381lb # [ 0.005864] Normal empty382worker2 # [ 0.031903] ACPI: Using ACPI (MADT) for SMP configuration information383lb # [ 0.005865] Device empty384worker1 # [ 0.002861] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)385worker2 # [ 0.031904] ACPI: HPET id: 0x8086a201 base: 0xfed00000386lb # [ 0.005865] Movable zone start for each node387client # [ 0.024614] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]388lb # [ 0.005866] Early memory node ranges389worker1 # [ 0.002863] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]390worker2 # [ 0.031909] TSC deadline timer available391client # [ 0.024616] [mem 0x60000000-0xafffffff] available for PCI devices392worker2 # [ 0.031913] CPU topo: Max. logical packages: 1393lb # [ 0.005866] node 0: [mem 0x0000000000001000-0x000000000009efff]394client # [ 0.024618] Booting paravirtualized kernel on KVM395worker1 # [ 0.002864] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]396worker2 # [ 0.031914] CPU topo: Max. logical dies: 1397lb # [ 0.005867] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]398worker2 # [ 0.031915] CPU topo: Max. dies per package: 1399worker1 # [ 0.002864] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]400client # [ 0.024621] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns401worker2 # [ 0.031918] CPU topo: Max. threads per core: 1402lb # [ 0.005868] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]403worker1 # [ 0.002865] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]404worker2 # [ 0.031919] CPU topo: Num. cores per package: 1405client # [ 0.029076] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1406lb # [ 0.005888] On node 0, zone DMA: 1 pages in unavailable ranges407worker2 # [ 0.031919] CPU topo: Num. threads per package: 1408worker1 # [ 0.002866] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]409client # [ 0.031164] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152410lb # [ 0.006167] On node 0, zone DMA: 97 pages in unavailable ranges411worker2 # [ 0.031920] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs412client # [ 0.031208] kvm-guest: PV spinlocks disabled, single CPU413worker1 # [ 0.002866] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]414lb # [ 0.032367] On node 0, zone DMA32: 40 pages in unavailable ranges415worker2 # [ 0.031942] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()416lb # [ 0.032831] ACPI: PM-Timer IO Port: 0x608417worker1 # [ 0.002867] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]418worker1 # [ 0.003096] No NUMA configuration found419lb # [ 0.032842] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])420worker2 # [ 0.031976] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]421worker1 # [ 0.003097] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]422lb # [ 0.032872] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23423worker2 # [ 0.031978] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]424worker1 # [ 0.003099] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]425lb # [ 0.032875] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)426worker1 # [ 0.005413] Zone ranges:427worker2 # [ 0.031979] [mem 0x60000000-0xafffffff] available for PCI devices428client # [ 0.031209] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test/init regInfo=/nix/store/5vv6z5wsbvfjs1wlzdz267if252sv9j6-closure-info/registration console=ttyS0,115200n8 console=tty0429worker2 # [ 0.031981] Booting paravirtualized kernel on KVM430lb # [ 0.032876] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)431worker1 # [ 0.005414] DMA [mem 0x0000000000001000-0x0000000000ffffff]432worker1 # [ 0.005416] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]433lb # [ 0.032877] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)434worker1 # [ 0.005417] Normal empty435worker2 # [ 0.031985] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns436worker1 # [ 0.005418] Device empty437lb # [ 0.032878] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)438worker1 # [ 0.005418] Movable zone start for each node439worker2 # [ 0.036482] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1440worker1 # [ 0.005419] Early memory node ranges441lb # [ 0.032879] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)442worker2 # [ 0.038612] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152443client # [ 0.031304] Unknown kernel command line parameters "regInfo=/nix/store/5vv6z5wsbvfjs1wlzdz267if252sv9j6-closure-info/registration", will be passed to user space.444worker1 # [ 0.005419] node 0: [mem 0x0000000000001000-0x000000000009efff]445lb # [ 0.032881] ACPI: Using ACPI (MADT) for SMP configuration information446client # [ 0.031317] random: crng init done447worker2 # [ 0.038660] kvm-guest: PV spinlocks disabled, single CPU448lb # [ 0.032882] ACPI: HPET id: 0x8086a201 base: 0xfed00000449worker1 # [ 0.005420] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]450client # [ 0.031318] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes451lb # [ 0.032885] TSC deadline timer available452worker1 # [ 0.005421] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]453lb # [ 0.032889] CPU topo: Max. logical packages: 1454client # [ 0.033368] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)455worker1 # [ 0.005441] On node 0, zone DMA: 1 pages in unavailable ranges456lb # [ 0.032890] CPU topo: Max. logical dies: 1457lb # [ 0.032890] CPU topo: Max. dies per package: 1458client # [ 0.034566] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)459worker1 # [ 0.005692] On node 0, zone DMA: 97 pages in unavailable ranges460client # [ 0.034598] Fallback order for Node 0: 0461lb # [ 0.032893] CPU topo: Max. threads per core: 1462worker1 # [ 0.032979] On node 0, zone DMA32: 40 pages in unavailable ranges463lb # [ 0.032894] CPU topo: Num. cores per package: 1464worker2 # [ 0.038662] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ajnaamjsnp85c2azh20np9g3bhal9f4k-nixos-system-worker2-test/init regInfo=/nix/store/lrq2vv6va7xb5ai7lrqfa5bclv9db1av-closure-info/registration console=ttyS0,115200n8 console=tty0465worker1 # [ 0.033472] ACPI: PM-Timer IO Port: 0x608466client # [ 0.034601] Built 1 zonelists, mobility grouping on. Total pages: 393078467client # [ 0.034602] Policy zone: DMA32468lb # [ 0.032894] CPU topo: Num. threads per package: 1469worker1 # [ 0.033486] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])470lb # [ 0.032895] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs471client # [ 0.036976] mem auto-init: stack:all(zero), heap alloc:on, heap free:off472worker1 # [ 0.033518] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23473worker2 # [ 0.038760] Unknown kernel command line parameters "regInfo=/nix/store/lrq2vv6va7xb5ai7lrqfa5bclv9db1av-closure-info/registration", will be passed to user space.474lb # [ 0.032912] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()475client # [ 0.040115] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1476worker2 # [ 0.038774] random: crng init done477worker1 # [ 0.033520] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)478client # [ 0.043141] allocated 3145728 bytes of page_ext479lb # [ 0.032942] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]480worker2 # [ 0.038775] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes481worker1 # [ 0.033522] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)482client # [ 0.053122] ftrace: allocating 48731 entries in 192 pages483client # [ 0.053123] ftrace: allocated 192 pages with 2 groups484worker1 # [ 0.033523] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)485lb # [ 0.032944] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]486worker2 # [ 0.040858] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)487client # [ 0.053944] Dynamic Preempt: lazy488lb # [ 0.032945] [mem 0x60000000-0xafffffff] available for PCI devices489worker1 # [ 0.033524] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)490worker2 # [ 0.041908] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)491client # [ 0.054138] rcu: Preemptible hierarchical RCU implementation.492lb # [ 0.032947] Booting paravirtualized kernel on KVM493worker2 # [ 0.041945] Fallback order for Node 0: 0494client # [ 0.054139] rcu: RCU event tracing is enabled.495worker1 # [ 0.033525] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)496worker2 # [ 0.041948] Built 1 zonelists, mobility grouping on. Total pages: 393078497worker1 # [ 0.033527] ACPI: Using ACPI (MADT) for SMP configuration information498client # [ 0.054139] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.499lb # [ 0.032949] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns500worker2 # [ 0.041949] Policy zone: DMA32501worker1 # [ 0.033528] ACPI: HPET id: 0x8086a201 base: 0xfed00000502client # [ 0.054141] Trampoline variant of Tasks RCU enabled.503worker2 # [ 0.044541] mem auto-init: stack:all(zero), heap alloc:on, heap free:off504lb # [ 0.037405] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1505worker1 # [ 0.033534] TSC deadline timer available506client # [ 0.054141] Rude variant of Tasks RCU enabled.507worker1 # [ 0.033539] CPU topo: Max. logical packages: 1508client # [ 0.054141] Tracing variant of Tasks RCU enabled.509worker2 # [ 0.047771] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1510lb # [ 0.039706] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152511worker2 # [ 0.051016] allocated 3145728 bytes of page_ext512worker1 # [ 0.033539] CPU topo: Max. logical dies: 1513lb # [ 0.039753] kvm-guest: PV spinlocks disabled, single CPU514client # [ 0.054142] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.515worker1 # [ 0.033540] CPU topo: Max. dies per package: 1516worker2 # [ 0.060626] ftrace: allocating 48731 entries in 192 pages517worker1 # [ 0.033544] CPU topo: Max. threads per core: 1518client # [ 0.054143] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1519worker2 # [ 0.060628] ftrace: allocated 192 pages with 2 groups520worker1 # [ 0.033544] CPU topo: Num. cores per package: 1521worker2 # [ 0.061440] Dynamic Preempt: lazy522worker1 # [ 0.033544] CPU topo: Num. threads per package: 1523client # [ 0.054164] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.524worker2 # [ 0.061591] rcu: Preemptible hierarchical RCU implementation.525worker1 # [ 0.033545] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs526worker2 # [ 0.061592] rcu: RCU event tracing is enabled.527client # [ 0.054166] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.528worker1 # [ 0.033569] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()529worker2 # [ 0.061593] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.530lb # [ 0.039754] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/c43v4kfblicdpldaa61n4j8w84hpmhmi-nixos-system-lb-test/init regInfo=/nix/store/jynfd7hpm82yx9w7xm3qqfqqxscm4jax-closure-info/registration console=ttyS0,115200n8 console=tty0531worker2 # [ 0.061594] Trampoline variant of Tasks RCU enabled.532client # [ 0.054167] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.533worker1 # [ 0.033604] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]534worker2 # [ 0.061595] Rude variant of Tasks RCU enabled.535client # [ 0.058453] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16536worker2 # [ 0.061595] Tracing variant of Tasks RCU enabled.537worker1 # [ 0.033606] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]538lb # [ 0.039846] Unknown kernel command line parameters "regInfo=/nix/store/jynfd7hpm82yx9w7xm3qqfqqxscm4jax-closure-info/registration", will be passed to user space.539client # [ 0.058908] rcu: srcu_init: Setting srcu_struct sizes based on contention.540lb # [ 0.039858] random: crng init done541worker1 # [ 0.033607] [mem 0x60000000-0xafffffff] available for PCI devices542worker2 # [ 0.061596] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.543worker1 # [ 0.033609] Booting paravirtualized kernel on KVM544lb # [ 0.039858] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes545client # [ 0.058914] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns546worker2 # [ 0.061596] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1547lb # [ 0.042065] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)548worker1 # [ 0.033613] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns549worker2 # [ 0.061606] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.550client # [ 0.059019] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)551client # [ 0.062461] Console: colour VGA+ 80x25552lb # [ 0.043114] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)553worker1 # [ 0.038053] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1554lb # [ 0.043144] Fallback order for Node 0: 0555client # [ 0.062464] printk: legacy console [tty0] enabled556worker2 # [ 0.061607] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.557worker1 # [ 0.040150] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152558client # [ 0.091732] printk: legacy console [ttyS0] enabled559lb # [ 0.043147] Built 1 zonelists, mobility grouping on. Total pages: 393078560worker1 # [ 0.040194] kvm-guest: PV spinlocks disabled, single CPU561worker2 # [ 0.061608] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.562lb # [ 0.043148] Policy zone: DMA32563client # [ 0.188489] ACPI: Core revision 20250807564worker2 # [ 0.065960] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16565lb # [ 0.046133] mem auto-init: stack:all(zero), heap alloc:on, heap free:off566lb # [ 0.049253] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1567worker2 # [ 0.066239] rcu: srcu_init: Setting srcu_struct sizes based on contention.568client # [ 0.189335] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns569lb # [ 0.052785] allocated 3145728 bytes of page_ext570worker2 # [ 0.066244] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns571client # [ 0.190793] APIC: Switch to symmetric I/O mode setup572lb # [ 0.062287] ftrace: allocating 48731 entries in 192 pages573lb # [ 0.062289] ftrace: allocated 192 pages with 2 groups574client # [ 0.191698] x2apic enabled575worker1 # [ 0.040196] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/dfinz1jb80f7kddpy2i5dm45jl5sdafn-nixos-system-worker1-test/init regInfo=/nix/store/vh6vnxsr6kpl55n796qpvfi7vmfgnn54-closure-info/registration console=ttyS0,115200n8 console=tty0576lb # [ 0.063140] Dynamic Preempt: lazy577worker2 # [ 0.066349] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)578worker2 # [ 0.069848] Console: colour VGA+ 80x25579client # [ 0.192418] APIC: Switched APIC routing to: physical x2apic580lb # [ 0.063272] rcu: Preemptible hierarchical RCU implementation.581worker2 # [ 0.069850] printk: legacy console [tty0] enabled582lb # [ 0.063272] rcu: RCU event tracing is enabled.583worker1 # [ 0.040289] Unknown kernel command line parameters "regInfo=/nix/store/vh6vnxsr6kpl55n796qpvfi7vmfgnn54-closure-info/registration", will be passed to user space.584worker2 # [ 0.099272] printk: legacy console [ttyS0] enabled585worker1 # [ 0.040303] random: crng init done586lb # [ 0.063273] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.587worker2 # [ 0.194457] ACPI: Core revision 20250807588client # [ 0.194167] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1589lb # [ 0.063274] Trampoline variant of Tasks RCU enabled.590worker1 # [ 0.040304] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes591lb # [ 0.063275] Rude variant of Tasks RCU enabled.592lb # [ 0.063275] Tracing variant of Tasks RCU enabled.593worker1 # [ 0.042564] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)594worker2 # [ 0.195649] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns595client # [ 0.195106] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns596lb # [ 0.063276] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.597worker1 # [ 0.043595] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)598worker1 # [ 0.043629] Fallback order for Node 0: 0599worker2 # [ 0.197240] APIC: Switch to symmetric I/O mode setup600lb # [ 0.063277] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1601client # [ 0.197046] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)602worker1 # [ 0.043632] Built 1 zonelists, mobility grouping on. Total pages: 393078603worker2 # [ 0.198173] x2apic enabled604worker1 # [ 0.043633] Policy zone: DMA32605lb # [ 0.063329] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.606client # [ 0.198128] x86/cpu: User Mode Instruction Prevention (UMIP) activated607worker1 # [ 0.046032] mem auto-init: stack:all(zero), heap alloc:on, heap free:off608worker2 # [ 0.198932] APIC: Switched APIC routing to: physical x2apic609lb # [ 0.063331] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.610worker1 # [ 0.049103] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1611client # [ 0.199196] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127612worker1 # [ 0.052294] allocated 3145728 bytes of page_ext613lb # [ 0.063332] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.614client # [ 0.200717] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0615worker2 # [ 0.200705] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1616worker1 # [ 0.062153] ftrace: allocating 48731 entries in 192 pages617lb # [ 0.067755] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16618worker1 # [ 0.062156] ftrace: allocated 192 pages with 2 groups619worker1 # [ 0.062964] Dynamic Preempt: lazy620lb # [ 0.068018] rcu: srcu_init: Setting srcu_struct sizes based on contention.621worker2 # [ 0.201653] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns622client # [ 0.201046] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto623worker1 # [ 0.063146] rcu: Preemptible hierarchical RCU implementation.624worker1 # [ 0.063146] rcu: RCU event tracing is enabled.625lb # [ 0.068026] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns626worker2 # [ 0.203248] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)627worker1 # [ 0.063147] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.628client # [ 0.203042] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl629worker1 # [ 0.063149] Trampoline variant of Tasks RCU enabled.630lb # [ 0.068132] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)631client # [ 0.204043] Transient Scheduler Attacks: Vulnerable: No microcode632lb # [ 0.071695] Console: colour VGA+ 80x25633worker2 # [ 0.204332] x86/cpu: User Mode Instruction Prevention (UMIP) activated634worker1 # [ 0.063149] Rude variant of Tasks RCU enabled.635lb # [ 0.071697] printk: legacy console [tty0] enabled636worker1 # [ 0.063149] Tracing variant of Tasks RCU enabled.637client # [ 0.205042] Spectre V2 : Mitigation: Enhanced / Automatic IBRS638lb # [ 0.101171] printk: legacy console [ttyS0] enabled639worker2 # [ 0.205414] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127640worker1 # [ 0.063150] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.641client # [ 0.206042] Speculative Return Stack Overflow: Mitigation: Safe RET642lb # [ 0.197040] ACPI: Core revision 20250807643worker2 # [ 0.206246] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0644worker1 # [ 0.063150] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1645worker1 # [ 0.063172] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.646lb # [ 0.197909] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns647worker2 # [ 0.207250] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto648lb # [ 0.199444] APIC: Switch to symmetric I/O mode setup649worker1 # [ 0.063174] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.650lb # [ 0.200408] x2apic enabled651worker2 # [ 0.208246] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl652worker1 # [ 0.063175] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.653client # [ 0.207042] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization654worker2 # [ 0.209245] Transient Scheduler Attacks: Vulnerable: No microcode655lb # [ 0.201173] APIC: Switched APIC routing to: physical x2apic656worker1 # [ 0.067425] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16657client # [ 0.209049] Spectre V2 : Enabling IBPB for BPF658worker2 # [ 0.210245] Spectre V2 : Mitigation: Enhanced / Automatic IBRS659worker1 # [ 0.067693] rcu: srcu_init: Setting srcu_struct sizes based on contention.660client # [ 0.210043] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier661worker2 # [ 0.211245] Speculative Return Stack Overflow: Mitigation: Safe RET662worker1 # [ 0.067698] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns663lb # [ 0.203010] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1664client # [ 0.211043] active return thunk: srso_alias_return_thunk665worker1 # [ 0.067793] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)666worker1 # [ 0.071317] Console: colour VGA+ 80x25667lb # [ 0.203985] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns668client # [ 0.212063] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'669worker1 # [ 0.071320] printk: legacy console [tty0] enabled670worker1 # [ 0.100756] printk: legacy console [ttyS0] enabled671client # [ 0.213042] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'672lb # [ 0.205665] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)673worker1 # [ 0.195594] ACPI: Core revision 20250807674worker2 # [ 0.212245] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization675client # [ 0.214043] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'676lb # [ 0.206751] x86/cpu: User Mode Instruction Prevention (UMIP) activated677worker2 # [ 0.214253] Spectre V2 : Enabling IBPB for BPF678client # [ 0.215042] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'679worker1 # [ 0.196417] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns680lb # [ 0.207797] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127681worker2 # [ 0.214959] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier682client # [ 0.216042] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'683worker1 # [ 0.197900] APIC: Switch to symmetric I/O mode setup684lb # [ 0.209332] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0685worker1 # [ 0.198829] x2apic enabled686worker2 # [ 0.216246] active return thunk: srso_alias_return_thunk687client # [ 0.217042] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'688worker1 # [ 0.199565] APIC: Switched APIC routing to: physical x2apic689worker2 # [ 0.217086] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'690lb # [ 0.209665] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto691client # [ 0.218043] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'692worker2 # [ 0.218245] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'693client # [ 0.219043] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256694lb # [ 0.211662] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl695worker1 # [ 0.201366] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1696worker2 # [ 0.219246] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'697client # [ 0.220042] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64698lb # [ 0.212662] Transient Scheduler Attacks: Vulnerable: No microcode699worker2 # [ 0.220245] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'700client # [ 0.221043] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512701worker1 # [ 0.202327] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns702lb # [ 0.213661] Spectre V2 : Mitigation: Enhanced / Automatic IBRS703worker2 # [ 0.221245] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'704client # [ 0.222042] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024705lb # [ 0.214662] Speculative Return Stack Overflow: Mitigation: Safe RET706worker1 # [ 0.203916] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)707worker2 # [ 0.222176] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'708client # [ 0.223042] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8709worker1 # [ 0.204999] x86/cpu: User Mode Instruction Prevention (UMIP) activated710worker2 # [ 0.222998] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'711client # [ 0.224042] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.712worker1 # [ 0.206032] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127713worker2 # [ 0.224246] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256714worker1 # [ 0.206914] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0715worker2 # [ 0.225245] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64716lb # [ 0.215661] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization717worker2 # [ 0.226245] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512718lb # [ 0.217669] Spectre V2 : Enabling IBPB for BPF719worker1 # [ 0.207917] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto720worker2 # [ 0.227161] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024721lb # [ 0.218662] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier722worker2 # [ 0.227951] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8723worker1 # [ 0.208914] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl724lb # [ 0.219662] active return thunk: srso_alias_return_thunk725worker1 # [ 0.209914] Transient Scheduler Attacks: Vulnerable: No microcode726worker2 # [ 0.228951] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.727lb # [ 0.220683] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'728worker1 # [ 0.210913] Spectre V2 : Mitigation: Enhanced / Automatic IBRS729lb # [ 0.221661] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'730worker1 # [ 0.211913] Speculative Return Stack Overflow: Mitigation: Safe RET731lb # [ 0.223427] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'732lb # [ 0.224471] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'733lb # [ 0.225473] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'734worker1 # [ 0.212913] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization735lb # [ 0.226661] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'736worker1 # [ 0.214921] Spectre V2 : Enabling IBPB for BPF737lb # [ 0.227661] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'738worker1 # [ 0.215621] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier739lb # [ 0.228662] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256740worker1 # [ 0.216914] active return thunk: srso_alias_return_thunk741lb # [ 0.229661] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64742worker1 # [ 0.217940] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'743lb # [ 0.230661] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512744worker1 # [ 0.218913] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'745lb # [ 0.231661] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024746worker1 # [ 0.219913] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'747lb # [ 0.232661] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8748worker1 # [ 0.220913] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'749worker1 # [ 0.221913] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'750lb # [ 0.233661] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.751worker1 # [ 0.222913] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'752worker1 # [ 0.223913] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'753worker1 # [ 0.224914] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256754worker1 # [ 0.225913] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64755worker1 # [ 0.226913] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512756worker1 # [ 0.227913] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024757worker1 # [ 0.228913] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8758worker1 # [ 0.229913] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759client # [ 0.258223] Freeing SMP alternatives memory: 44K760client # [ 0.258996] pid_max: default: 32768 minimum: 301761client # [ 0.259679] LSM: initializing lsm=capability,landlock,yama,bpf,ima762client # [ 0.260862] landlock: Up and running.763client # [ 0.261480] Yama: becoming mindful.764client # [ 0.262255] LSM support for eBPF active765client # [ 0.263098] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)766worker2 # [ 0.263182] Freeing SMP alternatives memory: 44K767client # [ 0.264093] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)768worker2 # [ 0.263794] pid_max: default: 32768 minimum: 301769worker2 # [ 0.264338] LSM: initializing lsm=capability,landlock,yama,bpf,ima770worker2 # [ 0.265346] landlock: Up and running.771client # [ 0.266613] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)772worker2 # [ 0.266246] Yama: becoming mindful.773worker2 # [ 0.267355] LSM support for eBPF active774client # [ 0.267603] Performance Events: Fam17h+ core perfctr, AMD PMU driver.775worker2 # [ 0.268175] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)776client # [ 0.268047] ... version: 2777client # [ 0.268784] ... bit width: 48778worker2 # [ 0.269290] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)779client # [ 0.269045] ... generic counters: 6780client # [ 0.269729] ... generic bitmap: 000000000000003f781client # [ 0.270050] ... fixed-purpose counters: 0782worker2 # [ 0.271744] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)783client # [ 0.270769] ... fixed-purpose bitmap: 0000000000000000784client # [ 0.271046] ... value mask: 0000ffffffffffff785worker2 # [ 0.273081] Performance Events: Fam17h+ core perfctr, AMD PMU driver.786client # [ 0.271945] ... max period: 00007fffffffffff787worker2 # [ 0.274059] ... version: 2788client # [ 0.272737] ... global_ctrl mask: 000000000000003f789worker2 # [ 0.274792] ... bit width: 48790client # [ 0.273159] signal: max sigframe size: 3376791worker2 # [ 0.275262] ... generic counters: 6792lb # [ 0.268291] Freeing SMP alternatives memory: 44K793client # [ 0.273935] rcu: Hierarchical SRCU implementation.794worker2 # [ 0.275979] ... generic bitmap: 000000000000003f795lb # [ 0.268665] pid_max: default: 32768 minimum: 301796client # [ 0.274608] rcu: Max phase no-delay instances is 400.797worker2 # [ 0.276248] ... fixed-purpose counters: 0798lb # [ 0.269763] LSM: initializing lsm=capability,landlock,yama,bpf,ima799worker2 # [ 0.276954] ... fixed-purpose bitmap: 0000000000000000800lb # [ 0.270763] landlock: Up and running.801worker2 # [ 0.277253] ... value mask: 0000ffffffffffff802lb # [ 0.271662] Yama: becoming mindful.803worker2 # [ 0.278146] ... max period: 00007fffffffffff804lb # [ 0.272471] LSM support for eBPF active805worker2 # [ 0.278940] ... global_ctrl mask: 000000000000003f806lb # [ 0.273208] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)807worker2 # [ 0.279370] signal: max sigframe size: 3376808client # [ 0.279586] smp: Bringing up secondary CPUs ...809worker2 # [ 0.280196] rcu: Hierarchical SRCU implementation.810lb # [ 0.274626] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)811client # [ 0.280059] smp: Brought up 1 node, 1 CPU812worker2 # [ 0.280836] rcu: Max phase no-delay instances is 400.813worker1 # [ 0.264001] Freeing SMP alternatives memory: 44K814client # [ 0.280731] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)815worker1 # [ 0.264776] pid_max: default: 32768 minimum: 301816lb # [ 0.277115] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)817worker1 # [ 0.265537] LSM: initializing lsm=capability,landlock,yama,bpf,ima818client # [ 0.281192] Memory: 1454564K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)819worker1 # [ 0.266014] landlock: Up and running.820client # [ 0.282281] devtmpfs: initialized821worker1 # [ 0.266914] Yama: becoming mindful.822lb # [ 0.278248] Performance Events: Fam17h+ core perfctr, AMD PMU driver.823worker1 # [ 0.268113] LSM support for eBPF active824client # [ 0.283071] x86/mm: Memory block size: 128MB825lb # [ 0.278670] ... version: 2826lb # [ 0.279367] ... bit width: 48827worker2 # [ 0.285739] smp: Bringing up secondary CPUs ...828worker1 # [ 0.268867] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)829lb # [ 0.279664] ... generic counters: 6830worker2 # [ 0.286263] smp: Brought up 1 node, 1 CPU831client # [ 0.284753] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)832lb # [ 0.280350] ... generic bitmap: 000000000000003f833worker1 # [ 0.269951] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)834worker2 # [ 0.286905] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)835lb # [ 0.280664] ... fixed-purpose counters: 0836client # [ 0.285944] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).837lb # [ 0.281379] ... fixed-purpose bitmap: 0000000000000000838client # [ 0.287140] pinctrl core: initialized pinctrl subsystem839lb # [ 0.281664] ... value mask: 0000ffffffffffff840worker2 # [ 0.287456] Memory: 1454596K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)841worker1 # [ 0.272379] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)842client # [ 0.288214] PM: RTC time: 09:11:52, date: 2026-09-21843worker2 # [ 0.288474] devtmpfs: initialized844lb # [ 0.282541] ... max period: 00007fffffffffff845lb # [ 0.283359] ... global_ctrl mask: 000000000000003f846worker1 # [ 0.273470] Performance Events: Fam17h+ core perfctr, AMD PMU driver.847worker2 # [ 0.289318] x86/mm: Memory block size: 128MB848worker1 # [ 0.273922] ... version: 2849lb # [ 0.283781] signal: max sigframe size: 3376850worker1 # [ 0.274650] ... bit width: 48851lb # [ 0.284600] rcu: Hierarchical SRCU implementation.852worker1 # [ 0.274916] ... generic counters: 6853worker2 # [ 0.291086] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)854lb # [ 0.285252] rcu: Max phase no-delay instances is 400.855client # [ 0.291883] NET: Registered PF_NETLINK/PF_ROUTE protocol family856worker1 # [ 0.275596] ... generic bitmap: 000000000000003f857worker2 # [ 0.292140] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).858worker1 # [ 0.275916] ... fixed-purpose counters: 0859client # [ 0.293324] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations860worker1 # [ 0.276626] ... fixed-purpose bitmap: 0000000000000000861worker2 # [ 0.293335] pinctrl core: initialized pinctrl subsystem862worker1 # [ 0.276916] ... value mask: 0000ffffffffffff863client # [ 0.294064] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations864worker2 # [ 0.294467] PM: RTC time: 09:11:52, date: 2026-09-21865worker1 # [ 0.277798] ... max period: 00007fffffffffff866client # [ 0.295324] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations867worker1 # [ 0.278607] ... global_ctrl mask: 000000000000003f868lb # [ 0.290133] smp: Bringing up secondary CPUs ...869worker1 # [ 0.279012] signal: max sigframe size: 3376870client # [ 0.296059] audit: initializing netlink subsys (disabled)871lb # [ 0.290678] smp: Brought up 1 node, 1 CPU872worker1 # [ 0.279779] rcu: Hierarchical SRCU implementation.873client # [ 0.297165] thermal_sys: Registered thermal governor 'fair_share'874lb # [ 0.291333] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)875worker2 # [ 0.298077] NET: Registered PF_NETLINK/PF_ROUTE protocol family876worker1 # [ 0.280510] rcu: Max phase no-delay instances is 400.877client # [ 0.297167] thermal_sys: Registered thermal governor 'bang_bang'878worker2 # [ 0.299470] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations879client # [ 0.298046] thermal_sys: Registered thermal governor 'step_wise'880lb # [ 0.291886] Memory: 1454572K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)881client # [ 0.299008] thermal_sys: Registered thermal governor 'user_space'882worker2 # [ 0.300271] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations883lb # [ 0.292927] devtmpfs: initialized884lb # [ 0.293716] x86/mm: Memory block size: 128MB885client # [ 0.299784] audit: type=2000 audit(1789981913.389:1): state=initialized audit_enabled=0 res=1886worker2 # [ 0.301536] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations887client # [ 0.301048] thermal_sys: Registered thermal governor 'power_allocator'888worker2 # [ 0.302258] audit: initializing netlink subsys (disabled)889worker1 # [ 0.285363] smp: Bringing up secondary CPUs ...890client # [ 0.301061] cpuidle: using governor menu891worker1 # [ 0.285930] smp: Brought up 1 node, 1 CPU892lb # [ 0.295462] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)893worker2 # [ 0.303408] thermal_sys: Registered thermal governor 'fair_share'894worker1 # [ 0.286589] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)895client # [ 0.303238] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5896worker2 # [ 0.303410] thermal_sys: Registered thermal governor 'bang_bang'897lb # [ 0.296537] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).898worker2 # [ 0.304249] thermal_sys: Registered thermal governor 'step_wise'899lb # [ 0.297775] pinctrl core: initialized pinctrl subsystem900client # [ 0.304300] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]901worker2 # [ 0.305161] thermal_sys: Registered thermal governor 'user_space'902worker1 # [ 0.287145] Memory: 1454560K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)903lb # [ 0.298840] PM: RTC time: 09:11:52, date: 2026-09-21904client # [ 0.305056] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry905worker1 # [ 0.288167] devtmpfs: initialized906worker2 # [ 0.305962] audit: type=2000 audit(1789981913.396:1): state=initialized audit_enabled=0 res=1907client # [ 0.306056] PCI: Using configuration type 1 for base access908worker1 # [ 0.288953] x86/mm: Memory block size: 128MB909worker2 # [ 0.307251] thermal_sys: Registered thermal governor 'power_allocator'910worker2 # [ 0.307265] cpuidle: using governor menu911client # [ 0.307153] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.912lb # [ 0.302490] NET: Registered PF_NETLINK/PF_ROUTE protocol family913worker1 # [ 0.290749] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)914worker2 # [ 0.309489] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5915worker1 # [ 0.291838] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).916lb # [ 0.303883] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations917worker2 # [ 0.310499] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]918worker1 # [ 0.293007] pinctrl core: initialized pinctrl subsystem919lb # [ 0.304687] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations920worker2 # [ 0.311251] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry921worker1 # [ 0.294077] PM: RTC time: 09:11:52, date: 2026-09-21922worker2 # [ 0.312260] PCI: Using configuration type 1 for base access923client # [ 0.312284] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages924lb # [ 0.305952] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations925lb # [ 0.306675] audit: initializing netlink subsys (disabled)926client # [ 0.313047] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page927worker2 # [ 0.313348] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.928lb # [ 0.307799] thermal_sys: Registered thermal governor 'fair_share'929worker1 # [ 0.297562] NET: Registered PF_NETLINK/PF_ROUTE protocol family930lb # [ 0.307801] thermal_sys: Registered thermal governor 'bang_bang'931lb # [ 0.308665] thermal_sys: Registered thermal governor 'step_wise'932worker1 # [ 0.298405] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations933worker1 # [ 0.298935] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations934lb # [ 0.309667] audit: type=2000 audit(1789981913.410:1): state=initialized audit_enabled=0 res=1935client # [ 0.318046] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages936lb # [ 0.311667] thermal_sys: Registered thermal governor 'user_space'937worker2 # [ 0.318533] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages938client # [ 0.319044] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page939worker1 # [ 0.300186] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations940lb # [ 0.311669] thermal_sys: Registered thermal governor 'power_allocator'941worker2 # [ 0.319249] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page942worker1 # [ 0.300927] audit: initializing netlink subsys (disabled)943lb # [ 0.312665] cpuidle: using governor menu944worker1 # [ 0.302019] thermal_sys: Registered thermal governor 'fair_share'945lb # [ 0.314729] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5946worker1 # [ 0.302021] thermal_sys: Registered thermal governor 'bang_bang'947worker1 # [ 0.302917] thermal_sys: Registered thermal governor 'step_wise'948lb # [ 0.315915] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]949worker1 # [ 0.303874] thermal_sys: Registered thermal governor 'user_space'950lb # [ 0.316667] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry951worker2 # [ 0.324249] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages952worker1 # [ 0.304666] audit: type=2000 audit(1789981913.418:1): state=initialized audit_enabled=0 res=1953lb # [ 0.317675] PCI: Using configuration type 1 for base access954worker2 # [ 0.325248] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page955worker1 # [ 0.305919] thermal_sys: Registered thermal governor 'power_allocator'956lb # [ 0.318794] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.957worker1 # [ 0.305932] cpuidle: using governor menu958worker1 # [ 0.308140] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5959client # [ 0.329404] ACPI: Added _OSI(Module Device)960client # [ 0.330046] ACPI: Added _OSI(Processor Device)961worker1 # [ 0.309168] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]962client # [ 0.330769] ACPI: Added _OSI(Processor Aggregator Device)963worker1 # [ 0.309919] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry964worker1 # [ 0.310928] PCI: Using configuration type 1 for base access965lb # [ 0.323953] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages966lb # [ 0.324666] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page967worker1 # [ 0.312062] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.968worker2 # [ 0.335743] ACPI: Added _OSI(Module Device)969worker2 # [ 0.336250] ACPI: Added _OSI(Processor Device)970worker2 # [ 0.337005] ACPI: Added _OSI(Processor Aggregator Device)971lb # [ 0.329666] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages972worker1 # [ 0.317046] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages973lb # [ 0.330664] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page974client # [ 0.339267] ACPI: 1 ACPI AML tables successfully acquired and loaded975worker1 # [ 0.317918] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page976client # [ 0.343465] ACPI: Interpreter enabled977client # [ 0.344061] ACPI: PM: (supports S0 S3 S4 S5)978worker1 # [ 0.322917] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages979client # [ 0.344787] ACPI: Using IOAPIC for interrupt routing980worker1 # [ 0.323916] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page981client # [ 0.345105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug982worker2 # [ 0.345833] ACPI: 1 ACPI AML tables successfully acquired and loaded983client # [ 0.346046] PCI: Using E820 reservations for host bridge windows984lb # [ 0.341075] ACPI: Added _OSI(Module Device)985lb # [ 0.341666] ACPI: Added _OSI(Processor Device)986client # [ 0.349152] ACPI: Enabled 2 GPEs in block 00 to 3F987worker2 # [ 0.349682] ACPI: Interpreter enabled988lb # [ 0.342423] ACPI: Added _OSI(Processor Aggregator Device)989worker2 # [ 0.350265] ACPI: PM: (supports S0 S3 S4 S5)990worker2 # [ 0.350999] ACPI: Using IOAPIC for interrupt routing991worker2 # [ 0.351325] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug992worker2 # [ 0.352249] PCI: Using E820 reservations for host bridge windows993worker1 # [ 0.334299] ACPI: Added _OSI(Module Device)994worker1 # [ 0.334917] ACPI: Added _OSI(Processor Device)995worker1 # [ 0.335696] ACPI: Added _OSI(Processor Aggregator Device)996worker2 # [ 0.355377] ACPI: Enabled 2 GPEs in block 00 to 3F997client # [ 0.357725] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])998client # [ 0.358051] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]999lb # [ 0.351217] ACPI: 1 ACPI AML tables successfully acquired and loaded1000client # [ 0.359985] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1001client # [ 0.361078] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1002lb # [ 0.355067] ACPI: Interpreter enabled1003client # [ 0.362387] PCI host bridge to bus 0000:001004lb # [ 0.355680] ACPI: PM: (supports S0 S3 S4 S5)1005client # [ 0.363049] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1006lb # [ 0.356399] ACPI: Using IOAPIC for interrupt routing1007worker2 # [ 0.364037] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1008worker1 # [ 0.344091] ACPI: 1 ACPI AML tables successfully acquired and loaded1009client # [ 0.364045] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1010lb # [ 0.356698] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1011worker2 # [ 0.364959] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1012client # [ 0.365045] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1013lb # [ 0.357665] PCI: Using E820 reservations for host bridge windows1014client # [ 0.366045] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1015worker2 # [ 0.366994] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1016client # [ 0.367045] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1017worker1 # [ 0.348321] ACPI: Interpreter enabled1018worker2 # [ 0.368328] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1019worker1 # [ 0.348933] ACPI: PM: (supports S0 S3 S4 S5)1020lb # [ 0.360783] ACPI: Enabled 2 GPEs in block 00 to 3F1021client # [ 0.368045] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1022worker2 # [ 0.369613] PCI host bridge to bus 0000:001023worker1 # [ 0.349629] ACPI: Using IOAPIC for interrupt routing1024client # [ 0.369050] pci_bus 0000:00: root bus resource [bus 00-ff]1025worker2 # [ 0.370253] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1026client # [ 0.370016] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1027worker2 # [ 0.371248] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1028worker2 # [ 0.372249] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1029worker1 # [ 0.351976] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1030client # [ 0.371576] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1031worker1 # [ 0.352916] PCI: Using E820 reservations for host bridge windows1032worker2 # [ 0.373248] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1033worker2 # [ 0.374248] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1034worker2 # [ 0.375249] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1035worker1 # [ 0.356036] ACPI: Enabled 2 GPEs in block 00 to 3F1036worker2 # [ 0.376248] pci_bus 0000:00: root bus resource [bus 00-ff]1037lb # [ 0.369534] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1038worker2 # [ 0.377225] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1039lb # [ 0.370454] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1040worker2 # [ 0.378678] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1041lb # [ 0.372162] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1042client # [ 0.374138] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1043client # [ 0.375058] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1044lb # [ 0.372786] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1045client # [ 0.376026] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1046lb # [ 0.374044] PCI host bridge to bus 0000:001047lb # [ 0.374668] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1048worker1 # [ 0.363549] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1049client # [ 0.376977] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1050lb # [ 0.375664] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1051worker1 # [ 0.363923] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1052lb # [ 0.376664] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1053client # [ 0.378790] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1054worker1 # [ 0.364996] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1055lb # [ 0.377665] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1056worker1 # [ 0.366044] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1057worker2 # [ 0.381374] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1058lb # [ 0.378664] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1059worker1 # [ 0.367248] PCI host bridge to bus 0000:001060worker2 # [ 0.382261] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1061client # [ 0.382082] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1062lb # [ 0.379664] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1063worker1 # [ 0.367921] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1064worker2 # [ 0.383200] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1065client # [ 0.382920] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1066lb # [ 0.380664] pci_bus 0000:00: root bus resource [bus 00-ff]1067worker1 # [ 0.368916] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1068client # [ 0.383776] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1069worker2 # [ 0.384201] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1070lb # [ 0.381621] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1071worker1 # [ 0.369916] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1072client # [ 0.384051] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1073worker1 # [ 0.370916] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1074lb # [ 0.383052] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1075worker2 # [ 0.386150] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1076client # [ 0.385602] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1077worker1 # [ 0.371916] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1078worker1 # [ 0.372916] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1079worker1 # [ 0.373916] pci_bus 0000:00: root bus resource [bus 00-ff]1080worker2 # [ 0.389265] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1081client # [ 0.387076] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1082worker2 # [ 0.390117] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1083client # [ 0.387933] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1084worker1 # [ 0.374923] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1085worker2 # [ 0.390979] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1086client # [ 0.388788] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1087worker1 # [ 0.376385] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1088worker2 # [ 0.391254] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1089client # [ 0.389621] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1090lb # [ 0.385846] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1091worker2 # [ 0.392795] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1092lb # [ 0.386726] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1093lb # [ 0.387687] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1094worker2 # [ 0.395260] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1095client # [ 0.392080] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1096lb # [ 0.388979] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1097worker2 # [ 0.396115] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1098client # [ 0.393660] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1099worker2 # [ 0.396948] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1100client # [ 0.394066] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1101lb # [ 0.390394] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1102worker1 # [ 0.378993] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1103worker2 # [ 0.397816] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1104client # [ 0.395614] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1105worker1 # [ 0.379929] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1106worker1 # [ 0.380884] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1107lb # [ 0.392701] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1108lb # [ 0.393548] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1109client # [ 0.397076] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1110worker1 # [ 0.382333] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1111client # [ 0.397926] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1112lb # [ 0.394403] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1113worker2 # [ 0.400265] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1114worker2 # [ 0.401256] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1115lb # [ 0.395482] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1116client # [ 0.398771] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1117worker1 # [ 0.383651] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1118worker2 # [ 0.402175] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1119lb # [ 0.396930] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1120client # [ 0.399580] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1121worker2 # [ 0.403652] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1122worker1 # [ 0.385947] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1123worker1 # [ 0.386792] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1124lb # [ 0.398672] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1125client # [ 0.402080] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1126worker2 # [ 0.405266] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1127lb # [ 0.399488] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1128worker1 # [ 0.387684] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1129client # [ 0.402927] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1130worker2 # [ 0.406102] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1131worker1 # [ 0.388720] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1132lb # [ 0.400386] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1133client # [ 0.403779] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1134worker2 # [ 0.406962] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1135lb # [ 0.401260] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1136worker1 # [ 0.390189] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1137client # [ 0.404689] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1138worker2 # [ 0.407822] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1139worker1 # [ 0.392945] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1140worker2 # [ 0.409265] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1141worker1 # [ 0.393828] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1142worker2 # [ 0.410098] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1143lb # [ 0.404271] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1144client # [ 0.407083] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1145worker1 # [ 0.394677] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1146worker2 # [ 0.410971] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1147client # [ 0.408052] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1148lb # [ 0.404671] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1149client # [ 0.408992] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1150lb # [ 0.405602] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1151worker1 # [ 0.396273] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1152worker2 # [ 0.411827] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1153client # [ 0.410416] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1154lb # [ 0.407044] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1155worker2 # [ 0.413260] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1156lb # [ 0.408699] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1157client # [ 0.412052] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1158worker1 # [ 0.399137] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1159lb # [ 0.409551] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1160client # [ 0.412917] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1161worker2 # [ 0.414147] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1162worker1 # [ 0.400566] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1163lb # [ 0.410418] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1164client # [ 0.413770] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1165worker2 # [ 0.415271] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1166worker1 # [ 0.400938] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1167client # [ 0.414051] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1168worker2 # [ 0.416872] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1169lb # [ 0.411216] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1170worker1 # [ 0.402473] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1171client # [ 0.416088] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1172worker2 # [ 0.418259] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1173lb # [ 0.412671] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1174worker2 # [ 0.419148] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1175worker1 # [ 0.403923] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1176lb # [ 0.413496] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1177client # [ 0.418090] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1178worker1 # [ 0.404761] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1179worker2 # [ 0.419992] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1180lb # [ 0.414381] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1181client # [ 0.418999] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1182worker1 # [ 0.405610] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1183worker2 # [ 0.420255] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1184client # [ 0.420377] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1185lb # [ 0.415311] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1186worker2 # [ 0.421786] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1187worker1 # [ 0.406492] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1188client # [ 0.422995] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1189worker2 # [ 0.423088] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1190lb # [ 0.417672] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1191worker1 # [ 0.408924] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1192worker2 # [ 0.423974] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1193client # [ 0.423664] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1194lb # [ 0.418511] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1195worker1 # [ 0.409780] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1196client # [ 0.424067] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1197lb # [ 0.419404] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1198worker1 # [ 0.410635] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1199worker2 # [ 0.424814] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1200client # [ 0.425591] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1201lb # [ 0.420212] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1202worker1 # [ 0.411518] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1203worker2 # [ 0.426266] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1204lb # [ 0.421697] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1205worker2 # [ 0.427224] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1206client # [ 0.427052] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1207lb # [ 0.422557] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1208client # [ 0.427933] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1209worker2 # [ 0.427971] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1210worker1 # [ 0.413950] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1211lb # [ 0.423391] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1212client # [ 0.428783] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1213worker1 # [ 0.414923] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1214lb # [ 0.423670] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1215worker2 # [ 0.429479] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1216client # [ 0.429604] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1217worker1 # [ 0.415857] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1218lb # [ 0.425201] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1219worker2 # [ 0.431256] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1220worker1 # [ 0.417380] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1221client # [ 0.431092] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1222worker2 # [ 0.432080] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1223lb # [ 0.426480] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1224client # [ 0.432129] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1225worker2 # [ 0.433012] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1226lb # [ 0.427417] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1227worker1 # [ 0.418947] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1228client # [ 0.433746] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1229worker1 # [ 0.419773] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1230worker2 # [ 0.434633] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1231lb # [ 0.428213] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1232worker1 # [ 0.420624] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1233worker2 # [ 0.435854] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1234worker1 # [ 0.420922] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1235lb # [ 0.429672] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1236worker2 # [ 0.436447] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1237lb # [ 0.430560] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1238worker1 # [ 0.423012] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1239worker2 # [ 0.437812] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1240lb # [ 0.431433] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1241worker1 # [ 0.424929] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1242lb # [ 0.433583] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1243worker1 # [ 0.425938] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1244lb # [ 0.435630] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1245worker1 # [ 0.427488] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1246lb # [ 0.436356] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1247lb # [ 0.436686] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1248worker1 # [ 0.429704] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1249client # [ 0.434234] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1250lb # [ 0.438243] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1251client # [ 0.436681] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1252worker1 # [ 0.430577] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1253lb # [ 0.439243] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1254worker1 # [ 0.430939] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1255client # [ 0.437349] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1256worker1 # [ 0.432458] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1257lb # [ 0.439870] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1258client # [ 0.438638] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1259worker2 # [ 0.438461] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1260lb # [ 0.441214] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1261worker2 # [ 0.440766] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1262client # [ 0.439979] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1263worker1 # [ 0.433949] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1264worker1 # [ 0.434808] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1265worker2 # [ 0.441488] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1266client # [ 0.441364] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1267worker1 # [ 0.435639] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1268worker2 # [ 0.442766] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1269client # [ 0.442302] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1270worker1 # [ 0.436468] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1271worker2 # [ 0.443542] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1272client # [ 0.444046] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1273client # [ 0.444923] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1274worker2 # [ 0.444544] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1275worker1 # [ 0.437917] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1276worker2 # [ 0.445501] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1277client # [ 0.446127] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1278worker1 # [ 0.438969] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1279client # [ 0.447733] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1280worker1 # [ 0.440599] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1281worker2 # [ 0.447081] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1282worker2 # [ 0.447878] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1283lb # [ 0.441860] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1284worker2 # [ 0.448568] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1285lb # [ 0.444226] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1286client # [ 0.451055] ACPI: PCI: Interrupt link LNKA configured for IRQ 101287worker2 # [ 0.449806] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1288client # [ 0.452146] ACPI: PCI: Interrupt link LNKB configured for IRQ 101289lb # [ 0.444935] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1290client # [ 0.453147] ACPI: PCI: Interrupt link LNKC configured for IRQ 111291lb # [ 0.446381] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1292client # [ 0.454144] ACPI: PCI: Interrupt link LNKD configured for IRQ 111293worker2 # [ 0.453568] ACPI: PCI: Interrupt link LNKA configured for IRQ 101294client # [ 0.455142] ACPI: PCI: Interrupt link LNKE configured for IRQ 101295lb # [ 0.446900] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1296worker2 # [ 0.454365] ACPI: PCI: Interrupt link LNKB configured for IRQ 101297client # [ 0.456152] ACPI: PCI: Interrupt link LNKF configured for IRQ 101298worker2 # [ 0.455350] ACPI: PCI: Interrupt link LNKC configured for IRQ 111299client # [ 0.457175] ACPI: PCI: Interrupt link LNKG configured for IRQ 111300lb # [ 0.447954] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1301worker2 # [ 0.456348] ACPI: PCI: Interrupt link LNKD configured for IRQ 111302client # [ 0.458147] ACPI: PCI: Interrupt link LNKH configured for IRQ 111303lb # [ 0.448947] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1304client # [ 0.459082] ACPI: PCI: Interrupt link GSIA configured for IRQ 161305worker2 # [ 0.457351] ACPI: PCI: Interrupt link LNKE configured for IRQ 101306worker1 # [ 0.441112] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1307client # [ 0.460013] ACPI: PCI: Interrupt link GSIB configured for IRQ 171308worker2 # [ 0.458347] ACPI: PCI: Interrupt link LNKF configured for IRQ 101309lb # [ 0.450488] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1310worker1 # [ 0.443512] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1311client # [ 0.460811] ACPI: PCI: Interrupt link GSIC configured for IRQ 181312worker2 # [ 0.459350] ACPI: PCI: Interrupt link LNKG configured for IRQ 111313lb # [ 0.451358] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1314client # [ 0.461057] ACPI: PCI: Interrupt link GSID configured for IRQ 191315worker1 # [ 0.444166] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1316worker2 # [ 0.460352] ACPI: PCI: Interrupt link LNKH configured for IRQ 111317client # [ 0.462055] ACPI: PCI: Interrupt link GSIE configured for IRQ 201318lb # [ 0.451991] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1319worker2 # [ 0.461288] ACPI: PCI: Interrupt link GSIA configured for IRQ 161320client # [ 0.463014] ACPI: PCI: Interrupt link GSIF configured for IRQ 211321worker1 # [ 0.445555] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1322worker2 # [ 0.462267] ACPI: PCI: Interrupt link GSIB configured for IRQ 171323lb # [ 0.453246] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1324client # [ 0.463758] ACPI: PCI: Interrupt link GSIG configured for IRQ 221325worker2 # [ 0.463210] ACPI: PCI: Interrupt link GSIC configured for IRQ 181326worker1 # [ 0.447161] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1327client # [ 0.464061] ACPI: PCI: Interrupt link GSIH configured for IRQ 231328worker2 # [ 0.463971] ACPI: PCI: Interrupt link GSID configured for IRQ 191329worker2 # [ 0.464265] ACPI: PCI: Interrupt link GSIE configured for IRQ 201330worker1 # [ 0.448214] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1331client # [ 0.466195] iommu: Default domain type: Translated1332worker2 # [ 0.465208] ACPI: PCI: Interrupt link GSIF configured for IRQ 211333client # [ 0.466962] iommu: DMA domain TLB invalidation policy: lazy mode1334lb # [ 0.457044] ACPI: PCI: Interrupt link LNKA configured for IRQ 101335worker2 # [ 0.465988] ACPI: PCI: Interrupt link GSIG configured for IRQ 221336worker1 # [ 0.449170] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1337client # [ 0.468004] ACPI: bus type USB registered1338lb # [ 0.457779] ACPI: PCI: Interrupt link LNKB configured for IRQ 101339worker2 # [ 0.466266] ACPI: PCI: Interrupt link GSIH configured for IRQ 231340client # [ 0.468572] usbcore: registered new interface driver usbfs1341lb # [ 0.458773] ACPI: PCI: Interrupt link LNKC configured for IRQ 111342worker1 # [ 0.450933] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1343client # [ 0.469064] usbcore: registered new interface driver hub1344worker2 # [ 0.468228] iommu: Default domain type: Translated1345worker1 # [ 0.451809] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1346lb # [ 0.459763] ACPI: PCI: Interrupt link LNKD configured for IRQ 111347client # [ 0.469939] usbcore: registered new device driver usb1348worker2 # [ 0.468803] iommu: DMA domain TLB invalidation policy: lazy mode1349lb # [ 0.460757] ACPI: PCI: Interrupt link LNKE configured for IRQ 101350worker1 # [ 0.452923] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1351worker2 # [ 0.469511] ACPI: bus type USB registered1352client # [ 0.471575] NetLabel: Initializing1353lb # [ 0.461761] ACPI: PCI: Interrupt link LNKF configured for IRQ 101354client # [ 0.472046] NetLabel: domain hash size = 1281355worker2 # [ 0.470227] usbcore: registered new interface driver usbfs1356lb # [ 0.462762] ACPI: PCI: Interrupt link LNKG configured for IRQ 111357client # [ 0.472765] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1358worker1 # [ 0.454801] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1359worker2 # [ 0.471035] usbcore: registered new interface driver hub1360lb # [ 0.463759] ACPI: PCI: Interrupt link LNKH configured for IRQ 111361client # [ 0.473102] NetLabel: unlabeled traffic allowed by default1362worker2 # [ 0.471887] usbcore: registered new device driver usb1363client # [ 0.474006] PCI: Using ACPI for IRQ routing1364lb # [ 0.464699] ACPI: PCI: Interrupt link GSIA configured for IRQ 161365worker2 # [ 0.473210] NetLabel: Initializing1366lb # [ 0.465674] ACPI: PCI: Interrupt link GSIB configured for IRQ 171367worker2 # [ 0.473674] NetLabel: domain hash size = 1281368lb # [ 0.466680] ACPI: PCI: Interrupt link GSIC configured for IRQ 181369worker1 # [ 0.458503] ACPI: PCI: Interrupt link LNKA configured for IRQ 101370worker2 # [ 0.474248] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1371lb # [ 0.467646] ACPI: PCI: Interrupt link GSID configured for IRQ 191372worker1 # [ 0.459020] ACPI: PCI: Interrupt link LNKB configured for IRQ 101373worker2 # [ 0.475174] NetLabel: unlabeled traffic allowed by default1374lb # [ 0.468452] ACPI: PCI: Interrupt link GSIE configured for IRQ 201375worker2 # [ 0.475946] PCI: Using ACPI for IRQ routing1376worker1 # [ 0.460017] ACPI: PCI: Interrupt link LNKC configured for IRQ 111377lb # [ 0.469406] ACPI: PCI: Interrupt link GSIF configured for IRQ 211378worker1 # [ 0.461013] ACPI: PCI: Interrupt link LNKD configured for IRQ 111379lb # [ 0.469676] ACPI: PCI: Interrupt link GSIG configured for IRQ 221380worker1 # [ 0.462013] ACPI: PCI: Interrupt link LNKE configured for IRQ 101381lb # [ 0.470676] ACPI: PCI: Interrupt link GSIH configured for IRQ 231382worker1 # [ 0.463012] ACPI: PCI: Interrupt link LNKF configured for IRQ 101383lb # [ 0.472637] iommu: Default domain type: Translated1384worker1 # [ 0.464023] ACPI: PCI: Interrupt link LNKG configured for IRQ 111385lb # [ 0.473226] iommu: DMA domain TLB invalidation policy: lazy mode1386worker1 # [ 0.465014] ACPI: PCI: Interrupt link LNKH configured for IRQ 111387lb # [ 0.473927] ACPI: bus type USB registered1388worker1 # [ 0.465955] ACPI: PCI: Interrupt link GSIA configured for IRQ 161389lb # [ 0.474674] usbcore: registered new interface driver usbfs1390worker1 # [ 0.466929] ACPI: PCI: Interrupt link GSIB configured for IRQ 171391lb # [ 0.475598] usbcore: registered new interface driver hub1392worker1 # [ 0.467922] ACPI: PCI: Interrupt link GSIC configured for IRQ 181393lb # [ 0.476353] usbcore: registered new device driver usb1394worker1 # [ 0.468900] ACPI: PCI: Interrupt link GSID configured for IRQ 191395worker1 # [ 0.469697] ACPI: PCI: Interrupt link GSIE configured for IRQ 201396lb # [ 0.477606] NetLabel: Initializing1397lb # [ 0.478105] NetLabel: domain hash size = 1281398worker1 # [ 0.469931] ACPI: PCI: Interrupt link GSIF configured for IRQ 211399lb # [ 0.478664] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1400worker1 # [ 0.470900] ACPI: PCI: Interrupt link GSIG configured for IRQ 221401lb # [ 0.479624] NetLabel: unlabeled traffic allowed by default1402worker1 # [ 0.471681] ACPI: PCI: Interrupt link GSIH configured for IRQ 231403lb # [ 0.480394] PCI: Using ACPI for IRQ routing1404worker1 # [ 0.473822] iommu: Default domain type: Translated1405worker1 # [ 0.474589] iommu: DMA domain TLB invalidation policy: lazy mode1406worker1 # [ 0.475166] ACPI: bus type USB registered1407worker1 # [ 0.475962] usbcore: registered new interface driver usbfs1408worker1 # [ 0.476900] usbcore: registered new interface driver hub1409worker1 # [ 0.477619] usbcore: registered new device driver usb1410worker1 # [ 0.478832] NetLabel: Initializing1411worker1 # [ 0.479374] NetLabel: domain hash size = 1281412worker1 # [ 0.479916] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413worker1 # [ 0.481001] NetLabel: unlabeled traffic allowed by default1414worker1 # [ 0.481930] PCI: Using ACPI for IRQ routing1415client # [ 0.517605] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416client # [ 0.518041] pci 0000:00:01.0: vgaarb: bridge control possible1417client # [ 0.518041] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418client # [ 0.518052] vgaarb: loaded1419worker2 # [ 0.519230] pci 0000:00:01.0: vgaarb: setting as boot VGA device1420client # [ 0.518781] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01421worker2 # [ 0.519244] pci 0000:00:01.0: vgaarb: bridge control possible1422client # [ 0.519047] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1423worker2 # [ 0.519244] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1424worker2 # [ 0.519252] vgaarb: loaded1425worker2 # [ 0.519996] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01426worker2 # [ 0.520248] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1427client # [ 0.523128] clocksource: Switched to clocksource kvm-clock1428worker2 # [ 0.523330] clocksource: Switched to clocksource kvm-clock1429client # [ 0.526634] VFS: Disk quotas dquot_6.6.01430client # [ 0.527370] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1431client # [ 0.528763] pnp: PnP ACPI init1432client # [ 0.529598] ACPI: IRQ 4 override to edge(!), high(!)1433worker2 # [ 0.526779] VFS: Disk quotas dquot_6.6.01434worker2 # [ 0.527512] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1435client # [ 0.530619] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1436worker2 # [ 0.528824] pnp: PnP ACPI init1437client # [ 0.532207] pnp: PnP ACPI: found 6 devices1438worker2 # [ 0.529620] ACPI: IRQ 4 override to edge(!), high(!)1439lb # [ 0.525396] pci 0000:00:01.0: vgaarb: setting as boot VGA device1440worker2 # [ 0.530593] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1441lb # [ 0.525660] pci 0000:00:01.0: vgaarb: bridge control possible1442worker2 # [ 0.532126] pnp: PnP ACPI: found 6 devices1443lb # [ 0.525660] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1444lb # [ 0.525669] vgaarb: loaded1445lb # [ 0.526439] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01446lb # [ 0.526666] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1447client # [ 0.539774] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1448client # [ 0.541285] clocksource: Switched to clocksource acpi_pm1449client # [ 0.542280] NET: Registered PF_INET protocol family1450lb # [ 0.530754] clocksource: Switched to clocksource kvm-clock1451worker2 # [ 0.539776] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1452worker1 # [ 0.525522] pci 0000:00:01.0: vgaarb: setting as boot VGA device1453client # [ 0.543480] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1454worker2 # [ 0.541164] clocksource: Switched to clocksource acpi_pm1455worker1 # [ 0.525912] pci 0000:00:01.0: vgaarb: bridge control possible1456lb # [ 0.532428] VFS: Disk quotas dquot_6.6.01457worker2 # [ 0.542126] NET: Registered PF_INET protocol family1458worker1 # [ 0.525912] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1459worker1 # [ 0.525920] vgaarb: loaded1460worker2 # [ 0.543288] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1461worker1 # [ 0.526678] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01462lb # [ 0.534926] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1463lb # [ 0.536234] pnp: PnP ACPI init1464worker1 # [ 0.526918] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1465lb # [ 0.537051] ACPI: IRQ 4 override to edge(!), high(!)1466lb # [ 0.538049] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1467lb # [ 0.539650] pnp: PnP ACPI: found 6 devices1468worker1 # [ 0.530007] clocksource: Switched to clocksource kvm-clock1469worker1 # [ 0.533449] VFS: Disk quotas dquot_6.6.01470worker1 # [ 0.534167] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1471worker1 # [ 0.535552] pnp: PnP ACPI init1472worker1 # [ 0.536405] ACPI: IRQ 4 override to edge(!), high(!)1473lb # [ 0.547311] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1474worker1 # [ 0.537440] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1475client # [ 0.560261] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1476lb # [ 0.548726] clocksource: Switched to clocksource acpi_pm1477worker1 # [ 0.538942] pnp: PnP ACPI: found 6 devices1478client # [ 0.561700] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1479lb # [ 0.549755] NET: Registered PF_INET protocol family1480worker2 # [ 0.559549] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1481client # [ 0.563030] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1482lb # [ 0.550969] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1483worker2 # [ 0.561065] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1484client # [ 0.564326] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1485client # [ 0.565566] TCP: Hash tables configured (established 16384 bind 16384)1486worker2 # [ 0.562346] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1487client # [ 0.566721] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1488worker2 # [ 0.563717] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1489client # [ 0.568000] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1490worker2 # [ 0.564961] TCP: Hash tables configured (established 16384 bind 16384)1491client # [ 0.569103] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1492worker1 # [ 0.546495] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1493worker2 # [ 0.566123] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1494worker1 # [ 0.547889] clocksource: Switched to clocksource acpi_pm1495client # [ 0.570295] NET: Registered PF_UNIX/PF_LOCAL protocol family1496worker2 # [ 0.567381] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1497client # [ 0.571273] NET: Registered PF_XDP protocol family1498worker1 # [ 0.548850] NET: Registered PF_INET protocol family1499worker2 # [ 0.568503] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1500client # [ 0.572083] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1501worker1 # [ 0.550029] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1502worker2 # [ 0.569715] NET: Registered PF_UNIX/PF_LOCAL protocol family1503client # [ 0.573166] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1504worker2 # [ 0.570662] NET: Registered PF_XDP protocol family1505client # [ 0.574156] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1506worker2 # [ 0.571505] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1507client # [ 0.575268] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1508worker2 # [ 0.572526] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1509client # [ 0.576431] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1510worker2 # [ 0.573549] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1511client # [ 0.577569] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1512worker2 # [ 0.574679] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1513worker2 # [ 0.575779] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1514client # [ 0.579400] ACPI: \_SB_.GSIA: Enabled at IRQ 161515worker2 # [ 0.576896] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1516lb # [ 0.567396] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1517client # [ 0.581607] ACPI: \_SB_.GSIB: Enabled at IRQ 171518worker2 # [ 0.578583] ACPI: \_SB_.GSIA: Enabled at IRQ 161519lb # [ 0.569077] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1520lb # [ 0.570362] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1521client # [ 0.583660] ACPI: \_SB_.GSIC: Enabled at IRQ 181522worker2 # [ 0.580747] ACPI: \_SB_.GSIB: Enabled at IRQ 171523lb # [ 0.571794] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1524client # [ 0.585628] ACPI: \_SB_.GSID: Enabled at IRQ 191525lb # [ 0.573044] TCP: Hash tables configured (established 16384 bind 16384)1526worker2 # [ 0.582887] ACPI: \_SB_.GSIC: Enabled at IRQ 181527lb # [ 0.574189] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1528client # [ 0.587340] PCI: CLS 0 bytes, default 641529worker2 # [ 0.584878] ACPI: \_SB_.GSID: Enabled at IRQ 191530lb # [ 0.575482] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1531client # [ 0.588266] Trying to unpack rootfs image as initramfs...1532lb # [ 0.576568] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1533worker2 # [ 0.586677] PCI: CLS 0 bytes, default 641534worker1 # [ 0.566705] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1535lb # [ 0.577832] NET: Registered PF_UNIX/PF_LOCAL protocol family1536worker1 # [ 0.568200] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1537lb # [ 0.578802] NET: Registered PF_XDP protocol family1538worker2 # [ 0.587523] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1539lb # [ 0.579606] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1540worker1 # [ 0.569485] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1541worker2 # [ 0.589184] Trying to unpack rootfs image as initramfs...1542lb # [ 0.580602] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1543client # [ 0.592523] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1544worker1 # [ 0.570781] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1545lb # [ 0.581592] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1546worker1 # [ 0.571976] TCP: Hash tables configured (established 16384 bind 16384)1547lb # [ 0.582740] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1548worker1 # [ 0.573078] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1549lb # [ 0.583877] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1550worker1 # [ 0.574361] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1551lb # [ 0.584979] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1552worker1 # [ 0.575430] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1553lb # [ 0.586680] ACPI: \_SB_.GSIA: Enabled at IRQ 161554worker1 # [ 0.576624] NET: Registered PF_UNIX/PF_LOCAL protocol family1555worker1 # [ 0.577574] NET: Registered PF_XDP protocol family1556worker1 # [ 0.578424] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1557lb # [ 0.589288] ACPI: \_SB_.GSIB: Enabled at IRQ 171558worker1 # [ 0.579451] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1559worker1 # [ 0.580471] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1560lb # [ 0.591276] ACPI: \_SB_.GSIC: Enabled at IRQ 181561worker1 # [ 0.581603] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1562worker1 # [ 0.582696] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1563lb # [ 0.593250] ACPI: \_SB_.GSID: Enabled at IRQ 191564worker1 # [ 0.583800] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1565lb # [ 0.594982] PCI: CLS 0 bytes, default 641566worker1 # [ 0.585526] ACPI: \_SB_.GSIA: Enabled at IRQ 161567lb # [ 0.595881] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1568lb # [ 0.597690] Trying to unpack rootfs image as initramfs...1569worker1 # [ 0.587845] ACPI: \_SB_.GSIB: Enabled at IRQ 171570worker1 # [ 0.590080] ACPI: \_SB_.GSIC: Enabled at IRQ 181571worker1 # [ 0.592101] ACPI: \_SB_.GSID: Enabled at IRQ 191572worker1 # [ 0.593854] PCI: CLS 0 bytes, default 641573worker1 # [ 0.594833] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1574worker1 # [ 0.596577] Trying to unpack rootfs image as initramfs...1575client # [ 0.634266] Initialise system trusted keyrings1576worker2 # [ 0.633037] Initialise system trusted keyrings1577client # [ 0.637199] workingset: timestamp_bits=40 max_order=19 bucket_order=01578worker2 # [ 0.636074] workingset: timestamp_bits=40 max_order=19 bucket_order=01579lb # [ 0.644108] Initialise system trusted keyrings1580client # [ 0.658640] Key type asymmetric registered1581client # [ 0.659421] Asymmetric key parser 'x509' registered1582lb # [ 0.647709] workingset: timestamp_bits=40 max_order=19 bucket_order=01583worker2 # [ 0.657414] Key type asymmetric registered1584worker2 # [ 0.658132] Asymmetric key parser 'x509' registered1585client # [ 0.662222] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1586worker2 # [ 0.661061] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1587worker1 # [ 0.642647] Initialise system trusted keyrings1588client # [ 0.665231] io scheduler mq-deadline registered1589client # [ 0.665942] io scheduler kyber registered1590worker1 # [ 0.643573] workingset: timestamp_bits=40 max_order=19 bucket_order=01591worker2 # [ 0.664090] io scheduler mq-deadline registered1592worker2 # [ 0.664847] io scheduler kyber registered1593client # [ 0.668681] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1594client # [ 0.669938] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1595worker2 # [ 0.670820] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1596client # [ 0.675940] Linux agpgart interface v0.1031597client # [ 0.676777] ACPI: bus type drm_connector registered1598worker2 # [ 0.674218] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1599lb # [ 0.667303] Key type asymmetric registered1600worker2 # [ 0.678092] Linux agpgart interface v0.1031601client # [ 0.681390] usbcore: registered new interface driver usbserial_generic1602worker2 # [ 0.678865] ACPI: bus type drm_connector registered1603client # [ 0.682460] usbserial: USB Serial support registered for generic1604lb # [ 0.670695] Asymmetric key parser 'x509' registered1605lb # [ 0.671581] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1606worker2 # [ 0.682173] usbcore: registered new interface driver usbserial_generic1607worker2 # [ 0.683292] usbserial: USB Serial support registered for generic1608client # [ 0.686158] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1609lb # [ 0.674895] io scheduler mq-deadline registered1610client # [ 0.686158] Please enable it if your BIOS has the CPPC option.1611client # [ 0.688406] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1612worker1 # [ 0.667296] Key type asymmetric registered1613worker2 # [ 0.686045] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1614lb # [ 0.677659] io scheduler kyber registered1615worker1 # [ 0.667998] Asymmetric key parser 'x509' registered1616worker2 # [ 0.686045] Please enable it if your BIOS has the CPPC option.1617client # [ 0.691305] drop_monitor: Initializing network drop monitor service1618worker1 # [ 0.668845] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1619worker2 # [ 0.688215] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1620client # [ 0.692415] NET: Registered PF_INET6 protocol family1621lb # [ 0.680680] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1622worker1 # [ 0.671364] io scheduler mq-deadline registered1623worker1 # [ 0.672084] io scheduler kyber registered1624lb # [ 0.682039] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1625client # [ 0.695616] Segment Routing with IPv61626worker2 # [ 0.692174] drop_monitor: Initializing network drop monitor service1627worker2 # [ 0.693363] NET: Registered PF_INET6 protocol family1628worker1 # [ 0.674874] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1629client # [ 0.698182] In-situ OAM (IOAM) with IPv61630worker1 # [ 0.676183] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1631client # [ 0.699256] IPI shorthand broadcast: enabled1632worker2 # [ 0.696586] Segment Routing with IPv61633lb # [ 0.687425] Linux agpgart interface v0.1031634lb # [ 0.688204] ACPI: bus type drm_connector registered1635worker2 # [ 0.699052] In-situ OAM (IOAM) with IPv61636worker2 # [ 0.700160] IPI shorthand broadcast: enabled1637worker1 # [ 0.681031] Linux agpgart interface v0.1031638lb # [ 0.691102] usbcore: registered new interface driver usbserial_generic1639worker1 # [ 0.681794] ACPI: bus type drm_connector registered1640lb # [ 0.692148] usbserial: USB Serial support registered for generic1641worker1 # [ 0.684633] usbcore: registered new interface driver usbserial_generic1642client # [ 0.707377] sched_clock: Marking stable (573013976, 134127449)->(778639487, -71498062)1643worker1 # [ 0.685699] usbserial: USB Serial support registered for generic1644lb # [ 0.695668] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1645lb # [ 0.695668] Please enable it if your BIOS has the CPPC option.1646lb # [ 0.697949] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1647worker2 # [ 0.708427] sched_clock: Marking stable (575015182, 132999924)->(782852985, -74837879)1648client # [ 0.712399] registered taskstats version 11649worker1 # [ 0.689258] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1650worker1 # [ 0.689258] Please enable it if your BIOS has the CPPC option.1651client # [ 0.713341] Loading compiled-in X.509 certificates1652worker1 # [ 0.691441] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1653lb # [ 0.701795] drop_monitor: Initializing network drop monitor service1654lb # [ 0.702949] NET: Registered PF_INET6 protocol family1655worker2 # [ 0.713202] registered taskstats version 11656worker2 # [ 0.714249] Loading compiled-in X.509 certificates1657lb # [ 0.706124] Segment Routing with IPv61658worker1 # [ 0.696354] drop_monitor: Initializing network drop monitor service1659worker1 # [ 0.697550] NET: Registered PF_INET6 protocol family1660lb # [ 0.709669] In-situ OAM (IOAM) with IPv61661worker1 # [ 0.700687] Segment Routing with IPv61662lb # [ 0.710744] IPI shorthand broadcast: enabled1663worker1 # [ 0.701350] In-situ OAM (IOAM) with IPv61664worker1 # [ 0.704518] IPI shorthand broadcast: enabled1665client # [ 0.729693] Demotion targets for Node 0: null1666lb # [ 0.718885] sched_clock: Marking stable (585014727, 133635272)->(793047595, -74397596)1667client # [ 0.733202] Key type .fscrypt registered1668client # [ 0.733886] Key type fscrypt-provisioning registered1669client # [ 0.734867] ima: No TPM chip found, activating TPM-bypass!1670worker1 # [ 0.712791] sched_clock: Marking stable (581014609, 131214578)->(785541372, -73312185)1671worker2 # [ 0.733032] Demotion targets for Node 0: null1672lb # [ 0.723896] registered taskstats version 11673worker2 # [ 0.733896] Key type .fscrypt registered1674client # [ 0.737154] ima: Allocated hash algorithm: sha11675lb # [ 0.724833] Loading compiled-in X.509 certificates1676worker2 # [ 0.734591] Key type fscrypt-provisioning registered1677client # [ 0.737903] ima: No architecture policies found1678worker1 # [ 0.716362] registered taskstats version 11679worker2 # [ 0.735517] ima: No TPM chip found, activating TPM-bypass!1680worker1 # [ 0.717320] Loading compiled-in X.509 certificates1681client # [ 0.740163] PM: Magic number: 10:167:1721682worker2 # [ 0.738035] ima: Allocated hash algorithm: sha11683client # [ 0.741755] RAS: Correctable Errors collector initialized.1684worker2 # [ 0.738847] ima: No architecture policies found1685worker2 # [ 0.743045] PM: Magic number: 10:167:1721686worker2 # [ 0.744683] RAS: Correctable Errors collector initialized.1687client # [ 0.750817] clk: Disabling unused clocks1688client # [ 0.754169] PM: genpd: Disabling unused power domains1689lb # [ 0.742665] Demotion targets for Node 0: null1690lb # [ 0.743567] Key type .fscrypt registered1691lb # [ 0.744311] Key type fscrypt-provisioning registered1692worker2 # [ 0.753767] clk: Disabling unused clocks1693lb # [ 0.745263] ima: No TPM chip found, activating TPM-bypass!1694worker1 # [ 0.735242] Demotion targets for Node 0: null1695worker2 # [ 0.756043] PM: genpd: Disabling unused power domains1696worker1 # [ 0.739320] Key type .fscrypt registered1697lb # [ 0.749671] ima: Allocated hash algorithm: sha11698worker1 # [ 0.740007] Key type fscrypt-provisioning registered1699lb # [ 0.750473] ima: No architecture policies found1700worker1 # [ 0.740958] ima: No TPM chip found, activating TPM-bypass!1701lb # [ 0.751449] PM: Magic number: 10:167:1721702worker1 # [ 0.743242] ima: Allocated hash algorithm: sha11703worker1 # [ 0.744074] ima: No architecture policies found1704lb # [ 0.755163] RAS: Correctable Errors collector initialized.1705worker1 # [ 0.748253] PM: Magic number: 10:167:1721706worker1 # [ 0.749820] RAS: Correctable Errors collector initialized.1707lb # [ 0.763842] clk: Disabling unused clocks1708lb # [ 0.764562] PM: genpd: Disabling unused power domains1709worker1 # [ 0.757880] clk: Disabling unused clocks1710worker1 # [ 0.762248] PM: genpd: Disabling unused power domains1711client # [ 0.915021] Freeing initrd memory: 29096K1712client # [ 0.918178] Freeing unused decrypted memory: 2028K1713client # [ 0.920777] Freeing unused kernel image (initmem) memory: 3644K1714client # [ 0.921912] Write protecting the kernel read-only data: 32768k1715client # [ 0.923836] Freeing unused kernel image (text/rodata gap) memory: 1208K1716client # [ 0.925387] Freeing unused kernel image (rodata/data gap) memory: 752K1717worker2 # [ 0.923682] Freeing initrd memory: 29092K1718worker2 # [ 0.926782] Freeing unused decrypted memory: 2028K1719worker2 # [ 0.929382] Freeing unused kernel image (initmem) memory: 3644K1720worker2 # [ 0.930476] Write protecting the kernel read-only data: 32768k1721worker2 # [ 0.932382] Freeing unused kernel image (text/rodata gap) memory: 1208K1722worker2 # [ 0.933919] Freeing unused kernel image (rodata/data gap) memory: 752K1723lb # [ 0.929464] Freeing initrd memory: 29092K1724lb # [ 0.932758] Freeing unused decrypted memory: 2028K1725lb # [ 0.935536] Freeing unused kernel image (initmem) memory: 3644K1726lb # [ 0.936685] Write protecting the kernel read-only data: 32768k1727lb # [ 0.938563] Freeing unused kernel image (text/rodata gap) memory: 1208K1728lb # [ 0.940112] Freeing unused kernel image (rodata/data gap) memory: 752K1729worker1 # [ 0.931801] Freeing initrd memory: 29092K1730worker1 # [ 0.934931] Freeing unused decrypted memory: 2028K1731worker1 # [ 0.937562] Freeing unused kernel image (initmem) memory: 3644K1732worker1 # [ 0.938656] Write protecting the kernel read-only data: 32768k1733worker1 # [ 0.940544] Freeing unused kernel image (text/rodata gap) memory: 1208K1734worker1 # [ 0.942097] Freeing unused kernel image (rodata/data gap) memory: 752K1735client # [ 0.976760] x86/mm: Checked W+X mappings: passed, no W+X pages found.1736client # [ 0.977841] Run /init as init process1737client # [ 0.987908] systemd[1]: Inserted module 'autofs4'1738worker2 # [ 0.985018] x86/mm: Checked W+X mappings: passed, no W+X pages found.1739worker2 # [ 0.986112] Run /init as init process1740worker2 # [ 0.996358] systemd[1]: Inserted module 'autofs4'1741lb # [ 0.991182] x86/mm: Checked W+X mappings: passed, no W+X pages found.1742client # [ 1.004081] fuse: init (API version 7.45)1743lb # [ 0.992281] Run /init as init process1744client # [ 1.009908] ACPI: \_SB_.GSIG: Enabled at IRQ 221745client # [ 1.012313] ACPI: \_SB_.GSIH: Enabled at IRQ 231746lb # [ 1.002512] systemd[1]: Inserted module 'autofs4'1747client # [ 1.015110] ACPI: \_SB_.GSIE: Enabled at IRQ 201748worker2 # [ 1.012623] fuse: init (API version 7.45)1749worker1 # [ 0.993197] x86/mm: Checked W+X mappings: passed, no W+X pages found.1750worker1 # [ 0.994304] Run /init as init process1751client # [ 1.017314] ACPI: \_SB_.GSIF: Enabled at IRQ 211752worker2 # [ 1.018350] ACPI: \_SB_.GSIG: Enabled at IRQ 221753worker2 # [ 1.020734] ACPI: \_SB_.GSIH: Enabled at IRQ 231754worker2 # [ 1.023653] ACPI: \_SB_.GSIE: Enabled at IRQ 201755worker1 # [ 1.004519] systemd[1]: Inserted module 'autofs4'1756worker2 # [ 1.025816] ACPI: \_SB_.GSIF: Enabled at IRQ 211757lb # [ 1.018887] fuse: init (API version 7.45)1758lb # [ 1.025029] ACPI: \_SB_.GSIG: Enabled at IRQ 221759lb # [ 1.027592] ACPI: \_SB_.GSIH: Enabled at IRQ 231760worker1 # [ 1.020678] fuse: init (API version 7.45)1761client # [ 1.042929] systemd[1]: Successfully made /usr/ read-only.1762lb # [ 1.030968] ACPI: \_SB_.GSIE: Enabled at IRQ 201763lb # [ 1.033423] ACPI: \_SB_.GSIF: Enabled at IRQ 211764worker1 # [ 1.026574] ACPI: \_SB_.GSIG: Enabled at IRQ 221765worker1 # [ 1.029019] ACPI: \_SB_.GSIH: Enabled at IRQ 231766worker1 # [ 1.032120] ACPI: \_SB_.GSIE: Enabled at IRQ 201767worker2 # [ 1.052194] systemd[1]: Successfully made /usr/ read-only.1768worker1 # [ 1.034425] ACPI: \_SB_.GSIF: Enabled at IRQ 211769lb # [ 1.061806] systemd[1]: Successfully made /usr/ read-only.1770worker1 # [ 1.059895] systemd[1]: Successfully made /usr/ read-only.1771client # [ 1.379610] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1772client # [ 1.391473] systemd[1]: Detected virtualization kvm.1773client # [ 1.393557] systemd[1]: Detected architecture x86-64.1774worker2 # [ 1.387827] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1775worker2 # [ 1.392856] systemd[1]: Detected virtualization kvm.1776client # [ 1.395635] systemd[1]: Running in initrd.1777worker2 # [ 1.393726] systemd[1]: Detected architecture x86-64.1778worker2 # [ 1.394604] systemd[1]: Running in initrd.1779worker2 # [ 1.395854] systemd[1]: Initializing machine ID from random generator.1780client # [ 1.398074] systemd[1]: Initializing machine ID from random generator.1781worker2 # [ 1.397202] systemd[1]: Hostname set to <worker2>.1782client # [ 1.400918] systemd[1]: Hostname set to <client>.1783lb # [ 1.398473] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1784worker1 # [ 1.395428] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1785worker1 # [ 1.400557] systemd[1]: Detected virtualization kvm.1786worker1 # [ 1.401396] systemd[1]: Detected architecture x86-64.1787lb # [ 1.410511] systemd[1]: Detected virtualization kvm.1788worker1 # [ 1.402252] systemd[1]: Running in initrd.1789worker1 # [ 1.403348] systemd[1]: Initializing machine ID from random generator.1790lb # [ 1.412724] systemd[1]: Detected architecture x86-64.1791worker1 # [ 1.404518] systemd[1]: Hostname set to <worker1>.1792lb # [ 1.414813] systemd[1]: Running in initrd.1793lb # [ 1.417426] systemd[1]: Initializing machine ID from random generator.1794lb # [ 1.420393] systemd[1]: Hostname set to <lb>.1795worker2 # [ 1.586530] systemd[1]: bpf-restrict-fs: LSM BPF program attached1796client # [ 1.611239] systemd[1]: bpf-restrict-fs: LSM BPF program attached1797worker1 # [ 1.593345] systemd[1]: bpf-restrict-fs: LSM BPF program attached1798worker2 # [ 1.626963] systemd[1]: Queued start job for default target Initrd Default Target.1799worker2 # [ 1.631455] systemd[1]: Created slice Slice /system/modprobe.1800worker2 # [ 1.632712] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1801worker2 # [ 1.634235] systemd[1]: Expecting device /dev/disk/by-label/nixos...1802worker2 # [ 1.635370] systemd[1]: Reached target Path Units.1803worker2 # [ 1.636275] systemd[1]: Reached target Slice Units.1804worker2 # [ 1.637150] systemd[1]: Reached target Swaps.1805worker2 # [ 1.637957] systemd[1]: Reached target Timer Units.1806worker2 # [ 1.639039] systemd[1]: Listening on D-Bus System Message Bus Socket.1807worker2 # [ 1.640316] systemd[1]: Listening on Journal Socket (/dev/log).1808worker2 # [ 1.641475] systemd[1]: Listening on Journal Sockets.1809worker2 # [ 1.642485] systemd[1]: Listening on udev Control Socket.1810worker2 # [ 1.643556] systemd[1]: Listening on udev Kernel Socket.1811worker2 # [ 1.644526] systemd[1]: Reached target Socket Units.1812worker2 # [ 1.646449] systemd[1]: Starting Create List of Static Device Nodes...1813client # [ 1.652264] systemd[1]: Queued start job for default target Initrd Default Target.1814worker1 # [ 1.631339] systemd[1]: Queued start job for default target Initrd Default Target.1815worker2 # [ 1.651110] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1816client # [ 1.655452] systemd[1]: Created slice Slice /system/modprobe.1817worker1 # [ 1.634510] systemd[1]: Created slice Slice /system/modprobe.1818client # [ 1.656652] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1819lb # [ 1.643514] systemd[1]: bpf-restrict-fs: LSM BPF program attached1820client # [ 1.658045] systemd[1]: Expecting device /dev/disk/by-label/nixos...1821worker1 # [ 1.635664] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1822client # [ 1.659155] systemd[1]: Reached target Path Units.1823worker1 # [ 1.637030] systemd[1]: Expecting device /dev/disk/by-label/nixos...1824client # [ 1.659988] systemd[1]: Reached target Slice Units.1825worker1 # [ 1.638145] systemd[1]: Reached target Path Units.1826client # [ 1.660935] systemd[1]: Reached target Swaps.1827worker1 # [ 1.639050] systemd[1]: Reached target Slice Units.1828client # [ 1.661764] systemd[1]: Reached target Timer Units.1829worker1 # [ 1.639945] systemd[1]: Reached target Swaps.1830worker2 # [ 1.659112] systemd[1]: Starting Load Kernel Module configfs...1831client # [ 1.662766] systemd[1]: Listening on D-Bus System Message Bus Socket.1832worker1 # [ 1.640749] systemd[1]: Reached target Timer Units.1833client # [ 1.663949] systemd[1]: Listening on Journal Socket (/dev/log).1834worker1 # [ 1.641714] systemd[1]: Listening on D-Bus System Message Bus Socket.1835client # [ 1.665197] systemd[1]: Listening on Journal Sockets.1836worker1 # [ 1.642998] systemd[1]: Listening on Journal Socket (/dev/log).1837client # [ 1.666193] systemd[1]: Listening on udev Control Socket.1838worker1 # [ 1.644197] systemd[1]: Listening on Journal Sockets.1839client # [ 1.667223] systemd[1]: Listening on udev Kernel Socket.1840worker1 # [ 1.645260] systemd[1]: Listening on udev Control Socket.1841client # [ 1.668149] systemd[1]: Reached target Socket Units.1842worker1 # [ 1.646291] systemd[1]: Listening on udev Kernel Socket.1843worker1 # [ 1.647271] systemd[1]: Reached target Socket Units.1844client # [ 1.669934] systemd[1]: Starting Create List of Static Device Nodes...1845worker1 # [ 1.649068] systemd[1]: Starting Create List of Static Device Nodes...1846client # [ 1.674222] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1847worker1 # [ 1.653031] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1848worker2 # [ 1.674220] systemd[1]: Starting Journal Service...1849client # [ 1.682232] systemd[1]: Starting Load Kernel Module configfs...1850worker1 # [ 1.660408] systemd[1]: Starting Load Kernel Module configfs...1851worker2 # [ 1.687100] systemd[1]: Starting Load Kernel Modules...1852worker1 # [ 1.673412] systemd[1]: Starting Journal Service...1853client # [ 1.696454] systemd[1]: Starting Journal Service...1854client # [ 1.705394] systemd[1]: Starting Load Kernel Modules...1855worker2 # [ 1.694111] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1856worker1 # [ 1.687358] systemd[1]: Starting Load Kernel Modules...1857lb # [ 1.700207] systemd[1]: Queued start job for default target Initrd Default Target.1858worker2 # [ 1.710423] netfs: FS-Cache loaded1859lb # [ 1.704158] systemd[1]: Created slice Slice /system/modprobe.1860worker2 # [ 1.715183] 9pnet: Installing 9P2000 support1861lb # [ 1.705382] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1862lb # [ 1.706806] systemd[1]: Expecting device /dev/disk/by-label/nixos...1863lb # [ 1.707993] systemd[1]: Reached target Path Units.1864lb # [ 1.708927] systemd[1]: Reached target Slice Units.1865lb # [ 1.709820] systemd[1]: Reached target Swaps.1866lb # [ 1.710653] systemd[1]: Reached target Timer Units.1867lb # [ 1.711714] systemd[1]: Listening on D-Bus System Message Bus Socket.1868client # [ 1.712230] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1869lb # [ 1.712916] systemd[1]: Listening on Journal Socket (/dev/log).1870lb # [ 1.714085] systemd[1]: Listening on Journal Sockets.1871worker2 # [ 1.723833] systemd[1]: Starting Coldplug All udev Devices...1872lb # [ 1.715172] systemd[1]: Listening on udev Control Socket.1873worker1 # [ 1.693335] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1874lb # [ 1.716208] systemd[1]: Listening on udev Kernel Socket.1875lb # [ 1.717199] systemd[1]: Reached target Socket Units.1876client # [ 1.730304] netfs: FS-Cache loaded1877lb # [ 1.719178] systemd[1]: Starting Create List of Static Device Nodes...1878worker1 # [ 1.710000] netfs: FS-Cache loaded1879client # [ 1.734917] 9pnet: Installing 9P2000 support1880lb # [ 1.723723] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1881worker1 # [ 1.715121] 9pnet: Installing 9P2000 support1882lb # [ 1.728767] systemd[1]: Starting Load Kernel Module configfs...1883worker2 # [ 1.738126] systemd[1]: Finished Create List of Static Device Nodes.1884client # [ 1.744914] systemd[1]: Starting Coldplug All udev Devices...1885worker2 # [ 1.744658] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1886worker1 # [ 1.726420] systemd[1]: Starting Coldplug All udev Devices...1887lb # [ 1.742622] systemd[1]: Starting Journal Service...1888worker2 # [ 1.753460] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1889client # [ 1.758573] systemd[1]: Finished Create List of Static Device Nodes.1890worker2 # [ 1.758471] systemd[1]: modprobe@configfs.service: Deactivated successfully.1891worker1 # [ 1.741358] systemd[1]: Finished Create List of Static Device Nodes.1892client # [ 1.763719] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1893worker2 # [ 1.761322] systemd-journald[67]: Collecting audit messages is disabled.1894worker1 # [ 1.742679] systemd-journald[67]: Collecting audit messages is disabled.1895worker2 # [ 1.765405] systemd[1]: Finished Load Kernel Module configfs.1896lb # [ 1.756736] systemd[1]: Starting Load Kernel Modules...1897worker1 # [ 1.747810] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1898client # [ 1.772559] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1899client # [ 1.775777] systemd-journald[67]: Collecting audit messages is disabled.1900worker2 # [ 1.771879] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1901worker1 # [ 1.755612] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1902worker2 # [ 1.774372] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1903client # [ 1.778543] systemd[1]: modprobe@configfs.service: Deactivated successfully.1904worker1 # [ 1.760625] systemd[1]: modprobe@configfs.service: Deactivated successfully.1905client # [ 1.785294] systemd[1]: Finished Load Kernel Module configfs.1906worker2 # [ 1.783102] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1907worker2 # [ 1.784614] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1908lb # [ 1.763754] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1909worker1 # [ 1.766593] systemd[1]: Finished Load Kernel Module configfs.1910client # [ 1.790003] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1911client # [ 1.792481] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1912worker1 # [ 1.772634] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1913lb # [ 1.784556] netfs: FS-Cache loaded1914worker1 # [ 1.775115] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1915lb # [ 1.789599] 9pnet: Installing 9P2000 support1916client # [ 1.802219] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1917client # [ 1.805275] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1918worker1 # [ 1.784358] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1919worker1 # [ 1.786099] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1920lb # [ 1.797725] systemd[1]: Starting Coldplug All udev Devices...1921worker2 # [ 1.817655] systemd[1]: Finished Load Kernel Modules.1922lb # [ 1.813991] systemd[1]: Finished Create List of Static Device Nodes.1923worker2 # [ 1.825812] systemd[1]: Starting Apply Kernel Variables...1924lb # [ 1.819365] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1925client # [ 1.833710] systemd[1]: Finished Load Kernel Modules.1926worker1 # [ 1.813778] systemd[1]: Finished Load Kernel Modules.1927lb # [ 1.828128] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1928worker2 # [ 1.838108] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1929client # [ 1.842228] systemd[1]: Starting Apply Kernel Variables...1930worker1 # [ 1.821422] systemd[1]: Starting Apply Kernel Variables...1931lb # [ 1.834140] systemd[1]: modprobe@configfs.service: Deactivated successfully.1932worker2 # [ 1.846228] systemd[1]: Starting Create Static Device Nodes in /dev...1933lb # [ 1.837475] systemd-journald[67]: Collecting audit messages is disabled.1934worker1 # [ 1.831411] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1935lb # [ 1.842066] systemd[1]: Finished Load Kernel Module configfs.1936client # [ 1.855226] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1937lb # [ 1.849168] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1938worker1 # [ 1.840927] systemd[1]: Starting Create Static Device Nodes in /dev...1939client # [ 1.864324] systemd[1]: Starting Create Static Device Nodes in /dev...1940lb # [ 1.853430] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1941worker2 # [ 1.867618] systemd[1]: Finished Apply Kernel Variables.1942lb # [ 1.860898] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1943client # [ 1.875725] systemd[1]: Finished Apply Kernel Variables.1944lb # [ 1.864851] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1945worker2 # [ 1.877599] systemd[1]: Started Journal Service.1946worker1 # [ 1.858537] systemd[1]: Finished Apply Kernel Variables.1947worker2 # [ 1.745616] systemd-modules-load[68]: Inserted module 'dm_mod'1948worker2 # [ 1.749528] systemd-modules-load[68]: Inserted module 'virtio_balloon'1949worker1 # [ 1.863434] systemd[1]: Started Journal Service.1950worker1 # [ 1.733108] systemd-modules-load[68]: Inserted module 'dm_mod'1951worker2 # [ 1.751548] systemd-modules-load[68]: Inserted module 'virtio_gpu'1952worker1 # [ 1.735668] systemd-modules-load[68]: Inserted module 'virtio_balloon'1953client # [ 1.889703] systemd[1]: Started Journal Service.1954worker2 # [ 1.755080] systemd[1]: Finished Create Static Device Nodes in /dev.1955worker1 # [ 1.738284] systemd-modules-load[68]: Inserted module 'virtio_gpu'1956client # [ 1.757230] systemd-modules-load[68]: Inserted module 'dm_mod'1957client # [ 1.760625] systemd-modules-load[68]: Inserted module 'virtio_balloon'1958client # [ 1.763194] systemd-modules-load[68]: Inserted module 'virtio_gpu'1959worker2 # [ 1.762150] systemd[1]: Reached target Preparation for Local File Systems.1960worker2 # [ 1.763231] systemd[1]: Reached target Local File Systems.1961worker2 # [ 1.766194] systemd[1]: Starting Create System Files and Directories...1962worker1 # [ 1.749133] systemd[1]: Finished Create Static Device Nodes in /dev.1963client # [ 1.770905] systemd[1]: Finished Create Static Device Nodes in /dev.1964worker1 # [ 1.751372] systemd[1]: Reached target Preparation for Local File Systems.1965worker2 # [ 1.770625] systemd[1]: Starting Rule-based Manager for Device Events and Files...1966worker1 # [ 1.754092] systemd[1]: Reached target Local File Systems.1967client # [ 1.773931] systemd[1]: Reached target Preparation for Local File Systems.1968worker1 # [ 1.755870] systemd[1]: Starting Create System Files and Directories...1969lb # [ 1.898296] systemd[1]: Finished Load Kernel Modules.1970client # [ 1.776776] systemd[1]: Reached target Local File Systems.1971client # [ 1.777823] systemd[1]: Starting Create System Files and Directories...1972worker1 # [ 1.763069] systemd[1]: Starting Rule-based Manager for Device Events and Files...1973client # [ 1.782784] systemd[1]: Starting Rule-based Manager for Device Events and Files...1974lb # [ 1.906894] systemd[1]: Starting Apply Kernel Variables...1975lb # [ 1.917793] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1976worker2 # [ 1.800573] systemd[1]: Finished Create System Files and Directories.1977worker1 # [ 1.786596] systemd[1]: Finished Create System Files and Directories.1978client # [ 1.810448] systemd[1]: Finished Create System Files and Directories.1979lb # [ 1.932742] systemd[1]: Starting Create Static Device Nodes in /dev...1980worker2 # [ 1.821439] systemd-udevd[83]: Using default interface naming scheme 'v261'.1981worker1 # [ 1.807355] systemd-udevd[84]: Using default interface naming scheme 'v261'.1982lb # [ 1.951822] systemd[1]: Finished Apply Kernel Variables.1983client # [ 1.831363] systemd-udevd[83]: Using default interface naming scheme 'v261'.1984lb # [ 1.959121] systemd[1]: Started Journal Service.1985lb # [ 1.826492] systemd-modules-load[68]: Inserted module 'dm_mod'1986lb # [ 1.830629] systemd-modules-load[68]: Inserted module 'virtio_balloon'1987lb # [ 1.831739] systemd-modules-load[68]: Inserted module 'virtio_gpu'1988worker2 # [ 1.846370] systemd[1]: Started Rule-based Manager for Device Events and Files.1989worker1 # [ 1.831277] systemd[1]: Started Rule-based Manager for Device Events and Files.1990lb # [ 1.842976] systemd[1]: Finished Create Static Device Nodes in /dev.1991client # [ 1.856064] systemd[1]: Started Rule-based Manager for Device Events and Files.1992lb # [ 1.846095] systemd[1]: Reached target Preparation for Local File Systems.1993lb # [ 1.847120] systemd[1]: Reached target Local File Systems.1994lb # [ 1.850174] systemd[1]: Starting Create System Files and Directories...1995lb # [ 1.854839] systemd[1]: Starting Rule-based Manager for Device Events and Files...1996lb # [ 1.883199] systemd[1]: Finished Create System Files and Directories.1997worker2 # [ 1.893761] systemd[1]: Finished Coldplug All udev Devices.1998worker2 # [ 1.894697] systemd[1]: Reached target System Initialization.1999worker2 # [ 1.895572] systemd[1]: Reached target Basic System.2000client # [ 1.902061] systemd[1]: Finished Coldplug All udev Devices.
loading 2,109 hidden lines…
4110lb # [ 18.402509] envoy[921]: [2026-09-21 09:12:10.969][921][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor.4111lb # [ 18.405506] envoy[921]: [2026-09-21 09:12:10.969][921][info][main] [source/server/server.cc:1077] starting main dispatch loop4112worker2 # [ 18.544747] systemd[1]: scheduler-token.service: Deactivated successfully.4113worker2 # [ 18.546486] systemd[1]: Finished scheduler-token.service.4114worker2 # [ 18.547835] systemd[1]: scheduler-token.service: Consumed 35ms CPU time over 145ms wall clock time, 3.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4115worker2 # [ 18.552902] systemd[1]: Starting Nix worker-protocol over gRPC...4116worker1 # [ 18.599535] nix-grpc-daemon[783]: ts=2026-09-21T09:12:11Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4117worker1 # [ 18.609650] nix-grpc-daemon[783]: ts=2026-09-21T09:12:11Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4118worker1 # [ 18.616298] nix-grpc-daemon[783]: ts=2026-09-21T09:12:11Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=14119worker1 # [ 18.649322] nix-grpc-daemon[783]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4120worker1 # [ 18.650669] nix-grpc-daemon[783]: E0000 00:00:1789981931.718854 783 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4121worker1 # [ 18.666424] nix-grpc-daemon[783]: E0000 00:00:1789981931.737144 783 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4122worker1 # [ 18.671472] nix-grpc-daemon[783]: ts=2026-09-21T09:12:11Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4123worker1 # [ 18.676861] systemd[1]: Started Nix worker-protocol over gRPC.4124worker1 # [ 18.705078] nix-grpc-daemon[783]: E0000 00:00:1789981931.775097 824 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4125worker1 # [ 18.726181] nix-grpc-daemon[783]: E0000 00:00:1789981931.796728 824 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4126worker1 # [ 18.730064] nix-grpc-daemon[783]: E0000 00:00:1789981931.800056 824 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4127lb # [ 18.749587] envoy[921]: [2026-09-21 09:12:11.316][921][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization4128worker1 # [ 18.744836] nix-grpc-daemon[783]: E0000 00:00:1789981931.815532 824 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4129lb # [ 18.753980] envoy[921]: [2026-09-21 09:12:11.316][921][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized4130lb # [ 18.758500] envoy[921]: [2026-09-21 09:12:11.316][921][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager4131worker1 # [ 18.757057] nix-grpc-daemon[783]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4132worker2 # [ 18.777792] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4133lb # [ 18.768798] envoy[921]: [2026-09-21 09:12:11.335][921][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers4134worker2 # [ 18.787739] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4135worker2 # [ 18.792696] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14136worker2 # [ 18.832895] nix-grpc-daemon[786]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4137worker2 # [ 18.834294] nix-grpc-daemon[786]: E0000 00:00:1789981931.758017 786 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4138worker2 # [ 18.854187] nix-grpc-daemon[786]: E0000 00:00:1789981931.780842 786 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4139worker2 # [ 18.858494] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4140worker2 # [ 18.861686] systemd[1]: Started Nix worker-protocol over gRPC.4141worker2 # [ 18.917446] nix-grpc-daemon[786]: E0000 00:00:1789981931.843325 828 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4142worker2 # [ 18.932807] nix-grpc-daemon[786]: E0000 00:00:1789981931.859444 828 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4143worker2 # [ 18.938071] nix-grpc-daemon[786]: E0000 00:00:1789981931.863281 828 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4144worker2 # [ 18.949768] nix-grpc-daemon[786]: E0000 00:00:1789981931.876389 828 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4145worker2 # [ 18.979162] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_connected addr=lb:500514146worker1 # [ 18.958167] nix-grpc-daemon[783]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4147worker2 # [ 18.992814] nix-grpc-daemon[786]: ts=2026-09-21T09:12:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4148worker2 # [ 19.195333] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514149worker2 # [ 19.199402] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4150worker1 # [ 19.358883] nix-grpc-daemon[783]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4151worker2 # [ 19.400169] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514152worker2 # [ 19.404111] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4153worker2 # [ 19.605539] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514154worker2 # [ 19.609292] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4155worker2 # [ 19.810982] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514156worker2 # [ 19.814855] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4157worker2 # [ 20.015515] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_connected addr=lb:500514158worker2 # [ 20.018868] nix-grpc-daemon[786]: ts=2026-09-21T09:12:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4159worker1 # [ 20.163699] nix-grpc-daemon[783]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514160worker1 # [ 20.168233] nix-grpc-daemon[783]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4161worker2 # [ 20.220408] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514162worker2 # [ 20.224843] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4163worker1 # [ 20.369373] nix-grpc-daemon[783]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514164worker1 # [ 20.373209] nix-grpc-daemon[783]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4165worker2 # [ 20.426129] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514166worker2 # [ 20.430513] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4167worker1 # [ 20.573984] nix-grpc-daemon[783]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514168worker1 # [ 20.577888] nix-grpc-daemon[783]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4169worker2 # [ 20.631639] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514170worker2 # [ 20.635882] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4171worker1 # [ 20.778338] nix-grpc-daemon[783]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514172worker1 # [ 20.781497] nix-grpc-daemon[783]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4173worker2 # [ 20.838321] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514174worker2 # [ 20.841477] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4175worker1 # [ 20.982386] nix-grpc-daemon[783]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514176worker1 # [ 20.986216] nix-grpc-daemon[783]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4177worker2 # [ 21.039562] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_connected addr=lb:500514178worker2 # [ 21.043712] nix-grpc-daemon[786]: ts=2026-09-21T09:12:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4179worker1 # [ 21.187850] nix-grpc-daemon[783]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514180worker1 # [ 21.191603] nix-grpc-daemon[783]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4181worker2 # [ 21.244921] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514182worker2 # [ 21.249238] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4183worker1 # [ 21.392635] nix-grpc-daemon[783]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514184worker1 # [ 21.396589] nix-grpc-daemon[783]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4185worker2 # [ 21.450212] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514186worker2 # [ 21.453973] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4187worker1 # [ 21.597502] nix-grpc-daemon[783]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514188worker1 # [ 21.601342] nix-grpc-daemon[783]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4189worker2 # [ 21.655263] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514190worker2 # [ 21.659766] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4191worker1 # [ 21.802184] nix-grpc-daemon[783]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514192worker1 # [ 21.806090] nix-grpc-daemon[783]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4193worker2 # [ 21.861604] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514194worker2 # [ 21.865760] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4195worker1 # [ 22.006658] nix-grpc-daemon[783]: ts=2026-09-21T09:12:15Z level=info event=scheduler_connected addr=lb:500514196worker1 # [ 22.010473] nix-grpc-daemon[783]: ts=2026-09-21T09:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4197worker2 # [ 22.066592] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_connected addr=lb:500514198worker2 # [ 22.070413] nix-grpc-daemon[786]: ts=2026-09-21T09:12:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4199worker1 # [ 22.212108] nix-grpc-daemon[783]: ts=2026-09-21T09:12:15Z level=info event=scheduler_connected addr=lb:500514200worker1 # [ 22.215809] nix-grpc-daemon[783]: ts=2026-09-21T09:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4201worker2 # [ 22.271627] nix-grpc-daemon[786]: ts=2026-09-21T09:12:15Z level=info event=scheduler_connected addr=lb:500514202worker2 # [ 22.275447] nix-grpc-daemon[786]: ts=2026-09-21T09:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4203worker1 # [ 22.416869] nix-grpc-daemon[783]: ts=2026-09-21T09:12:15Z level=info event=scheduler_connected addr=lb:500514204worker1 # [ 22.420726] nix-grpc-daemon[783]: ts=2026-09-21T09:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4205worker2 # [ 22.476682] nix-grpc-daemon[786]: ts=2026-09-21T09:12:15Z level=info event=scheduler_connected addr=lb:500514206worker2 # [ 22.480479] nix-grpc-daemon[786]: ts=2026-09-21T09:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4207worker1 # [ 22.621661] nix-grpc-daemon[783]: ts=2026-09-21T09:12:15Z level=info event=scheduler_connected addr=lb:500514208worker1 # [ 22.625461] nix-grpc-daemon[783]: ts=2026-09-21T09:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4209worker2 # [ 22.681606] nix-grpc-daemon[786]: ts=2026-09-21T09:12:15Z level=info event=scheduler_connected addr=lb:500514210worker2 # [ 22.685357] nix-grpc-daemon[786]: ts=2026-09-21T09:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4211worker1 # [ 22.826154] nix-grpc-daemon[783]: ts=2026-09-21T09:12:15Z level=info event=scheduler_connected addr=lb:500514212worker1 # [ 22.829942] nix-grpc-daemon[783]: ts=2026-09-21T09:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4213worker2 # [ 22.886938] nix-grpc-daemon[786]: ts=2026-09-21T09:12:15Z level=info event=scheduler_connected addr=lb:500514214worker2 # [ 22.890742] nix-grpc-daemon[786]: ts=2026-09-21T09:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4215worker1 # [ 23.030765] nix-grpc-daemon[783]: ts=2026-09-21T09:12:16Z level=info event=scheduler_connected addr=lb:500514216worker1 # [ 23.034646] nix-grpc-daemon[783]: ts=2026-09-21T09:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4217worker2 # [ 23.092139] nix-grpc-daemon[786]: ts=2026-09-21T09:12:16Z level=info event=scheduler_connected addr=lb:500514218worker2 # [ 23.097350] nix-grpc-daemon[786]: ts=2026-09-21T09:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4219worker1 # [ 23.236357] nix-grpc-daemon[783]: ts=2026-09-21T09:12:16Z level=info event=scheduler_connected addr=lb:500514220worker1 # [ 23.239956] nix-grpc-daemon[783]: ts=2026-09-21T09:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4221worker2 # [ 23.297396] nix-grpc-daemon[786]: ts=2026-09-21T09:12:16Z level=info event=scheduler_connected addr=lb:500514222worker2 # [ 23.301234] nix-grpc-daemon[786]: ts=2026-09-21T09:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4223worker1 # [ 23.440689] nix-grpc-daemon[783]: ts=2026-09-21T09:12:16Z level=info event=scheduler_connected addr=lb:500514224worker1 # [ 23.444488] nix-grpc-daemon[783]: ts=2026-09-21T09:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4225worker2 # [ 23.501969] nix-grpc-daemon[786]: ts=2026-09-21T09:12:16Z level=info event=scheduler_connected addr=lb:500514226worker2 # [ 23.505762] nix-grpc-daemon[786]: ts=2026-09-21T09:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4227worker1 # [ 23.645311] nix-grpc-daemon[783]: ts=2026-09-21T09:12:16Z level=info event=scheduler_connected addr=lb:500514228worker1 # [ 23.649141] nix-grpc-daemon[783]: ts=2026-09-21T09:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4229worker2 # [ 23.706926] nix-grpc-daemon[786]: ts=2026-09-21T09:12:16Z level=info event=scheduler_connected addr=lb:500514230worker2 # [ 23.710798] nix-grpc-daemon[786]: ts=2026-09-21T09:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4231lb # [ 23.857136] rustfs-bucket-start[985]: mb s3://farm4232worker1 # [ 23.849900] nix-grpc-daemon[783]: ts=2026-09-21T09:12:16Z level=info event=scheduler_connected addr=lb:500514233lb # [ 23.863244] systemd[1]: Finished rustfs-bucket.service.4234worker1 # [ 23.853794] nix-grpc-daemon[783]: ts=2026-09-21T09:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4235lb # [ 23.870457] systemd[1]: Starting niks3 server...4236worker2 # [ 23.913165] nix-grpc-daemon[786]: ts=2026-09-21T09:12:16Z level=info event=scheduler_connected addr=lb:500514237worker2 # [ 23.916444] nix-grpc-daemon[786]: ts=2026-09-21T09:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4238worker1 # [ 24.054412] nix-grpc-daemon[783]: ts=2026-09-21T09:12:17Z level=info event=scheduler_connected addr=lb:500514239worker1 # [ 24.057854] nix-grpc-daemon[783]: ts=2026-09-21T09:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4240lb # [ 24.105487] postgres[1010]: [1010] ERROR: relation "goose_db_version" does not exist at character 364241worker2 # [ 24.114532] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=scheduler_connected addr=lb:500514242lb # [ 24.106686] postgres[1010]: [1010] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4243worker2 # [ 24.118288] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4244lb # [ 24.147430] niks3-server[1003]: 2026/09/21 09:12:16 OK 20241026095416_initial_model.sql (21.48ms)4245lb # [ 24.151676] niks3-server[1003]: 2026/09/21 09:12:16 OK 20251210153512_drop_unused_gin_index.sql (2.87ms)4246lb # [ 24.157511] niks3-server[1003]: 2026/09/21 09:12:16 OK 20251218171726_add_pins.sql (4.42ms)4247lb # [ 24.162803] niks3-server[1003]: 2026/09/21 09:12:16 OK 20260628120000_add_object_size_and_stats.sql (5.34ms)4248lb # [ 24.167380] niks3-server[1003]: 2026/09/21 09:12:16 OK 20260905000000_add_claims.sql (3.17ms)4249lb # [ 24.172094] niks3-server[1003]: 2026/09/21 09:12:16 OK 20260920000000_drop_claims.sql (4.71ms)4250lb # [ 24.173363] niks3-server[1003]: 2026/09/21 09:12:16 goose: successfully migrated database to version: 202609200000004251lb # [ 24.179628] niks3-server[1003]: 2026/09/21 09:12:16 OK 1_commit_pending_closure.sql (4.88ms)4252lb # [ 24.182373] niks3-server[1003]: 2026/09/21 09:12:16 OK 2_object_stats_trigger.sql (1.58ms)4253lb # [ 24.183611] niks3-server[1003]: 2026/09/21 09:12:16 goose: up to current file version: 24254lb # [ 24.190687] niks3-server[1003]: 2026/09/21 09:12:16 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key4255worker1 # [ 24.260104] nix-grpc-daemon[783]: ts=2026-09-21T09:12:17Z level=info event=scheduler_connected addr=lb:500514256worker1 # [ 24.263309] nix-grpc-daemon[783]: ts=2026-09-21T09:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4257lb # [ 24.277645] niks3-server[1003]: 2026/09/21 09:12:16 INFO Read proxy enabled — serving cache objects from S34258lb # [ 24.279454] niks3-server[1003]: 2026/09/21 09:12:16 INFO Using socket-activated listener address=0.0.0.0:57514259lb # [ 24.284098] niks3-server[1003]: 2026/09/21 09:12:16 INFO systemd watchdog enabled interval=15s4260lb # [ 24.286247] systemd[1]: Started niks3 server.4261lb # [ 24.288282] niks3-server[1003]: 2026/09/21 09:12:16 INFO Starting HTTP server address=0.0.0.0:57514262lb # [ 24.290335] systemd[1]: Reached target Multi-User System.4263lb # [ 24.291324] systemd[1]: Startup finished in 862ms (kernel) + 4.210s (initrd) + 19.218s (userspace) = 24.291s.4264lb # [ 24.297227] niks3-server[1003]: 2026/09/21 09:12:16 INFO lead: acquired remote=192.168.1.4:546804265worker2 # [ 24.313696] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=scheduler_take_over4266worker2 # [ 24.318985] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=scheduler_connected addr=lb:500514267worker2 # [ 24.322685] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4268worker1 # [ 24.460980] nix-grpc-daemon[783]: ts=2026-09-21T09:12:17Z level=info event=scheduler_connected addr=lb:500514269worker1 # [ 24.464857] nix-grpc-daemon[783]: ts=2026-09-21T09:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4270worker2 # [ 24.523870] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=scheduler_connected addr=lb:500514271worker2 # [ 24.527701] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4272worker1 # [ 24.665783] nix-grpc-daemon[783]: ts=2026-09-21T09:12:17Z level=info event=scheduler_connected addr=lb:500514273worker1 # [ 24.669561] nix-grpc-daemon[783]: ts=2026-09-21T09:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4274worker2 # [ 24.728726] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=scheduler_connected addr=lb:500514275worker2 # [ 24.732807] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4276worker1 # [ 24.870747] nix-grpc-daemon[783]: ts=2026-09-21T09:12:17Z level=info event=scheduler_connected addr=lb:500514277worker1 # [ 24.874535] nix-grpc-daemon[783]: ts=2026-09-21T09:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4278worker2 # [ 24.934449] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=scheduler_connected addr=lb:500514279worker2 # [ 24.965849] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=oidc_jwks provider=mock keys=14280lb: (finished: waiting for unit niks3.service, in 13.36 seconds)4281worker2 # [ 24.969964] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:594064282lb: waiting for TCP port 5751 on localhost4283worker2 # [ 24.975261] nix-grpc-daemon[786]: ts=2026-09-21T09:12:17Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04284lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!4285lb: (finished: waiting for TCP port 5751 on localhost, in 0.07 seconds)4286lb: waiting for unit envoy.service4287worker1 # [ 25.075434] nix-grpc-daemon[783]: ts=2026-09-21T09:12:18Z level=info event=scheduler_connected addr=lb:500514288worker2 # [ 25.095211] nix-grpc-daemon[786]: ts=2026-09-21T09:12:18Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:594064289worker2 # [ 25.099962] nix-grpc-daemon[786]: ts=2026-09-21T09:12:18Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04290lb: (finished: waiting for unit envoy.service, in 0.07 seconds)4291lb: waiting for TCP port 50051 on localhost4292lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!4293lb # [ 25.138543] envoy[921]: conn peer=[::1]:34584 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=24294lb: (finished: waiting for TCP port 50051 on localhost, in 0.04 seconds)4295worker2 # [ 25.287224] systemd[1]: Starting scheduler-token.service...4296worker2 # [ 25.363081] systemd[1]: scheduler-token.service: Deactivated successfully.4297worker2 # [ 25.364716] systemd[1]: Finished scheduler-token.service.4298worker2 # [ 25.366333] systemd[1]: scheduler-token.service: Consumed 19ms CPU time over 78ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4299lb: waiting until exactly no endpoints of x86_64-linux are unhealthy4300lb: must succeed: curl -sf localhost:9901/clusters4301lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)4302lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.10 seconds)4303subtest: one node schedules and both builders hold a WorkerSession on it4304lb: must succeed: curl -sf localhost:9901/clusters4305lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4306worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4307worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4308lb: must succeed: curl -sf localhost:9901/clusters4309lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4310lb: must succeed: curl -sf localhost:9901/clusters4311lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4312lb: must succeed: curl -sf localhost:9901/clusters4313lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4314lb: must succeed: curl -sf localhost:9901/clusters4315lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4316worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4317worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.05 seconds)4318worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4319worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4320lb: must succeed: curl -sf localhost:9901/clusters4321lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4322lb: must succeed: curl -sf localhost:9901/clusters4323lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4324worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4325worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)4326worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4327worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4328worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4329worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4330worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4331worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4332(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.77 seconds)4333subtest: balancer auth: client cert, bearer token, nothing, unknown CN4334worker1 # [ 26.563805] nix-grpc-daemon[783]: ts=2026-09-21T09:12:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:335884335worker1 # [ 26.577490] systemd[1]: Started Nix Daemon.4336worker1 # [ 26.806563] nix-daemon[845]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4337worker1 # [ 26.860244] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 34 ms (attempt 1/5)4338worker1 # [ 26.896629] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 41 ms (attempt 2/5)4339worker1 # [ 26.941938] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 90 ms (attempt 3/5)4340worker1 # [ 27.037827] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 251 ms (attempt 4/5)4341worker1 # [ 27.300700] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4342worker2 # [ 27.364216] nix-grpc-daemon[786]: ts=2026-09-21T09:12:20Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:594104343worker2 # [ 27.375506] systemd[1]: Started Nix Daemon.4344worker2 # [ 27.615072] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4345worker2 # [ 27.645148] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 19 ms (attempt 1/5)4346worker2 # [ 27.664236] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 32 ms (attempt 2/5)4347worker2 # [ 27.698629] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 10 ms (attempt 3/5)4348worker2 # [ 27.711262] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 327 ms (attempt 4/5)4349worker2 # [ 28.050355] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4350worker2 # [ 28.073324] nix-grpc-daemon[786]: ts=2026-09-21T09:12:20Z level=debug event=path_info_miss path=bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test error="error: path '/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test' is required, but there is no substituter that can build it"4351worker1 # [ 28.071704] nix-grpc-daemon[783]: ts=2026-09-21T09:12:21Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335884352worker1 # [ 28.091164] nix-grpc-daemon[783]: ts=2026-09-21T09:12:21Z level=debug event=path_info_miss path=bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test error="error: path '/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test' is required, but there is no substituter that can build it"4353client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4354client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds)4355client: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4356client: (finished: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4357lb # [ 28.389497] envoy[921]: conn peer=[2001:db8:1::1]:45860 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=406 ms=15814358worker2 # [ 28.485712] nix-grpc-daemon[786]: ts=2026-09-21T09:12:21Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:594104359worker1 # [ 28.501879] nix-grpc-daemon[783]: ts=2026-09-21T09:12:21Z level=info event=oidc_jwks provider=mock keys=14360worker1 # [ 28.506270] nix-grpc-daemon[783]: ts=2026-09-21T09:12:21Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:335884361worker1 # [ 28.515850] nix-grpc-daemon[783]: ts=2026-09-21T09:12:21Z level=debug event=path_info_miss path=bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test error="error: path '/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test' is required, but there is no substituter that can build it"4362worker1 # [ 28.524792] nix-grpc-daemon[783]: ts=2026-09-21T09:12:21Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:335884363worker1 # [ 28.533627] nix-grpc-daemon[783]: ts=2026-09-21T09:12:21Z level=debug event=path_info_miss path=bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test error="error: path '/nix/store/bnxkji68im1y2fipk5gr7ybm5fgsnjir-nixos-system-client-test' is required, but there is no substituter that can build it"4364client: must succeed: rm -r /var/lib/nix-grpc-store4365client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds)4366worker2 # [ 28.814115] nix-grpc-daemon[786]: ts=2026-09-21T09:12:21Z level=info event=denied method=QueryMissing cn=- role=none4367worker1 # [ 29.029571] nix-grpc-daemon[783]: ts=2026-09-21T09:12:22Z level=info event=denied method=QueryMissing cn=stranger role=none4368(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 2.83 seconds)4369subtest: DAG build is scheduled across workers and lands in the cache4370client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24371worker2 # [ 29.275845] nix-grpc-daemon[786]: ts=2026-09-21T09:12:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:594104372client # don't know how to build these paths:4373client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4374worker2 # [ 29.289079] nix-grpc-daemon[786]: ts=2026-09-21T09:12:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104375worker2 # [ 29.327428] nix-grpc-daemon[786]: ts=2026-09-21T09:12:22Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04376worker2 # [ 29.331923] nix-grpc-daemon[786]: ts=2026-09-21T09:12:22Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.4:50051 clients=14377worker2 # [ 29.384207] nix-grpc-daemon[786]: ts=2026-09-21T09:12:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104378worker2 # [ 29.409513] nix-grpc-daemon[786]: ts=2026-09-21T09:12:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104379client # copying 1 paths to 'lb:50051'...4380worker2 # [ 29.422716] nix-grpc-daemon[786]: ts=2026-09-21T09:12:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:594104381client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4382worker2 # [ 29.449888] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4383worker2 # [ 29.479837] nix-grpc-daemon[786]: ts=2026-09-21T09:12:22Z level=info event=niks3_push_started pid=8804384worker2 # [ 29.823156] nix-grpc-daemon[880]: time=2026-09-21T09:12:22.745Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""4385worker2 # [ 30.061644] nix-daemon[869]: accepted connection from pid 894, user nix-grpc-daemon (trusted)4386worker2 # [ 30.076509] nix-daemon[869]: reaped child process 897, status = succeeded4387worker2 # [ 30.268071] nix-daemon[869]: accepted connection from pid 899, user nix-grpc-daemon (trusted)4388worker2 # [ 30.279608] nix-daemon[869]: reaped child process 902, status = succeeded4389lb # [ 30.290550] niks3-server[1003]: 2026/09/21 09:12:22 INFO Received uploads request method=POST path=/api/pending_closures4390worker2 # [ 30.327898] nix-grpc-daemon[880]: time=2026-09-21T09:12:23.254Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4391worker2 # [ 30.330647] nix-grpc-daemon[880]: time=2026-09-21T09:12:23.257Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4392lb # [ 30.367485] niks3-server[1003]: 2026/09/21 09:12:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4393lb # [ 30.384660] niks3-server[1003]: 2026/09/21 09:12:22 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4394lb # [ 30.393434] niks3-server[1003]: 2026/09/21 09:12:22 INFO Signed narinfos id=1 count=14395worker2 # [ 30.405389] nix-grpc-daemon[880]: time=2026-09-21T09:12:23.331Z level=INFO msg="Uploading 1 narinfos"4396lb # [ 30.403123] niks3-server[1003]: 2026/09/21 09:12:22 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4397lb # [ 30.410597] niks3-server[1003]: 2026/09/21 09:12:22 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4398lb # [ 30.420764] niks3-server[1003]: 2026/09/21 09:12:22 INFO Completed upload id=14399worker2 # [ 30.431185] nix-grpc-daemon[880]: time=2026-09-21T09:12:23.357Z level=INFO msg="Upload complete. (582ms)"4400lb # [ 30.426721] niks3-server[1003]: 2026/09/21 09:12:22 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4401worker2 # [ 30.436495] nix-grpc-daemon[786]: ts=2026-09-21T09:12:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=1 paths=1 nar_bytes_in=5044402worker2 # [ 30.443998] nix-grpc-daemon[786]: ts=2026-09-21T09:12:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104403worker2 # [ 30.459279] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4404worker2 # [ 30.463477] nix-daemon[869]: reaped child process 878, status = succeeded4405client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4406worker2 # [ 30.472829] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4407client # farm-a> worker2: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4408client # farm-a> LOG-farm-a-t14409worker2 # [ 31.898960] nix-daemon[869]: reaped child process 906, status = succeeded4410worker2 # [ 32.110888] nix-daemon[869]: accepted connection from pid 921, user nix-grpc-daemon (trusted)4411worker2 # [ 32.122731] nix-daemon[869]: reaped child process 924, status = succeeded4412lb # [ 32.133839] niks3-server[1003]: 2026/09/21 09:12:24 INFO Received uploads request method=POST path=/api/pending_closures4413worker2 # [ 32.156510] nix-grpc-daemon[880]: time=2026-09-21T09:12:25.082Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4414worker2 # [ 32.158168] nix-grpc-daemon[880]: time=2026-09-21T09:12:25.084Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4415lb # [ 32.184583] niks3-server[1003]: 2026/09/21 09:12:24 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4416lb # [ 32.194137] niks3-server[1003]: 2026/09/21 09:12:24 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4417lb # [ 32.203222] niks3-server[1003]: 2026/09/21 09:12:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4418lb # [ 32.208090] niks3-server[1003]: 2026/09/21 09:12:24 INFO Signed narinfos id=2 count=14419worker2 # [ 32.221081] nix-grpc-daemon[880]: time=2026-09-21T09:12:25.147Z level=INFO msg="Uploading 1 narinfos"4420lb # [ 32.212830] niks3-server[1003]: 2026/09/21 09:12:24 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4421lb # [ 32.223178] niks3-server[1003]: 2026/09/21 09:12:24 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4422lb # [ 32.226851] niks3-server[1003]: 2026/09/21 09:12:24 INFO Completed upload id=24423worker2 # [ 32.236899] nix-grpc-daemon[880]: time=2026-09-21T09:12:25.163Z level=INFO msg="Upload complete. (350ms)"4424lb # [ 32.228580] niks3-server[1003]: 2026/09/21 09:12:24 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4425worker2 # [ 32.242890] nix-grpc-daemon[786]: ts=2026-09-21T09:12:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14426worker2 # [ 32.262762] nix-daemon[869]: reaped child process 904, status = succeeded4427worker2 # [ 32.269154] nix-grpc-daemon[786]: ts=2026-09-21T09:12:25Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04428worker2 # [ 32.272454] nix-grpc-daemon[786]: ts=2026-09-21T09:12:25Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.4:50051 clients=14429worker2 # [ 32.314605] nix-grpc-daemon[786]: ts=2026-09-21T09:12:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104430worker2 # [ 32.328176] nix-grpc-daemon[786]: ts=2026-09-21T09:12:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104431worker2 # [ 32.341885] nix-grpc-daemon[880]: time=2026-09-21T09:12:25.268Z level=INFO msg="All 1 paths already cached"4432client # copying 1 paths to 'lb:50051'...4433worker2 # [ 32.350226] nix-grpc-daemon[786]: ts=2026-09-21T09:12:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:594104434client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4435worker2 # [ 32.370153] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4436worker2 # [ 32.586914] nix-daemon[869]: accepted connection from pid 930, user nix-grpc-daemon (trusted)4437worker2 # [ 32.599784] nix-daemon[869]: reaped child process 933, status = succeeded4438worker2 # [ 32.805255] nix-daemon[869]: accepted connection from pid 935, user nix-grpc-daemon (trusted)4439worker2 # [ 32.816632] nix-daemon[869]: reaped child process 939, status = succeeded4440lb # [ 32.827295] niks3-server[1003]: 2026/09/21 09:12:25 INFO Received uploads request method=POST path=/api/pending_closures4441worker2 # [ 32.848591] nix-grpc-daemon[880]: time=2026-09-21T09:12:25.775Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4442worker2 # [ 32.850571] nix-grpc-daemon[880]: time=2026-09-21T09:12:25.776Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4443lb # [ 32.861395] niks3-server[1003]: 2026/09/21 09:12:25 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4444lb # [ 32.871626] niks3-server[1003]: 2026/09/21 09:12:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4445worker2 # [ 32.886091] nix-grpc-daemon[880]: time=2026-09-21T09:12:25.812Z level=INFO msg="Uploading 1 narinfos"4446lb # [ 32.875785] niks3-server[1003]: 2026/09/21 09:12:25 INFO Signed narinfos id=3 count=14447lb # [ 32.891862] niks3-server[1003]: 2026/09/21 09:12:25 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4448lb # [ 32.899215] niks3-server[1003]: 2026/09/21 09:12:25 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4449lb # [ 32.909303] niks3-server[1003]: 2026/09/21 09:12:25 INFO Completed upload id=34450lb # [ 32.910604] niks3-server[1003]: 2026/09/21 09:12:25 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4451worker2 # [ 32.919492] nix-grpc-daemon[880]: time=2026-09-21T09:12:25.846Z level=INFO msg="Upload complete. (535ms)"4452worker2 # [ 32.923967] nix-grpc-daemon[786]: ts=2026-09-21T09:12:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=0 paths=1 nar_bytes_in=6244453worker2 # [ 32.930363] nix-grpc-daemon[786]: ts=2026-09-21T09:12:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104454worker2 # [ 32.942388] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4455worker2 # [ 32.950143] nix-daemon[869]: reaped child process 928, status = succeeded4456client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4457worker2 # [ 32.955395] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4458client # farm-b> worker2: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4459client # farm-b> LOG-farm-b-t14460worker2 # [ 33.778401] nix-daemon[869]: reaped child process 943, status = succeeded4461worker2 # [ 33.952926] nix-daemon[869]: accepted connection from pid 958, user nix-grpc-daemon (trusted)4462worker2 # [ 33.964904] nix-daemon[869]: reaped child process 961, status = succeeded4463lb # [ 33.975826] niks3-server[1003]: 2026/09/21 09:12:26 INFO Received uploads request method=POST path=/api/pending_closures4464worker2 # [ 33.997887] nix-grpc-daemon[880]: time=2026-09-21T09:12:26.923Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4465worker2 # [ 33.999362] nix-grpc-daemon[880]: time=2026-09-21T09:12:26.923Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4466lb # [ 34.016899] niks3-server[1003]: 2026/09/21 09:12:26 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4467lb # [ 34.027197] niks3-server[1003]: 2026/09/21 09:12:26 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4468lb # [ 34.038543] niks3-server[1003]: 2026/09/21 09:12:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4469lb # [ 34.046140] niks3-server[1003]: 2026/09/21 09:12:26 INFO Signed narinfos id=4 count=14470worker2 # [ 34.058657] nix-grpc-daemon[880]: time=2026-09-21T09:12:26.985Z level=INFO msg="Uploading 1 narinfos"4471lb # [ 34.048926] niks3-server[1003]: 2026/09/21 09:12:26 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4472lb # [ 34.059140] niks3-server[1003]: 2026/09/21 09:12:26 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4473lb # [ 34.065674] niks3-server[1003]: 2026/09/21 09:12:26 INFO Completed upload id=44474lb # [ 34.067875] niks3-server[1003]: 2026/09/21 09:12:26 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4475worker2 # [ 34.076311] nix-grpc-daemon[880]: time=2026-09-21T09:12:27.002Z level=INFO msg="Upload complete. (314ms)"4476worker2 # [ 34.086395] nix-grpc-daemon[786]: ts=2026-09-21T09:12:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14477worker2 # [ 34.097814] nix-grpc-daemon[786]: ts=2026-09-21T09:12:27Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04478worker2 # [ 34.102437] nix-grpc-daemon[786]: ts=2026-09-21T09:12:27Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.4:50051 clients=14479worker2 # [ 34.112388] nix-daemon[869]: reaped child process 941, status = succeeded4480worker2 # [ 34.141143] nix-grpc-daemon[786]: ts=2026-09-21T09:12:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104481worker2 # [ 34.154924] nix-grpc-daemon[786]: ts=2026-09-21T09:12:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104482worker2 # [ 34.170111] nix-grpc-daemon[880]: time=2026-09-21T09:12:27.096Z level=INFO msg="All 2 paths already cached"4483client # copying 1 paths to 'lb:50051'...4484client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4485worker2 # [ 34.178879] nix-grpc-daemon[786]: ts=2026-09-21T09:12:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:594104486worker2 # [ 34.187746] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4487worker2 # [ 34.403164] nix-daemon[869]: accepted connection from pid 967, user nix-grpc-daemon (trusted)4488worker2 # [ 34.415307] nix-daemon[869]: reaped child process 970, status = succeeded4489worker2 # [ 34.613819] nix-daemon[869]: accepted connection from pid 972, user nix-grpc-daemon (trusted)4490worker2 # [ 34.626132] nix-daemon[869]: reaped child process 975, status = succeeded4491lb # [ 34.635204] niks3-server[1003]: 2026/09/21 09:12:27 INFO Received uploads request method=POST path=/api/pending_closures4492worker2 # [ 34.655305] nix-grpc-daemon[880]: time=2026-09-21T09:12:27.581Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4493worker2 # [ 34.657209] nix-grpc-daemon[880]: time=2026-09-21T09:12:27.583Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4494lb # [ 34.665698] niks3-server[1003]: 2026/09/21 09:12:27 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4495lb # [ 34.679302] niks3-server[1003]: 2026/09/21 09:12:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4496lb # [ 34.685104] niks3-server[1003]: 2026/09/21 09:12:27 INFO Signed narinfos id=5 count=14497worker2 # [ 34.698395] nix-grpc-daemon[880]: time=2026-09-21T09:12:27.625Z level=INFO msg="Uploading 1 narinfos"4498lb # [ 34.691629] niks3-server[1003]: 2026/09/21 09:12:27 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4499lb # [ 34.706816] niks3-server[1003]: 2026/09/21 09:12:27 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4500lb # [ 34.712416] niks3-server[1003]: 2026/09/21 09:12:27 INFO Completed upload id=54501worker2 # [ 34.724802] nix-grpc-daemon[880]: time=2026-09-21T09:12:27.651Z level=INFO msg="Upload complete. (521ms)"4502lb # [ 34.715599] niks3-server[1003]: 2026/09/21 09:12:27 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4503worker2 # [ 34.729303] nix-grpc-daemon[786]: ts=2026-09-21T09:12:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=0 paths=1 nar_bytes_in=7524504worker2 # [ 34.735316] nix-grpc-daemon[786]: ts=2026-09-21T09:12:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104505worker2 # [ 34.746117] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4506worker2 # [ 34.754397] nix-daemon[869]: reaped child process 965, status = succeeded4507worker2 # [ 34.761517] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4508client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4509client # farm-top> worker2: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4510client # farm-top> LOG-farm-top-t14511worker2 # [ 35.979892] nix-daemon[869]: reaped child process 979, status = succeeded4512worker2 # [ 36.163061] nix-daemon[869]: accepted connection from pid 994, user nix-grpc-daemon (trusted)4513worker2 # [ 36.173661] nix-daemon[869]: reaped child process 998, status = succeeded4514lb # [ 36.184538] niks3-server[1003]: 2026/09/21 09:12:28 INFO Received uploads request method=POST path=/api/pending_closures4515worker2 # [ 36.199428] nix-grpc-daemon[880]: time=2026-09-21T09:12:29.125Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4516worker2 # [ 36.200929] nix-grpc-daemon[880]: time=2026-09-21T09:12:29.127Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4517lb # [ 36.203584] niks3-server[1003]: 2026/09/21 09:12:28 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4518lb # [ 36.211529] niks3-server[1003]: 2026/09/21 09:12:28 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4519lb # [ 36.213852] niks3-server[1003]: 2026/09/21 09:12:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4520lb # [ 36.216045] niks3-server[1003]: 2026/09/21 09:12:28 INFO Signed narinfos id=6 count=14521worker2 # [ 36.226121] nix-grpc-daemon[880]: time=2026-09-21T09:12:29.152Z level=INFO msg="Uploading 1 narinfos"4522lb # [ 36.219394] niks3-server[1003]: 2026/09/21 09:12:28 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4523lb # [ 36.224430] niks3-server[1003]: 2026/09/21 09:12:28 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4524lb # [ 36.228119] niks3-server[1003]: 2026/09/21 09:12:28 INFO Completed upload id=64525worker2 # [ 36.238078] nix-grpc-daemon[880]: time=2026-09-21T09:12:29.164Z level=INFO msg="Upload complete. (269ms)"4526lb # [ 36.229701] niks3-server[1003]: 2026/09/21 09:12:28 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4527worker2 # [ 36.240510] nix-grpc-daemon[786]: ts=2026-09-21T09:12:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14528worker2 # [ 36.247653] nix-daemon[869]: reaped child process 977, status = succeeded4529client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 7.21 seconds)4530client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4531client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.19 seconds)4532client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4533client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.01 seconds)4534client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4535client # copying 3 paths...4536client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4537client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4538client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4539client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.26 seconds)4540(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.77 seconds)4541subtest: repeat is answered Cached by the scheduler without building4542worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4543worker1 # finding garbage collector roots...4544worker1 # deleting specified paths...4545worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4546worker1 # deleting unused links...4547worker1 # note: hard linking is currently saving 0.0 KiB4548worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.27 seconds)4549worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4550worker2 # finding garbage collector roots...4551worker2 # deleting specified paths...4552worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4553worker2 # deleting unused links...4554worker2 # note: hard linking is currently saving 0.0 KiB4555worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.28 seconds)4556client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24557worker2 # [ 37.605212] nix-grpc-daemon[786]: ts=2026-09-21T09:12:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:594104558client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4559client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4560worker2 # [ 37.623837] nix-grpc-daemon[786]: ts=2026-09-21T09:12:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104561worker2 # [ 37.634232] nix-grpc-daemon[880]: time=2026-09-21T09:12:30.560Z level=INFO msg="All 2 paths already cached"4562worker2 # [ 37.648658] nix-grpc-daemon[786]: ts=2026-09-21T09:12:30Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14563client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.28 seconds)4564client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4565client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.18 seconds)4566lb: must succeed: curl -sf localhost:9901/clusters4567lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4568lb: must succeed: curl -sf localhost:9901/clusters4569lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4570worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4571worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4572(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.28 seconds)4573subtest: inputs already in the cache are not sent to the scheduler4574lb: must succeed: curl -sf localhost:9901/clusters4575lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4576lb: must succeed: curl -sf localhost:9901/clusters4577lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4578worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4579worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4580client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24581lb # [ 38.391116] envoy[921]: conn peer=[2001:db8:1::1]:45872 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=794582lb # [ 38.394635] envoy[921]: conn peer=[2001:db8:1::1]:45876 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=94583lb # [ 38.397880] envoy[921]: conn peer=[2001:db8:1::1]:45884 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=84584lb # [ 38.401303] envoy[921]: conn peer=[2001:db8:1::1]:45900 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9481 tx=3824 ms=70084585lb # [ 38.405565] envoy[921]: conn peer=[2001:db8:1::1]:59776 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=584586client # don't know how to build these paths:4587client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4588worker1 # [ 38.490078] nix-grpc-daemon[783]: ts=2026-09-21T09:12:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:335884589worker2 # [ 38.512348] nix-grpc-daemon[786]: ts=2026-09-21T09:12:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104590worker2 # [ 38.521446] nix-grpc-daemon[880]: time=2026-09-21T09:12:31.448Z level=INFO msg="All 2 paths already cached"4591worker2 # [ 38.534155] nix-grpc-daemon[786]: ts=2026-09-21T09:12:31Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04592worker2 # [ 38.539407] nix-grpc-daemon[786]: ts=2026-09-21T09:12:31Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.4:50051 clients=14593worker2 # [ 38.579283] nix-grpc-daemon[786]: ts=2026-09-21T09:12:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104594worker2 # [ 38.592189] nix-grpc-daemon[786]: ts=2026-09-21T09:12:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104595worker2 # [ 38.604860] nix-grpc-daemon[880]: time=2026-09-21T09:12:31.531Z level=INFO msg="All 2 paths already cached"4596client # copying 1 paths to 'lb:50051'...4597client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4598worker2 # [ 38.613545] nix-grpc-daemon[786]: ts=2026-09-21T09:12:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:594104599worker2 # [ 38.629388] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4600worker2 # [ 38.830216] nix-daemon[869]: accepted connection from pid 1027, user nix-grpc-daemon (trusted)4601worker2 # [ 38.843128] nix-daemon[869]: reaped child process 1030, status = succeeded4602worker2 # [ 39.021255] nix-daemon[869]: accepted connection from pid 1032, user nix-grpc-daemon (trusted)4603worker2 # [ 39.033092] nix-daemon[869]: reaped child process 1035, status = succeeded4604lb # [ 39.041936] niks3-server[1003]: 2026/09/21 09:12:31 INFO Received uploads request method=POST path=/api/pending_closures4605worker2 # [ 39.063505] nix-grpc-daemon[880]: time=2026-09-21T09:12:31.989Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4606worker2 # [ 39.065031] nix-grpc-daemon[880]: time=2026-09-21T09:12:31.991Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4607lb # [ 39.073671] niks3-server[1003]: 2026/09/21 09:12:31 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4608lb # [ 39.084372] niks3-server[1003]: 2026/09/21 09:12:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4609lb # [ 39.088622] niks3-server[1003]: 2026/09/21 09:12:31 INFO Signed narinfos id=7 count=14610worker2 # [ 39.101791] nix-grpc-daemon[880]: time=2026-09-21T09:12:32.028Z level=INFO msg="Uploading 1 narinfos"4611lb # [ 39.098481] niks3-server[1003]: 2026/09/21 09:12:31 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4612lb # [ 39.108883] niks3-server[1003]: 2026/09/21 09:12:31 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4613worker2 # [ 39.125565] nix-grpc-daemon[880]: time=2026-09-21T09:12:32.052Z level=INFO msg="Upload complete. (484ms)"4614lb # [ 39.115774] niks3-server[1003]: 2026/09/21 09:12:31 INFO Completed upload id=74615worker2 # [ 39.127739] nix-grpc-daemon[786]: ts=2026-09-21T09:12:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=0 paths=1 nar_bytes_in=7524616lb # [ 39.119262] niks3-server[1003]: 2026/09/21 09:12:31 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4617worker2 # [ 39.132132] nix-grpc-daemon[786]: ts=2026-09-21T09:12:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104618worker2 # [ 39.139499] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4619worker2 # [ 39.141356] nix-daemon[869]: reaped child process 1025, status = succeeded4620worker2 # [ 39.147307] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4621client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4622client # farm-top> worker2: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4623client # farm-top> LOG-farm-top-t1b4624worker2 # [ 40.572959] nix-daemon[869]: reaped child process 1039, status = succeeded4625worker2 # [ 40.759859] nix-daemon[869]: accepted connection from pid 1055, user nix-grpc-daemon (trusted)4626worker2 # [ 40.771674] nix-daemon[869]: reaped child process 1058, status = succeeded4627lb # [ 40.781356] niks3-server[1003]: 2026/09/21 09:12:33 INFO Received uploads request method=POST path=/api/pending_closures4628worker2 # [ 40.803173] nix-grpc-daemon[880]: time=2026-09-21T09:12:33.729Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4629worker2 # [ 40.805586] nix-grpc-daemon[880]: time=2026-09-21T09:12:33.731Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4630lb # [ 40.821208] niks3-server[1003]: 2026/09/21 09:12:33 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4631lb # [ 40.833301] niks3-server[1003]: 2026/09/21 09:12:33 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4632lb # [ 40.843681] niks3-server[1003]: 2026/09/21 09:12:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4633worker2 # [ 40.858143] nix-grpc-daemon[880]: time=2026-09-21T09:12:33.784Z level=INFO msg="Uploading 1 narinfos"4634lb # [ 40.848097] niks3-server[1003]: 2026/09/21 09:12:33 INFO Signed narinfos id=8 count=14635lb # [ 40.853138] niks3-server[1003]: 2026/09/21 09:12:33 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4636lb # [ 40.862261] niks3-server[1003]: 2026/09/21 09:12:33 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4637lb # [ 40.865332] niks3-server[1003]: 2026/09/21 09:12:33 INFO Completed upload id=84638lb # [ 40.867348] niks3-server[1003]: 2026/09/21 09:12:33 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4639worker2 # [ 40.876654] nix-grpc-daemon[880]: time=2026-09-21T09:12:33.803Z level=INFO msg="Upload complete. (318ms)"4640worker2 # [ 40.883704] nix-grpc-daemon[786]: ts=2026-09-21T09:12:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14641worker2 # [ 40.895895] nix-daemon[869]: reaped child process 1037, status = succeeded4642client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.64 seconds)4643client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4644client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.19 seconds)4645lb: must succeed: curl -sf localhost:9901/clusters4646lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4647lb: must succeed: curl -sf localhost:9901/clusters4648lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4649worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4650worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4651(finished: subtest: inputs already in the cache are not sent to the scheduler, in 3.17 seconds)4652subtest: two clients wanting the same drv build it once4653client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4654client # [ 41.350801] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4655client # Running as unit: dedup1.service; invocation ID: 060da656486943239efa83f4212fe27c4656client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.07 seconds)4657client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4658worker1 # [ 41.539377] nix-grpc-daemon[783]: ts=2026-09-21T09:12:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:335884659client # [ 41.560142] nix[891]: don't know how to build these paths:4660client # [ 41.561411] nix[891]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4661worker2 # [ 41.567283] nix-grpc-daemon[786]: ts=2026-09-21T09:12:34Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04662worker2 # [ 41.571487] nix-grpc-daemon[786]: ts=2026-09-21T09:12:34Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.3:50051 clients=14663worker1 # [ 41.561858] nix-grpc-daemon[783]: ts=2026-09-21T09:12:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335884664worker1 # [ 41.573722] nix-grpc-daemon[783]: ts=2026-09-21T09:12:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335884665client # [ 41.599960] nix[891]: copying 1 paths to 'lb:50051'...4666worker1 # [ 41.580670] nix-grpc-daemon[783]: ts=2026-09-21T09:12:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:335884667client # [ 41.600745] nix[891]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4668worker1 # [ 41.586905] nix-daemon[845]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4669worker1 # [ 41.601136] nix-grpc-daemon[783]: ts=2026-09-21T09:12:34Z level=info event=niks3_push_started pid=8724670worker1 # [ 41.904621] nix-grpc-daemon[872]: time=2026-09-21T09:12:34.971Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""4671worker1 # [ 42.128655] nix-daemon[845]: accepted connection from pid 885, user nix-grpc-daemon (trusted)4672worker1 # [ 42.145689] nix-daemon[845]: reaped child process 888, status = succeeded4673worker1 # [ 42.325079] nix-daemon[845]: accepted connection from pid 890, user nix-grpc-daemon (trusted)4674worker1 # [ 42.337136] nix-daemon[845]: reaped child process 893, status = succeeded4675lb # [ 42.365478] niks3-server[1003]: 2026/09/21 09:12:34 INFO Received uploads request method=POST path=/api/pending_closures4676worker1 # [ 42.365810] nix-grpc-daemon[872]: time=2026-09-21T09:12:35.436Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4677worker1 # [ 42.368407] nix-grpc-daemon[872]: time=2026-09-21T09:12:35.439Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4678lb # [ 42.402180] niks3-server[1003]: 2026/09/21 09:12:34 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4679lb # [ 42.414120] niks3-server[1003]: 2026/09/21 09:12:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4680worker1 # [ 42.411728] nix-grpc-daemon[872]: time=2026-09-21T09:12:35.482Z level=INFO msg="Uploading 1 narinfos"4681lb # [ 42.418962] niks3-server[1003]: 2026/09/21 09:12:34 INFO Signed narinfos id=9 count=14682client # [ 42.427280] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4683client # Running as unit: dedup2.service; invocation ID: 2e027f2c40d24507803bc3fc143bf4014684lb # [ 42.430712] niks3-server[1003]: 2026/09/21 09:12:34 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4685client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.08 seconds)4686lb # [ 42.439894] niks3-server[1003]: 2026/09/21 09:12:35 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4687worker1 # [ 42.438159] nix-grpc-daemon[872]: time=2026-09-21T09:12:35.508Z level=INFO msg="Upload complete. (507ms)"4688lb # [ 42.445483] niks3-server[1003]: 2026/09/21 09:12:35 INFO Completed upload id=94689lb # [ 42.447945] niks3-server[1003]: 2026/09/21 09:12:35 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4690worker1 # [ 42.440561] nix-grpc-daemon[783]: ts=2026-09-21T09:12:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33588 duration_s=0 paths=1 nar_bytes_in=4324691worker1 # [ 42.449066] nix-grpc-daemon[783]: ts=2026-09-21T09:12:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335884692worker1 # [ 42.453607] nix-daemon[845]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4693worker1 # [ 42.456158] nix-daemon[845]: reaped child process 870, status = succeeded4694worker1 # [ 42.463297] nix-daemon[845]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4695client # [ 42.484278] nix[891]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4696worker1 # [ 42.691864] nix-grpc-daemon[783]: ts=2026-09-21T09:12:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:335884697client # [ 42.722078] nix[907]: this derivation will be built:4698worker2 # [ 42.722393] nix-grpc-daemon[786]: ts=2026-09-21T09:12:35Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:594104699worker2 # [ 42.739674] nix-grpc-daemon[786]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' from 'http://lb:5751'...4700client # [ 42.760440] nix[907]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4701worker2 # [ 42.766256] nix-grpc-daemon[786]: ts=2026-09-21T09:12:35Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04702worker1 # [ 42.756305] nix-grpc-daemon[783]: ts=2026-09-21T09:12:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335884703client # [ 42.781264] nix[907]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4704client: must succeed: sleep 24705client: (finished: must succeed: sleep 2, in 2.05 seconds)4706client: waiting for success: ! systemctl is-active dedup1 dedup24707worker1 # [ 47.091608] nix-daemon[845]: reaped child process 901, status = succeeded4708worker1 # [ 47.275083] nix-daemon[845]: accepted connection from pid 931, user nix-grpc-daemon (trusted)4709worker1 # [ 47.285597] nix-daemon[845]: reaped child process 934, status = succeeded4710lb # [ 47.312953] niks3-server[1003]: 2026/09/21 09:12:39 INFO Received uploads request method=POST path=/api/pending_closures4711worker1 # [ 47.316028] nix-grpc-daemon[872]: time=2026-09-21T09:12:40.386Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4712worker1 # [ 47.317647] nix-grpc-daemon[872]: time=2026-09-21T09:12:40.388Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4713lb # [ 47.362168] niks3-server[1003]: 2026/09/21 09:12:39 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4714lb # [ 47.368750] niks3-server[1003]: 2026/09/21 09:12:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4715worker1 # [ 47.367463] nix-grpc-daemon[872]: time=2026-09-21T09:12:40.437Z level=INFO msg="Uploading 1 narinfos"4716lb # [ 47.374419] niks3-server[1003]: 2026/09/21 09:12:39 INFO Signed narinfos id=10 count=14717lb # [ 47.381561] niks3-server[1003]: 2026/09/21 09:12:39 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4718lb # [ 47.390187] niks3-server[1003]: 2026/09/21 09:12:39 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4719lb # [ 47.395954] niks3-server[1003]: 2026/09/21 09:12:39 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4720lb # [ 47.403266] niks3-server[1003]: 2026/09/21 09:12:39 INFO Completed upload id=104721worker1 # [ 47.396054] nix-grpc-daemon[872]: time=2026-09-21T09:12:40.466Z level=INFO msg="Upload complete. (318ms)"4722lb # [ 47.405613] niks3-server[1003]: 2026/09/21 09:12:39 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo4723worker1 # [ 47.402617] nix-grpc-daemon[783]: ts=2026-09-21T09:12:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33588 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14724worker1 # [ 47.409226] nix-grpc-daemon[783]: ts=2026-09-21T09:12:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33588 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14725worker1 # [ 47.418704] nix-daemon[845]: reaped child process 899, status = succeeded4726client # [ 47.471862] systemd[1]: dedup1.service: Deactivated successfully.4727client # [ 47.475714] systemd[1]: dedup1.service: Consumed 118ms CPU time over 6.124s wall clock time, 41.5M memory peak, 7K incoming IP traffic, 8.3K outgoing IP traffic.4728client # [ 47.481692] systemd[1]: dedup2.service: Deactivated successfully.4729client # [ 47.483627] systemd[1]: dedup2.service: Consumed 146ms CPU time over 5.054s wall clock time, 35.1M memory peak, 6.3K incoming IP traffic, 5.9K outgoing IP traffic.4730client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.11 seconds)4731client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4732client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)4733worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4734worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)4735worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4736worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)4737worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4738worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4739worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4740lb # [ 48.391887] envoy[921]: conn peer=[2001:db8:1::1]:59792 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4169 tx=1726 ms=23884741lb # [ 48.395798] envoy[921]: conn peer=[2001:db8:1::1]:59808 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2972 tx=1370 ms=58754742lb # [ 48.399364] envoy[921]: conn peer=[2001:db8:1::1]:59824 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1602 tx=1487 ms=47364743worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4744(finished: subtest: two clients wanting the same drv build it once, in 7.14 seconds)4745subtest: build hook: nix-daemon with builders = grpc://lb4746client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log4747client # [ 48.627715] systemd[1]: Started Nix Daemon.4748worker1 # [ 48.709824] nix-grpc-daemon[783]: E0000 00:00:1789981961.778561 825 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4749worker1 # [ 48.739697] nix-grpc-daemon[783]: E0000 00:00:1789981961.810200 825 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4750worker1 # [ 48.748307] nix-grpc-daemon[783]: E0000 00:00:1789981961.815469 825 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4751client # [ 48.786130] nix-daemon[950]: accepted connection from pid 947, user root (trusted)4752worker1 # [ 48.770783] nix-grpc-daemon[783]: E0000 00:00:1789981961.841106 825 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4753worker2 # [ 48.967898] nix-grpc-daemon[786]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:594104754worker2 # [ 48.975526] nix-grpc-daemon[786]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104755worker2 # [ 48.986907] nix-grpc-daemon[786]: ts=2026-09-21T09:12:41Z level=debug event=want client=6 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=04756worker2 # [ 48.992712] nix-grpc-daemon[786]: ts=2026-09-21T09:12:41Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.4:50051 clients=14757worker2 # [ 49.031670] nix-grpc-daemon[786]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104758worker2 # [ 49.043961] nix-grpc-daemon[786]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104759worker2 # [ 49.056137] nix-grpc-daemon[786]: ts=2026-09-21T09:12:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:594104760worker2 # [ 49.070322] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4761worker2 # [ 49.246819] nix-daemon[869]: accepted connection from pid 1100, user nix-grpc-daemon (trusted)4762worker2 # [ 49.259802] nix-daemon[869]: reaped child process 1103, status = succeeded4763worker2 # [ 49.411112] nix-daemon[869]: accepted connection from pid 1105, user nix-grpc-daemon (trusted)4764worker2 # [ 49.422771] nix-daemon[869]: reaped child process 1108, status = succeeded4765lb # [ 49.429171] niks3-server[1003]: 2026/09/21 09:12:41 INFO Received uploads request method=POST path=/api/pending_closures4766worker2 # [ 49.446235] nix-grpc-daemon[880]: time=2026-09-21T09:12:42.372Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4767worker2 # [ 49.447656] nix-grpc-daemon[880]: time=2026-09-21T09:12:42.372Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"4768lb # [ 49.449103] niks3-server[1003]: 2026/09/21 09:12:42 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst4769lb # [ 49.458889] niks3-server[1003]: 2026/09/21 09:12:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4770worker2 # [ 49.473919] nix-grpc-daemon[880]: time=2026-09-21T09:12:42.400Z level=INFO msg="Uploading 1 narinfos"4771lb # [ 49.464109] niks3-server[1003]: 2026/09/21 09:12:42 INFO Signed narinfos id=11 count=14772lb # [ 49.470507] niks3-server[1003]: 2026/09/21 09:12:42 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls4773lb # [ 49.480413] niks3-server[1003]: 2026/09/21 09:12:42 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4774lb # [ 49.486446] niks3-server[1003]: 2026/09/21 09:12:42 INFO Completed upload id=114775worker2 # [ 49.496541] nix-grpc-daemon[880]: time=2026-09-21T09:12:42.423Z level=INFO msg="Upload complete. (414ms)"4776lb # [ 49.488034] niks3-server[1003]: 2026/09/21 09:12:42 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo4777worker2 # [ 49.498650] nix-grpc-daemon[786]: ts=2026-09-21T09:12:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=0 paths=1 nar_bytes_in=5124778worker2 # [ 49.502654] nix-grpc-daemon[786]: ts=2026-09-21T09:12:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104779worker2 # [ 49.508391] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4780worker2 # [ 49.510648] nix-daemon[869]: reaped child process 1098, status = succeeded4781worker2 # [ 49.515817] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4782worker2 # [ 50.342808] nix-daemon[869]: reaped child process 1112, status = succeeded4783worker2 # [ 50.501269] nix-daemon[869]: accepted connection from pid 1127, user nix-grpc-daemon (trusted)4784worker2 # [ 50.512737] nix-daemon[869]: reaped child process 1130, status = succeeded4785lb # [ 50.520616] niks3-server[1003]: 2026/09/21 09:12:43 INFO Received uploads request method=POST path=/api/pending_closures4786worker2 # [ 50.538955] nix-grpc-daemon[880]: time=2026-09-21T09:12:43.465Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4787worker2 # [ 50.540425] nix-grpc-daemon[880]: time=2026-09-21T09:12:43.467Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"4788lb # [ 50.552876] niks3-server[1003]: 2026/09/21 09:12:43 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv4789lb # [ 50.564525] niks3-server[1003]: 2026/09/21 09:12:43 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4790lb # [ 50.576382] niks3-server[1003]: 2026/09/21 09:12:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4791worker2 # [ 50.591649] nix-grpc-daemon[880]: time=2026-09-21T09:12:43.518Z level=INFO msg="Uploading 1 narinfos"4792lb # [ 50.581725] niks3-server[1003]: 2026/09/21 09:12:43 INFO Signed narinfos id=12 count=14793lb # [ 50.586571] niks3-server[1003]: 2026/09/21 09:12:43 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls4794lb # [ 50.595416] niks3-server[1003]: 2026/09/21 09:12:43 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4795lb # [ 50.600796] niks3-server[1003]: 2026/09/21 09:12:43 INFO Completed upload id=124796worker2 # [ 50.610998] nix-grpc-daemon[880]: time=2026-09-21T09:12:43.537Z level=INFO msg="Upload complete. (288ms)"4797lb # [ 50.603588] niks3-server[1003]: 2026/09/21 09:12:43 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo4798worker2 # [ 50.616818] nix-grpc-daemon[786]: ts=2026-09-21T09:12:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=6 outputs=14799worker2 # [ 50.629990] nix-daemon[869]: reaped child process 1110, status = succeeded4800worker2 # [ 50.662115] nix-grpc-daemon[786]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:594104801worker2 # [ 50.688496] nix-grpc-daemon[786]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:594104802worker2 # [ 50.693745] nix-grpc-daemon[786]: ts=2026-09-21T09:12:43Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=0 paths=1 nar_bytes_out=1284803worker1 # [ 50.941939] nix-grpc-daemon[783]: ts=2026-09-21T09:12:44Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:335884804worker1 # [ 50.948605] nix-grpc-daemon[783]: ts=2026-09-21T09:12:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335884805worker2 # [ 50.987766] nix-grpc-daemon[786]: ts=2026-09-21T09:12:43Z level=debug event=want client=7 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=04806worker2 # [ 50.993195] nix-grpc-daemon[786]: ts=2026-09-21T09:12:43Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.4:50051 clients=14807worker2 # [ 50.998820] nix-grpc-daemon[786]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104808worker2 # [ 51.010167] nix-grpc-daemon[786]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104809worker2 # [ 51.022915] nix-grpc-daemon[880]: time=2026-09-21T09:12:43.949Z level=INFO msg="All 1 paths already cached"4810worker2 # [ 51.030383] nix-grpc-daemon[786]: ts=2026-09-21T09:12:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:594104811worker2 # [ 51.040172] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4812worker2 # [ 51.211203] nix-daemon[869]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)4813worker2 # [ 51.223787] nix-daemon[869]: reaped child process 1139, status = succeeded4814worker2 # [ 51.387612] nix-daemon[869]: accepted connection from pid 1141, user nix-grpc-daemon (trusted)4815worker2 # [ 51.399227] nix-daemon[869]: reaped child process 1144, status = succeeded4816lb # [ 51.406255] niks3-server[1003]: 2026/09/21 09:12:43 INFO Received uploads request method=POST path=/api/pending_closures4817worker2 # [ 51.424857] nix-grpc-daemon[880]: time=2026-09-21T09:12:44.351Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4818worker2 # [ 51.426316] nix-grpc-daemon[880]: time=2026-09-21T09:12:44.353Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"4819lb # [ 51.437191] niks3-server[1003]: 2026/09/21 09:12:44 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst4820lb # [ 51.445170] niks3-server[1003]: 2026/09/21 09:12:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4821lb # [ 51.450115] niks3-server[1003]: 2026/09/21 09:12:44 INFO Signed narinfos id=13 count=14822worker2 # [ 51.462900] nix-grpc-daemon[880]: time=2026-09-21T09:12:44.389Z level=INFO msg="Uploading 1 narinfos"4823lb # [ 51.461162] niks3-server[1003]: 2026/09/21 09:12:44 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls4824lb # [ 51.469267] niks3-server[1003]: 2026/09/21 09:12:44 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4825lb # [ 51.475291] niks3-server[1003]: 2026/09/21 09:12:44 INFO Completed upload id=134826worker2 # [ 51.485455] nix-grpc-daemon[880]: time=2026-09-21T09:12:44.412Z level=INFO msg="Upload complete. (433ms)"4827lb # [ 51.477904] niks3-server[1003]: 2026/09/21 09:12:44 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo4828worker2 # [ 51.488774] nix-grpc-daemon[786]: ts=2026-09-21T09:12:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=0 paths=1 nar_bytes_in=6244829worker2 # [ 51.494383] nix-grpc-daemon[786]: ts=2026-09-21T09:12:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104830worker2 # [ 51.504083] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4831worker2 # [ 51.507409] nix-daemon[869]: reaped child process 1134, status = succeeded4832worker2 # [ 51.512436] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4833worker2 # [ 52.331388] nix-daemon[869]: reaped child process 1148, status = succeeded4834worker2 # [ 52.497596] nix-daemon[869]: accepted connection from pid 1163, user nix-grpc-daemon (trusted)4835worker2 # [ 52.507982] nix-daemon[869]: reaped child process 1166, status = succeeded4836lb # [ 52.515328] niks3-server[1003]: 2026/09/21 09:12:45 INFO Received uploads request method=POST path=/api/pending_closures4837worker2 # [ 52.533665] nix-grpc-daemon[880]: time=2026-09-21T09:12:45.460Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4838worker2 # [ 52.536105] nix-grpc-daemon[880]: time=2026-09-21T09:12:45.461Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"4839lb # [ 52.555674] niks3-server[1003]: 2026/09/21 09:12:45 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst4840lb # [ 52.563059] niks3-server[1003]: 2026/09/21 09:12:45 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv4841lb # [ 52.569340] niks3-server[1003]: 2026/09/21 09:12:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4842lb # [ 52.574911] niks3-server[1003]: 2026/09/21 09:12:45 INFO Signed narinfos id=14 count=14843worker2 # [ 52.587892] nix-grpc-daemon[880]: time=2026-09-21T09:12:45.514Z level=INFO msg="Uploading 1 narinfos"4844lb # [ 52.581539] niks3-server[1003]: 2026/09/21 09:12:45 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls4845lb # [ 52.590428] niks3-server[1003]: 2026/09/21 09:12:45 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4846lb # [ 52.595796] niks3-server[1003]: 2026/09/21 09:12:45 INFO Completed upload id=144847worker2 # [ 52.605961] nix-grpc-daemon[880]: time=2026-09-21T09:12:45.532Z level=INFO msg="Upload complete. (286ms)"4848lb # [ 52.598890] niks3-server[1003]: 2026/09/21 09:12:45 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo4849worker2 # [ 52.614346] nix-grpc-daemon[786]: ts=2026-09-21T09:12:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=7 outputs=14850worker1 # [ 52.599858] nix-grpc-daemon[783]: ts=2026-09-21T09:12:45Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335884851worker2 # [ 52.627829] nix-daemon[869]: reaped child process 1146, status = succeeded4852worker1 # [ 52.615315] nix-grpc-daemon[783]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...4853worker1 # [ 52.631131] nix-grpc-daemon[783]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...4854worker1 # [ 52.662578] nix-grpc-daemon[783]: ts=2026-09-21T09:12:45Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:335884855worker1 # [ 52.667994] nix-grpc-daemon[783]: ts=2026-09-21T09:12:45Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:33588 duration_s=0 paths=1 nar_bytes_out=1764856worker1 # [ 52.946157] nix-grpc-daemon[783]: ts=2026-09-21T09:12:46Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:335884857worker2 # [ 52.968497] nix-grpc-daemon[786]: ts=2026-09-21T09:12:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104858worker2 # [ 52.977784] nix-grpc-daemon[880]: time=2026-09-21T09:12:45.904Z level=INFO msg="All 2 paths already cached"4859worker2 # [ 52.992654] nix-grpc-daemon[786]: ts=2026-09-21T09:12:45Z level=debug event=want client=8 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=04860worker2 # [ 52.997763] nix-grpc-daemon[786]: ts=2026-09-21T09:12:45Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.4:50051 clients=14861worker2 # [ 53.003889] nix-grpc-daemon[786]: ts=2026-09-21T09:12:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104862worker2 # [ 53.015308] nix-grpc-daemon[786]: ts=2026-09-21T09:12:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594104863worker2 # [ 53.027190] nix-grpc-daemon[880]: time=2026-09-21T09:12:45.953Z level=INFO msg="All 2 paths already cached"4864worker2 # [ 53.034758] nix-grpc-daemon[786]: ts=2026-09-21T09:12:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:594104865worker2 # [ 53.045144] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4866worker2 # [ 53.212348] nix-daemon[869]: accepted connection from pid 1172, user nix-grpc-daemon (trusted)4867worker2 # [ 53.224336] nix-daemon[869]: reaped child process 1175, status = succeeded4868worker2 # [ 53.384582] nix-daemon[869]: accepted connection from pid 1177, user nix-grpc-daemon (trusted)4869worker2 # [ 53.396270] nix-daemon[869]: reaped child process 1180, status = succeeded4870lb # [ 53.402181] niks3-server[1003]: 2026/09/21 09:12:45 INFO Received uploads request method=POST path=/api/pending_closures4871worker2 # [ 53.420761] nix-grpc-daemon[880]: time=2026-09-21T09:12:46.347Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4872worker2 # [ 53.422351] nix-grpc-daemon[880]: time=2026-09-21T09:12:46.348Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"4873lb # [ 53.432183] niks3-server[1003]: 2026/09/21 09:12:45 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst4874lb # [ 53.443274] niks3-server[1003]: 2026/09/21 09:12:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4875lb # [ 53.449996] niks3-server[1003]: 2026/09/21 09:12:46 INFO Signed narinfos id=15 count=14876worker2 # [ 53.463200] nix-grpc-daemon[880]: time=2026-09-21T09:12:46.389Z level=INFO msg="Uploading 1 narinfos"4877lb # [ 53.455141] niks3-server[1003]: 2026/09/21 09:12:46 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls4878lb # [ 53.465515] niks3-server[1003]: 2026/09/21 09:12:46 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4879worker2 # [ 53.480563] nix-grpc-daemon[880]: time=2026-09-21T09:12:46.407Z level=INFO msg="Upload complete. (424ms)"4880lb # [ 53.470544] niks3-server[1003]: 2026/09/21 09:12:46 INFO Completed upload id=154881worker2 # [ 53.482770] nix-grpc-daemon[786]: ts=2026-09-21T09:12:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=0 paths=1 nar_bytes_in=7524882lb # [ 53.475578] niks3-server[1003]: 2026/09/21 09:12:46 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo4883worker2 # [ 53.488776] nix-grpc-daemon[786]: ts=2026-09-21T09:12:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594104884worker2 # [ 53.500887] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4885worker2 # [ 53.502597] nix-daemon[869]: reaped child process 1170, status = succeeded4886worker2 # [ 53.507746] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4887worker2 # [ 54.534698] nix-daemon[869]: reaped child process 1184, status = succeeded4888worker2 # [ 54.687516] nix-daemon[869]: accepted connection from pid 1199, user nix-grpc-daemon (trusted)4889worker2 # [ 54.699931] nix-daemon[869]: reaped child process 1202, status = succeeded4890lb # [ 54.706921] niks3-server[1003]: 2026/09/21 09:12:47 INFO Received uploads request method=POST path=/api/pending_closures4891worker2 # [ 54.725416] nix-grpc-daemon[880]: time=2026-09-21T09:12:47.651Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4892worker2 # [ 54.726896] nix-grpc-daemon[880]: time=2026-09-21T09:12:47.653Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"4893lb # [ 54.741899] niks3-server[1003]: 2026/09/21 09:12:47 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst4894lb # [ 54.757833] niks3-server[1003]: 2026/09/21 09:12:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4895lb # [ 54.763441] niks3-server[1003]: 2026/09/21 09:12:47 INFO Signed narinfos id=16 count=14896worker2 # [ 54.776615] nix-grpc-daemon[880]: time=2026-09-21T09:12:47.703Z level=INFO msg="Uploading 1 narinfos"4897lb # [ 54.771316] niks3-server[1003]: 2026/09/21 09:12:47 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv4898lb # [ 54.781979] niks3-server[1003]: 2026/09/21 09:12:47 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4899lb # [ 54.789279] niks3-server[1003]: 2026/09/21 09:12:47 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo4900lb # [ 54.792959] niks3-server[1003]: 2026/09/21 09:12:47 INFO Completed upload id=164901worker2 # [ 54.802675] nix-grpc-daemon[880]: time=2026-09-21T09:12:47.729Z level=INFO msg="Upload complete. (287ms)"4902lb # [ 54.795272] niks3-server[1003]: 2026/09/21 09:12:47 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls4903worker2 # [ 54.810889] nix-grpc-daemon[786]: ts=2026-09-21T09:12:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=8 outputs=14904worker1 # [ 54.796864] nix-grpc-daemon[783]: ts=2026-09-21T09:12:47Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335884905worker2 # [ 54.825099] nix-daemon[869]: reaped child process 1182, status = succeeded4906worker1 # [ 54.807074] nix-grpc-daemon[783]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...4907worker2 # [ 54.850174] nix-grpc-daemon[786]: ts=2026-09-21T09:12:47Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:594104908worker2 # [ 54.855393] nix-grpc-daemon[786]: ts=2026-09-21T09:12:47Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=0 paths=1 nar_bytes_out=2324909client # [ 54.923946] nix-daemon[950]: reaped child process 952, status = succeeded4910client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.49 seconds)4911client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top4912client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.02 seconds)4913client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4914client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.03 seconds)4915client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4916client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.02 seconds)4917(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 6.56 seconds)4918subtest: an input only one worker has still reaches the builder4919client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only4920client # this derivation will be built:4921client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv4922client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...4923client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.33 seconds)4924client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure4925client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.17 seconds)4926worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4927worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.25 seconds)4928worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only4929worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)4930client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&24931client # [ 55.956521] nix-daemon[950]: accepted connection from pid 1032, user root (trusted)4932client # this derivation will be built:4933client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv4934client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...4935worker2 # [ 56.137838] nix-grpc-daemon[786]: ts=2026-09-21T09:12:49Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:594104936worker1 # [ 56.123829] nix-grpc-daemon[783]: ts=2026-09-21T09:12:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335884937worker1 # [ 56.324931] nix-daemon[845]: accepted connection from pid 953, user nix-grpc-daemon (trusted)4938worker1 # [ 56.335747] nix-daemon[845]: reaped child process 956, status = succeeded4939lb # [ 56.362459] niks3-server[1003]: 2026/09/21 09:12:48 INFO Received uploads request method=POST path=/api/pending_closures4940worker1 # [ 56.366232] nix-grpc-daemon[872]: time=2026-09-21T09:12:49.436Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4941worker1 # [ 56.367735] nix-grpc-daemon[872]: time=2026-09-21T09:12:49.436Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"4942lb # [ 56.390579] niks3-server[1003]: 2026/09/21 09:12:48 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4943lb # [ 56.403518] niks3-server[1003]: 2026/09/21 09:12:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4944worker1 # [ 56.402734] nix-grpc-daemon[872]: time=2026-09-21T09:12:49.473Z level=INFO msg="Uploading 1 narinfos"4945lb # [ 56.411133] niks3-server[1003]: 2026/09/21 09:12:48 INFO Signed narinfos id=17 count=14946lb # [ 56.418557] niks3-server[1003]: 2026/09/21 09:12:48 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls4947lb # [ 56.428916] niks3-server[1003]: 2026/09/21 09:12:48 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4948lb # [ 56.437857] niks3-server[1003]: 2026/09/21 09:12:49 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo4949lb # [ 56.440082] niks3-server[1003]: 2026/09/21 09:12:49 INFO Completed upload id=174950worker1 # [ 56.432776] nix-grpc-daemon[872]: time=2026-09-21T09:12:49.503Z level=INFO msg="Upload complete. (304ms)"4951worker2 # [ 56.467074] nix-grpc-daemon[786]: ts=2026-09-21T09:12:49Z level=debug event=want client=9 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=04952worker2 # [ 56.469191] nix-grpc-daemon[786]: ts=2026-09-21T09:12:49Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=14953worker1 # [ 56.493686] nix-grpc-daemon[783]: ts=2026-09-21T09:12:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335884954worker1 # [ 56.507643] nix-grpc-daemon[783]: ts=2026-09-21T09:12:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335884955client # copying 2 paths to 'lb:50051'...4956client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...4957worker1 # [ 56.526634] nix-grpc-daemon[783]: ts=2026-09-21T09:12:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:335884958client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...4959worker1 # [ 56.536439] nix-daemon[845]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4960worker1 # [ 56.738532] nix-daemon[845]: accepted connection from pid 960, user nix-grpc-daemon (trusted)4961worker1 # [ 56.751351] nix-daemon[845]: reaped child process 963, status = succeeded4962worker1 # [ 56.931110] nix-daemon[845]: accepted connection from pid 965, user nix-grpc-daemon (trusted)4963worker1 # [ 56.942983] nix-daemon[845]: reaped child process 968, status = succeeded4964lb # [ 56.968216] niks3-server[1003]: 2026/09/21 09:12:49 INFO Received uploads request method=POST path=/api/pending_closures4965lb # [ 56.978074] niks3-server[1003]: 2026/09/21 09:12:49 INFO Received uploads request method=POST path=/api/pending_closures4966worker1 # [ 56.980518] nix-grpc-daemon[872]: time=2026-09-21T09:12:50.051Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4967worker1 # [ 56.982139] nix-grpc-daemon[872]: time=2026-09-21T09:12:50.052Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"4968worker1 # [ 56.983764] nix-grpc-daemon[872]: time=2026-09-21T09:12:50.052Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"4969lb # [ 57.017359] niks3-server[1003]: 2026/09/21 09:12:49 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst4970lb # [ 57.024673] niks3-server[1003]: 2026/09/21 09:12:49 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst4971lb # [ 57.038124] niks3-server[1003]: 2026/09/21 09:12:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4972lb # [ 57.041058] niks3-server[1003]: 2026/09/21 09:12:49 INFO Signed narinfos id=18 count=14973lb # [ 57.043284] niks3-server[1003]: 2026/09/21 09:12:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4974lb # [ 57.046339] niks3-server[1003]: 2026/09/21 09:12:49 INFO Signed narinfos id=19 count=24975lb # [ 57.048419] niks3-server[1003]: 2026/09/21 09:12:49 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls4976worker1 # [ 57.039870] nix-grpc-daemon[872]: time=2026-09-21T09:12:50.109Z level=INFO msg="Uploading 3 narinfos"4977lb # [ 57.059637] niks3-server[1003]: 2026/09/21 09:12:49 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo4978lb # [ 57.061749] niks3-server[1003]: 2026/09/21 09:12:49 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls4979lb # [ 57.067367] niks3-server[1003]: 2026/09/21 09:12:49 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4980lb # [ 57.070838] niks3-server[1003]: 2026/09/21 09:12:49 INFO Completed upload id=184981lb # [ 57.072461] niks3-server[1003]: 2026/09/21 09:12:49 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo4982lb # [ 57.074139] niks3-server[1003]: 2026/09/21 09:12:49 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4983lb # [ 57.076542] niks3-server[1003]: 2026/09/21 09:12:49 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo4984lb # [ 57.078963] niks3-server[1003]: 2026/09/21 09:12:49 INFO Completed upload id=194985worker1 # [ 57.071675] nix-grpc-daemon[872]: time=2026-09-21T09:12:50.142Z level=INFO msg="Upload complete. (517ms)"4986worker1 # [ 57.076326] nix-grpc-daemon[783]: ts=2026-09-21T09:12:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33588 duration_s=0 paths=2 nar_bytes_in=9764987worker1 # [ 57.083229] nix-grpc-daemon[783]: ts=2026-09-21T09:12:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335884988worker1 # [ 57.092460] nix-daemon[845]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4989worker1 # [ 57.102570] nix-daemon[845]: reaped child process 958, status = succeeded4990client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...4991worker1 # [ 57.111890] nix-daemon[845]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4992client # uses-input-w1onlya> node-a: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv4993worker1 # [ 58.327344] nix-daemon[845]: reaped child process 972, status = succeeded4994lb # [ 58.391967] envoy[921]: conn peer=[2001:db8:1::1]:53836 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3402 tx=1634 ms=17794995lb # [ 58.396100] envoy[921]: conn peer=[2001:db8:1::1]:53848 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=594996lb # [ 58.399619] envoy[921]: conn peer=[2001:db8:1::1]:53862 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=594997lb # [ 58.403007] envoy[921]: conn peer=[2001:db8:1::1]:53850 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3854 tx=1966 ms=17794998lb # [ 58.406557] envoy[921]: conn peer=[2001:db8:1::1]:33534 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4328 tx=2197 ms=19034999lb # [ 58.410648] envoy[921]: conn peer=[2001:db8:1::1]:33544 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=411 tx=410 ms=165000worker1 # [ 58.494085] nix-daemon[845]: accepted connection from pid 987, user nix-grpc-daemon (trusted)5001worker1 # [ 58.504498] nix-daemon[845]: reaped child process 990, status = succeeded5002lb # [ 58.531795] niks3-server[1003]: 2026/09/21 09:12:51 INFO Received uploads request method=POST path=/api/pending_closures5003worker1 # [ 58.532693] nix-grpc-daemon[872]: time=2026-09-21T09:12:51.603Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5004lb # [ 58.565081] niks3-server[1003]: 2026/09/21 09:12:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5005worker1 # [ 58.564378] nix-grpc-daemon[872]: time=2026-09-21T09:12:51.634Z level=INFO msg="Uploading 1 narinfos"5006lb # [ 58.571785] niks3-server[1003]: 2026/09/21 09:12:51 INFO Signed narinfos id=20 count=15007lb # [ 58.576118] niks3-server[1003]: 2026/09/21 09:12:51 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5008lb # [ 58.582335] niks3-server[1003]: 2026/09/21 09:12:51 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5009lb # [ 58.593592] niks3-server[1003]: 2026/09/21 09:12:51 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5010lb # [ 58.599869] niks3-server[1003]: 2026/09/21 09:12:51 INFO Completed upload id=205011worker1 # [ 58.593997] nix-grpc-daemon[872]: time=2026-09-21T09:12:51.663Z level=INFO msg="Upload complete. (271ms)"5012lb # [ 58.603959] niks3-server[1003]: 2026/09/21 09:12:51 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5013worker1 # [ 58.597363] nix-grpc-daemon[783]: ts=2026-09-21T09:12:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33588 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=9 outputs=15014worker1 # [ 58.608211] nix-daemon[845]: reaped child process 970, status = succeeded5015client # copying 1 paths...5016worker1 # [ 58.642624] nix-grpc-daemon[783]: ts=2026-09-21T09:12:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335885017client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5018worker1 # [ 58.665169] nix-grpc-daemon[783]: ts=2026-09-21T09:12:51Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:335885019worker1 # [ 58.669373] nix-grpc-daemon[783]: ts=2026-09-21T09:12:51Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:33588 duration_s=0 paths=1 nar_bytes_out=1205020client # [ 58.787237] nix-daemon[950]: reaped child process 1035, status = succeeded5021client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.03 seconds)5022client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25023client # [ 58.960700] nix-daemon[950]: accepted connection from pid 1054, user root (trusted)5024client # this derivation will be built:5025client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5026client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5027worker2 # [ 59.136913] nix-grpc-daemon[786]: ts=2026-09-21T09:12:52Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:594105028worker1 # [ 59.121725] nix-grpc-daemon[783]: ts=2026-09-21T09:12:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335885029worker1 # [ 59.131624] nix-grpc-daemon[872]: time=2026-09-21T09:12:52.202Z level=INFO msg="All 1 paths already cached"5030worker2 # [ 59.164286] nix-grpc-daemon[786]: ts=2026-09-21T09:12:52Z level=debug event=want client=10 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05031worker2 # [ 59.168573] nix-grpc-daemon[786]: ts=2026-09-21T09:12:52Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=15032worker2 # [ 59.176613] nix-grpc-daemon[786]: ts=2026-09-21T09:12:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594105033worker2 # [ 59.189830] nix-grpc-daemon[786]: ts=2026-09-21T09:12:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594105034client # copying 1 paths to 'lb:50051'...5035client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5036worker2 # [ 59.209281] nix-grpc-daemon[786]: ts=2026-09-21T09:12:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:594105037worker2 # [ 59.218871] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5038worker2 # [ 59.236831] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 90 ms (attempt 1/5)5039worker2 # [ 59.332449] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 58 ms (attempt 2/5)5040worker2 # [ 59.390846] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 167 ms (attempt 3/5)5041worker2 # [ 59.564536] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 48 ms (attempt 4/5)5042worker2 # [ 59.619466] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5043worker2 # [ 59.625688] nix-grpc-daemon[786]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5044worker2 # [ 59.831638] nix-daemon[869]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5045worker2 # [ 59.843649] nix-daemon[869]: reaped child process 1220, status = succeeded5046worker2 # [ 60.001500] nix-daemon[869]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5047worker2 # [ 60.013469] nix-daemon[869]: reaped child process 1225, status = succeeded5048lb # [ 60.019227] niks3-server[1003]: 2026/09/21 09:12:52 INFO Received uploads request method=POST path=/api/pending_closures5049worker2 # [ 60.036455] nix-grpc-daemon[880]: time=2026-09-21T09:12:52.962Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5050worker2 # [ 60.038455] nix-grpc-daemon[880]: time=2026-09-21T09:12:52.964Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5051lb # [ 60.048693] niks3-server[1003]: 2026/09/21 09:12:52 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5052lb # [ 60.052889] niks3-server[1003]: 2026/09/21 09:12:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5053lb # [ 60.057986] niks3-server[1003]: 2026/09/21 09:12:52 INFO Signed narinfos id=21 count=15054worker2 # [ 60.071157] nix-grpc-daemon[880]: time=2026-09-21T09:12:52.997Z level=INFO msg="Uploading 1 narinfos"5055lb # [ 60.069429] niks3-server[1003]: 2026/09/21 09:12:52 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5056lb # [ 60.076462] niks3-server[1003]: 2026/09/21 09:12:52 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5057lb # [ 60.081501] niks3-server[1003]: 2026/09/21 09:12:52 INFO Completed upload id=215058worker2 # [ 60.092072] nix-grpc-daemon[880]: time=2026-09-21T09:12:53.018Z level=INFO msg="Upload complete. (447ms)"5059worker2 # [ 60.095078] nix-grpc-daemon[786]: ts=2026-09-21T09:12:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=0 paths=1 nar_bytes_in=5605060lb # [ 60.086391] niks3-server[1003]: 2026/09/21 09:12:52 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5061worker2 # [ 60.098732] nix-grpc-daemon[786]: ts=2026-09-21T09:12:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:594105062worker2 # [ 60.103056] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5063worker2 # [ 60.109165] nix-daemon[869]: reaped child process 1211, status = succeeded5064worker2 # [ 60.117843] nix-grpc-daemon[786]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5065client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5066worker2 # [ 60.127859] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5067client # uses-input-w1onlyb> worker2: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5068worker2 # [ 60.939875] nix-daemon[869]: reaped child process 1230, status = succeeded5069worker2 # [ 61.079324] nix-daemon[869]: accepted connection from pid 1245, user nix-grpc-daemon (trusted)5070worker2 # [ 61.091173] nix-daemon[869]: reaped child process 1248, status = succeeded5071lb # [ 61.099292] niks3-server[1003]: 2026/09/21 09:12:53 INFO Received uploads request method=POST path=/api/pending_closures5072worker2 # [ 61.116873] nix-grpc-daemon[880]: time=2026-09-21T09:12:54.043Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5073lb # [ 61.129727] niks3-server[1003]: 2026/09/21 09:12:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5074worker2 # [ 61.147424] nix-grpc-daemon[880]: time=2026-09-21T09:12:54.074Z level=INFO msg="Uploading 1 narinfos"5075lb # [ 61.137687] niks3-server[1003]: 2026/09/21 09:12:53 INFO Signed narinfos id=22 count=15076lb # [ 61.143742] niks3-server[1003]: 2026/09/21 09:12:53 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5077lb # [ 61.148263] niks3-server[1003]: 2026/09/21 09:12:53 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5078lb # [ 61.161793] niks3-server[1003]: 2026/09/21 09:12:53 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5079lb # [ 61.166783] niks3-server[1003]: 2026/09/21 09:12:53 INFO Completed upload id=225080worker2 # [ 61.176775] nix-grpc-daemon[880]: time=2026-09-21T09:12:54.103Z level=INFO msg="Upload complete. (243ms)"5081lb # [ 61.169076] niks3-server[1003]: 2026/09/21 09:12:53 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5082worker2 # [ 61.181324] nix-grpc-daemon[786]: ts=2026-09-21T09:12:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59410 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=10 outputs=15083worker2 # [ 61.194327] nix-daemon[869]: reaped child process 1227, status = succeeded5084client # copying 1 paths...5085worker2 # [ 61.228257] nix-grpc-daemon[786]: ts=2026-09-21T09:12:54Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:594105086client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5087worker1 # [ 61.235667] nix-grpc-daemon[783]: ts=2026-09-21T09:12:54Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:335885088worker1 # [ 61.246808] nix-grpc-daemon[783]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5089worker1 # [ 61.260347] nix-grpc-daemon[783]: ts=2026-09-21T09:12:54Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:33588 duration_s=0 paths=1 nar_bytes_out=1205090client # [ 61.385489] nix-daemon[950]: reaped child process 1057, status = succeeded5091client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.60 seconds)5092client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25093client # [ 61.571724] nix-daemon[950]: accepted connection from pid 1076, user root (trusted)5094client # this derivation will be built:5095client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5096worker2 # [ 61.764619] nix-grpc-daemon[786]: ts=2026-09-21T09:12:54Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:594105097client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5098worker2 # [ 61.771649] nix-grpc-daemon[786]: ts=2026-09-21T09:12:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:594105099worker2 # [ 61.776655] nix-grpc-daemon[880]: time=2026-09-21T09:12:54.703Z level=INFO msg="All 1 paths already cached"5100worker2 # [ 61.791226] nix-grpc-daemon[786]: ts=2026-09-21T09:12:54Z level=debug event=want client=11 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05101worker2 # [ 61.796218] nix-grpc-daemon[786]: ts=2026-09-21T09:12:54Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=15102worker1 # [ 61.786980] nix-grpc-daemon[783]: ts=2026-09-21T09:12:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335885103worker1 # [ 61.799568] nix-grpc-daemon[783]: ts=2026-09-21T09:12:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335885104worker1 # [ 61.812822] nix-grpc-daemon[872]: time=2026-09-21T09:12:54.882Z level=INFO msg="All 1 paths already cached"5105client # copying 1 paths to 'lb:50051'...5106client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5107worker1 # [ 61.820687] nix-grpc-daemon[783]: ts=2026-09-21T09:12:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:335885108worker1 # [ 61.831757] nix-daemon[845]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5109worker1 # [ 62.030956] nix-daemon[845]: accepted connection from pid 995, user nix-grpc-daemon (trusted)5110worker1 # [ 62.043735] nix-daemon[845]: reaped child process 998, status = succeeded5111worker1 # [ 62.223262] nix-daemon[845]: accepted connection from pid 1000, user nix-grpc-daemon (trusted)5112worker1 # [ 62.234419] nix-daemon[845]: reaped child process 1003, status = succeeded5113lb # [ 62.261339] niks3-server[1003]: 2026/09/21 09:12:54 INFO Received uploads request method=POST path=/api/pending_closures5114worker1 # [ 62.263463] nix-grpc-daemon[872]: time=2026-09-21T09:12:55.334Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5115worker1 # [ 62.265235] nix-grpc-daemon[872]: time=2026-09-21T09:12:55.335Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5116lb # [ 62.292779] niks3-server[1003]: 2026/09/21 09:12:54 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5117lb # [ 62.302901] niks3-server[1003]: 2026/09/21 09:12:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5118worker1 # [ 62.300386] nix-grpc-daemon[872]: time=2026-09-21T09:12:55.370Z level=INFO msg="Uploading 1 narinfos"5119lb # [ 62.308113] niks3-server[1003]: 2026/09/21 09:12:54 INFO Signed narinfos id=23 count=15120lb # [ 62.313127] niks3-server[1003]: 2026/09/21 09:12:54 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5121lb # [ 62.323823] niks3-server[1003]: 2026/09/21 09:12:54 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5122lb # [ 62.327833] niks3-server[1003]: 2026/09/21 09:12:54 INFO Completed upload id=235123worker1 # [ 62.320299] nix-grpc-daemon[872]: time=2026-09-21T09:12:55.391Z level=INFO msg="Upload complete. (476ms)"5124lb # [ 62.329751] niks3-server[1003]: 2026/09/21 09:12:54 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5125worker1 # [ 62.325221] nix-grpc-daemon[783]: ts=2026-09-21T09:12:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33588 duration_s=0 paths=1 nar_bytes_in=5605126worker1 # [ 62.329591] nix-grpc-daemon[783]: ts=2026-09-21T09:12:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335885127worker1 # [ 62.339610] nix-daemon[845]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5128worker1 # [ 62.342529] nix-daemon[845]: reaped child process 993, status = succeeded5129worker1 # [ 62.347460] nix-daemon[845]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5130client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5131client # uses-input-w1onlyc> node-a: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5132worker1 # [ 63.359519] nix-daemon[845]: reaped child process 1007, status = succeeded5133worker1 # [ 63.507267] nix-daemon[845]: accepted connection from pid 1022, user nix-grpc-daemon (trusted)5134worker1 # [ 63.518654] nix-daemon[845]: reaped child process 1025, status = succeeded5135lb # [ 63.545695] niks3-server[1003]: 2026/09/21 09:12:56 INFO Received uploads request method=POST path=/api/pending_closures5136worker1 # [ 63.547286] nix-grpc-daemon[872]: time=2026-09-21T09:12:56.617Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5137lb # [ 63.574126] niks3-server[1003]: 2026/09/21 09:12:56 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5138lb # [ 63.582203] niks3-server[1003]: 2026/09/21 09:12:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5139lb # [ 63.586269] niks3-server[1003]: 2026/09/21 09:12:56 INFO Signed narinfos id=24 count=15140worker1 # [ 63.582657] nix-grpc-daemon[872]: time=2026-09-21T09:12:56.653Z level=INFO msg="Uploading 1 narinfos"5141lb # [ 63.594400] niks3-server[1003]: 2026/09/21 09:12:56 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5142lb # [ 63.606227] niks3-server[1003]: 2026/09/21 09:12:56 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5143worker1 # [ 63.605295] nix-grpc-daemon[872]: time=2026-09-21T09:12:56.676Z level=INFO msg="Upload complete. (251ms)"5144lb # [ 63.612833] niks3-server[1003]: 2026/09/21 09:12:56 INFO Completed upload id=245145lb # [ 63.615983] niks3-server[1003]: 2026/09/21 09:12:56 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5146worker1 # [ 63.609372] nix-grpc-daemon[783]: ts=2026-09-21T09:12:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33588 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=11 outputs=15147worker1 # [ 63.619876] nix-daemon[845]: reaped child process 1005, status = succeeded5148client # copying 1 paths...5149worker1 # [ 63.655437] nix-grpc-daemon[783]: ts=2026-09-21T09:12:56Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335885150client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5151worker1 # [ 63.673970] nix-grpc-daemon[783]: ts=2026-09-21T09:12:56Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:335885152worker1 # [ 63.677833] nix-grpc-daemon[783]: ts=2026-09-21T09:12:56Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:33588 duration_s=0 paths=1 nar_bytes_out=1205153client # [ 63.799236] nix-daemon[950]: reaped child process 1079, status = succeeded5154client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.41 seconds)5155(finished: subtest: an input only one worker has still reaches the builder, in 8.81 seconds)5156subtest: upload whose reference the worker lacks is completed from the cache5157client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5158client # this derivation will be built:5159client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5160client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5161client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.40 seconds)5162client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5163client # this derivation will be built:5164client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5165client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5166client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.40 seconds)5167client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5168client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.19 seconds)5169worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5170worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.23 seconds)5171worker2 # [ 65.076103] systemd[1]: Stopping Nix worker-protocol over gRPC...5172worker2 # [ 65.880084] nix-grpc-daemon[786]: ts=2026-09-21T09:12:58Z level=info event=signal_exit5173worker2 # [ 66.139953] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=scheduler_yield5174worker1 # [ 66.126359] nix-grpc-daemon[783]: ts=2026-09-21T09:12:59Z level=info event=scheduler_restarting addr=lb:500515175lb # [ 66.135736] niks3-server[1003]: 2026/09/21 09:12:58 INFO lead: released remote=192.168.1.4:546805176worker2 # [ 66.150664] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=scheduler_restarting addr=lb:500515177worker2 # [ 66.157170] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:594065178worker2 # [ 66.160653] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=worker_gone addr=192.168.1.4:500515179worker2 # [ 66.163671] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:594065180worker2 # [ 66.166957] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=worker_gone addr=192.168.1.3:500515181worker1 # [ 66.325984] nix-grpc-daemon[783]: ts=2026-09-21T09:12:59Z level=info event=scheduler_connected addr=lb:500515182worker1 # [ 66.329988] nix-grpc-daemon[783]: ts=2026-09-21T09:12:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5183worker2 # [ 66.351977] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=scheduler_connected addr=lb:500515184worker2 # [ 66.356237] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5185worker1 # [ 66.530776] nix-grpc-daemon[783]: ts=2026-09-21T09:12:59Z level=info event=scheduler_connected addr=lb:500515186worker1 # [ 66.534810] nix-grpc-daemon[783]: ts=2026-09-21T09:12:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5187worker2 # [ 66.557314] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=scheduler_connected addr=lb:500515188worker2 # [ 66.561414] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5189worker1 # [ 66.735468] nix-grpc-daemon[783]: ts=2026-09-21T09:12:59Z level=info event=scheduler_connected addr=lb:500515190worker1 # [ 66.739358] nix-grpc-daemon[783]: ts=2026-09-21T09:12:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5191worker2 # [ 66.762492] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=scheduler_connected addr=lb:500515192worker2 # [ 66.766618] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5193worker1 # [ 66.940060] nix-grpc-daemon[783]: ts=2026-09-21T09:13:00Z level=info event=scheduler_connected addr=lb:500515194worker1 # [ 66.943908] nix-grpc-daemon[783]: ts=2026-09-21T09:13:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5195worker2 # [ 66.969506] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=scheduler_connected addr=lb:500515196worker2 # [ 66.972587] nix-grpc-daemon[786]: ts=2026-09-21T09:12:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5197worker2 # [ 67.145320] nix-grpc-daemon[786]: ts=2026-09-21T09:13:00Z level=info event=scheduler_restarting5198worker1 # [ 67.145705] nix-grpc-daemon[783]: ts=2026-09-21T09:13:00Z level=info event=scheduler_connected addr=lb:500515199worker1 # [ 67.149490] nix-grpc-daemon[783]: ts=2026-09-21T09:13:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5200worker2 # [ 67.182721] nix-daemon[869]: reaped child process 871, status = succeeded5201worker2 # [ 67.189745] nix-grpc-daemon[786]: ts=2026-09-21T09:13:00Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5202worker1 # [ 67.349982] nix-grpc-daemon[783]: ts=2026-09-21T09:13:00Z level=info event=scheduler_connected addr=lb:500515203worker1 # [ 67.351707] nix-grpc-daemon[783]: ts=2026-09-21T09:13:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5204worker1 # [ 67.552290] nix-grpc-daemon[783]: ts=2026-09-21T09:13:00Z level=info event=scheduler_connected addr=lb:500515205worker1 # [ 67.554107] nix-grpc-daemon[783]: ts=2026-09-21T09:13:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5206worker1 # [ 67.754602] nix-grpc-daemon[783]: ts=2026-09-21T09:13:00Z level=info event=scheduler_connected addr=lb:500515207worker1 # [ 67.756312] nix-grpc-daemon[783]: ts=2026-09-21T09:13:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5208worker1 # [ 67.956812] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515209worker1 # [ 67.958559] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5210worker2 # [ 68.016455] nix-grpc-daemon[786]: E0000 00:00:1789981980.942215 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5211worker2 # [ 68.021283] nix-grpc-daemon[786]: E0000 00:00:1789981980.942339 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5212worker1 # [ 68.159598] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515213worker1 # [ 68.161587] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5214worker1 # [ 68.362140] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515215worker1 # [ 68.363857] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5216lb # [ 68.392210] envoy[921]: conn peer=[2001:db8:1::1]:33564 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=535217lb # [ 68.393949] envoy[921]: conn peer=[2001:db8:1::1]:33556 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3842 tx=1763 ms=25985218lb # [ 68.395535] envoy[921]: conn peer=[2001:db8:1::1]:33570 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3715 tx=1806 ms=21935219lb # [ 68.397098] envoy[921]: conn peer=[2001:db8:1::1]:33580 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=329 ms=795220lb # [ 68.398633] envoy[921]: conn peer=[2001:db8:1::1]:58130 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=535221lb # [ 68.400358] envoy[921]: conn peer=[2001:db8:1::1]:33596 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3716 tx=1777 ms=19795222lb # [ 68.402164] envoy[921]: conn peer=[2001:db8:1::4]:48332 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=29967 tx=1996 ms=482165223worker2 # [ 68.482342] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5224worker2 # [ 68.485388] systemd[1]: Stopped Nix worker-protocol over gRPC.5225worker2 # [ 68.490377] systemd[1]: nix-grpc-daemon.service: Consumed 2.575s CPU time over 49.928s wall clock time, 60.5M memory peak, 88.3K incoming IP traffic, 221.4K outgoing IP traffic.5226worker2 # [ 68.500908] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5227worker2 # [ 68.504263] systemd[1]: Closed Nix worker-protocol over gRPC.5228worker2 # [ 68.507139] systemd[1]: nix-grpc-daemon.socket: Consumed 291us CPU time over 58.948s wall clock time, 256K memory peak, 87.1K incoming IP traffic, 68.2K outgoing IP traffic.5229lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5230lb: must succeed: curl -sf localhost:9901/clusters5231worker1 # [ 68.564920] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515232worker1 # [ 68.569092] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5233lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5234lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.08 seconds)5235client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25236worker1 # [ 68.769587] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=info event=scheduler_connected addr=lb:500515237worker1 # [ 68.774058] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5238worker1 # [ 68.791796] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:335885239client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5240worker1 # [ 68.797817] nix-grpc-daemon[783]: ts=2026-09-21T09:13:01Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335885241client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.25 seconds)5242worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5243worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.03 seconds)5244worker2 # [ 68.922782] systemd[1]: Listening on Nix worker-protocol over gRPC.5245worker1 # [ 68.968457] systemd[1]: Stopping Nix worker-protocol over gRPC...5246worker1 # [ 68.975653] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515247worker1 # [ 68.978729] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5248worker1 # [ 69.180554] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515249worker1 # [ 69.184377] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5250lb # [ 69.296934] niks3-server[1003]: 2026/09/21 09:13:01 INFO lead: acquired remote=192.168.1.3:537925251worker1 # [ 69.292981] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=scheduler_take_over5252worker1 # [ 69.385567] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515253worker1 # [ 69.389356] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5254worker2 # [ 69.468944] systemd[1]: Starting scheduler-token.service...5255worker2 # [ 69.563971] systemd[1]: scheduler-token.service: Deactivated successfully.5256worker2 # [ 69.567148] systemd[1]: Finished scheduler-token.service.5257worker2 # [ 69.569249] systemd[1]: scheduler-token.service: Consumed 35ms CPU time over 97ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5258worker2 # [ 69.575294] systemd[1]: Starting Nix worker-protocol over gRPC...5259worker1 # [ 69.590240] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515260worker1 # [ 69.594122] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5261worker1 # [ 69.693254] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=signal_exit5262worker2 # [ 69.730817] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:02Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5263worker2 # [ 69.733060] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:02Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5264worker2 # [ 69.736630] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:02Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15265worker2 # [ 69.743072] nix-grpc-daemon[1274]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5266worker2 # [ 69.744393] nix-grpc-daemon[1274]: E0000 00:00:1789981982.669678 1274 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5267worker2 # [ 69.751076] nix-grpc-daemon[1274]: E0000 00:00:1789981982.674883 1274 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5268worker2 # [ 69.753371] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:02Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5269worker2 # [ 69.754931] systemd[1]: Started Nix worker-protocol over gRPC.5270worker2 # [ 69.765078] nix-grpc-daemon[1274]: E0000 00:00:1789981982.691068 1315 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5271worker2 # [ 69.776364] nix-grpc-daemon[1274]: E0000 00:00:1789981982.703009 1315 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5272worker2 # [ 69.781039] nix-grpc-daemon[1274]: E0000 00:00:1789981982.705204 1315 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5273worker2 # [ 69.790152] nix-grpc-daemon[1274]: E0000 00:00:1789981982.716801 1315 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5274worker2 # [ 69.797232] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515275worker2 # [ 69.799157] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5276worker1 # [ 69.794850] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515277worker1 # [ 69.799123] nix-grpc-daemon[783]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5278worker1 # [ 69.951193] nix-grpc-daemon[783]: ts=2026-09-21T09:13:03Z level=info event=scheduler_yield5279lb # [ 69.960181] niks3-server[1003]: 2026/09/21 09:13:02 INFO lead: released remote=192.168.1.3:537925280worker2 # [ 70.000099] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:02Z level=info event=scheduler_connected addr=lb:500515281worker2 # [ 70.004085] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5282worker1 # [ 69.999837] nix-grpc-daemon[783]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515283worker1 # [ 70.003569] nix-grpc-daemon[783]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5284worker2 # [ 70.205431] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515285worker2 # [ 70.209344] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5286worker1 # [ 70.205766] nix-grpc-daemon[783]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515287worker1 # [ 70.208787] nix-grpc-daemon[783]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5288worker2 # [ 70.410300] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515289worker2 # [ 70.414157] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5290worker1 # [ 70.406360] nix-grpc-daemon[783]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515291worker1 # [ 70.410142] nix-grpc-daemon[783]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5292worker2 # [ 70.615184] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515293worker2 # [ 70.619104] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5294worker1 # [ 70.611103] nix-grpc-daemon[783]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515295worker1 # [ 70.614772] nix-grpc-daemon[783]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5296worker2 # [ 70.821253] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515297worker2 # [ 70.824371] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5298worker1 # [ 70.815562] nix-grpc-daemon[783]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515299worker1 # [ 70.819297] nix-grpc-daemon[783]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5300worker1 # [ 70.951774] nix-grpc-daemon[783]: ts=2026-09-21T09:13:04Z level=info event=scheduler_restarting5301worker1 # [ 70.975639] nix-daemon[845]: reaped child process 847, status = succeeded5302worker2 # [ 71.022311] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:03Z level=info event=scheduler_connected addr=lb:500515303worker2 # [ 71.025982] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5304worker1 # [ 71.031350] nix-grpc-daemon[783]: ts=2026-09-21T09:13:04Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5305worker2 # [ 71.227269] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:04Z level=info event=scheduler_connected addr=lb:500515306worker2 # [ 71.231165] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5307worker2 # [ 71.432322] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:04Z level=info event=scheduler_connected addr=lb:500515308worker2 # [ 71.436107] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5309worker2 # [ 71.637280] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:04Z level=info event=scheduler_connected addr=lb:500515310worker2 # [ 71.641154] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5311worker2 # [ 71.843421] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:04Z level=info event=scheduler_connected addr=lb:500515312worker2 # [ 71.846482] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5313worker1 # [ 71.834253] nix-grpc-daemon[783]: E0000 00:00:1789981984.904091 828 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true5314worker1 # [ 71.839090] nix-grpc-daemon[783]: E0000 00:00:1789981984.904204 828 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5315worker2 # [ 72.044741] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:04Z level=info event=scheduler_connected addr=lb:500515316worker2 # [ 72.048690] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5317worker2 # [ 72.250102] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515318worker2 # [ 72.253853] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5319worker1 # [ 72.415549] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5320worker1 # [ 72.419995] systemd[1]: Stopped Nix worker-protocol over gRPC.5321worker1 # [ 72.423290] systemd[1]: nix-grpc-daemon.service: Consumed 1.442s CPU time over 54.038s wall clock time, 51.6M memory peak, 48.8K incoming IP traffic, 89.6K outgoing IP traffic.5322worker1 # [ 72.434368] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5323worker1 # [ 72.437958] systemd[1]: Closed Nix worker-protocol over gRPC.5324worker2 # [ 72.455106] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515325worker1 # [ 72.440419] systemd[1]: nix-grpc-daemon.socket: Consumed 289us CPU time over 1min 2.024s wall clock time, 256K memory peak, 58.1K incoming IP traffic, 51.7K outgoing IP traffic.5326worker2 # [ 72.458884] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5327lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5328lb: must succeed: curl -sf localhost:9901/clusters5329lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5330lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.05 seconds)5331client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25332worker2 # [ 72.659988] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515333worker2 # [ 72.663871] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5334worker2 # [ 72.743184] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490685335worker2 # [ 72.755391] nix-daemon[869]: accepted connection from pid 1274, user nix-grpc-daemon (trusted)5336worker2 # [ 72.774215] nix-grpc-daemon[1274]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 31 ms (attempt 1/5)5337worker2 # [ 72.810934] nix-grpc-daemon[1274]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 101 ms (attempt 2/5)5338worker2 # [ 72.866354] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515339worker2 # [ 72.869395] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5340worker2 # [ 72.917830] nix-grpc-daemon[1274]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 267 ms (attempt 3/5)5341worker2 # [ 73.067402] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:05Z level=info event=scheduler_connected addr=lb:500515342worker2 # [ 73.071213] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5343worker2 # [ 73.190622] nix-grpc-daemon[1274]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 742 ms (attempt 4/5)5344worker2 # [ 73.272288] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:06Z level=info event=scheduler_connected addr=lb:500515345worker2 # [ 73.276301] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5346worker2 # [ 73.477455] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:06Z level=info event=scheduler_connected addr=lb:500515347worker2 # [ 73.481329] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5348worker2 # [ 73.682424] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:06Z level=info event=scheduler_connected addr=lb:500515349worker2 # [ 73.686282] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5350worker2 # [ 73.887930] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:06Z level=info event=scheduler_connected addr=lb:500515351worker2 # [ 73.891854] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5352worker2 # [ 73.938810] nix-grpc-daemon[1274]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5353client # copying 2 paths...5354client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5355worker2 # [ 73.961433] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490685356client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5357worker2 # [ 73.974989] nix-daemon[869]: accepted connection from pid 1274, user nix-grpc-daemon (trusted)5358worker2 # [ 73.995368] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:06Z level=info event=niks3_push_started pid=13335359worker2 # [ 74.092604] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=scheduler_connected addr=lb:500515360worker2 # [ 74.094557] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5361worker2 # [ 74.187920] nix-grpc-daemon[1333]: time=2026-09-21T09:13:07.113Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5362worker2 # [ 74.295081] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=scheduler_connected addr=lb:500515363worker2 # [ 74.296973] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5364worker2 # [ 74.352039] nix-daemon[869]: accepted connection from pid 1346, user nix-grpc-daemon (trusted)5365worker2 # [ 74.362602] nix-daemon[869]: reaped child process 1349, status = succeeded5366lb # [ 74.369881] niks3-server[1003]: 2026/09/21 09:13:06 INFO Received uploads request method=POST path=/api/pending_closures5367lb # [ 74.378761] niks3-server[1003]: 2026/09/21 09:13:06 INFO Received uploads request method=POST path=/api/pending_closures5368worker2 # [ 74.395686] nix-grpc-daemon[1333]: time=2026-09-21T09:13:07.322Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5369worker2 # [ 74.397232] nix-grpc-daemon[1333]: time=2026-09-21T09:13:07.323Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5370worker2 # [ 74.399176] nix-grpc-daemon[1333]: time=2026-09-21T09:13:07.325Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5371lb # [ 74.425870] niks3-server[1003]: 2026/09/21 09:13:06 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5372lb # [ 74.437208] niks3-server[1003]: 2026/09/21 09:13:07 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5373lb # [ 74.445741] niks3-server[1003]: 2026/09/21 09:13:07 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5374lb # [ 74.449691] niks3-server[1003]: 2026/09/21 09:13:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5375lb # [ 74.452380] niks3-server[1003]: 2026/09/21 09:13:07 INFO Signed narinfos id=25 count=15376lb # [ 74.453923] niks3-server[1003]: 2026/09/21 09:13:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5377lb # [ 74.455838] niks3-server[1003]: 2026/09/21 09:13:07 INFO Signed narinfos id=26 count=25378lb # [ 74.458951] niks3-server[1003]: 2026/09/21 09:13:07 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5379worker2 # [ 74.468242] nix-grpc-daemon[1333]: time=2026-09-21T09:13:07.394Z level=INFO msg="Uploading 3 narinfos"5380lb # [ 74.469627] niks3-server[1003]: 2026/09/21 09:13:07 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5381lb # [ 74.474976] niks3-server[1003]: 2026/09/21 09:13:07 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5382lb # [ 74.478056] niks3-server[1003]: 2026/09/21 09:13:07 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5383lb # [ 74.480824] niks3-server[1003]: 2026/09/21 09:13:07 INFO Completed upload id=255384lb # [ 74.482933] niks3-server[1003]: 2026/09/21 09:13:07 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5385lb # [ 74.485259] niks3-server[1003]: 2026/09/21 09:13:07 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5386lb # [ 74.488848] niks3-server[1003]: 2026/09/21 09:13:07 INFO Completed upload id=265387worker2 # [ 74.498113] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=scheduler_connected addr=lb:500515388worker2 # [ 74.502341] nix-grpc-daemon[1333]: time=2026-09-21T09:13:07.428Z level=INFO msg="Upload complete. (310ms)"5389worker2 # [ 74.507326] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5390worker2 # [ 74.512182] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49068 duration_s=0 paths=2 nar_bytes_in=3125391worker2 # [ 74.522291] nix-daemon[869]: reaped child process 1331, status = succeeded5392client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 2.02 seconds)5393worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5394worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.03 seconds)5395worker1 # [ 74.614737] systemd[1]: Listening on Nix worker-protocol over gRPC.5396worker1 # [ 74.622414] systemd[1]: Starting Nix worker-protocol over gRPC...5397worker2 # [ 74.707207] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=scheduler_connected addr=lb:500515398worker2 # [ 74.711405] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5399lb # [ 74.753721] niks3-server[1003]: 2026/09/21 09:13:07 INFO lead: acquired remote=192.168.1.4:549285400worker2 # [ 74.767746] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=scheduler_take_over5401worker1 # [ 74.793783] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:07Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5402worker1 # [ 74.795884] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:07Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5403worker1 # [ 74.799460] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:07Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15404worker1 # [ 74.805065] nix-grpc-daemon[1048]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5405worker1 # [ 74.806375] nix-grpc-daemon[1048]: E0000 00:00:1789981987.875729 1048 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5406worker1 # [ 74.812380] nix-grpc-daemon[1048]: E0000 00:00:1789981987.880921 1048 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5407worker1 # [ 74.815403] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:07Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5408worker1 # [ 74.818743] systemd[1]: Started Nix worker-protocol over gRPC.5409worker1 # [ 74.825224] nix-grpc-daemon[1048]: E0000 00:00:1789981987.894102 1089 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5410lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5411lb: must succeed: curl -sf localhost:9901/clusters5412worker1 # [ 74.836482] nix-grpc-daemon[1048]: E0000 00:00:1789981987.907228 1089 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5413worker1 # [ 74.841057] nix-grpc-daemon[1048]: E0000 00:00:1789981987.909433 1089 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5414worker1 # [ 74.850929] nix-grpc-daemon[1048]: E0000 00:00:1789981987.921632 1089 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5415worker1 # [ 74.861453] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:07Z level=info event=scheduler_connected addr=lb:500515416worker2 # [ 74.884632] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:477705417worker2 # [ 74.890209] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05418worker2 # [ 74.911860] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=scheduler_connected addr=lb:500515419lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5420worker2 # [ 74.922150] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=oidc_jwks provider=mock keys=15421worker2 # [ 74.926274] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:477705422worker2 # [ 74.930160] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:07Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05423lb: must succeed: curl -sf localhost:9901/clusters5424lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5425lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.15 seconds)5426(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 12.19 seconds)5427subtest: scheduler down: the other node takes the lock and keeps it5428lb: must succeed: curl -sf localhost:9901/clusters5429lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5430lb: must succeed: curl -sf localhost:9901/clusters5431lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5432lb: must succeed: curl -sf localhost:9901/clusters5433lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5434lb: must succeed: curl -sf localhost:9901/clusters5435lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5436worker2 # [ 76.228543] systemd[1]: Stopping Nix worker-protocol over gRPC...5437worker2 # [ 76.752507] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:09Z level=info event=signal_exit5438worker2 # [ 76.969351] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:09Z level=info event=scheduler_yield5439lb # [ 76.963914] niks3-server[1003]: 2026/09/21 09:13:09 INFO lead: released remote=192.168.1.4:549285440worker2 # [ 76.974568] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:09Z level=info event=scheduler_restarting addr=lb:500515441worker2 # [ 76.977580] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:09Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:477705442worker1 # [ 76.961929] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:10Z level=info event=scheduler_restarting addr=lb:500515443worker2 # [ 76.982325] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:09Z level=info event=worker_gone addr=192.168.1.4:500515444worker2 # [ 76.985431] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:09Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:477705445worker2 # [ 76.988821] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:09Z level=info event=worker_gone addr=192.168.1.3:500515446worker2 # [ 77.174537] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515447worker2 # [ 77.178370] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5448worker1 # [ 77.162161] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515449worker1 # [ 77.166286] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5450worker2 # [ 77.379416] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515451worker2 # [ 77.383242] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5452worker1 # [ 77.367199] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515453worker1 # [ 77.371164] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5454worker2 # [ 77.584085] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515455worker2 # [ 77.587944] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5456worker1 # [ 77.572269] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515457worker1 # [ 77.576643] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5458worker2 # [ 77.789363] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515459worker2 # [ 77.792958] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5460worker1 # [ 77.777395] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:10Z level=info event=scheduler_connected addr=lb:500515461worker1 # [ 77.781324] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5462worker2 # [ 77.973760] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:10Z level=info event=scheduler_restarting5463worker1 # [ 77.983146] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:11Z level=info event=scheduler_connected addr=lb:500515464worker2 # [ 78.004774] nix-grpc-daemon[1274]: ts=2026-09-21T09:13:10Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5465worker1 # [ 77.987683] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5466worker2 # [ 78.009348] nix-daemon[869]: reaped child process 1326, status = succeeded5467worker1 # [ 78.188416] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:11Z level=info event=scheduler_connected addr=lb:500515468worker1 # [ 78.192384] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5469lb # [ 78.393698] envoy[921]: conn peer=[2001:db8:1::1]:58144 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=145470lb # [ 78.397705] envoy[921]: conn peer=192.168.1.3:49012 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=10533 tx=2057 ms=512035471worker1 # [ 78.393358] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:11Z level=info event=scheduler_connected addr=lb:500515472worker1 # [ 78.397237] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5473lb # [ 78.402980] envoy[921]: conn peer=[2001:db8:1::1]:58148 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1010 tx=305 ms=17745474lb # [ 78.408873] envoy[921]: conn peer=[2001:db8:1::4]:35744 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=25103 tx=1187 ms=82005475worker1 # [ 78.598205] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:11Z level=info event=scheduler_connected addr=lb:500515476worker1 # [ 78.602195] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5477worker2 # [ 78.781532] nix-grpc-daemon[1274]: E0000 00:00:1789981991.707976 1319 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5478worker2 # [ 78.786814] nix-grpc-daemon[1274]: E0000 00:00:1789981991.713209 1319 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5479worker1 # [ 78.802840] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:11Z level=info event=scheduler_connected addr=lb:500515480worker1 # [ 78.806806] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5481worker1 # [ 79.008584] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:12Z level=info event=scheduler_connected addr=lb:500515482worker1 # [ 79.012334] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5483worker1 # [ 79.213176] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:12Z level=info event=scheduler_connected addr=lb:500515484worker1 # [ 79.217158] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5485worker1 # [ 79.418700] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:12Z level=info event=scheduler_connected addr=lb:500515486worker1 # [ 79.423464] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5487worker1 # [ 79.624090] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:12Z level=info event=scheduler_connected addr=lb:500515488worker1 # [ 79.627992] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5489worker2 # [ 79.783350] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5490worker2 # [ 79.787386] systemd[1]: Stopped Nix worker-protocol over gRPC.5491worker2 # [ 79.791164] systemd[1]: nix-grpc-daemon.service: Consumed 390ms CPU time over 10.210s wall clock time, 32.8M memory peak, 23.5K incoming IP traffic, 51.9K outgoing IP traffic.5492worker2 # [ 79.801985] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5493worker2 # [ 79.805703] systemd[1]: Closed Nix worker-protocol over gRPC.5494worker2 # [ 79.808441] systemd[1]: nix-grpc-daemon.socket: Consumed 272us CPU time over 10.886s wall clock time, 256K memory peak, 20.7K incoming IP traffic, 15.4K outgoing IP traffic.5495worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5496worker1 # [ 79.824489] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:12Z level=info event=scheduler_take_over5497lb # [ 79.832287] niks3-server[1003]: 2026/09/21 09:13:12 INFO lead: acquired remote=192.168.1.3:597225498worker1 # [ 79.828709] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:12Z level=info event=scheduler_connected addr=lb:500515499worker1 # [ 79.832618] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5500worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds)5501worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5502worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5503worker1 # [ 80.034264] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:13Z level=info event=scheduler_connected addr=lb:500515504worker1 # [ 80.038398] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5505worker1 # [ 80.239343] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:13Z level=info event=scheduler_connected addr=lb:500515506worker1 # [ 80.243322] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5507worker1 # [ 80.443948] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:13Z level=info event=scheduler_connected addr=lb:500515508worker1 # [ 80.451839] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:13Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:494845509worker1 # [ 80.456532] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:13Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05510worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5511worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5512client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&25513worker1 # [ 81.224199] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:494885514worker1 # [ 81.235331] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5515client # don't know how to build these paths:5516client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5517worker1 # [ 81.249671] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:494885518worker1 # [ 81.261161] nix-grpc-daemon[1048]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 1/5)5519worker1 # [ 81.319237] nix-grpc-daemon[1048]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 62 ms (attempt 2/5)5520worker1 # [ 81.387067] nix-grpc-daemon[1048]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 19 ms (attempt 3/5)5521worker1 # [ 81.411759] nix-grpc-daemon[1048]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 93 ms (attempt 4/5)5522worker1 # [ 81.511357] nix-grpc-daemon[1048]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5523worker1 # [ 81.552715] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:14Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=05524worker1 # [ 81.557695] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:14Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=15525worker1 # [ 81.598372] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:494885526worker1 # [ 81.611479] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:494885527client # copying 1 paths to 'lb:50051'...5528client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...5529worker1 # [ 81.624240] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:494885530worker1 # [ 81.632822] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5531worker1 # [ 81.655327] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:14Z level=info event=niks3_push_started pid=11215532worker1 # [ 81.865678] nix-grpc-daemon[1121]: time=2026-09-21T09:13:14.936Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""5533worker1 # [ 82.040541] nix-daemon[845]: accepted connection from pid 1134, user nix-grpc-daemon (trusted)5534worker1 # [ 82.050978] nix-daemon[845]: reaped child process 1137, status = succeeded5535worker1 # [ 82.229248] nix-daemon[845]: accepted connection from pid 1139, user nix-grpc-daemon (trusted)5536worker1 # [ 82.241147] nix-daemon[845]: reaped child process 1142, status = succeeded5537lb # [ 82.265935] niks3-server[1003]: 2026/09/21 09:13:14 INFO Received uploads request method=POST path=/api/pending_closures5538worker1 # [ 82.271068] nix-grpc-daemon[1121]: time=2026-09-21T09:13:15.340Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5539worker1 # [ 82.272460] nix-grpc-daemon[1121]: time=2026-09-21T09:13:15.340Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"5540lb # [ 82.313736] niks3-server[1003]: 2026/09/21 09:13:14 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst5541lb # [ 82.322913] niks3-server[1003]: 2026/09/21 09:13:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5542lb # [ 82.331654] niks3-server[1003]: 2026/09/21 09:13:14 INFO Signed narinfos id=27 count=15543worker1 # [ 82.324611] nix-grpc-daemon[1121]: time=2026-09-21T09:13:15.394Z level=INFO msg="Uploading 1 narinfos"5544lb # [ 82.350699] niks3-server[1003]: 2026/09/21 09:13:14 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5545lb # [ 82.356510] niks3-server[1003]: 2026/09/21 09:13:14 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls5546lb # [ 82.365298] niks3-server[1003]: 2026/09/21 09:13:14 INFO Completed upload id=275547worker1 # [ 82.358189] nix-grpc-daemon[1121]: time=2026-09-21T09:13:15.428Z level=INFO msg="Upload complete. (488ms)"5548lb # [ 82.371177] niks3-server[1003]: 2026/09/21 09:13:14 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo5549worker1 # [ 82.365306] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49488 duration_s=0 paths=1 nar_bytes_in=5205550worker1 # [ 82.371811] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:494885551worker1 # [ 82.381877] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5552worker1 # [ 82.391266] nix-daemon[845]: reaped child process 1119, status = succeeded5553worker1 # [ 82.397932] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5554client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...5555client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5556client # farm-a> LOG-farm-a-on-standby5557worker1 # [ 83.622900] nix-daemon[845]: reaped child process 1146, status = succeeded5558worker1 # [ 83.810563] nix-daemon[845]: accepted connection from pid 1161, user nix-grpc-daemon (trusted)5559worker1 # [ 83.822151] nix-daemon[845]: reaped child process 1164, status = succeeded5560lb # [ 83.847594] niks3-server[1003]: 2026/09/21 09:13:16 INFO Received uploads request method=POST path=/api/pending_closures5561worker1 # [ 83.852269] nix-grpc-daemon[1121]: time=2026-09-21T09:13:16.922Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5562worker1 # [ 83.853878] nix-grpc-daemon[1121]: time=2026-09-21T09:13:16.924Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"5563lb # [ 83.893747] niks3-server[1003]: 2026/09/21 09:13:16 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5564lb # [ 83.905364] niks3-server[1003]: 2026/09/21 09:13:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5565lb # [ 83.911146] niks3-server[1003]: 2026/09/21 09:13:16 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5566lb # [ 83.915388] niks3-server[1003]: 2026/09/21 09:13:16 INFO Signed narinfos id=28 count=15567worker1 # [ 83.910665] nix-grpc-daemon[1121]: time=2026-09-21T09:13:16.980Z level=INFO msg="Uploading 1 narinfos"5568lb # [ 83.917655] niks3-server[1003]: 2026/09/21 09:13:16 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls5569lb # [ 83.925057] niks3-server[1003]: 2026/09/21 09:13:16 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5570lb # [ 83.931524] niks3-server[1003]: 2026/09/21 09:13:16 INFO Completed upload id=285571worker1 # [ 83.924224] nix-grpc-daemon[1121]: time=2026-09-21T09:13:16.994Z level=INFO msg="Upload complete. (318ms)"5572lb # [ 83.933544] niks3-server[1003]: 2026/09/21 09:13:16 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo5573worker1 # [ 83.931649] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49488 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=15574worker1 # [ 83.949500] nix-daemon[845]: reaped child process 1144, status = succeeded5575worker1 # [ 83.952498] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:17Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=05576worker1 # [ 83.954487] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:17Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=15577worker1 # [ 83.997989] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:494885578worker1 # [ 84.010650] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:494885579client # copying 1 paths to 'lb:50051'...5580worker1 # [ 84.025326] nix-grpc-daemon[1121]: time=2026-09-21T09:13:17.095Z level=INFO msg="All 1 paths already cached"5581client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...5582worker1 # [ 84.030757] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:494885583worker1 # [ 84.039228] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5584worker1 # [ 84.226101] nix-daemon[845]: accepted connection from pid 1170, user nix-grpc-daemon (trusted)5585worker1 # [ 84.237931] nix-daemon[845]: reaped child process 1173, status = succeeded5586worker1 # [ 84.414503] nix-daemon[845]: accepted connection from pid 1175, user nix-grpc-daemon (trusted)5587worker1 # [ 84.425994] nix-daemon[845]: reaped child process 1178, status = succeeded5588lb # [ 84.452501] niks3-server[1003]: 2026/09/21 09:13:17 INFO Received uploads request method=POST path=/api/pending_closures5589worker1 # [ 84.454321] nix-grpc-daemon[1121]: time=2026-09-21T09:13:17.524Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5590worker1 # [ 84.455758] nix-grpc-daemon[1121]: time=2026-09-21T09:13:17.524Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"5591lb # [ 84.478923] niks3-server[1003]: 2026/09/21 09:13:17 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst5592lb # [ 84.491130] niks3-server[1003]: 2026/09/21 09:13:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5593lb # [ 84.495736] niks3-server[1003]: 2026/09/21 09:13:17 INFO Signed narinfos id=29 count=15594worker1 # [ 84.491544] nix-grpc-daemon[1121]: time=2026-09-21T09:13:17.562Z level=INFO msg="Uploading 1 narinfos"5595lb # [ 84.507101] niks3-server[1003]: 2026/09/21 09:13:17 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls5596lb # [ 84.514880] niks3-server[1003]: 2026/09/21 09:13:17 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5597lb # [ 84.522564] niks3-server[1003]: 2026/09/21 09:13:17 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo5598lb # [ 84.527305] niks3-server[1003]: 2026/09/21 09:13:17 INFO Completed upload id=295599worker1 # [ 84.520071] nix-grpc-daemon[1121]: time=2026-09-21T09:13:17.590Z level=INFO msg="Upload complete. (467ms)"5600worker1 # [ 84.524154] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49488 duration_s=0 paths=1 nar_bytes_in=6405601worker1 # [ 84.529646] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:494885602worker1 # [ 84.539128] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5603worker1 # [ 84.546431] nix-daemon[845]: reaped child process 1168, status = succeeded5604worker1 # [ 84.551470] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5605client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...5606client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5607client # farm-b> LOG-farm-b-on-standby5608worker1 # [ 85.975565] nix-daemon[845]: reaped child process 1182, status = succeeded5609worker1 # [ 86.150802] nix-daemon[845]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5610worker1 # [ 86.161586] nix-daemon[845]: reaped child process 1200, status = succeeded5611lb # [ 86.187670] niks3-server[1003]: 2026/09/21 09:13:18 INFO Received uploads request method=POST path=/api/pending_closures5612worker1 # [ 86.191053] nix-grpc-daemon[1121]: time=2026-09-21T09:13:19.261Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5613worker1 # [ 86.192650] nix-grpc-daemon[1121]: time=2026-09-21T09:13:19.263Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"5614lb # [ 86.232126] niks3-server[1003]: 2026/09/21 09:13:18 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst5615lb # [ 86.235769] niks3-server[1003]: 2026/09/21 09:13:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5616lb # [ 86.242896] niks3-server[1003]: 2026/09/21 09:13:18 INFO Signed narinfos id=30 count=15617worker1 # [ 86.239839] nix-grpc-daemon[1121]: time=2026-09-21T09:13:19.310Z level=INFO msg="Uploading 1 narinfos"5618lb # [ 86.250123] niks3-server[1003]: 2026/09/21 09:13:18 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5619lb # [ 86.253762] niks3-server[1003]: 2026/09/21 09:13:18 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls5620lb # [ 86.262197] niks3-server[1003]: 2026/09/21 09:13:18 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5621lb # [ 86.266082] niks3-server[1003]: 2026/09/21 09:13:18 INFO Completed upload id=305622lb # [ 86.267589] niks3-server[1003]: 2026/09/21 09:13:18 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo5623worker1 # [ 86.259123] nix-grpc-daemon[1121]: time=2026-09-21T09:13:19.329Z level=INFO msg="Upload complete. (297ms)"5624worker1 # [ 86.269475] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49488 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=15625worker1 # [ 86.276233] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:19Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=05626worker1 # [ 86.280219] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:19Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=15627worker1 # [ 86.288241] nix-daemon[845]: reaped child process 1180, status = succeeded5628worker1 # [ 86.319580] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:494885629worker1 # [ 86.332765] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:494885630worker1 # [ 86.345825] nix-grpc-daemon[1121]: time=2026-09-21T09:13:19.416Z level=INFO msg="All 2 paths already cached"5631client # copying 1 paths to 'lb:50051'...5632client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...5633worker1 # [ 86.354847] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:494885634worker1 # [ 86.365263] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5635worker1 # [ 86.551827] nix-daemon[845]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)5636worker1 # [ 86.563469] nix-daemon[845]: reaped child process 1209, status = succeeded5637worker1 # [ 86.736412] nix-daemon[845]: accepted connection from pid 1211, user nix-grpc-daemon (trusted)5638worker1 # [ 86.748829] nix-daemon[845]: reaped child process 1214, status = succeeded5639lb # [ 86.773160] niks3-server[1003]: 2026/09/21 09:13:19 INFO Received uploads request method=POST path=/api/pending_closures5640worker1 # [ 86.777426] nix-grpc-daemon[1121]: time=2026-09-21T09:13:19.848Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5641worker1 # [ 86.778911] nix-grpc-daemon[1121]: time=2026-09-21T09:13:19.849Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"5642lb # [ 86.811512] niks3-server[1003]: 2026/09/21 09:13:19 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst5643lb # [ 86.820692] niks3-server[1003]: 2026/09/21 09:13:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5644worker1 # [ 86.818577] nix-grpc-daemon[1121]: time=2026-09-21T09:13:19.889Z level=INFO msg="Uploading 1 narinfos"5645lb # [ 86.826700] niks3-server[1003]: 2026/09/21 09:13:19 INFO Signed narinfos id=31 count=15646lb # [ 86.834223] niks3-server[1003]: 2026/09/21 09:13:19 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls5647lb # [ 86.843660] niks3-server[1003]: 2026/09/21 09:13:19 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5648lb # [ 86.850133] niks3-server[1003]: 2026/09/21 09:13:19 INFO Completed upload id=315649worker1 # [ 86.844996] nix-grpc-daemon[1121]: time=2026-09-21T09:13:19.915Z level=INFO msg="Upload complete. (468ms)"5650lb # [ 86.852913] niks3-server[1003]: 2026/09/21 09:13:19 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo5651worker1 # [ 86.849258] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49488 duration_s=0 paths=1 nar_bytes_in=7685652worker1 # [ 86.853690] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:494885653worker1 # [ 86.863109] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5654worker1 # [ 86.872265] nix-daemon[845]: reaped child process 1204, status = succeeded5655worker1 # [ 86.875939] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5656client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...5657client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5658client # farm-top> LOG-farm-top-on-standby5659worker1 # [ 88.299760] nix-daemon[845]: reaped child process 1218, status = succeeded5660worker1 # [ 88.472152] nix-daemon[845]: accepted connection from pid 1233, user nix-grpc-daemon (trusted)5661worker1 # [ 88.483936] nix-daemon[845]: reaped child process 1236, status = succeeded5662lb # [ 88.510674] niks3-server[1003]: 2026/09/21 09:13:21 INFO Received uploads request method=POST path=/api/pending_closures5663worker1 # [ 88.512939] nix-grpc-daemon[1121]: time=2026-09-21T09:13:21.583Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5664worker1 # [ 88.514412] nix-grpc-daemon[1121]: time=2026-09-21T09:13:21.585Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"5665lb # [ 88.552363] niks3-server[1003]: 2026/09/21 09:13:21 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5666lb # [ 88.562993] niks3-server[1003]: 2026/09/21 09:13:21 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst5667lb # [ 88.572611] niks3-server[1003]: 2026/09/21 09:13:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5668lb # [ 88.577920] niks3-server[1003]: 2026/09/21 09:13:21 INFO Signed narinfos id=32 count=15669worker1 # [ 88.573389] nix-grpc-daemon[1121]: time=2026-09-21T09:13:21.643Z level=INFO msg="Uploading 1 narinfos"5670lb # [ 88.583491] niks3-server[1003]: 2026/09/21 09:13:21 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls5671lb # [ 88.590756] niks3-server[1003]: 2026/09/21 09:13:21 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5672lb # [ 88.593682] niks3-server[1003]: 2026/09/21 09:13:21 INFO Completed upload id=325673worker1 # [ 88.586432] nix-grpc-daemon[1121]: time=2026-09-21T09:13:21.657Z level=INFO msg="Upload complete. (300ms)"5674lb # [ 88.596265] niks3-server[1003]: 2026/09/21 09:13:21 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo5675worker1 # [ 88.593563] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49488 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=15676worker1 # [ 88.606728] nix-daemon[845]: reaped child process 1216, status = succeeded5677client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 7.66 seconds)5678client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath5679client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.22 seconds)5680worker2 # [ 88.945693] systemd[1]: Listening on Nix worker-protocol over gRPC.5681worker2 # [ 88.952489] systemd[1]: Starting scheduler-token.service...5682worker2 # [ 89.023785] systemd[1]: scheduler-token.service: Deactivated successfully.5683worker2 # [ 89.024957] systemd[1]: Finished scheduler-token.service.5684worker2 # [ 89.026625] systemd[1]: scheduler-token.service: Consumed 19ms CPU time over 72ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5685worker2 # [ 89.032191] systemd[1]: Starting Nix worker-protocol over gRPC...5686worker2 # [ 89.198430] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:22Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5687worker2 # [ 89.200242] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:22Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5688worker2 # [ 89.204223] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:22Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15689worker2 # [ 89.211077] nix-grpc-daemon[1373]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5690worker2 # [ 89.212588] nix-grpc-daemon[1373]: E0000 00:00:1789982002.136902 1373 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5691worker2 # [ 89.217883] nix-grpc-daemon[1373]: E0000 00:00:1789982002.142168 1373 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5692worker2 # [ 89.221592] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:22Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5693worker2 # [ 89.223957] systemd[1]: Started Nix worker-protocol over gRPC.5694lb: must succeed: curl -sf localhost:9901/clusters5695worker2 # [ 89.229744] nix-grpc-daemon[1373]: E0000 00:00:1789982002.156432 1415 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5696worker2 # [ 89.242062] nix-grpc-daemon[1373]: E0000 00:00:1789982002.168672 1415 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5697worker2 # [ 89.246064] nix-grpc-daemon[1373]: E0000 00:00:1789982002.170988 1415 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5698worker2 # [ 89.256377] nix-grpc-daemon[1373]: E0000 00:00:1789982002.183016 1415 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5699worker2 # [ 89.267363] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:22Z level=info event=scheduler_connected addr=lb:500515700worker1 # [ 89.259172] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:22Z level=info event=oidc_jwks provider=mock keys=15701worker1 # [ 89.263707] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:22Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:494845702worker1 # [ 89.267566] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:22Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05703lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5704worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5705worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5706lb: must succeed: curl -sf localhost:9901/clusters5707lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5708lb: must succeed: curl -sf localhost:9901/clusters5709lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5710worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5711worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5712(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 13.54 seconds)5713subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5714client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs5715client # [ 89.594779] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.5716client # Running as unit: rs.service; invocation ID: 05c1d402ca774222b0bb99b273839ad25717client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.07 seconds)5718worker2 # [ 89.813746] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:454845719worker2 # [ 89.824848] nix-daemon[869]: accepted connection from pid 1373, user nix-grpc-daemon (trusted)5720client # [ 89.838859] nix[1180]: don't know how to build these paths:5721client # [ 89.839987] nix[1180]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5722worker1 # [ 89.829243] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:22Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=05723worker1 # [ 89.833207] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:22Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=15724worker1 # [ 89.872170] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:494885725worker1 # [ 89.885574] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:494885726client # [ 89.914894] nix[1180]: copying 1 paths to 'lb:50051'...5727client # [ 89.917918] nix[1180]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...5728worker1 # [ 89.899610] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:494885729worker1 # [ 89.911496] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5730worker1 # [ 90.094338] nix-daemon[845]: accepted connection from pid 1249, user nix-grpc-daemon (trusted)5731worker1 # [ 90.106095] nix-daemon[845]: reaped child process 1252, status = succeeded5732worker1 # [ 90.269055] nix-daemon[845]: accepted connection from pid 1254, user nix-grpc-daemon (trusted)5733worker1 # [ 90.281249] nix-daemon[845]: reaped child process 1257, status = succeeded5734lb # [ 90.304885] niks3-server[1003]: 2026/09/21 09:13:22 INFO Received uploads request method=POST path=/api/pending_closures5735worker1 # [ 90.306194] nix-grpc-daemon[1121]: time=2026-09-21T09:13:23.376Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5736worker1 # [ 90.307679] nix-grpc-daemon[1121]: time=2026-09-21T09:13:23.378Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"5737lb # [ 90.335632] niks3-server[1003]: 2026/09/21 09:13:22 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst5738lb # [ 90.345746] niks3-server[1003]: 2026/09/21 09:13:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5739worker1 # [ 90.343248] nix-grpc-daemon[1121]: time=2026-09-21T09:13:23.413Z level=INFO msg="Uploading 1 narinfos"5740lb # [ 90.351133] niks3-server[1003]: 2026/09/21 09:13:22 INFO Signed narinfos id=33 count=15741lb # [ 90.357100] niks3-server[1003]: 2026/09/21 09:13:22 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls5742lb # [ 90.370588] niks3-server[1003]: 2026/09/21 09:13:22 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5743lb # [ 90.375548] niks3-server[1003]: 2026/09/21 09:13:22 INFO Completed upload id=335744worker1 # [ 90.370537] nix-grpc-daemon[1121]: time=2026-09-21T09:13:23.441Z level=INFO msg="Upload complete. (447ms)"5745lb # [ 90.378361] niks3-server[1003]: 2026/09/21 09:13:22 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo5746worker1 # [ 90.374405] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49488 duration_s=0 paths=1 nar_bytes_in=4165747worker1 # [ 90.379285] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:494885748worker1 # [ 90.389106] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5749worker1 # [ 90.396341] nix-daemon[845]: reaped child process 1247, status = succeeded5750worker1 # [ 90.401281] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)5751client # [ 90.420545] nix[1180]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...5752lb: must succeed: curl -sf localhost:9901/clusters5753lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5754lb: must succeed: curl -sf localhost:9901/clusters5755lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5756lb: must succeed: curl -sf localhost:9901/clusters5757lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5758lb: must succeed: curl -sf localhost:9901/clusters5759lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5760worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5761worker2 # [ 92.485949] systemd-journald[286]: Received client request to rotate journal, rotating.5762worker2 # [ 92.505336] systemd-journald[286]: Vacuuming done, freed 0B of archived journals from /var/log/journal/55085b0d8a3d46019eeecd54b6676f21.5763worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.07 seconds)5764worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5765worker1 # [ 92.408684] systemd[1]: Stopping Nix worker-protocol over gRPC...5766worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)5767client: must succeed: sleep 5.05768worker1 # [ 92.817555] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:25Z level=info event=draining5769client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5770worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5771worker1 # [ 97.836258] nix-daemon[845]: reaped child process 1261, status = succeeded5772worker1 # [ 97.990683] nix-daemon[845]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)5773worker1 # [ 98.002717] nix-daemon[845]: reaped child process 1304, status = succeeded5774lb # [ 98.027231] niks3-server[1003]: 2026/09/21 09:13:30 INFO Received uploads request method=POST path=/api/pending_closures5775worker1 # [ 98.029480] nix-grpc-daemon[1121]: time=2026-09-21T09:13:31.100Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5776worker1 # [ 98.031477] nix-grpc-daemon[1121]: time=2026-09-21T09:13:31.101Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"5777lb # [ 98.072394] niks3-server[1003]: 2026/09/21 09:13:30 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5778lb # [ 98.080261] niks3-server[1003]: 2026/09/21 09:13:30 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5779lb # [ 98.088593] niks3-server[1003]: 2026/09/21 09:13:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5780lb # [ 98.092752] niks3-server[1003]: 2026/09/21 09:13:30 INFO Signed narinfos id=34 count=15781worker1 # [ 98.085437] nix-grpc-daemon[1121]: time=2026-09-21T09:13:31.155Z level=INFO msg="Uploading 1 narinfos"5782lb # [ 98.097752] niks3-server[1003]: 2026/09/21 09:13:30 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls5783lb # [ 98.106096] niks3-server[1003]: 2026/09/21 09:13:30 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5784lb # [ 98.109708] niks3-server[1003]: 2026/09/21 09:13:30 INFO Completed upload id=345785lb # [ 98.112181] niks3-server[1003]: 2026/09/21 09:13:30 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo5786worker1 # [ 98.103697] nix-grpc-daemon[1121]: time=2026-09-21T09:13:31.174Z level=INFO msg="Upload complete. (287ms)"5787worker1 # [ 98.112398] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49488 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=15788worker1 # [ 98.124392] nix-daemon[845]: reaped child process 1259, status = succeeded5789client # [ 98.168179] systemd[1]: rs.service: Deactivated successfully.5790client # [ 98.171739] systemd[1]: rs.service: Consumed 127ms CPU time over 8.576s wall clock time, 41.4M memory peak, 7K incoming IP traffic, 8.2K outgoing IP traffic.5791lb # [ 98.394605] envoy[921]: conn peer=[2001:db8:1::1]:48218 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9624 tx=3890 ms=73905792lb # [ 98.399271] envoy[921]: conn peer=[2001:db8:1::1]:33236 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2934 tx=1325 ms=83275793worker1 # [ 98.820473] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:31Z level=info event=signal_exit5794worker1 # [ 99.050207] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=scheduler_yield5795lb # [ 99.062133] niks3-server[1003]: 2026/09/21 09:13:31 INFO lead: released remote=192.168.1.3:597225796worker2 # [ 99.071608] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:31Z level=info event=scheduler_restarting addr=lb:500515797worker1 # [ 99.055910] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=scheduler_restarting addr=lb:500515798worker1 # [ 99.059634] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:494845799worker1 # [ 99.064147] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=worker_gone addr=192.168.1.4:500515800worker1 # [ 99.067271] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:494845801worker1 # [ 99.070577] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=worker_gone addr=192.168.1.3:500515802lb # [ 99.217163] niks3-server[1003]: 2026/09/21 09:13:31 INFO lead: acquired remote=192.168.1.4:410145803worker2 # [ 99.230964] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:32Z level=info event=scheduler_take_over5804worker2 # [ 99.271728] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:32Z level=info event=scheduler_connected addr=lb:500515805worker1 # [ 99.259163] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=scheduler_connected addr=lb:500515806worker2 # [ 99.276371] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:32Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5807worker1 # [ 99.263325] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5808worker2 # [ 99.477624] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:32Z level=info event=scheduler_connected addr=lb:500515809worker1 # [ 99.463703] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=scheduler_connected addr=lb:500515810worker2 # [ 99.482152] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5811worker1 # [ 99.467561] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5812worker2 # [ 99.683310] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:32Z level=info event=scheduler_connected addr=lb:500515813worker1 # [ 99.668496] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=scheduler_connected addr=lb:500515814worker2 # [ 99.687902] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5815worker1 # [ 99.672552] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5816worker2 # [ 99.889197] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:32Z level=info event=scheduler_connected addr=lb:500515817worker1 # [ 99.873391] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=scheduler_connected addr=lb:500515818worker2 # [ 99.893892] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5819worker1 # [ 99.877370] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5820worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.18 seconds)5821client: waiting for success: ! systemctl is-active rs5822client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)5823client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5824client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.04 seconds)5825client: must fail: journalctl -u rs | grep -q 'reconnecting'5826client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.02 seconds)5827lb: must succeed: curl -sf localhost:9901/clusters5828worker1 # [ 100.055171] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:33Z level=info event=scheduler_restarting5829lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5830worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5831worker2 # [ 100.096290] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=scheduler_connected addr=lb:500515832worker2 # [ 100.100815] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5833worker1 # [ 100.088139] nix-daemon[845]: reaped child process 1112, status = succeeded5834worker1 # [ 100.090493] nix-grpc-daemon[1048]: ts=2026-09-21T09:13:33Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5835worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5836worker2 # [ 100.296870] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=scheduler_connected addr=lb:500515837worker2 # [ 100.300788] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5838worker1 # [ 100.458121] nix-grpc-daemon[1048]: E0000 00:00:1789982013.528082 1093 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5839worker1 # [ 100.463088] nix-grpc-daemon[1048]: E0000 00:00:1789982013.533476 1093 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5840worker2 # [ 100.501763] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=scheduler_connected addr=lb:500515841worker2 # [ 100.505575] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5842worker2 # [ 100.706690] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=scheduler_connected addr=lb:500515843worker2 # [ 100.710718] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5844worker2 # [ 100.911766] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=scheduler_connected addr=lb:500515845worker2 # [ 100.924160] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=oidc_jwks provider=mock keys=15846worker2 # [ 100.927937] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:376285847worker2 # [ 100.931882] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:33Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05848lb: must succeed: curl -sf localhost:9901/clusters5849lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5850worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5851worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5852worker1 # [ 101.398794] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5853worker1 # [ 101.402529] systemd[1]: Stopped Nix worker-protocol over gRPC.5854worker1 # [ 101.404496] systemd[1]: nix-grpc-daemon.service: Consumed 1.343s CPU time over 26.775s wall clock time, 35.9M memory peak, 48.8K incoming IP traffic, 96.4K outgoing IP traffic.5855worker1 # [ 101.419390] systemd[1]: Starting Nix worker-protocol over gRPC...5856worker1 # [ 101.580350] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:34Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5857worker1 # [ 101.582853] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:34Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5858worker1 # [ 101.586572] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:34Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15859worker1 # [ 101.594065] nix-grpc-daemon[1311]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5860worker1 # [ 101.595372] nix-grpc-daemon[1311]: E0000 00:00:1789982014.663791 1311 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5861worker1 # [ 101.599933] nix-grpc-daemon[1311]: E0000 00:00:1789982014.668920 1311 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5862worker1 # [ 101.603114] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:34Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5863worker1 # [ 101.604693] systemd[1]: Started Nix worker-protocol over gRPC.5864worker1 # [ 101.612062] nix-grpc-daemon[1311]: E0000 00:00:1789982014.681394 1352 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5865worker1 # [ 101.622457] nix-grpc-daemon[1311]: E0000 00:00:1789982014.693151 1352 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5866worker1 # [ 101.627055] nix-grpc-daemon[1311]: E0000 00:00:1789982014.695465 1352 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5867worker1 # [ 101.636630] nix-grpc-daemon[1311]: E0000 00:00:1789982014.707337 1352 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5868worker1 # [ 101.644051] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:34Z level=info event=scheduler_connected addr=lb:500515869worker2 # [ 101.663547] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:34Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:376285870worker2 # [ 101.667831] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:34Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05871lb: must succeed: curl -sf localhost:9901/clusters5872lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5873worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5874worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5875(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.89 seconds)5876subtest: scheduler restart mid-build: build finishes, no double build5877client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid5878client # [ 102.486722] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.5879client # Running as unit: mid.service; invocation ID: 4f2a164121ca4df881f23262501830ae5880client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)5881worker2 # [ 102.700669] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:454845882client # [ 102.708328] nix[1216]: don't know how to build these paths:5883client # [ 102.709177] nix[1216]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv5884worker2 # [ 102.716176] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:35Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=05885worker2 # [ 102.721430] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:35Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=15886worker2 # [ 102.759557] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454845887worker2 # [ 102.771452] nix-grpc-daemon[1373]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 13 ms (attempt 1/5)5888worker2 # [ 102.790172] nix-grpc-daemon[1373]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 154 ms (attempt 2/5)5889worker2 # [ 102.950184] nix-grpc-daemon[1373]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 217 ms (attempt 3/5)5890worker2 # [ 103.173424] nix-grpc-daemon[1373]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 43 ms (attempt 4/5)5891worker2 # [ 103.223329] nix-grpc-daemon[1373]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5892worker2 # [ 103.238482] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454845893client # [ 103.248901] nix[1216]: copying 1 paths to 'lb:50051'...5894client # [ 103.253176] nix[1216]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...5895worker2 # [ 103.250931] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:454845896worker2 # [ 103.259860] nix-daemon[869]: accepted connection from pid 1373, user nix-grpc-daemon (trusted)5897worker2 # [ 103.282117] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:36Z level=info event=niks3_push_started pid=14975898worker2 # [ 103.513368] nix-grpc-daemon[1497]: time=2026-09-21T09:13:36.439Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5899worker2 # [ 103.710411] nix-daemon[869]: accepted connection from pid 1510, user nix-grpc-daemon (trusted)5900worker2 # [ 103.723957] nix-daemon[869]: reaped child process 1515, status = succeeded5901worker2 # [ 103.923444] nix-daemon[869]: accepted connection from pid 1519, user nix-grpc-daemon (trusted)5902worker2 # [ 103.934956] nix-daemon[869]: reaped child process 1522, status = succeeded5903lb # [ 103.947170] niks3-server[1003]: 2026/09/21 09:13:36 INFO Received uploads request method=POST path=/api/pending_closures5904worker2 # [ 103.971780] nix-grpc-daemon[1497]: time=2026-09-21T09:13:36.898Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5905worker2 # [ 103.973318] nix-grpc-daemon[1497]: time=2026-09-21T09:13:36.900Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"5906lb # [ 103.986126] niks3-server[1003]: 2026/09/21 09:13:36 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst5907lb # [ 103.995761] niks3-server[1003]: 2026/09/21 09:13:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5908lb # [ 104.001468] niks3-server[1003]: 2026/09/21 09:13:36 INFO Signed narinfos id=35 count=15909worker2 # [ 104.014257] nix-grpc-daemon[1497]: time=2026-09-21T09:13:36.940Z level=INFO msg="Uploading 1 narinfos"5910lb # [ 104.012298] niks3-server[1003]: 2026/09/21 09:13:36 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls5911lb # [ 104.019373] niks3-server[1003]: 2026/09/21 09:13:36 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5912lb # [ 104.023619] niks3-server[1003]: 2026/09/21 09:13:36 INFO Completed upload id=355913lb # [ 104.026057] niks3-server[1003]: 2026/09/21 09:13:36 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo5914worker2 # [ 104.035612] nix-grpc-daemon[1497]: time=2026-09-21T09:13:36.962Z level=INFO msg="Upload complete. (518ms)"5915worker2 # [ 104.039801] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45484 duration_s=0 paths=1 nar_bytes_in=4245916worker2 # [ 104.045264] nix-grpc-daemon[1373]: ts=2026-09-21T09:13:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454845917worker2 # [ 104.055114] nix-daemon[869]: accepted connection from pid 1373, user nix-grpc-daemon (trusted)5918worker2 # [ 104.059135] nix-daemon[869]: reaped child process 1495, status = succeeded5919worker2 # [ 104.062938] nix-daemon[869]: accepted connection from pid 1373, user nix-grpc-daemon (trusted)5920client # [ 104.064739] nix[1216]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...5921lb: must succeed: curl -sf localhost:9901/clusters5922lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5923lb: must succeed: curl -sf localhost:9901/clusters5924lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5925worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5926worker2 # [ 105.203199] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1373 (nix-grpc-daemon) on client request.5927worker2 # [ 105.206355] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5928lb # [ 105.204701] niks3-server[1003]: 2026/09/21 09:13:37 INFO lead: released remote=192.168.1.4:410145929client # [ 105.224671] nix[1216]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5930worker1 # [ 105.204502] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5931worker2 # [ 105.227232] systemd[1]: Starting scheduler-token.service...5932client # [ 105.229408] nix[1216]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5933worker2 # [ 105.241906] nix-daemon[869]: reaped child process 1524, status = succeeded5934worker2 # [ 105.243121] nix-daemon[869]: reaped child process 1432, status = succeeded5935worker2 # [ 105.247609] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5936worker2 # [ 105.249666] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5937worker2 # [ 105.250978] systemd[1]: nix-grpc-daemon.service: Consumed 452ms CPU time over 16.213s wall clock time, 33.5M memory peak, 16.9K incoming IP traffic, 38K outgoing IP traffic.5938worker2 # [ 105.259254] nix-daemon[1526]: unexpected Nix daemon error: error: interrupted by the user5939worker2 # [ 105.263907] nix-daemon[869]: reaped child process 1526, status = failed with exit code 15940worker2 # [ 105.325530] systemd[1]: scheduler-token.service: Deactivated successfully.5941worker2 # [ 105.327159] systemd[1]: Finished scheduler-token.service.5942worker2 # [ 105.328949] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 99ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5943worker2 # [ 105.331286] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.5944worker2 # [ 105.336159] systemd[1]: Starting scheduler-token.service...5945worker2 # [ 105.415655] systemd[1]: scheduler-token.service: Deactivated successfully.5946worker2 # [ 105.416736] systemd[1]: Finished scheduler-token.service.5947worker2 # [ 105.418476] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 79ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5948worker2 # [ 105.423856] systemd[1]: Starting Nix worker-protocol over gRPC...5949worker1 # [ 105.404754] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500515950worker2 # [ 105.619686] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5951worker2 # [ 105.622139] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5952worker2 # [ 105.625802] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15953worker2 # [ 105.634079] nix-grpc-daemon[1564]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5954worker2 # [ 105.635599] nix-grpc-daemon[1564]: E0000 00:00:1789982018.560015 1564 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5955worker2 # [ 105.641767] nix-grpc-daemon[1564]: E0000 00:00:1789982018.565204 1564 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5956worker2 # [ 105.644887] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5957worker2 # [ 105.648358] systemd[1]: Started Nix worker-protocol over gRPC.5958worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.49 seconds)5959client: must succeed: sleep 5.05960worker1 # [ 105.634502] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5961lb # [ 105.644374] niks3-server[1003]: 2026/09/21 09:13:38 INFO lead: acquired remote=192.168.1.4:583345962worker2 # [ 105.655741] nix-grpc-daemon[1564]: E0000 00:00:1789982018.582438 1606 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5963worker2 # [ 105.659220] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=info event=scheduler_take_over5964worker2 # [ 105.668978] nix-grpc-daemon[1564]: E0000 00:00:1789982018.595672 1606 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5965worker2 # [ 105.673060] nix-grpc-daemon[1564]: E0000 00:00:1789982018.597924 1606 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5966worker2 # [ 105.682869] nix-grpc-daemon[1564]: E0000 00:00:1789982018.609515 1606 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5967worker2 # [ 105.691633] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500515968worker2 # [ 105.696675] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=info event=oidc_jwks provider=mock keys=15969worker2 # [ 105.698089] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:584305970worker2 # [ 105.699787] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05971worker2 # [ 105.734045] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=05972worker2 # [ 105.736201] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=15973worker2 # [ 105.785134] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584465974worker2 # [ 105.793142] nix-daemon[869]: accepted connection from pid 1564, user nix-grpc-daemon (trusted)5975worker2 # [ 105.804608] nix-daemon[869]: accepted connection from pid 1564, user nix-grpc-daemon (trusted)5976worker2 # [ 105.813357] nix-daemon[869]: accepted connection from pid 1564, user nix-grpc-daemon (trusted)5977client # [ 105.814938] nix[1216]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...5978worker2 # [ 105.853853] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:584305979worker1 # [ 105.835677] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500515980worker2 # [ 105.855759] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:38Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05981lb # [ 108.394933] envoy[921]: conn peer=[2001:db8:1::3]:55102 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=5568 tx=1321 ms=252215982lb # [ 108.399385] envoy[921]: conn peer=[2001:db8:1::4]:36004 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=8646 tx=646 ms=159595983client: (finished: must succeed: sleep 5.0, in 5.04 seconds)5984client: waiting for success: ! systemctl is-active mid5985worker2 # [ 111.026624] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:43Z level=info event=niks3_push_started pid=16465986worker2 # [ 111.037306] nix-daemon[869]: reaped child process 1622, status = succeeded5987worker2 # [ 111.244695] nix-grpc-daemon[1646]: time=2026-09-21T09:13:44.170Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5988worker2 # [ 111.444377] nix-daemon[869]: accepted connection from pid 1659, user nix-grpc-daemon (trusted)5989worker2 # [ 111.456215] nix-daemon[869]: reaped child process 1662, status = succeeded5990lb # [ 111.468880] niks3-server[1003]: 2026/09/21 09:13:44 INFO Received uploads request method=POST path=/api/pending_closures5991worker2 # [ 111.487499] nix-grpc-daemon[1646]: time=2026-09-21T09:13:44.414Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5992worker2 # [ 111.488979] nix-grpc-daemon[1646]: time=2026-09-21T09:13:44.415Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"5993lb # [ 111.512844] niks3-server[1003]: 2026/09/21 09:13:44 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5994lb # [ 111.520639] niks3-server[1003]: 2026/09/21 09:13:44 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv5995lb # [ 111.528757] niks3-server[1003]: 2026/09/21 09:13:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5996lb # [ 111.539355] niks3-server[1003]: 2026/09/21 09:13:44 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls5997lb # [ 111.546667] niks3-server[1003]: 2026/09/21 09:13:44 INFO Signed narinfos id=36 count=15998worker2 # [ 111.556781] nix-grpc-daemon[1646]: time=2026-09-21T09:13:44.483Z level=INFO msg="Uploading 1 narinfos"5999lb # [ 111.557501] niks3-server[1003]: 2026/09/21 09:13:44 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete6000lb # [ 111.561703] niks3-server[1003]: 2026/09/21 09:13:44 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6001lb # [ 111.567451] niks3-server[1003]: 2026/09/21 09:13:44 INFO Completed upload id=366002worker2 # [ 111.577803] nix-grpc-daemon[1646]: time=2026-09-21T09:13:44.504Z level=INFO msg="Upload complete. (329ms)"6003worker2 # [ 111.586096] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58446 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16004worker2 # [ 111.599285] nix-daemon[869]: reaped child process 1620, status = succeeded6005client # [ 111.630668] systemd[1]: mid.service: Deactivated successfully.6006client # [ 111.634288] systemd[1]: mid.service: Consumed 129ms CPU time over 9.146s wall clock time, 39.4M memory peak, 8K incoming IP traffic, 9.9K outgoing IP traffic.6007client: (finished: waiting for success: ! systemctl is-active mid, in 1.11 seconds)6008client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6009client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.03 seconds)6010(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 9.67 seconds)6011subtest: interrupting the client stops the build on the worker6012client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6013client # [ 112.132447] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6014client # Running as unit: intr.service; invocation ID: 3107365d0f934661b6d9fe65138b920b6015client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.03 seconds)6016worker2 # [ 112.331698] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:584466017client # [ 112.340107] nix[1252]: don't know how to build these paths:6018client # [ 112.341048] nix[1252]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6019worker2 # [ 112.347969] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:45Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06020worker2 # [ 112.352232] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:45Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.3:50051 clients=16021worker1 # [ 112.384682] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806022worker1 # [ 112.395865] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6023worker1 # [ 112.414838] nix-grpc-daemon[1311]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 76 ms (attempt 1/5)6024worker1 # [ 112.496638] nix-grpc-daemon[1311]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 90 ms (attempt 2/5)6025worker1 # [ 112.592546] nix-grpc-daemon[1311]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 268 ms (attempt 3/5)6026worker1 # [ 112.866442] nix-grpc-daemon[1311]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 730 ms (attempt 4/5)6027worker1 # [ 113.603985] nix-grpc-daemon[1311]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6028worker1 # [ 113.621854] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806029client # [ 113.655057] nix[1252]: copying 1 paths to 'lb:50051'...6030worker1 # [ 113.636083] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806031client # [ 113.657647] nix[1252]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6032worker1 # [ 113.649538] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6033worker1 # [ 113.668834] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:46Z level=info event=niks3_push_started pid=14026034worker1 # [ 113.876674] nix-grpc-daemon[1402]: time=2026-09-21T09:13:46.946Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6035worker1 # [ 114.045476] nix-daemon[845]: accepted connection from pid 1415, user nix-grpc-daemon (trusted)6036worker1 # [ 114.055912] nix-daemon[845]: reaped child process 1418, status = succeeded6037worker1 # [ 114.225737] nix-daemon[845]: accepted connection from pid 1420, user nix-grpc-daemon (trusted)6038worker1 # [ 114.235822] nix-daemon[845]: reaped child process 1423, status = succeeded6039lb # [ 114.262407] niks3-server[1003]: 2026/09/21 09:13:46 INFO Received uploads request method=POST path=/api/pending_closures6040worker1 # [ 114.266768] nix-grpc-daemon[1402]: time=2026-09-21T09:13:47.337Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6041worker1 # [ 114.268279] nix-grpc-daemon[1402]: time=2026-09-21T09:13:47.339Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6042lb # [ 114.307142] niks3-server[1003]: 2026/09/21 09:13:46 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6043lb # [ 114.311324] niks3-server[1003]: 2026/09/21 09:13:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6044lb # [ 114.313592] niks3-server[1003]: 2026/09/21 09:13:46 INFO Signed narinfos id=37 count=16045worker1 # [ 114.306216] nix-grpc-daemon[1402]: time=2026-09-21T09:13:47.376Z level=INFO msg="Uploading 1 narinfos"6046lb # [ 114.320179] niks3-server[1003]: 2026/09/21 09:13:46 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6047lb # [ 114.324831] niks3-server[1003]: 2026/09/21 09:13:46 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6048lb # [ 114.328103] niks3-server[1003]: 2026/09/21 09:13:46 INFO Completed upload id=376049worker1 # [ 114.320536] nix-grpc-daemon[1402]: time=2026-09-21T09:13:47.391Z level=INFO msg="Upload complete. (440ms)"6050worker1 # [ 114.322607] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=4246051worker1 # [ 114.325491] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806052worker1 # [ 114.327702] nix-grpc-daemon[1402]: time=2026-09-21T09:13:47.398Z level=WARN msg="Failed to register uploaded object" key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo error="server returned 404: object is not pending upload\n"6053worker1 # [ 114.332970] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6054worker1 # [ 114.337166] nix-daemon[845]: reaped child process 1400, status = succeeded6055worker1 # [ 114.341935] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6056client # [ 114.360962] nix[1252]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6057client: must succeed: systemctl kill -s INT intr6058client # [ 116.839186] systemd[1]: intr.service: Sent signal SIGINT to main process 1252 (nix) on client request.6059worker1 # [ 116.821335] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:49Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6060client # [ 116.844823] nix[1252]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6061client: (finished: must succeed: systemctl kill -s INT intr, in 0.05 seconds)6062client # [ 116.851881] nix[1252]: error: interrupted by the user6063client # [ 116.879450] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6064client # [ 116.881096] systemd[1]: intr.service: Failed with result 'exit-code'.6065client # [ 116.883059] systemd[1]: intr.service: Consumed 114ms CPU time over 4.748s wall clock time, 43.4M memory peak, 6.2K incoming IP traffic, 8K outgoing IP traffic.6066worker1 # [ 116.959396] nix-daemon[845]: reaped child process 1425, status = succeeded6067worker1 # [ 116.963225] nix-daemon[1427]: unexpected Nix daemon error: error: interrupted by the user6068worker1 # [ 116.967362] nix-daemon[845]: reaped child process 1427, status = failed with exit code 16069(finished: subtest: interrupting the client stops the build on the worker, in 6.02 seconds)6070subtest: a deploy mid-build drains: the build finishes, the new generation takes over6071client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6072client # [ 118.174971] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6073client # Running as unit: sw.service; invocation ID: 72a544acfeb04aeab4dffa2bfad22c846074client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.07 seconds)6075client # [ 118.379509] nix[1273]: don't know how to build these paths:6076client # [ 118.380348] nix[1273]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6077worker2 # [ 118.376648] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:584466078worker2 # [ 118.390628] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:51Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06079worker2 # [ 118.395663] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:51Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.4:50051 clients=16080lb # [ 118.395301] envoy[921]: conn peer=[2001:db8:1::1]:52646 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3835 tx=1637 ms=89026081lb # [ 118.399084] envoy[921]: conn peer=[2001:db8:1::1]:57408 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2924 tx=731 ms=45306082worker2 # [ 118.434654] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584466083worker2 # [ 118.446877] nix-grpc-daemon[1564]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 81 ms (attempt 1/5)6084worker2 # [ 118.533792] nix-grpc-daemon[1564]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 3 ms (attempt 2/5)6085worker2 # [ 118.542320] nix-grpc-daemon[1564]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 341 ms (attempt 3/5)6086worker2 # [ 118.889113] nix-grpc-daemon[1564]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 687 ms (attempt 4/5)6087worker2 # [ 119.582844] nix-grpc-daemon[1564]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6088worker2 # [ 119.600962] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584466089client # [ 119.614537] nix[1273]: copying 1 paths to 'lb:50051'...6090client # [ 119.617845] nix[1273]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6091worker2 # [ 119.616323] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:584466092worker2 # [ 119.624908] nix-daemon[869]: accepted connection from pid 1564, user nix-grpc-daemon (trusted)6093worker2 # [ 119.845252] nix-daemon[869]: accepted connection from pid 1709, user nix-grpc-daemon (trusted)6094worker2 # [ 119.857738] nix-daemon[869]: reaped child process 1712, status = succeeded6095worker2 # [ 120.056208] nix-daemon[869]: accepted connection from pid 1714, user nix-grpc-daemon (trusted)6096worker2 # [ 120.067697] nix-daemon[869]: reaped child process 1717, status = succeeded6097lb # [ 120.077778] niks3-server[1003]: 2026/09/21 09:13:52 INFO Received uploads request method=POST path=/api/pending_closures6098worker2 # [ 120.099618] nix-grpc-daemon[1646]: time=2026-09-21T09:13:53.025Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6099worker2 # [ 120.101115] nix-grpc-daemon[1646]: time=2026-09-21T09:13:53.025Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6100lb # [ 120.117269] niks3-server[1003]: 2026/09/21 09:13:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6101lb # [ 120.128169] niks3-server[1003]: 2026/09/21 09:13:52 INFO Signed narinfos id=38 count=16102worker2 # [ 120.141438] nix-grpc-daemon[1646]: time=2026-09-21T09:13:53.067Z level=INFO msg="Uploading 1 narinfos"6103lb # [ 120.131519] niks3-server[1003]: 2026/09/21 09:13:52 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6104lb # [ 120.139341] niks3-server[1003]: 2026/09/21 09:13:52 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6105lb # [ 120.147162] niks3-server[1003]: 2026/09/21 09:13:52 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6106lb # [ 120.152401] niks3-server[1003]: 2026/09/21 09:13:52 INFO Completed upload id=386107lb # [ 120.154230] niks3-server[1003]: 2026/09/21 09:13:52 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6108worker2 # [ 120.163820] nix-grpc-daemon[1646]: time=2026-09-21T09:13:53.090Z level=INFO msg="Upload complete. (524ms)"6109worker2 # [ 120.167786] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58446 duration_s=0 paths=1 nar_bytes_in=4166110worker2 # [ 120.174324] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584466111worker2 # [ 120.182537] nix-daemon[869]: accepted connection from pid 1564, user nix-grpc-daemon (trusted)6112worker2 # [ 120.192125] nix-daemon[869]: reaped child process 1707, status = succeeded6113worker2 # [ 120.195529] nix-daemon[869]: accepted connection from pid 1564, user nix-grpc-daemon (trusted)6114client # [ 120.197228] nix[1273]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6115worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&26116worker2 # Checking switch inhibitors... done6117worker2 # [ 122.821825] nixos[1752]: switching to system configuration /nix/store/v11lvsn6lvm6c4sin29rwv464gq3s2s3-nixos-system-worker2-test6118worker2 # [ 122.824510] systemd[1]: Stopped target Local File Systems.6119worker2 # activating the configuration...6120worker2 # [ 122.827246] systemd[1]: Stopped target Remote File Systems.6121worker2 # setting up /etc...6122worker2 # [ 123.583523] systemd[1]: Reload requested from client PID 1752 ('.switch-to-conf') (unit backdoor.service)...6123worker2 # [ 123.585218] systemd[1]: Reloading...6124worker2 # [ 124.432263] systemd[1]: Reloading finished in 845 ms.6125worker2 # restarting sysinit-reactivation.target6126worker2 # [ 124.490641] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....6127worker2 # [ 124.636395] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..6128worker2 # [ 124.639933] systemd[1]: Reached target Reactivate sysinit units.6129worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service6130worker2 # [ 124.644253] systemd[1]: Reloading D-Bus System Message Bus...6131worker2 # [ 124.645485] dbus-broker-launch[533]: Caught SIGHUP, trigger reload.6132worker2 # [ 124.648714] systemd[1]: Reloading Nix worker-protocol over gRPC...6133worker2 # [ 124.680508] dbus-broker-launch[533]: Looking up NSS user entry for 'systemd-timesync'...6134worker2 # [ 124.683183] dbus-broker-launch[533]: NSS returned no entry for 'systemd-timesync'6135worker2 # [ 124.685143] dbus-broker-launch[533]: Invalid user-name in /nix/store/nl02z4vnl25dc1df60lqifjnw6c89zpg-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"6136worker2 # [ 124.696761] systemd[1]: Reloaded Nix worker-protocol over gRPC.6137worker2 # [ 124.720341] systemd[1]: Reloaded D-Bus System Message Bus.6138worker2 # [ 124.725070] systemd[1]: Reached target Local File Systems.6139worker2 # [ 124.725834] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore6140worker2 # [ 124.732778] systemd[1]: Starting Create SUID/SGID Wrappers...6141worker2 # [ 124.733601] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.6142worker2 # [ 124.734756] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc676143worker2 # [ 124.736720] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore6144worker2 # [ 124.738176] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6145worker2 # [ 124.739352] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6146worker2 # [ 124.745177] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os6147worker2 # [ 124.747134] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki6148worker2 # [ 124.752836] systemd[1]: Reached target Remote File Systems.6149worker2 # [ 125.293678] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.6150worker2 # [ 125.296378] systemd[1]: Finished Create SUID/SGID Wrappers.6151worker2 # [ 125.301108] systemd[1]: Starting linger-users.service...6152worker2 # [ 125.350974] systemd[1]: linger-users.service: Deactivated successfully.6153worker2 # [ 125.354229] systemd[1]: Finished linger-users.service.6154worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service6155worker2 # [ 125.611219] nixos[1752]: finished switching to system configuration /nix/store/v11lvsn6lvm6c4sin29rwv464gq3s2s3-nixos-system-worker2-test6156worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 3.79 seconds)6157worker2: must succeed: pgrep -f 'read -t [9]0 x'6158worker2 # [ 125.646148] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:58Z level=info event=draining6159worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)6160client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26161worker1 # [ 125.899595] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:386806162client # don't know how to build these paths:6163client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6164worker1 # [ 125.908190] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806165worker2 # [ 125.948375] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:58Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06166worker2 # [ 125.950229] nix-grpc-daemon[1564]: ts=2026-09-21T09:13:58Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.3:50051 clients=16167worker1 # [ 125.980964] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806168worker1 # [ 125.993706] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806169client # copying 1 paths to 'lb:50051'...6170worker1 # [ 126.006541] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806171client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6172worker1 # [ 126.032666] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6173worker1 # [ 126.227267] nix-daemon[845]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6174worker1 # [ 126.237384] nix-daemon[845]: reaped child process 1488, status = succeeded6175worker1 # [ 126.403623] nix-daemon[845]: accepted connection from pid 1490, user nix-grpc-daemon (trusted)6176worker1 # [ 126.414856] nix-daemon[845]: reaped child process 1493, status = succeeded6177lb # [ 126.441461] niks3-server[1003]: 2026/09/21 09:13:59 INFO Received uploads request method=POST path=/api/pending_closures6178worker1 # [ 126.442170] nix-grpc-daemon[1402]: time=2026-09-21T09:13:59.512Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6179worker1 # [ 126.444050] nix-grpc-daemon[1402]: time=2026-09-21T09:13:59.514Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6180lb # [ 126.469824] niks3-server[1003]: 2026/09/21 09:13:59 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6181lb # [ 126.482223] niks3-server[1003]: 2026/09/21 09:13:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6182worker1 # [ 126.479403] nix-grpc-daemon[1402]: time=2026-09-21T09:13:59.549Z level=INFO msg="Uploading 1 narinfos"6183lb # [ 126.487236] niks3-server[1003]: 2026/09/21 09:13:59 INFO Signed narinfos id=39 count=16184lb # [ 126.493142] niks3-server[1003]: 2026/09/21 09:13:59 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6185lb # [ 126.501902] niks3-server[1003]: 2026/09/21 09:13:59 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6186lb # [ 126.508512] niks3-server[1003]: 2026/09/21 09:13:59 INFO Completed upload id=396187worker1 # [ 126.501181] nix-grpc-daemon[1402]: time=2026-09-21T09:13:59.571Z level=INFO msg="Upload complete. (456ms)"6188lb # [ 126.513336] niks3-server[1003]: 2026/09/21 09:13:59 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6189worker1 # [ 126.505130] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=5286190worker1 # [ 126.510220] nix-grpc-daemon[1311]: ts=2026-09-21T09:13:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806191worker1 # [ 126.520105] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6192worker1 # [ 126.523815] nix-daemon[845]: reaped child process 1483, status = succeeded6193worker1 # [ 126.528702] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6194client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6195client # farm-a> node-a: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6196worker1 # [ 127.541350] nix-daemon[845]: reaped child process 1497, status = succeeded6197client # farm-a> LOG-farm-a-during-drain6198worker1 # [ 127.722077] nix-daemon[845]: accepted connection from pid 1512, user nix-grpc-daemon (trusted)6199worker1 # [ 127.733604] nix-daemon[845]: reaped child process 1515, status = succeeded6200lb # [ 127.761858] niks3-server[1003]: 2026/09/21 09:14:00 INFO Received uploads request method=POST path=/api/pending_closures6201worker1 # [ 127.764282] nix-grpc-daemon[1402]: time=2026-09-21T09:14:00.833Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6202worker1 # [ 127.765687] nix-grpc-daemon[1402]: time=2026-09-21T09:14:00.833Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6203lb # [ 127.810886] niks3-server[1003]: 2026/09/21 09:14:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6204lb # [ 127.817924] niks3-server[1003]: 2026/09/21 09:14:00 INFO Signed narinfos id=40 count=16205worker1 # [ 127.813684] nix-grpc-daemon[1402]: time=2026-09-21T09:14:00.883Z level=INFO msg="Uploading 1 narinfos"6206lb # [ 127.825313] niks3-server[1003]: 2026/09/21 09:14:00 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6207lb # [ 127.830439] niks3-server[1003]: 2026/09/21 09:14:00 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6208lb # [ 127.838638] niks3-server[1003]: 2026/09/21 09:14:00 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6209lb # [ 127.842409] niks3-server[1003]: 2026/09/21 09:14:00 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6210lb # [ 127.846121] niks3-server[1003]: 2026/09/21 09:14:00 INFO Completed upload id=406211lb # [ 127.847968] niks3-server[1003]: 2026/09/21 09:14:00 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6212worker1 # [ 127.840095] nix-grpc-daemon[1402]: time=2026-09-21T09:14:00.910Z level=INFO msg="Upload complete. (305ms)"6213worker1 # [ 127.846184] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16214worker2 # [ 127.866973] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:00Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06215worker2 # [ 127.870298] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:00Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.3:50051 clients=16216worker1 # [ 127.858484] nix-daemon[845]: reaped child process 1495, status = succeeded6217worker1 # [ 127.893489] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806218worker1 # [ 127.906162] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806219worker1 # [ 127.919682] nix-grpc-daemon[1402]: time=2026-09-21T09:14:00.990Z level=INFO msg="All 1 paths already cached"6220client # copying 1 paths to 'lb:50051'...6221worker1 # [ 127.927881] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806222client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6223worker1 # [ 127.945439] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6224worker1 # [ 128.150067] nix-daemon[845]: accepted connection from pid 1519, user nix-grpc-daemon (trusted)6225worker1 # [ 128.161407] nix-daemon[845]: reaped child process 1522, status = succeeded6226worker1 # [ 128.354906] nix-daemon[845]: accepted connection from pid 1524, user nix-grpc-daemon (trusted)6227worker1 # [ 128.366382] nix-daemon[845]: reaped child process 1527, status = succeeded6228lb # [ 128.393161] niks3-server[1003]: 2026/09/21 09:14:00 INFO Received uploads request method=POST path=/api/pending_closures6229worker1 # [ 128.396720] nix-grpc-daemon[1402]: time=2026-09-21T09:14:01.467Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6230worker1 # [ 128.398210] nix-grpc-daemon[1402]: time=2026-09-21T09:14:01.468Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6231lb # [ 128.424163] niks3-server[1003]: 2026/09/21 09:14:00 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6232lb # [ 128.438555] niks3-server[1003]: 2026/09/21 09:14:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6233lb # [ 128.444994] niks3-server[1003]: 2026/09/21 09:14:01 INFO Signed narinfos id=41 count=16234worker1 # [ 128.440705] nix-grpc-daemon[1402]: time=2026-09-21T09:14:01.511Z level=INFO msg="Uploading 1 narinfos"6235lb # [ 128.448561] niks3-server[1003]: 2026/09/21 09:14:01 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6236lb # [ 128.462249] niks3-server[1003]: 2026/09/21 09:14:01 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6237lb # [ 128.468852] niks3-server[1003]: 2026/09/21 09:14:01 INFO Completed upload id=416238worker1 # [ 128.461328] nix-grpc-daemon[1402]: time=2026-09-21T09:14:01.532Z level=INFO msg="Upload complete. (502ms)"6239lb # [ 128.471807] niks3-server[1003]: 2026/09/21 09:14:01 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6240worker1 # [ 128.465062] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=6406241worker1 # [ 128.470471] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806242worker1 # [ 128.480104] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6243worker1 # [ 128.486747] nix-daemon[845]: reaped child process 1517, status = succeeded6244worker1 # [ 128.492055] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6245client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6246client # farm-b> node-a: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6247client # farm-b> LOG-farm-b-during-drain6248worker1 # [ 129.316811] nix-daemon[845]: reaped child process 1531, status = succeeded6249worker1 # [ 129.502521] nix-daemon[845]: accepted connection from pid 1546, user nix-grpc-daemon (trusted)6250worker1 # [ 129.513946] nix-daemon[845]: reaped child process 1549, status = succeeded6251lb # [ 129.541905] niks3-server[1003]: 2026/09/21 09:14:02 INFO Received uploads request method=POST path=/api/pending_closures6252worker1 # [ 129.539640] nix-grpc-daemon[1402]: time=2026-09-21T09:14:02.610Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6253worker1 # [ 129.541265] nix-grpc-daemon[1402]: time=2026-09-21T09:14:02.611Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6254lb # [ 129.559672] niks3-server[1003]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6255lb # [ 129.566334] niks3-server[1003]: 2026/09/21 09:14:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6256lb # [ 129.568579] niks3-server[1003]: 2026/09/21 09:14:02 INFO Signed narinfos id=42 count=16257worker1 # [ 129.561390] nix-grpc-daemon[1402]: time=2026-09-21T09:14:02.631Z level=INFO msg="Uploading 1 narinfos"6258lb # [ 129.572757] niks3-server[1003]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6259lb # [ 129.575529] niks3-server[1003]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6260lb # [ 129.579735] niks3-server[1003]: 2026/09/21 09:14:02 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6261lb # [ 129.583964] niks3-server[1003]: 2026/09/21 09:14:02 INFO Completed upload id=426262worker1 # [ 129.576567] nix-grpc-daemon[1402]: time=2026-09-21T09:14:02.647Z level=INFO msg="Upload complete. (277ms)"6263lb # [ 129.585913] niks3-server[1003]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6264worker1 # [ 129.579256] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16265worker1 # [ 129.585202] nix-daemon[845]: reaped child process 1529, status = succeeded6266worker2 # [ 129.604666] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:02Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06267worker2 # [ 129.606504] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:02Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.3:50051 clients=16268worker1 # [ 129.630690] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806269worker1 # [ 129.641566] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806270worker1 # [ 129.652366] nix-grpc-daemon[1402]: time=2026-09-21T09:14:02.723Z level=INFO msg="All 2 paths already cached"6271client # copying 1 paths to 'lb:50051'...6272client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6273worker1 # [ 129.660614] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806274worker1 # [ 129.668704] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6275worker1 # [ 129.864088] nix-daemon[845]: accepted connection from pid 1553, user nix-grpc-daemon (trusted)6276worker1 # [ 129.875578] nix-daemon[845]: reaped child process 1556, status = succeeded6277worker1 # [ 130.067535] nix-daemon[845]: accepted connection from pid 1558, user nix-grpc-daemon (trusted)6278worker1 # [ 130.078792] nix-daemon[845]: reaped child process 1561, status = succeeded6279lb # [ 130.106560] niks3-server[1003]: 2026/09/21 09:14:02 INFO Received uploads request method=POST path=/api/pending_closures6280worker1 # [ 130.108314] nix-grpc-daemon[1402]: time=2026-09-21T09:14:03.178Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6281worker1 # [ 130.109748] nix-grpc-daemon[1402]: time=2026-09-21T09:14:03.178Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6282lb # [ 130.136860] niks3-server[1003]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6283lb # [ 130.145930] niks3-server[1003]: 2026/09/21 09:14:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6284lb # [ 130.153098] niks3-server[1003]: 2026/09/21 09:14:02 INFO Signed narinfos id=43 count=16285worker1 # [ 130.148887] nix-grpc-daemon[1402]: time=2026-09-21T09:14:03.219Z level=INFO msg="Uploading 1 narinfos"6286lb # [ 130.156718] niks3-server[1003]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6287lb # [ 130.170417] niks3-server[1003]: 2026/09/21 09:14:02 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6288worker1 # [ 130.169375] nix-grpc-daemon[1402]: time=2026-09-21T09:14:03.240Z level=INFO msg="Upload complete. (489ms)"6289lb # [ 130.176891] niks3-server[1003]: 2026/09/21 09:14:02 INFO Completed upload id=436290lb # [ 130.180679] niks3-server[1003]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6291worker1 # [ 130.172146] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=7686292worker1 # [ 130.177259] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806293worker1 # [ 130.186106] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6294worker1 # [ 130.194110] nix-daemon[845]: reaped child process 1551, status = succeeded6295worker1 # [ 130.197822] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6296client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6297client # farm-top> node-a: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6298client # farm-top> LOG-farm-top-during-drain6299worker1 # [ 131.010287] nix-daemon[845]: reaped child process 1565, status = succeeded6300worker1 # [ 131.179787] nix-daemon[845]: accepted connection from pid 1580, user nix-grpc-daemon (trusted)6301worker1 # [ 131.191356] nix-daemon[845]: reaped child process 1583, status = succeeded6302lb # [ 131.219516] niks3-server[1003]: 2026/09/21 09:14:03 INFO Received uploads request method=POST path=/api/pending_closures6303worker1 # [ 131.222840] nix-grpc-daemon[1402]: time=2026-09-21T09:14:04.293Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6304worker1 # [ 131.224413] nix-grpc-daemon[1402]: time=2026-09-21T09:14:04.295Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6305lb # [ 131.254484] niks3-server[1003]: 2026/09/21 09:14:03 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6306lb # [ 131.268288] niks3-server[1003]: 2026/09/21 09:14:03 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6307lb # [ 131.277508] niks3-server[1003]: 2026/09/21 09:14:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6308worker1 # [ 131.274940] nix-grpc-daemon[1402]: time=2026-09-21T09:14:04.345Z level=INFO msg="Uploading 1 narinfos"6309lb # [ 131.283307] niks3-server[1003]: 2026/09/21 09:14:03 INFO Signed narinfos id=44 count=16310lb # [ 131.285295] niks3-server[1003]: 2026/09/21 09:14:03 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6311lb # [ 131.290549] niks3-server[1003]: 2026/09/21 09:14:03 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6312lb # [ 131.294856] niks3-server[1003]: 2026/09/21 09:14:03 INFO Completed upload id=446313worker1 # [ 131.287568] nix-grpc-daemon[1402]: time=2026-09-21T09:14:04.358Z level=INFO msg="Upload complete. (281ms)"6314lb # [ 131.297334] niks3-server[1003]: 2026/09/21 09:14:03 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6315worker1 # [ 131.293307] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16316worker1 # [ 131.305471] nix-daemon[845]: reaped child process 1563, status = succeeded6317client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 5.68 seconds)6318client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6319client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.21 seconds)6320worker2: must succeed: pkill -f 'read -t [9]0 x'6321worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds)6322client: waiting for success: ! systemctl is-active sw6323worker2 # [ 131.834926] nix-daemon[869]: reaped child process 1721, status = succeeded6324worker2 # [ 132.017929] nix-daemon[869]: accepted connection from pid 1898, user nix-grpc-daemon (trusted)6325worker2 # [ 132.028290] nix-daemon[869]: reaped child process 1901, status = succeeded6326lb # [ 132.035326] niks3-server[1003]: 2026/09/21 09:14:04 INFO Received uploads request method=POST path=/api/pending_closures6327worker2 # [ 132.055118] nix-grpc-daemon[1646]: time=2026-09-21T09:14:04.981Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6328worker2 # [ 132.057302] nix-grpc-daemon[1646]: time=2026-09-21T09:14:04.983Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6329lb # [ 132.073320] niks3-server[1003]: 2026/09/21 09:14:04 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6330lb # [ 132.081723] niks3-server[1003]: 2026/09/21 09:14:04 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6331lb # [ 132.091565] niks3-server[1003]: 2026/09/21 09:14:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6332worker2 # [ 132.107218] nix-grpc-daemon[1646]: time=2026-09-21T09:14:05.033Z level=INFO msg="Uploading 1 narinfos"6333lb # [ 132.096873] niks3-server[1003]: 2026/09/21 09:14:04 INFO Signed narinfos id=45 count=16334lb # [ 132.101445] niks3-server[1003]: 2026/09/21 09:14:04 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6335lb # [ 132.111482] niks3-server[1003]: 2026/09/21 09:14:04 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6336lb # [ 132.117674] niks3-server[1003]: 2026/09/21 09:14:04 INFO Completed upload id=456337lb # [ 132.119828] niks3-server[1003]: 2026/09/21 09:14:04 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6338worker2 # [ 132.129218] nix-grpc-daemon[1646]: time=2026-09-21T09:14:05.055Z level=INFO msg="Upload complete. (310ms)"6339worker2 # [ 132.139639] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58446 duration_s=11 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16340worker2 # [ 132.151946] nix-daemon[869]: reaped child process 1719, status = succeeded6341client # [ 132.178232] systemd[1]: sw.service: Deactivated successfully.6342client # [ 132.181967] systemd[1]: sw.service: Consumed 128ms CPU time over 14.006s wall clock time, 41.4M memory peak, 7.1K incoming IP traffic, 8.3K outgoing IP traffic.6343worker2 # [ 132.647413] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:05Z level=info event=signal_exit6344client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)6345client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6346client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds)6347worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6348worker2 # [ 132.832815] systemd[1]: Starting scheduler-token.service...6349worker2 # [ 132.878514] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:05Z level=info event=scheduler_yield6350worker1 # [ 132.863893] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:05Z level=info event=scheduler_restarting addr=lb:500516351lb # [ 132.871905] niks3-server[1003]: 2026/09/21 09:14:05 INFO lead: released remote=192.168.1.4:583346352worker2 # [ 132.885513] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:05Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:584306353worker2 # [ 132.887428] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:05Z level=info event=worker_gone addr=192.168.1.3:500516354worker2 # [ 132.889555] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:05Z level=info event=scheduler_restarting addr=lb:500516355worker2 # [ 132.890924] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:05Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:584306356worker2 # [ 132.892306] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:05Z level=info event=worker_gone addr=192.168.1.4:500516357worker2 # [ 132.922096] systemd[1]: scheduler-token.service: Deactivated successfully.6358worker2 # [ 132.923667] systemd[1]: Finished scheduler-token.service.6359worker2 # [ 132.924845] systemd[1]: scheduler-token.service: Consumed 18ms CPU time over 91ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6360worker1 # [ 133.067993] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:06Z level=info event=scheduler_connected addr=lb:500516361worker2 # [ 133.088921] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:06Z level=info event=scheduler_connected addr=lb:500516362worker1 # [ 133.072237] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6363worker2 # [ 133.092822] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6364worker1 # [ 133.273060] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:06Z level=info event=scheduler_connected addr=lb:500516365worker1 # [ 133.277156] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6366worker2 # [ 133.295400] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:06Z level=info event=scheduler_connected addr=lb:500516367worker2 # [ 133.298492] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6368worker1 # [ 133.477731] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:06Z level=info event=scheduler_connected addr=lb:500516369worker2 # [ 133.496466] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:06Z level=info event=scheduler_connected addr=lb:500516370worker1 # [ 133.481739] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6371worker2 # [ 133.500403] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6372worker1 # [ 133.682609] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:06Z level=info event=scheduler_connected addr=lb:500516373worker2 # [ 133.701582] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:06Z level=info event=scheduler_connected addr=lb:500516374worker1 # [ 133.687302] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6375worker2 # [ 133.705567] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6376worker2 # [ 133.879185] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:06Z level=info event=scheduler_restarting6377worker2 # [ 133.903535] nix-daemon[869]: reaped child process 1618, status = succeeded6378worker1 # [ 133.888776] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:06Z level=info event=scheduler_connected addr=lb:500516379worker1 # [ 133.892715] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6380worker2 # [ 133.916620] nix-grpc-daemon[1564]: ts=2026-09-21T09:14:06Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6381worker1 # [ 134.090501] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:07Z level=info event=scheduler_connected addr=lb:500516382worker1 # [ 134.094370] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6383worker1 # [ 134.295155] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:07Z level=info event=scheduler_connected addr=lb:500516384worker1 # [ 134.298998] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6385worker1 # [ 134.499646] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:07Z level=info event=scheduler_connected addr=lb:500516386worker1 # [ 134.503471] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6387worker1 # [ 134.704328] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:07Z level=info event=scheduler_connected addr=lb:500516388worker1 # [ 134.708207] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6389worker2 # [ 134.881339] nix-grpc-daemon[1564]: E0000 00:00:1789982047.807073 1609 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true6390worker2 # [ 134.886745] nix-grpc-daemon[1564]: E0000 00:00:1789982047.813061 1609 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6391worker1 # [ 134.909749] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:07Z level=info event=scheduler_connected addr=lb:500516392worker1 # [ 134.913316] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6393worker1 # [ 135.113984] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:08Z level=info event=scheduler_connected addr=lb:500516394worker1 # [ 135.117876] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6395worker1 # [ 135.318796] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:08Z level=info event=scheduler_connected addr=lb:500516396worker1 # [ 135.322877] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6397worker1 # [ 135.523506] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:08Z level=info event=scheduler_connected addr=lb:500516398worker1 # [ 135.527482] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6399worker2 # [ 135.691818] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6400worker2 # [ 135.695451] systemd[1]: nix-grpc-daemon.service: Consumed 682ms CPU time over 30.444s wall clock time, 33.1M memory peak, 25.6K incoming IP traffic, 36.4K outgoing IP traffic.6401worker2 # [ 135.711266] systemd[1]: Starting Nix worker-protocol over gRPC...6402worker1 # [ 135.728148] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:08Z level=info event=scheduler_connected addr=lb:500516403worker1 # [ 135.732179] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6404worker2 # [ 135.904911] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:08Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6405worker2 # [ 135.907281] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:08Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6406worker2 # [ 135.910783] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:08Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16407worker2 # [ 135.918071] nix-grpc-daemon[1918]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6408worker2 # [ 135.920858] nix-grpc-daemon[1918]: E0000 00:00:1789982048.844435 1918 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6409worker2 # [ 135.926445] nix-grpc-daemon[1918]: E0000 00:00:1789982048.847394 1918 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6410worker2 # [ 135.928972] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:08Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6411worker2 # [ 135.931635] systemd[1]: Started Nix worker-protocol over gRPC.6412worker2 # [ 135.937231] nix-grpc-daemon[1918]: E0000 00:00:1789982048.863922 1960 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6413lb # [ 135.927700] niks3-server[1003]: 2026/09/21 09:14:08 INFO lead: acquired remote=192.168.1.4:344726414worker2 # [ 135.943068] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:08Z level=info event=scheduler_take_over6415worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 3.19 seconds)6416lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6417lb: must succeed: curl -sf localhost:9901/clusters6418worker2 # [ 135.951395] nix-grpc-daemon[1918]: E0000 00:00:1789982048.878089 1960 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6419worker1 # [ 135.933850] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:09Z level=info event=scheduler_connected addr=lb:500516420worker2 # [ 135.955138] nix-grpc-daemon[1918]: E0000 00:00:1789982048.880205 1960 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6421worker1 # [ 135.937655] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6422worker2 # [ 135.965427] nix-grpc-daemon[1918]: E0000 00:00:1789982048.892056 1960 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6423worker2 # [ 135.976760] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:08Z level=info event=scheduler_connected addr=lb:500516424worker2 # [ 135.978262] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6425lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6426worker1 # [ 136.138654] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:09Z level=info event=scheduler_connected addr=lb:500516427worker1 # [ 136.142595] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6428worker2 # [ 136.179800] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=scheduler_connected addr=lb:500516429worker2 # [ 136.183789] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6430worker1 # [ 136.343432] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:09Z level=info event=scheduler_connected addr=lb:500516431worker1 # [ 136.347532] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6432worker2 # [ 136.384876] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=scheduler_connected addr=lb:500516433worker2 # [ 136.388929] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6434worker1 # [ 136.548136] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:09Z level=info event=scheduler_connected addr=lb:500516435worker1 # [ 136.551982] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6436worker2 # [ 136.590137] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=scheduler_connected addr=lb:500516437worker2 # [ 136.593981] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6438worker1 # [ 136.752818] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:09Z level=info event=scheduler_connected addr=lb:500516439worker2 # [ 136.776331] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:343306440worker2 # [ 136.780612] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06441worker2 # [ 136.794717] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=scheduler_connected addr=lb:500516442worker2 # [ 136.804161] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=oidc_jwks provider=mock keys=16443worker2 # [ 136.807773] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:343306444worker2 # [ 136.811621] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:09Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06445lb: must succeed: curl -sf localhost:9901/clusters6446lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6447lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.15 seconds)6448lb: must succeed: curl -sf localhost:9901/clusters6449lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6450worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6451worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6452(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 19.10 seconds)6453subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6454client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16455worker2 # [ 137.454280] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:343346456worker2 # [ 137.459533] nix-daemon[869]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)6457worker1 # [ 137.452301] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806458worker2 # [ 137.514212] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:10Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06459worker2 # [ 137.516326] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:10Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16460worker2 # [ 137.518168] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:10Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06461worker2 # [ 137.519712] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:10Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06462worker1 # [ 137.543681] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806463worker1 # [ 137.556407] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806464worker1 # [ 137.568212] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806465worker1 # [ 137.579090] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6466worker1 # [ 137.800224] nix-daemon[845]: accepted connection from pid 1587, user nix-grpc-daemon (trusted)6467worker1 # [ 137.811661] nix-daemon[845]: reaped child process 1590, status = succeeded6468worker1 # [ 138.014832] nix-daemon[845]: accepted connection from pid 1592, user nix-grpc-daemon (trusted)6469worker1 # [ 138.026122] nix-daemon[845]: reaped child process 1595, status = succeeded6470lb # [ 138.054735] niks3-server[1003]: 2026/09/21 09:14:10 INFO Received uploads request method=POST path=/api/pending_closures6471worker1 # [ 138.056124] nix-grpc-daemon[1402]: time=2026-09-21T09:14:11.126Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6472worker1 # [ 138.057974] nix-grpc-daemon[1402]: time=2026-09-21T09:14:11.128Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6473lb # [ 138.085411] niks3-server[1003]: 2026/09/21 09:14:10 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6474lb # [ 138.097882] niks3-server[1003]: 2026/09/21 09:14:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6475worker1 # [ 138.096597] nix-grpc-daemon[1402]: time=2026-09-21T09:14:11.167Z level=INFO msg="Uploading 1 narinfos"6476lb # [ 138.103821] niks3-server[1003]: 2026/09/21 09:14:10 INFO Signed narinfos id=46 count=16477lb # [ 138.114408] niks3-server[1003]: 2026/09/21 09:14:10 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6478lb # [ 138.122731] niks3-server[1003]: 2026/09/21 09:14:10 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6479worker1 # [ 138.122335] nix-grpc-daemon[1402]: time=2026-09-21T09:14:11.193Z level=INFO msg="Upload complete. (532ms)"6480lb # [ 138.129859] niks3-server[1003]: 2026/09/21 09:14:10 INFO Completed upload id=466481lb # [ 138.133654] niks3-server[1003]: 2026/09/21 09:14:10 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6482worker1 # [ 138.124841] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=5046483worker1 # [ 138.130293] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806484worker1 # [ 138.138753] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6485worker1 # [ 138.149143] nix-daemon[845]: reaped child process 1585, status = succeeded6486worker1 # [ 138.152481] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6487lb # [ 138.399393] envoy[921]: conn peer=[2001:db8:1::1]:47860 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9529 tx=3826 ms=54076488lb # [ 138.403947] envoy[921]: conn peer=[2001:db8:1::1]:54026 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2936 tx=1353 ms=137726489lb # [ 138.408143] envoy[921]: conn peer=[2001:db8:1::4]:38474 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=4876 tx=539 ms=282226490worker1 # [ 138.974962] nix-daemon[845]: reaped child process 1599, status = succeeded6491worker1 # [ 139.171577] nix-daemon[845]: accepted connection from pid 1614, user nix-grpc-daemon (trusted)6492worker1 # [ 139.181740] nix-daemon[845]: reaped child process 1617, status = succeeded6493lb # [ 139.210772] niks3-server[1003]: 2026/09/21 09:14:11 INFO Received uploads request method=POST path=/api/pending_closures6494worker1 # [ 139.211323] nix-grpc-daemon[1402]: time=2026-09-21T09:14:12.281Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6495worker1 # [ 139.213329] nix-grpc-daemon[1402]: time=2026-09-21T09:14:12.283Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6496lb # [ 139.244184] niks3-server[1003]: 2026/09/21 09:14:11 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6497lb # [ 139.257891] niks3-server[1003]: 2026/09/21 09:14:11 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6498lb # [ 139.267813] niks3-server[1003]: 2026/09/21 09:14:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6499lb # [ 139.271624] niks3-server[1003]: 2026/09/21 09:14:11 INFO Signed narinfos id=47 count=16500worker1 # [ 139.267072] nix-grpc-daemon[1402]: time=2026-09-21T09:14:12.337Z level=INFO msg="Uploading 1 narinfos"6501lb # [ 139.277907] niks3-server[1003]: 2026/09/21 09:14:11 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6502lb # [ 139.283213] niks3-server[1003]: 2026/09/21 09:14:11 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6503lb # [ 139.286448] niks3-server[1003]: 2026/09/21 09:14:11 INFO Completed upload id=476504worker1 # [ 139.279141] nix-grpc-daemon[1402]: time=2026-09-21T09:14:12.349Z level=INFO msg="Upload complete. (320ms)"6505lb # [ 139.288391] niks3-server[1003]: 2026/09/21 09:14:11 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6506worker2 # [ 139.302481] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:12Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16507worker1 # [ 139.285792] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=16508worker2 # [ 139.308376] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:12Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06509worker1 # [ 139.292584] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806510worker1 # [ 139.303509] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806511worker1 # [ 139.307858] nix-daemon[845]: reaped child process 1597, status = succeeded6512worker1 # [ 139.312418] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806513worker1 # [ 139.316452] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6514worker1 # [ 139.521330] nix-daemon[845]: accepted connection from pid 1621, user nix-grpc-daemon (trusted)6515worker1 # [ 139.533829] nix-daemon[845]: reaped child process 1624, status = succeeded6516worker1 # [ 139.738056] nix-daemon[845]: accepted connection from pid 1626, user nix-grpc-daemon (trusted)6517worker1 # [ 139.749534] nix-daemon[845]: reaped child process 1629, status = succeeded6518lb # [ 139.776779] niks3-server[1003]: 2026/09/21 09:14:12 INFO Received uploads request method=POST path=/api/pending_closures6519worker1 # [ 139.778062] nix-grpc-daemon[1402]: time=2026-09-21T09:14:12.848Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6520worker1 # [ 139.779421] nix-grpc-daemon[1402]: time=2026-09-21T09:14:12.848Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6521lb # [ 139.803470] niks3-server[1003]: 2026/09/21 09:14:12 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6522lb # [ 139.815993] niks3-server[1003]: 2026/09/21 09:14:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6523lb # [ 139.822881] niks3-server[1003]: 2026/09/21 09:14:12 INFO Signed narinfos id=48 count=16524worker1 # [ 139.818600] nix-grpc-daemon[1402]: time=2026-09-21T09:14:12.889Z level=INFO msg="Uploading 1 narinfos"6525lb # [ 139.826222] niks3-server[1003]: 2026/09/21 09:14:12 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6526lb # [ 139.839599] niks3-server[1003]: 2026/09/21 09:14:12 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6527worker1 # [ 139.838400] nix-grpc-daemon[1402]: time=2026-09-21T09:14:12.909Z level=INFO msg="Upload complete. (516ms)"6528lb # [ 139.845906] niks3-server[1003]: 2026/09/21 09:14:12 INFO Completed upload id=486529worker1 # [ 139.841100] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=5046530lb # [ 139.849642] niks3-server[1003]: 2026/09/21 09:14:12 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6531worker1 # [ 139.846498] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806532worker1 # [ 139.854867] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6533worker1 # [ 139.863614] nix-daemon[845]: reaped child process 1619, status = succeeded6534worker1 # [ 139.867412] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6535worker1 # [ 140.678835] nix-daemon[845]: reaped child process 1633, status = succeeded6536worker1 # [ 140.869431] nix-daemon[845]: accepted connection from pid 1648, user nix-grpc-daemon (trusted)6537worker1 # [ 140.880835] nix-daemon[845]: reaped child process 1651, status = succeeded6538lb # [ 140.910382] niks3-server[1003]: 2026/09/21 09:14:13 INFO Received uploads request method=POST path=/api/pending_closures6539worker1 # [ 140.911258] nix-grpc-daemon[1402]: time=2026-09-21T09:14:13.981Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6540worker1 # [ 140.912777] nix-grpc-daemon[1402]: time=2026-09-21T09:14:13.983Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6541lb # [ 140.945876] niks3-server[1003]: 2026/09/21 09:14:13 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6542lb # [ 140.958932] niks3-server[1003]: 2026/09/21 09:14:13 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6543lb # [ 140.969322] niks3-server[1003]: 2026/09/21 09:14:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6544lb # [ 140.975475] niks3-server[1003]: 2026/09/21 09:14:13 INFO Signed narinfos id=49 count=16545worker1 # [ 140.967809] nix-grpc-daemon[1402]: time=2026-09-21T09:14:14.038Z level=INFO msg="Uploading 1 narinfos"6546lb # [ 140.981406] niks3-server[1003]: 2026/09/21 09:14:13 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6547lb # [ 140.989840] niks3-server[1003]: 2026/09/21 09:14:13 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6548lb # [ 140.994025] niks3-server[1003]: 2026/09/21 09:14:13 INFO Completed upload id=496549worker1 # [ 140.986805] nix-grpc-daemon[1402]: time=2026-09-21T09:14:14.057Z level=INFO msg="Upload complete. (314ms)"6550lb # [ 140.996463] niks3-server[1003]: 2026/09/21 09:14:13 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6551worker1 # [ 140.992395] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=16552worker2 # [ 141.010966] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:13Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16553worker1 # [ 141.000549] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806554worker2 # [ 141.021518] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:13Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06555worker1 # [ 141.010916] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806556worker1 # [ 141.017592] nix-daemon[845]: reaped child process 1631, status = succeeded6557worker1 # [ 141.023258] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806558worker1 # [ 141.031748] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6559worker1 # [ 141.239253] nix-daemon[845]: accepted connection from pid 1655, user nix-grpc-daemon (trusted)6560worker1 # [ 141.252314] nix-daemon[845]: reaped child process 1658, status = succeeded6561worker1 # [ 141.455847] nix-daemon[845]: accepted connection from pid 1660, user nix-grpc-daemon (trusted)6562worker1 # [ 141.467350] nix-daemon[845]: reaped child process 1663, status = succeeded6563lb # [ 141.495975] niks3-server[1003]: 2026/09/21 09:14:14 INFO Received uploads request method=POST path=/api/pending_closures6564worker1 # [ 141.496435] nix-grpc-daemon[1402]: time=2026-09-21T09:14:14.567Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6565worker1 # [ 141.498288] nix-grpc-daemon[1402]: time=2026-09-21T09:14:14.568Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6566lb # [ 141.522236] niks3-server[1003]: 2026/09/21 09:14:14 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6567lb # [ 141.534676] niks3-server[1003]: 2026/09/21 09:14:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6568lb # [ 141.541122] niks3-server[1003]: 2026/09/21 09:14:14 INFO Signed narinfos id=50 count=16569worker1 # [ 141.536729] nix-grpc-daemon[1402]: time=2026-09-21T09:14:14.607Z level=INFO msg="Uploading 1 narinfos"6570lb # [ 141.546376] niks3-server[1003]: 2026/09/21 09:14:14 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6571lb # [ 141.556957] niks3-server[1003]: 2026/09/21 09:14:14 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6572worker1 # [ 141.555074] nix-grpc-daemon[1402]: time=2026-09-21T09:14:14.625Z level=INFO msg="Upload complete. (518ms)"6573lb # [ 141.562601] niks3-server[1003]: 2026/09/21 09:14:14 INFO Completed upload id=506574worker1 # [ 141.557603] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=5046575lb # [ 141.568254] niks3-server[1003]: 2026/09/21 09:14:14 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6576worker1 # [ 141.562837] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806577worker1 # [ 141.572103] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6578worker1 # [ 141.576157] nix-daemon[845]: reaped child process 1653, status = succeeded6579worker1 # [ 141.581158] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6580worker1 # [ 142.595293] nix-daemon[845]: reaped child process 1667, status = succeeded6581worker1 # [ 142.760299] nix-daemon[845]: accepted connection from pid 1682, user nix-grpc-daemon (trusted)6582worker1 # [ 142.772112] nix-daemon[845]: reaped child process 1685, status = succeeded6583lb # [ 142.799364] niks3-server[1003]: 2026/09/21 09:14:15 INFO Received uploads request method=POST path=/api/pending_closures6584worker1 # [ 142.800479] nix-grpc-daemon[1402]: time=2026-09-21T09:14:15.871Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6585worker1 # [ 142.801993] nix-grpc-daemon[1402]: time=2026-09-21T09:14:15.872Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"6586lb # [ 142.845844] niks3-server[1003]: 2026/09/21 09:14:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6587worker1 # [ 142.845961] nix-grpc-daemon[1402]: time=2026-09-21T09:14:15.916Z level=INFO msg="Uploading 1 narinfos"6588lb # [ 142.853099] niks3-server[1003]: 2026/09/21 09:14:15 INFO Signed narinfos id=51 count=16589lb # [ 142.860550] niks3-server[1003]: 2026/09/21 09:14:15 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6590lb # [ 142.864393] niks3-server[1003]: 2026/09/21 09:14:15 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls6591lb # [ 142.877520] niks3-server[1003]: 2026/09/21 09:14:15 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6592lb # [ 142.880302] niks3-server[1003]: 2026/09/21 09:14:15 INFO Completed upload id=516593worker1 # [ 142.874201] nix-grpc-daemon[1402]: time=2026-09-21T09:14:15.944Z level=INFO msg="Upload complete. (286ms)"6594lb # [ 142.883162] niks3-server[1003]: 2026/09/21 09:14:15 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo6595worker1 # [ 142.876579] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=16596lb # [ 142.888136] niks3-server[1003]: 2026/09/21 09:14:15 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv6597worker2 # [ 142.897747] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:15Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=16598worker1 # [ 142.883871] nix-daemon[845]: reaped child process 1665, status = succeeded6599worker1 # [ 142.922356] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806600worker2 # [ 142.946776] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:15Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=06601worker1 # [ 142.932335] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806602worker1 # [ 142.941352] nix-grpc-daemon[1402]: time=2026-09-21T09:14:16.012Z level=INFO msg="All 1 paths already cached"6603worker1 # [ 142.945872] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806604worker1 # [ 142.950934] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6605worker1 # [ 143.127562] nix-daemon[845]: accepted connection from pid 1689, user nix-grpc-daemon (trusted)6606worker1 # [ 143.139645] nix-daemon[845]: reaped child process 1692, status = succeeded6607worker1 # [ 143.317529] nix-daemon[845]: accepted connection from pid 1694, user nix-grpc-daemon (trusted)6608worker1 # [ 143.329133] nix-daemon[845]: reaped child process 1697, status = succeeded6609lb # [ 143.355675] niks3-server[1003]: 2026/09/21 09:14:15 INFO Received uploads request method=POST path=/api/pending_closures6610worker1 # [ 143.360883] nix-grpc-daemon[1402]: time=2026-09-21T09:14:16.431Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6611worker1 # [ 143.362282] nix-grpc-daemon[1402]: time=2026-09-21T09:14:16.431Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"6612lb # [ 143.389694] niks3-server[1003]: 2026/09/21 09:14:15 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst6613lb # [ 143.400062] niks3-server[1003]: 2026/09/21 09:14:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6614lb # [ 143.407433] niks3-server[1003]: 2026/09/21 09:14:15 INFO Signed narinfos id=52 count=16615worker1 # [ 143.403432] nix-grpc-daemon[1402]: time=2026-09-21T09:14:16.473Z level=INFO msg="Uploading 1 narinfos"6616lb # [ 143.412614] niks3-server[1003]: 2026/09/21 09:14:15 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls6617lb # [ 143.423741] niks3-server[1003]: 2026/09/21 09:14:15 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6618worker1 # [ 143.422302] nix-grpc-daemon[1402]: time=2026-09-21T09:14:16.493Z level=INFO msg="Upload complete. (466ms)"6619lb # [ 143.429815] niks3-server[1003]: 2026/09/21 09:14:15 INFO Completed upload id=526620lb # [ 143.431609] niks3-server[1003]: 2026/09/21 09:14:15 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo6621worker1 # [ 143.424403] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=6246622worker1 # [ 143.428334] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806623worker1 # [ 143.434323] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6624worker1 # [ 143.436548] nix-daemon[845]: reaped child process 1687, status = succeeded6625worker1 # [ 143.442053] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6626worker1 # [ 144.453966] nix-daemon[845]: reaped child process 1701, status = succeeded6627worker1 # [ 144.630403] nix-daemon[845]: accepted connection from pid 1716, user nix-grpc-daemon (trusted)6628worker1 # [ 144.640476] nix-daemon[845]: reaped child process 1719, status = succeeded6629lb # [ 144.668536] niks3-server[1003]: 2026/09/21 09:14:17 INFO Received uploads request method=POST path=/api/pending_closures6630worker1 # [ 144.670122] nix-grpc-daemon[1402]: time=2026-09-21T09:14:17.740Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6631worker1 # [ 144.671683] nix-grpc-daemon[1402]: time=2026-09-21T09:14:17.742Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"6632lb # [ 144.706916] niks3-server[1003]: 2026/09/21 09:14:17 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv6633lb # [ 144.720121] niks3-server[1003]: 2026/09/21 09:14:17 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst6634lb # [ 144.727052] niks3-server[1003]: 2026/09/21 09:14:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6635lb # [ 144.731981] niks3-server[1003]: 2026/09/21 09:14:17 INFO Signed narinfos id=53 count=16636worker1 # [ 144.724789] nix-grpc-daemon[1402]: time=2026-09-21T09:14:17.795Z level=INFO msg="Uploading 1 narinfos"6637lb # [ 144.735396] niks3-server[1003]: 2026/09/21 09:14:17 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls6638lb # [ 144.740800] niks3-server[1003]: 2026/09/21 09:14:17 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6639lb # [ 144.743565] niks3-server[1003]: 2026/09/21 09:14:17 INFO Completed upload id=536640worker1 # [ 144.736158] nix-grpc-daemon[1402]: time=2026-09-21T09:14:17.806Z level=INFO msg="Upload complete. (288ms)"6641lb # [ 144.746604] niks3-server[1003]: 2026/09/21 09:14:17 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo6642worker1 # [ 144.738481] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=16643worker1 # [ 144.745237] nix-daemon[845]: reaped child process 1699, status = succeeded6644worker2 # [ 144.757824] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:17Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=16645worker2 # [ 144.767365] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:17Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=06646worker1 # [ 144.789235] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806647worker1 # [ 144.798134] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806648worker1 # [ 144.812932] nix-grpc-daemon[1402]: time=2026-09-21T09:14:17.883Z level=INFO msg="All 1 paths already cached"6649worker1 # [ 144.817185] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806650worker1 # [ 144.821145] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6651worker1 # [ 145.013384] nix-daemon[845]: accepted connection from pid 1723, user nix-grpc-daemon (trusted)6652worker1 # [ 145.026368] nix-daemon[845]: reaped child process 1726, status = succeeded6653worker1 # [ 145.204793] nix-daemon[845]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6654worker1 # [ 145.216177] nix-daemon[845]: reaped child process 1731, status = succeeded6655lb # [ 145.241588] niks3-server[1003]: 2026/09/21 09:14:17 INFO Received uploads request method=POST path=/api/pending_closures6656worker1 # [ 145.243625] nix-grpc-daemon[1402]: time=2026-09-21T09:14:18.314Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6657worker1 # [ 145.244970] nix-grpc-daemon[1402]: time=2026-09-21T09:14:18.314Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"6658lb # [ 145.268300] niks3-server[1003]: 2026/09/21 09:14:17 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst6659lb # [ 145.280188] niks3-server[1003]: 2026/09/21 09:14:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6660lb # [ 145.285568] niks3-server[1003]: 2026/09/21 09:14:17 INFO Signed narinfos id=54 count=16661worker1 # [ 145.280486] nix-grpc-daemon[1402]: time=2026-09-21T09:14:18.351Z level=INFO msg="Uploading 1 narinfos"6662lb # [ 145.293953] niks3-server[1003]: 2026/09/21 09:14:17 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls6663lb # [ 145.302314] niks3-server[1003]: 2026/09/21 09:14:17 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6664lb # [ 145.305418] niks3-server[1003]: 2026/09/21 09:14:17 INFO Completed upload id=546665worker1 # [ 145.297995] nix-grpc-daemon[1402]: time=2026-09-21T09:14:18.368Z level=INFO msg="Upload complete. (469ms)"6666lb # [ 145.307106] niks3-server[1003]: 2026/09/21 09:14:17 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo6667worker1 # [ 145.299876] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=6246668worker1 # [ 145.304654] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806669worker1 # [ 145.313056] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6670worker1 # [ 145.316358] nix-daemon[845]: reaped child process 1721, status = succeeded6671worker1 # [ 145.321162] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6672worker1 # [ 146.547224] nix-daemon[845]: reaped child process 1735, status = succeeded6673worker1 # [ 146.742474] nix-daemon[845]: accepted connection from pid 1750, user nix-grpc-daemon (trusted)6674worker1 # [ 146.752641] nix-daemon[845]: reaped child process 1753, status = succeeded6675lb # [ 146.780643] niks3-server[1003]: 2026/09/21 09:14:19 INFO Received uploads request method=POST path=/api/pending_closures6676worker1 # [ 146.781649] nix-grpc-daemon[1402]: time=2026-09-21T09:14:19.852Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6677worker1 # [ 146.784520] nix-grpc-daemon[1402]: time=2026-09-21T09:14:19.853Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"6678lb # [ 146.817251] niks3-server[1003]: 2026/09/21 09:14:19 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst6679lb # [ 146.829207] niks3-server[1003]: 2026/09/21 09:14:19 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv6680lb # [ 146.836101] niks3-server[1003]: 2026/09/21 09:14:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6681lb # [ 146.840635] niks3-server[1003]: 2026/09/21 09:14:19 INFO Signed narinfos id=55 count=16682worker1 # [ 146.836200] nix-grpc-daemon[1402]: time=2026-09-21T09:14:19.906Z level=INFO msg="Uploading 1 narinfos"6683lb # [ 146.846120] niks3-server[1003]: 2026/09/21 09:14:19 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls6684lb # [ 146.855592] niks3-server[1003]: 2026/09/21 09:14:19 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6685lb # [ 146.860273] niks3-server[1003]: 2026/09/21 09:14:19 INFO Completed upload id=556686worker1 # [ 146.852944] nix-grpc-daemon[1402]: time=2026-09-21T09:14:19.923Z level=INFO msg="Upload complete. (321ms)"6687lb # [ 146.862973] niks3-server[1003]: 2026/09/21 09:14:19 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo6688worker1 # [ 146.857774] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=16689worker2 # [ 146.876863] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:19Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=16690worker2 # [ 146.886380] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:19Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=06691worker1 # [ 146.873223] nix-daemon[845]: reaped child process 1733, status = succeeded6692worker1 # [ 146.910579] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806693worker1 # [ 146.923646] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806694worker1 # [ 146.937604] nix-grpc-daemon[1402]: time=2026-09-21T09:14:20.008Z level=INFO msg="All 1 paths already cached"6695worker1 # [ 146.945163] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806696worker1 # [ 146.955168] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6697worker1 # [ 147.146790] nix-daemon[845]: accepted connection from pid 1757, user nix-grpc-daemon (trusted)6698worker1 # [ 147.158612] nix-daemon[845]: reaped child process 1760, status = succeeded6699worker1 # [ 147.345117] nix-daemon[845]: accepted connection from pid 1762, user nix-grpc-daemon (trusted)6700worker1 # [ 147.356329] nix-daemon[845]: reaped child process 1765, status = succeeded6701lb # [ 147.383613] niks3-server[1003]: 2026/09/21 09:14:19 INFO Received uploads request method=POST path=/api/pending_closures6702worker1 # [ 147.383838] nix-grpc-daemon[1402]: time=2026-09-21T09:14:20.454Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6703worker1 # [ 147.385339] nix-grpc-daemon[1402]: time=2026-09-21T09:14:20.456Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"6704lb # [ 147.412065] niks3-server[1003]: 2026/09/21 09:14:19 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst6705lb # [ 147.422463] niks3-server[1003]: 2026/09/21 09:14:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6706lb # [ 147.427060] niks3-server[1003]: 2026/09/21 09:14:19 INFO Signed narinfos id=56 count=16707worker1 # [ 147.422633] nix-grpc-daemon[1402]: time=2026-09-21T09:14:20.493Z level=INFO msg="Uploading 1 narinfos"6708lb # [ 147.437976] niks3-server[1003]: 2026/09/21 09:14:20 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls6709lb # [ 147.446753] niks3-server[1003]: 2026/09/21 09:14:20 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6710worker1 # [ 147.444474] nix-grpc-daemon[1402]: time=2026-09-21T09:14:20.515Z level=INFO msg="Upload complete. (477ms)"6711lb # [ 147.452223] niks3-server[1003]: 2026/09/21 09:14:20 INFO Completed upload id=566712worker1 # [ 147.447073] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=6246713lb # [ 147.455923] niks3-server[1003]: 2026/09/21 09:14:20 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo6714worker1 # [ 147.452761] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806715worker1 # [ 147.454407] nix-daemon[845]: reaped child process 1755, status = succeeded6716worker1 # [ 147.457431] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6717worker1 # [ 147.463953] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6718worker1 # [ 147.874467] nix-daemon[845]: reaped child process 1769, status = succeeded6719worker1 # [ 148.052552] nix-daemon[845]: accepted connection from pid 1784, user nix-grpc-daemon (trusted)6720worker1 # [ 148.064066] nix-daemon[845]: reaped child process 1787, status = succeeded6721lb # [ 148.092342] niks3-server[1003]: 2026/09/21 09:14:20 INFO Received uploads request method=POST path=/api/pending_closures6722worker1 # [ 148.093844] nix-grpc-daemon[1402]: time=2026-09-21T09:14:21.164Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6723worker1 # [ 148.095287] nix-grpc-daemon[1402]: time=2026-09-21T09:14:21.166Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"6724lb # [ 148.125819] niks3-server[1003]: 2026/09/21 09:14:20 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv6725lb # [ 148.141653] niks3-server[1003]: 2026/09/21 09:14:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6726lb # [ 148.147375] niks3-server[1003]: 2026/09/21 09:14:20 INFO Signed narinfos id=57 count=16727worker1 # [ 148.142940] nix-grpc-daemon[1402]: time=2026-09-21T09:14:21.213Z level=INFO msg="Uploading 1 narinfos"6728lb # [ 148.155828] niks3-server[1003]: 2026/09/21 09:14:20 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst6729lb # [ 148.163201] niks3-server[1003]: 2026/09/21 09:14:20 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls6730lb # [ 148.171289] niks3-server[1003]: 2026/09/21 09:14:20 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6731lb # [ 148.176386] niks3-server[1003]: 2026/09/21 09:14:20 INFO Completed upload id=576732lb # [ 148.178202] niks3-server[1003]: 2026/09/21 09:14:20 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo6733worker1 # [ 148.169214] nix-grpc-daemon[1402]: time=2026-09-21T09:14:21.239Z level=INFO msg="Upload complete. (298ms)"6734worker2 # [ 148.193365] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:21Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=16735worker1 # [ 148.174669] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=16736worker1 # [ 148.183661] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806737worker2 # [ 148.202828] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:21Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=06738worker1 # [ 148.194567] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806739worker1 # [ 148.200478] nix-daemon[845]: reaped child process 1767, status = succeeded6740worker1 # [ 148.205933] nix-grpc-daemon[1402]: time=2026-09-21T09:14:21.276Z level=INFO msg="All 2 paths already cached"6741worker1 # [ 148.212100] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806742worker1 # [ 148.218789] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6743worker1 # [ 148.403129] nix-daemon[845]: accepted connection from pid 1791, user nix-grpc-daemon (trusted)6744worker1 # [ 148.414643] nix-daemon[845]: reaped child process 1794, status = succeeded6745worker1 # [ 148.604555] nix-daemon[845]: accepted connection from pid 1796, user nix-grpc-daemon (trusted)6746worker1 # [ 148.616256] nix-daemon[845]: reaped child process 1799, status = succeeded6747lb # [ 148.643257] niks3-server[1003]: 2026/09/21 09:14:21 INFO Received uploads request method=POST path=/api/pending_closures6748worker1 # [ 148.645160] nix-grpc-daemon[1402]: time=2026-09-21T09:14:21.715Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6749worker1 # [ 148.646715] nix-grpc-daemon[1402]: time=2026-09-21T09:14:21.717Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"6750lb # [ 148.676543] niks3-server[1003]: 2026/09/21 09:14:21 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst6751lb # [ 148.685100] niks3-server[1003]: 2026/09/21 09:14:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6752lb # [ 148.690375] niks3-server[1003]: 2026/09/21 09:14:21 INFO Signed narinfos id=58 count=16753worker1 # [ 148.686157] nix-grpc-daemon[1402]: time=2026-09-21T09:14:21.756Z level=INFO msg="Uploading 1 narinfos"6754lb # [ 148.696891] niks3-server[1003]: 2026/09/21 09:14:21 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls6755lb # [ 148.709320] niks3-server[1003]: 2026/09/21 09:14:21 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6756worker1 # [ 148.707560] nix-grpc-daemon[1402]: time=2026-09-21T09:14:21.778Z level=INFO msg="Upload complete. (482ms)"6757lb # [ 148.715248] niks3-server[1003]: 2026/09/21 09:14:21 INFO Completed upload id=586758lb # [ 148.718728] niks3-server[1003]: 2026/09/21 09:14:21 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo6759worker1 # [ 148.710207] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=7526760worker1 # [ 148.715755] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806761worker1 # [ 148.724571] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6762worker1 # [ 148.737281] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6763worker1 # [ 148.738573] nix-daemon[845]: reaped child process 1789, status = succeeded6764worker1 # [ 149.556906] nix-daemon[845]: reaped child process 1803, status = succeeded6765worker1 # [ 149.744042] nix-daemon[845]: accepted connection from pid 1818, user nix-grpc-daemon (trusted)6766worker1 # [ 149.754351] nix-daemon[845]: reaped child process 1821, status = succeeded6767lb # [ 149.782801] niks3-server[1003]: 2026/09/21 09:14:22 INFO Received uploads request method=POST path=/api/pending_closures6768worker1 # [ 149.784718] nix-grpc-daemon[1402]: time=2026-09-21T09:14:22.855Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6769worker1 # [ 149.786179] nix-grpc-daemon[1402]: time=2026-09-21T09:14:22.856Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"6770lb # [ 149.822055] niks3-server[1003]: 2026/09/21 09:14:22 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv6771lb # [ 149.828579] niks3-server[1003]: 2026/09/21 09:14:22 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst6772lb # [ 149.841780] niks3-server[1003]: 2026/09/21 09:14:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6773lb # [ 149.846377] niks3-server[1003]: 2026/09/21 09:14:22 INFO Signed narinfos id=59 count=16774worker1 # [ 149.841593] nix-grpc-daemon[1402]: time=2026-09-21T09:14:22.911Z level=INFO msg="Uploading 1 narinfos"6775lb # [ 149.852406] niks3-server[1003]: 2026/09/21 09:14:22 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls6776lb # [ 149.861801] niks3-server[1003]: 2026/09/21 09:14:22 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6777lb # [ 149.866750] niks3-server[1003]: 2026/09/21 09:14:22 INFO Completed upload id=596778worker1 # [ 149.859317] nix-grpc-daemon[1402]: time=2026-09-21T09:14:22.930Z level=INFO msg="Upload complete. (315ms)"6779lb # [ 149.869276] niks3-server[1003]: 2026/09/21 09:14:22 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo6780worker2 # [ 149.881834] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:22Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=16781worker1 # [ 149.864642] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=16782worker1 # [ 149.871498] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806783worker1 # [ 149.882499] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806784worker1 # [ 149.887966] nix-daemon[845]: reaped child process 1801, status = succeeded6785worker1 # [ 149.894142] nix-grpc-daemon[1402]: time=2026-09-21T09:14:22.964Z level=INFO msg="All 2 paths already cached"6786worker1 # [ 149.900520] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806787worker1 # [ 149.906974] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6788worker1 # [ 150.094929] nix-daemon[845]: accepted connection from pid 1825, user nix-grpc-daemon (trusted)6789worker1 # [ 150.106483] nix-daemon[845]: reaped child process 1828, status = succeeded6790worker1 # [ 150.295647] nix-daemon[845]: accepted connection from pid 1830, user nix-grpc-daemon (trusted)6791worker1 # [ 150.307140] nix-daemon[845]: reaped child process 1833, status = succeeded6792lb # [ 150.333963] niks3-server[1003]: 2026/09/21 09:14:22 INFO Received uploads request method=POST path=/api/pending_closures6793worker1 # [ 150.335153] nix-grpc-daemon[1402]: time=2026-09-21T09:14:23.405Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6794worker1 # [ 150.336952] nix-grpc-daemon[1402]: time=2026-09-21T09:14:23.407Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"6795lb # [ 150.360794] niks3-server[1003]: 2026/09/21 09:14:22 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst6796lb # [ 150.372908] niks3-server[1003]: 2026/09/21 09:14:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6797worker1 # [ 150.370257] nix-grpc-daemon[1402]: time=2026-09-21T09:14:23.440Z level=INFO msg="Uploading 1 narinfos"6798lb # [ 150.379435] niks3-server[1003]: 2026/09/21 09:14:22 INFO Signed narinfos id=60 count=16799lb # [ 150.384780] niks3-server[1003]: 2026/09/21 09:14:22 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls6800lb # [ 150.396597] niks3-server[1003]: 2026/09/21 09:14:22 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6801worker1 # [ 150.393911] nix-grpc-daemon[1402]: time=2026-09-21T09:14:23.464Z level=INFO msg="Upload complete. (479ms)"6802lb # [ 150.401402] niks3-server[1003]: 2026/09/21 09:14:22 INFO Completed upload id=606803lb # [ 150.404719] niks3-server[1003]: 2026/09/21 09:14:22 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo6804worker1 # [ 150.396265] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=7526805worker1 # [ 150.401483] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806806worker1 # [ 150.411106] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6807worker1 # [ 150.414987] nix-daemon[845]: reaped child process 1823, status = succeeded6808worker1 # [ 150.419985] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6809worker1 # [ 151.243378] nix-daemon[845]: reaped child process 1837, status = succeeded6810worker1 # [ 151.437358] nix-daemon[845]: accepted connection from pid 1852, user nix-grpc-daemon (trusted)6811worker1 # [ 151.448851] nix-daemon[845]: reaped child process 1855, status = succeeded6812lb # [ 151.476522] niks3-server[1003]: 2026/09/21 09:14:24 INFO Received uploads request method=POST path=/api/pending_closures6813worker1 # [ 151.478004] nix-grpc-daemon[1402]: time=2026-09-21T09:14:24.548Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6814worker1 # [ 151.479515] nix-grpc-daemon[1402]: time=2026-09-21T09:14:24.550Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"6815lb # [ 151.520253] niks3-server[1003]: 2026/09/21 09:14:24 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst6816lb # [ 151.526558] niks3-server[1003]: 2026/09/21 09:14:24 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv6817lb # [ 151.534675] niks3-server[1003]: 2026/09/21 09:14:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6818lb # [ 151.539893] niks3-server[1003]: 2026/09/21 09:14:24 INFO Signed narinfos id=61 count=16819worker1 # [ 151.532999] nix-grpc-daemon[1402]: time=2026-09-21T09:14:24.603Z level=INFO msg="Uploading 1 narinfos"6820lb # [ 151.545537] niks3-server[1003]: 2026/09/21 09:14:24 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls6821lb # [ 151.553743] niks3-server[1003]: 2026/09/21 09:14:24 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6822lb # [ 151.558170] niks3-server[1003]: 2026/09/21 09:14:24 INFO Completed upload id=616823worker1 # [ 151.552256] nix-grpc-daemon[1402]: time=2026-09-21T09:14:24.622Z level=INFO msg="Upload complete. (325ms)"6824lb # [ 151.561371] niks3-server[1003]: 2026/09/21 09:14:24 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo6825worker1 # [ 151.557617] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=16826worker2 # [ 151.576360] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:24Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=16827worker1 # [ 151.571547] nix-daemon[845]: reaped child process 1835, status = succeeded6828worker1 # [ 151.608981] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806829worker1 # [ 151.622607] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806830worker1 # [ 151.635128] nix-grpc-daemon[1402]: time=2026-09-21T09:14:24.705Z level=INFO msg="All 2 paths already cached"6831worker1 # [ 151.642007] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:386806832worker1 # [ 151.652154] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6833worker1 # [ 151.839135] nix-daemon[845]: accepted connection from pid 1859, user nix-grpc-daemon (trusted)6834worker1 # [ 151.851246] nix-daemon[845]: reaped child process 1862, status = succeeded6835worker1 # [ 152.022166] nix-daemon[845]: accepted connection from pid 1864, user nix-grpc-daemon (trusted)6836worker1 # [ 152.033476] nix-daemon[845]: reaped child process 1867, status = succeeded6837lb # [ 152.059993] niks3-server[1003]: 2026/09/21 09:14:24 INFO Received uploads request method=POST path=/api/pending_closures6838worker1 # [ 152.060407] nix-grpc-daemon[1402]: time=2026-09-21T09:14:25.131Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6839worker1 # [ 152.062108] nix-grpc-daemon[1402]: time=2026-09-21T09:14:25.132Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"6840lb # [ 152.088226] niks3-server[1003]: 2026/09/21 09:14:24 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst6841lb # [ 152.100752] niks3-server[1003]: 2026/09/21 09:14:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6842lb # [ 152.107091] niks3-server[1003]: 2026/09/21 09:14:24 INFO Signed narinfos id=62 count=16843worker1 # [ 152.102661] nix-grpc-daemon[1402]: time=2026-09-21T09:14:25.173Z level=INFO msg="Uploading 1 narinfos"6844lb # [ 152.110602] niks3-server[1003]: 2026/09/21 09:14:24 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls6845lb # [ 152.123061] niks3-server[1003]: 2026/09/21 09:14:24 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6846worker1 # [ 152.122694] nix-grpc-daemon[1402]: time=2026-09-21T09:14:25.193Z level=INFO msg="Upload complete. (459ms)"6847lb # [ 152.130290] niks3-server[1003]: 2026/09/21 09:14:24 INFO Completed upload id=626848worker1 # [ 152.125507] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=0 paths=1 nar_bytes_in=7526849lb # [ 152.134243] niks3-server[1003]: 2026/09/21 09:14:24 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo6850worker1 # [ 152.130833] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:386806851worker1 # [ 152.140130] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6852worker1 # [ 152.146476] nix-daemon[845]: reaped child process 1857, status = succeeded6853worker1 # [ 152.151390] nix-daemon[845]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6854worker1 # [ 153.162562] nix-daemon[845]: reaped child process 1871, status = succeeded6855worker1 # [ 153.331265] nix-daemon[845]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)6856worker1 # [ 153.342879] nix-daemon[845]: reaped child process 1889, status = succeeded6857lb # [ 153.371498] niks3-server[1003]: 2026/09/21 09:14:25 INFO Received uploads request method=POST path=/api/pending_closures6858worker1 # [ 153.371776] nix-grpc-daemon[1402]: time=2026-09-21T09:14:26.442Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6859worker1 # [ 153.373268] nix-grpc-daemon[1402]: time=2026-09-21T09:14:26.444Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"6860lb # [ 153.409686] niks3-server[1003]: 2026/09/21 09:14:25 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv6861lb # [ 153.418886] niks3-server[1003]: 2026/09/21 09:14:25 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst6862lb # [ 153.429185] niks3-server[1003]: 2026/09/21 09:14:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6863worker1 # [ 153.426232] nix-grpc-daemon[1402]: time=2026-09-21T09:14:26.496Z level=INFO msg="Uploading 1 narinfos"6864lb # [ 153.433630] niks3-server[1003]: 2026/09/21 09:14:26 INFO Signed narinfos id=63 count=16865lb # [ 153.440138] niks3-server[1003]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls6866lb # [ 153.450625] niks3-server[1003]: 2026/09/21 09:14:26 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6867lb # [ 153.455078] niks3-server[1003]: 2026/09/21 09:14:26 INFO Completed upload id=636868lb # [ 153.457188] niks3-server[1003]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo6869worker1 # [ 153.448987] nix-grpc-daemon[1402]: time=2026-09-21T09:14:26.519Z level=INFO msg="Upload complete. (289ms)"6870worker1 # [ 153.454685] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38680 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=16871worker1 # [ 153.467880] nix-daemon[845]: reaped child process 1869, status = succeeded6872client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 16.31 seconds)6873worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6874worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6875client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'6876client # [ 153.662687] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6877client # Running as unit: gpu.service; invocation ID: 2d68d5adc779454e889c7d9ebe7750126878client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.07 seconds)6879lb: must succeed: curl -sf localhost:9901/clusters6880lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6881lb: must succeed: curl -sf localhost:9901/clusters6882lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6883worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6884client # [ 153.892667] nix[1353]: don't know how to build these paths:6885client # [ 153.894050] nix[1353]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv6886worker1 # [ 153.871979] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:386806887worker2 # [ 153.896621] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343346888worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6889worker2 # [ 153.902943] nix-grpc-daemon[1918]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 70 ms (attempt 1/5)6890worker2 # [ 153.975720] nix-grpc-daemon[1918]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 73 ms (attempt 2/5)6891worker2 # [ 154.054693] nix-grpc-daemon[1918]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 296 ms (attempt 3/5)6892worker2 # [ 154.356621] nix-grpc-daemon[1918]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 27 ms (attempt 4/5)6893worker2 # [ 154.389741] nix-grpc-daemon[1918]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6894worker2 # [ 154.419602] nix-grpc-daemon[1918]: ts=2026-09-21T09:14:27Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=06895client # [ 154.422592] nix[1353]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker6896lb: must succeed: curl -sf localhost:9901/clusters6897lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6898lb: must succeed: curl -sf localhost:9901/clusters6899lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6900worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6901worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6902client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6903lb # [ 158.400249] envoy[921]: conn peer=[2001:db8:1::1]:33394 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27055 tx=10623 ms=160326904client # [ 184.423801] nix[1353]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}6905client # [ 184.432897] nix[1353]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed6906client # [ 184.479519] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6907client # [ 184.482301] systemd[1]: gpu.service: Failed with result 'exit-code'.6908client # [ 184.485913] systemd[1]: gpu.service: Consumed 132ms CPU time over 30.822s wall clock time, 37.4M memory peak, 5.3K incoming IP traffic, 5.1K outgoing IP traffic.6909client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.72 seconds)6910client: must succeed: journalctl -u gpu -o cat6911client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)6912lb: must succeed: curl -sf localhost:9901/clusters6913lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6914lb: must succeed: curl -sf localhost:9901/clusters6915lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6916worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6917worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6918(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 47.84 seconds)6919subtest: low disk drains a worker and builds go to the other6920worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6921worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.95 seconds)6922worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6923worker1 # [ 186.630775] nix-grpc-daemon[1311]: ts=2026-09-21T09:14:59Z level=info event=unhealthy reason=min_free6924worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.12 seconds)6925client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&26926client # don't know how to build these paths:6927client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv6928worker2 # [ 187.402716] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:343346929worker1 # [ 187.389314] nix-grpc-daemon[1311]: ts=2026-09-21T09:15:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:386806930worker2 # [ 187.427997] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:00Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=06931worker2 # [ 187.433440] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:00Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=16932worker2 # [ 187.473328] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343346933worker2 # [ 187.485949] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343346934client # copying 1 paths to 'lb:50051'...6935client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...6936worker2 # [ 187.498251] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:343346937worker2 # [ 187.509200] nix-daemon[869]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)6938worker2 # [ 187.529445] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:00Z level=info event=niks3_push_started pid=20076939worker2 # [ 187.761424] nix-grpc-daemon[2007]: time=2026-09-21T09:15:00.687Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6940worker2 # [ 187.958299] nix-daemon[869]: accepted connection from pid 2020, user nix-grpc-daemon (trusted)6941worker2 # [ 187.969966] nix-daemon[869]: reaped child process 2023, status = succeeded6942worker2 # [ 188.174224] nix-daemon[869]: accepted connection from pid 2025, user nix-grpc-daemon (trusted)6943worker2 # [ 188.185788] nix-daemon[869]: reaped child process 2028, status = succeeded6944lb # [ 188.200248] niks3-server[1003]: 2026/09/21 09:15:00 INFO Received uploads request method=POST path=/api/pending_closures6945worker2 # [ 188.219679] nix-grpc-daemon[2007]: time=2026-09-21T09:15:01.145Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6946worker2 # [ 188.221239] nix-grpc-daemon[2007]: time=2026-09-21T09:15:01.145Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"6947lb # [ 188.234768] niks3-server[1003]: 2026/09/21 09:15:00 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst6948lb # [ 188.247182] niks3-server[1003]: 2026/09/21 09:15:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6949lb # [ 188.255055] niks3-server[1003]: 2026/09/21 09:15:00 INFO Signed narinfos id=64 count=16950worker2 # [ 188.268592] nix-grpc-daemon[2007]: time=2026-09-21T09:15:01.194Z level=INFO msg="Uploading 1 narinfos"6951lb # [ 188.265236] niks3-server[1003]: 2026/09/21 09:15:00 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls6952lb # [ 188.275156] niks3-server[1003]: 2026/09/21 09:15:00 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6953lb # [ 188.281134] niks3-server[1003]: 2026/09/21 09:15:00 INFO Completed upload id=646954lb # [ 188.282982] niks3-server[1003]: 2026/09/21 09:15:00 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo6955worker2 # [ 188.292584] nix-grpc-daemon[2007]: time=2026-09-21T09:15:01.219Z level=INFO msg="Upload complete. (526ms)"6956worker2 # [ 188.296901] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34334 duration_s=0 paths=1 nar_bytes_in=5126957worker2 # [ 188.304330] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343346958worker2 # [ 188.312837] nix-daemon[869]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)6959worker2 # [ 188.321944] nix-daemon[869]: reaped child process 2005, status = succeeded6960worker2 # [ 188.327530] nix-daemon[869]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)6961client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...6962client # farm-a> worker2-next: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv6963lb # [ 188.401288] envoy[921]: conn peer=[2001:db8:1::1]:48836 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=912 tx=516 ms=305536964client # farm-a> LOG-farm-a-drain6965worker2 # [ 188.939734] nix-daemon[869]: reaped child process 2032, status = succeeded6966worker2 # [ 189.131385] nix-daemon[869]: accepted connection from pid 2047, user nix-grpc-daemon (trusted)6967worker2 # [ 189.143220] nix-daemon[869]: reaped child process 2050, status = succeeded6968lb # [ 189.154338] niks3-server[1003]: 2026/09/21 09:15:01 INFO Received uploads request method=POST path=/api/pending_closures6969worker2 # [ 189.175538] nix-grpc-daemon[2007]: time=2026-09-21T09:15:02.101Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6970worker2 # [ 189.177090] nix-grpc-daemon[2007]: time=2026-09-21T09:15:02.103Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"6971lb # [ 189.194815] niks3-server[1003]: 2026/09/21 09:15:01 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv6972lb # [ 189.201528] niks3-server[1003]: 2026/09/21 09:15:01 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6973lb # [ 189.210921] niks3-server[1003]: 2026/09/21 09:15:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6974lb # [ 189.214969] niks3-server[1003]: 2026/09/21 09:15:01 INFO Signed narinfos id=65 count=16975worker2 # [ 189.228677] nix-grpc-daemon[2007]: time=2026-09-21T09:15:02.154Z level=INFO msg="Uploading 1 narinfos"6976lb # [ 189.220243] niks3-server[1003]: 2026/09/21 09:15:01 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls6977lb # [ 189.225477] niks3-server[1003]: 2026/09/21 09:15:01 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6978lb # [ 189.230290] niks3-server[1003]: 2026/09/21 09:15:01 INFO Completed upload id=656979worker2 # [ 189.240519] nix-grpc-daemon[2007]: time=2026-09-21T09:15:02.167Z level=INFO msg="Upload complete. (307ms)"6980lb # [ 189.232216] niks3-server[1003]: 2026/09/21 09:15:01 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo6981worker2 # [ 189.247499] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34334 duration_s=0 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=16982worker2 # [ 189.257189] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:02Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=06983worker2 # [ 189.259276] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:02Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=16984worker2 # [ 189.263936] nix-daemon[869]: reaped child process 2030, status = succeeded6985worker2 # [ 189.300627] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343346986worker2 # [ 189.312944] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343346987worker2 # [ 189.325963] nix-grpc-daemon[2007]: time=2026-09-21T09:15:02.252Z level=INFO msg="All 1 paths already cached"6988client # copying 1 paths to 'lb:50051'...6989client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...6990worker2 # [ 189.334343] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:343346991worker2 # [ 189.342729] nix-daemon[869]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)6992worker2 # [ 189.547753] nix-daemon[869]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)6993worker2 # [ 189.559635] nix-daemon[869]: reaped child process 2059, status = succeeded6994worker2 # [ 189.757043] nix-daemon[869]: accepted connection from pid 2061, user nix-grpc-daemon (trusted)6995worker2 # [ 189.768616] nix-daemon[869]: reaped child process 2064, status = succeeded6996lb # [ 189.778783] niks3-server[1003]: 2026/09/21 09:15:02 INFO Received uploads request method=POST path=/api/pending_closures6997worker2 # [ 189.799707] nix-grpc-daemon[2007]: time=2026-09-21T09:15:02.726Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6998worker2 # [ 189.801717] nix-grpc-daemon[2007]: time=2026-09-21T09:15:02.727Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"6999lb # [ 189.808732] niks3-server[1003]: 2026/09/21 09:15:02 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7000lb # [ 189.820804] niks3-server[1003]: 2026/09/21 09:15:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7001lb # [ 189.824663] niks3-server[1003]: 2026/09/21 09:15:02 INFO Signed narinfos id=66 count=17002worker2 # [ 189.837526] nix-grpc-daemon[2007]: time=2026-09-21T09:15:02.764Z level=INFO msg="Uploading 1 narinfos"7003lb # [ 189.831309] niks3-server[1003]: 2026/09/21 09:15:02 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7004lb # [ 189.839972] niks3-server[1003]: 2026/09/21 09:15:02 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7005lb # [ 189.844281] niks3-server[1003]: 2026/09/21 09:15:02 INFO Completed upload id=667006worker2 # [ 189.854203] nix-grpc-daemon[2007]: time=2026-09-21T09:15:02.780Z level=INFO msg="Upload complete. (499ms)"7007lb # [ 189.846064] niks3-server[1003]: 2026/09/21 09:15:02 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7008worker2 # [ 189.857661] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34334 duration_s=0 paths=1 nar_bytes_in=6247009worker2 # [ 189.863501] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343347010worker2 # [ 189.871148] nix-daemon[869]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)7011worker2 # [ 189.876207] nix-daemon[869]: reaped child process 2054, status = succeeded7012worker2 # [ 189.881193] nix-daemon[869]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)7013client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7014client # farm-b> worker2-next: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7015worker2 # [ 191.094401] nix-daemon[869]: reaped child process 2068, status = succeeded7016client # farm-b> LOG-farm-b-drain7017worker2 # [ 191.258402] nix-daemon[869]: accepted connection from pid 2083, user nix-grpc-daemon (trusted)7018worker2 # [ 191.269527] nix-daemon[869]: reaped child process 2086, status = succeeded7019lb # [ 191.277846] niks3-server[1003]: 2026/09/21 09:15:03 INFO Received uploads request method=POST path=/api/pending_closures7020worker2 # [ 191.297306] nix-grpc-daemon[2007]: time=2026-09-21T09:15:04.223Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7021worker2 # [ 191.298833] nix-grpc-daemon[2007]: time=2026-09-21T09:15:04.225Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7022lb # [ 191.315220] niks3-server[1003]: 2026/09/21 09:15:03 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7023lb # [ 191.321110] niks3-server[1003]: 2026/09/21 09:15:03 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7024lb # [ 191.336071] niks3-server[1003]: 2026/09/21 09:15:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7025worker2 # [ 191.350862] nix-grpc-daemon[2007]: time=2026-09-21T09:15:04.277Z level=INFO msg="Uploading 1 narinfos"7026lb # [ 191.342127] niks3-server[1003]: 2026/09/21 09:15:03 INFO Signed narinfos id=67 count=17027lb # [ 191.346522] niks3-server[1003]: 2026/09/21 09:15:03 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7028lb # [ 191.354534] niks3-server[1003]: 2026/09/21 09:15:03 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7029lb # [ 191.360263] niks3-server[1003]: 2026/09/21 09:15:03 INFO Completed upload id=677030lb # [ 191.361858] niks3-server[1003]: 2026/09/21 09:15:03 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7031worker2 # [ 191.371587] nix-grpc-daemon[2007]: time=2026-09-21T09:15:04.298Z level=INFO msg="Upload complete. (283ms)"7032worker2 # [ 191.380665] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34334 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=17033worker2 # [ 191.390157] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:04Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07034worker2 # [ 191.393440] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:04Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17035worker2 # [ 191.402553] nix-daemon[869]: reaped child process 2066, status = succeeded7036worker2 # [ 191.432871] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343347037worker2 # [ 191.445628] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343347038worker2 # [ 191.457999] nix-grpc-daemon[2007]: time=2026-09-21T09:15:04.384Z level=INFO msg="All 2 paths already cached"7039client # copying 1 paths to 'lb:50051'...7040client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7041worker2 # [ 191.466277] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:343347042worker2 # [ 191.476172] nix-daemon[869]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)7043worker2 # [ 191.645579] nix-daemon[869]: accepted connection from pid 2092, user nix-grpc-daemon (trusted)7044worker2 # [ 191.657653] nix-daemon[869]: reaped child process 2095, status = succeeded7045worker2 # [ 191.826497] nix-daemon[869]: accepted connection from pid 2097, user nix-grpc-daemon (trusted)7046worker2 # [ 191.838307] nix-daemon[869]: reaped child process 2100, status = succeeded7047lb # [ 191.845803] niks3-server[1003]: 2026/09/21 09:15:04 INFO Received uploads request method=POST path=/api/pending_closures7048worker2 # [ 191.864786] nix-grpc-daemon[2007]: time=2026-09-21T09:15:04.791Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7049worker2 # [ 191.866178] nix-grpc-daemon[2007]: time=2026-09-21T09:15:04.791Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7050lb # [ 191.879340] niks3-server[1003]: 2026/09/21 09:15:04 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7051lb # [ 191.888637] niks3-server[1003]: 2026/09/21 09:15:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7052lb # [ 191.894117] niks3-server[1003]: 2026/09/21 09:15:04 INFO Signed narinfos id=68 count=17053worker2 # [ 191.907376] nix-grpc-daemon[2007]: time=2026-09-21T09:15:04.833Z level=INFO msg="Uploading 1 narinfos"7054lb # [ 191.901341] niks3-server[1003]: 2026/09/21 09:15:04 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7055lb # [ 191.912163] niks3-server[1003]: 2026/09/21 09:15:04 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7056lb # [ 191.916944] niks3-server[1003]: 2026/09/21 09:15:04 INFO Completed upload id=687057lb # [ 191.920407] niks3-server[1003]: 2026/09/21 09:15:04 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7058worker2 # [ 191.929544] nix-grpc-daemon[2007]: time=2026-09-21T09:15:04.856Z level=INFO msg="Upload complete. (441ms)"7059worker2 # [ 191.932722] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34334 duration_s=0 paths=1 nar_bytes_in=7527060worker2 # [ 191.938293] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343347061worker2 # [ 191.947927] nix-daemon[869]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)7062worker2 # [ 191.955527] nix-daemon[869]: reaped child process 2090, status = succeeded7063worker2 # [ 191.961121] nix-daemon[869]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)7064client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7065client # farm-top> worker2-next: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7066client # farm-top> LOG-farm-top-drain7067worker2 # [ 193.374971] nix-daemon[869]: reaped child process 2104, status = succeeded7068worker2 # [ 193.530917] nix-daemon[869]: accepted connection from pid 2119, user nix-grpc-daemon (trusted)7069worker2 # [ 193.543151] nix-daemon[869]: reaped child process 2122, status = succeeded7070lb # [ 193.550738] niks3-server[1003]: 2026/09/21 09:15:06 INFO Received uploads request method=POST path=/api/pending_closures7071worker2 # [ 193.569787] nix-grpc-daemon[2007]: time=2026-09-21T09:15:06.496Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7072worker2 # [ 193.572411] nix-grpc-daemon[2007]: time=2026-09-21T09:15:06.498Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7073lb # [ 193.595419] niks3-server[1003]: 2026/09/21 09:15:06 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7074lb # [ 193.600603] niks3-server[1003]: 2026/09/21 09:15:06 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7075lb # [ 193.614583] niks3-server[1003]: 2026/09/21 09:15:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7076worker2 # [ 193.628716] nix-grpc-daemon[2007]: time=2026-09-21T09:15:06.555Z level=INFO msg="Uploading 1 narinfos"7077lb # [ 193.618334] niks3-server[1003]: 2026/09/21 09:15:06 INFO Signed narinfos id=69 count=17078lb # [ 193.623957] niks3-server[1003]: 2026/09/21 09:15:06 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7079lb # [ 193.632924] niks3-server[1003]: 2026/09/21 09:15:06 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7080lb # [ 193.636231] niks3-server[1003]: 2026/09/21 09:15:06 INFO Completed upload id=697081lb # [ 193.637911] niks3-server[1003]: 2026/09/21 09:15:06 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7082worker2 # [ 193.647849] nix-grpc-daemon[2007]: time=2026-09-21T09:15:06.574Z level=INFO msg="Upload complete. (279ms)"7083worker2 # [ 193.655700] nix-grpc-daemon[1918]: ts=2026-09-21T09:15:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34334 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=17084worker2 # [ 193.668953] nix-daemon[869]: reaped child process 2102, status = succeeded7085client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 6.56 seconds)7086client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7087client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.23 seconds)7088worker1: must succeed: rm /nix/.rw-store/fill7089worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.13 seconds)7090worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7091worker1 # [ 194.634353] nix-grpc-daemon[1311]: ts=2026-09-21T09:15:07Z level=info event=healthy reason=min_free7092worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.12 seconds)7093(finished: subtest: low disk drains a worker and builds go to the other, in 10.19 seconds)7094(finished: run the VM test script, in 196.17 seconds)7095test script finished in 196.44s7096cleanup7097kill QemuMachine (pid 49)7098client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7099kill QemuMachine (pid 50)7100lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7101kill QemuMachine (pid 51)7102worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7103kill QemuMachine (pid 52)7104worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7105(finished: cleanup, in 0.95 seconds)7106additionally exposed symbols:7107 client, lb, worker1, worker2,7108 vlan1,7109 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh