vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #494
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker2: QEMU running (pid 52)13worker2 # Disk image does not exist, creating the virtualisation disk image...14worker2 # Formatting '/build/vm-state-worker2/tmp.2fOtyxRwgL', fmt=raw size=107374182415worker1 # Disk image does not exist, creating the virtualisation disk image...16worker1: QEMU running (pid 51)17worker1 # Formatting '/build/vm-state-worker1/tmp.kTQ0W1BM0I', fmt=raw size=107374182418worker2 # mke2fs 1.47.4 (6-Mar-2025)19worker1 # mke2fs 1.47.4 (6-Mar-2025)20worker2 # Discarding device blocks: 0/262144 done21lb # Disk image does not exist, creating the virtualisation disk image...22lb: QEMU running (pid 49)23worker1 # Discarding device blocks: 0/262144 done24worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes25worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes26worker2 # Filesystem UUID: c28dc7c7-33bd-4c82-abcb-54de9793e18127lb # Formatting '/build/vm-state-lb/tmp.oF03DsbxCP', fmt=raw size=107374182428worker1 # Filesystem UUID: 2c76696f-d741-40ba-8b57-9f4ab0085c7429worker2 # Superblock backups stored on blocks:30lb # mke2fs 1.47.4 (6-Mar-2025)31worker1 # Superblock backups stored on blocks:32worker2 # 32768, 98304, 163840, 22937633lb # Discarding device blocks: 0/262144 done34worker1 # 32768, 98304, 163840, 22937635worker2 # 36lb # Creating filesystem with 262144 4k blocks and 65536 inodes37worker1 # 38worker2 # Allocating group tables: 0/8 done39lb # Filesystem UUID: f0540d8d-dc46-40b7-b3f7-f6f93ed7ab3440worker1 # Allocating group tables: 0/8 done41worker2 # Writing inode tables: 0/8 done42lb # Superblock backups stored on blocks:43worker1 # Writing inode tables: 0/8 done44worker2 # Creating journal (8192 blocks): done45lb # 32768, 98304, 163840, 22937646worker1 # Creating journal (8192 blocks): done47worker2 # Writing superblocks and filesystem accounting information: 0/8 done48lb # 49worker1 # Writing superblocks and filesystem accounting information: 0/8 done50worker2 # 51lb # Allocating group tables: 0/8 done52worker1 # 53worker2 # Virtualisation disk image created.54lb # Writing inode tables: 0/8 done55worker1 # Virtualisation disk image created.56client # Disk image does not exist, creating the virtualisation disk image...57lb # Creating journal (8192 blocks): done58client # Formatting '/build/vm-state-client/tmp.kCFLYdbFRY', fmt=raw size=107374182459client: QEMU running (pid 50)60client # mke2fs 1.47.4 (6-Mar-2025)61lb # Writing superblocks and filesystem accounting information: 0/8 done62client # Discarding device blocks: 0/262144 done63client # Creating filesystem with 262144 4k blocks and 65536 inodes64(finished: start all VMs, in 0.20 seconds)65client # Filesystem UUID: 0b5df678-f7c4-4cdd-9397-56990a5593ac66client: waiting for unit network-addresses-eth1.service67lb # 68client: waiting for the VM to finish booting69lb # Virtualisation disk image created.70client # Superblock backups stored on blocks:71client # 32768, 98304, 163840, 22937672client # 73client # Allocating group tables: 0/8 done74client # Writing inode tables: 0/8 done75client # Creating journal (8192 blocks): done76client # Writing superblocks and filesystem accounting information: 0/8 done77client # 78client # Virtualisation disk image created.79lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83lb # 84lb # 85lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0086lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...87lb # 88lb # 89lb # 90lb # 91lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB0092worker2 # 93worker2 # 94worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0095client # 96client # 97lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...98lb # 99lb # 100client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00101worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...102worker2 # 103worker2 # 104worker2 # 105worker2 # 106worker1 # 107worker1 # 108worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00109worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00110client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...111client # 112client # 113client # 114client # 115client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00116worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...117worker1 # 118worker1 # 119worker1 # 120worker1 # 121worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...122worker2 # 123worker2 # 124worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00125client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...126client # 127client # 128worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129worker1 # 130worker1 # 131lb # Booting from ROM...132worker2 # Booting from ROM...133client # Booting from ROM...134worker1 # Booting from ROM...135lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/j1nn9bbhrn5xz154akfhw453f8i07i4d-nixos-system-lb-test/init regInfo=/nix/store/hq8v7nws787a9kwd3h009z1n9dqal4fl-closure-info/registration console=ttyS0,115200n8 console=tty0137lb # [ 0.000000] BIOS-provided physical RAM map:138lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved141lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable142lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved143lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved144lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved145lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved146client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026147lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved148worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026149lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved150lb # [ 0.000000] NX (Execute Disable) protection: active151lb # [ 0.000000] APIC: Static calls initialized152lb # [ 0.000000] SMBIOS 2.8 present.153lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014154client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/d0yzpmy03dgjykw182nqn9rx9qp34ayl-nixos-system-client-test/init regInfo=/nix/store/8qcbhkhv67h7qfb5lww8vbzpf1w4p6d9-closure-info/registration console=ttyS0,115200n8 console=tty0155lb # [ 0.000000] DMI: Memory slots populated: 1/1156client # [ 0.000000] BIOS-provided physical RAM map:157worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/97sck759bskxrmqr38axr5kzb0p084pz-nixos-system-worker2-test/init regInfo=/nix/store/dfij3l6x15yx3kznbya019zn219wy845-closure-info/registration console=ttyS0,115200n8 console=tty0158lb # [ 0.000000] Hypervisor detected: KVM159worker2 # [ 0.000000] BIOS-provided physical RAM map:160client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable161lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000162worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable163client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved164lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00165worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved166lb # [ 0.000000] kvm-clock: using sched offset of 475946022 cycles167client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved168worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved169client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable170lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns171worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026172worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable173client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved174lb # [ 0.000004] tsc: Detected 2400.012 MHz processor175worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved176lb # [ 0.000813] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000177client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved178worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved179lb # [ 0.000840] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs180client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved181worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved182client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved183lb # [ 0.000842] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT184worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved185lb # [ 0.002731] found SMP MP-table at [mem 0x000f5450-0x000f545f]186worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/pj67bngsjx4svw66my9vshw84i37agz7-nixos-system-worker1-test/init regInfo=/nix/store/j1ik3n9mca5rj6xyz1di0lygynxpmyha-closure-info/registration console=ttyS0,115200n8 console=tty0187client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved188lb # [ 0.002742] Using GB pages for direct mapping189worker1 # [ 0.000000] BIOS-provided physical RAM map:190worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved191client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved192lb # [ 0.002845] RAMDISK: [mem 0x5e367000-0x5ffcffff]193worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable194client # [ 0.000000] NX (Execute Disable) protection: active195worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved196lb # [ 0.002853] ACPI: Early table checksum verification disabled197client # [ 0.000000] APIC: Static calls initialized198worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved199worker2 # [ 0.000000] NX (Execute Disable) protection: active200lb # [ 0.002856] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )201client # [ 0.000000] SMBIOS 2.8 present.202worker2 # [ 0.000000] APIC: Static calls initialized203worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved204worker2 # [ 0.000000] SMBIOS 2.8 present.205lb # [ 0.002860] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)206client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014207worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable208client # [ 0.000000] DMI: Memory slots populated: 1/1209worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014210lb # [ 0.002864] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)211worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved212client # [ 0.000000] Hypervisor detected: KVM213worker2 # [ 0.000000] DMI: Memory slots populated: 1/1214client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000215worker2 # [ 0.000000] Hypervisor detected: KVM216lb # [ 0.002872] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)217worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved218client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00219lb # [ 0.002874] ACPI: FACS 0x000000005FFE0000 000040220worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000221worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved222worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00223client # [ 0.000000] kvm-clock: using sched offset of 471492823 cycles224lb # [ 0.002875] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)225worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved226worker2 # [ 0.000000] kvm-clock: using sched offset of 472421457 cycles227client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns228worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved229lb # [ 0.002877] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)230client # [ 0.000005] tsc: Detected 2400.012 MHz processor231worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns232worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved233lb # [ 0.002878] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)234client # [ 0.000815] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000235worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor236worker1 # [ 0.000000] NX (Execute Disable) protection: active237worker1 # [ 0.000000] APIC: Static calls initialized238worker2 # [ 0.000811] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000239lb # [ 0.002880] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)240client # [ 0.000842] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs241worker1 # [ 0.000000] SMBIOS 2.8 present.242worker2 # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs243lb # [ 0.002881] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]244client # [ 0.000845] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT245worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014246client # [ 0.002742] found SMP MP-table at [mem 0x000f5450-0x000f545f]247worker2 # [ 0.000840] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT248lb # [ 0.002882] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]249worker1 # [ 0.000000] DMI: Memory slots populated: 1/1250client # [ 0.002752] Using GB pages for direct mapping251worker2 # [ 0.002751] found SMP MP-table at [mem 0x000f5450-0x000f545f]252worker1 # [ 0.000000] Hypervisor detected: KVM253lb # [ 0.002883] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]254client # [ 0.002871] RAMDISK: [mem 0x5e366000-0x5ffcffff]255worker2 # [ 0.002762] Using GB pages for direct mapping256worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000257lb # [ 0.002883] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]258client # [ 0.002878] ACPI: Early table checksum verification disabled259worker2 # [ 0.002889] RAMDISK: [mem 0x5e367000-0x5ffcffff]260worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00261client # [ 0.002880] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )262worker2 # [ 0.002898] ACPI: Early table checksum verification disabled263lb # [ 0.002883] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]264worker1 # [ 0.000000] kvm-clock: using sched offset of 490673306 cycles265worker2 # [ 0.002901] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )266lb # [ 0.002884] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]267client # [ 0.002884] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)268worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns269worker2 # [ 0.002905] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)270lb # [ 0.002884] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]271client # [ 0.002888] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)272worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor273lb # [ 0.003106] No NUMA configuration found274worker1 # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000275worker2 # [ 0.002909] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)276lb # [ 0.003107] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]277client # [ 0.002894] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)278client # [ 0.002896] ACPI: FACS 0x000000005FFE0000 000040279lb # [ 0.003109] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]280worker1 # [ 0.000839] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs281worker2 # [ 0.002916] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)282lb # [ 0.005350] Zone ranges:283worker2 # [ 0.002918] ACPI: FACS 0x000000005FFE0000 000040284worker1 # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT285client # [ 0.002898] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)286lb # [ 0.005351] DMA [mem 0x0000000000001000-0x0000000000ffffff]287worker1 # [ 0.002734] found SMP MP-table at [mem 0x000f5450-0x000f545f]288worker2 # [ 0.002920] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)289lb # [ 0.005352] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]290client # [ 0.002899] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)291worker1 # [ 0.002745] Using GB pages for direct mapping292lb # [ 0.005353] Normal empty293lb # [ 0.005354] Device empty294worker1 # [ 0.002830] RAMDISK: [mem 0x5e367000-0x5ffcffff]295worker2 # [ 0.002921] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)296lb # [ 0.005355] Movable zone start for each node297client # [ 0.002901] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)298worker1 # [ 0.002838] ACPI: Early table checksum verification disabled299lb # [ 0.005355] Early memory node ranges300worker2 # [ 0.002923] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)301client # [ 0.002902] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)302worker1 # [ 0.002840] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )303lb # [ 0.005356] node 0: [mem 0x0000000000001000-0x000000000009efff]304worker2 # [ 0.002925] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)305client # [ 0.002904] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]306lb # [ 0.005357] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]307worker1 # [ 0.002844] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)308worker2 # [ 0.002926] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]309client # [ 0.002905] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]310lb # [ 0.005358] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]311worker1 # [ 0.002848] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)312worker2 # [ 0.002927] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]313lb # [ 0.005378] On node 0, zone DMA: 1 pages in unavailable ranges314client # [ 0.002905] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]315worker1 # [ 0.002855] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)316lb # [ 0.005640] On node 0, zone DMA: 97 pages in unavailable ranges317worker2 # [ 0.002927] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]318client # [ 0.002906] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]319worker1 # [ 0.002857] ACPI: FACS 0x000000005FFE0000 000040320lb # [ 0.031659] On node 0, zone DMA32: 40 pages in unavailable ranges321client # [ 0.002906] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]322worker2 # [ 0.002928] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]323lb # [ 0.032112] ACPI: PM-Timer IO Port: 0x608324worker1 # [ 0.002858] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)325lb # [ 0.032122] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])326client # [ 0.002907] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]327worker2 # [ 0.002928] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]328worker1 # [ 0.002860] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)329lb # [ 0.032148] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23330client # [ 0.002907] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]331worker2 # [ 0.002929] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]332client # [ 0.003127] No NUMA configuration found333lb # [ 0.032151] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)334worker1 # [ 0.002861] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)335worker2 # [ 0.002929] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]336client # [ 0.003128] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]337worker2 # [ 0.003153] No NUMA configuration found338lb # [ 0.032152] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)339worker1 # [ 0.002863] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)340client # [ 0.003130] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]341worker2 # [ 0.003154] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]342lb # [ 0.032153] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)343client # [ 0.005340] Zone ranges:344worker1 # [ 0.002864] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]345worker2 # [ 0.003156] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]346client # [ 0.005340] DMA [mem 0x0000000000001000-0x0000000000ffffff]347lb # [ 0.032154] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)348worker2 # [ 0.005588] Zone ranges:349worker1 # [ 0.002865] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]350client # [ 0.005342] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]351lb # [ 0.032155] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)352worker2 # [ 0.005589] DMA [mem 0x0000000000001000-0x0000000000ffffff]353client # [ 0.005343] Normal empty354worker1 # [ 0.002866] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]355client # [ 0.005344] Device empty356lb # [ 0.032157] ACPI: Using ACPI (MADT) for SMP configuration information357worker2 # [ 0.005591] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]358client # [ 0.005344] Movable zone start for each node359worker2 # [ 0.005600] Normal empty360worker1 # [ 0.002866] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]361lb # [ 0.032158] ACPI: HPET id: 0x8086a201 base: 0xfed00000362worker2 # [ 0.005600] Device empty363client # [ 0.005345] Early memory node ranges364lb # [ 0.032161] TSC deadline timer available365worker2 # [ 0.005601] Movable zone start for each node366worker1 # [ 0.002867] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]367client # [ 0.005345] node 0: [mem 0x0000000000001000-0x000000000009efff]368lb # [ 0.032165] CPU topo: Max. logical packages: 1369worker2 # [ 0.005602] Early memory node ranges370lb # [ 0.032166] CPU topo: Max. logical dies: 1371worker1 # [ 0.002867] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]372client # [ 0.005346] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]373worker2 # [ 0.005602] node 0: [mem 0x0000000000001000-0x000000000009efff]374lb # [ 0.032166] CPU topo: Max. dies per package: 1375worker1 # [ 0.002868] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]376client # [ 0.005347] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]377worker2 # [ 0.005603] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]378lb # [ 0.032170] CPU topo: Max. threads per core: 1379worker1 # [ 0.003086] No NUMA configuration found380client # [ 0.005366] On node 0, zone DMA: 1 pages in unavailable ranges381lb # [ 0.032170] CPU topo: Num. cores per package: 1382worker2 # [ 0.005604] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]383worker1 # [ 0.003087] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]384lb # [ 0.032171] CPU topo: Num. threads per package: 1385client # [ 0.005628] On node 0, zone DMA: 97 pages in unavailable ranges386worker2 # [ 0.005625] On node 0, zone DMA: 1 pages in unavailable ranges387worker1 # [ 0.003089] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]388lb # [ 0.032171] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs389client # [ 0.031407] On node 0, zone DMA32: 40 pages in unavailable ranges390worker1 # [ 0.005466] Zone ranges:391worker2 # [ 0.005903] On node 0, zone DMA: 97 pages in unavailable ranges392client # [ 0.031858] ACPI: PM-Timer IO Port: 0x608393lb # [ 0.032189] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()394worker1 # [ 0.005467] DMA [mem 0x0000000000001000-0x0000000000ffffff]395worker2 # [ 0.032401] On node 0, zone DMA32: 40 pages in unavailable ranges396client # [ 0.031869] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])397worker2 # [ 0.032864] ACPI: PM-Timer IO Port: 0x608398worker1 # [ 0.005469] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]399lb # [ 0.032220] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]400worker1 # [ 0.005470] Normal empty401client # [ 0.031894] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23402worker2 # [ 0.032876] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])403worker1 # [ 0.005471] Device empty404lb # [ 0.032222] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]405client # [ 0.031897] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)406worker1 # [ 0.005471] Movable zone start for each node407worker2 # [ 0.032905] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23408worker1 # [ 0.005472] Early memory node ranges409lb # [ 0.032223] [mem 0x60000000-0xafffffff] available for PCI devices410client # [ 0.031898] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)411worker2 # [ 0.032908] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)412lb # [ 0.032224] Booting paravirtualized kernel on KVM413worker1 # [ 0.005472] node 0: [mem 0x0000000000001000-0x000000000009efff]414client # [ 0.031899] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)415worker2 # [ 0.032909] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)416worker1 # [ 0.005473] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]417lb # [ 0.032227] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns418client # [ 0.031900] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)419worker2 # [ 0.032910] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)420worker1 # [ 0.005474] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]421client # [ 0.031901] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)422lb # [ 0.036668] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1423worker1 # [ 0.005493] On node 0, zone DMA: 1 pages in unavailable ranges424worker2 # [ 0.032911] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)425lb # [ 0.038813] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152426client # [ 0.031903] ACPI: Using ACPI (MADT) for SMP configuration information427worker1 # [ 0.005752] On node 0, zone DMA: 97 pages in unavailable ranges428worker2 # [ 0.032912] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)429lb # [ 0.038857] kvm-guest: PV spinlocks disabled, single CPU430client # [ 0.031904] ACPI: HPET id: 0x8086a201 base: 0xfed00000431worker1 # [ 0.031404] On node 0, zone DMA32: 40 pages in unavailable ranges432client # [ 0.031908] TSC deadline timer available433worker2 # [ 0.032914] ACPI: Using ACPI (MADT) for SMP configuration information434worker1 # [ 0.031861] ACPI: PM-Timer IO Port: 0x608435client # [ 0.031912] CPU topo: Max. logical packages: 1436worker2 # [ 0.032915] ACPI: HPET id: 0x8086a201 base: 0xfed00000437worker1 # [ 0.031872] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])438client # [ 0.031913] CPU topo: Max. logical dies: 1439worker2 # [ 0.032919] TSC deadline timer available440client # [ 0.031913] CPU topo: Max. dies per package: 1441worker2 # [ 0.032923] CPU topo: Max. logical packages: 1442worker1 # [ 0.031897] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23443client # [ 0.031917] CPU topo: Max. threads per core: 1444worker2 # [ 0.032924] CPU topo: Max. logical dies: 1445worker1 # [ 0.031899] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)446client # [ 0.031918] CPU topo: Num. cores per package: 1447worker2 # [ 0.032924] CPU topo: Max. dies per package: 1448lb # [ 0.038858] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/j1nn9bbhrn5xz154akfhw453f8i07i4d-nixos-system-lb-test/init regInfo=/nix/store/hq8v7nws787a9kwd3h009z1n9dqal4fl-closure-info/registration console=ttyS0,115200n8 console=tty0449worker1 # [ 0.031901] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)450worker2 # [ 0.032928] CPU topo: Max. threads per core: 1451client # [ 0.031918] CPU topo: Num. threads per package: 1452worker2 # [ 0.032928] CPU topo: Num. cores per package: 1453client # [ 0.031918] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs454worker1 # [ 0.031902] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)455worker2 # [ 0.032929] CPU topo: Num. threads per package: 1456client # [ 0.031936] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()457lb # [ 0.038952] Unknown kernel command line parameters "regInfo=/nix/store/hq8v7nws787a9kwd3h009z1n9dqal4fl-closure-info/registration", will be passed to user space.458worker1 # [ 0.031903] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)459worker2 # [ 0.032929] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs460lb # [ 0.038965] random: crng init done461worker1 # [ 0.031904] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)462client # [ 0.031965] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]463worker2 # [ 0.032947] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()464lb # [ 0.038966] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes465worker1 # [ 0.031906] ACPI: Using ACPI (MADT) for SMP configuration information466client # [ 0.031967] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]467worker2 # [ 0.032980] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]468worker1 # [ 0.031907] ACPI: HPET id: 0x8086a201 base: 0xfed00000469lb # [ 0.041224] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)470client # [ 0.031968] [mem 0x60000000-0xafffffff] available for PCI devices471worker1 # [ 0.031910] TSC deadline timer available472client # [ 0.031969] Booting paravirtualized kernel on KVM473worker2 # [ 0.032982] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]474lb # [ 0.042263] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)475worker1 # [ 0.031914] CPU topo: Max. logical packages: 1476lb # [ 0.042293] Fallback order for Node 0: 0477worker1 # [ 0.031915] CPU topo: Max. logical dies: 1478worker2 # [ 0.032983] [mem 0x60000000-0xafffffff] available for PCI devices479worker1 # [ 0.031916] CPU topo: Max. dies per package: 1480client # [ 0.031972] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns481lb # [ 0.042295] Built 1 zonelists, mobility grouping on. Total pages: 393078482worker2 # [ 0.032985] Booting paravirtualized kernel on KVM483lb # [ 0.042296] Policy zone: DMA32484worker1 # [ 0.031919] CPU topo: Max. threads per core: 1485client # [ 0.036411] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1486worker1 # [ 0.031919] CPU topo: Num. cores per package: 1487lb # [ 0.044718] mem auto-init: stack:all(zero), heap alloc:on, heap free:off488worker2 # [ 0.032988] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns489client # [ 0.038711] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152490worker1 # [ 0.031920] CPU topo: Num. threads per package: 1491lb # [ 0.047692] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1492worker2 # [ 0.037435] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1493client # [ 0.038754] kvm-guest: PV spinlocks disabled, single CPU494worker1 # [ 0.031920] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs495lb # [ 0.051127] allocated 3145728 bytes of page_ext496worker2 # [ 0.039614] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152497lb # [ 0.061136] ftrace: allocating 48731 entries in 192 pages498worker1 # [ 0.031937] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()499worker2 # [ 0.039660] kvm-guest: PV spinlocks disabled, single CPU500lb # [ 0.061138] ftrace: allocated 192 pages with 2 groups501worker1 # [ 0.031967] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]502lb # [ 0.061962] Dynamic Preempt: lazy503lb # [ 0.062087] rcu: Preemptible hierarchical RCU implementation.504worker1 # [ 0.031969] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]505lb # [ 0.062087] rcu: RCU event tracing is enabled.506worker1 # [ 0.031970] [mem 0x60000000-0xafffffff] available for PCI devices507client # [ 0.038755] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/d0yzpmy03dgjykw182nqn9rx9qp34ayl-nixos-system-client-test/init regInfo=/nix/store/8qcbhkhv67h7qfb5lww8vbzpf1w4p6d9-closure-info/registration console=ttyS0,115200n8 console=tty0508lb # [ 0.062088] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.509worker1 # [ 0.031971] Booting paravirtualized kernel on KVM510lb # [ 0.062089] Trampoline variant of Tasks RCU enabled.511lb # [ 0.062090] Rude variant of Tasks RCU enabled.512worker2 # [ 0.039661] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/97sck759bskxrmqr38axr5kzb0p084pz-nixos-system-worker2-test/init regInfo=/nix/store/dfij3l6x15yx3kznbya019zn219wy845-closure-info/registration console=ttyS0,115200n8 console=tty0513worker1 # [ 0.031974] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns514client # [ 0.038846] Unknown kernel command line parameters "regInfo=/nix/store/8qcbhkhv67h7qfb5lww8vbzpf1w4p6d9-closure-info/registration", will be passed to user space.515lb # [ 0.062090] Tracing variant of Tasks RCU enabled.516client # [ 0.038859] random: crng init done517worker1 # [ 0.036446] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1518lb # [ 0.062091] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.519client # [ 0.038860] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes520worker1 # [ 0.038543] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152521worker2 # [ 0.039757] Unknown kernel command line parameters "regInfo=/nix/store/dfij3l6x15yx3kznbya019zn219wy845-closure-info/registration", will be passed to user space.522lb # [ 0.062092] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1523worker2 # [ 0.039770] random: crng init done524client # [ 0.040935] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)525worker1 # [ 0.038584] kvm-guest: PV spinlocks disabled, single CPU526worker2 # [ 0.039771] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes527lb # [ 0.062106] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.528client # [ 0.041970] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)529client # [ 0.042000] Fallback order for Node 0: 0530worker2 # [ 0.041882] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)531lb # [ 0.062108] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.532client # [ 0.042002] Built 1 zonelists, mobility grouping on. Total pages: 393078533worker2 # [ 0.043121] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)534client # [ 0.042003] Policy zone: DMA32535lb # [ 0.062109] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.536worker2 # [ 0.043151] Fallback order for Node 0: 0537client # [ 0.044414] mem auto-init: stack:all(zero), heap alloc:on, heap free:off538lb # [ 0.066462] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16539worker1 # [ 0.038586] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/pj67bngsjx4svw66my9vshw84i37agz7-nixos-system-worker1-test/init regInfo=/nix/store/j1ik3n9mca5rj6xyz1di0lygynxpmyha-closure-info/registration console=ttyS0,115200n8 console=tty0540worker2 # [ 0.043154] Built 1 zonelists, mobility grouping on. Total pages: 393078541client # [ 0.047392] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1542worker2 # [ 0.043155] Policy zone: DMA32543lb # [ 0.066771] rcu: srcu_init: Setting srcu_struct sizes based on contention.544client # [ 0.051295] allocated 3145728 bytes of page_ext545worker2 # [ 0.045652] mem auto-init: stack:all(zero), heap alloc:on, heap free:off546client # [ 0.061129] ftrace: allocating 48731 entries in 192 pages547worker1 # [ 0.038684] Unknown kernel command line parameters "regInfo=/nix/store/j1ik3n9mca5rj6xyz1di0lygynxpmyha-closure-info/registration", will be passed to user space.548lb # [ 0.066778] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns549worker1 # [ 0.038697] random: crng init done550worker2 # [ 0.048805] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1551client # [ 0.061130] ftrace: allocated 192 pages with 2 groups552client # [ 0.061957] Dynamic Preempt: lazy553worker2 # [ 0.052107] allocated 3145728 bytes of page_ext554worker1 # [ 0.038697] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes555lb # [ 0.066879] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)556client # [ 0.062117] rcu: Preemptible hierarchical RCU implementation.557worker2 # [ 0.061670] ftrace: allocating 48731 entries in 192 pages558lb # [ 0.070406] Console: colour VGA+ 80x25559client # [ 0.062118] rcu: RCU event tracing is enabled.560worker1 # [ 0.040769] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)561worker2 # [ 0.061672] ftrace: allocated 192 pages with 2 groups562lb # [ 0.070409] printk: legacy console [tty0] enabled563worker2 # [ 0.062509] Dynamic Preempt: lazy564lb # [ 0.099794] printk: legacy console [ttyS0] enabled565client # [ 0.062118] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.566worker1 # [ 0.041805] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)567worker2 # [ 0.062692] rcu: Preemptible hierarchical RCU implementation.568client # [ 0.062120] Trampoline variant of Tasks RCU enabled.569worker1 # [ 0.041840] Fallback order for Node 0: 0570lb # [ 0.196186] ACPI: Core revision 20250807571worker2 # [ 0.062693] rcu: RCU event tracing is enabled.572client # [ 0.062121] Rude variant of Tasks RCU enabled.573worker1 # [ 0.041842] Built 1 zonelists, mobility grouping on. Total pages: 393078574client # [ 0.062121] Tracing variant of Tasks RCU enabled.575worker2 # [ 0.062694] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.576worker1 # [ 0.041843] Policy zone: DMA32577lb # [ 0.197068] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns578worker2 # [ 0.062695] Trampoline variant of Tasks RCU enabled.579worker1 # [ 0.045028] mem auto-init: stack:all(zero), heap alloc:on, heap free:off580client # [ 0.062122] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.581worker2 # [ 0.062696] Rude variant of Tasks RCU enabled.582lb # [ 0.198542] APIC: Switch to symmetric I/O mode setup583worker1 # [ 0.048108] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1584client # [ 0.062122] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1585worker2 # [ 0.062696] Tracing variant of Tasks RCU enabled.586lb # [ 0.199472] x2apic enabled587worker1 # [ 0.051724] allocated 3145728 bytes of page_ext588client # [ 0.062138] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.589worker2 # [ 0.062697] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.590worker1 # [ 0.061404] ftrace: allocating 48731 entries in 192 pages591lb # [ 0.200206] APIC: Switched APIC routing to: physical x2apic592worker1 # [ 0.061406] ftrace: allocated 192 pages with 2 groups593worker2 # [ 0.062698] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1594client # [ 0.062140] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.595worker1 # [ 0.062244] Dynamic Preempt: lazy596worker1 # [ 0.062376] rcu: Preemptible hierarchical RCU implementation.597worker2 # [ 0.062722] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.598client # [ 0.062140] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.599worker1 # [ 0.062376] rcu: RCU event tracing is enabled.600lb # [ 0.202020] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1601client # [ 0.066501] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16602worker2 # [ 0.062723] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.603worker1 # [ 0.062377] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.604client # [ 0.066789] rcu: srcu_init: Setting srcu_struct sizes based on contention.605worker1 # [ 0.062378] Trampoline variant of Tasks RCU enabled.606worker2 # [ 0.062724] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.607lb # [ 0.202981] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns608worker1 # [ 0.062378] Rude variant of Tasks RCU enabled.609client # [ 0.066796] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns610worker2 # [ 0.067278] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16611worker1 # [ 0.062379] Tracing variant of Tasks RCU enabled.612lb # [ 0.204604] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)613worker1 # [ 0.062379] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.614worker2 # [ 0.067548] rcu: srcu_init: Setting srcu_struct sizes based on contention.615client # [ 0.066895] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)616client # [ 0.070414] Console: colour VGA+ 80x25617worker1 # [ 0.062380] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1618lb # [ 0.205688] x86/cpu: User Mode Instruction Prevention (UMIP) activated619worker2 # [ 0.067556] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns620client # [ 0.070417] printk: legacy console [tty0] enabled621worker1 # [ 0.062436] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.622client # [ 0.099840] printk: legacy console [ttyS0] enabled623lb # [ 0.206720] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127624worker2 # [ 0.067665] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)625client # [ 0.195739] ACPI: Core revision 20250807626worker2 # [ 0.071206] Console: colour VGA+ 80x25627worker1 # [ 0.062438] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.628lb # [ 0.207602] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0629worker2 # [ 0.071209] printk: legacy console [tty0] enabled630worker2 # [ 0.100628] printk: legacy console [ttyS0] enabled631worker1 # [ 0.062439] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.632client # [ 0.196601] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns633lb # [ 0.208606] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto634worker1 # [ 0.066800] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16635worker2 # [ 0.198060] ACPI: Core revision 20250807636client # [ 0.198081] APIC: Switch to symmetric I/O mode setup637worker1 # [ 0.067082] rcu: srcu_init: Setting srcu_struct sizes based on contention.638client # [ 0.199010] x2apic enabled639lb # [ 0.209602] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl640worker2 # [ 0.198959] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns641worker1 # [ 0.067089] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns642client # [ 0.199740] APIC: Switched APIC routing to: physical x2apic643lb # [ 0.210602] Transient Scheduler Attacks: Vulnerable: No microcode644worker2 # [ 0.200470] APIC: Switch to symmetric I/O mode setup645lb # [ 0.211602] Spectre V2 : Mitigation: Enhanced / Automatic IBRS646worker1 # [ 0.067188] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)647worker2 # [ 0.201463] x2apic enabled648worker1 # [ 0.070735] Console: colour VGA+ 80x25649client # [ 0.201534] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1650lb # [ 0.212602] Speculative Return Stack Overflow: Mitigation: Safe RET651worker1 # [ 0.070738] printk: legacy console [tty0] enabled652worker2 # [ 0.202250] APIC: Switched APIC routing to: physical x2apic653worker1 # [ 0.100038] printk: legacy console [ttyS0] enabled654client # [ 0.202473] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns655worker1 # [ 0.195608] ACPI: Core revision 20250807656worker2 # [ 0.204091] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1657client # [ 0.204059] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)658worker1 # [ 0.196523] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns659client # [ 0.206145] x86/cpu: User Mode Instruction Prevention (UMIP) activated660lb # [ 0.213601] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization661worker2 # [ 0.205093] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns662worker1 # [ 0.198071] APIC: Switch to symmetric I/O mode setup663lb # [ 0.215610] Spectre V2 : Enabling IBPB for BPF664worker1 # [ 0.199013] x2apic enabled665client # [ 0.207219] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127666worker2 # [ 0.206782] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)667worker1 # [ 0.199746] APIC: Switched APIC routing to: physical x2apic668client # [ 0.208052] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0669lb # [ 0.216313] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier670worker2 # [ 0.207868] x86/cpu: User Mode Instruction Prevention (UMIP) activated671lb # [ 0.217602] active return thunk: srso_alias_return_thunk672worker2 # [ 0.208923] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127673client # [ 0.209056] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto674worker1 # [ 0.201538] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1675lb # [ 0.218622] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'676worker2 # [ 0.210474] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0677client # [ 0.210052] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl678lb # [ 0.219601] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'679worker1 # [ 0.202609] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns680client # [ 0.211052] Transient Scheduler Attacks: Vulnerable: No microcode681lb # [ 0.220601] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'682worker2 # [ 0.211554] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto683client # [ 0.212052] Spectre V2 : Mitigation: Enhanced / Automatic IBRS684worker1 # [ 0.204293] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)685lb # [ 0.221589] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'686worker2 # [ 0.212780] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl687client # [ 0.213052] Speculative Return Stack Overflow: Mitigation: Safe RET688worker1 # [ 0.205380] x86/cpu: User Mode Instruction Prevention (UMIP) activated689lb # [ 0.222601] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'690worker2 # [ 0.213780] Transient Scheduler Attacks: Vulnerable: No microcode691worker1 # [ 0.206425] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127692lb # [ 0.223601] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'693worker2 # [ 0.215511] Spectre V2 : Mitigation: Enhanced / Automatic IBRS694worker1 # [ 0.207995] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0695lb # [ 0.224601] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'696worker2 # [ 0.216486] Speculative Return Stack Overflow: Mitigation: Safe RET697lb # [ 0.225602] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256698client # [ 0.214052] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization699worker1 # [ 0.208294] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto700client # [ 0.216059] Spectre V2 : Enabling IBPB for BPF701lb # [ 0.226601] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64702worker1 # [ 0.210290] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl703lb # [ 0.227601] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512704client # [ 0.217568] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier705worker1 # [ 0.211290] Transient Scheduler Attacks: Vulnerable: No microcode706lb # [ 0.228602] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024707client # [ 0.219053] active return thunk: srso_alias_return_thunk708worker2 # [ 0.217534] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization709worker1 # [ 0.212990] Spectre V2 : Mitigation: Enhanced / Automatic IBRS710lb # [ 0.229601] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8711worker2 # [ 0.218787] Spectre V2 : Enabling IBPB for BPF712client # [ 0.219888] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'713worker1 # [ 0.213976] Speculative Return Stack Overflow: Mitigation: Safe RET714client # [ 0.221052] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'715worker2 # [ 0.219780] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier716lb # [ 0.230602] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.717worker2 # [ 0.220780] active return thunk: srso_alias_return_thunk718client # [ 0.221989] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'719client # [ 0.222782] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'720worker2 # [ 0.222463] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'721client # [ 0.223791] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'722worker2 # [ 0.223779] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'723worker1 # [ 0.214290] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization724client # [ 0.224776] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'725worker1 # [ 0.216298] Spectre V2 : Enabling IBPB for BPF726worker2 # [ 0.224758] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'727client # [ 0.225814] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'728worker2 # [ 0.225779] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'729worker1 # [ 0.217291] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier730client # [ 0.227052] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256731worker2 # [ 0.226780] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'732worker1 # [ 0.218291] active return thunk: srso_alias_return_thunk733client # [ 0.227965] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64734worker2 # [ 0.227779] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'735worker1 # [ 0.219946] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'736client # [ 0.228754] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512737worker2 # [ 0.228779] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'738worker1 # [ 0.221203] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'739client # [ 0.229758] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024740worker2 # [ 0.229779] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256741worker1 # [ 0.222031] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'742client # [ 0.230754] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8743worker2 # [ 0.230779] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64744worker1 # [ 0.223025] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'745client # [ 0.231052] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.746worker2 # [ 0.231779] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512747worker1 # [ 0.224057] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'748worker2 # [ 0.232779] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024749worker1 # [ 0.225052] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'750worker2 # [ 0.233779] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8751worker1 # [ 0.226115] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'752worker2 # [ 0.234779] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.753worker1 # [ 0.227290] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256754worker1 # [ 0.228290] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64755worker1 # [ 0.229290] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512756worker1 # [ 0.230290] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024757worker1 # [ 0.231290] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8758worker1 # [ 0.232290] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759lb # [ 0.265000] Freeing SMP alternatives memory: 44K760lb # [ 0.265604] pid_max: default: 32768 minimum: 301761lb # [ 0.266618] LSM: initializing lsm=capability,landlock,yama,bpf,ima762lb # [ 0.267678] landlock: Up and running.763lb # [ 0.268284] Yama: becoming mindful.764lb # [ 0.268814] LSM support for eBPF active765lb # [ 0.269708] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)766lb # [ 0.270639] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)767lb # [ 0.273298] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)768client # [ 0.264106] Freeing SMP alternatives memory: 44K769lb # [ 0.274124] Performance Events: Fam17h+ core perfctr, AMD PMU driver.770client # [ 0.264863] pid_max: default: 32768 minimum: 301771lb # [ 0.274606] ... version: 2772lb # [ 0.275308] ... bit width: 48773client # [ 0.265680] LSM: initializing lsm=capability,landlock,yama,bpf,ima774lb # [ 0.275604] ... generic counters: 6775client # [ 0.266162] landlock: Up and running.776lb # [ 0.276278] ... generic bitmap: 000000000000003f777client # [ 0.267052] Yama: becoming mindful.778lb # [ 0.276655] ... fixed-purpose counters: 0779client # [ 0.268179] LSM support for eBPF active780worker2 # [ 0.268613] Freeing SMP alternatives memory: 44K781lb # [ 0.277367] ... fixed-purpose bitmap: 0000000000000000782worker2 # [ 0.269360] pid_max: default: 32768 minimum: 301783client # [ 0.268909] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)784lb # [ 0.277604] ... value mask: 0000ffffffffffff785worker2 # [ 0.269873] LSM: initializing lsm=capability,landlock,yama,bpf,ima786client # [ 0.270098] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)787lb # [ 0.278507] ... max period: 00007fffffffffff788worker2 # [ 0.270885] landlock: Up and running.789worker2 # [ 0.271780] Yama: becoming mindful.790lb # [ 0.279275] ... global_ctrl mask: 000000000000003f791worker2 # [ 0.272881] LSM support for eBPF active792lb # [ 0.279731] signal: max sigframe size: 3376793client # [ 0.272345] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)794lb # [ 0.280474] rcu: Hierarchical SRCU implementation.795worker2 # [ 0.273690] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)796worker1 # [ 0.266222] Freeing SMP alternatives memory: 44K797lb # [ 0.281191] rcu: Max phase no-delay instances is 400.798worker1 # [ 0.266868] pid_max: default: 32768 minimum: 301799client # [ 0.273651] Performance Events: Fam17h+ core perfctr, AMD PMU driver.800worker2 # [ 0.274829] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)801client # [ 0.274102] ... version: 2802worker1 # [ 0.267372] LSM: initializing lsm=capability,landlock,yama,bpf,ima803client # [ 0.274781] ... bit width: 48804worker1 # [ 0.268391] landlock: Up and running.805client # [ 0.275054] ... generic counters: 6806worker1 # [ 0.269291] Yama: becoming mindful.807worker2 # [ 0.277231] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)808client # [ 0.275746] ... generic bitmap: 000000000000003f809worker1 # [ 0.270478] LSM support for eBPF active810client # [ 0.276054] ... fixed-purpose counters: 0811worker2 # [ 0.278329] Performance Events: Fam17h+ core perfctr, AMD PMU driver.812client # [ 0.276730] ... fixed-purpose bitmap: 0000000000000000813worker1 # [ 0.271244] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)814lb # [ 0.286113] smp: Bringing up secondary CPUs ...815worker2 # [ 0.278788] ... version: 2816lb # [ 0.286618] smp: Brought up 1 node, 1 CPU817client # [ 0.277054] ... value mask: 0000ffffffffffff818worker2 # [ 0.279495] ... bit width: 48819worker1 # [ 0.272333] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)820client # [ 0.277906] ... max period: 00007fffffffffff821lb # [ 0.287293] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)822worker2 # [ 0.279782] ... generic counters: 6823client # [ 0.278698] ... global_ctrl mask: 000000000000003f824worker2 # [ 0.280484] ... generic bitmap: 000000000000003f825client # [ 0.279147] signal: max sigframe size: 3376826worker2 # [ 0.280782] ... fixed-purpose counters: 0827worker1 # [ 0.274790] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)828lb # [ 0.287754] Memory: 1454564K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)829client # [ 0.279901] rcu: Hierarchical SRCU implementation.830worker2 # [ 0.281467] ... fixed-purpose bitmap: 0000000000000000831client # [ 0.280596] rcu: Max phase no-delay instances is 400.832lb # [ 0.288838] devtmpfs: initialized833worker2 # [ 0.281782] ... value mask: 0000ffffffffffff834worker1 # [ 0.275875] Performance Events: Fam17h+ core perfctr, AMD PMU driver.835lb # [ 0.289700] x86/mm: Memory block size: 128MB836worker1 # [ 0.276299] ... version: 2837worker2 # [ 0.282649] ... max period: 00007fffffffffff838worker1 # [ 0.277004] ... bit width: 48839worker2 # [ 0.283441] ... global_ctrl mask: 000000000000003f840worker1 # [ 0.277292] ... generic counters: 6841worker2 # [ 0.283879] signal: max sigframe size: 3376842lb # [ 0.291468] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)843worker1 # [ 0.278013] ... generic bitmap: 000000000000003f844worker2 # [ 0.284683] rcu: Hierarchical SRCU implementation.845worker1 # [ 0.278292] ... fixed-purpose counters: 0846worker2 # [ 0.285386] rcu: Max phase no-delay instances is 400.847lb # [ 0.292525] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).848worker1 # [ 0.279008] ... fixed-purpose bitmap: 0000000000000000849client # [ 0.285527] smp: Bringing up secondary CPUs ...850lb # [ 0.293723] pinctrl core: initialized pinctrl subsystem851client # [ 0.286068] smp: Brought up 1 node, 1 CPU852worker1 # [ 0.279292] ... value mask: 0000ffffffffffff853lb # [ 0.294867] PM: RTC time: 11:21:23, date: 2026-09-22854client # [ 0.286709] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)855worker1 # [ 0.280185] ... max period: 00007fffffffffff856worker1 # [ 0.280930] ... global_ctrl mask: 000000000000003f857worker1 # [ 0.281386] signal: max sigframe size: 3376858client # [ 0.287246] Memory: 1454580K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)859worker1 # [ 0.282178] rcu: Hierarchical SRCU implementation.860client # [ 0.288259] devtmpfs: initialized861worker2 # [ 0.290246] smp: Bringing up secondary CPUs ...862lb # [ 0.298271] NET: Registered PF_NETLINK/PF_ROUTE protocol family863worker1 # [ 0.282907] rcu: Max phase no-delay instances is 400.864worker2 # [ 0.290796] smp: Brought up 1 node, 1 CPU865client # [ 0.288958] x86/mm: Memory block size: 128MB866worker2 # [ 0.291466] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)867lb # [ 0.299103] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations868client # [ 0.290609] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)869lb # [ 0.299632] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations870worker2 # [ 0.291992] Memory: 1454560K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)871client # [ 0.291079] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).872worker2 # [ 0.293024] devtmpfs: initialized873lb # [ 0.300880] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations874client # [ 0.292138] pinctrl core: initialized pinctrl subsystem875worker1 # [ 0.287838] smp: Bringing up secondary CPUs ...876worker2 # [ 0.293832] x86/mm: Memory block size: 128MB877lb # [ 0.301613] audit: initializing netlink subsys (disabled)878client # [ 0.293245] PM: RTC time: 11:21:23, date: 2026-09-22879worker1 # [ 0.288306] smp: Brought up 1 node, 1 CPU880lb # [ 0.302606] audit: type=2000 audit(1790076084.249:1): state=initialized audit_enabled=0 res=1881worker1 # [ 0.288983] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)882worker2 # [ 0.295580] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)883lb # [ 0.303764] thermal_sys: Registered thermal governor 'fair_share'884lb # [ 0.303767] thermal_sys: Registered thermal governor 'bang_bang'885worker2 # [ 0.296701] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).886worker1 # [ 0.289467] Memory: 1454564K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)887worker2 # [ 0.297858] pinctrl core: initialized pinctrl subsystem888client # [ 0.296944] NET: Registered PF_NETLINK/PF_ROUTE protocol family889lb # [ 0.304605] thermal_sys: Registered thermal governor 'step_wise'890worker1 # [ 0.290516] devtmpfs: initialized891lb # [ 0.305539] thermal_sys: Registered thermal governor 'user_space'892worker2 # [ 0.298929] PM: RTC time: 11:21:23, date: 2026-09-22893worker1 # [ 0.291279] x86/mm: Memory block size: 128MB894client # [ 0.298237] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations895lb # [ 0.306304] thermal_sys: Registered thermal governor 'power_allocator'896lb # [ 0.306616] cpuidle: using governor menu897client # [ 0.299078] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations898worker1 # [ 0.292838] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)899lb # [ 0.308837] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5900client # [ 0.300338] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations901worker1 # [ 0.293317] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).902worker2 # [ 0.302473] NET: Registered PF_NETLINK/PF_ROUTE protocol family903client # [ 0.301063] audit: initializing netlink subsys (disabled)904worker1 # [ 0.294374] pinctrl core: initialized pinctrl subsystem905lb # [ 0.309830] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]906client # [ 0.302134] thermal_sys: Registered thermal governor 'fair_share'907worker2 # [ 0.303285] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations908worker1 # [ 0.295454] PM: RTC time: 11:21:23, date: 2026-09-22909lb # [ 0.310607] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry910client # [ 0.302136] thermal_sys: Registered thermal governor 'bang_bang'911worker2 # [ 0.303805] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations912lb # [ 0.311617] PCI: Using configuration type 1 for base access913client # [ 0.303055] thermal_sys: Registered thermal governor 'step_wise'914client # [ 0.303960] thermal_sys: Registered thermal governor 'user_space'915worker2 # [ 0.305053] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations916lb # [ 0.312692] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.917worker2 # [ 0.305791] audit: initializing netlink subsys (disabled)918client # [ 0.304750] audit: type=2000 audit(1790076084.261:1): state=initialized audit_enabled=0 res=1919worker1 # [ 0.299301] NET: Registered PF_NETLINK/PF_ROUTE protocol family920worker2 # [ 0.306915] thermal_sys: Registered thermal governor 'fair_share'921client # [ 0.306057] thermal_sys: Registered thermal governor 'power_allocator'922client # [ 0.306071] cpuidle: using governor menu923worker2 # [ 0.306918] thermal_sys: Registered thermal governor 'bang_bang'924worker1 # [ 0.300789] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations925worker2 # [ 0.307783] thermal_sys: Registered thermal governor 'step_wise'926client # [ 0.308242] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5927worker1 # [ 0.301313] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations928worker2 # [ 0.308760] thermal_sys: Registered thermal governor 'user_space'929worker1 # [ 0.302568] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations930client # [ 0.309290] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]931worker2 # [ 0.309532] audit: type=2000 audit(1790076084.264:1): state=initialized audit_enabled=0 res=1932worker1 # [ 0.303302] audit: initializing netlink subsys (disabled)933client # [ 0.310057] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry934lb # [ 0.319622] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages935worker2 # [ 0.310785] thermal_sys: Registered thermal governor 'power_allocator'936client # [ 0.311066] PCI: Using configuration type 1 for base access937worker1 # [ 0.304429] thermal_sys: Registered thermal governor 'fair_share'938worker2 # [ 0.310799] cpuidle: using governor menu939lb # [ 0.320605] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page940worker1 # [ 0.304432] thermal_sys: Registered thermal governor 'bang_bang'941client # [ 0.312109] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.942worker2 # [ 0.313010] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5943worker1 # [ 0.305294] thermal_sys: Registered thermal governor 'step_wise'944worker1 # [ 0.306254] thermal_sys: Registered thermal governor 'user_space'945worker2 # [ 0.314026] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]946worker1 # [ 0.307043] audit: type=2000 audit(1790076084.271:1): state=initialized audit_enabled=0 res=1947worker2 # [ 0.314785] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry948worker1 # [ 0.308296] thermal_sys: Registered thermal governor 'power_allocator'949worker2 # [ 0.315793] PCI: Using configuration type 1 for base access950lb # [ 0.325606] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages951worker1 # [ 0.308310] cpuidle: using governor menu952lb # [ 0.326605] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page953client # [ 0.317325] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages954worker2 # [ 0.316883] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.955worker1 # [ 0.310499] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5956client # [ 0.318055] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page957worker1 # [ 0.311526] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]958worker1 # [ 0.312296] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry959worker1 # [ 0.313305] PCI: Using configuration type 1 for base access960worker2 # [ 0.321867] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages961client # [ 0.323056] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages962worker1 # [ 0.314413] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.963worker2 # [ 0.322783] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page964client # [ 0.324054] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page965worker2 # [ 0.323773] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages966lb # [ 0.337983] ACPI: Added _OSI(Module Device)967worker1 # [ 0.319377] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages968lb # [ 0.338606] ACPI: Added _OSI(Processor Device)969worker1 # [ 0.320294] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page970lb # [ 0.339400] ACPI: Added _OSI(Processor Aggregator Device)971worker2 # [ 0.328782] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page972client # [ 0.334478] ACPI: Added _OSI(Module Device)973worker1 # [ 0.325294] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages974client # [ 0.335056] ACPI: Added _OSI(Processor Device)975worker2 # [ 0.334656] ACPI: Added _OSI(Module Device)976worker1 # [ 0.326292] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page977client # [ 0.335764] ACPI: Added _OSI(Processor Aggregator Device)978worker2 # [ 0.335291] ACPI: Added _OSI(Processor Device)979worker2 # [ 0.335783] ACPI: Added _OSI(Processor Aggregator Device)980lb # [ 0.348155] ACPI: 1 ACPI AML tables successfully acquired and loaded981lb # [ 0.352014] ACPI: Interpreter enabled982lb # [ 0.352619] ACPI: PM: (supports S0 S3 S4 S5)983lb # [ 0.353374] ACPI: Using IOAPIC for interrupt routing984client # [ 0.344564] ACPI: 1 ACPI AML tables successfully acquired and loaded985worker1 # [ 0.336574] ACPI: Added _OSI(Module Device)986lb # [ 0.353664] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug987worker1 # [ 0.337270] ACPI: Added _OSI(Processor Device)988lb # [ 0.354605] PCI: Using E820 reservations for host bridge windows989worker1 # [ 0.337869] ACPI: Added _OSI(Processor Aggregator Device)990client # [ 0.348476] ACPI: Interpreter enabled991lb # [ 0.357741] ACPI: Enabled 2 GPEs in block 00 to 3F992client # [ 0.349071] ACPI: PM: (supports S0 S3 S4 S5)993worker2 # [ 0.349276] ACPI: 1 ACPI AML tables successfully acquired and loaded994client # [ 0.349749] ACPI: Using IOAPIC for interrupt routing995worker2 # [ 0.350677] ACPI: Interpreter enabled996client # [ 0.350129] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug997worker2 # [ 0.351243] ACPI: PM: (supports S0 S3 S4 S5)998client # [ 0.351055] PCI: Using E820 reservations for host bridge windows999worker2 # [ 0.351782] ACPI: Using IOAPIC for interrupt routing1000worker2 # [ 0.352698] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1001client # [ 0.354124] ACPI: Enabled 2 GPEs in block 00 to 3F1002worker1 # [ 0.346638] ACPI: 1 ACPI AML tables successfully acquired and loaded1003lb # [ 0.365196] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1004lb # [ 0.365611] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1005worker2 # [ 0.357783] PCI: Using E820 reservations for host bridge windows1006lb # [ 0.366681] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1007worker1 # [ 0.350705] ACPI: Interpreter enabled1008worker2 # [ 0.358869] ACPI: Enabled 2 GPEs in block 00 to 3F1009worker1 # [ 0.351308] ACPI: PM: (supports S0 S3 S4 S5)1010lb # [ 0.367727] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1011worker1 # [ 0.351982] ACPI: Using IOAPIC for interrupt routing1012lb # [ 0.368957] PCI host bridge to bus 0000:001013worker1 # [ 0.352326] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1014lb # [ 0.369609] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1015worker1 # [ 0.353293] PCI: Using E820 reservations for host bridge windows1016lb # [ 0.370605] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1017client # [ 0.362779] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1018lb # [ 0.371604] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1019client # [ 0.363061] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1020lb # [ 0.372604] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1021worker1 # [ 0.356381] ACPI: Enabled 2 GPEs in block 00 to 3F1022client # [ 0.364337] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1023lb # [ 0.373604] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1024client # [ 0.365177] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1025lb # [ 0.374604] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1026client # [ 0.366404] PCI host bridge to bus 0000:001027worker2 # [ 0.367544] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1028lb # [ 0.375604] pci_bus 0000:00: root bus resource [bus 00-ff]1029client # [ 0.367060] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1030client # [ 0.368055] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1031lb # [ 0.376606] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1032worker2 # [ 0.369783] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1033client # [ 0.369055] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1034lb # [ 0.378074] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1035worker2 # [ 0.370860] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1036client # [ 0.370055] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1037worker2 # [ 0.371903] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1038client # [ 0.371054] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1039worker2 # [ 0.373135] PCI host bridge to bus 0000:001040worker1 # [ 0.364967] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1041client # [ 0.372054] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1042worker2 # [ 0.373787] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1043client # [ 0.373055] pci_bus 0000:00: root bus resource [bus 00-ff]1044worker1 # [ 0.365300] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1045worker2 # [ 0.374782] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1046client # [ 0.373953] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1047worker2 # [ 0.375782] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1048worker1 # [ 0.367257] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1049client # [ 0.375458] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1050worker2 # [ 0.376782] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1051worker1 # [ 0.368282] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1052lb # [ 0.380690] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1053worker2 # [ 0.377782] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1054worker1 # [ 0.369504] PCI host bridge to bus 0000:001055lb # [ 0.381617] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1056worker1 # [ 0.370187] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1057worker2 # [ 0.378787] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1058lb # [ 0.382550] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1059worker2 # [ 0.379811] pci_bus 0000:00: root bus resource [bus 00-ff]1060worker1 # [ 0.371144] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1061worker1 # [ 0.372156] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1062worker2 # [ 0.380775] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1063lb # [ 0.383562] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1064worker1 # [ 0.373158] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1065worker2 # [ 0.382243] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1066lb # [ 0.385385] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1067worker1 # [ 0.374250] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1068client # [ 0.378113] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1069worker1 # [ 0.375253] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1070client # [ 0.379067] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1071worker1 # [ 0.376300] pci_bus 0000:00: root bus resource [bus 00-ff]1072client # [ 0.380000] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1073lb # [ 0.388635] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1074worker1 # [ 0.377375] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1075lb # [ 0.389500] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1076client # [ 0.381269] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1077lb # [ 0.390367] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1078worker1 # [ 0.378732] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1079lb # [ 0.390611] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1080client # [ 0.382753] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1081worker2 # [ 0.384866] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1082worker2 # [ 0.385794] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1083lb # [ 0.392554] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1084client # [ 0.385064] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1085worker2 # [ 0.386787] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1086client # [ 0.385887] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1087client # [ 0.386774] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1088worker2 # [ 0.388007] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1089lb # [ 0.395635] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1090client # [ 0.387061] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1091lb # [ 0.396492] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1092worker2 # [ 0.389512] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1093lb # [ 0.397376] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1094client # [ 0.388566] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1095worker1 # [ 0.381353] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1096worker1 # [ 0.382305] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1097lb # [ 0.398977] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1098worker1 # [ 0.383315] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1099worker2 # [ 0.392813] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1100client # [ 0.390071] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1101worker2 # [ 0.393655] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1102client # [ 0.390891] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1103worker1 # [ 0.384712] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1104worker2 # [ 0.394541] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1105client # [ 0.391754] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1106worker2 # [ 0.394788] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1107worker1 # [ 0.385990] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1108client # [ 0.392597] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1109lb # [ 0.401612] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1110lb # [ 0.403212] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1111worker2 # [ 0.396357] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1112lb # [ 0.403626] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1113worker1 # [ 0.389323] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1114worker1 # [ 0.390165] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1115worker2 # [ 0.398816] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1116lb # [ 0.405194] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1117client # [ 0.395071] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1118worker1 # [ 0.391012] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1119worker2 # [ 0.399707] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1120client # [ 0.396061] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1121worker1 # [ 0.391299] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1122worker2 # [ 0.400531] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1123client # [ 0.396962] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1124lb # [ 0.407483] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1125lb # [ 0.408270] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1126client # [ 0.398392] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1127worker2 # [ 0.401364] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1128worker1 # [ 0.392994] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1129lb # [ 0.408626] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1130client # [ 0.400063] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1131client # [ 0.400895] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1132worker1 # [ 0.396251] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1133lb # [ 0.410141] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1134worker1 # [ 0.396941] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1135client # [ 0.401762] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1136worker2 # [ 0.404000] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1137worker2 # [ 0.405391] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1138worker1 # [ 0.397315] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1139lb # [ 0.413312] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1140worker2 # [ 0.405804] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1141client # [ 0.402594] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1142lb # [ 0.413611] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1143worker1 # [ 0.398853] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1144lb # [ 0.414551] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1145worker2 # [ 0.407359] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1146client # [ 0.404062] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1147client # [ 0.404890] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1148lb # [ 0.416036] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1149client # [ 0.405781] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1150worker2 # [ 0.408814] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1151worker2 # [ 0.409678] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1152worker1 # [ 0.401589] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1153client # [ 0.406667] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1154worker1 # [ 0.402956] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1155worker2 # [ 0.410570] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1156worker1 # [ 0.403314] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1157lb # [ 0.418635] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1158lb # [ 0.419612] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1159worker2 # [ 0.412141] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1160client # [ 0.408989] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1161worker1 # [ 0.404863] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1162lb # [ 0.420555] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1163client # [ 0.409664] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1164worker2 # [ 0.414814] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1165lb # [ 0.422016] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1166client # [ 0.410076] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1167worker1 # [ 0.407127] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1168worker2 # [ 0.415675] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1169worker1 # [ 0.407944] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1170client # [ 0.411599] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1171worker2 # [ 0.416579] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1172lb # [ 0.424383] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1173worker1 # [ 0.408315] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1174lb # [ 0.424611] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1175client # [ 0.413061] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1176worker2 # [ 0.418215] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1177lb # [ 0.425522] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1178client # [ 0.413893] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1179lb # [ 0.426424] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1180worker1 # [ 0.410409] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1181client # [ 0.414773] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1182client # [ 0.415061] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1183worker2 # [ 0.420814] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1184lb # [ 0.428431] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1185worker1 # [ 0.412324] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1186worker2 # [ 0.421789] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1187client # [ 0.416566] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1188worker1 # [ 0.413157] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1189worker2 # [ 0.422710] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1190lb # [ 0.430617] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1191worker1 # [ 0.414068] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1192client # [ 0.418077] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1193lb # [ 0.431567] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1194worker2 # [ 0.424254] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1195worker1 # [ 0.415711] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1196client # [ 0.419077] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1197lb # [ 0.433002] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1198worker2 # [ 0.425815] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1199client # [ 0.420615] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1200worker2 # [ 0.426676] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1201client # [ 0.422062] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1202worker2 # [ 0.427579] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1203lb # [ 0.435647] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1204worker1 # [ 0.418679] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1205client # [ 0.422881] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1206worker1 # [ 0.419924] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1207worker2 # [ 0.428636] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1208lb # [ 0.436536] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1209client # [ 0.423772] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1210worker1 # [ 0.420315] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1211lb # [ 0.437339] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1212worker2 # [ 0.430321] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1213client # [ 0.424610] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1214worker1 # [ 0.421900] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1215lb # [ 0.438152] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1216worker2 # [ 0.431788] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1217client # [ 0.426596] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1218worker2 # [ 0.432775] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1219worker1 # [ 0.423944] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1220lb # [ 0.440279] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1221client # [ 0.427062] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1222worker1 # [ 0.424300] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1223lb # [ 0.440611] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1224worker2 # [ 0.434327] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1225client # [ 0.427986] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1226worker1 # [ 0.425207] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1227lb # [ 0.441581] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1228client # [ 0.429404] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1229worker1 # [ 0.426107] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1230worker2 # [ 0.435805] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1231client # [ 0.430589] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1232lb # [ 0.443013] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1233worker1 # [ 0.427848] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1234worker2 # [ 0.436646] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1235client # [ 0.431257] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1236lb # [ 0.444450] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1237worker2 # [ 0.437804] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1238client # [ 0.432627] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1239worker1 # [ 0.429306] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1240lb # [ 0.445439] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1241worker2 # [ 0.439502] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1242worker1 # [ 0.430249] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1243lb # [ 0.447311] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1244worker1 # [ 0.431679] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1245worker2 # [ 0.441790] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1246worker2 # [ 0.442674] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1247worker2 # [ 0.443588] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1248worker1 # [ 0.434040] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1249worker2 # [ 0.445250] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1250worker1 # [ 0.434300] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1251worker1 # [ 0.435316] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1252worker2 # [ 0.447182] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1253worker1 # [ 0.436845] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1254worker2 # [ 0.448016] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1255client # [ 0.433250] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1256worker2 # [ 0.449416] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1257worker1 # [ 0.438300] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1258worker1 # [ 0.439135] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1259lb # [ 0.447822] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1260worker1 # [ 0.440021] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1261lb # [ 0.450397] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1262client # [ 0.436643] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1263worker1 # [ 0.440881] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1264lb # [ 0.450837] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1265client # [ 0.437333] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1266worker1 # [ 0.442779] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1267lb # [ 0.452287] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1268worker1 # [ 0.443489] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1269lb # [ 0.452893] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1270client # [ 0.439066] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1271worker1 # [ 0.445038] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1272lb # [ 0.453886] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1273client # [ 0.440337] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1274lb # [ 0.454893] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1275client # [ 0.441344] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1276worker2 # [ 0.449982] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1277lb # [ 0.456633] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1278client # [ 0.442319] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1279lb # [ 0.457481] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1280worker2 # [ 0.452441] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1281client # [ 0.444092] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1282worker2 # [ 0.453055] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1283lb # [ 0.458632] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1284client # [ 0.444909] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1285worker2 # [ 0.454726] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1286lb # [ 0.461432] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1287client # [ 0.446132] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1288worker2 # [ 0.455769] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1289client # [ 0.448082] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1290worker2 # [ 0.457081] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1291lb # [ 0.465198] ACPI: PCI: Interrupt link LNKA configured for IRQ 101292worker1 # [ 0.445500] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1293worker2 # [ 0.458028] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1294lb # [ 0.465711] ACPI: PCI: Interrupt link LNKB configured for IRQ 101295worker1 # [ 0.448047] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1296lb # [ 0.466706] ACPI: PCI: Interrupt link LNKC configured for IRQ 111297client # [ 0.452007] ACPI: PCI: Interrupt link LNKA configured for IRQ 101298worker2 # [ 0.459740] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1299lb # [ 0.467700] ACPI: PCI: Interrupt link LNKD configured for IRQ 111300client # [ 0.452856] ACPI: PCI: Interrupt link LNKB configured for IRQ 101301worker2 # [ 0.460391] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1302worker1 # [ 0.448532] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1303client # [ 0.453835] ACPI: PCI: Interrupt link LNKC configured for IRQ 111304lb # [ 0.468701] ACPI: PCI: Interrupt link LNKE configured for IRQ 101305worker2 # [ 0.461093] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1306client # [ 0.454150] ACPI: PCI: Interrupt link LNKD configured for IRQ 111307lb # [ 0.469702] ACPI: PCI: Interrupt link LNKF configured for IRQ 101308worker1 # [ 0.450299] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1309client # [ 0.455149] ACPI: PCI: Interrupt link LNKE configured for IRQ 101310lb # [ 0.470704] ACPI: PCI: Interrupt link LNKG configured for IRQ 111311worker2 # [ 0.462442] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1312worker1 # [ 0.451481] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1313client # [ 0.456152] ACPI: PCI: Interrupt link LNKF configured for IRQ 101314lb # [ 0.471700] ACPI: PCI: Interrupt link LNKH configured for IRQ 111315lb # [ 0.472639] ACPI: PCI: Interrupt link GSIA configured for IRQ 161316client # [ 0.457152] ACPI: PCI: Interrupt link LNKG configured for IRQ 111317worker1 # [ 0.452593] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1318lb # [ 0.473554] ACPI: PCI: Interrupt link GSIB configured for IRQ 171319client # [ 0.458149] ACPI: PCI: Interrupt link LNKH configured for IRQ 111320lb # [ 0.474338] ACPI: PCI: Interrupt link GSIC configured for IRQ 181321worker1 # [ 0.453547] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1322client # [ 0.459092] ACPI: PCI: Interrupt link GSIA configured for IRQ 161323worker2 # [ 0.465735] ACPI: PCI: Interrupt link LNKA configured for IRQ 101324lb # [ 0.474616] ACPI: PCI: Interrupt link GSID configured for IRQ 191325client # [ 0.460026] ACPI: PCI: Interrupt link GSIB configured for IRQ 171326worker2 # [ 0.466597] ACPI: PCI: Interrupt link LNKB configured for IRQ 101327lb # [ 0.475553] ACPI: PCI: Interrupt link GSIE configured for IRQ 201328worker1 # [ 0.455347] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1329client # [ 0.460984] ACPI: PCI: Interrupt link GSIC configured for IRQ 181330worker2 # [ 0.467589] ACPI: PCI: Interrupt link LNKC configured for IRQ 111331lb # [ 0.476329] ACPI: PCI: Interrupt link GSIF configured for IRQ 211332worker1 # [ 0.456188] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1333client # [ 0.461825] ACPI: PCI: Interrupt link GSID configured for IRQ 191334worker2 # [ 0.468580] ACPI: PCI: Interrupt link LNKD configured for IRQ 111335lb # [ 0.476617] ACPI: PCI: Interrupt link GSIG configured for IRQ 221336client # [ 0.462799] ACPI: PCI: Interrupt link GSIE configured for IRQ 201337worker2 # [ 0.469584] ACPI: PCI: Interrupt link LNKE configured for IRQ 101338lb # [ 0.477620] ACPI: PCI: Interrupt link GSIH configured for IRQ 231339client # [ 0.463070] ACPI: PCI: Interrupt link GSIF configured for IRQ 211340worker1 # [ 0.457320] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1341worker2 # [ 0.470605] ACPI: PCI: Interrupt link LNKF configured for IRQ 101342client # [ 0.464001] ACPI: PCI: Interrupt link GSIG configured for IRQ 221343worker2 # [ 0.471596] ACPI: PCI: Interrupt link LNKG configured for IRQ 111344lb # [ 0.479615] iommu: Default domain type: Translated1345worker1 # [ 0.459094] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1346client # [ 0.464802] ACPI: PCI: Interrupt link GSIH configured for IRQ 231347lb # [ 0.480396] iommu: DMA domain TLB invalidation policy: lazy mode1348worker2 # [ 0.472589] ACPI: PCI: Interrupt link LNKH configured for IRQ 111349lb # [ 0.481606] ACPI: bus type USB registered1350worker2 # [ 0.473579] ACPI: PCI: Interrupt link GSIA configured for IRQ 161351client # [ 0.466126] iommu: Default domain type: Translated1352lb # [ 0.482337] usbcore: registered new interface driver usbfs1353worker2 # [ 0.474523] ACPI: PCI: Interrupt link GSIB configured for IRQ 171354client # [ 0.466898] iommu: DMA domain TLB invalidation policy: lazy mode1355lb # [ 0.482623] usbcore: registered new interface driver hub1356worker2 # [ 0.474795] ACPI: PCI: Interrupt link GSIC configured for IRQ 181357client # [ 0.468030] ACPI: bus type USB registered1358worker1 # [ 0.462904] ACPI: PCI: Interrupt link LNKA configured for IRQ 101359lb # [ 0.483463] usbcore: registered new device driver usb1360worker2 # [ 0.475725] ACPI: PCI: Interrupt link GSID configured for IRQ 191361client # [ 0.468574] usbcore: registered new interface driver usbfs1362worker1 # [ 0.463425] ACPI: PCI: Interrupt link LNKB configured for IRQ 101363worker2 # [ 0.476490] ACPI: PCI: Interrupt link GSIE configured for IRQ 201364client # [ 0.469069] usbcore: registered new interface driver hub1365lb # [ 0.485051] NetLabel: Initializing1366worker1 # [ 0.464405] ACPI: PCI: Interrupt link LNKC configured for IRQ 111367client # [ 0.469925] usbcore: registered new device driver usb1368worker2 # [ 0.476794] ACPI: PCI: Interrupt link GSIF configured for IRQ 211369lb # [ 0.485605] NetLabel: domain hash size = 1281370worker1 # [ 0.465417] ACPI: PCI: Interrupt link LNKD configured for IRQ 111371worker2 # [ 0.477713] ACPI: PCI: Interrupt link GSIG configured for IRQ 221372lb # [ 0.486322] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1373client # [ 0.471560] NetLabel: Initializing1374worker1 # [ 0.466391] ACPI: PCI: Interrupt link LNKE configured for IRQ 101375worker2 # [ 0.478535] ACPI: PCI: Interrupt link GSIH configured for IRQ 231376lb # [ 0.486658] NetLabel: unlabeled traffic allowed by default1377client # [ 0.472055] NetLabel: domain hash size = 1281378worker1 # [ 0.467390] ACPI: PCI: Interrupt link LNKF configured for IRQ 101379lb # [ 0.487617] PCI: Using ACPI for IRQ routing1380client # [ 0.472759] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1381worker1 # [ 0.468393] ACPI: PCI: Interrupt link LNKG configured for IRQ 111382worker2 # [ 0.479908] iommu: Default domain type: Translated1383client # [ 0.473112] NetLabel: unlabeled traffic allowed by default1384client # [ 0.473993] PCI: Using ACPI for IRQ routing1385worker1 # [ 0.469390] ACPI: PCI: Interrupt link LNKH configured for IRQ 111386worker2 # [ 0.480677] iommu: DMA domain TLB invalidation policy: lazy mode1387worker1 # [ 0.470327] ACPI: PCI: Interrupt link GSIA configured for IRQ 161388worker2 # [ 0.481761] ACPI: bus type USB registered1389worker2 # [ 0.482321] usbcore: registered new interface driver usbfs1390worker1 # [ 0.471281] ACPI: PCI: Interrupt link GSIB configured for IRQ 171391worker2 # [ 0.482813] usbcore: registered new interface driver hub1392worker1 # [ 0.472026] ACPI: PCI: Interrupt link GSIC configured for IRQ 181393worker2 # [ 0.483667] usbcore: registered new device driver usb1394worker1 # [ 0.472307] ACPI: PCI: Interrupt link GSID configured for IRQ 191395worker1 # [ 0.473257] ACPI: PCI: Interrupt link GSIE configured for IRQ 201396worker2 # [ 0.485447] NetLabel: Initializing1397worker1 # [ 0.474042] ACPI: PCI: Interrupt link GSIF configured for IRQ 211398worker2 # [ 0.485783] NetLabel: domain hash size = 1281399worker1 # [ 0.474304] ACPI: PCI: Interrupt link GSIG configured for IRQ 221400worker2 # [ 0.486496] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1401worker1 # [ 0.475245] ACPI: PCI: Interrupt link GSIH configured for IRQ 231402worker2 # [ 0.486819] NetLabel: unlabeled traffic allowed by default1403worker2 # [ 0.487794] PCI: Using ACPI for IRQ routing1404worker1 # [ 0.477099] iommu: Default domain type: Translated1405worker1 # [ 0.477907] iommu: DMA domain TLB invalidation policy: lazy mode1406worker1 # [ 0.478545] ACPI: bus type USB registered1407worker1 # [ 0.479279] usbcore: registered new interface driver usbfs1408worker1 # [ 0.479938] usbcore: registered new interface driver hub1409worker1 # [ 0.480319] usbcore: registered new device driver usb1410worker1 # [ 0.482163] NetLabel: Initializing1411worker1 # [ 0.482723] NetLabel: domain hash size = 1281412worker1 # [ 0.483293] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413worker1 # [ 0.484330] NetLabel: unlabeled traffic allowed by default1414worker1 # [ 0.485305] PCI: Using ACPI for IRQ routing1415lb # [ 0.531570] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416client # [ 0.517297] pci 0000:00:01.0: vgaarb: setting as boot VGA device1417lb # [ 0.531600] pci 0000:00:01.0: vgaarb: bridge control possible1418client # [ 0.518051] pci 0000:00:01.0: vgaarb: bridge control possible1419lb # [ 0.531600] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1420lb # [ 0.531609] vgaarb: loaded1421client # [ 0.518051] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1422client # [ 0.518059] vgaarb: loaded1423lb # [ 0.532309] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01424client # [ 0.518774] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01425lb # [ 0.532604] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1426client # [ 0.519056] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1427lb # [ 0.535688] clocksource: Switched to clocksource kvm-clock1428client # [ 0.523137] clocksource: Switched to clocksource kvm-clock1429worker2 # [ 0.532327] pci 0000:00:01.0: vgaarb: setting as boot VGA device1430worker2 # [ 0.532778] pci 0000:00:01.0: vgaarb: bridge control possible1431lb # [ 0.539122] VFS: Disk quotas dquot_6.6.01432worker2 # [ 0.532778] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1433worker2 # [ 0.532790] vgaarb: loaded1434lb # [ 0.539789] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1435client # [ 0.526776] VFS: Disk quotas dquot_6.6.01436worker2 # [ 0.533492] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01437lb # [ 0.541100] pnp: PnP ACPI init1438client # [ 0.527484] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1439worker2 # [ 0.533784] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1440lb # [ 0.541850] ACPI: IRQ 4 override to edge(!), high(!)1441client # [ 0.528808] pnp: PnP ACPI init1442client # [ 0.529614] ACPI: IRQ 4 override to edge(!), high(!)1443lb # [ 0.542791] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1444lb # [ 0.544297] pnp: PnP ACPI: found 6 devices1445client # [ 0.530620] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1446client # [ 0.532159] pnp: PnP ACPI: found 6 devices1447worker2 # [ 0.536875] clocksource: Switched to clocksource kvm-clock1448worker1 # [ 0.529667] pci 0000:00:01.0: vgaarb: setting as boot VGA device1449worker1 # [ 0.530289] pci 0000:00:01.0: vgaarb: bridge control possible1450worker2 # [ 0.540462] VFS: Disk quotas dquot_6.6.01451worker2 # [ 0.541152] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1452worker1 # [ 0.530289] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1453worker1 # [ 0.530297] vgaarb: loaded1454worker2 # [ 0.542526] pnp: PnP ACPI init1455worker1 # [ 0.531028] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01456worker2 # [ 0.543314] ACPI: IRQ 4 override to edge(!), high(!)1457lb # [ 0.551861] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1458worker1 # [ 0.531295] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1459lb # [ 0.553290] clocksource: Switched to clocksource acpi_pm1460worker2 # [ 0.544282] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1461client # [ 0.539777] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1462lb # [ 0.554222] NET: Registered PF_INET protocol family1463worker2 # [ 0.545759] pnp: PnP ACPI: found 6 devices1464client # [ 0.541198] clocksource: Switched to clocksource acpi_pm1465client # [ 0.542145] NET: Registered PF_INET protocol family1466lb # [ 0.555357] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1467client # [ 0.543337] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1468worker1 # [ 0.535381] clocksource: Switched to clocksource kvm-clock1469worker1 # [ 0.537055] VFS: Disk quotas dquot_6.6.01470worker1 # [ 0.539519] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1471worker2 # [ 0.553353] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1472worker1 # [ 0.540851] pnp: PnP ACPI init1473worker2 # [ 0.554777] clocksource: Switched to clocksource acpi_pm1474worker1 # [ 0.541700] ACPI: IRQ 4 override to edge(!), high(!)1475worker2 # [ 0.555780] NET: Registered PF_INET protocol family1476worker1 # [ 0.542693] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1477worker2 # [ 0.557010] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1478worker1 # [ 0.544192] pnp: PnP ACPI: found 6 devices1479lb # [ 0.571323] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1480lb # [ 0.572853] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1481client # [ 0.559682] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1482lb # [ 0.574157] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1483worker1 # [ 0.551770] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1484client # [ 0.561196] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1485worker1 # [ 0.553195] clocksource: Switched to clocksource acpi_pm1486lb # [ 0.575446] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1487client # [ 0.562491] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1488worker1 # [ 0.554147] NET: Registered PF_INET protocol family1489lb # [ 0.576651] TCP: Hash tables configured (established 16384 bind 16384)1490client # [ 0.563821] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1491worker1 # [ 0.555304] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1492lb # [ 0.577812] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1493client # [ 0.565052] TCP: Hash tables configured (established 16384 bind 16384)1494lb # [ 0.579124] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1495client # [ 0.566169] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1496lb # [ 0.580236] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1497client # [ 0.567453] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1498lb # [ 0.581397] NET: Registered PF_UNIX/PF_LOCAL protocol family1499lb # [ 0.582345] NET: Registered PF_XDP protocol family1500client # [ 0.568553] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1501worker2 # [ 0.573937] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1502client # [ 0.569785] NET: Registered PF_UNIX/PF_LOCAL protocol family1503lb # [ 0.583202] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1504client # [ 0.570715] NET: Registered PF_XDP protocol family1505lb # [ 0.584173] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1506worker2 # [ 0.575471] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1507client # [ 0.571542] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1508lb # [ 0.585148] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1509worker2 # [ 0.576735] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1510client # [ 0.572534] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1511lb # [ 0.586204] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1512client # [ 0.573496] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1513worker2 # [ 0.578049] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1514lb # [ 0.587262] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1515client # [ 0.574558] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1516worker2 # [ 0.579285] TCP: Hash tables configured (established 16384 bind 16384)1517lb # [ 0.588317] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1518client # [ 0.575641] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1519worker2 # [ 0.580390] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1520lb # [ 0.589984] ACPI: \_SB_.GSIA: Enabled at IRQ 161521client # [ 0.576778] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1522worker2 # [ 0.581697] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1523client # [ 0.578455] ACPI: \_SB_.GSIA: Enabled at IRQ 161524worker2 # [ 0.582793] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1525lb # [ 0.592388] ACPI: \_SB_.GSIB: Enabled at IRQ 171526worker2 # [ 0.584017] NET: Registered PF_UNIX/PF_LOCAL protocol family1527worker2 # [ 0.585031] NET: Registered PF_XDP protocol family1528lb # [ 0.594377] ACPI: \_SB_.GSIC: Enabled at IRQ 181529client # [ 0.580947] ACPI: \_SB_.GSIB: Enabled at IRQ 171530worker1 # [ 0.571590] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1531worker2 # [ 0.585921] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1532worker2 # [ 0.586897] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1533worker1 # [ 0.573040] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1534lb # [ 0.596274] ACPI: \_SB_.GSID: Enabled at IRQ 191535client # [ 0.583060] ACPI: \_SB_.GSIC: Enabled at IRQ 181536worker2 # [ 0.587890] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1537worker1 # [ 0.574309] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1538lb # [ 0.597917] PCI: CLS 0 bytes, default 641539worker2 # [ 0.588936] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1540worker1 # [ 0.575630] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1541client # [ 0.585074] ACPI: \_SB_.GSID: Enabled at IRQ 191542worker2 # [ 0.589980] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1543worker1 # [ 0.576814] TCP: Hash tables configured (established 16384 bind 16384)1544lb # [ 0.598760] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1545client # [ 0.586839] PCI: CLS 0 bytes, default 641546worker2 # [ 0.591022] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1547lb # [ 0.600498] Trying to unpack rootfs image as initramfs...1548worker1 # [ 0.577943] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1549worker2 # [ 0.592803] ACPI: \_SB_.GSIA: Enabled at IRQ 161550worker1 # [ 0.579246] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1551client # [ 0.587688] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1552client # [ 0.589375] Trying to unpack rootfs image as initramfs...1553worker1 # [ 0.580376] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1554worker2 # [ 0.595036] ACPI: \_SB_.GSIB: Enabled at IRQ 171555worker1 # [ 0.581595] NET: Registered PF_UNIX/PF_LOCAL protocol family1556worker1 # [ 0.582511] NET: Registered PF_XDP protocol family1557worker1 # [ 0.583320] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1558worker2 # [ 0.597422] ACPI: \_SB_.GSIC: Enabled at IRQ 181559worker1 # [ 0.584284] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1560worker1 # [ 0.585248] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1561worker2 # [ 0.599469] ACPI: \_SB_.GSID: Enabled at IRQ 191562worker1 # [ 0.586312] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1563worker2 # [ 0.601259] PCI: CLS 0 bytes, default 641564worker1 # [ 0.587365] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1565worker1 # [ 0.588413] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1566worker2 # [ 0.602168] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1567worker1 # [ 0.590116] ACPI: \_SB_.GSIA: Enabled at IRQ 161568worker2 # [ 0.603944] Trying to unpack rootfs image as initramfs...1569worker1 # [ 0.592533] ACPI: \_SB_.GSIB: Enabled at IRQ 171570worker1 # [ 0.594414] ACPI: \_SB_.GSIC: Enabled at IRQ 181571worker1 # [ 0.596331] ACPI: \_SB_.GSID: Enabled at IRQ 191572worker1 # [ 0.597941] PCI: CLS 0 bytes, default 641573worker1 # [ 0.598791] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1574worker1 # [ 0.600441] Trying to unpack rootfs image as initramfs...1575lb # [ 0.643238] Initialise system trusted keyrings1576client # [ 0.633305] Initialise system trusted keyrings1577lb # [ 0.646888] workingset: timestamp_bits=40 max_order=19 bucket_order=01578client # [ 0.636324] workingset: timestamp_bits=40 max_order=19 bucket_order=01579worker2 # [ 0.648166] Initialise system trusted keyrings1580worker2 # [ 0.651291] workingset: timestamp_bits=40 max_order=19 bucket_order=01581lb # [ 0.669071] Key type asymmetric registered1582worker1 # [ 0.646651] Initialise system trusted keyrings1583lb # [ 0.669784] Asymmetric key parser 'x509' registered1584worker1 # [ 0.647582] workingset: timestamp_bits=40 max_order=19 bucket_order=01585lb # [ 0.670640] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1586client # [ 0.658384] Key type asymmetric registered1587client # [ 0.659110] Asymmetric key parser 'x509' registered1588lb # [ 0.673974] io scheduler mq-deadline registered1589client # [ 0.659997] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1590lb # [ 0.674721] io scheduler kyber registered1591client # [ 0.665424] io scheduler mq-deadline registered1592lb # [ 0.678877] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1593client # [ 0.666178] io scheduler kyber registered1594lb # [ 0.680150] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1595worker2 # [ 0.672468] Key type asymmetric registered1596client # [ 0.669827] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1597worker2 # [ 0.675271] Asymmetric key parser 'x509' registered1598client # [ 0.671128] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1599worker2 # [ 0.676138] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1600lb # [ 0.685634] Linux agpgart interface v0.1031601lb # [ 0.686410] ACPI: bus type drm_connector registered1602worker2 # [ 0.679399] io scheduler mq-deadline registered1603client # [ 0.675049] Linux agpgart interface v0.1031604worker2 # [ 0.680186] io scheduler kyber registered1605client # [ 0.675803] ACPI: bus type drm_connector registered1606lb # [ 0.689321] usbcore: registered new interface driver usbserial_generic1607lb # [ 0.690393] usbserial: USB Serial support registered for generic1608worker1 # [ 0.669261] Key type asymmetric registered1609worker2 # [ 0.682793] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1610worker1 # [ 0.669952] Asymmetric key parser 'x509' registered1611client # [ 0.678770] usbcore: registered new interface driver usbserial_generic1612worker2 # [ 0.684097] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1613client # [ 0.679839] usbserial: USB Serial support registered for generic1614worker1 # [ 0.670793] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1615lb # [ 0.693856] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1616lb # [ 0.693856] Please enable it if your BIOS has the CPPC option.1617client # [ 0.682288] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1618lb # [ 0.696152] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1619client # [ 0.682288] Please enable it if your BIOS has the CPPC option.1620worker2 # [ 0.689623] Linux agpgart interface v0.1031621client # [ 0.684517] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1622worker1 # [ 0.676361] io scheduler mq-deadline registered1623worker2 # [ 0.690401] ACPI: bus type drm_connector registered1624worker1 # [ 0.677095] io scheduler kyber registered1625lb # [ 0.699965] drop_monitor: Initializing network drop monitor service1626lb # [ 0.701231] NET: Registered PF_INET6 protocol family1627client # [ 0.688413] drop_monitor: Initializing network drop monitor service1628worker1 # [ 0.679809] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1629worker2 # [ 0.693671] usbcore: registered new interface driver usbserial_generic1630client # [ 0.689582] NET: Registered PF_INET6 protocol family1631worker2 # [ 0.694721] usbserial: USB Serial support registered for generic1632worker1 # [ 0.681033] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1633lb # [ 0.704356] Segment Routing with IPv61634client # [ 0.692759] Segment Routing with IPv61635lb # [ 0.706871] In-situ OAM (IOAM) with IPv61636worker1 # [ 0.685035] Linux agpgart interface v0.1031637worker2 # [ 0.698260] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1638lb # [ 0.707961] IPI shorthand broadcast: enabled1639worker1 # [ 0.685778] ACPI: bus type drm_connector registered1640worker2 # [ 0.698260] Please enable it if your BIOS has the CPPC option.1641client # [ 0.695309] In-situ OAM (IOAM) with IPv61642worker2 # [ 0.700413] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1643client # [ 0.696353] IPI shorthand broadcast: enabled1644worker2 # [ 0.701758] drop_monitor: Initializing network drop monitor service1645worker2 # [ 0.702905] NET: Registered PF_INET6 protocol family1646worker1 # [ 0.690708] usbcore: registered new interface driver usbserial_generic1647worker1 # [ 0.691764] usbserial: USB Serial support registered for generic1648worker2 # [ 0.707718] Segment Routing with IPv61649lb # [ 0.716118] sched_clock: Marking stable (582013490, 133823491)->(787491132, -71654151)1650worker2 # [ 0.708446] In-situ OAM (IOAM) with IPv61651worker1 # [ 0.694234] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1652worker1 # [ 0.694234] Please enable it if your BIOS has the CPPC option.1653client # [ 0.704896] sched_clock: Marking stable (571014915, 133255474)->(777136254, -72865865)1654worker1 # [ 0.696361] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1655worker2 # [ 0.711540] IPI shorthand broadcast: enabled1656lb # [ 0.721094] registered taskstats version 11657lb # [ 0.722112] Loading compiled-in X.509 certificates1658worker1 # [ 0.700446] drop_monitor: Initializing network drop monitor service1659client # [ 0.710438] registered taskstats version 11660worker1 # [ 0.701691] NET: Registered PF_INET6 protocol family1661client # [ 0.711398] Loading compiled-in X.509 certificates1662worker1 # [ 0.704713] Segment Routing with IPv61663worker2 # [ 0.719818] sched_clock: Marking stable (584015336, 135222654)->(787649232, -68411242)1664worker1 # [ 0.707249] In-situ OAM (IOAM) with IPv61665worker1 # [ 0.708292] IPI shorthand broadcast: enabled1666worker2 # [ 0.722342] registered taskstats version 11667worker2 # [ 0.723284] Loading compiled-in X.509 certificates1668worker1 # [ 0.715533] sched_clock: Marking stable (582014666, 133200594)->(789283834, -74068574)1669lb # [ 0.740856] Demotion targets for Node 0: null1670lb # [ 0.741773] Key type .fscrypt registered1671lb # [ 0.742490] Key type fscrypt-provisioning registered1672lb # [ 0.743454] ima: No TPM chip found, activating TPM-bypass!1673client # [ 0.730286] Demotion targets for Node 0: null1674worker1 # [ 0.721406] registered taskstats version 11675client # [ 0.731133] Key type .fscrypt registered1676worker1 # [ 0.722342] Loading compiled-in X.509 certificates1677client # [ 0.731823] Key type fscrypt-provisioning registered1678lb # [ 0.747858] ima: Allocated hash algorithm: sha11679client # [ 0.734386] ima: No TPM chip found, activating TPM-bypass!1680lb # [ 0.748634] ima: No architecture policies found1681client # [ 0.735319] ima: Allocated hash algorithm: sha11682lb # [ 0.749580] PM: Magic number: 10:660:3781683client # [ 0.736100] ima: No architecture policies found1684worker2 # [ 0.742261] Demotion targets for Node 0: null1685worker2 # [ 0.743093] Key type .fscrypt registered1686worker2 # [ 0.743759] Key type fscrypt-provisioning registered1687lb # [ 0.753243] RAS: Correctable Errors collector initialized.1688worker2 # [ 0.744674] ima: No TPM chip found, activating TPM-bypass!1689client # [ 0.740457] PM: Magic number: 10:660:3781690client # [ 0.742000] RAS: Correctable Errors collector initialized.1691worker2 # [ 0.749259] ima: Allocated hash algorithm: sha11692worker2 # [ 0.750009] ima: No architecture policies found1693worker2 # [ 0.750951] PM: Magic number: 10:660:3781694worker1 # [ 0.738710] Demotion targets for Node 0: null1695lb # [ 0.761926] clk: Disabling unused clocks1696lb # [ 0.762625] PM: genpd: Disabling unused power domains1697client # [ 0.749905] clk: Disabling unused clocks1698worker2 # [ 0.754547] RAS: Correctable Errors collector initialized.1699worker1 # [ 0.742268] Key type .fscrypt registered1700worker1 # [ 0.742921] Key type fscrypt-provisioning registered1701worker1 # [ 0.743863] ima: No TPM chip found, activating TPM-bypass!1702client # [ 0.754288] PM: genpd: Disabling unused power domains1703worker1 # [ 0.746234] ima: Allocated hash algorithm: sha11704worker1 # [ 0.747049] ima: No architecture policies found1705worker1 # [ 0.749230] PM: Magic number: 10:660:3781706worker2 # [ 0.763097] clk: Disabling unused clocks1707worker1 # [ 0.750791] RAS: Correctable Errors collector initialized.1708worker2 # [ 0.765249] PM: genpd: Disabling unused power domains1709worker1 # [ 0.760875] clk: Disabling unused clocks1710worker1 # [ 0.763234] PM: genpd: Disabling unused power domains1711lb # [ 0.926223] Freeing initrd memory: 29092K1712client # [ 0.913033] Freeing initrd memory: 29096K1713lb # [ 0.929331] Freeing unused decrypted memory: 2028K1714client # [ 0.916103] Freeing unused decrypted memory: 2028K1715lb # [ 0.931927] Freeing unused kernel image (initmem) memory: 3644K1716client # [ 0.918672] Freeing unused kernel image (initmem) memory: 3644K1717lb # [ 0.932973] Write protecting the kernel read-only data: 32768k1718client # [ 0.919749] Write protecting the kernel read-only data: 32768k1719lb # [ 0.934810] Freeing unused kernel image (text/rodata gap) memory: 1208K1720client # [ 0.921605] Freeing unused kernel image (text/rodata gap) memory: 1208K1721lb # [ 0.936336] Freeing unused kernel image (rodata/data gap) memory: 752K1722client # [ 0.923125] Freeing unused kernel image (rodata/data gap) memory: 752K1723worker2 # [ 0.932096] Freeing initrd memory: 29092K1724worker2 # [ 0.935274] Freeing unused decrypted memory: 2028K1725worker2 # [ 0.937813] Freeing unused kernel image (initmem) memory: 3644K1726worker2 # [ 0.938877] Write protecting the kernel read-only data: 32768k1727worker2 # [ 0.940690] Freeing unused kernel image (text/rodata gap) memory: 1208K1728worker1 # [ 0.928587] Freeing initrd memory: 29092K1729worker2 # [ 0.942160] Freeing unused kernel image (rodata/data gap) memory: 752K1730worker1 # [ 0.931676] Freeing unused decrypted memory: 2028K1731worker1 # [ 0.934242] Freeing unused kernel image (initmem) memory: 3644K1732worker1 # [ 0.935356] Write protecting the kernel read-only data: 32768k1733worker1 # [ 0.937238] Freeing unused kernel image (text/rodata gap) memory: 1208K1734worker1 # [ 0.938750] Freeing unused kernel image (rodata/data gap) memory: 752K1735lb # [ 0.987325] x86/mm: Checked W+X mappings: passed, no W+X pages found.1736client # [ 0.974101] x86/mm: Checked W+X mappings: passed, no W+X pages found.1737lb # [ 0.988424] Run /init as init process1738client # [ 0.975143] Run /init as init process1739client # [ 0.984996] systemd[1]: Inserted module 'autofs4'1740lb # [ 0.998777] systemd[1]: Inserted module 'autofs4'1741worker2 # [ 0.993585] x86/mm: Checked W+X mappings: passed, no W+X pages found.1742worker2 # [ 0.994627] Run /init as init process1743worker1 # [ 0.989992] x86/mm: Checked W+X mappings: passed, no W+X pages found.1744worker1 # [ 0.991082] Run /init as init process1745worker2 # [ 1.004775] systemd[1]: Inserted module 'autofs4'1746client # [ 1.001163] fuse: init (API version 7.45)1747lb # [ 1.014980] fuse: init (API version 7.45)1748client # [ 1.006995] ACPI: \_SB_.GSIG: Enabled at IRQ 221749lb # [ 1.020684] ACPI: \_SB_.GSIG: Enabled at IRQ 221750client # [ 1.009385] ACPI: \_SB_.GSIH: Enabled at IRQ 231751lb # [ 1.023076] ACPI: \_SB_.GSIH: Enabled at IRQ 231752worker1 # [ 1.001000] systemd[1]: Inserted module 'autofs4'1753client # [ 1.012178] ACPI: \_SB_.GSIE: Enabled at IRQ 201754lb # [ 1.026171] ACPI: \_SB_.GSIE: Enabled at IRQ 201755client # [ 1.014396] ACPI: \_SB_.GSIF: Enabled at IRQ 211756lb # [ 1.028375] ACPI: \_SB_.GSIF: Enabled at IRQ 211757worker2 # [ 1.020919] fuse: init (API version 7.45)1758worker2 # [ 1.026733] ACPI: \_SB_.GSIG: Enabled at IRQ 221759worker2 # [ 1.029133] ACPI: \_SB_.GSIH: Enabled at IRQ 231760worker1 # [ 1.017187] fuse: init (API version 7.45)1761worker2 # [ 1.031994] ACPI: \_SB_.GSIE: Enabled at IRQ 201762worker2 # [ 1.034171] ACPI: \_SB_.GSIF: Enabled at IRQ 211763worker1 # [ 1.022872] ACPI: \_SB_.GSIG: Enabled at IRQ 221764worker1 # [ 1.025273] ACPI: \_SB_.GSIH: Enabled at IRQ 231765worker1 # [ 1.028265] ACPI: \_SB_.GSIE: Enabled at IRQ 201766worker1 # [ 1.030446] ACPI: \_SB_.GSIF: Enabled at IRQ 211767client # [ 1.042469] systemd[1]: Successfully made /usr/ read-only.1768lb # [ 1.056030] systemd[1]: Successfully made /usr/ read-only.1769worker2 # [ 1.060035] systemd[1]: Successfully made /usr/ read-only.1770worker1 # [ 1.057786] systemd[1]: Successfully made /usr/ read-only.1771lb # [ 1.391712] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1772lb # [ 1.396787] systemd[1]: Detected virtualization kvm.1773lb # [ 1.397599] systemd[1]: Detected architecture x86-64.1774lb # [ 1.398482] systemd[1]: Running in initrd.1775lb # [ 1.399717] systemd[1]: Initializing machine ID from random generator.1776lb # [ 1.401027] systemd[1]: Hostname set to <lb>.1777client # [ 1.379169] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1778client # [ 1.392951] systemd[1]: Detected virtualization kvm.1779client # [ 1.395077] systemd[1]: Detected architecture x86-64.1780client # [ 1.397198] systemd[1]: Running in initrd.1781client # [ 1.399892] systemd[1]: Initializing machine ID from random generator.1782worker2 # [ 1.396568] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1783client # [ 1.402864] systemd[1]: Hostname set to <client>.1784worker2 # [ 1.408537] systemd[1]: Detected virtualization kvm.1785worker1 # [ 1.393455] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1786worker2 # [ 1.410637] systemd[1]: Detected architecture x86-64.1787worker1 # [ 1.398466] systemd[1]: Detected virtualization kvm.1788worker1 # [ 1.399324] systemd[1]: Detected architecture x86-64.1789worker2 # [ 1.412724] systemd[1]: Running in initrd.1790worker1 # [ 1.400189] systemd[1]: Running in initrd.1791worker1 # [ 1.401465] systemd[1]: Initializing machine ID from random generator.1792worker1 # [ 1.402704] systemd[1]: Hostname set to <worker1>.1793worker2 # [ 1.415218] systemd[1]: Initializing machine ID from random generator.1794worker2 # [ 1.418069] systemd[1]: Hostname set to <worker2>.1795lb # [ 1.595347] systemd[1]: bpf-restrict-fs: LSM BPF program attached1796worker1 # [ 1.595818] systemd[1]: bpf-restrict-fs: LSM BPF program attached1797lb # [ 1.636027] systemd[1]: Queued start job for default target Initrd Default Target.1798worker2 # [ 1.629457] systemd[1]: bpf-restrict-fs: LSM BPF program attached1799client # [ 1.624537] systemd[1]: bpf-restrict-fs: LSM BPF program attached1800lb # [ 1.640227] systemd[1]: Created slice Slice /system/modprobe.1801lb # [ 1.641383] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1802lb # [ 1.642764] systemd[1]: Expecting device /dev/disk/by-label/nixos...1803lb # [ 1.643808] systemd[1]: Reached target Path Units.1804lb # [ 1.644702] systemd[1]: Reached target Slice Units.1805lb # [ 1.645535] systemd[1]: Reached target Swaps.1806lb # [ 1.646327] systemd[1]: Reached target Timer Units.1807lb # [ 1.647316] systemd[1]: Listening on D-Bus System Message Bus Socket.1808lb # [ 1.648487] systemd[1]: Listening on Journal Socket (/dev/log).1809lb # [ 1.649603] systemd[1]: Listening on Journal Sockets.1810lb # [ 1.650582] systemd[1]: Listening on udev Control Socket.1811lb # [ 1.651580] systemd[1]: Listening on udev Kernel Socket.1812lb # [ 1.652493] systemd[1]: Reached target Socket Units.1813lb # [ 1.654361] systemd[1]: Starting Create List of Static Device Nodes...1814lb # [ 1.658908] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1815worker1 # [ 1.636492] systemd[1]: Queued start job for default target Initrd Default Target.1816worker1 # [ 1.640619] systemd[1]: Created slice Slice /system/modprobe.1817worker1 # [ 1.641839] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1818worker1 # [ 1.643269] systemd[1]: Expecting device /dev/disk/by-label/nixos...1819lb # [ 1.666102] systemd[1]: Starting Load Kernel Module configfs...1820worker1 # [ 1.644384] systemd[1]: Reached target Path Units.1821worker1 # [ 1.645294] systemd[1]: Reached target Slice Units.1822worker1 # [ 1.646183] systemd[1]: Reached target Swaps.1823worker1 # [ 1.647006] systemd[1]: Reached target Timer Units.1824worker1 # [ 1.648090] systemd[1]: Listening on D-Bus System Message Bus Socket.1825worker1 # [ 1.649349] systemd[1]: Listening on Journal Socket (/dev/log).1826worker1 # [ 1.650524] systemd[1]: Listening on Journal Sockets.1827worker1 # [ 1.651573] systemd[1]: Listening on udev Control Socket.1828worker1 # [ 1.652625] systemd[1]: Listening on udev Kernel Socket.1829worker1 # [ 1.653593] systemd[1]: Reached target Socket Units.1830worker2 # [ 1.668468] systemd[1]: Queued start job for default target Initrd Default Target.1831worker1 # [ 1.655478] systemd[1]: Starting Create List of Static Device Nodes...1832worker2 # [ 1.672514] systemd[1]: Created slice Slice /system/modprobe.1833lb # [ 1.681532] systemd[1]: Starting Journal Service...1834worker1 # [ 1.659090] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1835worker2 # [ 1.673694] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1836worker2 # [ 1.675130] systemd[1]: Expecting device /dev/disk/by-label/nixos...1837worker2 # [ 1.676287] systemd[1]: Reached target Path Units.1838worker2 # [ 1.677135] systemd[1]: Reached target Slice Units.1839worker2 # [ 1.678026] systemd[1]: Reached target Swaps.1840worker1 # [ 1.664291] systemd[1]: Starting Load Kernel Module configfs...1841worker2 # [ 1.678837] systemd[1]: Reached target Timer Units.1842worker2 # [ 1.679889] systemd[1]: Listening on D-Bus System Message Bus Socket.1843worker2 # [ 1.681149] systemd[1]: Listening on Journal Socket (/dev/log).1844worker2 # [ 1.682391] systemd[1]: Listening on Journal Sockets.1845worker2 # [ 1.683438] systemd[1]: Listening on udev Control Socket.1846client # [ 1.679308] systemd[1]: Queued start job for default target Initrd Default Target.1847worker2 # [ 1.684497] systemd[1]: Listening on udev Kernel Socket.1848worker2 # [ 1.685481] systemd[1]: Reached target Socket Units.1849lb # [ 1.694931] systemd[1]: Starting Load Kernel Modules...1850client # [ 1.682737] systemd[1]: Created slice Slice /system/modprobe.1851worker2 # [ 1.687305] systemd[1]: Starting Create List of Static Device Nodes...1852client # [ 1.683875] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1853client # [ 1.685211] systemd[1]: Expecting device /dev/disk/by-label/nixos...1854client # [ 1.686491] systemd[1]: Reached target Path Units.1855worker2 # [ 1.691088] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1856worker1 # [ 1.678280] systemd[1]: Starting Journal Service...1857client # [ 1.687402] systemd[1]: Reached target Slice Units.1858client # [ 1.688304] systemd[1]: Reached target Swaps.1859client # [ 1.689101] systemd[1]: Reached target Timer Units.1860client # [ 1.690106] systemd[1]: Listening on D-Bus System Message Bus Socket.1861client # [ 1.691377] systemd[1]: Listening on Journal Socket (/dev/log).1862client # [ 1.692588] systemd[1]: Listening on Journal Sockets.1863worker2 # [ 1.697793] systemd[1]: Starting Load Kernel Module configfs...1864client # [ 1.693625] systemd[1]: Listening on udev Control Socket.1865client # [ 1.694663] systemd[1]: Listening on udev Kernel Socket.1866client # [ 1.695662] systemd[1]: Reached target Socket Units.1867client # [ 1.697727] systemd[1]: Starting Create List of Static Device Nodes...1868worker1 # [ 1.691372] systemd[1]: Starting Load Kernel Modules...1869lb # [ 1.701889] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1870client # [ 1.702339] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1871lb # [ 1.718327] netfs: FS-Cache loaded1872worker2 # [ 1.711421] systemd[1]: Starting Journal Service...1873client # [ 1.709016] systemd[1]: Starting Load Kernel Module configfs...1874lb # [ 1.723279] 9pnet: Installing 9P2000 support1875worker1 # [ 1.699310] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1876lb # [ 1.733920] systemd[1]: Starting Coldplug All udev Devices...1877worker2 # [ 1.725310] systemd[1]: Starting Load Kernel Modules...1878client # [ 1.724972] systemd[1]: Starting Journal Service...1879worker1 # [ 1.718085] netfs: FS-Cache loaded1880worker1 # [ 1.722982] 9pnet: Installing 9P2000 support1881lb # [ 1.748981] systemd[1]: Finished Create List of Static Device Nodes.1882worker2 # [ 1.730266] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1883client # [ 1.739423] systemd[1]: Starting Load Kernel Modules...1884lb # [ 1.755515] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1885worker1 # [ 1.733537] systemd[1]: Starting Coldplug All udev Devices...1886worker2 # [ 1.748108] netfs: FS-Cache loaded1887worker2 # [ 1.752761] 9pnet: Installing 9P2000 support1888lb # [ 1.763292] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1889lb # [ 1.768597] systemd-journald[67]: Collecting audit messages is disabled.1890client # [ 1.744355] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1891worker2 # [ 1.761486] systemd[1]: Starting Coldplug All udev Devices...1892lb # [ 1.770202] systemd[1]: modprobe@configfs.service: Deactivated successfully.1893worker1 # [ 1.750628] systemd[1]: Finished Create List of Static Device Nodes.1894client # [ 1.762274] netfs: FS-Cache loaded1895lb # [ 1.776221] systemd[1]: Finished Load Kernel Module configfs.1896worker1 # [ 1.754918] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1897client # [ 1.767127] 9pnet: Installing 9P2000 support1898lb # [ 1.783324] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1899worker2 # [ 1.775364] systemd[1]: Finished Create List of Static Device Nodes.1900worker1 # [ 1.763298] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1901client # [ 1.775542] systemd[1]: Starting Coldplug All udev Devices...1902lb # [ 1.788393] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1903worker2 # [ 1.781806] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1904worker1 # [ 1.770579] systemd-journald[67]: Collecting audit messages is disabled.1905lb # [ 1.793731] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1906worker1 # [ 1.772271] systemd[1]: modprobe@configfs.service: Deactivated successfully.1907worker2 # [ 1.788900] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1908lb # [ 1.797866] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1909worker1 # [ 1.777457] systemd[1]: Finished Load Kernel Module configfs.1910worker1 # [ 1.781754] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1911worker2 # [ 1.796656] systemd[1]: modprobe@configfs.service: Deactivated successfully.1912client # [ 1.792365] systemd[1]: Finished Create List of Static Device Nodes.1913worker2 # [ 1.798415] systemd-journald[67]: Collecting audit messages is disabled.1914worker1 # [ 1.786663] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1915worker2 # [ 1.801580] systemd[1]: Finished Load Kernel Module configfs.1916client # [ 1.797187] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1917worker1 # [ 1.794253] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1918worker2 # [ 1.807681] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1919worker1 # [ 1.796025] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1920client # [ 1.805815] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1921worker2 # [ 1.811034] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1922client # [ 1.812709] systemd[1]: modprobe@configfs.service: Deactivated successfully.1923worker2 # [ 1.819317] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1924client # [ 1.815303] systemd-journald[67]: Collecting audit messages is disabled.1925lb # [ 1.829373] systemd[1]: Finished Load Kernel Modules.1926worker2 # [ 1.820791] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1927client # [ 1.819675] systemd[1]: Finished Load Kernel Module configfs.1928lb # [ 1.834960] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1929client # [ 1.823850] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1930client # [ 1.828536] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1931lb # [ 1.843070] systemd[1]: Starting Apply Kernel Variables...1932worker1 # [ 1.825747] systemd[1]: Finished Load Kernel Modules.1933client # [ 1.837525] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1934client # [ 1.839048] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1935lb # [ 1.854917] systemd[1]: Starting Create Static Device Nodes in /dev...1936worker1 # [ 1.833494] systemd[1]: Starting Apply Kernel Variables...1937worker2 # [ 1.850769] systemd[1]: Finished Load Kernel Modules.1938worker2 # [ 1.858198] systemd[1]: Starting Apply Kernel Variables...1939worker1 # [ 1.848287] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1940worker2 # [ 1.868345] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1941worker1 # [ 1.857443] systemd[1]: Starting Create Static Device Nodes in /dev...1942client # [ 1.870832] systemd[1]: Finished Load Kernel Modules.1943lb # [ 1.886429] systemd[1]: Finished Apply Kernel Variables.1944worker2 # [ 1.878481] systemd[1]: Starting Create Static Device Nodes in /dev...1945lb # [ 1.890500] systemd[1]: Started Journal Service.1946lb # [ 1.754676] systemd-modules-load[68]: Inserted module 'dm_mod'1947lb # [ 1.758472] systemd-modules-load[68]: Inserted module 'virtio_balloon'1948client # [ 1.879527] systemd[1]: Starting Apply Kernel Variables...1949lb # [ 1.759566] systemd-modules-load[68]: Inserted module 'virtio_gpu'1950worker1 # [ 1.871363] systemd[1]: Finished Apply Kernel Variables.1951lb # [ 1.764882] systemd[1]: Finished Create Static Device Nodes in /dev.1952client # [ 1.890809] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1953lb # [ 1.771683] systemd[1]: Reached target Preparation for Local File Systems.1954worker2 # [ 1.897361] systemd[1]: Finished Apply Kernel Variables.1955lb # [ 1.773356] systemd[1]: Reached target Local File Systems.1956lb # [ 1.776066] systemd[1]: Starting Create System Files and Directories...1957worker1 # [ 1.888759] systemd[1]: Finished Create Static Device Nodes in /dev.1958worker1 # [ 1.892615] systemd[1]: Started Journal Service.1959client # [ 1.901472] systemd[1]: Starting Create Static Device Nodes in /dev...1960lb # [ 1.781486] systemd[1]: Starting Rule-based Manager for Device Events and Files...1961worker1 # [ 1.758771] systemd-modules-load[68]: Inserted module 'dm_mod'1962worker1 # [ 1.762216] systemd-modules-load[68]: Inserted module 'virtio_balloon'1963worker1 # [ 1.764349] systemd-modules-load[68]: Inserted module 'virtio_gpu'1964worker2 # [ 1.776536] systemd-modules-load[68]: Inserted module 'dm_mod'[ 1.912934] systemd[1]: Started Journal Service.1965worker2 # 1966worker1 # [ 1.766383] systemd[1]: Reached target Preparation for Local File Systems.1967worker2 # [ 1.778608] systemd-modules-load[68]: Inserted module 'virtio_balloon'1968worker1 # [ 1.768419] systemd[1]: Reached target Local File Systems.1969worker2 # [ 1.781321] systemd-modules-load[68]: Inserted module 'virtio_gpu'1970worker1 # [ 1.770434] systemd[1]: Starting Create System Files and Directories...1971client # [ 1.914920] systemd[1]: Finished Apply Kernel Variables.1972worker2 # [ 1.784785] systemd[1]: Finished Create Static Device Nodes in /dev.1973worker1 # [ 1.778073] systemd[1]: Starting Rule-based Manager for Device Events and Files...1974worker2 # [ 1.792124] systemd[1]: Reached target Preparation for Local File Systems.1975worker2 # [ 1.793149] systemd[1]: Reached target Local File Systems.1976client # [ 1.926387] systemd[1]: Started Journal Service.1977worker2 # [ 1.796091] systemd[1]: Starting Create System Files and Directories...1978lb # [ 1.807958] systemd[1]: Finished Create System Files and Directories.1979client # [ 1.796222] systemd-modules-load[68]: Inserted module 'dm_mod'1980worker2 # [ 1.801072] systemd[1]: Starting Rule-based Manager for Device Events and Files...1981client # [ 1.800109] systemd-modules-load[68]: Inserted module 'virtio_balloon'1982client # [ 1.801193] systemd-modules-load[68]: Inserted module 'virtio_gpu'1983client # [ 1.810682] systemd[1]: Finished Create Static Device Nodes in /dev.1984client # [ 1.814076] systemd[1]: Reached target Preparation for Local File Systems.1985client # [ 1.815069] systemd[1]: Reached target Local File Systems.1986worker1 # [ 1.808166] systemd[1]: Finished Create System Files and Directories.1987client # [ 1.818148] systemd[1]: Starting Create System Files and Directories...1988lb # [ 1.833276] systemd-udevd[84]: Using default interface naming scheme 'v261'.1989client # [ 1.822805] systemd[1]: Starting Rule-based Manager for Device Events and Files...1990worker2 # [ 1.826285] systemd[1]: Finished Create System Files and Directories.1991lb # [ 1.857316] systemd[1]: Started Rule-based Manager for Device Events and Files.1992worker2 # [ 1.848716] systemd-udevd[83]: Using default interface naming scheme 'v261'.1993worker1 # [ 1.837313] systemd-udevd[76]: Using default interface naming scheme 'v261'.1994client # [ 1.849183] systemd[1]: Finished Create System Files and Directories.1995worker1 # [ 1.861074] systemd[1]: Started Rule-based Manager for Device Events and Files.1996worker2 # [ 1.872774] systemd[1]: Started Rule-based Manager for Device Events and Files.1997client # [ 1.874350] systemd-udevd[83]: Using default interface naming scheme 'v261'.1998lb # [ 1.903195] systemd[1]: Finished Coldplug All udev Devices.1999lb # [ 1.905124] systemd[1]: Reached target System Initialization.2000lb # [ 1.905909] systemd[1]: Reached target Basic System.
loading 2,951 hidden lines…
4952worker1 # [ 55.334848] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4953worker1 # [ 55.505502] nix-daemon[855]: accepted connection from pid 998, user nix-grpc-daemon (trusted)4954worker1 # [ 55.517891] nix-daemon[855]: reaped child process 1001, status = succeeded4955worker1 # [ 55.660865] nix-daemon[855]: accepted connection from pid 1003, user nix-grpc-daemon (trusted)4956worker1 # [ 55.673446] nix-daemon[855]: reaped child process 1006, status = succeeded4957lb # [ 55.709826] niks3-server[1047]: 2026/09/22 11:22:19 INFO Received uploads request method=POST path=/api/pending_closures4958worker1 # [ 55.695960] nix-grpc-daemon[867]: time=2026-09-22T11:22:18.928Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4959worker1 # [ 55.697500] nix-grpc-daemon[867]: time=2026-09-22T11:22:18.930Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4960lb # [ 55.738238] niks3-server[1047]: 2026/09/22 11:22:19 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4961lb # [ 55.749834] niks3-server[1047]: 2026/09/22 11:22:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4962lb # [ 55.756501] niks3-server[1047]: 2026/09/22 11:22:19 INFO Signed narinfos id=7 count=14963worker1 # [ 55.737645] nix-grpc-daemon[867]: time=2026-09-22T11:22:18.970Z level=INFO msg="Uploading 1 narinfos"4964lb # [ 55.760108] niks3-server[1047]: 2026/09/22 11:22:19 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4965lb # [ 55.773297] niks3-server[1047]: 2026/09/22 11:22:19 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4966worker1 # [ 55.757990] nix-grpc-daemon[867]: time=2026-09-22T11:22:18.990Z level=INFO msg="Upload complete. (410ms)"4967lb # [ 55.779802] niks3-server[1047]: 2026/09/22 11:22:19 INFO Completed upload id=74968worker1 # [ 55.760520] nix-grpc-daemon[786]: ts=2026-09-22T11:22:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=0 paths=1 nar_bytes_in=7524969lb # [ 55.782948] niks3-server[1047]: 2026/09/22 11:22:19 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4970worker1 # [ 55.765239] nix-grpc-daemon[786]: ts=2026-09-22T11:22:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361344971worker1 # [ 55.770542] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4972worker1 # [ 55.772743] nix-daemon[855]: reaped child process 996, status = succeeded4973worker1 # [ 55.777913] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4974client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4975client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4976client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4977lb # [ 56.778750] envoy[899]: conn peer=[2001:db8:1::1]:36196 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9550 tx=4426 ms=77334978lb # [ 56.782597] envoy[899]: conn peer=[2001:db8:1::1]:57098 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=664979client # farm-top> LOG-farm-top-t1b4980worker1 # [ 57.004948] nix-daemon[855]: reaped child process 1010, status = succeeded4981worker1 # [ 57.146252] nix-daemon[855]: accepted connection from pid 1026, user nix-grpc-daemon (trusted)4982worker1 # [ 57.157925] nix-daemon[855]: reaped child process 1029, status = succeeded4983lb # [ 57.194969] niks3-server[1047]: 2026/09/22 11:22:20 INFO Received uploads request method=POST path=/api/pending_closures4984worker1 # [ 57.184286] nix-grpc-daemon[867]: time=2026-09-22T11:22:20.415Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4985worker1 # [ 57.185626] nix-grpc-daemon[867]: time=2026-09-22T11:22:20.415Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4986lb # [ 57.228071] niks3-server[1047]: 2026/09/22 11:22:20 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4987lb # [ 57.244927] niks3-server[1047]: 2026/09/22 11:22:20 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4988lb # [ 57.254784] niks3-server[1047]: 2026/09/22 11:22:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4989lb # [ 57.258907] niks3-server[1047]: 2026/09/22 11:22:20 INFO Signed narinfos id=8 count=14990worker1 # [ 57.239595] nix-grpc-daemon[867]: time=2026-09-22T11:22:20.471Z level=INFO msg="Uploading 1 narinfos"4991lb # [ 57.263730] niks3-server[1047]: 2026/09/22 11:22:20 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4992lb # [ 57.273071] niks3-server[1047]: 2026/09/22 11:22:20 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4993lb # [ 57.275950] niks3-server[1047]: 2026/09/22 11:22:20 INFO Completed upload id=84994lb # [ 57.277917] niks3-server[1047]: 2026/09/22 11:22:20 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4995worker1 # [ 57.255660] nix-grpc-daemon[867]: time=2026-09-22T11:22:20.488Z level=INFO msg="Upload complete. (270ms)"4996worker1 # [ 57.261689] nix-grpc-daemon[786]: ts=2026-09-22T11:22:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14997worker1 # [ 57.274861] nix-daemon[855]: reaped child process 1008, status = succeeded4998client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.27 seconds)4999client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath5000client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.18 seconds)5001lb: must succeed: curl -sf localhost:9901/clusters5002lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5003lb: must succeed: curl -sf localhost:9901/clusters5004lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5005worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true5006worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.03 seconds)5007(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.74 seconds)5008subtest: outputs read back through any worker carry the cache signature5009client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag sig >&25010worker2 # [ 57.820325] nix-grpc-daemon[789]: ts=2026-09-22T11:22:21Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:362445011client # don't know how to build these paths:5012client # /nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv5013worker1 # [ 57.815311] nix-grpc-daemon[786]: ts=2026-09-22T11:22:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361345014worker2 # [ 57.841694] nix-grpc-daemon[789]: ts=2026-09-22T11:22:21Z level=debug event=want client=4 drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv cached=05015worker2 # [ 57.844188] nix-grpc-daemon[789]: ts=2026-09-22T11:22:21Z level=debug event=assigned drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv worker=192.168.1.4:50051 clients=15016worker2 # [ 57.846875] nix-grpc-daemon[789]: ts=2026-09-22T11:22:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445017worker2 # [ 57.854737] nix-grpc-daemon[789]: ts=2026-09-22T11:22:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:362445018client # copying 1 paths to 'lb:50051'...5019client # copying path '/nix/store/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv' to 'lb:50051'...5020worker2 # [ 57.860178] nix-grpc-daemon[789]: ts=2026-09-22T11:22:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:362445021worker2 # [ 57.864622] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5022worker2 # [ 57.879119] nix-grpc-daemon[789]: ts=2026-09-22T11:22:21Z level=info event=niks3_push_started pid=9285023worker2 # [ 58.117834] nix-grpc-daemon[928]: time=2026-09-22T11:22:22.184Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5024worker2 # [ 58.307621] nix-daemon[877]: accepted connection from pid 940, user nix-grpc-daemon (trusted)5025worker2 # [ 58.324699] nix-daemon[877]: reaped child process 943, status = succeeded5026worker2 # [ 58.479785] nix-daemon[877]: accepted connection from pid 945, user nix-grpc-daemon (trusted)5027worker2 # [ 58.493645] nix-daemon[877]: reaped child process 948, status = succeeded5028lb # [ 58.519424] niks3-server[1047]: 2026/09/22 11:22:21 INFO Received uploads request method=POST path=/api/pending_closures5029worker2 # [ 58.514767] nix-grpc-daemon[928]: time=2026-09-22T11:22:22.585Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5030worker2 # [ 58.517727] nix-grpc-daemon[928]: time=2026-09-22T11:22:22.588Z level=INFO msg="Uploading lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv (504B)"5031lb # [ 58.544996] niks3-server[1047]: 2026/09/22 11:22:21 INFO Registered completed upload object_key=nar/0bddczcnpndyzwmafx9yqfqrkrp1jg9rr5hsixz6rpy99kg84g1g.nar.zst5032lb # [ 58.548052] niks3-server[1047]: 2026/09/22 11:22:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign5033lb # [ 58.550697] niks3-server[1047]: 2026/09/22 11:22:21 INFO Signed narinfos id=9 count=15034worker2 # [ 58.540923] nix-grpc-daemon[928]: time=2026-09-22T11:22:22.611Z level=INFO msg="Uploading 1 narinfos"5035lb # [ 58.553144] niks3-server[1047]: 2026/09/22 11:22:21 INFO Registered completed upload object_key=lqi1rvcqjj8q9l279yzgrgaghp394zs5.ls5036lb # [ 58.558620] niks3-server[1047]: 2026/09/22 11:22:21 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete5037lb # [ 58.562156] niks3-server[1047]: 2026/09/22 11:22:21 INFO Completed upload id=95038worker2 # [ 58.553273] nix-grpc-daemon[928]: time=2026-09-22T11:22:22.623Z level=INFO msg="Upload complete. (408ms)"5039lb # [ 58.563683] niks3-server[1047]: 2026/09/22 11:22:21 INFO Registered completed upload object_key=lqi1rvcqjj8q9l279yzgrgaghp394zs5.narinfo5040worker2 # [ 58.555929] nix-grpc-daemon[789]: ts=2026-09-22T11:22:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_in=5045041worker2 # [ 58.561480] nix-grpc-daemon[789]: ts=2026-09-22T11:22:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445042worker2 # [ 58.567589] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5043worker2 # [ 58.570217] nix-daemon[877]: reaped child process 926, status = succeeded5044client # building '/nix/store/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv' on lb:50051...5045worker2 # [ 58.576182] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5046client # farm-a> worker2: building lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv5047client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5048client # farm-a> LOG-farm-a-sig5049worker2 # [ 59.791390] nix-daemon[877]: reaped child process 952, status = succeeded5050worker2 # [ 59.936741] nix-daemon[877]: accepted connection from pid 968, user nix-grpc-daemon (trusted)5051worker2 # [ 59.950606] nix-daemon[877]: reaped child process 971, status = succeeded5052lb # [ 59.976580] niks3-server[1047]: 2026/09/22 11:22:23 INFO Received uploads request method=POST path=/api/pending_closures5053worker2 # [ 59.974494] nix-grpc-daemon[928]: time=2026-09-22T11:22:24.044Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5054worker2 # [ 59.976199] nix-grpc-daemon[928]: time=2026-09-22T11:22:24.046Z level=INFO msg="Uploading qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a (128B)"5055lb # [ 60.010562] niks3-server[1047]: 2026/09/22 11:22:23 INFO Registered completed upload object_key=log/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv5056lb # [ 60.023837] niks3-server[1047]: 2026/09/22 11:22:23 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst5057lb # [ 60.035619] niks3-server[1047]: 2026/09/22 11:22:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign5058lb # [ 60.041312] niks3-server[1047]: 2026/09/22 11:22:23 INFO Signed narinfos id=10 count=15059worker2 # [ 60.034289] nix-grpc-daemon[928]: time=2026-09-22T11:22:24.104Z level=INFO msg="Uploading 1 narinfos"5060lb # [ 60.044245] niks3-server[1047]: 2026/09/22 11:22:23 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.ls5061lb # [ 60.054473] niks3-server[1047]: 2026/09/22 11:22:23 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete5062lb # [ 60.061355] niks3-server[1047]: 2026/09/22 11:22:23 INFO Completed upload id=105063worker2 # [ 60.051533] nix-grpc-daemon[928]: time=2026-09-22T11:22:24.121Z level=INFO msg="Upload complete. (266ms)"5064lb # [ 60.062982] niks3-server[1047]: 2026/09/22 11:22:23 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.narinfo5065worker2 # [ 60.058549] nix-grpc-daemon[789]: ts=2026-09-22T11:22:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=1 drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv assign_id=5 outputs=15066worker2 # [ 60.068825] nix-grpc-daemon[789]: ts=2026-09-22T11:22:24Z level=debug event=want client=4 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv cached=05067worker2 # [ 60.072867] nix-grpc-daemon[789]: ts=2026-09-22T11:22:24Z level=debug event=assigned drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv worker=192.168.1.4:50051 clients=15068worker2 # [ 60.079995] nix-daemon[877]: reaped child process 950, status = succeeded5069worker2 # [ 60.112531] nix-grpc-daemon[789]: ts=2026-09-22T11:22:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445070worker2 # [ 60.124574] nix-grpc-daemon[789]: ts=2026-09-22T11:22:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:362445071worker2 # [ 60.136597] nix-grpc-daemon[928]: time=2026-09-22T11:22:24.206Z level=INFO msg="All 1 paths already cached"5072client # copying 1 paths to 'lb:50051'...5073client # copying path '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' to 'lb:50051'...5074worker2 # [ 60.145138] nix-grpc-daemon[789]: ts=2026-09-22T11:22:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:362445075worker2 # [ 60.154628] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5076worker2 # [ 60.332164] nix-daemon[877]: accepted connection from pid 978, user nix-grpc-daemon (trusted)5077worker2 # [ 60.345395] nix-daemon[877]: reaped child process 981, status = succeeded5078worker2 # [ 60.511627] nix-daemon[877]: accepted connection from pid 983, user nix-grpc-daemon (trusted)5079worker2 # [ 60.526593] nix-daemon[877]: reaped child process 986, status = succeeded5080lb # [ 60.550616] niks3-server[1047]: 2026/09/22 11:22:23 INFO Received uploads request method=POST path=/api/pending_closures5081worker2 # [ 60.550828] nix-grpc-daemon[928]: time=2026-09-22T11:22:24.621Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5082worker2 # [ 60.553120] nix-grpc-daemon[928]: time=2026-09-22T11:22:24.623Z level=INFO msg="Uploading 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv (624B)"5083lb # [ 60.582541] niks3-server[1047]: 2026/09/22 11:22:23 INFO Registered completed upload object_key=nar/1m1mvw8bmb91glp26qj699lxqv98q2iz8wm4ik3miz0xj6vi3mw6.nar.zst5084lb # [ 60.594555] niks3-server[1047]: 2026/09/22 11:22:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign5085lb # [ 60.599136] niks3-server[1047]: 2026/09/22 11:22:24 INFO Signed narinfos id=11 count=15086worker2 # [ 60.592441] nix-grpc-daemon[928]: time=2026-09-22T11:22:24.662Z level=INFO msg="Uploading 1 narinfos"5087lb # [ 60.608544] niks3-server[1047]: 2026/09/22 11:22:24 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.ls5088lb # [ 60.617490] niks3-server[1047]: 2026/09/22 11:22:24 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete5089lb # [ 60.623645] niks3-server[1047]: 2026/09/22 11:22:24 INFO Completed upload id=115090worker2 # [ 60.613681] nix-grpc-daemon[928]: time=2026-09-22T11:22:24.684Z level=INFO msg="Upload complete. (446ms)"5091lb # [ 60.625844] niks3-server[1047]: 2026/09/22 11:22:24 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.narinfo5092worker2 # [ 60.617183] nix-grpc-daemon[789]: ts=2026-09-22T11:22:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_in=6245093worker2 # [ 60.622234] nix-grpc-daemon[789]: ts=2026-09-22T11:22:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445094worker2 # [ 60.631484] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5095worker2 # [ 60.640260] nix-daemon[877]: reaped child process 975, status = succeeded5096worker2 # [ 60.644664] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5097client # building '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' on lb:50051...5098client # farm-b> worker2: building 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv5099client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5100client # farm-b> LOG-farm-b-sig5101worker2 # [ 61.460147] nix-daemon[877]: reaped child process 990, status = succeeded5102worker2 # [ 61.609699] nix-daemon[877]: accepted connection from pid 1006, user nix-grpc-daemon (trusted)5103worker2 # [ 61.623708] nix-daemon[877]: reaped child process 1009, status = succeeded5104lb # [ 61.648623] niks3-server[1047]: 2026/09/22 11:22:25 INFO Received uploads request method=POST path=/api/pending_closures5105worker2 # [ 61.646649] nix-grpc-daemon[928]: time=2026-09-22T11:22:25.716Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5106worker2 # [ 61.648215] nix-grpc-daemon[928]: time=2026-09-22T11:22:25.718Z level=INFO msg="Uploading 2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b (176B)"5107lb # [ 61.682921] niks3-server[1047]: 2026/09/22 11:22:25 INFO Registered completed upload object_key=nar/0y3xlf15z3z540fgrc9sgnn2rl0aj3w6x9mjqd50hhqgmaxbiivd.nar.zst5108lb # [ 61.694687] niks3-server[1047]: 2026/09/22 11:22:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign5109lb # [ 61.701924] niks3-server[1047]: 2026/09/22 11:22:25 INFO Registered completed upload object_key=log/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv5110lb # [ 61.707261] niks3-server[1047]: 2026/09/22 11:22:25 INFO Signed narinfos id=12 count=15111worker2 # [ 61.700302] nix-grpc-daemon[928]: time=2026-09-22T11:22:25.770Z level=INFO msg="Uploading 1 narinfos"5112lb # [ 61.712624] niks3-server[1047]: 2026/09/22 11:22:25 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.ls5113lb # [ 61.722356] niks3-server[1047]: 2026/09/22 11:22:25 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete5114lb # [ 61.729769] niks3-server[1047]: 2026/09/22 11:22:25 INFO Completed upload id=125115lb # [ 61.731371] niks3-server[1047]: 2026/09/22 11:22:25 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.narinfo5116worker2 # [ 61.721564] nix-grpc-daemon[928]: time=2026-09-22T11:22:25.791Z level=INFO msg="Upload complete. (268ms)"5117worker2 # [ 61.728607] nix-grpc-daemon[789]: ts=2026-09-22T11:22:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=1 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv assign_id=6 outputs=15118worker2 # [ 61.740299] nix-grpc-daemon[789]: ts=2026-09-22T11:22:25Z level=debug event=want client=4 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv cached=05119worker2 # [ 61.744217] nix-grpc-daemon[789]: ts=2026-09-22T11:22:25Z level=debug event=assigned drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv worker=192.168.1.4:50051 clients=15120worker2 # [ 61.753844] nix-daemon[877]: reaped child process 988, status = succeeded5121worker2 # [ 61.783212] nix-grpc-daemon[789]: ts=2026-09-22T11:22:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445122worker2 # [ 61.794749] nix-grpc-daemon[789]: ts=2026-09-22T11:22:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:362445123worker2 # [ 61.806878] nix-grpc-daemon[928]: time=2026-09-22T11:22:25.877Z level=INFO msg="All 2 paths already cached"5124client # copying 1 paths to 'lb:50051'...5125client # copying path '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' to 'lb:50051'...5126worker2 # [ 61.814590] nix-grpc-daemon[789]: ts=2026-09-22T11:22:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:362445127worker2 # [ 61.823133] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5128worker2 # [ 61.988220] nix-daemon[877]: accepted connection from pid 1015, user nix-grpc-daemon (trusted)5129worker2 # [ 62.002736] nix-daemon[877]: reaped child process 1018, status = succeeded5130worker2 # [ 62.158120] nix-daemon[877]: accepted connection from pid 1020, user nix-grpc-daemon (trusted)5131worker2 # [ 62.173609] nix-daemon[877]: reaped child process 1023, status = succeeded5132lb # [ 62.197126] niks3-server[1047]: 2026/09/22 11:22:25 INFO Received uploads request method=POST path=/api/pending_closures5133worker2 # [ 62.195302] nix-grpc-daemon[928]: time=2026-09-22T11:22:26.265Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5134worker2 # [ 62.196692] nix-grpc-daemon[928]: time=2026-09-22T11:22:26.265Z level=INFO msg="Uploading cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv (752B)"5135lb # [ 62.222582] niks3-server[1047]: 2026/09/22 11:22:25 INFO Registered completed upload object_key=nar/0w21abkvlzg18nqkl2x4zxs7d0376kbld1n0fkr68vcpqpq1g5zg.nar.zst5136lb # [ 62.231129] niks3-server[1047]: 2026/09/22 11:22:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign5137worker2 # [ 62.227529] nix-grpc-daemon[928]: time=2026-09-22T11:22:26.297Z level=INFO msg="Uploading 1 narinfos"5138lb # [ 62.237361] niks3-server[1047]: 2026/09/22 11:22:25 INFO Signed narinfos id=13 count=15139lb # [ 62.243531] niks3-server[1047]: 2026/09/22 11:22:25 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.ls5140lb # [ 62.256529] niks3-server[1047]: 2026/09/22 11:22:25 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete5141lb # [ 62.261728] niks3-server[1047]: 2026/09/22 11:22:25 INFO Completed upload id=135142worker2 # [ 62.253866] nix-grpc-daemon[928]: time=2026-09-22T11:22:26.324Z level=INFO msg="Upload complete. (418ms)"5143lb # [ 62.264641] niks3-server[1047]: 2026/09/22 11:22:25 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.narinfo5144worker2 # [ 62.258223] nix-grpc-daemon[789]: ts=2026-09-22T11:22:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_in=7525145worker2 # [ 62.262702] nix-grpc-daemon[789]: ts=2026-09-22T11:22:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445146worker2 # [ 62.271431] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5147worker2 # [ 62.277128] nix-daemon[877]: reaped child process 1013, status = succeeded5148client # building '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' on lb:50051...5149client # farm-top> worker2: building cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv5150worker2 # [ 62.282343] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5151client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5152client # farm-top> LOG-farm-top-sig5153worker2 # [ 63.304242] nix-daemon[877]: reaped child process 1027, status = succeeded5154worker2 # [ 63.453213] nix-daemon[877]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5155worker2 # [ 63.466744] nix-daemon[877]: reaped child process 1046, status = succeeded5156lb # [ 63.488292] niks3-server[1047]: 2026/09/22 11:22:26 INFO Received uploads request method=POST path=/api/pending_closures5157worker2 # [ 63.486648] nix-grpc-daemon[928]: time=2026-09-22T11:22:27.556Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5158worker2 # [ 63.488203] nix-grpc-daemon[928]: time=2026-09-22T11:22:27.558Z level=INFO msg="Uploading dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top (232B)"5159lb # [ 63.521143] niks3-server[1047]: 2026/09/22 11:22:26 INFO Registered completed upload object_key=log/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv5160lb # [ 63.535151] niks3-server[1047]: 2026/09/22 11:22:26 INFO Registered completed upload object_key=nar/0h4zvhqjxsvdqli2l6i5ajh5sh40hz57gzchdirnvbisq2aks0k8.nar.zst5161lb # [ 63.543578] niks3-server[1047]: 2026/09/22 11:22:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign5162lb # [ 63.548721] niks3-server[1047]: 2026/09/22 11:22:26 INFO Signed narinfos id=14 count=15163worker2 # [ 63.541659] nix-grpc-daemon[928]: time=2026-09-22T11:22:27.611Z level=INFO msg="Uploading 1 narinfos"5164lb # [ 63.561670] niks3-server[1047]: 2026/09/22 11:22:26 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete5165lb # [ 63.685406] niks3-server[1047]: 2026/09/22 11:22:27 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.ls5166lb # [ 63.691087] niks3-server[1047]: 2026/09/22 11:22:27 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.narinfo5167lb # [ 63.698736] niks3-server[1047]: 2026/09/22 11:22:27 INFO Completed upload id=145168worker2 # [ 63.688812] nix-grpc-daemon[928]: time=2026-09-22T11:22:27.759Z level=INFO msg="Upload complete. (396ms)"5169worker2 # [ 63.696897] nix-grpc-daemon[789]: ts=2026-09-22T11:22:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=1 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv assign_id=7 outputs=15170worker2 # [ 63.711427] nix-daemon[877]: reaped child process 1025, status = succeeded5171client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag sig >&2, in 6.15 seconds)5172client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag sig outPath5173client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag sig outPath, in 0.20 seconds)5174client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5175worker1 # [ 64.152473] nix-grpc-daemon[786]: ts=2026-09-22T11:22:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:356905176client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):5177client # /nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a5178client # /nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b5179client # /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5180worker1 # [ 64.198247] nix-grpc-daemon[786]: ts=2026-09-22T11:22:27Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:356905181worker1 # [ 64.206296] nix-grpc-daemon[786]: copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'http://lb:5751'...5182worker1 # [ 64.222929] nix-grpc-daemon[786]: copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'http://lb:5751'...5183worker1 # [ 64.234995] nix-grpc-daemon[786]: copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'http://lb:5751'...5184client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.31 seconds)5185client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5186worker2 # [ 64.484982] nix-grpc-daemon[789]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:577125187worker2 # [ 64.487611] nix-grpc-daemon[789]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:577125188client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.21 seconds)5189client: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5190client # finding garbage collector roots...5191client # removing stale temporary roots file "/nix/var/nix/temproots/878"5192client # removing stale temporary roots file "/nix/var/nix/temproots/809"5193client # removing stale temporary roots file "/nix/var/nix/temproots/824"5194client # removing stale temporary roots file "/nix/var/nix/temproots/845"5195client # removing stale temporary roots file "/nix/var/nix/temproots/900"5196client # removing stale temporary roots file "/nix/var/nix/temproots/865"5197client # removing stale temporary roots file "/nix/var/nix/temproots/886"5198client # removing stale temporary roots file "/nix/var/nix/temproots/857"5199client # deleting specified paths...5200client # deleting '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top'5201client # deleting unused links...5202client # note: hard linking is currently saving 0.0 KiB5203worker2 # [ 64.906702] nix-grpc-daemon[789]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:362445204worker2 # [ 64.908789] nix-grpc-daemon[789]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:362445205worker2 # [ 64.914749] nix-grpc-daemon[928]: time=2026-09-22T11:22:28.985Z level=INFO msg="All 1 paths already cached"5206worker1 # [ 64.907488] nix-grpc-daemon[786]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:361345207worker1 # [ 64.910176] nix-grpc-daemon[786]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:361345208worker2 # [ 64.925256] nix-grpc-daemon[789]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:362445209client # copying 3 paths...5210worker1 # [ 64.920500] nix-grpc-daemon[786]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:361345211client # copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'grpc://lb:50051'...5212worker1 # [ 64.936299] nix-grpc-daemon[786]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:361345213worker1 # [ 64.938812] nix-grpc-daemon[786]: ts=2026-09-22T11:22:28Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=0 paths=1 nar_bytes_out=2325214worker2 # [ 64.956618] nix-grpc-daemon[789]: ts=2026-09-22T11:22:29Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:362445215worker2 # [ 64.959314] nix-grpc-daemon[789]: ts=2026-09-22T11:22:29Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_out=1765216worker2 # [ 64.964696] nix-grpc-daemon[789]: ts=2026-09-22T11:22:29Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:362445217worker2 # [ 64.966919] nix-grpc-daemon[789]: ts=2026-09-22T11:22:29Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_out=1285218client # copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'grpc://lb:50051'...5219client # copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'grpc://lb:50051'...5220client: (finished: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.54 seconds)5221(finished: subtest: outputs read back through any worker carry the cache signature, in 7.42 seconds)5222subtest: two clients wanting the same drv build it once5223client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup5224client # [ 65.083907] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.5225client # Running as unit: dedup1.service; invocation ID: bd2f7dff8def44e39a31ecc9efeb43145226client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.04 seconds)5227client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup5228client # [ 65.299228] nix[953]: don't know how to build these paths:5229client # [ 65.300115] nix[953]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5230worker1 # [ 65.289549] nix-grpc-daemon[786]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:361345231worker2 # [ 65.309650] nix-grpc-daemon[789]: ts=2026-09-22T11:22:29Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05232worker2 # [ 65.313914] nix-grpc-daemon[789]: ts=2026-09-22T11:22:29Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.3:50051 clients=15233worker1 # [ 65.341747] nix-grpc-daemon[786]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345234worker1 # [ 65.352401] nix-grpc-daemon[786]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361345235client # [ 65.371041] nix[953]: copying 1 paths to 'lb:50051'...5236client # [ 65.371782] nix[953]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...5237worker1 # [ 65.362845] nix-grpc-daemon[786]: ts=2026-09-22T11:22:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361345238worker1 # [ 65.373817] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5239worker1 # [ 65.546911] nix-daemon[855]: accepted connection from pid 1036, user nix-grpc-daemon (trusted)5240worker1 # [ 65.560504] nix-daemon[855]: reaped child process 1039, status = succeeded5241worker1 # [ 65.711724] nix-daemon[855]: accepted connection from pid 1041, user nix-grpc-daemon (trusted)5242worker1 # [ 65.723152] nix-daemon[855]: reaped child process 1044, status = succeeded5243lb # [ 65.759551] niks3-server[1047]: 2026/09/22 11:22:29 INFO Received uploads request method=POST path=/api/pending_closures5244worker1 # [ 65.745315] nix-grpc-daemon[867]: time=2026-09-22T11:22:28.977Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5245worker1 # [ 65.746807] nix-grpc-daemon[867]: time=2026-09-22T11:22:28.979Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"5246lb # [ 65.785963] niks3-server[1047]: 2026/09/22 11:22:29 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst5247lb # [ 65.796261] niks3-server[1047]: 2026/09/22 11:22:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5248lb # [ 65.801065] niks3-server[1047]: 2026/09/22 11:22:29 INFO Signed narinfos id=15 count=15249worker1 # [ 65.782344] nix-grpc-daemon[867]: time=2026-09-22T11:22:29.014Z level=INFO msg="Uploading 1 narinfos"5250lb # [ 65.813088] niks3-server[1047]: 2026/09/22 11:22:29 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5251lb # [ 65.818180] niks3-server[1047]: 2026/09/22 11:22:29 INFO Completed upload id=155252worker1 # [ 65.799053] nix-grpc-daemon[867]: time=2026-09-22T11:22:29.031Z level=INFO msg="Upload complete. (414ms)"5253worker1 # [ 65.801320] nix-grpc-daemon[786]: ts=2026-09-22T11:22:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=0 paths=1 nar_bytes_in=4325254lb # [ 65.823662] niks3-server[1047]: 2026/09/22 11:22:29 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls5255lb # [ 65.827946] niks3-server[1047]: 2026/09/22 11:22:29 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo5256worker1 # [ 65.806509] nix-grpc-daemon[786]: ts=2026-09-22T11:22:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345257worker1 # [ 65.811223] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5258worker1 # [ 65.813360] nix-daemon[855]: reaped child process 1034, status = succeeded5259worker1 # [ 65.818534] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5260client # [ 65.827301] nix[953]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5261client # [ 66.155403] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.5262client # Running as unit: dedup2.service; invocation ID: 2d4d6284a93b4f1f9e603a9086ba2db45263client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.08 seconds)5264worker1 # [ 66.415868] nix-grpc-daemon[786]: ts=2026-09-22T11:22:29Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:361345265client # [ 66.435476] nix[969]: this derivation will be built:5266worker2 # [ 66.439682] nix-grpc-daemon[789]: ts=2026-09-22T11:22:30Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:362445267worker2 # [ 66.447735] nix-grpc-daemon[789]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' from 'http://lb:5751'...5268client # [ 66.462035] nix[969]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5269worker2 # [ 66.471198] nix-grpc-daemon[789]: ts=2026-09-22T11:22:30Z level=debug event=want client=6 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05270worker1 # [ 66.468779] nix-grpc-daemon[786]: ts=2026-09-22T11:22:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345271client # [ 66.482683] nix[969]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5272lb # [ 66.778985] envoy[899]: conn peer=[2001:db8:1::1]:57114 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4169 tx=1895 ms=20585273lb # [ 66.782664] envoy[899]: conn peer=[2001:db8:1::1]:57116 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9524 tx=4372 ms=59295274lb # [ 66.786119] envoy[899]: conn peer=[2001:db8:1::1]:36694 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1108 tx=2927 ms=1245275lb # [ 66.789582] envoy[899]: conn peer=[2001:db8:1::1]:36714 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=330 ms=665276lb # [ 66.792884] envoy[899]: conn peer=[2001:db8:1::1]:36700 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=376 ms=745277lb # [ 66.796279] envoy[899]: conn peer=[2001:db8:1::1]:36696 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=410 ms=835278client: must succeed: sleep 25279client: (finished: must succeed: sleep 2, in 2.06 seconds)5280client: waiting for success: ! systemctl is-active dedup1 dedup25281worker1 # [ 70.845683] nix-daemon[855]: reaped child process 1048, status = succeeded5282worker1 # [ 71.005070] nix-daemon[855]: accepted connection from pid 1082, user nix-grpc-daemon (trusted)5283worker1 # [ 71.016960] nix-daemon[855]: reaped child process 1085, status = succeeded5284lb # [ 71.054627] niks3-server[1047]: 2026/09/22 11:22:34 INFO Received uploads request method=POST path=/api/pending_closures5285worker1 # [ 71.040755] nix-grpc-daemon[867]: time=2026-09-22T11:22:34.273Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5286worker1 # [ 71.042858] nix-grpc-daemon[867]: time=2026-09-22T11:22:34.274Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"5287lb # [ 71.093218] niks3-server[1047]: 2026/09/22 11:22:34 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst5288lb # [ 71.105858] niks3-server[1047]: 2026/09/22 11:22:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5289lb # [ 71.113976] niks3-server[1047]: 2026/09/22 11:22:34 INFO Signed narinfos id=16 count=15290worker1 # [ 71.095191] nix-grpc-daemon[867]: time=2026-09-22T11:22:34.327Z level=INFO msg="Uploading 1 narinfos"5291lb # [ 71.119205] niks3-server[1047]: 2026/09/22 11:22:34 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5292lb # [ 71.124215] niks3-server[1047]: 2026/09/22 11:22:34 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls5293lb # [ 71.133603] niks3-server[1047]: 2026/09/22 11:22:34 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5294lb # [ 71.139555] niks3-server[1047]: 2026/09/22 11:22:34 INFO Completed upload id=165295worker1 # [ 71.119657] nix-grpc-daemon[867]: time=2026-09-22T11:22:34.352Z level=INFO msg="Upload complete. (288ms)"5296lb # [ 71.141870] niks3-server[1047]: 2026/09/22 11:22:34 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo5297worker1 # [ 71.122566] nix-grpc-daemon[786]: ts=2026-09-22T11:22:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15298worker1 # [ 71.126135] nix-grpc-daemon[786]: ts=2026-09-22T11:22:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15299worker1 # [ 71.131477] nix-daemon[855]: reaped child process 1046, status = succeeded5300client # [ 71.162789] systemd[1]: dedup1.service: Deactivated successfully.5301client # [ 71.165144] systemd[1]: dedup1.service: Consumed 114ms CPU time over 6.080s wall clock time, 43.5M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.5302client # [ 71.204032] systemd[1]: dedup2.service: Deactivated successfully.5303client # [ 71.205440] systemd[1]: dedup2.service: Consumed 134ms CPU time over 5.050s wall clock time, 35.3M memory peak, 6.4K incoming IP traffic, 5.8K outgoing IP traffic.5304client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.11 seconds)5305client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }5306client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.05 seconds)5307worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5308worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)5309worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5310worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)5311worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5312worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5313worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5314worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5315(finished: subtest: two clients wanting the same drv build it once, in 7.06 seconds)5316subtest: --timeout and --max-silent-time reach the builder5317worker2 # [ 72.340807] nix-grpc-daemon[789]: ts=2026-09-22T11:22:36Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:362445318worker2 # [ 72.352513] nix-grpc-daemon[789]: ts=2026-09-22T11:22:36Z level=debug event=want client=7 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv cached=05319worker2 # [ 72.358589] nix-grpc-daemon[789]: ts=2026-09-22T11:22:36Z level=debug event=assigned drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv worker=192.168.1.4:50051 clients=15320worker2 # [ 72.364871] nix-grpc-daemon[789]: ts=2026-09-22T11:22:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445321worker2 # [ 72.371973] nix-grpc-daemon[789]: ts=2026-09-22T11:22:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:362445322worker2 # [ 72.380755] nix-grpc-daemon[789]: ts=2026-09-22T11:22:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:362445323worker2 # [ 72.391641] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5324worker2 # [ 72.561072] nix-daemon[877]: accepted connection from pid 1082, user nix-grpc-daemon (trusted)5325worker2 # [ 72.574576] nix-daemon[877]: reaped child process 1085, status = succeeded5326worker2 # [ 72.734395] nix-daemon[877]: accepted connection from pid 1087, user nix-grpc-daemon (trusted)5327worker2 # [ 72.746803] nix-daemon[877]: reaped child process 1090, status = succeeded5328lb # [ 72.771908] niks3-server[1047]: 2026/09/22 11:22:36 INFO Received uploads request method=POST path=/api/pending_closures5329worker2 # [ 72.765875] nix-grpc-daemon[928]: time=2026-09-22T11:22:36.836Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5330worker2 # [ 72.767451] nix-grpc-daemon[928]: time=2026-09-22T11:22:36.837Z level=INFO msg="Uploading qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv (424B)"5331lb # [ 72.787731] niks3-server[1047]: 2026/09/22 11:22:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5332worker2 # [ 72.783077] nix-grpc-daemon[928]: time=2026-09-22T11:22:36.853Z level=INFO msg="Uploading 1 narinfos"5333lb # [ 72.793695] niks3-server[1047]: 2026/09/22 11:22:36 INFO Signed narinfos id=17 count=15334lb # [ 72.800968] niks3-server[1047]: 2026/09/22 11:22:36 INFO Registered completed upload object_key=nar/1s4sciw269ksj1s7ymwz5cy4dw9n7bxv3k8wqw69h694spdh4ha2.nar.zst5335lb # [ 72.805551] niks3-server[1047]: 2026/09/22 11:22:36 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.ls5336lb # [ 72.813243] niks3-server[1047]: 2026/09/22 11:22:36 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5337lb # [ 72.818987] niks3-server[1047]: 2026/09/22 11:22:36 INFO Completed upload id=175338worker2 # [ 72.811488] nix-grpc-daemon[928]: time=2026-09-22T11:22:36.881Z level=INFO msg="Upload complete. (408ms)"5339lb # [ 72.823404] niks3-server[1047]: 2026/09/22 11:22:36 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.narinfo5340worker2 # [ 72.814282] nix-grpc-daemon[789]: ts=2026-09-22T11:22:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_in=4245341worker2 # [ 72.817144] nix-grpc-daemon[789]: ts=2026-09-22T11:22:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445342worker2 # [ 72.823524] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5343worker2 # [ 72.826560] nix-daemon[877]: reaped child process 1080, status = succeeded5344worker2 # [ 72.832090] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5345lb # [ 76.779609] envoy[899]: conn peer=[2001:db8:1::1]:36726 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3011 tx=1560 ms=58435346lb # [ 76.783422] envoy[899]: conn peer=[2001:db8:1::1]:36734 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1576 tx=1599 ms=47655347worker2 # [ 78.648868] nix-grpc-daemon[789]: ts=2026-09-22T11:22:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=5 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv assign_id=9 outputs=05348worker2 # [ 78.673195] nix-daemon[877]: reaped child process 1092, status = succeeded5349worker2 # [ 78.675294] nix-daemon[877]: reaped child process 1094, status = succeeded5350worker1 # [ 78.888190] nix-grpc-daemon[786]: ts=2026-09-22T11:22:42Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:361345351worker2 # [ 78.915061] nix-grpc-daemon[789]: ts=2026-09-22T11:22:42Z level=debug event=want client=8 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv cached=05352worker2 # [ 78.919695] nix-grpc-daemon[789]: ts=2026-09-22T11:22:42Z level=debug event=assigned drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv worker=192.168.1.3:50051 clients=15353worker1 # [ 78.910418] nix-grpc-daemon[786]: ts=2026-09-22T11:22:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345354worker1 # [ 78.922676] nix-grpc-daemon[786]: ts=2026-09-22T11:22:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361345355worker1 # [ 78.932922] nix-grpc-daemon[786]: ts=2026-09-22T11:22:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361345356worker1 # [ 78.943673] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5357worker1 # [ 79.111128] nix-daemon[855]: accepted connection from pid 1098, user nix-grpc-daemon (trusted)5358worker1 # [ 79.123509] nix-daemon[855]: reaped child process 1101, status = succeeded5359worker1 # [ 79.275255] nix-daemon[855]: accepted connection from pid 1103, user nix-grpc-daemon (trusted)5360worker1 # [ 79.286670] nix-daemon[855]: reaped child process 1106, status = succeeded5361lb # [ 79.323374] niks3-server[1047]: 2026/09/22 11:22:42 INFO Received uploads request method=POST path=/api/pending_closures5362worker1 # [ 79.310100] nix-grpc-daemon[867]: time=2026-09-22T11:22:42.542Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5363worker1 # [ 79.311667] nix-grpc-daemon[867]: time=2026-09-22T11:22:42.544Z level=INFO msg="Uploading fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv (424B)"5364lb # [ 79.354150] niks3-server[1047]: 2026/09/22 11:22:42 INFO Registered completed upload object_key=nar/013y5sy2r6v0qdwv0dzpgvdfy08bds4mfh2qpvbgjazzrg2v71hp.nar.zst5365lb # [ 79.362759] niks3-server[1047]: 2026/09/22 11:22:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5366worker1 # [ 79.346657] nix-grpc-daemon[867]: time=2026-09-22T11:22:42.579Z level=INFO msg="Uploading 1 narinfos"5367lb # [ 79.367842] niks3-server[1047]: 2026/09/22 11:22:42 INFO Signed narinfos id=18 count=15368lb # [ 79.373130] niks3-server[1047]: 2026/09/22 11:22:42 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.ls5369lb # [ 79.384964] niks3-server[1047]: 2026/09/22 11:22:42 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5370worker1 # [ 79.368414] nix-grpc-daemon[867]: time=2026-09-22T11:22:42.600Z level=INFO msg="Upload complete. (413ms)"5371lb # [ 79.390390] niks3-server[1047]: 2026/09/22 11:22:42 INFO Completed upload id=185372worker1 # [ 79.370452] nix-grpc-daemon[786]: ts=2026-09-22T11:22:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=0 paths=1 nar_bytes_in=4245373lb # [ 79.393862] niks3-server[1047]: 2026/09/22 11:22:42 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.narinfo5374worker1 # [ 79.376397] nix-grpc-daemon[786]: ts=2026-09-22T11:22:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345375worker1 # [ 79.380933] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5376worker1 # [ 79.383306] nix-daemon[855]: reaped child process 1096, status = succeeded5377worker1 # [ 79.388461] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5378worker1 # [ 85.200971] nix-grpc-daemon[786]: ts=2026-09-22T11:22:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=5 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv assign_id=10 outputs=05379worker1 # [ 85.225410] nix-daemon[855]: reaped child process 1108, status = succeeded5380worker1 # [ 85.227788] nix-daemon[855]: reaped child process 1110, status = succeeded5381(finished: subtest: --timeout and --max-silent-time reach the builder, in 13.32 seconds)5382subtest: the first client leaving does not fail the second5383client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share5384client # [ 85.458726] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share.5385client # Running as unit: share1.service; invocation ID: b679ca1c2e7e44f6a0f48ca272c9d7275386client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share, in 0.04 seconds)5387client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share5388worker2 # [ 85.651490] nix-grpc-daemon[789]: ts=2026-09-22T11:22:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:362445389client # [ 85.655699] nix[1042]: don't know how to build these paths:5390client # [ 85.656979] nix[1042]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5391worker2 # [ 85.669673] nix-grpc-daemon[789]: ts=2026-09-22T11:22:49Z level=debug event=want client=9 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05392worker2 # [ 85.676466] nix-grpc-daemon[789]: ts=2026-09-22T11:22:49Z level=debug event=assigned drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv worker=192.168.1.4:50051 clients=15393worker2 # [ 85.683363] nix-grpc-daemon[789]: ts=2026-09-22T11:22:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445394worker2 # [ 85.696490] nix-grpc-daemon[789]: ts=2026-09-22T11:22:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:362445395client # [ 85.704106] nix[1042]: copying 1 paths to 'lb:50051'...5396client # [ 85.704905] nix[1042]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' to 'lb:50051'...5397worker2 # [ 85.708150] nix-grpc-daemon[789]: ts=2026-09-22T11:22:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:362445398worker2 # [ 85.720664] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5399worker2 # [ 85.900807] nix-daemon[877]: accepted connection from pid 1121, user nix-grpc-daemon (trusted)5400worker2 # [ 85.917562] nix-daemon[877]: reaped child process 1124, status = succeeded5401worker2 # [ 86.089359] nix-daemon[877]: accepted connection from pid 1126, user nix-grpc-daemon (trusted)5402worker2 # [ 86.104440] nix-daemon[877]: reaped child process 1129, status = succeeded5403lb # [ 86.129664] niks3-server[1047]: 2026/09/22 11:22:49 INFO Received uploads request method=POST path=/api/pending_closures5404worker2 # [ 86.129559] nix-grpc-daemon[928]: time=2026-09-22T11:22:50.199Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5405worker2 # [ 86.131095] nix-grpc-daemon[928]: time=2026-09-22T11:22:50.199Z level=INFO msg="Uploading j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv (432B)"5406lb # [ 86.151317] niks3-server[1047]: 2026/09/22 11:22:49 INFO Registered completed upload object_key=nar/0s0b0gfhn36p0aylk1qa64cgw01bi7a8ca93iv4zg2q7wqcjcqhs.nar.zst5407lb # [ 86.158742] niks3-server[1047]: 2026/09/22 11:22:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5408lb # [ 86.161835] niks3-server[1047]: 2026/09/22 11:22:49 INFO Signed narinfos id=19 count=15409worker2 # [ 86.152197] nix-grpc-daemon[928]: time=2026-09-22T11:22:50.222Z level=INFO msg="Uploading 1 narinfos"5410lb # [ 86.165092] niks3-server[1047]: 2026/09/22 11:22:49 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.ls5411lb # [ 86.171108] niks3-server[1047]: 2026/09/22 11:22:49 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5412lb # [ 86.175129] niks3-server[1047]: 2026/09/22 11:22:49 INFO Completed upload id=195413worker2 # [ 86.165197] nix-grpc-daemon[928]: time=2026-09-22T11:22:50.235Z level=INFO msg="Upload complete. (432ms)"5414lb # [ 86.176941] niks3-server[1047]: 2026/09/22 11:22:49 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.narinfo5415worker2 # [ 86.167609] nix-grpc-daemon[789]: ts=2026-09-22T11:22:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_in=4325416worker2 # [ 86.171526] nix-grpc-daemon[789]: ts=2026-09-22T11:22:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445417worker2 # [ 86.178806] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5418worker2 # [ 86.183664] nix-daemon[877]: reaped child process 1119, status = succeeded5419client # [ 86.185751] nix[1042]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5420worker2 # [ 86.189407] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5421client # [ 86.512578] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share.5422client # Running as unit: share2.service; invocation ID: 63a953a177154bb8b4c63cff816dd8175423client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share, in 1.05 seconds)5424worker1 # [ 86.691412] nix-grpc-daemon[786]: ts=2026-09-22T11:22:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:361345425client # [ 86.705388] nix[1058]: don't know how to build these paths:5426client # [ 86.706213] nix[1058]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5427worker2 # [ 86.716638] nix-grpc-daemon[789]: ts=2026-09-22T11:22:50Z level=debug event=want client=10 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05428client # [ 86.716693] nix[1058]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5429worker2 # [ 86.719372] nix-grpc-daemon[789]: ts=2026-09-22T11:22:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445430lb # [ 86.779158] envoy[899]: conn peer=[2001:db8:1::1]:36742 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2970 tx=977 ms=63175431lb # [ 86.780949] envoy[899]: conn peer=[2001:db8:1::1]:57936 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2982 tx=948 ms=63695432client: must succeed: sleep 2; systemctl kill -s INT share15433client # [ 89.802615] systemd[1]: share1.service: Sent signal SIGINT to main process 1042 (nix) on client request.5434client # [ 89.806164] nix[1042]: shutting downj2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5435client # [ 89.809071] nix[1042]: error: interrupted by the user5436client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.04 seconds)5437client: waiting for success: ! systemctl is-active share15438client # [ 89.840616] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE5439client # [ 89.841969] systemd[1]: share1.service: Failed with result 'exit-code'.5440client # [ 89.844065] systemd[1]: share1.service: Consumed 101ms CPU time over 4.381s wall clock time, 41.4M memory peak, 6.2K incoming IP traffic, 8.1K outgoing IP traffic.5441client: (finished: waiting for success: ! systemctl is-active share1, in 0.04 seconds)5442client: waiting for success: ! systemctl is-active share25443worker2 # [ 90.208393] nix-daemon[877]: reaped child process 1133, status = succeeded5444worker2 # [ 90.363812] nix-daemon[877]: accepted connection from pid 1169, user nix-grpc-daemon (trusted)5445worker2 # [ 90.377406] nix-daemon[877]: reaped child process 1172, status = succeeded5446lb # [ 90.404960] niks3-server[1047]: 2026/09/22 11:22:53 INFO Received uploads request method=POST path=/api/pending_closures5447worker2 # [ 90.403224] nix-grpc-daemon[928]: time=2026-09-22T11:22:54.473Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5448worker2 # [ 90.405870] nix-grpc-daemon[928]: time=2026-09-22T11:22:54.475Z level=INFO msg="Uploading b9lg6znd0rinl36gy6ixq1dcj6b8bhhw-slow-share (120B)"5449lb # [ 90.447217] niks3-server[1047]: 2026/09/22 11:22:53 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst5450lb # [ 90.453298] niks3-server[1047]: 2026/09/22 11:22:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5451lb # [ 90.457592] niks3-server[1047]: 2026/09/22 11:22:53 INFO Signed narinfos id=20 count=15452worker2 # [ 90.450031] nix-grpc-daemon[928]: time=2026-09-22T11:22:54.520Z level=INFO msg="Uploading 1 narinfos"5453lb # [ 90.465493] niks3-server[1047]: 2026/09/22 11:22:53 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.ls5454lb # [ 90.469800] niks3-server[1047]: 2026/09/22 11:22:53 INFO Registered completed upload object_key=log/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5455lb # [ 90.478257] niks3-server[1047]: 2026/09/22 11:22:53 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5456lb # [ 90.481393] niks3-server[1047]: 2026/09/22 11:22:53 INFO Completed upload id=205457worker2 # [ 90.471475] nix-grpc-daemon[928]: time=2026-09-22T11:22:54.541Z level=INFO msg="Upload complete. (271ms)"5458lb # [ 90.484214] niks3-server[1047]: 2026/09/22 11:22:53 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.narinfo5459worker2 # [ 90.478463] nix-grpc-daemon[789]: ts=2026-09-22T11:22:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=4 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15460worker2 # [ 90.483554] nix-grpc-daemon[789]: ts=2026-09-22T11:22:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=3 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15461worker2 # [ 90.494903] nix-daemon[877]: reaped child process 1131, status = succeeded5462client # [ 90.555907] systemd[1]: share2.service: Deactivated successfully.5463client # [ 90.560305] systemd[1]: share2.service: Consumed 103ms CPU time over 4.046s wall clock time, 35.1M memory peak, 6.1K incoming IP traffic, 5.7K outgoing IP traffic.5464client: (finished: waiting for success: ! systemctl is-active share2, in 1.09 seconds)5465client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }5466client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.04 seconds)5467(finished: subtest: the first client leaving does not fail the second, in 5.80 seconds)5468subtest: build hook: nix-daemon with builders = grpc://lb5469client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log5470client # [ 91.404725] systemd[1]: Started Nix Daemon.5471client # [ 91.590485] nix-daemon[1104]: accepted connection from pid 1101, user root (trusted)5472worker1 # [ 91.773469] nix-grpc-daemon[786]: ts=2026-09-22T11:22:55Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:361345473worker1 # [ 91.778752] nix-grpc-daemon[786]: ts=2026-09-22T11:22:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361345474worker2 # [ 91.802797] nix-grpc-daemon[789]: ts=2026-09-22T11:22:55Z level=debug event=want client=11 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05475worker1 # [ 91.793185] nix-grpc-daemon[786]: ts=2026-09-22T11:22:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345476worker2 # [ 91.806872] nix-grpc-daemon[789]: ts=2026-09-22T11:22:55Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=15477worker1 # [ 91.804315] nix-grpc-daemon[786]: ts=2026-09-22T11:22:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361345478worker1 # [ 91.814974] nix-grpc-daemon[786]: ts=2026-09-22T11:22:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361345479worker1 # [ 91.825498] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5480worker1 # [ 92.007073] nix-daemon[855]: accepted connection from pid 1149, user nix-grpc-daemon (trusted)5481worker1 # [ 92.018795] nix-daemon[855]: reaped child process 1152, status = succeeded5482worker1 # [ 92.181751] nix-daemon[855]: accepted connection from pid 1154, user nix-grpc-daemon (trusted)5483worker1 # [ 92.192222] nix-daemon[855]: reaped child process 1157, status = succeeded5484lb # [ 92.230649] niks3-server[1047]: 2026/09/22 11:22:55 INFO Received uploads request method=POST path=/api/pending_closures5485worker1 # [ 92.217323] nix-grpc-daemon[867]: time=2026-09-22T11:22:55.449Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5486worker1 # [ 92.218969] nix-grpc-daemon[867]: time=2026-09-22T11:22:55.451Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5487lb # [ 92.265935] niks3-server[1047]: 2026/09/22 11:22:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5488worker1 # [ 92.250556] nix-grpc-daemon[867]: time=2026-09-22T11:22:55.482Z level=INFO msg="Uploading 1 narinfos"5489lb # [ 92.272156] niks3-server[1047]: 2026/09/22 11:22:55 INFO Signed narinfos id=21 count=15490lb # [ 92.279292] niks3-server[1047]: 2026/09/22 11:22:55 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5491lb # [ 92.290421] niks3-server[1047]: 2026/09/22 11:22:55 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5492lb # [ 92.299399] niks3-server[1047]: 2026/09/22 11:22:55 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5493lb # [ 92.304763] niks3-server[1047]: 2026/09/22 11:22:55 INFO Completed upload id=215494worker1 # [ 92.285359] nix-grpc-daemon[867]: time=2026-09-22T11:22:55.517Z level=INFO msg="Upload complete. (448ms)"5495lb # [ 92.309598] niks3-server[1047]: 2026/09/22 11:22:55 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo5496worker1 # [ 92.289482] nix-grpc-daemon[786]: ts=2026-09-22T11:22:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=0 paths=1 nar_bytes_in=5125497worker1 # [ 92.296325] nix-grpc-daemon[786]: ts=2026-09-22T11:22:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345498worker1 # [ 92.304514] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5499worker1 # [ 92.312631] nix-daemon[855]: reaped child process 1147, status = succeeded5500worker1 # [ 92.320760] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5501worker1 # [ 93.344488] nix-daemon[855]: reaped child process 1161, status = succeeded5502worker1 # [ 93.486707] nix-daemon[855]: accepted connection from pid 1177, user nix-grpc-daemon (trusted)5503worker1 # [ 93.498128] nix-daemon[855]: reaped child process 1180, status = succeeded5504lb # [ 93.532968] niks3-server[1047]: 2026/09/22 11:22:56 INFO Received uploads request method=POST path=/api/pending_closures5505worker1 # [ 93.521606] nix-grpc-daemon[867]: time=2026-09-22T11:22:56.753Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5506worker1 # [ 93.523354] nix-grpc-daemon[867]: time=2026-09-22T11:22:56.755Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5507lb # [ 93.575669] niks3-server[1047]: 2026/09/22 11:22:56 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5508lb # [ 93.584074] niks3-server[1047]: 2026/09/22 11:22:56 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5509lb # [ 93.589640] niks3-server[1047]: 2026/09/22 11:22:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5510lb # [ 93.592399] niks3-server[1047]: 2026/09/22 11:22:57 INFO Signed narinfos id=22 count=15511worker1 # [ 93.572417] nix-grpc-daemon[867]: time=2026-09-22T11:22:56.804Z level=INFO msg="Uploading 1 narinfos"5512lb # [ 93.595343] niks3-server[1047]: 2026/09/22 11:22:57 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5513lb # [ 93.600285] niks3-server[1047]: 2026/09/22 11:22:57 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5514lb # [ 93.604128] niks3-server[1047]: 2026/09/22 11:22:57 INFO Completed upload id=225515worker1 # [ 93.582635] nix-grpc-daemon[867]: time=2026-09-22T11:22:56.815Z level=INFO msg="Upload complete. (256ms)"5516lb # [ 93.605790] niks3-server[1047]: 2026/09/22 11:22:57 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5517worker1 # [ 93.587374] nix-grpc-daemon[786]: ts=2026-09-22T11:22:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=12 outputs=15518worker1 # [ 93.598392] nix-daemon[855]: reaped child process 1159, status = succeeded5519worker2 # [ 93.648265] nix-grpc-daemon[789]: ts=2026-09-22T11:22:57Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:362445520worker2 # [ 93.663208] nix-grpc-daemon[789]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5521worker1 # [ 93.681626] nix-grpc-daemon[786]: ts=2026-09-22T11:22:56Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:361345522worker1 # [ 93.686867] nix-grpc-daemon[786]: ts=2026-09-22T11:22:56Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=0 paths=1 nar_bytes_out=1285523worker1 # [ 93.925523] nix-grpc-daemon[786]: ts=2026-09-22T11:22:57Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:361345524worker1 # [ 93.932895] nix-grpc-daemon[786]: ts=2026-09-22T11:22:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361345525worker1 # [ 93.942304] nix-grpc-daemon[867]: time=2026-09-22T11:22:57.174Z level=INFO msg="All 1 paths already cached"5526worker2 # [ 93.967645] nix-grpc-daemon[789]: ts=2026-09-22T11:22:58Z level=debug event=want client=12 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05527worker2 # [ 93.969711] nix-grpc-daemon[789]: ts=2026-09-22T11:22:58Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=15528worker1 # [ 93.962557] nix-grpc-daemon[786]: ts=2026-09-22T11:22:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345529worker1 # [ 93.974762] nix-grpc-daemon[786]: ts=2026-09-22T11:22:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361345530worker1 # [ 93.986569] nix-grpc-daemon[867]: time=2026-09-22T11:22:57.218Z level=INFO msg="All 1 paths already cached"5531worker1 # [ 93.993670] nix-grpc-daemon[786]: ts=2026-09-22T11:22:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361345532worker1 # [ 94.003157] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5533worker1 # [ 94.178696] nix-daemon[855]: accepted connection from pid 1184, user nix-grpc-daemon (trusted)5534worker1 # [ 94.191412] nix-daemon[855]: reaped child process 1187, status = succeeded5535worker1 # [ 94.341097] nix-daemon[855]: accepted connection from pid 1189, user nix-grpc-daemon (trusted)5536worker1 # [ 94.353211] nix-daemon[855]: reaped child process 1192, status = succeeded5537lb # [ 94.385473] niks3-server[1047]: 2026/09/22 11:22:57 INFO Received uploads request method=POST path=/api/pending_closures5538worker1 # [ 94.372605] nix-grpc-daemon[867]: time=2026-09-22T11:22:57.604Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5539worker1 # [ 94.373945] nix-grpc-daemon[867]: time=2026-09-22T11:22:57.604Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5540lb # [ 94.409755] niks3-server[1047]: 2026/09/22 11:22:57 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5541lb # [ 94.422273] niks3-server[1047]: 2026/09/22 11:22:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5542lb # [ 94.428747] niks3-server[1047]: 2026/09/22 11:22:57 INFO Signed narinfos id=23 count=15543worker1 # [ 94.410321] nix-grpc-daemon[867]: time=2026-09-22T11:22:57.642Z level=INFO msg="Uploading 1 narinfos"5544lb # [ 94.433881] niks3-server[1047]: 2026/09/22 11:22:57 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5545lb # [ 94.443666] niks3-server[1047]: 2026/09/22 11:22:57 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5546worker1 # [ 94.428746] nix-grpc-daemon[867]: time=2026-09-22T11:22:57.661Z level=INFO msg="Upload complete. (413ms)"5547lb # [ 94.450364] niks3-server[1047]: 2026/09/22 11:22:57 INFO Completed upload id=235548worker1 # [ 94.430842] nix-grpc-daemon[786]: ts=2026-09-22T11:22:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=0 paths=1 nar_bytes_in=6245549lb # [ 94.455509] niks3-server[1047]: 2026/09/22 11:22:57 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5550worker1 # [ 94.436832] nix-daemon[855]: reaped child process 1182, status = succeeded5551worker1 # [ 94.438400] nix-grpc-daemon[786]: ts=2026-09-22T11:22:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345552worker1 # [ 94.442447] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5553worker1 # [ 94.447841] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5554worker1 # [ 96.072700] nix-daemon[855]: reaped child process 1196, status = succeeded5555worker1 # [ 96.227870] nix-daemon[855]: accepted connection from pid 1212, user nix-grpc-daemon (trusted)5556worker1 # [ 96.240656] nix-daemon[855]: reaped child process 1215, status = succeeded5557lb # [ 96.280364] niks3-server[1047]: 2026/09/22 11:22:59 INFO Received uploads request method=POST path=/api/pending_closures5558worker1 # [ 96.267982] nix-grpc-daemon[867]: time=2026-09-22T11:22:59.500Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5559worker1 # [ 96.269719] nix-grpc-daemon[867]: time=2026-09-22T11:22:59.502Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5560lb # [ 96.317795] niks3-server[1047]: 2026/09/22 11:22:59 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5561lb # [ 96.331913] niks3-server[1047]: 2026/09/22 11:22:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5562lb # [ 96.336328] niks3-server[1047]: 2026/09/22 11:22:59 INFO Signed narinfos id=24 count=15563worker1 # [ 96.316970] nix-grpc-daemon[867]: time=2026-09-22T11:22:59.549Z level=INFO msg="Uploading 1 narinfos"5564lb # [ 96.343538] niks3-server[1047]: 2026/09/22 11:22:59 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5565lb # [ 96.347469] niks3-server[1047]: 2026/09/22 11:22:59 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5566lb # [ 96.356268] niks3-server[1047]: 2026/09/22 11:22:59 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5567lb # [ 96.362852] niks3-server[1047]: 2026/09/22 11:22:59 INFO Completed upload id=245568lb # [ 96.364602] niks3-server[1047]: 2026/09/22 11:22:59 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5569worker1 # [ 96.342636] nix-grpc-daemon[867]: time=2026-09-22T11:22:59.575Z level=INFO msg="Upload complete. (285ms)"5570worker1 # [ 96.348840] nix-grpc-daemon[786]: ts=2026-09-22T11:22:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=13 outputs=15571worker1 # [ 96.358297] nix-grpc-daemon[786]: ts=2026-09-22T11:22:59Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:361345572worker1 # [ 96.367700] nix-daemon[855]: reaped child process 1194, status = succeeded5573worker2 # [ 96.388510] nix-grpc-daemon[789]: ts=2026-09-22T11:23:00Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:362445574worker2 # [ 96.395493] nix-grpc-daemon[789]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...5575worker2 # [ 96.403313] nix-grpc-daemon[789]: ts=2026-09-22T11:23:00Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_out=1765576worker1 # [ 96.678476] nix-grpc-daemon[786]: ts=2026-09-22T11:22:59Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:361345577worker2 # [ 96.695423] nix-grpc-daemon[789]: ts=2026-09-22T11:23:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:362445578worker2 # [ 96.705104] nix-grpc-daemon[928]: time=2026-09-22T11:23:00.774Z level=INFO msg="All 2 paths already cached"5579worker2 # [ 96.719228] nix-grpc-daemon[789]: ts=2026-09-22T11:23:00Z level=debug event=want client=13 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05580worker2 # [ 96.723164] nix-grpc-daemon[789]: ts=2026-09-22T11:23:00Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=15581worker1 # [ 96.760461] nix-grpc-daemon[786]: ts=2026-09-22T11:22:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345582lb # [ 96.781963] envoy[899]: conn peer=[2001:db8:1::1]:44778 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2956 tx=814 ms=41705583lb # [ 96.786649] envoy[899]: conn peer=[2001:db8:1::1]:44782 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1440 tx=1197 ms=38275584lb # [ 96.790237] envoy[899]: conn peer=[2001:db8:1::1]:43382 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=275585lb # [ 96.794951] envoy[899]: conn peer=[2001:db8:1::1]:44790 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3421 tx=1960 ms=19385586lb # [ 96.798622] envoy[899]: conn peer=[2001:db8:1::1]:43398 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=379 ms=675587lb # [ 96.802395] envoy[899]: conn peer=[2001:db8:1::1]:43392 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3893 tx=2154 ms=25235588worker1 # [ 96.789258] nix-grpc-daemon[786]: ts=2026-09-22T11:23:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361345589worker1 # [ 96.798027] nix-grpc-daemon[867]: time=2026-09-22T11:23:00.030Z level=INFO msg="All 2 paths already cached"5590worker1 # [ 96.803108] nix-grpc-daemon[786]: ts=2026-09-22T11:23:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361345591worker1 # [ 96.806995] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5592worker1 # [ 96.957373] nix-daemon[855]: accepted connection from pid 1219, user nix-grpc-daemon (trusted)5593worker1 # [ 96.969220] nix-daemon[855]: reaped child process 1222, status = succeeded5594worker1 # [ 97.129249] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted)5595worker1 # [ 97.140419] nix-daemon[855]: reaped child process 1227, status = succeeded5596lb # [ 97.178837] niks3-server[1047]: 2026/09/22 11:23:00 INFO Received uploads request method=POST path=/api/pending_closures5597worker1 # [ 97.165120] nix-grpc-daemon[867]: time=2026-09-22T11:23:00.397Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5598worker1 # [ 97.166588] nix-grpc-daemon[867]: time=2026-09-22T11:23:00.399Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5599lb # [ 97.206479] niks3-server[1047]: 2026/09/22 11:23:00 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5600lb # [ 97.218513] niks3-server[1047]: 2026/09/22 11:23:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5601worker1 # [ 97.201383] nix-grpc-daemon[867]: time=2026-09-22T11:23:00.433Z level=INFO msg="Uploading 1 narinfos"5602lb # [ 97.223569] niks3-server[1047]: 2026/09/22 11:23:00 INFO Signed narinfos id=25 count=15603lb # [ 97.229613] niks3-server[1047]: 2026/09/22 11:23:00 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5604lb # [ 97.242098] niks3-server[1047]: 2026/09/22 11:23:00 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5605worker1 # [ 97.225604] nix-grpc-daemon[867]: time=2026-09-22T11:23:00.458Z level=INFO msg="Upload complete. (412ms)"5606lb # [ 97.247238] niks3-server[1047]: 2026/09/22 11:23:00 INFO Completed upload id=255607lb # [ 97.249620] niks3-server[1047]: 2026/09/22 11:23:00 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5608worker1 # [ 97.228350] nix-grpc-daemon[786]: ts=2026-09-22T11:23:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=0 paths=1 nar_bytes_in=7525609worker1 # [ 97.234788] nix-grpc-daemon[786]: ts=2026-09-22T11:23:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345610worker1 # [ 97.241935] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5611worker1 # [ 97.245837] nix-daemon[855]: reaped child process 1217, status = succeeded5612worker1 # [ 97.250158] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5613worker1 # [ 98.067902] nix-daemon[855]: reaped child process 1231, status = succeeded5614worker1 # [ 98.219908] nix-daemon[855]: accepted connection from pid 1247, user nix-grpc-daemon (trusted)5615worker1 # [ 98.232173] nix-daemon[855]: reaped child process 1250, status = succeeded5616lb # [ 98.271251] niks3-server[1047]: 2026/09/22 11:23:01 INFO Received uploads request method=POST path=/api/pending_closures5617worker1 # [ 98.259320] nix-grpc-daemon[867]: time=2026-09-22T11:23:01.491Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5618worker1 # [ 98.260774] nix-grpc-daemon[867]: time=2026-09-22T11:23:01.493Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5619lb # [ 98.307262] niks3-server[1047]: 2026/09/22 11:23:01 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5620lb # [ 98.315092] niks3-server[1047]: 2026/09/22 11:23:01 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5621lb # [ 98.323907] niks3-server[1047]: 2026/09/22 11:23:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5622lb # [ 98.327749] niks3-server[1047]: 2026/09/22 11:23:01 INFO Signed narinfos id=26 count=15623worker1 # [ 98.308358] nix-grpc-daemon[867]: time=2026-09-22T11:23:01.540Z level=INFO msg="Uploading 1 narinfos"5624lb # [ 98.334361] niks3-server[1047]: 2026/09/22 11:23:01 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5625lb # [ 98.339554] niks3-server[1047]: 2026/09/22 11:23:01 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5626lb # [ 98.342656] niks3-server[1047]: 2026/09/22 11:23:01 INFO Completed upload id=265627worker1 # [ 98.321221] nix-grpc-daemon[867]: time=2026-09-22T11:23:01.553Z level=INFO msg="Upload complete. (264ms)"5628lb # [ 98.345165] niks3-server[1047]: 2026/09/22 11:23:01 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5629worker1 # [ 98.327106] nix-grpc-daemon[786]: ts=2026-09-22T11:23:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=14 outputs=15630worker1 # [ 98.337134] nix-daemon[855]: reaped child process 1229, status = succeeded5631worker1 # [ 98.372517] nix-grpc-daemon[786]: ts=2026-09-22T11:23:01Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:361345632worker2 # [ 98.402785] nix-grpc-daemon[789]: ts=2026-09-22T11:23:02Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:362445633worker2 # [ 98.411709] nix-grpc-daemon[789]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...5634worker2 # [ 98.420836] nix-grpc-daemon[789]: ts=2026-09-22T11:23:02Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_out=2325635client # [ 98.531941] nix-daemon[1104]: reaped child process 1106, status = succeeded5636client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.32 seconds)5637client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5638client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.03 seconds)5639client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5640client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.04 seconds)5641client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5642client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)5643(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.42 seconds)5644subtest: an input only one worker has still reaches the builder5645client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only5646client # this derivation will be built:5647client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5648client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5649client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.43 seconds)5650client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5651client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.20 seconds)5652worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5653worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.20 seconds)5654worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5655worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)5656client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&25657client # [ 99.709565] nix-daemon[1104]: accepted connection from pid 1187, user root (trusted)5658client # this derivation will be built:5659client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5660worker2 # [ 99.926184] nix-grpc-daemon[789]: ts=2026-09-22T11:23:03Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:362445661client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5662worker1 # [ 99.917738] nix-grpc-daemon[786]: ts=2026-09-22T11:23:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361345663worker1 # [ 100.071185] nix-daemon[855]: accepted connection from pid 1257, user nix-grpc-daemon (trusted)5664worker1 # [ 100.083611] nix-daemon[855]: reaped child process 1260, status = succeeded5665lb # [ 100.119616] niks3-server[1047]: 2026/09/22 11:23:03 INFO Received uploads request method=POST path=/api/pending_closures5666worker1 # [ 100.106855] nix-grpc-daemon[867]: time=2026-09-22T11:23:03.339Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5667worker1 # [ 100.108695] nix-grpc-daemon[867]: time=2026-09-22T11:23:03.340Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5668lb # [ 100.149606] niks3-server[1047]: 2026/09/22 11:23:03 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5669lb # [ 100.159594] niks3-server[1047]: 2026/09/22 11:23:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5670worker1 # [ 100.142488] nix-grpc-daemon[867]: time=2026-09-22T11:23:03.374Z level=INFO msg="Uploading 1 narinfos"5671lb # [ 100.164113] niks3-server[1047]: 2026/09/22 11:23:03 INFO Signed narinfos id=27 count=15672lb # [ 100.171113] niks3-server[1047]: 2026/09/22 11:23:03 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5673lb # [ 100.181969] niks3-server[1047]: 2026/09/22 11:23:03 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5674lb # [ 100.188149] niks3-server[1047]: 2026/09/22 11:23:03 INFO Completed upload id=275675worker1 # [ 100.166628] nix-grpc-daemon[867]: time=2026-09-22T11:23:03.399Z level=INFO msg="Upload complete. (247ms)"5676lb # [ 100.190411] niks3-server[1047]: 2026/09/22 11:23:03 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5677worker2 # [ 100.198159] nix-grpc-daemon[789]: ts=2026-09-22T11:23:04Z level=debug event=want client=14 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05678worker2 # [ 100.200778] nix-grpc-daemon[789]: ts=2026-09-22T11:23:04Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=15679worker2 # [ 100.243045] nix-grpc-daemon[789]: ts=2026-09-22T11:23:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445680worker2 # [ 100.252122] nix-grpc-daemon[789]: ts=2026-09-22T11:23:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:362445681client # copying 2 paths to 'lb:50051'...5682worker2 # [ 100.263451] nix-grpc-daemon[789]: ts=2026-09-22T11:23:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:362445683client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5684client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5685worker2 # [ 100.273449] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5686worker2 # [ 100.425689] nix-daemon[877]: accepted connection from pid 1190, user nix-grpc-daemon (trusted)5687worker2 # [ 100.440567] nix-daemon[877]: reaped child process 1193, status = succeeded5688worker2 # [ 100.582762] nix-daemon[877]: accepted connection from pid 1195, user nix-grpc-daemon (trusted)5689worker2 # [ 100.597495] nix-daemon[877]: reaped child process 1198, status = succeeded5690lb # [ 100.619569] niks3-server[1047]: 2026/09/22 11:23:04 INFO Received uploads request method=POST path=/api/pending_closures5691lb # [ 100.625450] niks3-server[1047]: 2026/09/22 11:23:04 INFO Received uploads request method=POST path=/api/pending_closures5692worker2 # [ 100.620380] nix-grpc-daemon[928]: time=2026-09-22T11:23:04.689Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5693worker2 # [ 100.621733] nix-grpc-daemon[928]: time=2026-09-22T11:23:04.689Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5694worker2 # [ 100.623354] nix-grpc-daemon[928]: time=2026-09-22T11:23:04.690Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5695lb # [ 100.642024] niks3-server[1047]: 2026/09/22 11:23:04 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5696lb # [ 100.646181] niks3-server[1047]: 2026/09/22 11:23:04 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5697lb # [ 100.648898] niks3-server[1047]: 2026/09/22 11:23:04 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5698lb # [ 100.652943] niks3-server[1047]: 2026/09/22 11:23:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5699lb # [ 100.654642] niks3-server[1047]: 2026/09/22 11:23:04 INFO Signed narinfos id=28 count=15700lb # [ 100.655980] niks3-server[1047]: 2026/09/22 11:23:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5701lb # [ 100.658525] niks3-server[1047]: 2026/09/22 11:23:04 INFO Signed narinfos id=29 count=25702worker2 # [ 100.648723] nix-grpc-daemon[928]: time=2026-09-22T11:23:04.718Z level=INFO msg="Uploading 3 narinfos"5703lb # [ 100.663133] niks3-server[1047]: 2026/09/22 11:23:04 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5704lb # [ 100.670625] niks3-server[1047]: 2026/09/22 11:23:04 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5705lb # [ 100.675442] niks3-server[1047]: 2026/09/22 11:23:04 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5706lb # [ 100.677442] niks3-server[1047]: 2026/09/22 11:23:04 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5707lb # [ 100.680812] niks3-server[1047]: 2026/09/22 11:23:04 INFO Completed upload id=285708lb # [ 100.682301] niks3-server[1047]: 2026/09/22 11:23:04 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5709lb # [ 100.684318] niks3-server[1047]: 2026/09/22 11:23:04 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5710lb # [ 100.686541] niks3-server[1047]: 2026/09/22 11:23:04 INFO Completed upload id=295711worker2 # [ 100.676515] nix-grpc-daemon[928]: time=2026-09-22T11:23:04.746Z level=INFO msg="Upload complete. (395ms)"5712worker2 # [ 100.679608] nix-grpc-daemon[789]: ts=2026-09-22T11:23:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=2 nar_bytes_in=9765713worker2 # [ 100.683419] nix-grpc-daemon[789]: ts=2026-09-22T11:23:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445714worker2 # [ 100.691397] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5715worker2 # [ 100.693578] nix-daemon[877]: reaped child process 1188, status = succeeded5716worker2 # [ 100.700186] nix-grpc-daemon[789]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5717worker2 # [ 100.707052] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5718client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5719client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5720client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5721worker2 # [ 101.720735] nix-daemon[877]: reaped child process 1203, status = succeeded5722worker2 # [ 101.868727] nix-daemon[877]: accepted connection from pid 1219, user nix-grpc-daemon (trusted)5723worker2 # [ 101.880797] nix-daemon[877]: reaped child process 1222, status = succeeded5724lb # [ 101.906246] niks3-server[1047]: 2026/09/22 11:23:05 INFO Received uploads request method=POST path=/api/pending_closures5725worker2 # [ 101.904134] nix-grpc-daemon[928]: time=2026-09-22T11:23:05.974Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5726lb # [ 101.934569] niks3-server[1047]: 2026/09/22 11:23:05 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5727lb # [ 101.941879] niks3-server[1047]: 2026/09/22 11:23:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5728lb # [ 101.946118] niks3-server[1047]: 2026/09/22 11:23:05 INFO Signed narinfos id=30 count=15729worker2 # [ 101.938837] nix-grpc-daemon[928]: time=2026-09-22T11:23:06.009Z level=INFO msg="Uploading 1 narinfos"5730lb # [ 101.955764] niks3-server[1047]: 2026/09/22 11:23:05 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5731lb # [ 101.967952] niks3-server[1047]: 2026/09/22 11:23:05 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5732lb # [ 101.974224] niks3-server[1047]: 2026/09/22 11:23:05 INFO Completed upload id=305733worker2 # [ 101.963925] nix-grpc-daemon[928]: time=2026-09-22T11:23:06.034Z level=INFO msg="Upload complete. (248ms)"5734lb # [ 101.976914] niks3-server[1047]: 2026/09/22 11:23:05 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5735worker2 # [ 101.968512] nix-grpc-daemon[789]: ts=2026-09-22T11:23:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=15 outputs=15736worker2 # [ 101.980208] nix-daemon[877]: reaped child process 1200, status = succeeded5737client # copying 1 paths...5738worker1 # [ 102.006467] nix-grpc-daemon[786]: ts=2026-09-22T11:23:05Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:361345739worker1 # [ 102.015120] nix-grpc-daemon[786]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5740client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5741worker1 # [ 102.041658] nix-grpc-daemon[786]: ts=2026-09-22T11:23:05Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:361345742worker1 # [ 102.043938] nix-grpc-daemon[786]: ts=2026-09-22T11:23:05Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=0 paths=1 nar_bytes_out=1205743client # [ 102.156061] nix-daemon[1104]: reaped child process 1190, status = succeeded5744client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.67 seconds)5745client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25746client # [ 102.336193] nix-daemon[1104]: accepted connection from pid 1209, user root (trusted)5747client # this derivation will be built:5748client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5749worker1 # [ 102.498262] nix-grpc-daemon[786]: ts=2026-09-22T11:23:05Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:361345750client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5751worker1 # [ 102.501264] nix-grpc-daemon[786]: ts=2026-09-22T11:23:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361345752worker1 # [ 102.506832] nix-grpc-daemon[867]: time=2026-09-22T11:23:05.739Z level=INFO msg="All 1 paths already cached"5753worker2 # [ 102.529040] nix-grpc-daemon[789]: ts=2026-09-22T11:23:06Z level=debug event=want client=15 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05754worker2 # [ 102.531258] nix-grpc-daemon[789]: ts=2026-09-22T11:23:06Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15755worker1 # [ 102.521597] nix-grpc-daemon[786]: ts=2026-09-22T11:23:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345756worker1 # [ 102.529513] nix-grpc-daemon[786]: ts=2026-09-22T11:23:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361345757client # copying 1 paths to 'lb:50051'...5758client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5759worker1 # [ 102.541520] nix-grpc-daemon[786]: ts=2026-09-22T11:23:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361345760worker1 # [ 102.545909] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5761worker1 # [ 102.556555] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 10 ms (attempt 1/5)5762worker1 # [ 102.569049] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 194 ms (attempt 2/5)5763worker1 # [ 102.765792] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 189 ms (attempt 3/5)5764worker1 # [ 102.957891] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 205 ms (attempt 4/5)5765worker1 # [ 103.170428] nix-grpc-daemon[786]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5766worker1 # [ 103.174829] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5767worker1 # [ 103.180123] nix-grpc-daemon[786]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5768worker1 # [ 103.381188] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5769worker1 # [ 103.393421] nix-daemon[855]: reaped child process 1273, status = succeeded5770worker1 # [ 103.539256] nix-daemon[855]: accepted connection from pid 1275, user nix-grpc-daemon (trusted)5771worker1 # [ 103.549732] nix-daemon[855]: reaped child process 1278, status = succeeded5772lb # [ 103.587431] niks3-server[1047]: 2026/09/22 11:23:07 INFO Received uploads request method=POST path=/api/pending_closures5773worker1 # [ 103.576197] nix-grpc-daemon[867]: time=2026-09-22T11:23:06.808Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5774worker1 # [ 103.578104] nix-grpc-daemon[867]: time=2026-09-22T11:23:06.810Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5775lb # [ 103.619273] niks3-server[1047]: 2026/09/22 11:23:07 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5776lb # [ 103.627694] niks3-server[1047]: 2026/09/22 11:23:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5777worker1 # [ 103.610973] nix-grpc-daemon[867]: time=2026-09-22T11:23:06.843Z level=INFO msg="Uploading 1 narinfos"5778lb # [ 103.632806] niks3-server[1047]: 2026/09/22 11:23:07 INFO Signed narinfos id=31 count=15779lb # [ 103.638670] niks3-server[1047]: 2026/09/22 11:23:07 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5780lb # [ 103.648701] niks3-server[1047]: 2026/09/22 11:23:07 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5781lb # [ 103.655328] niks3-server[1047]: 2026/09/22 11:23:07 INFO Completed upload id=315782worker1 # [ 103.633718] nix-grpc-daemon[867]: time=2026-09-22T11:23:06.866Z level=INFO msg="Upload complete. (434ms)"5783worker1 # [ 103.637237] nix-grpc-daemon[786]: ts=2026-09-22T11:23:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=1 paths=1 nar_bytes_in=5605784lb # [ 103.659950] niks3-server[1047]: 2026/09/22 11:23:07 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5785worker1 # [ 103.642544] nix-grpc-daemon[786]: ts=2026-09-22T11:23:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361345786worker1 # [ 103.652950] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5787worker1 # [ 103.657494] nix-daemon[855]: reaped child process 1263, status = succeeded5788worker1 # [ 103.661781] nix-daemon[855]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5789client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5790client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5791client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5792worker1 # [ 104.486310] nix-daemon[855]: reaped child process 1282, status = succeeded5793worker1 # [ 104.622435] nix-daemon[855]: accepted connection from pid 1298, user nix-grpc-daemon (trusted)5794worker1 # [ 104.633940] nix-daemon[855]: reaped child process 1301, status = succeeded5795lb # [ 104.667928] niks3-server[1047]: 2026/09/22 11:23:08 INFO Received uploads request method=POST path=/api/pending_closures5796worker1 # [ 104.654143] nix-grpc-daemon[867]: time=2026-09-22T11:23:07.886Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5797lb # [ 104.696521] niks3-server[1047]: 2026/09/22 11:23:08 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5798lb # [ 104.705462] niks3-server[1047]: 2026/09/22 11:23:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5799lb # [ 104.711258] niks3-server[1047]: 2026/09/22 11:23:08 INFO Signed narinfos id=32 count=15800worker1 # [ 104.692633] nix-grpc-daemon[867]: time=2026-09-22T11:23:07.925Z level=INFO msg="Uploading 1 narinfos"5801lb # [ 104.717051] niks3-server[1047]: 2026/09/22 11:23:08 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5802lb # [ 104.729401] niks3-server[1047]: 2026/09/22 11:23:08 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5803worker1 # [ 104.712418] nix-grpc-daemon[867]: time=2026-09-22T11:23:07.944Z level=INFO msg="Upload complete. (244ms)"5804lb # [ 104.734064] niks3-server[1047]: 2026/09/22 11:23:08 INFO Completed upload id=325805worker1 # [ 104.714931] nix-grpc-daemon[786]: ts=2026-09-22T11:23:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36134 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=16 outputs=15806lb # [ 104.738925] niks3-server[1047]: 2026/09/22 11:23:08 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5807client # copying 1 paths...5808worker1 # [ 104.724571] nix-grpc-daemon[786]: ts=2026-09-22T11:23:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:361345809worker1 # [ 104.727608] nix-daemon[855]: reaped child process 1280, status = succeeded5810client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5811worker2 # [ 104.757052] nix-grpc-daemon[789]: ts=2026-09-22T11:23:08Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:362445812worker2 # [ 104.765271] nix-grpc-daemon[789]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5813worker2 # [ 104.773390] nix-grpc-daemon[789]: ts=2026-09-22T11:23:08Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_out=1205814client # [ 104.860316] nix-daemon[1104]: reaped child process 1212, status = succeeded5815client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.70 seconds)5816client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25817client # [ 105.034232] nix-daemon[1104]: accepted connection from pid 1231, user root (trusted)5818client # this derivation will be built:5819client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5820worker2 # [ 105.217770] nix-grpc-daemon[789]: ts=2026-09-22T11:23:09Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:362445821client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5822worker2 # [ 105.220769] nix-grpc-daemon[789]: ts=2026-09-22T11:23:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:362445823worker2 # [ 105.226594] nix-grpc-daemon[928]: time=2026-09-22T11:23:09.296Z level=INFO msg="All 1 paths already cached"5824worker2 # [ 105.235845] nix-grpc-daemon[789]: ts=2026-09-22T11:23:09Z level=debug event=want client=16 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05825worker2 # [ 105.238108] nix-grpc-daemon[789]: ts=2026-09-22T11:23:09Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15826worker2 # [ 105.240516] nix-grpc-daemon[789]: ts=2026-09-22T11:23:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445827worker2 # [ 105.247759] nix-grpc-daemon[789]: ts=2026-09-22T11:23:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:362445828worker2 # [ 105.255377] nix-grpc-daemon[928]: time=2026-09-22T11:23:09.325Z level=INFO msg="All 1 paths already cached"5829client # copying 1 paths to 'lb:50051'...5830client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5831worker2 # [ 105.259861] nix-grpc-daemon[789]: ts=2026-09-22T11:23:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:362445832worker2 # [ 105.265601] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5833worker2 # [ 105.396442] nix-daemon[877]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5834worker2 # [ 105.410320] nix-daemon[877]: reaped child process 1234, status = succeeded5835worker2 # [ 105.543403] nix-daemon[877]: accepted connection from pid 1236, user nix-grpc-daemon (trusted)5836worker2 # [ 105.556623] nix-daemon[877]: reaped child process 1239, status = succeeded5837lb # [ 105.578128] niks3-server[1047]: 2026/09/22 11:23:08 INFO Received uploads request method=POST path=/api/pending_closures5838worker2 # [ 105.577212] nix-grpc-daemon[928]: time=2026-09-22T11:23:09.647Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5839worker2 # [ 105.578549] nix-grpc-daemon[928]: time=2026-09-22T11:23:09.647Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5840lb # [ 105.604668] niks3-server[1047]: 2026/09/22 11:23:09 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5841lb # [ 105.614748] niks3-server[1047]: 2026/09/22 11:23:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5842lb # [ 105.620991] niks3-server[1047]: 2026/09/22 11:23:09 INFO Signed narinfos id=33 count=15843worker2 # [ 105.613868] nix-grpc-daemon[928]: time=2026-09-22T11:23:09.684Z level=INFO msg="Uploading 1 narinfos"5844lb # [ 105.628258] niks3-server[1047]: 2026/09/22 11:23:09 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5845lb # [ 105.637641] niks3-server[1047]: 2026/09/22 11:23:09 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5846worker2 # [ 105.632872] nix-grpc-daemon[928]: time=2026-09-22T11:23:09.703Z level=INFO msg="Upload complete. (362ms)"5847lb # [ 105.643127] niks3-server[1047]: 2026/09/22 11:23:09 INFO Completed upload id=335848worker2 # [ 105.636070] nix-grpc-daemon[789]: ts=2026-09-22T11:23:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_in=5605849lb # [ 105.646607] niks3-server[1047]: 2026/09/22 11:23:09 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5850worker2 # [ 105.640545] nix-grpc-daemon[789]: ts=2026-09-22T11:23:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:362445851worker2 # [ 105.647323] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5852worker2 # [ 105.649720] nix-daemon[877]: reaped child process 1229, status = succeeded5853client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5854worker2 # [ 105.655082] nix-daemon[877]: accepted connection from pid 789, user nix-grpc-daemon (trusted)5855client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5856client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5857worker2 # [ 106.687636] nix-daemon[877]: reaped child process 1243, status = succeeded5858lb # [ 106.781351] envoy[899]: conn peer=[2001:db8:1::1]:43414 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4353 tx=2410 ms=17845859lb # [ 106.783137] envoy[899]: conn peer=[2001:db8:1::1]:43426 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=411 ms=745860lb # [ 106.784874] envoy[899]: conn peer=[2001:db8:1::1]:43432 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3879 tx=2059 ms=21825861lb # [ 106.786491] envoy[899]: conn peer=[2001:db8:1::1]:43442 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=421 tx=328 ms=545862lb # [ 106.788060] envoy[899]: conn peer=[2001:db8:1::1]:43454 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3740 tx=2119 ms=23115863lb # [ 106.789757] envoy[899]: conn peer=[2001:db8:1::1]:42414 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=329 ms=685864worker2 # [ 106.827978] nix-daemon[877]: accepted connection from pid 1259, user nix-grpc-daemon (trusted)5865worker2 # [ 106.841849] nix-daemon[877]: reaped child process 1262, status = succeeded5866lb # [ 106.864450] niks3-server[1047]: 2026/09/22 11:23:10 INFO Received uploads request method=POST path=/api/pending_closures5867worker2 # [ 106.860397] nix-grpc-daemon[928]: time=2026-09-22T11:23:10.929Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5868lb # [ 106.877716] niks3-server[1047]: 2026/09/22 11:23:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5869lb # [ 106.880319] niks3-server[1047]: 2026/09/22 11:23:10 INFO Signed narinfos id=34 count=15870worker2 # [ 106.870406] nix-grpc-daemon[928]: time=2026-09-22T11:23:10.940Z level=INFO msg="Uploading 1 narinfos"5871lb # [ 106.883596] niks3-server[1047]: 2026/09/22 11:23:10 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5872lb # [ 106.886066] niks3-server[1047]: 2026/09/22 11:23:10 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5873lb # [ 106.890152] niks3-server[1047]: 2026/09/22 11:23:10 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5874lb # [ 106.892827] niks3-server[1047]: 2026/09/22 11:23:10 INFO Completed upload id=345875worker2 # [ 106.882782] nix-grpc-daemon[928]: time=2026-09-22T11:23:10.953Z level=INFO msg="Upload complete. (220ms)"5876lb # [ 106.895142] niks3-server[1047]: 2026/09/22 11:23:10 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5877worker2 # [ 106.885476] nix-grpc-daemon[789]: ts=2026-09-22T11:23:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=17 outputs=15878worker2 # [ 106.893022] nix-daemon[877]: reaped child process 1241, status = succeeded5879client # copying 1 paths...5880worker2 # [ 106.931413] nix-grpc-daemon[789]: ts=2026-09-22T11:23:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:362445881client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5882worker2 # [ 106.950592] nix-grpc-daemon[789]: ts=2026-09-22T11:23:11Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:362445883worker2 # [ 106.954614] nix-grpc-daemon[789]: ts=2026-09-22T11:23:11Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36244 duration_s=0 paths=1 nar_bytes_out=1205884client # [ 107.056397] nix-daemon[1104]: reaped child process 1234, status = succeeded5885client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.20 seconds)5886(finished: subtest: an input only one worker has still reaches the builder, in 8.43 seconds)5887subtest: upload whose reference the worker lacks is completed from the cache5888client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5889client # this derivation will be built:5890client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5891client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5892client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.36 seconds)5893client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5894client # this derivation will be built:5895client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5896client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5897client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.37 seconds)5898client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5899client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.16 seconds)5900worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5901worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.18 seconds)5902worker2 # [ 108.192611] systemd[1]: Stopping Nix worker-protocol over gRPC...5903worker2 # [ 108.435878] nix-grpc-daemon[789]: ts=2026-09-22T11:23:12Z level=info event=signal_exit5904lb # [ 108.727909] niks3-server[1047]: 2026/09/22 11:23:12 INFO lead: released remote=192.168.1.4:415445905worker1 # [ 109.709121] nix-grpc-daemon[786]: ts=2026-09-22T11:23:12Z level=info event=scheduler_restarting addr=lb:500515906worker2 # [ 109.725319] nix-grpc-daemon[789]: ts=2026-09-22T11:23:13Z level=info event=scheduler_restarting5907worker2 # [ 109.728378] nix-grpc-daemon[789]: ts=2026-09-22T11:23:13Z level=info event=scheduler_restarting addr=lb:500515908worker2 # [ 109.731874] nix-grpc-daemon[789]: ts=2026-09-22T11:23:13Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:362405909worker2 # [ 109.735219] nix-grpc-daemon[789]: ts=2026-09-22T11:23:13Z level=info event=worker_gone addr=192.168.1.4:500515910worker2 # [ 109.738268] nix-grpc-daemon[789]: ts=2026-09-22T11:23:13Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:362405911worker2 # [ 109.742433] nix-grpc-daemon[789]: ts=2026-09-22T11:23:13Z level=info event=worker_gone addr=192.168.1.3:500515912worker2 # [ 109.761649] nix-daemon[877]: reaped child process 879, status = succeeded5913worker1 # [ 109.909431] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=info event=scheduler_connected addr=lb:500515914worker1 # [ 109.913971] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5915worker2 # [ 109.936279] nix-grpc-daemon[789]: ts=2026-09-22T11:23:14Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5916worker1 # [ 110.114319] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=info event=scheduler_connected addr=lb:500515917worker1 # [ 110.116091] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5918worker2 # [ 110.169919] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5919worker2 # [ 110.171535] systemd[1]: Stopped Nix worker-protocol over gRPC.5920worker2 # [ 110.173315] systemd[1]: nix-grpc-daemon.service: Consumed 2.626s CPU time over 1min 33.028s wall clock time, 58.4M memory peak, 97.1K incoming IP traffic, 284.4K outgoing IP traffic.5921worker2 # [ 110.177330] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5922worker2 # [ 110.179063] systemd[1]: Closed Nix worker-protocol over gRPC.5923worker2 # [ 110.180469] systemd[1]: nix-grpc-daemon.socket: Consumed 279us CPU time over 1min 40.849s wall clock time, 256K memory peak, 114.1K incoming IP traffic, 96.2K outgoing IP traffic.5924lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5925lb: must succeed: curl -sf localhost:9901/clusters5926lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5927lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.04 seconds)5928client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25929worker1 # [ 110.316805] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=info event=scheduler_connected addr=lb:500515930worker1 # [ 110.320710] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5931lb # [ 110.365735] niks3-server[1047]: 2026/09/22 11:23:13 INFO lead: acquired remote=192.168.1.3:439185932worker1 # [ 110.347190] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=info event=scheduler_take_over5933worker1 # [ 110.416593] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:361345934worker1 # [ 110.422988] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:361345935client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5936client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.24 seconds)5937worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5938worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.02 seconds)5939worker2 # [ 110.515797] systemd[1]: Listening on Nix worker-protocol over gRPC.5940worker1 # [ 110.521678] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=info event=scheduler_connected addr=lb:500515941worker1 # [ 110.526820] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5942worker1 # [ 110.572584] systemd[1]: Stopping Nix worker-protocol over gRPC...5943worker2 # [ 110.615918] systemd[1]: Starting scheduler-token.service...5944worker2 # [ 110.674557] systemd[1]: scheduler-token.service: Deactivated successfully.5945worker2 # [ 110.675871] systemd[1]: Finished scheduler-token.service.5946worker2 # [ 110.677327] systemd[1]: scheduler-token.service: Consumed 22ms CPU time over 58ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5947worker2 # [ 110.682291] systemd[1]: Starting Nix worker-protocol over gRPC...5948worker1 # [ 110.729147] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=info event=scheduler_connected addr=lb:500515949worker1 # [ 110.732461] nix-grpc-daemon[786]: ts=2026-09-22T11:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5950worker2 # [ 110.815030] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:14Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5951worker2 # [ 110.817244] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:14Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5952worker2 # [ 110.820785] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:14Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15953worker2 # [ 110.827065] nix-grpc-daemon[1286]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5954worker2 # [ 110.830109] nix-grpc-daemon[1286]: E0000 00:00:1790076194.897336 1286 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5955worker2 # [ 110.834379] nix-grpc-daemon[1286]: E0000 00:00:1790076194.900574 1286 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5956worker2 # [ 110.838136] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:14Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5957worker2 # [ 110.839667] systemd[1]: Started Nix worker-protocol over gRPC.5958worker2 # [ 110.848073] nix-grpc-daemon[1286]: E0000 00:00:1790076194.917995 1327 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5959worker2 # [ 110.861062] nix-grpc-daemon[1286]: E0000 00:00:1790076194.931483 1327 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5960worker2 # [ 110.864212] nix-grpc-daemon[1286]: E0000 00:00:1790076194.933681 1327 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5961worker2 # [ 110.875524] nix-grpc-daemon[1286]: E0000 00:00:1790076194.945961 1327 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5962worker2 # [ 110.883256] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:14Z level=info event=scheduler_connected addr=lb:500515963worker2 # [ 110.884987] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5964worker1 # [ 110.929627] nix-grpc-daemon[786]: ts=2026-09-22T11:23:14Z level=info event=scheduler_connected addr=lb:500515965worker1 # [ 110.933777] nix-grpc-daemon[786]: ts=2026-09-22T11:23:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5966worker2 # [ 111.085580] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:15Z level=info event=scheduler_connected addr=lb:500515967worker2 # [ 111.087366] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5968worker1 # [ 111.087966] nix-grpc-daemon[786]: ts=2026-09-22T11:23:14Z level=info event=signal_exit5969worker1 # [ 111.134743] nix-grpc-daemon[786]: ts=2026-09-22T11:23:14Z level=info event=scheduler_connected addr=lb:500515970worker1 # [ 111.138876] nix-grpc-daemon[786]: ts=2026-09-22T11:23:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5971worker2 # [ 111.288257] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:15Z level=info event=scheduler_connected addr=lb:500515972worker2 # [ 111.290207] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5973worker1 # [ 111.340202] nix-grpc-daemon[786]: ts=2026-09-22T11:23:14Z level=info event=scheduler_connected addr=lb:500515974worker1 # [ 111.344184] nix-grpc-daemon[786]: ts=2026-09-22T11:23:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5975lb # [ 111.410627] niks3-server[1047]: 2026/09/22 11:23:14 INFO lead: released remote=192.168.1.3:439185976worker2 # [ 111.490825] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:15Z level=info event=scheduler_connected addr=lb:500515977worker2 # [ 111.492538] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5978worker1 # [ 111.545498] nix-grpc-daemon[786]: ts=2026-09-22T11:23:14Z level=info event=scheduler_connected addr=lb:500515979worker1 # [ 111.549675] nix-grpc-daemon[786]: ts=2026-09-22T11:23:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5980worker2 # [ 111.693293] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:15Z level=info event=scheduler_connected addr=lb:500515981worker2 # [ 111.694969] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5982worker1 # [ 111.752247] nix-grpc-daemon[786]: ts=2026-09-22T11:23:14Z level=info event=scheduler_connected addr=lb:500515983worker1 # [ 111.755492] nix-grpc-daemon[786]: ts=2026-09-22T11:23:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5984worker2 # [ 111.897501] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:15Z level=info event=scheduler_connected addr=lb:500515985worker2 # [ 111.900599] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5986worker1 # [ 111.952915] nix-grpc-daemon[786]: ts=2026-09-22T11:23:15Z level=info event=scheduler_connected addr=lb:500515987worker1 # [ 111.957386] nix-grpc-daemon[786]: ts=2026-09-22T11:23:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5988worker2 # [ 112.098335] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:16Z level=info event=scheduler_connected addr=lb:500515989worker2 # [ 112.102303] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5990worker1 # [ 112.158288] nix-grpc-daemon[786]: ts=2026-09-22T11:23:15Z level=info event=scheduler_connected addr=lb:500515991worker1 # [ 112.162238] nix-grpc-daemon[786]: ts=2026-09-22T11:23:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5992worker2 # [ 112.303319] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:16Z level=info event=scheduler_connected addr=lb:500515993worker2 # [ 112.307194] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5994worker1 # [ 112.363190] nix-grpc-daemon[786]: ts=2026-09-22T11:23:15Z level=info event=scheduler_connected addr=lb:500515995worker1 # [ 112.367252] nix-grpc-daemon[786]: ts=2026-09-22T11:23:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5996worker1 # [ 112.387618] nix-grpc-daemon[786]: ts=2026-09-22T11:23:15Z level=info event=scheduler_restarting5997worker1 # [ 112.413227] nix-daemon[855]: reaped child process 857, status = succeeded5998worker2 # [ 112.508384] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:16Z level=info event=scheduler_connected addr=lb:500515999worker2 # [ 112.512197] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6000worker1 # [ 112.581179] nix-grpc-daemon[786]: ts=2026-09-22T11:23:15Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6001worker2 # [ 112.713318] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:16Z level=info event=scheduler_connected addr=lb:500516002worker2 # [ 112.717190] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6003worker2 # [ 112.919425] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:16Z level=info event=scheduler_connected addr=lb:500516004worker2 # [ 112.922525] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6005worker2 # [ 113.120256] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:17Z level=info event=scheduler_connected addr=lb:500516006worker2 # [ 113.122041] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6007worker1 # [ 113.262667] nix-grpc-daemon[786]: E0000 00:00:1790076196.493982 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true6008worker1 # [ 113.267562] nix-grpc-daemon[786]: E0000 00:00:1790076196.494259 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6009worker2 # [ 113.323166] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:17Z level=info event=scheduler_connected addr=lb:500516010worker2 # [ 113.327099] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6011worker2 # [ 113.527653] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:17Z level=info event=scheduler_connected addr=lb:500516012worker2 # [ 113.529511] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6013worker2 # [ 113.730279] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:17Z level=info event=scheduler_connected addr=lb:500516014worker2 # [ 113.732116] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6015worker2 # [ 113.933939] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:18Z level=info event=scheduler_connected addr=lb:500516016worker2 # [ 113.937768] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6017worker1 # [ 114.105507] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6018worker1 # [ 114.109818] systemd[1]: Stopped Nix worker-protocol over gRPC.6019worker1 # [ 114.111989] systemd[1]: nix-grpc-daemon.service: Consumed 3.253s CPU time over 1min 37.356s wall clock time, 62.1M memory peak, 112.4K incoming IP traffic, 213.3K outgoing IP traffic.6020worker1 # [ 114.122457] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.6021worker1 # [ 114.125862] systemd[1]: Closed Nix worker-protocol over gRPC.6022worker2 # [ 114.139329] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:18Z level=info event=scheduler_connected addr=lb:500516023worker1 # [ 114.128677] systemd[1]: nix-grpc-daemon.socket: Consumed 284us CPU time over 1min 43.957s wall clock time, 256K memory peak, 110.3K incoming IP traffic, 96K outgoing IP traffic.6024worker2 # [ 114.143409] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6025lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy6026lb: must succeed: curl -sf localhost:9901/clusters6027lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6028lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.06 seconds)6029client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&26030worker2 # [ 114.344639] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:18Z level=info event=scheduler_connected addr=lb:500516031worker2 # [ 114.348635] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6032worker2 # [ 114.405617] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389326033worker2 # [ 114.414953] nix-daemon[877]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)6034worker2 # [ 114.434517] nix-grpc-daemon[1286]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 25 ms (attempt 1/5)6035worker2 # [ 114.465510] nix-grpc-daemon[1286]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 140 ms (attempt 2/5)6036worker2 # [ 114.549645] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:18Z level=info event=scheduler_connected addr=lb:500516037worker2 # [ 114.553602] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6038worker2 # [ 114.611612] nix-grpc-daemon[1286]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 184 ms (attempt 3/5)6039worker2 # [ 114.754364] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:18Z level=info event=scheduler_connected addr=lb:500516040worker2 # [ 114.756386] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6041worker2 # [ 114.801176] nix-grpc-daemon[1286]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 797 ms (attempt 4/5)6042worker2 # [ 114.957795] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=scheduler_connected addr=lb:500516043worker2 # [ 114.959925] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6044worker2 # [ 115.161386] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=scheduler_connected addr=lb:500516045worker2 # [ 115.165251] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6046worker2 # [ 115.365811] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=scheduler_connected addr=lb:500516047worker2 # [ 115.367670] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6048worker2 # [ 115.568416] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=scheduler_connected addr=lb:500516049worker2 # [ 115.570156] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6050worker2 # [ 115.601518] nix-grpc-daemon[1286]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6051worker2 # [ 115.603746] nix-grpc-daemon[1286]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6052client # copying 2 paths...6053client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...6054worker2 # [ 115.623164] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389326055client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...6056worker2 # [ 115.636697] nix-daemon[877]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)6057worker2 # [ 115.655451] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=niks3_push_started pid=13466058worker2 # [ 115.771057] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=scheduler_connected addr=lb:500516059worker2 # [ 115.773125] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6060worker2 # [ 115.838912] nix-grpc-daemon[1346]: time=2026-09-22T11:23:19.908Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6061worker2 # [ 115.847760] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:19Z level=info event=scheduler_take_over6062lb # [ 115.858745] niks3-server[1047]: 2026/09/22 11:23:19 INFO lead: acquired remote=192.168.1.4:359126063worker2 # [ 115.975691] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:20Z level=info event=scheduler_connected addr=lb:500516064worker2 # [ 115.985062] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:20Z level=info event=oidc_jwks provider=mock keys=16065worker2 # [ 115.986897] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:20Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:389466066worker2 # [ 115.988474] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:20Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06067worker2 # [ 115.997075] nix-daemon[877]: accepted connection from pid 1359, user nix-grpc-daemon (trusted)6068worker2 # [ 116.009543] nix-daemon[877]: reaped child process 1362, status = succeeded6069lb # [ 116.031863] niks3-server[1047]: 2026/09/22 11:23:19 INFO Received uploads request method=POST path=/api/pending_closures6070lb # [ 116.038793] niks3-server[1047]: 2026/09/22 11:23:19 INFO Received uploads request method=POST path=/api/pending_closures6071worker2 # [ 116.033427] nix-grpc-daemon[1346]: time=2026-09-22T11:23:20.103Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"6072worker2 # [ 116.035058] nix-grpc-daemon[1346]: time=2026-09-22T11:23:20.105Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"6073worker2 # [ 116.037132] nix-grpc-daemon[1346]: time=2026-09-22T11:23:20.107Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"6074lb # [ 116.076460] niks3-server[1047]: 2026/09/22 11:23:19 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst6075lb # [ 116.090593] niks3-server[1047]: 2026/09/22 11:23:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign6076lb # [ 116.092554] niks3-server[1047]: 2026/09/22 11:23:19 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst6077lb # [ 116.097120] niks3-server[1047]: 2026/09/22 11:23:19 INFO Signed narinfos id=35 count=16078lb # [ 116.100105] niks3-server[1047]: 2026/09/22 11:23:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign6079lb # [ 116.102222] niks3-server[1047]: 2026/09/22 11:23:19 INFO Signed narinfos id=36 count=26080worker2 # [ 116.094106] nix-grpc-daemon[1346]: time=2026-09-22T11:23:20.163Z level=INFO msg="Uploading 3 narinfos"6081lb # [ 116.127780] niks3-server[1047]: 2026/09/22 11:23:19 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls6082lb # [ 116.138926] niks3-server[1047]: 2026/09/22 11:23:19 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete6083lb # [ 116.166405] niks3-server[1047]: 2026/09/22 11:23:19 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo6084lb # [ 116.172858] niks3-server[1047]: 2026/09/22 11:23:19 INFO Completed upload id=356085lb # [ 116.174563] niks3-server[1047]: 2026/09/22 11:23:19 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete6086lb # [ 116.176689] niks3-server[1047]: 2026/09/22 11:23:19 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls6087lb # [ 116.179117] niks3-server[1047]: 2026/09/22 11:23:19 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo6088lb # [ 116.183257] niks3-server[1047]: 2026/09/22 11:23:19 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo6089lb # [ 116.187130] niks3-server[1047]: 2026/09/22 11:23:19 INFO Completed upload id=366090worker2 # [ 116.177251] nix-grpc-daemon[1346]: time=2026-09-22T11:23:20.247Z level=INFO msg="Upload complete. (334ms)"6091worker2 # [ 116.181525] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38932 duration_s=0 paths=2 nar_bytes_in=3126092worker2 # [ 116.191711] nix-daemon[877]: reaped child process 1344, status = succeeded6093client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 2.01 seconds)6094worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache6095worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.02 seconds)6096worker1 # [ 116.290564] systemd[1]: Listening on Nix worker-protocol over gRPC.6097worker1 # [ 116.298335] systemd[1]: Starting Nix worker-protocol over gRPC...6098worker1 # [ 116.449307] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:19Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6099worker1 # [ 116.451061] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:19Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6100worker1 # [ 116.454919] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:19Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16101worker1 # [ 116.460065] nix-grpc-daemon[1324]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6102worker1 # [ 116.461347] nix-grpc-daemon[1324]: E0000 00:00:1790076199.692241 1324 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6103worker1 # [ 116.467110] nix-grpc-daemon[1324]: E0000 00:00:1790076199.697382 1324 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6104worker1 # [ 116.472867] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:19Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6105worker1 # [ 116.475279] systemd[1]: Started Nix worker-protocol over gRPC.6106worker1 # [ 116.480122] nix-grpc-daemon[1324]: E0000 00:00:1790076199.711390 1365 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6107lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6108lb: must succeed: curl -sf localhost:9901/clusters6109worker1 # [ 116.492735] nix-grpc-daemon[1324]: E0000 00:00:1790076199.725213 1365 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6110worker1 # [ 116.497849] nix-grpc-daemon[1324]: E0000 00:00:1790076199.727795 1365 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6111worker1 # [ 116.507548] nix-grpc-daemon[1324]: E0000 00:00:1790076199.739967 1365 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6112worker1 # [ 116.517257] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:19Z level=info event=scheduler_connected addr=lb:500516113worker2 # [ 116.530329] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:20Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:389466114worker2 # [ 116.531925] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:20Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06115lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6116lb # [ 116.781836] envoy[899]: conn peer=[2001:db8:1::1]:42426 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=546117lb # [ 116.784224] envoy[899]: conn peer=[2001:db8:1::1]:42416 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3755 tx=2099 ms=17896118lb # [ 116.786234] envoy[899]: conn peer=[2001:db8:1::4]:39682 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=95747 tx=4537 ms=924336119lb # [ 116.788412] envoy[899]: conn peer=[2001:db8:1::1]:42430 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=176120lb # [ 116.790542] envoy[899]: conn peer=[2001:db8:1::3]:54536 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=26532 tx=5045 ms=946276121lb # [ 116.792621] envoy[899]: conn peer=[2001:db8:1::1]:45784 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1006 tx=305 ms=17906122lb: must succeed: curl -sf localhost:9901/clusters6123lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6124lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.11 seconds)6125(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.53 seconds)6126subtest: a niks3 restart does not cost the leader its role6127lb: must succeed: curl -sf localhost:9901/clusters6128lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6129lb: must succeed: curl -sf localhost:9901/clusters6130lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6131lb # [ 117.731432] niks3-server[1047]: 2026/09/22 11:23:21 INFO Shutdown signal received, draining in-flight requests timeout=10s6132lb # [ 117.735069] niks3-server[1047]: 2026/09/22 11:23:21 INFO lead: released remote=192.168.1.4:359126133lb # [ 117.737716] systemd[1]: Stopping niks3 server...6134worker1 # [ 117.714980] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:20Z level=info event=scheduler_lock_lost status=2006135worker2 # [ 117.727260] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:21Z level=info event=scheduler_lock_lost status=2006136lb # [ 117.756588] systemd[1]: niks3.service: Deactivated successfully.6137lb # [ 117.758435] systemd[1]: Stopped niks3 server.6138lb # [ 117.759226] systemd[1]: niks3.service: Consumed 766ms CPU time over 1min 27.806s wall clock time, 22.5M memory peak, 100K incoming IP traffic, 142.3K outgoing IP traffic.6139lb # [ 117.768929] systemd[1]: Starting niks3 server...6140lb # [ 117.846130] niks3-server[1194]: 2026/09/22 11:23:21 goose: no migrations to run. current version: 202609200000006141lb # [ 117.854315] niks3-server[1194]: 2026/09/22 11:23:21 OK 1_commit_pending_closure.sql (6.27ms)6142lb # [ 117.859166] niks3-server[1194]: 2026/09/22 11:23:21 OK 2_object_stats_trigger.sql (2.71ms)6143lb # [ 117.860423] niks3-server[1194]: 2026/09/22 11:23:21 goose: up to current file version: 26144lb # [ 117.863545] niks3-server[1194]: 2026/09/22 11:23:21 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key6145lb # [ 117.878845] niks3-server[1194]: 2026/09/22 11:23:21 INFO Read proxy enabled — serving cache objects from S36146lb # [ 117.880483] niks3-server[1194]: 2026/09/22 11:23:21 INFO Using socket-activated listener address=0.0.0.0:57516147lb # [ 117.883056] niks3-server[1194]: 2026/09/22 11:23:21 INFO systemd watchdog enabled interval=15s6148lb # [ 117.884347] systemd[1]: Started niks3 server.6149lb # [ 117.886117] niks3-server[1194]: 2026/09/22 11:23:21 INFO Starting HTTP server address=0.0.0.0:57516150lb: waiting for unit niks3.service6151lb: (finished: waiting for unit niks3.service, in 0.04 seconds)6152lb: must succeed: curl -sf localhost:9901/clusters6153lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6154worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6155worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6156lb: must succeed: curl -sf localhost:9901/clusters6157lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6158lb: must succeed: curl -sf localhost:9901/clusters6159lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6160worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield6161worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.05 seconds)6162client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&26163worker2 # [ 118.346899] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:389326164client # don't know how to build these paths:6165client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6166worker2 # [ 118.356279] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389326167worker2 # [ 118.382164] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:22Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=06168worker2 # [ 118.386141] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:22Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.4:50051 clients=16169worker2 # [ 118.435709] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389326170worker2 # [ 118.449450] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389326171client # copying 1 paths to 'lb:50051'...6172client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...6173worker2 # [ 118.462386] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389326174worker2 # [ 118.473124] nix-daemon[877]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)6175worker2 # [ 118.638158] nix-daemon[877]: accepted connection from pid 1379, user nix-grpc-daemon (trusted)6176worker2 # [ 118.652715] nix-daemon[877]: reaped child process 1382, status = succeeded6177lb # [ 118.739895] niks3-server[1194]: 2026/09/22 11:23:22 INFO lead: acquired remote=192.168.1.4:356826178worker2 # [ 118.793271] nix-daemon[877]: accepted connection from pid 1384, user nix-grpc-daemon (trusted)6179worker2 # [ 118.806713] nix-daemon[877]: reaped child process 1389, status = succeeded6180lb # [ 118.832126] niks3-server[1194]: 2026/09/22 11:23:22 INFO Received uploads request method=POST path=/api/pending_closures6181worker2 # [ 118.835849] nix-grpc-daemon[1346]: time=2026-09-22T11:23:22.906Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6182worker2 # [ 118.837434] nix-grpc-daemon[1346]: time=2026-09-22T11:23:22.907Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"6183lb # [ 118.866911] niks3-server[1194]: 2026/09/22 11:23:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6184lb # [ 118.870259] niks3-server[1194]: 2026/09/22 11:23:22 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst6185lb # [ 118.882872] niks3-server[1194]: 2026/09/22 11:23:22 INFO Signed narinfos id=37 count=16186worker2 # [ 118.874615] nix-grpc-daemon[1346]: time=2026-09-22T11:23:22.945Z level=INFO msg="Uploading 1 narinfos"6187lb # [ 118.894602] niks3-server[1194]: 2026/09/22 11:23:22 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6188lb # [ 118.897156] niks3-server[1194]: 2026/09/22 11:23:22 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls6189lb # [ 118.904052] niks3-server[1194]: 2026/09/22 11:23:22 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo6190lb # [ 118.908673] niks3-server[1194]: 2026/09/22 11:23:22 INFO Completed upload id=376191worker2 # [ 118.898730] nix-grpc-daemon[1346]: time=2026-09-22T11:23:22.969Z level=INFO msg="Upload complete. (413ms)"6192worker2 # [ 118.902055] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38932 duration_s=0 paths=1 nar_bytes_in=5366193worker2 # [ 118.906456] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389326194worker2 # [ 118.913646] nix-daemon[877]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)6195worker2 # [ 118.917599] nix-daemon[877]: reaped child process 1377, status = succeeded6196client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...6197worker2 # [ 118.923286] nix-daemon[877]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)6198client # farm-a> worker2: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv6199client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6200client # farm-a> LOG-farm-a-after-niks3-restart6201worker2 # [ 120.341716] nix-daemon[877]: reaped child process 1393, status = succeeded6202worker2 # [ 120.463066] nix-daemon[877]: accepted connection from pid 1409, user nix-grpc-daemon (trusted)6203worker2 # [ 120.475577] nix-daemon[877]: reaped child process 1412, status = succeeded6204lb # [ 120.496163] niks3-server[1194]: 2026/09/22 11:23:23 INFO Received uploads request method=POST path=/api/pending_closures6205worker2 # [ 120.525695] nix-grpc-daemon[1346]: time=2026-09-22T11:23:24.595Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6206worker2 # [ 120.529066] nix-grpc-daemon[1346]: time=2026-09-22T11:23:24.597Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"6207lb # [ 120.570758] niks3-server[1194]: 2026/09/22 11:23:23 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv6208lb # [ 120.581382] niks3-server[1194]: 2026/09/22 11:23:23 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst6209lb # [ 120.589961] niks3-server[1194]: 2026/09/22 11:23:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6210lb # [ 120.595048] niks3-server[1194]: 2026/09/22 11:23:24 INFO Signed narinfos id=38 count=16211worker2 # [ 120.585678] nix-grpc-daemon[1346]: time=2026-09-22T11:23:24.655Z level=INFO msg="Uploading 1 narinfos"6212lb # [ 120.600440] niks3-server[1194]: 2026/09/22 11:23:24 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls6213lb # [ 120.610385] niks3-server[1194]: 2026/09/22 11:23:24 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6214lb # [ 120.613733] niks3-server[1194]: 2026/09/22 11:23:24 INFO Completed upload id=386215worker2 # [ 120.604095] nix-grpc-daemon[1346]: time=2026-09-22T11:23:24.674Z level=INFO msg="Upload complete. (272ms)"6216lb # [ 120.616754] niks3-server[1194]: 2026/09/22 11:23:24 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo6217worker2 # [ 120.616695] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38932 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=16218worker2 # [ 120.624827] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:24Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=06219worker2 # [ 120.629259] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:24Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.4:50051 clients=16220worker2 # [ 120.640154] nix-daemon[877]: reaped child process 1391, status = succeeded6221worker2 # [ 120.667564] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389326222worker2 # [ 120.678516] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389326223worker2 # [ 120.689092] nix-grpc-daemon[1346]: time=2026-09-22T11:23:24.759Z level=INFO msg="All 1 paths already cached"6224client # copying 1 paths to 'lb:50051'...6225client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...6226worker2 # [ 120.695221] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389326227worker2 # [ 120.701689] nix-daemon[877]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)6228worker2 # [ 120.855443] nix-daemon[877]: accepted connection from pid 1418, user nix-grpc-daemon (trusted)6229worker2 # [ 120.875056] nix-daemon[877]: reaped child process 1421, status = succeeded6230worker2 # [ 121.006576] nix-daemon[877]: accepted connection from pid 1423, user nix-grpc-daemon (trusted)6231worker2 # [ 121.020620] nix-daemon[877]: reaped child process 1426, status = succeeded6232lb # [ 121.040790] niks3-server[1194]: 2026/09/22 11:23:24 INFO Received uploads request method=POST path=/api/pending_closures6233worker2 # [ 121.036139] nix-grpc-daemon[1346]: time=2026-09-22T11:23:25.106Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6234worker2 # [ 121.037531] nix-grpc-daemon[1346]: time=2026-09-22T11:23:25.106Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"6235lb # [ 121.057694] niks3-server[1194]: 2026/09/22 11:23:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6236lb # [ 121.060928] niks3-server[1194]: 2026/09/22 11:23:24 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst6237lb # [ 121.063424] niks3-server[1194]: 2026/09/22 11:23:24 INFO Signed narinfos id=39 count=16238worker2 # [ 121.053657] nix-grpc-daemon[1346]: time=2026-09-22T11:23:25.123Z level=INFO msg="Uploading 1 narinfos"6239lb # [ 121.066768] niks3-server[1194]: 2026/09/22 11:23:24 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls6240lb # [ 121.071649] niks3-server[1194]: 2026/09/22 11:23:24 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6241lb # [ 121.077090] niks3-server[1194]: 2026/09/22 11:23:24 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo6242lb # [ 121.080474] niks3-server[1194]: 2026/09/22 11:23:24 INFO Completed upload id=396243worker2 # [ 121.070479] nix-grpc-daemon[1346]: time=2026-09-22T11:23:25.140Z level=INFO msg="Upload complete. (363ms)"6244worker2 # [ 121.072964] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38932 duration_s=0 paths=1 nar_bytes_in=6566245worker2 # [ 121.075625] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389326246worker2 # [ 121.082072] nix-daemon[877]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)6247worker2 # [ 121.087491] nix-daemon[877]: reaped child process 1416, status = succeeded6248worker2 # [ 121.092202] nix-daemon[877]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)6249client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...6250client # farm-b> worker2: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6251client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6252client # farm-b> LOG-farm-b-after-niks3-restart6253worker2 # [ 121.706314] nix-daemon[877]: reaped child process 1430, status = succeeded6254worker2 # [ 121.819279] nix-daemon[877]: accepted connection from pid 1446, user nix-grpc-daemon (trusted)6255worker2 # [ 121.833200] nix-daemon[877]: reaped child process 1449, status = succeeded6256lb # [ 121.854297] niks3-server[1194]: 2026/09/22 11:23:25 INFO Received uploads request method=POST path=/api/pending_closures6257worker2 # [ 121.858449] nix-grpc-daemon[1346]: time=2026-09-22T11:23:25.928Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6258worker2 # [ 121.860118] nix-grpc-daemon[1346]: time=2026-09-22T11:23:25.930Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"6259lb # [ 121.897113] niks3-server[1194]: 2026/09/22 11:23:25 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6260lb # [ 121.911374] niks3-server[1194]: 2026/09/22 11:23:25 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst6261lb # [ 121.921347] niks3-server[1194]: 2026/09/22 11:23:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6262lb # [ 121.926627] niks3-server[1194]: 2026/09/22 11:23:25 INFO Signed narinfos id=40 count=16263worker2 # [ 121.919464] nix-grpc-daemon[1346]: time=2026-09-22T11:23:25.989Z level=INFO msg="Uploading 1 narinfos"6264lb # [ 121.934046] niks3-server[1194]: 2026/09/22 11:23:25 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls6265lb # [ 121.941347] niks3-server[1194]: 2026/09/22 11:23:25 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6266lb # [ 121.945841] niks3-server[1194]: 2026/09/22 11:23:25 INFO Completed upload id=406267worker2 # [ 121.936096] nix-grpc-daemon[1346]: time=2026-09-22T11:23:26.006Z level=INFO msg="Upload complete. (237ms)"6268lb # [ 121.947888] niks3-server[1194]: 2026/09/22 11:23:25 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo6269worker2 # [ 121.939032] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38932 duration_s=0 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=16270worker2 # [ 121.945874] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:26Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=06271worker2 # [ 121.948325] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:26Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.4:50051 clients=16272worker2 # [ 121.952882] nix-daemon[877]: reaped child process 1428, status = succeeded6273worker2 # [ 121.989918] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389326274worker2 # [ 121.998925] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389326275worker2 # [ 122.007142] nix-grpc-daemon[1346]: time=2026-09-22T11:23:26.077Z level=INFO msg="All 2 paths already cached"6276client # copying 1 paths to 'lb:50051'...6277client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...6278worker2 # [ 122.012627] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389326279worker2 # [ 122.016405] nix-daemon[877]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)6280worker2 # [ 122.133598] nix-daemon[877]: accepted connection from pid 1455, user nix-grpc-daemon (trusted)6281worker2 # [ 122.148705] nix-daemon[877]: reaped child process 1458, status = succeeded6282worker2 # [ 122.265367] nix-daemon[877]: accepted connection from pid 1460, user nix-grpc-daemon (trusted)6283worker2 # [ 122.278847] nix-daemon[877]: reaped child process 1463, status = succeeded6284lb # [ 122.299295] niks3-server[1194]: 2026/09/22 11:23:25 INFO Received uploads request method=POST path=/api/pending_closures6285worker2 # [ 122.301450] nix-grpc-daemon[1346]: time=2026-09-22T11:23:26.371Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6286worker2 # [ 122.303060] nix-grpc-daemon[1346]: time=2026-09-22T11:23:26.373Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"6287lb # [ 122.336242] niks3-server[1194]: 2026/09/22 11:23:25 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst6288lb # [ 122.345959] niks3-server[1194]: 2026/09/22 11:23:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6289worker2 # [ 122.340425] nix-grpc-daemon[1346]: time=2026-09-22T11:23:26.410Z level=INFO msg="Uploading 1 narinfos"6290lb # [ 122.350492] niks3-server[1194]: 2026/09/22 11:23:25 INFO Signed narinfos id=41 count=16291lb # [ 122.358222] niks3-server[1194]: 2026/09/22 11:23:25 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls6292lb # [ 122.366874] niks3-server[1194]: 2026/09/22 11:23:25 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6293lb # [ 122.371331] niks3-server[1194]: 2026/09/22 11:23:25 INFO Completed upload id=416294worker2 # [ 122.361340] nix-grpc-daemon[1346]: time=2026-09-22T11:23:26.431Z level=INFO msg="Upload complete. (338ms)"6295lb # [ 122.373857] niks3-server[1194]: 2026/09/22 11:23:25 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo6296worker2 # [ 122.363775] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38932 duration_s=0 paths=1 nar_bytes_in=7846297worker2 # [ 122.368698] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389326298worker2 # [ 122.374781] nix-daemon[877]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)6299worker2 # [ 122.377158] nix-daemon[877]: reaped child process 1453, status = succeeded6300worker2 # [ 122.382707] nix-daemon[877]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)6301client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...6302client # farm-top> worker2: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6303client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6304client # farm-top> LOG-farm-top-after-niks3-restart6305worker2 # [ 123.398793] nix-daemon[877]: reaped child process 1467, status = succeeded6306worker2 # [ 123.518761] nix-daemon[877]: accepted connection from pid 1483, user nix-grpc-daemon (trusted)6307worker2 # [ 123.533077] nix-daemon[877]: reaped child process 1486, status = succeeded6308lb # [ 123.552693] niks3-server[1194]: 2026/09/22 11:23:26 INFO Received uploads request method=POST path=/api/pending_closures6309worker2 # [ 123.550031] nix-grpc-daemon[1346]: time=2026-09-22T11:23:27.620Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6310worker2 # [ 123.553322] nix-grpc-daemon[1346]: time=2026-09-22T11:23:27.622Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"6311lb # [ 123.578689] niks3-server[1194]: 2026/09/22 11:23:26 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6312lb # [ 123.584674] niks3-server[1194]: 2026/09/22 11:23:26 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst6313lb # [ 123.594349] niks3-server[1194]: 2026/09/22 11:23:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6314worker2 # [ 123.589604] nix-grpc-daemon[1346]: time=2026-09-22T11:23:27.659Z level=INFO msg="Uploading 1 narinfos"6315lb # [ 123.599478] niks3-server[1194]: 2026/09/22 11:23:27 INFO Signed narinfos id=42 count=16316lb # [ 123.601902] niks3-server[1194]: 2026/09/22 11:23:27 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls6317lb # [ 123.607145] niks3-server[1194]: 2026/09/22 11:23:27 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6318lb # [ 123.610232] niks3-server[1194]: 2026/09/22 11:23:27 INFO Completed upload id=426319worker2 # [ 123.600259] nix-grpc-daemon[1346]: time=2026-09-22T11:23:27.670Z level=INFO msg="Upload complete. (210ms)"6320lb # [ 123.611742] niks3-server[1194]: 2026/09/22 11:23:27 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo6321worker2 # [ 123.602943] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38932 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=16322worker2 # [ 123.611671] nix-daemon[877]: reaped child process 1465, status = succeeded6323client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&2, in 5.54 seconds)6324client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath6325client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath, in 0.17 seconds)6326(finished: subtest: a niks3 restart does not cost the leader its role, in 6.25 seconds)6327subtest: scheduler down: the other node takes the lock and keeps it6328lb: must succeed: curl -sf localhost:9901/clusters6329lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6330lb: must succeed: curl -sf localhost:9901/clusters6331lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6332lb: must succeed: curl -sf localhost:9901/clusters6333lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6334lb: must succeed: curl -sf localhost:9901/clusters6335lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6336worker2 # [ 124.020485] systemd[1]: Stopping Nix worker-protocol over gRPC...6337worker2 # [ 124.837981] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:28Z level=info event=signal_exit6338lb # [ 124.946735] niks3-server[1194]: 2026/09/22 11:23:28 INFO lead: released remote=192.168.1.4:356826339worker1 # [ 125.925651] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:29Z level=info event=scheduler_restarting addr=lb:500516340worker2 # [ 125.941590] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:30Z level=info event=scheduler_restarting6341worker2 # [ 125.945552] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:30Z level=info event=scheduler_restarting addr=lb:500516342worker2 # [ 125.948925] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:30Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:389466343worker2 # [ 125.952537] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:30Z level=info event=worker_gone addr=192.168.1.3:500516344worker2 # [ 125.955898] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:30Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:389466345worker2 # [ 125.959375] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:30Z level=info event=worker_gone addr=192.168.1.4:500516346worker2 # [ 125.976711] nix-daemon[877]: reaped child process 1338, status = succeeded6347worker1 # [ 126.125458] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:29Z level=info event=scheduler_connected addr=lb:500516348worker1 # [ 126.127516] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6349worker2 # [ 126.149817] nix-grpc-daemon[1286]: ts=2026-09-22T11:23:30Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6350worker1 # [ 126.328723] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:29Z level=info event=scheduler_connected addr=lb:500516351worker1 # [ 126.332944] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6352worker1 # [ 126.533828] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:29Z level=info event=scheduler_connected addr=lb:500516353worker1 # [ 126.537774] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6354worker1 # [ 126.738811] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:29Z level=info event=scheduler_connected addr=lb:500516355worker1 # [ 126.742906] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6356lb # [ 126.783152] envoy[899]: conn peer=[2001:db8:1::1]:45788 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9739 tx=4464 ms=53136357lb # [ 126.786842] envoy[899]: conn peer=[2001:db8:1::4]:40740 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=22397 tx=1359 ms=152596358worker2 # [ 126.877231] nix-grpc-daemon[1286]: E0000 00:00:1790076210.947584 1331 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true6359worker2 # [ 126.879449] nix-grpc-daemon[1286]: E0000 00:00:1790076210.947676 1331 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6360worker1 # [ 126.945735] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:30Z level=info event=scheduler_connected addr=lb:500516361worker1 # [ 126.948909] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6362worker1 # [ 127.146362] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:30Z level=info event=scheduler_connected addr=lb:500516363worker1 # [ 127.150141] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6364worker1 # [ 127.350591] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:30Z level=info event=scheduler_connected addr=lb:500516365worker1 # [ 127.352405] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6366worker1 # [ 127.553409] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:30Z level=info event=scheduler_connected addr=lb:500516367worker1 # [ 127.557314] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6368worker1 # [ 127.757613] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:30Z level=info event=scheduler_connected addr=lb:500516369worker1 # [ 127.761170] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6370worker1 # [ 127.962113] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:31Z level=info event=scheduler_connected addr=lb:500516371worker1 # [ 127.963689] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6372worker2 # [ 128.022685] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6373worker2 # [ 128.024305] systemd[1]: Stopped Nix worker-protocol over gRPC.6374worker2 # [ 128.025351] systemd[1]: nix-grpc-daemon.service: Consumed 1.126s CPU time over 17.339s wall clock time, 35.2M memory peak, 49.8K incoming IP traffic, 105.1K outgoing IP traffic.6375worker2 # [ 128.029592] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.6376worker2 # [ 128.031358] systemd[1]: Closed Nix worker-protocol over gRPC.6377worker2 # [ 128.032542] systemd[1]: nix-grpc-daemon.socket: Consumed 330us CPU time over 17.520s wall clock time, 256K memory peak, 38.7K incoming IP traffic, 28.9K outgoing IP traffic.6378worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over6379worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.06 seconds)6380worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6381worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6382worker1 # [ 128.164067] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:31Z level=info event=scheduler_connected addr=lb:500516383worker1 # [ 128.165765] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6384worker1 # [ 128.366377] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:31Z level=info event=scheduler_connected addr=lb:500516385worker1 # [ 128.368119] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6386worker1 # [ 128.568650] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:31Z level=info event=scheduler_connected addr=lb:500516387worker1 # [ 128.570502] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6388lb # [ 128.741362] niks3-server[1194]: 2026/09/22 11:23:32 INFO lead: acquired remote=192.168.1.3:374786389worker1 # [ 128.719710] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:31Z level=info event=scheduler_take_over6390worker1 # [ 128.771084] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:32Z level=info event=scheduler_connected addr=lb:500516391worker1 # [ 128.772870] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6392worker1 # [ 128.973901] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:32Z level=info event=scheduler_connected addr=lb:500516393worker1 # [ 128.975656] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6394worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6395worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6396worker1 # [ 129.176065] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:32Z level=info event=scheduler_connected addr=lb:500516397worker1 # [ 129.179997] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:32Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:336926398worker1 # [ 129.182410] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:32Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06399worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6400worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6401client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&26402worker1 # [ 130.389248] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:33Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:337066403worker1 # [ 130.400266] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6404client # don't know how to build these paths:6405client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6406worker1 # [ 130.414604] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337066407worker1 # [ 130.425224] nix-grpc-daemon[1324]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 79 ms (attempt 1/5)6408worker1 # [ 130.510082] nix-grpc-daemon[1324]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 107 ms (attempt 2/5)6409worker1 # [ 130.623352] nix-grpc-daemon[1324]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 143 ms (attempt 3/5)6410worker1 # [ 130.772296] nix-grpc-daemon[1324]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 60 ms (attempt 4/5)6411worker1 # [ 130.838426] nix-grpc-daemon[1324]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6412worker1 # [ 130.842188] nix-grpc-daemon[1324]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6413worker1 # [ 130.868832] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:34Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=06414worker1 # [ 130.874132] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:34Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=16415worker1 # [ 130.913180] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337066416worker1 # [ 130.925717] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337066417client # copying 1 paths to 'lb:50051'...6418client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...6419worker1 # [ 130.937087] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:337066420worker1 # [ 130.945509] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6421worker1 # [ 130.966395] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:34Z level=info event=niks3_push_started pid=14036422worker1 # [ 131.120514] nix-grpc-daemon[1403]: time=2026-09-22T11:23:34.352Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6423worker1 # [ 131.257957] nix-daemon[855]: accepted connection from pid 1416, user nix-grpc-daemon (trusted)6424worker1 # [ 131.270195] nix-daemon[855]: reaped child process 1419, status = succeeded6425worker1 # [ 131.386226] nix-daemon[855]: accepted connection from pid 1421, user nix-grpc-daemon (trusted)6426worker1 # [ 131.398808] nix-daemon[855]: reaped child process 1424, status = succeeded6427lb # [ 131.431430] niks3-server[1194]: 2026/09/22 11:23:34 INFO Received uploads request method=POST path=/api/pending_closures6428worker1 # [ 131.414489] nix-grpc-daemon[1403]: time=2026-09-22T11:23:34.646Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6429worker1 # [ 131.417906] nix-grpc-daemon[1403]: time=2026-09-22T11:23:34.648Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"6430lb # [ 131.453195] niks3-server[1194]: 2026/09/22 11:23:34 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst6431lb # [ 131.455833] niks3-server[1194]: 2026/09/22 11:23:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6432lb # [ 131.457901] niks3-server[1194]: 2026/09/22 11:23:34 INFO Signed narinfos id=43 count=16433worker1 # [ 131.436810] nix-grpc-daemon[1403]: time=2026-09-22T11:23:34.669Z level=INFO msg="Uploading 1 narinfos"6434lb # [ 131.462535] niks3-server[1194]: 2026/09/22 11:23:34 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls6435lb # [ 131.466414] niks3-server[1194]: 2026/09/22 11:23:34 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6436lb # [ 131.470722] niks3-server[1194]: 2026/09/22 11:23:34 INFO Completed upload id=436437worker1 # [ 131.448830] nix-grpc-daemon[1403]: time=2026-09-22T11:23:34.681Z level=INFO msg="Upload complete. (324ms)"6438lb # [ 131.472280] niks3-server[1194]: 2026/09/22 11:23:34 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo6439worker1 # [ 131.451449] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33706 duration_s=0 paths=1 nar_bytes_in=5206440worker1 # [ 131.454992] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337066441worker1 # [ 131.461970] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6442worker1 # [ 131.464329] nix-daemon[855]: reaped child process 1401, status = succeeded6443client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...6444client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6445worker1 # [ 131.469607] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6446client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6447client # farm-a> LOG-farm-a-on-standby6448worker1 # [ 132.082994] nix-daemon[855]: reaped child process 1428, status = succeeded6449worker1 # [ 132.214081] nix-daemon[855]: accepted connection from pid 1444, user nix-grpc-daemon (trusted)6450worker1 # [ 132.227526] nix-daemon[855]: reaped child process 1447, status = succeeded6451lb # [ 132.256580] niks3-server[1194]: 2026/09/22 11:23:35 INFO Received uploads request method=POST path=/api/pending_closures6452worker1 # [ 132.242318] nix-grpc-daemon[1403]: time=2026-09-22T11:23:35.473Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6453worker1 # [ 132.243714] nix-grpc-daemon[1403]: time=2026-09-22T11:23:35.473Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"6454lb # [ 132.281647] niks3-server[1194]: 2026/09/22 11:23:35 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6455lb # [ 132.290724] niks3-server[1194]: 2026/09/22 11:23:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6456lb # [ 132.295488] niks3-server[1194]: 2026/09/22 11:23:35 INFO Signed narinfos id=44 count=16457worker1 # [ 132.274386] nix-grpc-daemon[1403]: time=2026-09-22T11:23:35.506Z level=INFO msg="Uploading 1 narinfos"6458lb # [ 132.299414] niks3-server[1194]: 2026/09/22 11:23:35 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst6459lb # [ 132.309517] niks3-server[1194]: 2026/09/22 11:23:35 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6460lb # [ 132.314611] niks3-server[1194]: 2026/09/22 11:23:35 INFO Completed upload id=446461worker1 # [ 132.292671] nix-grpc-daemon[1403]: time=2026-09-22T11:23:35.525Z level=INFO msg="Upload complete. (216ms)"6462worker1 # [ 132.295141] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33706 duration_s=0 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=16463lb # [ 132.317744] niks3-server[1194]: 2026/09/22 11:23:35 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo6464lb # [ 132.324544] niks3-server[1194]: 2026/09/22 11:23:35 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls6465worker1 # [ 132.306714] nix-daemon[855]: reaped child process 1426, status = succeeded6466worker1 # [ 132.308924] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:35Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=06467worker1 # [ 132.311058] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:35Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=16468worker1 # [ 132.352915] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337066469worker1 # [ 132.364207] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337066470worker1 # [ 132.378322] nix-grpc-daemon[1403]: time=2026-09-22T11:23:35.610Z level=INFO msg="All 1 paths already cached"6471client # copying 1 paths to 'lb:50051'...6472client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...6473worker1 # [ 132.387141] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:337066474worker1 # [ 132.398117] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6475worker1 # [ 132.547443] nix-daemon[855]: accepted connection from pid 1453, user nix-grpc-daemon (trusted)6476worker1 # [ 132.566129] nix-daemon[855]: reaped child process 1456, status = succeeded6477worker1 # [ 132.705127] nix-daemon[855]: accepted connection from pid 1458, user nix-grpc-daemon (trusted)6478worker1 # [ 132.717991] nix-daemon[855]: reaped child process 1461, status = succeeded6479lb # [ 132.749968] niks3-server[1194]: 2026/09/22 11:23:36 INFO Received uploads request method=POST path=/api/pending_closures6480worker1 # [ 132.738922] nix-grpc-daemon[1403]: time=2026-09-22T11:23:35.971Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6481worker1 # [ 132.740560] nix-grpc-daemon[1403]: time=2026-09-22T11:23:35.973Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"6482lb # [ 132.783134] niks3-server[1194]: 2026/09/22 11:23:36 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst6483lb # [ 132.793791] niks3-server[1194]: 2026/09/22 11:23:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6484worker1 # [ 132.778934] nix-grpc-daemon[1403]: time=2026-09-22T11:23:36.011Z level=INFO msg="Uploading 1 narinfos"6485lb # [ 132.800442] niks3-server[1194]: 2026/09/22 11:23:36 INFO Signed narinfos id=45 count=16486lb # [ 132.803942] niks3-server[1194]: 2026/09/22 11:23:36 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls6487lb # [ 132.815391] niks3-server[1194]: 2026/09/22 11:23:36 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6488lb # [ 132.822215] niks3-server[1194]: 2026/09/22 11:23:36 INFO Completed upload id=456489worker1 # [ 132.800475] nix-grpc-daemon[1403]: time=2026-09-22T11:23:36.032Z level=INFO msg="Upload complete. (392ms)"6490lb # [ 132.823860] niks3-server[1194]: 2026/09/22 11:23:36 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo6491worker1 # [ 132.803865] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33706 duration_s=0 paths=1 nar_bytes_in=6406492worker1 # [ 132.806802] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337066493worker1 # [ 132.814302] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6494worker1 # [ 132.817083] nix-daemon[855]: reaped child process 1451, status = succeeded6495worker1 # [ 132.822342] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6496client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...6497client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6498client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6499client # farm-b> LOG-farm-b-on-standby6500worker1 # [ 133.436360] nix-daemon[855]: reaped child process 1465, status = succeeded6501worker1 # [ 133.580384] nix-daemon[855]: accepted connection from pid 1481, user nix-grpc-daemon (trusted)6502worker1 # [ 133.593537] nix-daemon[855]: reaped child process 1484, status = succeeded6503lb # [ 133.630904] niks3-server[1194]: 2026/09/22 11:23:37 INFO Received uploads request method=POST path=/api/pending_closures6504worker1 # [ 133.615965] nix-grpc-daemon[1403]: time=2026-09-22T11:23:36.847Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6505worker1 # [ 133.617441] nix-grpc-daemon[1403]: time=2026-09-22T11:23:36.847Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"6506lb # [ 133.649628] niks3-server[1194]: 2026/09/22 11:23:37 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst6507lb # [ 133.657748] niks3-server[1194]: 2026/09/22 11:23:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6508lb # [ 133.660604] niks3-server[1194]: 2026/09/22 11:23:37 INFO Signed narinfos id=46 count=16509worker1 # [ 133.640711] nix-grpc-daemon[1403]: time=2026-09-22T11:23:36.872Z level=INFO msg="Uploading 1 narinfos"6510lb # [ 133.665867] niks3-server[1194]: 2026/09/22 11:23:37 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6511lb # [ 133.668586] niks3-server[1194]: 2026/09/22 11:23:37 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls6512lb # [ 133.675662] niks3-server[1194]: 2026/09/22 11:23:37 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6513lb # [ 133.680332] niks3-server[1194]: 2026/09/22 11:23:37 INFO Completed upload id=466514worker1 # [ 133.658739] nix-grpc-daemon[1403]: time=2026-09-22T11:23:36.891Z level=INFO msg="Upload complete. (230ms)"6515lb # [ 133.681723] niks3-server[1194]: 2026/09/22 11:23:37 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo6516worker1 # [ 133.661283] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33706 duration_s=0 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=16517worker1 # [ 133.668643] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:36Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=06518worker1 # [ 133.672284] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:36Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=16519worker1 # [ 133.674930] nix-daemon[855]: reaped child process 1463, status = succeeded6520worker1 # [ 133.711519] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337066521worker1 # [ 133.720743] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337066522worker1 # [ 133.731440] nix-grpc-daemon[1403]: time=2026-09-22T11:23:36.963Z level=INFO msg="All 2 paths already cached"6523client # copying 1 paths to 'lb:50051'...6524client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...6525worker1 # [ 133.739804] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:337066526worker1 # [ 133.746835] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6527worker1 # [ 133.897892] nix-daemon[855]: accepted connection from pid 1490, user nix-grpc-daemon (trusted)6528worker1 # [ 133.912070] nix-daemon[855]: reaped child process 1493, status = succeeded6529worker1 # [ 134.067600] nix-daemon[855]: accepted connection from pid 1495, user nix-grpc-daemon (trusted)6530worker1 # [ 134.080891] nix-daemon[855]: reaped child process 1498, status = succeeded6531lb # [ 134.111631] niks3-server[1194]: 2026/09/22 11:23:37 INFO Received uploads request method=POST path=/api/pending_closures6532worker1 # [ 134.093978] nix-grpc-daemon[1403]: time=2026-09-22T11:23:37.326Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6533worker1 # [ 134.095530] nix-grpc-daemon[1403]: time=2026-09-22T11:23:37.328Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6534lb # [ 134.127622] niks3-server[1194]: 2026/09/22 11:23:37 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6535lb # [ 134.131074] niks3-server[1194]: 2026/09/22 11:23:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6536lb # [ 134.133891] niks3-server[1194]: 2026/09/22 11:23:37 INFO Signed narinfos id=47 count=16537worker1 # [ 134.112265] nix-grpc-daemon[1403]: time=2026-09-22T11:23:37.344Z level=INFO msg="Uploading 1 narinfos"6538lb # [ 134.137234] niks3-server[1194]: 2026/09/22 11:23:37 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6539lb # [ 134.141668] niks3-server[1194]: 2026/09/22 11:23:37 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6540lb # [ 134.146781] niks3-server[1194]: 2026/09/22 11:23:37 INFO Completed upload id=476541worker1 # [ 134.125117] nix-grpc-daemon[1403]: time=2026-09-22T11:23:37.357Z level=INFO msg="Upload complete. (369ms)"6542lb # [ 134.147954] niks3-server[1194]: 2026/09/22 11:23:37 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6543worker1 # [ 134.127627] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33706 duration_s=0 paths=1 nar_bytes_in=7686544worker1 # [ 134.130463] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337066545worker1 # [ 134.136410] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6546worker1 # [ 134.141392] nix-daemon[855]: reaped child process 1488, status = succeeded6547worker1 # [ 134.146217] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6548client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6549client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6550client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6551client # farm-top> LOG-farm-top-on-standby6552worker1 # [ 135.360736] nix-daemon[855]: reaped child process 1502, status = succeeded6553worker1 # [ 135.510521] nix-daemon[855]: accepted connection from pid 1518, user nix-grpc-daemon (trusted)6554worker1 # [ 135.523402] nix-daemon[855]: reaped child process 1521, status = succeeded6555lb # [ 135.557791] niks3-server[1194]: 2026/09/22 11:23:38 INFO Received uploads request method=POST path=/api/pending_closures6556worker1 # [ 135.541792] nix-grpc-daemon[1403]: time=2026-09-22T11:23:38.774Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6557worker1 # [ 135.543341] nix-grpc-daemon[1403]: time=2026-09-22T11:23:38.775Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6558lb # [ 135.577647] niks3-server[1194]: 2026/09/22 11:23:38 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6559lb # [ 135.580920] niks3-server[1194]: 2026/09/22 11:23:38 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6560lb # [ 135.584949] niks3-server[1194]: 2026/09/22 11:23:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6561lb # [ 135.587563] niks3-server[1194]: 2026/09/22 11:23:39 INFO Signed narinfos id=48 count=16562worker1 # [ 135.566095] nix-grpc-daemon[1403]: time=2026-09-22T11:23:38.798Z level=INFO msg="Uploading 1 narinfos"6563lb # [ 135.592857] niks3-server[1194]: 2026/09/22 11:23:39 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6564lb # [ 135.595728] niks3-server[1194]: 2026/09/22 11:23:39 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6565lb # [ 135.598121] niks3-server[1194]: 2026/09/22 11:23:39 INFO Completed upload id=486566worker1 # [ 135.576406] nix-grpc-daemon[1403]: time=2026-09-22T11:23:38.808Z level=INFO msg="Upload complete. (222ms)"6567lb # [ 135.599444] niks3-server[1194]: 2026/09/22 11:23:39 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6568worker1 # [ 135.578905] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33706 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16569worker1 # [ 135.586420] nix-daemon[855]: reaped child process 1500, status = succeeded6570client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 5.42 seconds)6571client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath6572client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.18 seconds)6573worker2 # [ 135.850053] systemd[1]: Listening on Nix worker-protocol over gRPC.6574worker2 # [ 135.854137] systemd[1]: Starting scheduler-token.service...6575worker2 # [ 135.914841] systemd[1]: scheduler-token.service: Deactivated successfully.6576worker2 # [ 135.916502] systemd[1]: Finished scheduler-token.service.6577worker2 # [ 135.919109] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 62ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6578worker2 # [ 135.923221] systemd[1]: Starting Nix worker-protocol over gRPC...6579worker2 # [ 136.067706] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:40Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6580worker2 # [ 136.070123] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:40Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6581worker2 # [ 136.073380] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:40Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16582worker2 # [ 136.080064] nix-grpc-daemon[1507]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6583worker2 # [ 136.081345] nix-grpc-daemon[1507]: E0000 00:00:1790076220.149835 1507 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6584worker2 # [ 136.086071] nix-grpc-daemon[1507]: E0000 00:00:1790076220.154863 1507 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6585worker2 # [ 136.091209] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:40Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6586worker2 # [ 136.097991] systemd[1]: Started Nix worker-protocol over gRPC.6587worker2 # [ 136.105311] nix-grpc-daemon[1507]: E0000 00:00:1790076220.174397 1546 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6588lb: must succeed: curl -sf localhost:9901/clusters6589worker2 # [ 136.117151] nix-grpc-daemon[1507]: E0000 00:00:1790076220.187631 1546 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6590worker2 # [ 136.120168] nix-grpc-daemon[1507]: E0000 00:00:1790076220.189819 1546 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6591worker2 # [ 136.131251] nix-grpc-daemon[1507]: E0000 00:00:1790076220.201687 1546 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6592worker2 # [ 136.138967] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:40Z level=info event=scheduler_connected addr=lb:500516593worker1 # [ 136.134161] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:39Z level=info event=oidc_jwks provider=mock keys=16594worker1 # [ 136.139309] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:39Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:336926595worker1 # [ 136.143813] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:39Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06596lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6597worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6598worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6599lb: must succeed: curl -sf localhost:9901/clusters6600lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6601lb: must succeed: curl -sf localhost:9901/clusters6602lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6603worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6604worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6605(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.53 seconds)6606subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6607client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs6608client # [ 136.427914] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.6609client # Running as unit: rs.service; invocation ID: 8da5c30f8dca4880873ae9a5eed8d4986610client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.07 seconds)6611worker2 # [ 136.640614] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:576366612worker2 # [ 136.652937] nix-daemon[877]: accepted connection from pid 1507, user nix-grpc-daemon (trusted)6613client # [ 136.663429] nix[1357]: don't know how to build these paths:6614client # [ 136.664386] nix[1357]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6615worker1 # [ 136.663102] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:39Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06616worker1 # [ 136.667085] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:39Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=16617worker1 # [ 136.704865] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337066618worker1 # [ 136.715538] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337066619client # [ 136.733276] nix[1357]: copying 1 paths to 'lb:50051'...6620worker1 # [ 136.724762] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:337066621client # [ 136.735137] nix[1357]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6622worker1 # [ 136.737316] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6623lb # [ 136.784529] envoy[899]: conn peer=[2001:db8:1::1]:59506 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9577 tx=4413 ms=52106624worker1 # [ 136.917563] nix-daemon[855]: accepted connection from pid 1534, user nix-grpc-daemon (trusted)6625worker1 # [ 136.929487] nix-daemon[855]: reaped child process 1537, status = succeeded6626worker1 # [ 137.090687] nix-daemon[855]: accepted connection from pid 1539, user nix-grpc-daemon (trusted)6627worker1 # [ 137.102730] nix-daemon[855]: reaped child process 1542, status = succeeded6628lb # [ 137.135824] niks3-server[1194]: 2026/09/22 11:23:40 INFO Received uploads request method=POST path=/api/pending_closures6629worker1 # [ 137.118764] nix-grpc-daemon[1403]: time=2026-09-22T11:23:40.350Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6630worker1 # [ 137.120126] nix-grpc-daemon[1403]: time=2026-09-22T11:23:40.350Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6631lb # [ 137.151674] niks3-server[1194]: 2026/09/22 11:23:40 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6632lb # [ 137.165539] niks3-server[1194]: 2026/09/22 11:23:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6633lb # [ 137.171621] niks3-server[1194]: 2026/09/22 11:23:40 INFO Signed narinfos id=49 count=16634worker1 # [ 137.153276] nix-grpc-daemon[1403]: time=2026-09-22T11:23:40.385Z level=INFO msg="Uploading 1 narinfos"6635lb # [ 137.176598] niks3-server[1194]: 2026/09/22 11:23:40 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6636lb # [ 137.189345] niks3-server[1194]: 2026/09/22 11:23:40 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6637worker1 # [ 137.173832] nix-grpc-daemon[1403]: time=2026-09-22T11:23:40.406Z level=INFO msg="Upload complete. (425ms)"6638lb # [ 137.195550] niks3-server[1194]: 2026/09/22 11:23:40 INFO Completed upload id=496639worker1 # [ 137.175818] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33706 duration_s=0 paths=1 nar_bytes_in=4166640lb # [ 137.199115] niks3-server[1194]: 2026/09/22 11:23:40 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6641worker1 # [ 137.182167] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337066642worker1 # [ 137.183868] nix-daemon[855]: reaped child process 1532, status = succeeded6643worker1 # [ 137.186669] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6644worker1 # [ 137.192686] nix-daemon[855]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)6645client # [ 137.201249] nix[1357]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6646lb: must succeed: curl -sf localhost:9901/clusters6647lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6648lb: must succeed: curl -sf localhost:9901/clusters6649lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6650lb: must succeed: curl -sf localhost:9901/clusters6651lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6652lb: must succeed: curl -sf localhost:9901/clusters6653lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6654worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6655worker2 # [ 139.211300] systemd-journald[289]: Received client request to rotate journal, rotating.6656worker2 # [ 139.228652] systemd-journald[289]: Vacuuming done, freed 0B of archived journals from /var/log/journal/589e47287e214817a9242c1f065477a7.6657worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.06 seconds)6658worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service6659worker1 # [ 139.134285] systemd[1]: Stopping Nix worker-protocol over gRPC...6660worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)6661client: must succeed: sleep 5.06662worker1 # [ 139.476866] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:42Z level=info event=draining6663client: (finished: must succeed: sleep 5.0, in 5.05 seconds)6664worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6665worker1 # [ 144.411244] nix-daemon[855]: reaped child process 1546, status = succeeded6666worker1 # [ 144.540987] nix-daemon[855]: accepted connection from pid 1587, user nix-grpc-daemon (trusted)6667worker1 # [ 144.553491] nix-daemon[855]: reaped child process 1590, status = succeeded6668lb # [ 144.592427] niks3-server[1194]: 2026/09/22 11:23:48 INFO Received uploads request method=POST path=/api/pending_closures6669worker1 # [ 144.580147] nix-grpc-daemon[1403]: time=2026-09-22T11:23:47.812Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6670worker1 # [ 144.582145] nix-grpc-daemon[1403]: time=2026-09-22T11:23:47.814Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6671lb # [ 144.625593] niks3-server[1194]: 2026/09/22 11:23:48 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6672lb # [ 144.641130] niks3-server[1194]: 2026/09/22 11:23:48 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6673lb # [ 144.652755] niks3-server[1194]: 2026/09/22 11:23:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6674worker1 # [ 144.636690] nix-grpc-daemon[1403]: time=2026-09-22T11:23:47.869Z level=INFO msg="Uploading 1 narinfos"6675lb # [ 144.658499] niks3-server[1194]: 2026/09/22 11:23:48 INFO Signed narinfos id=50 count=16676lb # [ 144.663838] niks3-server[1194]: 2026/09/22 11:23:48 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6677lb # [ 144.672351] niks3-server[1194]: 2026/09/22 11:23:48 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6678lb # [ 144.679610] niks3-server[1194]: 2026/09/22 11:23:48 INFO Completed upload id=506679worker1 # [ 144.657840] nix-grpc-daemon[1403]: time=2026-09-22T11:23:47.890Z level=INFO msg="Upload complete. (251ms)"6680lb # [ 144.682757] niks3-server[1194]: 2026/09/22 11:23:48 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6681worker1 # [ 144.661527] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33706 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16682worker1 # [ 144.669852] nix-daemon[855]: reaped child process 1544, status = succeeded6683client # [ 144.709556] systemd[1]: rs.service: Deactivated successfully.6684client # [ 144.713180] systemd[1]: rs.service: Consumed 133ms CPU time over 8.284s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6685worker1 # [ 145.476268] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:48Z level=info event=signal_exit6686lb # [ 145.969849] niks3-server[1194]: 2026/09/22 11:23:49 INFO lead: released remote=192.168.1.3:374786687lb # [ 146.112112] niks3-server[1194]: 2026/09/22 11:23:49 INFO lead: acquired remote=192.168.1.4:548026688worker2 # [ 146.105755] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:50Z level=info event=scheduler_take_over6689lb # [ 146.785489] envoy[899]: conn peer=[2001:db8:1::1]:42176 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2973 tx=1539 ms=80476690worker2 # [ 146.960414] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:51Z level=info event=scheduler_restarting addr=lb:500516691worker1 # [ 146.952918] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:50Z level=info event=scheduler_restarting6692worker1 # [ 146.957239] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:50Z level=info event=scheduler_restarting addr=lb:500516693worker1 # [ 146.960697] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:336926694worker1 # [ 146.964331] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:50Z level=info event=worker_gone addr=192.168.1.4:500516695worker1 # [ 146.967672] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:336926696worker1 # [ 146.971145] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:50Z level=info event=worker_gone addr=192.168.1.3:500516697worker1 # [ 146.987330] nix-daemon[855]: reaped child process 1393, status = succeeded6698worker2 # [ 147.162818] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:51Z level=info event=scheduler_connected addr=lb:500516699worker1 # [ 147.154868] nix-grpc-daemon[1324]: E0000 00:00:1790076230.387176 1369 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true6700worker2 # [ 147.175586] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:51Z level=info event=oidc_jwks provider=mock keys=16701worker1 # [ 147.163508] nix-grpc-daemon[1324]: E0000 00:00:1790076230.387290 1369 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6702worker2 # [ 147.179996] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:51Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:357726703worker1 # [ 147.172545] nix-grpc-daemon[1324]: ts=2026-09-22T11:23:50Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6704worker2 # [ 147.184399] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:51Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06705worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.16 seconds)6706client: waiting for success: ! systemctl is-active rs6707client: (finished: waiting for success: ! systemctl is-active rs, in 0.05 seconds)6708client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6709client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.04 seconds)6710client: must fail: journalctl -u rs | grep -q 'reconnecting'6711client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)6712lb: must succeed: curl -sf localhost:9901/clusters6713lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6714worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6715worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6716worker1 # [ 148.236306] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6717worker1 # [ 148.237448] systemd[1]: Stopped Nix worker-protocol over gRPC.6718worker1 # [ 148.238398] systemd[1]: nix-grpc-daemon.service: Consumed 1.285s CPU time over 31.937s wall clock time, 35.9M memory peak, 48.1K incoming IP traffic, 96.4K outgoing IP traffic.6719worker1 # [ 148.245481] systemd[1]: Starting Nix worker-protocol over gRPC...6720worker1 # [ 148.368742] nix-grpc-daemon[1597]: ts=2026-09-22T11:23:51Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6721worker1 # [ 148.370487] nix-grpc-daemon[1597]: ts=2026-09-22T11:23:51Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6722worker1 # [ 148.373924] nix-grpc-daemon[1597]: ts=2026-09-22T11:23:51Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16723worker1 # [ 148.381063] nix-grpc-daemon[1597]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6724worker1 # [ 148.383094] nix-grpc-daemon[1597]: E0000 00:00:1790076231.613212 1597 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6725worker1 # [ 148.386946] nix-grpc-daemon[1597]: E0000 00:00:1790076231.618347 1597 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6726worker1 # [ 148.389551] nix-grpc-daemon[1597]: ts=2026-09-22T11:23:51Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6727worker1 # [ 148.392803] systemd[1]: Started Nix worker-protocol over gRPC.6728worker1 # [ 148.399067] nix-grpc-daemon[1597]: E0000 00:00:1790076231.631094 1639 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6729worker1 # [ 148.410581] nix-grpc-daemon[1597]: E0000 00:00:1790076231.643058 1639 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6730worker1 # [ 148.414117] nix-grpc-daemon[1597]: E0000 00:00:1790076231.645293 1639 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6731worker1 # [ 148.424587] nix-grpc-daemon[1597]: E0000 00:00:1790076231.657064 1639 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6732worker1 # [ 148.432087] nix-grpc-daemon[1597]: ts=2026-09-22T11:23:51Z level=info event=scheduler_connected addr=lb:500516733worker2 # [ 148.446120] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:52Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:357726734worker2 # [ 148.449179] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:52Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06735lb: must succeed: curl -sf localhost:9901/clusters6736lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6737worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6738worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6739(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.50 seconds)6740subtest: scheduler restart mid-build: build finishes, no double build6741client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid6742client # [ 148.904651] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.6743client # Running as unit: mid.service; invocation ID: 956d99a393694d3c8ae778ad6ee92cc66744client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.04 seconds)6745worker2 # [ 149.101754] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:576366746client # [ 149.102790] nix[1393]: don't know how to build these paths:6747client # [ 149.103697] nix[1393]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6748worker2 # [ 149.110630] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:53Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06749worker2 # [ 149.112719] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:53Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=16750worker2 # [ 149.154550] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:576366751worker2 # [ 149.166745] nix-grpc-daemon[1507]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 61 ms (attempt 1/5)6752worker2 # [ 149.233609] nix-grpc-daemon[1507]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 103 ms (attempt 2/5)6753worker2 # [ 149.342806] nix-grpc-daemon[1507]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 339 ms (attempt 3/5)6754worker2 # [ 149.684827] nix-grpc-daemon[1507]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 17 ms (attempt 4/5)6755worker2 # [ 149.701722] nix-grpc-daemon[1507]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6756worker2 # [ 149.703454] nix-grpc-daemon[1507]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6757worker2 # [ 149.715126] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:576366758client # [ 149.719086] nix[1393]: copying 1 paths to 'lb:50051'...6759client # [ 149.719895] nix[1393]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6760worker2 # [ 149.722926] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:576366761worker2 # [ 149.726691] nix-daemon[877]: accepted connection from pid 1507, user nix-grpc-daemon (trusted)6762worker2 # [ 149.738891] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:53Z level=info event=niks3_push_started pid=16356763worker2 # [ 149.899429] nix-grpc-daemon[1635]: time=2026-09-22T11:23:53.969Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6764worker2 # [ 150.029227] nix-daemon[877]: accepted connection from pid 1648, user nix-grpc-daemon (trusted)6765worker2 # [ 150.040159] nix-daemon[877]: reaped child process 1651, status = succeeded6766worker2 # [ 150.188428] nix-daemon[877]: accepted connection from pid 1653, user nix-grpc-daemon (trusted)6767worker2 # [ 150.200478] nix-daemon[877]: reaped child process 1660, status = succeeded6768lb # [ 150.221362] niks3-server[1194]: 2026/09/22 11:23:53 INFO Received uploads request method=POST path=/api/pending_closures6769worker2 # [ 150.219059] nix-grpc-daemon[1635]: time=2026-09-22T11:23:54.289Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6770worker2 # [ 150.221873] nix-grpc-daemon[1635]: time=2026-09-22T11:23:54.291Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6771lb # [ 150.252425] niks3-server[1194]: 2026/09/22 11:23:53 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6772lb # [ 150.262120] niks3-server[1194]: 2026/09/22 11:23:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6773lb # [ 150.268124] niks3-server[1194]: 2026/09/22 11:23:53 INFO Signed narinfos id=51 count=16774worker2 # [ 150.261141] nix-grpc-daemon[1635]: time=2026-09-22T11:23:54.331Z level=INFO msg="Uploading 1 narinfos"6775lb # [ 150.278517] niks3-server[1194]: 2026/09/22 11:23:53 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6776lb # [ 150.287401] niks3-server[1194]: 2026/09/22 11:23:53 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6777lb # [ 150.295042] niks3-server[1194]: 2026/09/22 11:23:53 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6778worker2 # [ 150.288795] nix-grpc-daemon[1635]: time=2026-09-22T11:23:54.359Z level=INFO msg="Upload complete. (385ms)"6779lb # [ 150.298803] niks3-server[1194]: 2026/09/22 11:23:53 INFO Completed upload id=516780worker2 # [ 150.291718] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57636 duration_s=0 paths=1 nar_bytes_in=4246781worker2 # [ 150.296435] nix-grpc-daemon[1507]: ts=2026-09-22T11:23:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:576366782worker2 # [ 150.305465] nix-daemon[877]: accepted connection from pid 1507, user nix-grpc-daemon (trusted)6783worker2 # [ 150.310537] nix-daemon[877]: reaped child process 1633, status = succeeded6784worker2 # [ 150.316367] nix-daemon[877]: accepted connection from pid 1507, user nix-grpc-daemon (trusted)6785client # [ 150.313131] nix[1393]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6786lb: must succeed: curl -sf localhost:9901/clusters6787lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6788lb: must succeed: curl -sf localhost:9901/clusters6789lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6790worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6791worker2 # [ 152.631276] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1507 (nix-grpc-daemon) on client request.6792worker2 # [ 152.634422] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6793lb # [ 152.649688] niks3-server[1194]: 2026/09/22 11:23:56 INFO lead: released remote=192.168.1.4:548026794client # [ 152.644899] nix[1393]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6795client # [ 152.647099] nix[1393]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6796worker1 # [ 152.635938] nix-grpc-daemon[1597]: ts=2026-09-22T11:23:55Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6797worker2 # [ 152.653135] systemd[1]: Starting scheduler-token.service...6798worker2 # [ 152.663253] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6799worker2 # [ 152.665859] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6800worker2 # [ 152.670299] systemd[1]: nix-grpc-daemon.service: Consumed 416ms CPU time over 16.738s wall clock time, 33.5M memory peak, 14.5K incoming IP traffic, 27.7K outgoing IP traffic.6801worker2 # [ 152.678679] nix-daemon[877]: reaped child process 1566, status = succeeded6802worker2 # [ 152.680207] nix-daemon[877]: reaped child process 1662, status = succeeded6803worker2 # [ 152.691277] nix-daemon[1664]: unexpected Nix daemon error: error: interrupted by the user6804worker2 # [ 152.695168] nix-daemon[877]: reaped child process 1664, status = failed with exit code 16805worker2 # [ 152.745504] systemd[1]: scheduler-token.service: Deactivated successfully.6806worker2 # [ 152.747269] systemd[1]: Finished scheduler-token.service.6807worker2 # [ 152.748708] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 94ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6808worker2 # [ 152.751970] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6809worker2 # [ 152.756637] systemd[1]: Starting scheduler-token.service...6810worker2 # [ 152.810882] systemd[1]: scheduler-token.service: Deactivated successfully.6811worker2 # [ 152.812235] systemd[1]: Finished scheduler-token.service.6812worker2 # [ 152.815128] systemd[1]: scheduler-token.service: Consumed 16ms CPU time over 54ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6813worker2 # [ 152.819121] systemd[1]: Starting Nix worker-protocol over gRPC...6814worker1 # [ 152.835892] nix-grpc-daemon[1597]: ts=2026-09-22T11:23:56Z level=info event=scheduler_connected addr=lb:500516815worker2 # [ 152.973723] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6816worker2 # [ 152.976302] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6817worker2 # [ 152.980498] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16818worker2 # [ 152.988061] nix-grpc-daemon[1707]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6819worker2 # [ 152.991105] nix-grpc-daemon[1707]: E0000 00:00:1790076237.058227 1707 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6820worker2 # [ 152.995120] nix-grpc-daemon[1707]: E0000 00:00:1790076237.061450 1707 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6821worker2 # [ 153.000372] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6822worker2 # [ 153.004718] systemd[1]: Started Nix worker-protocol over gRPC.6823worker1 # [ 152.995773] nix-grpc-daemon[1597]: ts=2026-09-22T11:23:56Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6824worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.42 seconds)6825client: must succeed: sleep 5.06826lb # [ 153.020281] niks3-server[1194]: 2026/09/22 11:23:56 INFO lead: acquired remote=192.168.1.4:484106827worker2 # [ 153.012068] nix-grpc-daemon[1707]: E0000 00:00:1790076237.081526 1749 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6828worker2 # [ 153.015754] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=info event=scheduler_take_over6829worker2 # [ 153.025120] nix-grpc-daemon[1707]: E0000 00:00:1790076237.095536 1749 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6830worker2 # [ 153.030056] nix-grpc-daemon[1707]: E0000 00:00:1790076237.098095 1749 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6831worker2 # [ 153.039788] nix-grpc-daemon[1707]: E0000 00:00:1790076237.110199 1749 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6832worker2 # [ 153.048948] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=info event=scheduler_connected addr=lb:500516833worker2 # [ 153.054713] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=info event=oidc_jwks provider=mock keys=16834worker2 # [ 153.056334] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:387346835worker2 # [ 153.058193] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06836worker2 # [ 153.159217] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06837worker2 # [ 153.164155] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=16838worker1 # [ 153.197146] nix-grpc-daemon[1597]: ts=2026-09-22T11:23:56Z level=info event=scheduler_connected addr=lb:500516839worker2 # [ 153.207145] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:387586840worker2 # [ 153.212992] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:387346841worker2 # [ 153.216504] nix-grpc-daemon[1707]: ts=2026-09-22T11:23:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06842worker2 # [ 153.226394] nix-daemon[877]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6843worker2 # [ 153.241512] nix-daemon[877]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6844client # [ 153.249734] nix[1393]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6845worker2 # [ 153.253762] nix-daemon[877]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6846lb # [ 156.784682] envoy[899]: conn peer=[2001:db8:1::3]:37638 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=4581 tx=1106 ms=306386847lb # [ 156.786231] envoy[899]: conn peer=[2001:db8:1::4]:47426 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=2046 tx=311 ms=165106848client: (finished: must succeed: sleep 5.0, in 5.05 seconds)6849client: waiting for success: ! systemctl is-active mid6850worker2 # [ 158.471763] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:02Z level=info event=niks3_push_started pid=17906851worker2 # [ 158.500914] nix-daemon[877]: reaped child process 1765, status = succeeded6852worker2 # [ 158.661741] nix-grpc-daemon[1790]: time=2026-09-22T11:24:02.731Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6853worker2 # [ 158.811633] nix-daemon[877]: accepted connection from pid 1803, user nix-grpc-daemon (trusted)6854worker2 # [ 158.823827] nix-daemon[877]: reaped child process 1806, status = succeeded6855lb # [ 158.852221] niks3-server[1194]: 2026/09/22 11:24:02 INFO Received uploads request method=POST path=/api/pending_closures6856worker2 # [ 158.851179] nix-grpc-daemon[1790]: time=2026-09-22T11:24:02.921Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6857worker2 # [ 158.852719] nix-grpc-daemon[1790]: time=2026-09-22T11:24:02.923Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6858lb # [ 158.902234] niks3-server[1194]: 2026/09/22 11:24:02 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6859lb # [ 158.907980] niks3-server[1194]: 2026/09/22 11:24:02 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6860lb # [ 158.918800] niks3-server[1194]: 2026/09/22 11:24:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6861lb # [ 158.922397] niks3-server[1194]: 2026/09/22 11:24:02 INFO Signed narinfos id=52 count=16862worker2 # [ 158.915226] nix-grpc-daemon[1790]: time=2026-09-22T11:24:02.985Z level=INFO msg="Uploading 1 narinfos"6863lb # [ 158.929188] niks3-server[1194]: 2026/09/22 11:24:02 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6864lb # [ 158.938177] niks3-server[1194]: 2026/09/22 11:24:02 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6865lb # [ 158.941946] niks3-server[1194]: 2026/09/22 11:24:02 INFO Completed upload id=526866worker2 # [ 158.932252] nix-grpc-daemon[1790]: time=2026-09-22T11:24:03.002Z level=INFO msg="Upload complete. (266ms)"6867worker2 # [ 158.940108] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38758 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16868worker2 # [ 158.952163] nix-daemon[877]: reaped child process 1763, status = succeeded6869client # [ 158.970967] systemd[1]: mid.service: Deactivated successfully.6870client # [ 158.972476] systemd[1]: mid.service: Consumed 124ms CPU time over 10.067s wall clock time, 41.4M memory peak, 8.6K incoming IP traffic, 10.4K outgoing IP traffic.6871client: (finished: waiting for success: ! systemctl is-active mid, in 1.11 seconds)6872client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6873client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds)6874(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.56 seconds)6875subtest: interrupting the client stops the build on the worker6876client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6877client # [ 159.464237] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6878client # Running as unit: intr.service; invocation ID: 3ad0f94aea244623b044dbf41ed30da16879client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.04 seconds)6880worker1 # [ 159.669974] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:377206881worker1 # [ 159.674896] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6882client # [ 159.689897] nix[1429]: don't know how to build these paths:6883client # [ 159.690839] nix[1429]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6884worker2 # [ 159.700382] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:03Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06885worker2 # [ 159.704499] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:03Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.3:50051 clients=16886worker1 # [ 159.731069] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377206887worker1 # [ 159.737221] nix-grpc-daemon[1597]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 64 ms (attempt 1/5)6888worker1 # [ 159.803721] nix-grpc-daemon[1597]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 134 ms (attempt 2/5)6889worker1 # [ 159.940350] nix-grpc-daemon[1597]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 315 ms (attempt 3/5)6890worker1 # [ 160.255539] nix-grpc-daemon[1597]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 482 ms (attempt 4/5)6891worker1 # [ 160.738200] nix-grpc-daemon[1597]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6892worker1 # [ 160.740257] nix-grpc-daemon[1597]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6893worker1 # [ 160.756829] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377206894worker1 # [ 160.764758] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377206895client # [ 160.773815] nix[1429]: copying 1 paths to 'lb:50051'...6896client # [ 160.777076] nix[1429]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6897worker1 # [ 160.771982] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6898worker1 # [ 160.781893] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:04Z level=info event=niks3_push_started pid=16936899worker1 # [ 160.941950] nix-grpc-daemon[1693]: time=2026-09-22T11:24:04.174Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6900worker1 # [ 161.091117] nix-daemon[855]: accepted connection from pid 1706, user nix-grpc-daemon (trusted)6901worker1 # [ 161.101558] nix-daemon[855]: reaped child process 1709, status = succeeded6902worker1 # [ 161.246918] nix-daemon[855]: accepted connection from pid 1711, user nix-grpc-daemon (trusted)6903worker1 # [ 161.258397] nix-daemon[855]: reaped child process 1714, status = succeeded6904lb # [ 161.294252] niks3-server[1194]: 2026/09/22 11:24:04 INFO Received uploads request method=POST path=/api/pending_closures6905worker1 # [ 161.283191] nix-grpc-daemon[1693]: time=2026-09-22T11:24:04.515Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6906worker1 # [ 161.285562] nix-grpc-daemon[1693]: time=2026-09-22T11:24:04.517Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6907lb # [ 161.335950] niks3-server[1194]: 2026/09/22 11:24:04 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6908lb # [ 161.341767] niks3-server[1194]: 2026/09/22 11:24:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6909lb # [ 161.347247] niks3-server[1194]: 2026/09/22 11:24:04 INFO Signed narinfos id=53 count=16910worker1 # [ 161.328711] nix-grpc-daemon[1693]: time=2026-09-22T11:24:04.560Z level=INFO msg="Uploading 1 narinfos"6911lb # [ 161.358440] niks3-server[1194]: 2026/09/22 11:24:04 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6912lb # [ 161.363532] niks3-server[1194]: 2026/09/22 11:24:04 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6913lb # [ 161.372831] niks3-server[1194]: 2026/09/22 11:24:04 INFO Completed upload id=536914worker1 # [ 161.351224] nix-grpc-daemon[1693]: time=2026-09-22T11:24:04.583Z level=INFO msg="Upload complete. (405ms)"6915lb # [ 161.374847] niks3-server[1194]: 2026/09/22 11:24:04 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6916worker1 # [ 161.354800] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=4246917worker1 # [ 161.360131] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377206918worker1 # [ 161.368504] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6919worker1 # [ 161.375128] nix-daemon[855]: reaped child process 1691, status = succeeded6920worker1 # [ 161.378089] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6921client # [ 161.386421] nix[1429]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6922client: must succeed: systemctl kill -s INT intr6923client # [ 162.837547] systemd[1]: intr.service: Sent signal SIGINT to main process 1429 (nix) on client request.6924worker1 # [ 162.830433] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:06Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6925client # [ 162.843888] nix[1429]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6926client # [ 162.849667] nix[1429]: error: interrupted by the user6927client: (finished: must succeed: systemctl kill -s INT intr, in 0.05 seconds)6928client # [ 162.884396] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6929client # [ 162.885765] systemd[1]: intr.service: Failed with result 'exit-code'.6930client # [ 162.887728] systemd[1]: intr.service: Consumed 114ms CPU time over 3.422s wall clock time, 39.4M memory peak, 6.5K incoming IP traffic, 8.3K outgoing IP traffic.6931worker1 # [ 162.995255] nix-daemon[855]: reaped child process 1716, status = succeeded6932worker1 # [ 162.998685] nix-daemon[1718]: unexpected Nix daemon error: error: interrupted by the user6933worker1 # [ 163.002434] nix-daemon[855]: reaped child process 1718, status = failed with exit code 16934(finished: subtest: interrupting the client stops the build on the worker, in 4.63 seconds)6935subtest: a deploy mid-build drains: the build finishes, the new generation takes over6936client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6937client # [ 164.117235] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6938client # Running as unit: sw.service; invocation ID: c1c3d2c5c6584ce4a993bbaa96c4a5616939client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.07 seconds)6940client # [ 164.324055] nix[1450]: don't know how to build these paths:6941worker2 # [ 164.325778] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:387586942client # [ 164.324858] nix[1450]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6943worker2 # [ 164.331481] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:08Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06944worker2 # [ 164.333522] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:08Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.4:50051 clients=16945worker2 # [ 164.374102] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:387586946worker2 # [ 164.380033] nix-grpc-daemon[1707]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 36 ms (attempt 1/5)6947worker2 # [ 164.418502] nix-grpc-daemon[1707]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 22 ms (attempt 2/5)6948worker2 # [ 164.442952] nix-grpc-daemon[1707]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 194 ms (attempt 3/5)6949worker2 # [ 164.639629] nix-grpc-daemon[1707]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 777 ms (attempt 4/5)6950worker2 # [ 165.420396] nix-grpc-daemon[1707]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6951worker2 # [ 165.421828] nix-grpc-daemon[1707]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6952worker2 # [ 165.433964] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:387586953client # [ 165.438271] nix[1450]: copying 1 paths to 'lb:50051'...6954worker2 # [ 165.444238] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:387586955client # [ 165.441576] nix[1450]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6956worker2 # [ 165.450945] nix-daemon[877]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6957worker2 # [ 165.603801] nix-daemon[877]: accepted connection from pid 1850, user nix-grpc-daemon (trusted)6958worker2 # [ 165.615910] nix-daemon[877]: reaped child process 1853, status = succeeded6959worker2 # [ 165.762447] nix-daemon[877]: accepted connection from pid 1855, user nix-grpc-daemon (trusted)6960worker2 # [ 165.773230] nix-daemon[877]: reaped child process 1858, status = succeeded6961lb # [ 165.795646] niks3-server[1194]: 2026/09/22 11:24:09 INFO Received uploads request method=POST path=/api/pending_closures6962worker2 # [ 165.794633] nix-grpc-daemon[1790]: time=2026-09-22T11:24:09.864Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6963worker2 # [ 165.796273] nix-grpc-daemon[1790]: time=2026-09-22T11:24:09.866Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6964lb # [ 165.824481] niks3-server[1194]: 2026/09/22 11:24:09 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6965lb # [ 165.834998] niks3-server[1194]: 2026/09/22 11:24:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6966lb # [ 165.839841] niks3-server[1194]: 2026/09/22 11:24:09 INFO Signed narinfos id=54 count=16967worker2 # [ 165.833106] nix-grpc-daemon[1790]: time=2026-09-22T11:24:09.903Z level=INFO msg="Uploading 1 narinfos"6968lb # [ 165.851135] niks3-server[1194]: 2026/09/22 11:24:09 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6969lb # [ 165.857408] niks3-server[1194]: 2026/09/22 11:24:09 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6970worker2 # [ 165.852508] nix-grpc-daemon[1790]: time=2026-09-22T11:24:09.922Z level=INFO msg="Upload complete. (395ms)"6971lb # [ 165.863286] niks3-server[1194]: 2026/09/22 11:24:09 INFO Completed upload id=546972worker2 # [ 165.854589] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38758 duration_s=0 paths=1 nar_bytes_in=4166973lb # [ 165.867254] niks3-server[1194]: 2026/09/22 11:24:09 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6974worker2 # [ 165.859533] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:387586975worker2 # [ 165.865922] nix-daemon[877]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6976worker2 # [ 165.868210] nix-daemon[877]: reaped child process 1848, status = succeeded6977client # [ 165.869795] nix[1450]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6978worker2 # [ 165.873387] nix-daemon[877]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6979lb # [ 166.785391] envoy[899]: conn peer=[2001:db8:1::1]:53116 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3934 tx=1941 ms=98556980lb # [ 166.786971] envoy[899]: conn peer=[2001:db8:1::1]:57326 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2993 tx=851 ms=31886981worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&26982worker2 # Checking switch inhibitors... done6983worker2 # [ 168.230848] nixos[1894]: switching to system configuration /nix/store/f4fgvcyl12l15s7pxq6ja191ka2vx57d-nixos-system-worker2-test6984worker2 # [ 168.234637] systemd[1]: Stopped target Remote File Systems.6985worker2 # activating the configuration...6986worker2 # [ 168.236865] systemd[1]: Stopped target Local File Systems.6987worker2 # setting up /etc...6988worker2 # [ 168.829587] systemd[1]: Reload requested from client PID 1894 ('.switch-to-conf') (unit backdoor.service)...6989worker2 # [ 168.831271] systemd[1]: Reloading...6990worker2 # [ 169.482265] systemd[1]: Reloading finished in 646 ms.6991worker2 # restarting sysinit-reactivation.target6992worker2 # [ 169.535316] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....6993worker2 # [ 169.668250] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..6994worker2 # [ 169.673033] systemd[1]: Reached target Reactivate sysinit units.6995worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service6996worker2 # [ 169.677221] systemd[1]: Reloading D-Bus System Message Bus...6997worker2 # [ 169.679290] dbus-broker-launch[536]: Caught SIGHUP, trigger reload.6998worker2 # [ 169.681775] systemd[1]: Reloading Nix worker-protocol over gRPC...6999worker2 # [ 169.713074] dbus-broker-launch[536]: Looking up NSS user entry for 'systemd-timesync'...7000worker2 # [ 169.714916] dbus-broker-launch[536]: NSS returned no entry for 'systemd-timesync'7001worker2 # [ 169.717098] dbus-broker-launch[536]: Invalid user-name in /nix/store/s93qka0z548yjbxqnmj4c4b32apww6jv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"7002worker2 # [ 169.726228] systemd[1]: Reloaded Nix worker-protocol over gRPC.7003worker2 # [ 169.744572] systemd[1]: Reloaded D-Bus System Message Bus.7004worker2 # [ 169.746396] systemd[1]: Reached target Local File Systems.7005worker2 # [ 169.753071] systemd[1]: Reached target Remote File Systems.7006worker2 # [ 169.757434] systemd[1]: Starting linger-users.service...7007worker2 # [ 169.760318] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os7008worker2 # [ 169.761801] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki7009worker2 # [ 169.773842] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore7010worker2 # [ 169.778080] systemd[1]: Starting Create SUID/SGID Wrappers...7011worker2 # [ 169.779261] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.7012worker2 # [ 169.782173] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc677013worker2 # [ 169.786755] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore7014worker2 # [ 169.788383] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os7015worker2 # [ 169.790108] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os7016worker2 # [ 169.822814] systemd[1]: linger-users.service: Deactivated successfully.7017worker2 # [ 169.824127] systemd[1]: Finished linger-users.service.7018worker2 # [ 170.000559] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:14Z level=info event=draining7019worker2 # [ 170.185584] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.7020worker2 # [ 170.188763] systemd[1]: Finished Create SUID/SGID Wrappers.7021worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service7022worker2 # [ 170.446349] nixos[1894]: finished switching to system configuration /nix/store/f4fgvcyl12l15s7pxq6ja191ka2vx57d-nixos-system-worker2-test7023worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 2.83 seconds)7024worker2: must succeed: pgrep -f 'read -t [9]0 x'7025worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)7026client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&27027worker1 # [ 170.742708] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:377207028client # don't know how to build these paths:7029client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv7030worker1 # [ 170.751492] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207031worker2 # [ 170.783607] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:14Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=07032worker2 # [ 170.785580] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:14Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.3:50051 clients=17033worker1 # [ 170.823181] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207034worker1 # [ 170.837229] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207035client # copying 1 paths to 'lb:50051'...7036worker1 # [ 170.849326] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207037client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...7038worker1 # [ 170.868932] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7039worker1 # [ 171.016140] nix-daemon[855]: accepted connection from pid 1773, user nix-grpc-daemon (trusted)7040worker1 # [ 171.027691] nix-daemon[855]: reaped child process 1776, status = succeeded7041worker1 # [ 171.178361] nix-daemon[855]: accepted connection from pid 1778, user nix-grpc-daemon (trusted)7042worker1 # [ 171.189786] nix-daemon[855]: reaped child process 1781, status = succeeded7043lb # [ 171.225810] niks3-server[1194]: 2026/09/22 11:24:14 INFO Received uploads request method=POST path=/api/pending_closures7044worker1 # [ 171.215222] nix-grpc-daemon[1693]: time=2026-09-22T11:24:14.447Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7045worker1 # [ 171.216896] nix-grpc-daemon[1693]: time=2026-09-22T11:24:14.449Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"7046lb # [ 171.256348] niks3-server[1194]: 2026/09/22 11:24:14 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst7047lb # [ 171.266563] niks3-server[1194]: 2026/09/22 11:24:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign7048lb # [ 171.269860] niks3-server[1194]: 2026/09/22 11:24:14 INFO Signed narinfos id=55 count=17049worker1 # [ 171.251202] nix-grpc-daemon[1693]: time=2026-09-22T11:24:14.483Z level=INFO msg="Uploading 1 narinfos"7050lb # [ 171.278994] niks3-server[1194]: 2026/09/22 11:24:14 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls7051lb # [ 171.287130] niks3-server[1194]: 2026/09/22 11:24:14 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete7052lb # [ 171.294609] niks3-server[1194]: 2026/09/22 11:24:14 INFO Completed upload id=557053worker1 # [ 171.273222] nix-grpc-daemon[1693]: time=2026-09-22T11:24:14.505Z level=INFO msg="Upload complete. (390ms)"7054lb # [ 171.296281] niks3-server[1194]: 2026/09/22 11:24:14 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo7055worker1 # [ 171.278384] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=5287056worker1 # [ 171.283295] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207057worker1 # [ 171.291356] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7058worker1 # [ 171.296323] nix-daemon[855]: reaped child process 1771, status = succeeded7059worker1 # [ 171.301123] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7060client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...7061client # farm-a> node-a: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv7062client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7063client # farm-a> LOG-farm-a-during-drain7064worker1 # [ 172.731725] nix-daemon[855]: reaped child process 1785, status = succeeded7065worker1 # [ 172.853049] nix-daemon[855]: accepted connection from pid 1801, user nix-grpc-daemon (trusted)7066worker1 # [ 172.863465] nix-daemon[855]: reaped child process 1804, status = succeeded7067lb # [ 172.898134] niks3-server[1194]: 2026/09/22 11:24:16 INFO Received uploads request method=POST path=/api/pending_closures7068worker1 # [ 172.880941] nix-grpc-daemon[1693]: time=2026-09-22T11:24:16.113Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7069worker1 # [ 172.882439] nix-grpc-daemon[1693]: time=2026-09-22T11:24:16.114Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"7070lb # [ 172.915477] niks3-server[1194]: 2026/09/22 11:24:16 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv7071lb # [ 172.921671] niks3-server[1194]: 2026/09/22 11:24:16 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst7072lb # [ 172.927600] niks3-server[1194]: 2026/09/22 11:24:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign7073worker1 # [ 172.911193] nix-grpc-daemon[1693]: time=2026-09-22T11:24:16.143Z level=INFO msg="Uploading 1 narinfos"7074lb # [ 172.932834] niks3-server[1194]: 2026/09/22 11:24:16 INFO Signed narinfos id=56 count=17075lb # [ 172.937392] niks3-server[1194]: 2026/09/22 11:24:16 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls7076lb # [ 172.942726] niks3-server[1194]: 2026/09/22 11:24:16 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete7077lb # [ 172.945904] niks3-server[1194]: 2026/09/22 11:24:16 INFO Completed upload id=567078worker1 # [ 172.924206] nix-grpc-daemon[1693]: time=2026-09-22T11:24:16.156Z level=INFO msg="Upload complete. (215ms)"7079lb # [ 172.947374] niks3-server[1194]: 2026/09/22 11:24:16 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo7080worker1 # [ 172.926649] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=17081worker1 # [ 172.933102] nix-daemon[855]: reaped child process 1783, status = succeeded7082worker2 # [ 172.945363] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:17Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=07083worker2 # [ 172.948269] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:17Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.3:50051 clients=17084worker1 # [ 172.978329] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207085worker1 # [ 172.989297] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207086worker1 # [ 172.999137] nix-grpc-daemon[1693]: time=2026-09-22T11:24:16.231Z level=INFO msg="All 1 paths already cached"7087client # copying 1 paths to 'lb:50051'...7088worker1 # [ 173.006858] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207089client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...7090worker1 # [ 173.027161] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7091worker1 # [ 173.156273] nix-daemon[855]: accepted connection from pid 1808, user nix-grpc-daemon (trusted)7092worker1 # [ 173.168106] nix-daemon[855]: reaped child process 1811, status = succeeded7093worker1 # [ 173.297237] nix-daemon[855]: accepted connection from pid 1813, user nix-grpc-daemon (trusted)7094worker1 # [ 173.308746] nix-daemon[855]: reaped child process 1816, status = succeeded7095lb # [ 173.344196] niks3-server[1194]: 2026/09/22 11:24:16 INFO Received uploads request method=POST path=/api/pending_closures7096worker1 # [ 173.330951] nix-grpc-daemon[1693]: time=2026-09-22T11:24:16.563Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7097worker1 # [ 173.332707] nix-grpc-daemon[1693]: time=2026-09-22T11:24:16.565Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"7098lb # [ 173.376929] niks3-server[1194]: 2026/09/22 11:24:16 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst7099lb # [ 173.388864] niks3-server[1194]: 2026/09/22 11:24:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign7100worker1 # [ 173.373084] nix-grpc-daemon[1693]: time=2026-09-22T11:24:16.605Z level=INFO msg="Uploading 1 narinfos"7101lb # [ 173.394978] niks3-server[1194]: 2026/09/22 11:24:16 INFO Signed narinfos id=57 count=17102lb # [ 173.399603] niks3-server[1194]: 2026/09/22 11:24:16 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls7103lb # [ 173.410321] niks3-server[1194]: 2026/09/22 11:24:16 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete7104lb # [ 173.416470] niks3-server[1194]: 2026/09/22 11:24:16 INFO Completed upload id=577105worker1 # [ 173.394761] nix-grpc-daemon[1693]: time=2026-09-22T11:24:16.627Z level=INFO msg="Upload complete. (355ms)"7106lb # [ 173.417901] niks3-server[1194]: 2026/09/22 11:24:16 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo7107worker1 # [ 173.398426] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=6407108worker1 # [ 173.403082] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207109worker1 # [ 173.412111] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7110worker1 # [ 173.416457] nix-daemon[855]: reaped child process 1806, status = succeeded7111worker1 # [ 173.421336] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7112client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...7113client # farm-b> node-a: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv7114client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7115client # farm-b> LOG-farm-b-during-drain7116worker1 # [ 174.048144] nix-daemon[855]: reaped child process 1820, status = succeeded7117worker1 # [ 174.201086] nix-daemon[855]: accepted connection from pid 1836, user nix-grpc-daemon (trusted)7118worker1 # [ 174.212892] nix-daemon[855]: reaped child process 1839, status = succeeded7119lb # [ 174.243862] niks3-server[1194]: 2026/09/22 11:24:17 INFO Received uploads request method=POST path=/api/pending_closures7120worker1 # [ 174.227140] nix-grpc-daemon[1693]: time=2026-09-22T11:24:17.459Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7121worker1 # [ 174.228611] nix-grpc-daemon[1693]: time=2026-09-22T11:24:17.461Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"7122lb # [ 174.263595] niks3-server[1194]: 2026/09/22 11:24:17 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv7123lb # [ 174.267807] niks3-server[1194]: 2026/09/22 11:24:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign7124lb # [ 174.270140] niks3-server[1194]: 2026/09/22 11:24:17 INFO Signed narinfos id=58 count=17125worker1 # [ 174.249937] nix-grpc-daemon[1693]: time=2026-09-22T11:24:17.482Z level=INFO msg="Uploading 1 narinfos"7126lb # [ 174.272804] niks3-server[1194]: 2026/09/22 11:24:17 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst7127lb # [ 174.276852] niks3-server[1194]: 2026/09/22 11:24:17 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls7128lb # [ 174.281056] niks3-server[1194]: 2026/09/22 11:24:17 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete7129lb # [ 174.285076] niks3-server[1194]: 2026/09/22 11:24:17 INFO Completed upload id=587130worker1 # [ 174.264632] nix-grpc-daemon[1693]: time=2026-09-22T11:24:17.497Z level=INFO msg="Upload complete. (237ms)"7131lb # [ 174.286792] niks3-server[1194]: 2026/09/22 11:24:17 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo7132worker1 # [ 174.267054] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=17133worker2 # [ 174.283658] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:18Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=07134worker1 # [ 174.273801] nix-daemon[855]: reaped child process 1818, status = succeeded7135worker2 # [ 174.285400] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:18Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.3:50051 clients=17136worker1 # [ 174.314841] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207137worker1 # [ 174.324190] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207138worker1 # [ 174.333549] nix-grpc-daemon[1693]: time=2026-09-22T11:24:17.565Z level=INFO msg="All 2 paths already cached"7139client # copying 1 paths to 'lb:50051'...7140worker1 # [ 174.340101] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207141client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...7142worker1 # [ 174.348948] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7143worker1 # [ 174.492129] nix-daemon[855]: accepted connection from pid 1843, user nix-grpc-daemon (trusted)7144worker1 # [ 174.503759] nix-daemon[855]: reaped child process 1846, status = succeeded7145worker1 # [ 174.651294] nix-daemon[855]: accepted connection from pid 1848, user nix-grpc-daemon (trusted)7146worker1 # [ 174.662771] nix-daemon[855]: reaped child process 1851, status = succeeded7147lb # [ 174.697972] niks3-server[1194]: 2026/09/22 11:24:18 INFO Received uploads request method=POST path=/api/pending_closures7148worker1 # [ 174.680308] nix-grpc-daemon[1693]: time=2026-09-22T11:24:17.912Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7149worker1 # [ 174.681834] nix-grpc-daemon[1693]: time=2026-09-22T11:24:17.914Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"7150lb # [ 174.713635] niks3-server[1194]: 2026/09/22 11:24:18 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst7151lb # [ 174.716564] niks3-server[1194]: 2026/09/22 11:24:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign7152lb # [ 174.718333] niks3-server[1194]: 2026/09/22 11:24:18 INFO Signed narinfos id=59 count=17153worker1 # [ 174.698680] nix-grpc-daemon[1693]: time=2026-09-22T11:24:17.931Z level=INFO msg="Uploading 1 narinfos"7154lb # [ 174.722081] niks3-server[1194]: 2026/09/22 11:24:18 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls7155lb # [ 174.726363] niks3-server[1194]: 2026/09/22 11:24:18 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete7156lb # [ 174.729714] niks3-server[1194]: 2026/09/22 11:24:18 INFO Completed upload id=597157worker1 # [ 174.707918] nix-grpc-daemon[1693]: time=2026-09-22T11:24:17.940Z level=INFO msg="Upload complete. (350ms)"7158lb # [ 174.731228] niks3-server[1194]: 2026/09/22 11:24:18 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo7159worker1 # [ 174.710042] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=7687160worker1 # [ 174.713478] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207161worker1 # [ 174.717466] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7162worker1 # [ 174.721846] nix-daemon[855]: reaped child process 1841, status = succeeded7163client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...7164client # farm-top> node-a: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv7165worker1 # [ 174.726358] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7166client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7167client # farm-top> LOG-farm-top-during-drain7168worker1 # [ 175.139381] nix-daemon[855]: reaped child process 1855, status = succeeded7169worker1 # [ 175.247751] nix-daemon[855]: accepted connection from pid 1871, user nix-grpc-daemon (trusted)7170worker1 # [ 175.259575] nix-daemon[855]: reaped child process 1874, status = succeeded7171lb # [ 175.292800] niks3-server[1194]: 2026/09/22 11:24:18 INFO Received uploads request method=POST path=/api/pending_closures7172worker1 # [ 175.279601] nix-grpc-daemon[1693]: time=2026-09-22T11:24:18.511Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7173worker1 # [ 175.281050] nix-grpc-daemon[1693]: time=2026-09-22T11:24:18.513Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"7174lb # [ 175.329834] niks3-server[1194]: 2026/09/22 11:24:18 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst7175lb # [ 175.338203] niks3-server[1194]: 2026/09/22 11:24:18 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv7176lb # [ 175.346213] niks3-server[1194]: 2026/09/22 11:24:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign7177worker1 # [ 175.328484] nix-grpc-daemon[1693]: time=2026-09-22T11:24:18.560Z level=INFO msg="Uploading 1 narinfos"7178lb # [ 175.350226] niks3-server[1194]: 2026/09/22 11:24:18 INFO Signed narinfos id=60 count=17179lb # [ 175.356384] niks3-server[1194]: 2026/09/22 11:24:18 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls7180lb # [ 175.364985] niks3-server[1194]: 2026/09/22 11:24:18 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete7181lb # [ 175.368256] niks3-server[1194]: 2026/09/22 11:24:18 INFO Completed upload id=607182worker1 # [ 175.346240] nix-grpc-daemon[1693]: time=2026-09-22T11:24:18.578Z level=INFO msg="Upload complete. (214ms)"7183lb # [ 175.370902] niks3-server[1194]: 2026/09/22 11:24:18 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo7184worker1 # [ 175.350629] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=17185worker1 # [ 175.362189] nix-daemon[855]: reaped child process 1853, status = succeeded7186client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 4.93 seconds)7187client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath7188client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.17 seconds)7189worker2: must succeed: pkill -f 'read -t [9]0 x'7190worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)7191client: waiting for success: ! systemctl is-active sw7192worker2 # [ 175.697323] nix-daemon[877]: reaped child process 1862, status = succeeded7193worker2 # [ 175.822363] nix-daemon[877]: accepted connection from pid 2040, user nix-grpc-daemon (trusted)7194worker2 # [ 175.835272] nix-daemon[877]: reaped child process 2043, status = succeeded7195lb # [ 175.856122] niks3-server[1194]: 2026/09/22 11:24:19 INFO Received uploads request method=POST path=/api/pending_closures7196worker2 # [ 175.854085] nix-grpc-daemon[1790]: time=2026-09-22T11:24:19.924Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7197worker2 # [ 175.855687] nix-grpc-daemon[1790]: time=2026-09-22T11:24:19.926Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"7198lb # [ 175.889768] niks3-server[1194]: 2026/09/22 11:24:19 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv7199lb # [ 175.900990] niks3-server[1194]: 2026/09/22 11:24:19 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst7200lb # [ 175.910107] niks3-server[1194]: 2026/09/22 11:24:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign7201worker2 # [ 175.905451] nix-grpc-daemon[1790]: time=2026-09-22T11:24:19.975Z level=INFO msg="Uploading 1 narinfos"7202lb # [ 175.915128] niks3-server[1194]: 2026/09/22 11:24:19 INFO Signed narinfos id=61 count=17203lb # [ 175.923091] niks3-server[1194]: 2026/09/22 11:24:19 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls7204lb # [ 175.932876] niks3-server[1194]: 2026/09/22 11:24:19 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete7205lb # [ 175.938343] niks3-server[1194]: 2026/09/22 11:24:19 INFO Completed upload id=617206worker2 # [ 175.928691] nix-grpc-daemon[1790]: time=2026-09-22T11:24:19.999Z level=INFO msg="Upload complete. (239ms)"7207lb # [ 175.941331] niks3-server[1194]: 2026/09/22 11:24:19 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo7208worker2 # [ 175.937357] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38758 duration_s=10 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=17209worker2 # [ 175.950192] nix-daemon[877]: reaped child process 1860, status = succeeded7210client # [ 175.969568] systemd[1]: sw.service: Deactivated successfully.7211client # [ 175.972880] systemd[1]: sw.service: Consumed 130ms CPU time over 11.852s wall clock time, 41.4M memory peak, 7.5K incoming IP traffic, 8.5K outgoing IP traffic.7212worker2 # [ 176.001479] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:20Z level=info event=signal_exit7213lb # [ 176.250094] niks3-server[1194]: 2026/09/22 11:24:19 INFO lead: released remote=192.168.1.4:484107214client: (finished: waiting for success: ! systemctl is-active sw, in 1.09 seconds)7215client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }7216lb # [ 176.785914] envoy[899]: conn peer=[2001:db8:1::1]:36738 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9525 tx=4353 ms=46657217lb # [ 176.787512] envoy[899]: conn peer=[2001:db8:1::1]:36736 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2992 tx=1560 ms=116217218client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.03 seconds)7219worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service7220worker2 # [ 176.852646] systemd[1]: Starting scheduler-token.service...7221worker2 # [ 176.923596] systemd[1]: scheduler-token.service: Deactivated successfully.7222worker2 # [ 176.925167] systemd[1]: Finished scheduler-token.service.7223worker2 # [ 176.926520] systemd[1]: scheduler-token.service: Consumed 19ms CPU time over 72ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.7224worker1 # [ 177.228895] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:20Z level=info event=scheduler_restarting addr=lb:500517225worker2 # [ 177.243469] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:21Z level=info event=scheduler_restarting7226worker2 # [ 177.248403] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:387347227worker2 # [ 177.252294] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:21Z level=info event=worker_gone addr=192.168.1.3:500517228worker2 # [ 177.255691] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:21Z level=info event=scheduler_restarting addr=lb:500517229worker2 # [ 177.259609] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:387347230worker2 # [ 177.263453] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:21Z level=info event=worker_gone addr=192.168.1.4:500517231worker2 # [ 177.278528] nix-daemon[877]: reaped child process 1761, status = succeeded7232worker1 # [ 177.429431] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:20Z level=info event=scheduler_connected addr=lb:500517233worker1 # [ 177.433173] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7234worker2 # [ 177.455584] nix-grpc-daemon[1707]: ts=2026-09-22T11:24:21Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"7235worker1 # [ 177.633552] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:20Z level=info event=scheduler_connected addr=lb:500517236worker1 # [ 177.635268] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7237worker1 # [ 177.836305] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:21Z level=info event=scheduler_connected addr=lb:500517238worker1 # [ 177.840265] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7239worker1 # [ 178.040591] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:21Z level=info event=scheduler_connected addr=lb:500517240worker1 # [ 178.042456] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7241worker2 # [ 178.171255] nix-grpc-daemon[1707]: E0000 00:00:1790076262.241442 1752 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true7242worker2 # [ 178.176376] nix-grpc-daemon[1707]: E0000 00:00:1790076262.241647 1752 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true7243worker1 # [ 178.244162] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:21Z level=info event=scheduler_connected addr=lb:500517244worker1 # [ 178.247791] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7245worker1 # [ 178.397385] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:21Z level=info event=scheduler_take_over7246lb # [ 178.419357] niks3-server[1194]: 2026/09/22 11:24:21 INFO lead: acquired remote=192.168.1.3:523847247worker1 # [ 178.448578] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:21Z level=info event=scheduler_connected addr=lb:500517248worker1 # [ 178.452494] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7249worker1 # [ 178.653047] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:21Z level=info event=scheduler_connected addr=lb:500517250worker1 # [ 178.654761] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7251worker1 # [ 178.855670] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=scheduler_connected addr=lb:500517252worker1 # [ 178.859647] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7253worker2 # [ 179.019355] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.7254worker2 # [ 179.020530] systemd[1]: nix-grpc-daemon.service: Consumed 637ms CPU time over 26.355s wall clock time, 33.3M memory peak, 23.9K incoming IP traffic, 31.7K outgoing IP traffic.7255worker2 # [ 179.027968] systemd[1]: Starting Nix worker-protocol over gRPC...7256worker1 # [ 179.060451] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=scheduler_connected addr=lb:500517257worker1 # [ 179.064269] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7258worker2 # [ 179.173955] nix-grpc-daemon[2060]: ts=2026-09-22T11:24:23Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"7259worker2 # [ 179.176348] nix-grpc-daemon[2060]: ts=2026-09-22T11:24:23Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc7260worker2 # [ 179.180233] nix-grpc-daemon[2060]: ts=2026-09-22T11:24:23Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=17261worker2 # [ 179.187063] nix-grpc-daemon[2060]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR7262worker2 # [ 179.188679] nix-grpc-daemon[2060]: E0000 00:00:1790076263.256903 2060 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7263worker2 # [ 179.193608] nix-grpc-daemon[2060]: E0000 00:00:1790076263.262088 2060 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7264worker2 # [ 179.197845] nix-grpc-daemon[2060]: ts=2026-09-22T11:24:23Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket7265worker2 # [ 179.200985] systemd[1]: Started Nix worker-protocol over gRPC.7266worker2 # [ 179.207083] nix-grpc-daemon[2060]: E0000 00:00:1790076263.275414 2102 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7267worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.43 seconds)7268lb: waiting until exactly no endpoints of x86_64-linux are unhealthy7269lb: must succeed: curl -sf localhost:9901/clusters7270worker2 # [ 179.219084] nix-grpc-daemon[2060]: E0000 00:00:1790076263.288766 2102 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7271worker2 # [ 179.222107] nix-grpc-daemon[2060]: E0000 00:00:1790076263.288908 2102 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7272worker2 # [ 179.232371] nix-grpc-daemon[2060]: E0000 00:00:1790076263.302815 2102 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7273worker2 # [ 179.239438] nix-grpc-daemon[2060]: ts=2026-09-22T11:24:23Z level=info event=scheduler_connected addr=lb:500517274worker2 # [ 179.241038] nix-grpc-daemon[2060]: ts=2026-09-22T11:24:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7275lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)7276worker1 # [ 179.266326] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=scheduler_connected addr=lb:500517277worker1 # [ 179.269474] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7278worker2 # [ 179.441773] nix-grpc-daemon[2060]: ts=2026-09-22T11:24:23Z level=info event=scheduler_connected addr=lb:500517279worker1 # [ 179.443254] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=oidc_jwks provider=mock keys=17280worker1 # [ 179.446299] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:377607281worker1 # [ 179.450714] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=07282worker1 # [ 179.466965] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=scheduler_connected addr=lb:500517283worker1 # [ 179.470897] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:377607284worker1 # [ 179.474426] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:22Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=07285lb: must succeed: curl -sf localhost:9901/clusters7286lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7287lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.11 seconds)7288lb: must succeed: curl -sf localhost:9901/clusters7289lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7290worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7291worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)7292(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 16.37 seconds)7293subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does7294client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&17295worker2 # [ 180.638328] nix-grpc-daemon[2060]: ts=2026-09-22T11:24:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:336547296worker2 # [ 180.649369] nix-daemon[877]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7297worker1 # [ 180.652062] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207298worker1 # [ 180.684623] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:23Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=07299worker1 # [ 180.686550] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:23Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=17300worker1 # [ 180.688464] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:23Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=07301worker1 # [ 180.690053] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:23Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=07302worker1 # [ 180.728423] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207303worker1 # [ 180.736732] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207304worker1 # [ 180.744337] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207305worker1 # [ 180.752071] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7306worker1 # [ 180.911690] nix-daemon[855]: accepted connection from pid 1883, user nix-grpc-daemon (trusted)7307worker1 # [ 180.924621] nix-daemon[855]: reaped child process 1886, status = succeeded7308worker1 # [ 181.081845] nix-daemon[855]: accepted connection from pid 1888, user nix-grpc-daemon (trusted)7309worker1 # [ 181.094343] nix-daemon[855]: reaped child process 1891, status = succeeded7310lb # [ 181.126439] niks3-server[1194]: 2026/09/22 11:24:24 INFO Received uploads request method=POST path=/api/pending_closures7311worker1 # [ 181.113107] nix-grpc-daemon[1693]: time=2026-09-22T11:24:24.345Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7312worker1 # [ 181.114514] nix-grpc-daemon[1693]: time=2026-09-22T11:24:24.345Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"7313lb # [ 181.158099] niks3-server[1194]: 2026/09/22 11:24:24 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst7314lb # [ 181.166057] niks3-server[1194]: 2026/09/22 11:24:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign7315worker1 # [ 181.150966] nix-grpc-daemon[1693]: time=2026-09-22T11:24:24.383Z level=INFO msg="Uploading 1 narinfos"7316lb # [ 181.172430] niks3-server[1194]: 2026/09/22 11:24:24 INFO Signed narinfos id=62 count=17317lb # [ 181.191749] niks3-server[1194]: 2026/09/22 11:24:24 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete7318lb # [ 181.201565] niks3-server[1194]: 2026/09/22 11:24:24 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls7319lb # [ 181.206289] niks3-server[1194]: 2026/09/22 11:24:24 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo7320lb # [ 181.210842] niks3-server[1194]: 2026/09/22 11:24:24 INFO Completed upload id=627321worker1 # [ 181.189442] nix-grpc-daemon[1693]: time=2026-09-22T11:24:24.421Z level=INFO msg="Upload complete. (428ms)"7322worker1 # [ 181.193858] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=5047323worker1 # [ 181.199694] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207324worker1 # [ 181.212107] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7325worker1 # [ 181.221220] nix-daemon[855]: reaped child process 1881, status = succeeded7326worker1 # [ 181.227867] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7327worker1 # [ 182.455258] nix-daemon[855]: reaped child process 1895, status = succeeded7328worker1 # [ 182.606085] nix-daemon[855]: accepted connection from pid 1911, user nix-grpc-daemon (trusted)7329worker1 # [ 182.616596] nix-daemon[855]: reaped child process 1914, status = succeeded7330lb # [ 182.653123] niks3-server[1194]: 2026/09/22 11:24:26 INFO Received uploads request method=POST path=/api/pending_closures7331worker1 # [ 182.641698] nix-grpc-daemon[1693]: time=2026-09-22T11:24:25.873Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7332worker1 # [ 182.644149] nix-grpc-daemon[1693]: time=2026-09-22T11:24:25.875Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"7333lb # [ 182.690452] niks3-server[1194]: 2026/09/22 11:24:26 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv7334lb # [ 182.702817] niks3-server[1194]: 2026/09/22 11:24:26 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst7335lb # [ 182.713804] niks3-server[1194]: 2026/09/22 11:24:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign7336lb # [ 182.719122] niks3-server[1194]: 2026/09/22 11:24:26 INFO Signed narinfos id=63 count=17337worker1 # [ 182.697404] nix-grpc-daemon[1693]: time=2026-09-22T11:24:25.929Z level=INFO msg="Uploading 1 narinfos"7338lb # [ 182.724441] niks3-server[1194]: 2026/09/22 11:24:26 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls7339lb # [ 182.734059] niks3-server[1194]: 2026/09/22 11:24:26 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete7340lb # [ 182.739227] niks3-server[1194]: 2026/09/22 11:24:26 INFO Completed upload id=637341worker1 # [ 182.717903] nix-grpc-daemon[1693]: time=2026-09-22T11:24:25.950Z level=INFO msg="Upload complete. (280ms)"7342lb # [ 182.741463] niks3-server[1194]: 2026/09/22 11:24:26 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo7343worker1 # [ 182.724633] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=17344worker1 # [ 182.738231] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:25Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=17345worker1 # [ 182.741663] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:25Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=07346worker1 # [ 182.749672] nix-daemon[855]: reaped child process 1893, status = succeeded7347worker1 # [ 182.778397] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207348worker1 # [ 182.788333] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207349worker1 # [ 182.797546] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207350worker1 # [ 182.805531] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7351worker1 # [ 182.957834] nix-daemon[855]: accepted connection from pid 1920, user nix-grpc-daemon (trusted)7352worker1 # [ 182.969466] nix-daemon[855]: reaped child process 1923, status = succeeded7353worker1 # [ 183.104647] nix-daemon[855]: accepted connection from pid 1925, user nix-grpc-daemon (trusted)7354worker1 # [ 183.115935] nix-daemon[855]: reaped child process 1928, status = succeeded7355lb # [ 183.149721] niks3-server[1194]: 2026/09/22 11:24:26 INFO Received uploads request method=POST path=/api/pending_closures7356worker1 # [ 183.136677] nix-grpc-daemon[1693]: time=2026-09-22T11:24:26.369Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7357worker1 # [ 183.138621] nix-grpc-daemon[1693]: time=2026-09-22T11:24:26.370Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"7358lb # [ 183.173234] niks3-server[1194]: 2026/09/22 11:24:26 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst7359lb # [ 183.182227] niks3-server[1194]: 2026/09/22 11:24:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7360worker1 # [ 183.166181] nix-grpc-daemon[1693]: time=2026-09-22T11:24:26.398Z level=INFO msg="Uploading 1 narinfos"7361lb # [ 183.187752] niks3-server[1194]: 2026/09/22 11:24:26 INFO Signed narinfos id=64 count=17362lb # [ 183.191113] niks3-server[1194]: 2026/09/22 11:24:26 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls7363lb # [ 183.201359] niks3-server[1194]: 2026/09/22 11:24:26 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7364worker1 # [ 183.185713] nix-grpc-daemon[1693]: time=2026-09-22T11:24:26.418Z level=INFO msg="Upload complete. (369ms)"7365lb # [ 183.207641] niks3-server[1194]: 2026/09/22 11:24:26 INFO Completed upload id=647366worker1 # [ 183.187923] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=5047367lb # [ 183.210832] niks3-server[1194]: 2026/09/22 11:24:26 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo7368worker1 # [ 183.191703] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207369worker1 # [ 183.197749] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7370worker1 # [ 183.199984] nix-daemon[855]: reaped child process 1918, status = succeeded7371worker1 # [ 183.205041] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7372worker1 # [ 184.420541] nix-daemon[855]: reaped child process 1932, status = succeeded7373worker1 # [ 184.557165] nix-daemon[855]: accepted connection from pid 1948, user nix-grpc-daemon (trusted)7374worker1 # [ 184.569273] nix-daemon[855]: reaped child process 1951, status = succeeded7375lb # [ 184.602825] niks3-server[1194]: 2026/09/22 11:24:28 INFO Received uploads request method=POST path=/api/pending_closures7376worker1 # [ 184.590353] nix-grpc-daemon[1693]: time=2026-09-22T11:24:27.822Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7377worker1 # [ 184.591844] nix-grpc-daemon[1693]: time=2026-09-22T11:24:27.824Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"7378lb # [ 184.639076] niks3-server[1194]: 2026/09/22 11:24:28 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst7379lb # [ 184.649624] niks3-server[1194]: 2026/09/22 11:24:28 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv7380lb # [ 184.655530] niks3-server[1194]: 2026/09/22 11:24:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7381lb # [ 184.659090] niks3-server[1194]: 2026/09/22 11:24:28 INFO Signed narinfos id=65 count=17382worker1 # [ 184.639986] nix-grpc-daemon[1693]: time=2026-09-22T11:24:27.872Z level=INFO msg="Uploading 1 narinfos"7383lb # [ 184.664525] niks3-server[1194]: 2026/09/22 11:24:28 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls7384lb # [ 184.673637] niks3-server[1194]: 2026/09/22 11:24:28 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7385lb # [ 184.682417] niks3-server[1194]: 2026/09/22 11:24:28 INFO Completed upload id=657386worker1 # [ 184.660713] nix-grpc-daemon[1693]: time=2026-09-22T11:24:27.893Z level=INFO msg="Upload complete. (249ms)"7387lb # [ 184.683648] niks3-server[1194]: 2026/09/22 11:24:28 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo7388worker1 # [ 184.668858] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=17389worker1 # [ 184.675755] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:27Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=17390worker1 # [ 184.681212] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207391worker1 # [ 184.684381] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:27Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=07392worker1 # [ 184.691024] nix-daemon[855]: reaped child process 1930, status = succeeded7393worker1 # [ 184.692399] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207394worker1 # [ 184.698291] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207395worker1 # [ 184.702114] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7396worker1 # [ 184.849932] nix-daemon[855]: accepted connection from pid 1957, user nix-grpc-daemon (trusted)7397worker1 # [ 184.862857] nix-daemon[855]: reaped child process 1960, status = succeeded7398worker1 # [ 185.003891] nix-daemon[855]: accepted connection from pid 1962, user nix-grpc-daemon (trusted)7399worker1 # [ 185.014597] nix-daemon[855]: reaped child process 1965, status = succeeded7400lb # [ 185.047636] niks3-server[1194]: 2026/09/22 11:24:28 INFO Received uploads request method=POST path=/api/pending_closures7401worker1 # [ 185.033307] nix-grpc-daemon[1693]: time=2026-09-22T11:24:28.265Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7402worker1 # [ 185.034809] nix-grpc-daemon[1693]: time=2026-09-22T11:24:28.267Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"7403lb # [ 185.075084] niks3-server[1194]: 2026/09/22 11:24:28 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst7404lb # [ 185.086596] niks3-server[1194]: 2026/09/22 11:24:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7405lb # [ 185.093117] niks3-server[1194]: 2026/09/22 11:24:28 INFO Signed narinfos id=66 count=17406worker1 # [ 185.074326] nix-grpc-daemon[1693]: time=2026-09-22T11:24:28.306Z level=INFO msg="Uploading 1 narinfos"7407lb # [ 185.096224] niks3-server[1194]: 2026/09/22 11:24:28 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls7408lb # [ 185.109380] niks3-server[1194]: 2026/09/22 11:24:28 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7409lb # [ 185.115163] niks3-server[1194]: 2026/09/22 11:24:28 INFO Completed upload id=667410worker1 # [ 185.095681] nix-grpc-daemon[1693]: time=2026-09-22T11:24:28.328Z level=INFO msg="Upload complete. (387ms)"7411lb # [ 185.118148] niks3-server[1194]: 2026/09/22 11:24:28 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo7412worker1 # [ 185.099093] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=5047413worker1 # [ 185.104307] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207414worker1 # [ 185.112913] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7415worker1 # [ 185.119724] nix-daemon[855]: reaped child process 1955, status = succeeded7416worker1 # [ 185.123975] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7417worker1 # [ 186.349837] nix-daemon[855]: reaped child process 1969, status = succeeded7418worker1 # [ 186.471163] nix-daemon[855]: accepted connection from pid 1985, user nix-grpc-daemon (trusted)7419worker1 # [ 186.482905] nix-daemon[855]: reaped child process 1988, status = succeeded7420lb # [ 186.514816] niks3-server[1194]: 2026/09/22 11:24:29 INFO Received uploads request method=POST path=/api/pending_closures7421worker1 # [ 186.501392] nix-grpc-daemon[1693]: time=2026-09-22T11:24:29.733Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7422worker1 # [ 186.503039] nix-grpc-daemon[1693]: time=2026-09-22T11:24:29.735Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"7423lb # [ 186.553486] niks3-server[1194]: 2026/09/22 11:24:29 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst7424lb # [ 186.559275] niks3-server[1194]: 2026/09/22 11:24:29 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv7425lb # [ 186.569702] niks3-server[1194]: 2026/09/22 11:24:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7426lb # [ 186.573808] niks3-server[1194]: 2026/09/22 11:24:29 INFO Signed narinfos id=67 count=17427worker1 # [ 186.555516] nix-grpc-daemon[1693]: time=2026-09-22T11:24:29.787Z level=INFO msg="Uploading 1 narinfos"7428lb # [ 186.581344] niks3-server[1194]: 2026/09/22 11:24:29 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls7429lb # [ 186.589472] niks3-server[1194]: 2026/09/22 11:24:30 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7430lb # [ 186.596115] niks3-server[1194]: 2026/09/22 11:24:30 INFO Completed upload id=677431worker1 # [ 186.574594] nix-grpc-daemon[1693]: time=2026-09-22T11:24:29.807Z level=INFO msg="Upload complete. (243ms)"7432lb # [ 186.598929] niks3-server[1194]: 2026/09/22 11:24:30 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo7433worker1 # [ 186.587828] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=17434worker1 # [ 186.592254] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:29Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=17435worker1 # [ 186.596237] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207436worker1 # [ 186.599247] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:29Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=07437worker1 # [ 186.609342] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207438worker1 # [ 186.611979] nix-daemon[855]: reaped child process 1967, status = succeeded7439worker1 # [ 186.617286] nix-grpc-daemon[1693]: time=2026-09-22T11:24:29.849Z level=INFO msg="All 1 paths already cached"7440worker1 # [ 186.621677] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207441worker1 # [ 186.625347] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7442worker1 # [ 186.751791] nix-daemon[855]: accepted connection from pid 1994, user nix-grpc-daemon (trusted)7443worker1 # [ 186.763823] nix-daemon[855]: reaped child process 1997, status = succeeded7444lb # [ 186.785789] envoy[899]: conn peer=[2001:db8:1::4]:60460 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=1584 tx=344 ms=244017445worker1 # [ 186.910035] nix-daemon[855]: accepted connection from pid 1999, user nix-grpc-daemon (trusted)7446worker1 # [ 186.922085] nix-daemon[855]: reaped child process 2002, status = succeeded7447lb # [ 186.957277] niks3-server[1194]: 2026/09/22 11:24:30 INFO Received uploads request method=POST path=/api/pending_closures7448worker1 # [ 186.942975] nix-grpc-daemon[1693]: time=2026-09-22T11:24:30.175Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7449worker1 # [ 186.944512] nix-grpc-daemon[1693]: time=2026-09-22T11:24:30.177Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"7450lb # [ 186.983807] niks3-server[1194]: 2026/09/22 11:24:30 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst7451lb # [ 186.995911] niks3-server[1194]: 2026/09/22 11:24:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7452worker1 # [ 186.979507] nix-grpc-daemon[1693]: time=2026-09-22T11:24:30.212Z level=INFO msg="Uploading 1 narinfos"7453lb # [ 187.001609] niks3-server[1194]: 2026/09/22 11:24:30 INFO Signed narinfos id=68 count=17454lb # [ 187.006508] niks3-server[1194]: 2026/09/22 11:24:30 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls7455lb # [ 187.019356] niks3-server[1194]: 2026/09/22 11:24:30 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7456lb # [ 187.024506] niks3-server[1194]: 2026/09/22 11:24:30 INFO Completed upload id=687457worker1 # [ 187.004989] nix-grpc-daemon[1693]: time=2026-09-22T11:24:30.237Z level=INFO msg="Upload complete. (373ms)"7458worker1 # [ 187.007197] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=6247459lb # [ 187.028130] niks3-server[1194]: 2026/09/22 11:24:30 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo7460worker1 # [ 187.012389] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207461worker1 # [ 187.016976] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7462worker1 # [ 187.019187] nix-daemon[855]: reaped child process 1992, status = succeeded7463worker1 # [ 187.024384] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7464worker1 # [ 188.249403] nix-daemon[855]: reaped child process 2006, status = succeeded7465worker1 # [ 188.401215] nix-daemon[855]: accepted connection from pid 2022, user nix-grpc-daemon (trusted)7466worker1 # [ 188.413372] nix-daemon[855]: reaped child process 2025, status = succeeded7467lb # [ 188.450085] niks3-server[1194]: 2026/09/22 11:24:31 INFO Received uploads request method=POST path=/api/pending_closures7468worker1 # [ 188.432645] nix-grpc-daemon[1693]: time=2026-09-22T11:24:31.664Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7469worker1 # [ 188.436883] nix-grpc-daemon[1693]: time=2026-09-22T11:24:31.666Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"7470lb # [ 188.472374] niks3-server[1194]: 2026/09/22 11:24:31 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv7471lb # [ 188.476056] niks3-server[1194]: 2026/09/22 11:24:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7472lb # [ 188.478234] niks3-server[1194]: 2026/09/22 11:24:31 INFO Signed narinfos id=69 count=17473worker1 # [ 188.456767] nix-grpc-daemon[1693]: time=2026-09-22T11:24:31.689Z level=INFO msg="Uploading 1 narinfos"7474lb # [ 188.481634] niks3-server[1194]: 2026/09/22 11:24:31 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst7475lb # [ 188.484972] niks3-server[1194]: 2026/09/22 11:24:31 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls7476lb # [ 188.488944] niks3-server[1194]: 2026/09/22 11:24:31 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7477lb # [ 188.491921] niks3-server[1194]: 2026/09/22 11:24:31 INFO Completed upload id=697478worker1 # [ 188.471533] nix-grpc-daemon[1693]: time=2026-09-22T11:24:31.704Z level=INFO msg="Upload complete. (240ms)"7479lb # [ 188.493838] niks3-server[1194]: 2026/09/22 11:24:31 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo7480worker1 # [ 188.474191] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=17481worker1 # [ 188.478940] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:31Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=17482worker1 # [ 188.481716] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:31Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=07483worker1 # [ 188.486466] nix-daemon[855]: reaped child process 2004, status = succeeded7484worker1 # [ 188.519384] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207485worker1 # [ 188.524493] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207486worker1 # [ 188.530282] nix-grpc-daemon[1693]: time=2026-09-22T11:24:31.762Z level=INFO msg="All 1 paths already cached"7487worker1 # [ 188.533488] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207488worker1 # [ 188.537156] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7489worker1 # [ 188.665215] nix-daemon[855]: accepted connection from pid 2031, user nix-grpc-daemon (trusted)7490worker1 # [ 188.676842] nix-daemon[855]: reaped child process 2034, status = succeeded7491worker1 # [ 188.808767] nix-daemon[855]: accepted connection from pid 2036, user nix-grpc-daemon (trusted)7492worker1 # [ 188.820655] nix-daemon[855]: reaped child process 2039, status = succeeded7493lb # [ 188.851804] niks3-server[1194]: 2026/09/22 11:24:32 INFO Received uploads request method=POST path=/api/pending_closures7494worker1 # [ 188.838545] nix-grpc-daemon[1693]: time=2026-09-22T11:24:32.070Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7495worker1 # [ 188.839885] nix-grpc-daemon[1693]: time=2026-09-22T11:24:32.070Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"7496lb # [ 188.891615] niks3-server[1194]: 2026/09/22 11:24:32 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst7497lb # [ 188.900598] niks3-server[1194]: 2026/09/22 11:24:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7498worker1 # [ 188.883247] nix-grpc-daemon[1693]: time=2026-09-22T11:24:32.115Z level=INFO msg="Uploading 1 narinfos"7499lb # [ 188.905831] niks3-server[1194]: 2026/09/22 11:24:32 INFO Signed narinfos id=70 count=17500lb # [ 188.912929] niks3-server[1194]: 2026/09/22 11:24:32 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls7501lb # [ 188.917240] niks3-server[1194]: 2026/09/22 11:24:32 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7502lb # [ 188.921232] niks3-server[1194]: 2026/09/22 11:24:32 INFO Completed upload id=707503worker1 # [ 188.899600] nix-grpc-daemon[1693]: time=2026-09-22T11:24:32.132Z level=INFO msg="Upload complete. (356ms)"7504lb # [ 188.923392] niks3-server[1194]: 2026/09/22 11:24:32 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo7505worker1 # [ 188.901835] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=6247506worker1 # [ 188.905352] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207507worker1 # [ 188.914056] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7508worker1 # [ 188.916258] nix-daemon[855]: reaped child process 2029, status = succeeded7509worker1 # [ 188.921537] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7510worker1 # [ 189.734823] nix-daemon[855]: reaped child process 2043, status = succeeded7511worker1 # [ 189.862287] nix-daemon[855]: accepted connection from pid 2059, user nix-grpc-daemon (trusted)7512worker1 # [ 189.874377] nix-daemon[855]: reaped child process 2062, status = succeeded7513lb # [ 189.907475] niks3-server[1194]: 2026/09/22 11:24:33 INFO Received uploads request method=POST path=/api/pending_closures7514worker1 # [ 189.894456] nix-grpc-daemon[1693]: time=2026-09-22T11:24:33.126Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7515worker1 # [ 189.896975] nix-grpc-daemon[1693]: time=2026-09-22T11:24:33.128Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"7516lb # [ 189.935563] niks3-server[1194]: 2026/09/22 11:24:33 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv7517lb # [ 189.960489] niks3-server[1194]: 2026/09/22 11:24:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7518worker1 # [ 189.942375] nix-grpc-daemon[1693]: time=2026-09-22T11:24:33.174Z level=INFO msg="Uploading 1 narinfos"7519lb # [ 189.965128] niks3-server[1194]: 2026/09/22 11:24:33 INFO Signed narinfos id=71 count=17520lb # [ 189.969908] niks3-server[1194]: 2026/09/22 11:24:33 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst7521lb # [ 189.975211] niks3-server[1194]: 2026/09/22 11:24:33 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls7522lb # [ 189.983365] niks3-server[1194]: 2026/09/22 11:24:33 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7523lb # [ 189.986049] niks3-server[1194]: 2026/09/22 11:24:33 INFO Completed upload id=717524worker1 # [ 189.964257] nix-grpc-daemon[1693]: time=2026-09-22T11:24:33.196Z level=INFO msg="Upload complete. (236ms)"7525lb # [ 189.988385] niks3-server[1194]: 2026/09/22 11:24:33 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo7526worker1 # [ 189.970753] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=17527worker1 # [ 189.980412] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:33Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=17528worker1 # [ 189.985311] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207529worker1 # [ 189.988658] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:33Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=07530worker1 # [ 189.998084] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207531worker1 # [ 190.000458] nix-daemon[855]: reaped child process 2041, status = succeeded7532worker1 # [ 190.004189] nix-grpc-daemon[1693]: time=2026-09-22T11:24:33.236Z level=INFO msg="All 1 paths already cached"7533worker1 # [ 190.008125] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207534worker1 # [ 190.013154] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7535worker1 # [ 190.155995] nix-daemon[855]: accepted connection from pid 2068, user nix-grpc-daemon (trusted)7536worker1 # [ 190.167895] nix-daemon[855]: reaped child process 2071, status = succeeded7537worker1 # [ 190.294187] nix-daemon[855]: accepted connection from pid 2073, user nix-grpc-daemon (trusted)7538worker1 # [ 190.306863] nix-daemon[855]: reaped child process 2076, status = succeeded7539lb # [ 190.337833] niks3-server[1194]: 2026/09/22 11:24:33 INFO Received uploads request method=POST path=/api/pending_closures7540worker1 # [ 190.320678] nix-grpc-daemon[1693]: time=2026-09-22T11:24:33.553Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7541worker1 # [ 190.322438] nix-grpc-daemon[1693]: time=2026-09-22T11:24:33.554Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"7542lb # [ 190.366441] niks3-server[1194]: 2026/09/22 11:24:33 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst7543lb # [ 190.373966] niks3-server[1194]: 2026/09/22 11:24:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7544worker1 # [ 190.356956] nix-grpc-daemon[1693]: time=2026-09-22T11:24:33.589Z level=INFO msg="Uploading 1 narinfos"7545lb # [ 190.379095] niks3-server[1194]: 2026/09/22 11:24:33 INFO Signed narinfos id=72 count=17546lb # [ 190.385489] niks3-server[1194]: 2026/09/22 11:24:33 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls7547lb # [ 190.396132] niks3-server[1194]: 2026/09/22 11:24:33 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7548lb # [ 190.402437] niks3-server[1194]: 2026/09/22 11:24:33 INFO Completed upload id=727549worker1 # [ 190.381275] nix-grpc-daemon[1693]: time=2026-09-22T11:24:33.613Z level=INFO msg="Upload complete. (362ms)"7550worker1 # [ 190.383578] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=6247551lb # [ 190.404769] niks3-server[1194]: 2026/09/22 11:24:33 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo7552worker1 # [ 190.390043] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207553worker1 # [ 190.391931] nix-daemon[855]: reaped child process 2066, status = succeeded7554worker1 # [ 190.394986] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7555worker1 # [ 190.402037] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7556worker1 # [ 191.815701] nix-daemon[855]: reaped child process 2080, status = succeeded7557worker1 # [ 191.930285] nix-daemon[855]: accepted connection from pid 2096, user nix-grpc-daemon (trusted)7558worker1 # [ 191.942188] nix-daemon[855]: reaped child process 2099, status = succeeded7559lb # [ 191.976125] niks3-server[1194]: 2026/09/22 11:24:35 INFO Received uploads request method=POST path=/api/pending_closures7560worker1 # [ 191.961388] nix-grpc-daemon[1693]: time=2026-09-22T11:24:35.193Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7561worker1 # [ 191.963082] nix-grpc-daemon[1693]: time=2026-09-22T11:24:35.195Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"7562lb # [ 192.002604] niks3-server[1194]: 2026/09/22 11:24:35 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv7563lb # [ 192.018277] niks3-server[1194]: 2026/09/22 11:24:35 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst7564lb # [ 192.027709] niks3-server[1194]: 2026/09/22 11:24:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7565lb # [ 192.032485] niks3-server[1194]: 2026/09/22 11:24:35 INFO Signed narinfos id=73 count=17566worker1 # [ 192.013246] nix-grpc-daemon[1693]: time=2026-09-22T11:24:35.245Z level=INFO msg="Uploading 1 narinfos"7567lb # [ 192.042891] niks3-server[1194]: 2026/09/22 11:24:35 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls7568lb # [ 192.050373] niks3-server[1194]: 2026/09/22 11:24:35 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7569lb # [ 192.053432] niks3-server[1194]: 2026/09/22 11:24:35 INFO Completed upload id=737570lb # [ 192.054864] niks3-server[1194]: 2026/09/22 11:24:35 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo7571worker1 # [ 192.031919] nix-grpc-daemon[1693]: time=2026-09-22T11:24:35.264Z level=INFO msg="Upload complete. (221ms)"7572worker1 # [ 192.041850] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=17573worker1 # [ 192.048160] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:35Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=17574worker1 # [ 192.052535] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207575worker1 # [ 192.057187] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:35Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=07576worker1 # [ 192.064154] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207577worker1 # [ 192.066572] nix-daemon[855]: reaped child process 2078, status = succeeded7578worker1 # [ 192.070215] nix-grpc-daemon[1693]: time=2026-09-22T11:24:35.302Z level=INFO msg="All 2 paths already cached"7579worker1 # [ 192.074217] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207580worker1 # [ 192.079070] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7581worker1 # [ 192.209382] nix-daemon[855]: accepted connection from pid 2105, user nix-grpc-daemon (trusted)7582worker1 # [ 192.221272] nix-daemon[855]: reaped child process 2108, status = succeeded7583worker1 # [ 192.372032] nix-daemon[855]: accepted connection from pid 2110, user nix-grpc-daemon (trusted)7584worker1 # [ 192.383675] nix-daemon[855]: reaped child process 2113, status = succeeded7585lb # [ 192.417813] niks3-server[1194]: 2026/09/22 11:24:35 INFO Received uploads request method=POST path=/api/pending_closures7586worker1 # [ 192.403612] nix-grpc-daemon[1693]: time=2026-09-22T11:24:35.635Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7587worker1 # [ 192.404980] nix-grpc-daemon[1693]: time=2026-09-22T11:24:35.635Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7588lb # [ 192.443500] niks3-server[1194]: 2026/09/22 11:24:35 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7589lb # [ 192.451722] niks3-server[1194]: 2026/09/22 11:24:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7590lb # [ 192.456625] niks3-server[1194]: 2026/09/22 11:24:35 INFO Signed narinfos id=74 count=17591worker1 # [ 192.437936] nix-grpc-daemon[1693]: time=2026-09-22T11:24:35.670Z level=INFO msg="Uploading 1 narinfos"7592lb # [ 192.468951] niks3-server[1194]: 2026/09/22 11:24:35 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7593worker1 # [ 192.454086] nix-grpc-daemon[1693]: time=2026-09-22T11:24:35.686Z level=INFO msg="Upload complete. (370ms)"7594lb # [ 192.475719] niks3-server[1194]: 2026/09/22 11:24:35 INFO Completed upload id=747595worker1 # [ 192.456067] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=7527596worker1 # [ 192.459539] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207597lb # [ 192.482186] niks3-server[1194]: 2026/09/22 11:24:35 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7598lb # [ 192.486154] niks3-server[1194]: 2026/09/22 11:24:35 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7599worker1 # [ 192.466431] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7600worker1 # [ 192.468340] nix-daemon[855]: reaped child process 2103, status = succeeded7601worker1 # [ 192.473193] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7602worker1 # [ 193.687490] nix-daemon[855]: reaped child process 2117, status = succeeded7603worker1 # [ 193.793504] nix-daemon[855]: accepted connection from pid 2133, user nix-grpc-daemon (trusted)7604worker1 # [ 193.805113] nix-daemon[855]: reaped child process 2136, status = succeeded7605lb # [ 193.840553] niks3-server[1194]: 2026/09/22 11:24:37 INFO Received uploads request method=POST path=/api/pending_closures7606worker1 # [ 193.827114] nix-grpc-daemon[1693]: time=2026-09-22T11:24:37.059Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7607worker1 # [ 193.828567] nix-grpc-daemon[1693]: time=2026-09-22T11:24:37.061Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7608lb # [ 193.871154] niks3-server[1194]: 2026/09/22 11:24:37 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7609lb # [ 193.883950] niks3-server[1194]: 2026/09/22 11:24:37 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7610lb # [ 193.897745] niks3-server[1194]: 2026/09/22 11:24:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7611worker1 # [ 193.880851] nix-grpc-daemon[1693]: time=2026-09-22T11:24:37.113Z level=INFO msg="Uploading 1 narinfos"7612lb # [ 193.902450] niks3-server[1194]: 2026/09/22 11:24:37 INFO Signed narinfos id=75 count=17613lb # [ 193.908549] niks3-server[1194]: 2026/09/22 11:24:37 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7614lb # [ 193.917285] niks3-server[1194]: 2026/09/22 11:24:37 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7615lb # [ 193.921897] niks3-server[1194]: 2026/09/22 11:24:37 INFO Completed upload id=757616worker1 # [ 193.900229] nix-grpc-daemon[1693]: time=2026-09-22T11:24:37.132Z level=INFO msg="Upload complete. (220ms)"7617lb # [ 193.924388] niks3-server[1194]: 2026/09/22 11:24:37 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7618worker1 # [ 193.908465] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=17619worker1 # [ 193.915259] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:37Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17620worker1 # [ 193.924267] nix-daemon[855]: reaped child process 2115, status = succeeded7621worker1 # [ 193.950844] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207622worker1 # [ 193.962939] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207623worker1 # [ 193.976375] nix-grpc-daemon[1693]: time=2026-09-22T11:24:37.208Z level=INFO msg="All 2 paths already cached"7624worker1 # [ 193.983469] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207625worker1 # [ 193.994128] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7626worker1 # [ 194.112620] nix-daemon[855]: accepted connection from pid 2140, user nix-grpc-daemon (trusted)7627worker1 # [ 194.124506] nix-daemon[855]: reaped child process 2143, status = succeeded7628worker1 # [ 194.261204] nix-daemon[855]: accepted connection from pid 2145, user nix-grpc-daemon (trusted)7629worker1 # [ 194.271498] nix-daemon[855]: reaped child process 2148, status = succeeded7630lb # [ 194.303751] niks3-server[1194]: 2026/09/22 11:24:37 INFO Received uploads request method=POST path=/api/pending_closures7631worker1 # [ 194.289792] nix-grpc-daemon[1693]: time=2026-09-22T11:24:37.522Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7632worker1 # [ 194.291568] nix-grpc-daemon[1693]: time=2026-09-22T11:24:37.524Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7633lb # [ 194.333845] niks3-server[1194]: 2026/09/22 11:24:37 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7634lb # [ 194.344585] niks3-server[1194]: 2026/09/22 11:24:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign7635lb # [ 194.349779] niks3-server[1194]: 2026/09/22 11:24:37 INFO Signed narinfos id=76 count=17636worker1 # [ 194.331084] nix-grpc-daemon[1693]: time=2026-09-22T11:24:37.563Z level=INFO msg="Uploading 1 narinfos"7637lb # [ 194.356290] niks3-server[1194]: 2026/09/22 11:24:37 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7638lb # [ 194.368483] niks3-server[1194]: 2026/09/22 11:24:37 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete7639lb # [ 194.372608] niks3-server[1194]: 2026/09/22 11:24:37 INFO Completed upload id=767640worker1 # [ 194.353103] nix-grpc-daemon[1693]: time=2026-09-22T11:24:37.585Z level=INFO msg="Upload complete. (349ms)"7641lb # [ 194.376219] niks3-server[1194]: 2026/09/22 11:24:37 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7642worker1 # [ 194.355706] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=7527643worker1 # [ 194.361315] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207644worker1 # [ 194.372392] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7645worker1 # [ 194.374470] nix-daemon[855]: reaped child process 2138, status = succeeded7646worker1 # [ 194.379926] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7647worker1 # [ 195.207792] nix-daemon[855]: reaped child process 2152, status = succeeded7648worker1 # [ 195.320911] nix-daemon[855]: accepted connection from pid 2168, user nix-grpc-daemon (trusted)7649worker1 # [ 195.332609] nix-daemon[855]: reaped child process 2171, status = succeeded7650lb # [ 195.365415] niks3-server[1194]: 2026/09/22 11:24:38 INFO Received uploads request method=POST path=/api/pending_closures7651worker1 # [ 195.352039] nix-grpc-daemon[1693]: time=2026-09-22T11:24:38.584Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7652worker1 # [ 195.354848] nix-grpc-daemon[1693]: time=2026-09-22T11:24:38.585Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7653lb # [ 195.402919] niks3-server[1194]: 2026/09/22 11:24:38 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7654lb # [ 195.408284] niks3-server[1194]: 2026/09/22 11:24:38 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7655lb # [ 195.416777] niks3-server[1194]: 2026/09/22 11:24:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign7656lb # [ 195.421419] niks3-server[1194]: 2026/09/22 11:24:38 INFO Signed narinfos id=77 count=17657worker1 # [ 195.402838] nix-grpc-daemon[1693]: time=2026-09-22T11:24:38.635Z level=INFO msg="Uploading 1 narinfos"7658lb # [ 195.431343] niks3-server[1194]: 2026/09/22 11:24:38 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7659lb # [ 195.439122] niks3-server[1194]: 2026/09/22 11:24:38 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete7660lb # [ 195.444344] niks3-server[1194]: 2026/09/22 11:24:38 INFO Completed upload id=777661lb # [ 195.447282] niks3-server[1194]: 2026/09/22 11:24:38 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7662worker1 # [ 195.424557] nix-grpc-daemon[1693]: time=2026-09-22T11:24:38.657Z level=INFO msg="Upload complete. (237ms)"7663worker1 # [ 195.432761] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=17664worker1 # [ 195.439535] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:38Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17665worker1 # [ 195.448702] nix-daemon[855]: reaped child process 2150, status = succeeded7666worker1 # [ 195.475210] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207667worker1 # [ 195.486280] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207668worker1 # [ 195.498373] nix-grpc-daemon[1693]: time=2026-09-22T11:24:38.730Z level=INFO msg="All 2 paths already cached"7669worker1 # [ 195.505832] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:377207670worker1 # [ 195.513552] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7671worker1 # [ 195.651575] nix-daemon[855]: accepted connection from pid 2175, user nix-grpc-daemon (trusted)7672worker1 # [ 195.663618] nix-daemon[855]: reaped child process 2178, status = succeeded7673worker1 # [ 195.807575] nix-daemon[855]: accepted connection from pid 2180, user nix-grpc-daemon (trusted)7674worker1 # [ 195.819124] nix-daemon[855]: reaped child process 2183, status = succeeded7675lb # [ 195.850725] niks3-server[1194]: 2026/09/22 11:24:39 INFO Received uploads request method=POST path=/api/pending_closures7676worker1 # [ 195.836659] nix-grpc-daemon[1693]: time=2026-09-22T11:24:39.069Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7677worker1 # [ 195.838195] nix-grpc-daemon[1693]: time=2026-09-22T11:24:39.070Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7678lb # [ 195.877834] niks3-server[1194]: 2026/09/22 11:24:39 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7679lb # [ 195.888617] niks3-server[1194]: 2026/09/22 11:24:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign7680lb # [ 195.892524] niks3-server[1194]: 2026/09/22 11:24:39 INFO Signed narinfos id=78 count=17681worker1 # [ 195.874720] nix-grpc-daemon[1693]: time=2026-09-22T11:24:39.107Z level=INFO msg="Uploading 1 narinfos"7682lb # [ 195.900244] niks3-server[1194]: 2026/09/22 11:24:39 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7683lb # [ 195.910499] niks3-server[1194]: 2026/09/22 11:24:39 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete7684worker1 # [ 195.893327] nix-grpc-daemon[1693]: time=2026-09-22T11:24:39.125Z level=INFO msg="Upload complete. (368ms)"7685lb # [ 195.915270] niks3-server[1194]: 2026/09/22 11:24:39 INFO Completed upload id=787686worker1 # [ 195.895354] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=0 paths=1 nar_bytes_in=7527687lb # [ 195.918382] niks3-server[1194]: 2026/09/22 11:24:39 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7688worker1 # [ 195.900865] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:377207689worker1 # [ 195.905963] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7690worker1 # [ 195.908115] nix-daemon[855]: reaped child process 2173, status = succeeded7691worker1 # [ 195.913290] nix-daemon[855]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7692worker1 # [ 196.925652] nix-daemon[855]: reaped child process 2187, status = succeeded7693worker1 # [ 197.044272] nix-daemon[855]: accepted connection from pid 2203, user nix-grpc-daemon (trusted)7694worker1 # [ 197.054776] nix-daemon[855]: reaped child process 2206, status = succeeded7695lb # [ 197.089631] niks3-server[1194]: 2026/09/22 11:24:40 INFO Received uploads request method=POST path=/api/pending_closures7696worker1 # [ 197.075628] nix-grpc-daemon[1693]: time=2026-09-22T11:24:40.307Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7697worker1 # [ 197.077389] nix-grpc-daemon[1693]: time=2026-09-22T11:24:40.309Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7698lb # [ 197.126432] niks3-server[1194]: 2026/09/22 11:24:40 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7699lb # [ 197.131581] niks3-server[1194]: 2026/09/22 11:24:40 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7700lb # [ 197.145587] niks3-server[1194]: 2026/09/22 11:24:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign7701lb # [ 197.150943] niks3-server[1194]: 2026/09/22 11:24:40 INFO Signed narinfos id=79 count=17702worker1 # [ 197.131844] nix-grpc-daemon[1693]: time=2026-09-22T11:24:40.364Z level=INFO msg="Uploading 1 narinfos"7703lb # [ 197.155595] niks3-server[1194]: 2026/09/22 11:24:40 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7704lb # [ 197.166921] niks3-server[1194]: 2026/09/22 11:24:40 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete7705lb # [ 197.170273] niks3-server[1194]: 2026/09/22 11:24:40 INFO Completed upload id=797706worker1 # [ 197.148662] nix-grpc-daemon[1693]: time=2026-09-22T11:24:40.381Z level=INFO msg="Upload complete. (229ms)"7707lb # [ 197.171852] niks3-server[1194]: 2026/09/22 11:24:40 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7708worker1 # [ 197.154640] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37720 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=17709worker1 # [ 197.165955] nix-daemon[855]: reaped child process 2185, status = succeeded7710client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 16.77 seconds)7711worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7712worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7713client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'7714client # [ 197.299985] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7715client # Running as unit: gpu.service; invocation ID: ceb3cbd975b4437aa4b5ad688a8f5f327716client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.06 seconds)7717lb: must succeed: curl -sf localhost:9901/clusters7718lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7719lb: must succeed: curl -sf localhost:9901/clusters7720lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)7721worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7722worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)7723worker1 # [ 197.482615] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:377207724client # [ 197.497141] nix[1530]: don't know how to build these paths:7725client # [ 197.498208] nix[1530]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7726worker1 # [ 197.489164] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207727client # [ 197.516088] nix[1530]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker7728worker1 # [ 197.506503] nix-grpc-daemon[1597]: ts=2026-09-22T11:24:40Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07729lb: must succeed: curl -sf localhost:9901/clusters7730lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)7731lb: must succeed: curl -sf localhost:9901/clusters7732lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)7733worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7734worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)7735client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7736lb # [ 206.788195] envoy[899]: conn peer=[2001:db8:1::1]:56110 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27098 tx=12420 ms=165517737client # [ 227.517306] nix[1530]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}7738client # [ 227.525396] nix[1530]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7739client # [ 227.557570] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7740client # [ 227.562300] systemd[1]: gpu.service: Failed with result 'exit-code'.7741client # [ 227.565773] systemd[1]: gpu.service: Consumed 128ms CPU time over 30.262s wall clock time, 35.4M memory peak, 5.1K incoming IP traffic, 5.1K outgoing IP traffic.7742client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.45 seconds)7743client: must succeed: journalctl -u gpu -o cat7744client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)7745lb: must succeed: curl -sf localhost:9901/clusters7746lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7747lb: must succeed: curl -sf localhost:9901/clusters7748lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7749worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7750worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7751(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 47.86 seconds)7752subtest: low disk drains a worker and builds go to the other7753worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7754worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.91 seconds)7755worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7756worker1 # [ 229.413997] nix-grpc-daemon[1597]: ts=2026-09-22T11:25:12Z level=info event=unhealthy reason=min_free7757worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.12 seconds)7758client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&27759client # don't know how to build these paths:7760worker1 # [ 230.518377] nix-grpc-daemon[1597]: ts=2026-09-22T11:25:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:377207761client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7762worker1 # [ 230.521353] nix-grpc-daemon[1597]: ts=2026-09-22T11:25:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:377207763worker1 # [ 230.537258] nix-grpc-daemon[1597]: ts=2026-09-22T11:25:13Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07764worker1 # [ 230.539047] nix-grpc-daemon[1597]: ts=2026-09-22T11:25:13Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17765worker2 # [ 230.594388] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:336547766worker2 # [ 230.600797] nix-grpc-daemon[2060]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 58 ms (attempt 1/5)7767worker2 # [ 230.661576] nix-grpc-daemon[2060]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 195 ms (attempt 2/5)7768worker2 # [ 230.862578] nix-grpc-daemon[2060]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 12 ms (attempt 3/5)7769worker2 # [ 230.880411] nix-grpc-daemon[2060]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 505 ms (attempt 4/5)7770worker2 # [ 231.391328] nix-grpc-daemon[2060]: disabling binary cache 'https://cache.nixos.org' for 60 seconds7771worker2 # [ 231.393149] nix-grpc-daemon[2060]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org7772worker2 # [ 231.405300] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:336547773client # copying 1 paths to 'lb:50051'...7774worker2 # [ 231.414477] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:336547775client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7776worker2 # [ 231.421599] nix-daemon[877]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7777worker2 # [ 231.432916] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:15Z level=info event=niks3_push_started pid=21207778worker2 # [ 231.616675] nix-grpc-daemon[2120]: time=2026-09-22T11:25:15.686Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""7779worker2 # [ 231.776395] nix-daemon[877]: accepted connection from pid 2133, user nix-grpc-daemon (trusted)7780worker2 # [ 231.788903] nix-daemon[877]: reaped child process 2136, status = succeeded7781worker2 # [ 231.954202] nix-daemon[877]: accepted connection from pid 2138, user nix-grpc-daemon (trusted)7782worker2 # [ 231.966316] nix-daemon[877]: reaped child process 2141, status = succeeded7783lb # [ 231.993126] niks3-server[1194]: 2026/09/22 11:25:15 INFO Received uploads request method=POST path=/api/pending_closures7784worker2 # [ 231.992399] nix-grpc-daemon[2120]: time=2026-09-22T11:25:16.062Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7785worker2 # [ 231.994373] nix-grpc-daemon[2120]: time=2026-09-22T11:25:16.064Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7786lb # [ 232.031683] niks3-server[1194]: 2026/09/22 11:25:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign7787worker2 # [ 232.034475] nix-grpc-daemon[2120]: time=2026-09-22T11:25:16.104Z level=INFO msg="Uploading 1 narinfos"7788lb # [ 232.044344] niks3-server[1194]: 2026/09/22 11:25:15 INFO Signed narinfos id=80 count=17789lb # [ 232.047295] niks3-server[1194]: 2026/09/22 11:25:15 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7790lb # [ 232.056482] niks3-server[1194]: 2026/09/22 11:25:15 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7791lb # [ 232.068856] niks3-server[1194]: 2026/09/22 11:25:15 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete7792lb # [ 232.074334] niks3-server[1194]: 2026/09/22 11:25:15 INFO Completed upload id=807793worker2 # [ 232.066553] nix-grpc-daemon[2120]: time=2026-09-22T11:25:16.136Z level=INFO msg="Upload complete. (445ms)"7794lb # [ 232.077899] niks3-server[1194]: 2026/09/22 11:25:15 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7795worker2 # [ 232.071565] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33654 duration_s=0 paths=1 nar_bytes_in=5127796worker2 # [ 232.077663] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:336547797worker2 # [ 232.088114] nix-daemon[877]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7798worker2 # [ 232.092296] nix-daemon[877]: reaped child process 2118, status = succeeded7799worker2 # [ 232.098142] nix-daemon[877]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7800client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7801client # farm-a> worker2-next: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7802client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7803client # farm-a> LOG-farm-a-drain7804worker2 # [ 232.912580] nix-daemon[877]: reaped child process 2145, status = succeeded7805worker2 # [ 233.057381] nix-daemon[877]: accepted connection from pid 2161, user nix-grpc-daemon (trusted)7806worker2 # [ 233.068546] nix-daemon[877]: reaped child process 2164, status = succeeded7807lb # [ 233.089884] niks3-server[1194]: 2026/09/22 11:25:16 INFO Received uploads request method=POST path=/api/pending_closures7808worker2 # [ 233.086706] nix-grpc-daemon[2120]: time=2026-09-22T11:25:17.156Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7809worker2 # [ 233.088304] nix-grpc-daemon[2120]: time=2026-09-22T11:25:17.158Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7810lb # [ 233.112494] niks3-server[1194]: 2026/09/22 11:25:16 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7811lb # [ 233.117439] niks3-server[1194]: 2026/09/22 11:25:16 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7812lb # [ 233.121050] niks3-server[1194]: 2026/09/22 11:25:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign7813lb # [ 233.122960] niks3-server[1194]: 2026/09/22 11:25:16 INFO Signed narinfos id=81 count=17814worker2 # [ 233.115073] nix-grpc-daemon[2120]: time=2026-09-22T11:25:17.184Z level=INFO msg="Uploading 1 narinfos"7815lb # [ 233.126257] niks3-server[1194]: 2026/09/22 11:25:16 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7816lb # [ 233.131154] niks3-server[1194]: 2026/09/22 11:25:16 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete7817lb # [ 233.135044] niks3-server[1194]: 2026/09/22 11:25:16 INFO Completed upload id=817818worker2 # [ 233.124887] nix-grpc-daemon[2120]: time=2026-09-22T11:25:17.195Z level=INFO msg="Upload complete. (220ms)"7819lb # [ 233.136838] niks3-server[1194]: 2026/09/22 11:25:16 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7820worker2 # [ 233.127548] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33654 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=17821worker2 # [ 233.134303] nix-daemon[877]: reaped child process 2143, status = succeeded7822worker1 # [ 233.125811] nix-grpc-daemon[1597]: ts=2026-09-22T11:25:16Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07823worker1 # [ 233.129781] nix-grpc-daemon[1597]: ts=2026-09-22T11:25:16Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17824worker2 # [ 233.180101] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:336547825worker2 # [ 233.187533] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:336547826worker2 # [ 233.194924] nix-grpc-daemon[2120]: time=2026-09-22T11:25:17.265Z level=INFO msg="All 1 paths already cached"7827client # copying 1 paths to 'lb:50051'...7828worker2 # [ 233.200534] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:336547829client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7830worker2 # [ 233.206717] nix-daemon[877]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7831worker2 # [ 233.337534] nix-daemon[877]: accepted connection from pid 2168, user nix-grpc-daemon (trusted)7832worker2 # [ 233.350548] nix-daemon[877]: reaped child process 2171, status = succeeded7833worker2 # [ 233.489665] nix-daemon[877]: accepted connection from pid 2173, user nix-grpc-daemon (trusted)7834worker2 # [ 233.502559] nix-daemon[877]: reaped child process 2176, status = succeeded7835lb # [ 233.523213] niks3-server[1194]: 2026/09/22 11:25:16 INFO Received uploads request method=POST path=/api/pending_closures7836worker2 # [ 233.522509] nix-grpc-daemon[2120]: time=2026-09-22T11:25:17.592Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7837worker2 # [ 233.523870] nix-grpc-daemon[2120]: time=2026-09-22T11:25:17.592Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7838lb # [ 233.552166] niks3-server[1194]: 2026/09/22 11:25:16 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7839lb # [ 233.560155] niks3-server[1194]: 2026/09/22 11:25:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign7840lb # [ 233.564992] niks3-server[1194]: 2026/09/22 11:25:16 INFO Signed narinfos id=82 count=17841worker2 # [ 233.557822] nix-grpc-daemon[2120]: time=2026-09-22T11:25:17.628Z level=INFO msg="Uploading 1 narinfos"7842lb # [ 233.568300] niks3-server[1194]: 2026/09/22 11:25:16 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7843lb # [ 233.579318] niks3-server[1194]: 2026/09/22 11:25:16 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete7844lb # [ 233.582031] niks3-server[1194]: 2026/09/22 11:25:16 INFO Completed upload id=827845worker2 # [ 233.572040] nix-grpc-daemon[2120]: time=2026-09-22T11:25:17.642Z level=INFO msg="Upload complete. (360ms)"7846lb # [ 233.583373] niks3-server[1194]: 2026/09/22 11:25:16 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7847worker2 # [ 233.574221] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33654 duration_s=0 paths=1 nar_bytes_in=6247848worker2 # [ 233.576977] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:336547849worker2 # [ 233.580751] nix-daemon[877]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7850worker2 # [ 233.586985] nix-daemon[877]: reaped child process 2166, status = succeeded7851client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7852worker2 # [ 233.591520] nix-daemon[877]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7853client # farm-b> worker2-next: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7854client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7855client # farm-b> LOG-farm-b-drain7856worker2 # [ 234.817324] nix-daemon[877]: reaped child process 2180, status = succeeded7857worker2 # [ 234.968388] nix-daemon[877]: accepted connection from pid 2196, user nix-grpc-daemon (trusted)7858worker2 # [ 234.979388] nix-daemon[877]: reaped child process 2199, status = succeeded7859lb # [ 235.004854] niks3-server[1194]: 2026/09/22 11:25:18 INFO Received uploads request method=POST path=/api/pending_closures7860worker2 # [ 234.999118] nix-grpc-daemon[2120]: time=2026-09-22T11:25:19.069Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7861worker2 # [ 235.000666] nix-grpc-daemon[2120]: time=2026-09-22T11:25:19.071Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7862lb # [ 235.022824] niks3-server[1194]: 2026/09/22 11:25:18 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7863lb # [ 235.026941] niks3-server[1194]: 2026/09/22 11:25:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign7864lb # [ 235.029362] niks3-server[1194]: 2026/09/22 11:25:18 INFO Signed narinfos id=83 count=17865worker2 # [ 235.020866] nix-grpc-daemon[2120]: time=2026-09-22T11:25:19.091Z level=INFO msg="Uploading 1 narinfos"7866lb # [ 235.032347] niks3-server[1194]: 2026/09/22 11:25:18 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7867lb # [ 235.034723] niks3-server[1194]: 2026/09/22 11:25:18 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7868lb # [ 235.038814] niks3-server[1194]: 2026/09/22 11:25:18 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete7869lb # [ 235.042085] niks3-server[1194]: 2026/09/22 11:25:18 INFO Completed upload id=837870worker2 # [ 235.033210] nix-grpc-daemon[2120]: time=2026-09-22T11:25:19.103Z level=INFO msg="Upload complete. (232ms)"7871lb # [ 235.043873] niks3-server[1194]: 2026/09/22 11:25:18 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7872worker2 # [ 235.035923] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33654 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=17873worker2 # [ 235.042686] nix-daemon[877]: reaped child process 2178, status = succeeded7874worker1 # [ 235.031812] nix-grpc-daemon[1597]: ts=2026-09-22T11:25:18Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07875worker1 # [ 235.036200] nix-grpc-daemon[1597]: ts=2026-09-22T11:25:18Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17876worker2 # [ 235.086069] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:336547877worker2 # [ 235.094440] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:336547878worker2 # [ 235.103715] nix-grpc-daemon[2120]: time=2026-09-22T11:25:19.174Z level=INFO msg="All 2 paths already cached"7879client # copying 1 paths to 'lb:50051'...7880client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7881worker2 # [ 235.109737] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:336547882worker2 # [ 235.116051] nix-daemon[877]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7883worker2 # [ 235.274527] nix-daemon[877]: accepted connection from pid 2203, user nix-grpc-daemon (trusted)7884worker2 # [ 235.287106] nix-daemon[877]: reaped child process 2206, status = succeeded7885worker2 # [ 235.442198] nix-daemon[877]: accepted connection from pid 2208, user nix-grpc-daemon (trusted)7886worker2 # [ 235.454777] nix-daemon[877]: reaped child process 2211, status = succeeded7887lb # [ 235.478419] niks3-server[1194]: 2026/09/22 11:25:18 INFO Received uploads request method=POST path=/api/pending_closures7888worker2 # [ 235.476183] nix-grpc-daemon[2120]: time=2026-09-22T11:25:19.546Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7889worker2 # [ 235.477586] nix-grpc-daemon[2120]: time=2026-09-22T11:25:19.546Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7890lb # [ 235.501187] niks3-server[1194]: 2026/09/22 11:25:18 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7891lb # [ 235.513735] niks3-server[1194]: 2026/09/22 11:25:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign7892worker2 # [ 235.508463] nix-grpc-daemon[2120]: time=2026-09-22T11:25:19.578Z level=INFO msg="Uploading 1 narinfos"7893lb # [ 235.518827] niks3-server[1194]: 2026/09/22 11:25:18 INFO Signed narinfos id=84 count=17894lb # [ 235.524838] niks3-server[1194]: 2026/09/22 11:25:18 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7895lb # [ 235.536950] niks3-server[1194]: 2026/09/22 11:25:18 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete7896worker2 # [ 235.531645] nix-grpc-daemon[2120]: time=2026-09-22T11:25:19.602Z level=INFO msg="Upload complete. (409ms)"7897lb # [ 235.541944] niks3-server[1194]: 2026/09/22 11:25:18 INFO Completed upload id=847898worker2 # [ 235.533798] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33654 duration_s=0 paths=1 nar_bytes_in=7527899lb # [ 235.545120] niks3-server[1194]: 2026/09/22 11:25:18 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7900worker2 # [ 235.538990] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:336547901worker2 # [ 235.544510] nix-daemon[877]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7902worker2 # [ 235.546831] nix-daemon[877]: reaped child process 2201, status = succeeded7903worker2 # [ 235.552052] nix-daemon[877]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7904client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7905client # farm-top> worker2-next: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7906lb # [ 236.791160] envoy[899]: conn peer=[2001:db8:1::1]:49758 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300477907client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7908client # farm-top> LOG-farm-top-drain7909worker2 # [ 237.168849] nix-daemon[877]: reaped child process 2215, status = succeeded7910worker2 # [ 237.305332] nix-daemon[877]: accepted connection from pid 2231, user nix-grpc-daemon (trusted)7911worker2 # [ 237.318346] nix-daemon[877]: reaped child process 2234, status = succeeded7912lb # [ 237.343293] niks3-server[1194]: 2026/09/22 11:25:20 INFO Received uploads request method=POST path=/api/pending_closures7913worker2 # [ 237.342112] nix-grpc-daemon[2120]: time=2026-09-22T11:25:21.412Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7914worker2 # [ 237.343826] nix-grpc-daemon[2120]: time=2026-09-22T11:25:21.414Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7915lb # [ 237.385630] niks3-server[1194]: 2026/09/22 11:25:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign7916lb # [ 237.392127] niks3-server[1194]: 2026/09/22 11:25:20 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7917lb # [ 237.397871] niks3-server[1194]: 2026/09/22 11:25:20 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7918worker2 # [ 237.394031] nix-grpc-daemon[2120]: time=2026-09-22T11:25:21.464Z level=INFO msg="Uploading 1 narinfos"7919lb # [ 237.404144] niks3-server[1194]: 2026/09/22 11:25:20 INFO Signed narinfos id=85 count=17920lb # [ 237.414544] niks3-server[1194]: 2026/09/22 11:25:20 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7921lb # [ 237.419616] niks3-server[1194]: 2026/09/22 11:25:20 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete7922worker2 # [ 237.415757] nix-grpc-daemon[2120]: time=2026-09-22T11:25:21.486Z level=INFO msg="Upload complete. (256ms)"7923lb # [ 237.425814] niks3-server[1194]: 2026/09/22 11:25:20 INFO Completed upload id=857924worker2 # [ 237.418488] nix-grpc-daemon[2060]: ts=2026-09-22T11:25:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33654 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=17925lb # [ 237.430280] niks3-server[1194]: 2026/09/22 11:25:20 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7926worker2 # [ 237.425746] nix-daemon[877]: reaped child process 2213, status = succeeded7927client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 7.13 seconds)7928client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7929client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.20 seconds)7930worker1: must succeed: rm /nix/.rw-store/fill7931worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.13 seconds)7932worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7933worker1 # [ 238.418260] nix-grpc-daemon[1597]: ts=2026-09-22T11:25:21Z level=info event=healthy reason=min_free7934worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.12 seconds)7935(finished: subtest: low disk drains a worker and builds go to the other, in 10.69 seconds)7936(finished: run the VM test script, in 239.85 seconds)7937test script finished in 240.11s7938cleanup7939kill QemuMachine (pid 50)7940client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7941kill QemuMachine (pid 49)7942lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7943kill QemuMachine (pid 51)7944worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7945kill QemuMachine (pid 52)7946worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7947(finished: cleanup, in 0.93 seconds)7948additionally exposed symbols:7949 client, lb, worker1, worker2,7950 vlan1,7951 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh