vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #488
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12client # Disk image does not exist, creating the virtualisation disk image...13client: QEMU running (pid 49)14client # Formatting '/build/vm-state-client/tmp.Cl138Hvb2q', fmt=raw size=107374182415client # mke2fs 1.47.4 (6-Mar-2025)16lb: QEMU running (pid 50)17client # Discarding device blocks: 0/262144 done18lb # Disk image does not exist, creating the virtualisation disk image...19worker1 # Disk image does not exist, creating the virtualisation disk image...20lb # Formatting '/build/vm-state-lb/tmp.7gkPP0O52I', fmt=raw size=107374182421worker1: QEMU running (pid 52)22client # Creating filesystem with 262144 4k blocks and 65536 inodes23worker1 # Formatting '/build/vm-state-worker1/tmp.V3TAw9cEjQ', fmt=raw size=107374182424client # Filesystem UUID: c5faa8fb-fdc6-4740-98ef-e25fb2fde93f25worker1 # mke2fs 1.47.4 (6-Mar-2025)26client # Superblock backups stored on blocks:27worker1 # Discarding device blocks: 0/262144 done28client # 32768, 98304, 163840, 22937629worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes30client # 31worker1 # Filesystem UUID: d1ac7301-38b1-4955-b31d-72d6a0c4eeb832client # Allocating group tables: 0/8 done33worker1 # Superblock backups stored on blocks:34client # Writing inode tables: 0/8 done35worker1 # 32768, 98304, 163840, 22937636client # Creating journal (8192 blocks): done37worker1 # 38client # Writing superblocks and filesystem accounting information: 0/8 done39worker1 # Allocating group tables: 0/8 done40client # 41worker1 # Writing inode tables: 0/8 done42client # Virtualisation disk image created.43worker1 # Creating journal (8192 blocks): done44lb # mke2fs 1.47.4 (6-Mar-2025)45worker1 # Writing superblocks and filesystem accounting information: 0/8 done46lb # Discarding device blocks: 0/262144 done47worker1 # 48lb # Creating filesystem with 262144 4k blocks and 65536 inodes49worker1 # Virtualisation disk image created.50lb # Filesystem UUID: c00a4a05-7933-4dfe-a294-b9d1bc646e1c51lb # Superblock backups stored on blocks:52lb # 32768, 98304, 163840, 22937653lb # 54lb # Allocating group tables: 0/8 done55lb # Writing inode tables: 0/8 done56lb # Creating journal (8192 blocks): done57lb # Writing superblocks and filesystem accounting information: 0/8 done58lb # 59lb # Virtualisation disk image created.60worker2 # Disk image does not exist, creating the virtualisation disk image...61worker2: QEMU running (pid 51)62worker2 # Formatting '/build/vm-state-worker2/tmp.1j47e49kQV', fmt=raw size=107374182463worker2 # mke2fs 1.47.4 (6-Mar-2025)64(finished: start all VMs, in 0.26 seconds)65worker2 # Discarding device blocks: 0/262144 done66client: waiting for unit network-addresses-eth1.service67worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes68client: waiting for the VM to finish booting69worker2 # Filesystem UUID: 05f2a824-5c12-4135-be00-fd414e24052170worker2 # Superblock backups stored on blocks:71worker2 # 32768, 98304, 163840, 22937672worker2 # 73worker2 # Allocating group tables: 0/8 done74worker2 # Writing inode tables: 0/8 done75worker2 # Creating journal (8192 blocks): done76worker2 # Writing superblocks and filesystem accounting information: 0/8 done77worker2 # 78worker2 # Virtualisation disk image created.79lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83lb # 84lb # 85lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0086lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...87lb # 88lb # 89lb # 90lb # 91lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB0092client # 93client # 94lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...95lb # 96lb # 97client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0098worker2 # 99worker2 # 100worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00101client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...102client # 103client # 104client # 105client # 106worker1 # 107worker1 # 108client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00109worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00110worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...111worker2 # 112worker2 # 113worker2 # 114worker2 # 115worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00116worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...117worker1 # 118worker1 # 119worker1 # 120worker1 # 121client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...122client # 123client # 124worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00125worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...126worker2 # 127worker2 # 128worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129worker1 # 130worker1 # 131lb # Booting from ROM...132client # Booting from ROM...133worker2 # Booting from ROM...134worker1 # Booting from ROM...135lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/8mvx6cn1l6frdr55gdj0p01an8nxnxzn-nixos-system-lb-test/init regInfo=/nix/store/0a2f35z8lm3xm6ffdv71jfscn9vyk5l1-closure-info/registration console=ttyS0,115200n8 console=tty0137lb # [ 0.000000] BIOS-provided physical RAM map:138lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved141lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable142client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026143lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved144lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved145lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved146lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved147lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved148client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/zch17lfr063cj7f5rrjws8177ikhh5yq-nixos-system-client-test/init regInfo=/nix/store/qnavl2rn6zqsfxv612a754dx1lmnpc7g-closure-info/registration console=ttyS0,115200n8 console=tty0149client # [ 0.000000] BIOS-provided physical RAM map:150lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved151lb # [ 0.000000] NX (Execute Disable) protection: active152client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable153lb # [ 0.000000] APIC: Static calls initialized154lb # [ 0.000000] SMBIOS 2.8 present.155client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved156client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved157lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014158worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026159client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable160lb # [ 0.000000] DMI: Memory slots populated: 1/1161lb # [ 0.000000] Hypervisor detected: KVM162client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved163worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026164lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000165client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved166lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00167client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved168lb # [ 0.000000] kvm-clock: using sched offset of 486018240 cycles169client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved170worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/b4bykh1l473ki3g46f1yx84d1vv0a4hm-nixos-system-worker1-test/init regInfo=/nix/store/a8pnrrvlyaaij5bfwn387nrjbs9aiqwz-closure-info/registration console=ttyS0,115200n8 console=tty0171lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns172client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved173worker1 # [ 0.000000] BIOS-provided physical RAM map:174lb # [ 0.000005] tsc: Detected 2400.012 MHz processor175worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/kfz0w6wji9w61zllsspgwfdm3jmk8ba9-nixos-system-worker2-test/init regInfo=/nix/store/a2dpll57mdc301scrcj612nkyhxaivga-closure-info/registration console=ttyS0,115200n8 console=tty0176client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved177worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable178lb # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000179worker2 # [ 0.000000] BIOS-provided physical RAM map:180client # [ 0.000000] NX (Execute Disable) protection: active181worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved182lb # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs183worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable184client # [ 0.000000] APIC: Static calls initialized185client # [ 0.000000] SMBIOS 2.8 present.186worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved187lb # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT188worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved189worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable190lb # [ 0.002728] found SMP MP-table at [mem 0x000f5450-0x000f545f]191worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved192client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014193lb # [ 0.002742] Using GB pages for direct mapping194client # [ 0.000000] DMI: Memory slots populated: 1/1195worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved196worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable197lb # [ 0.002821] RAMDISK: [mem 0x5e367000-0x5ffcffff]198client # [ 0.000000] Hypervisor detected: KVM199worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved200client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000201lb # [ 0.002830] ACPI: Early table checksum verification disabled202worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved203worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved204lb # [ 0.002835] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )205client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00206worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved207client # [ 0.000001] kvm-clock: using sched offset of 500996085 cycles208worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved209worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved210lb # [ 0.002839] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)211worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved212client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns213worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved214lb # [ 0.002844] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)215worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved216client # [ 0.000005] tsc: Detected 2400.012 MHz processor217worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved218worker1 # [ 0.000000] NX (Execute Disable) protection: active219lb # [ 0.002852] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)220client # [ 0.000815] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000221worker1 # [ 0.000000] APIC: Static calls initialized222worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved223lb # [ 0.002854] ACPI: FACS 0x000000005FFE0000 000040224worker1 # [ 0.000000] SMBIOS 2.8 present.225client # [ 0.000842] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs226worker2 # [ 0.000000] NX (Execute Disable) protection: active227worker2 # [ 0.000000] APIC: Static calls initialized228lb # [ 0.002855] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)229client # [ 0.000845] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT230worker2 # [ 0.000000] SMBIOS 2.8 present.231worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014232worker1 # [ 0.000000] DMI: Memory slots populated: 1/1233client # [ 0.002740] found SMP MP-table at [mem 0x000f5450-0x000f545f]234lb # [ 0.002857] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)235worker1 # [ 0.000000] Hypervisor detected: KVM236client # [ 0.002751] Using GB pages for direct mapping237worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014238client # [ 0.002845] RAMDISK: [mem 0x5e366000-0x5ffcffff]239worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000240lb # [ 0.002859] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)241worker2 # [ 0.000000] DMI: Memory slots populated: 1/1242worker2 # [ 0.000000] Hypervisor detected: KVM243client # [ 0.002853] ACPI: Early table checksum verification disabled244worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00245lb # [ 0.002860] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)246worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000247worker1 # [ 0.000000] kvm-clock: using sched offset of 514373989 cycles248client # [ 0.002856] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )249lb # [ 0.002861] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]250worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00251worker2 # [ 0.000000] kvm-clock: using sched offset of 487419813 cycles252client # [ 0.002860] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)253worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns254lb # [ 0.002862] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]255worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor256client # [ 0.002864] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)257lb # [ 0.002863] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]258worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns259worker1 # [ 0.000810] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000260worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor261client # [ 0.002871] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)262lb # [ 0.002863] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]263worker1 # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs264client # [ 0.002873] ACPI: FACS 0x000000005FFE0000 000040265worker2 # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000266lb # [ 0.002864] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]267worker1 # [ 0.000840] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT268worker2 # [ 0.000839] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs269client # [ 0.002874] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)270lb # [ 0.002865] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]271worker1 # [ 0.002734] found SMP MP-table at [mem 0x000f5450-0x000f545f]272worker2 # [ 0.000842] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT273worker1 # [ 0.002745] Using GB pages for direct mapping274client # [ 0.002876] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)275lb # [ 0.002865] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]276worker1 # [ 0.002879] RAMDISK: [mem 0x5e367000-0x5ffcffff]277worker2 # [ 0.002744] found SMP MP-table at [mem 0x000f5450-0x000f545f]278lb # [ 0.003104] No NUMA configuration found279worker2 # [ 0.002757] Using GB pages for direct mapping280client # [ 0.002877] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)281worker1 # [ 0.002889] ACPI: Early table checksum verification disabled282lb # [ 0.003105] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]283worker2 # [ 0.002909] RAMDISK: [mem 0x5e367000-0x5ffcffff]284worker1 # [ 0.002892] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )285client # [ 0.002879] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)286lb # [ 0.003108] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]287worker2 # [ 0.002917] ACPI: Early table checksum verification disabled288lb # [ 0.005516] Zone ranges:289worker1 # [ 0.002896] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)290client # [ 0.002880] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]291worker2 # [ 0.002920] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )292lb # [ 0.005517] DMA [mem 0x0000000000001000-0x0000000000ffffff]293client # [ 0.002881] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]294lb # [ 0.005519] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]295worker2 # [ 0.002925] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)296worker1 # [ 0.002900] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)297lb # [ 0.005521] Normal empty298lb # [ 0.005522] Device empty299client # [ 0.002882] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]300worker1 # [ 0.002907] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)301worker2 # [ 0.002930] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)302lb # [ 0.005522] Movable zone start for each node303worker1 # [ 0.002909] ACPI: FACS 0x000000005FFE0000 000040304client # [ 0.002882] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]305lb # [ 0.005523] Early memory node ranges306worker2 # [ 0.002937] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)307lb # [ 0.005524] node 0: [mem 0x0000000000001000-0x000000000009efff]308client # [ 0.002883] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]309worker1 # [ 0.002911] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)310worker2 # [ 0.002939] ACPI: FACS 0x000000005FFE0000 000040311lb # [ 0.005525] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]312client # [ 0.002883] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]313worker1 # [ 0.002912] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)314worker2 # [ 0.002940] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)315lb # [ 0.005526] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]316client # [ 0.002884] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]317client # [ 0.003115] No NUMA configuration found318worker1 # [ 0.002914] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)319lb # [ 0.005549] On node 0, zone DMA: 1 pages in unavailable ranges320worker2 # [ 0.002942] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)321client # [ 0.003116] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]322lb # [ 0.005835] On node 0, zone DMA: 97 pages in unavailable ranges323worker1 # [ 0.002915] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)324worker2 # [ 0.002943] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)325lb # [ 0.033815] On node 0, zone DMA32: 40 pages in unavailable ranges326client # [ 0.003118] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]327worker1 # [ 0.002917] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]328client # [ 0.005864] Zone ranges:329worker2 # [ 0.002945] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)330lb # [ 0.034280] ACPI: PM-Timer IO Port: 0x608331client # [ 0.005865] DMA [mem 0x0000000000001000-0x0000000000ffffff]332worker1 # [ 0.002918] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]333lb # [ 0.034296] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])334worker2 # [ 0.002946] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]335client # [ 0.005867] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]336worker1 # [ 0.002918] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]337lb # [ 0.034325] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23338client # [ 0.005868] Normal empty339worker2 # [ 0.002947] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]340client # [ 0.005869] Device empty341lb # [ 0.034328] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)342worker1 # [ 0.002919] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]343client # [ 0.005869] Movable zone start for each node344worker2 # [ 0.002948] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]345client # [ 0.005870] Early memory node ranges346worker1 # [ 0.002919] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]347lb # [ 0.034329] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)348worker2 # [ 0.002948] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]349client # [ 0.005870] node 0: [mem 0x0000000000001000-0x000000000009efff]350lb # [ 0.034331] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)351worker1 # [ 0.002920] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]352worker2 # [ 0.002949] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]353client # [ 0.005871] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]354lb # [ 0.034332] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)355worker1 # [ 0.002920] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]356worker2 # [ 0.002949] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]357worker1 # [ 0.003155] No NUMA configuration found358client # [ 0.005872] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]359lb # [ 0.034332] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)360worker2 # [ 0.002950] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]361client # [ 0.005894] On node 0, zone DMA: 1 pages in unavailable ranges362worker1 # [ 0.003156] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]363lb # [ 0.034335] ACPI: Using ACPI (MADT) for SMP configuration information364worker2 # [ 0.003179] No NUMA configuration found365client # [ 0.006172] On node 0, zone DMA: 97 pages in unavailable ranges366worker1 # [ 0.003158] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]367lb # [ 0.034336] ACPI: HPET id: 0x8086a201 base: 0xfed00000368worker1 # [ 0.005597] Zone ranges:369worker2 # [ 0.003179] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]370lb # [ 0.034340] TSC deadline timer available371client # [ 0.033150] On node 0, zone DMA32: 40 pages in unavailable ranges372client # [ 0.033617] ACPI: PM-Timer IO Port: 0x608373worker1 # [ 0.005597] DMA [mem 0x0000000000001000-0x0000000000ffffff]374lb # [ 0.034345] CPU topo: Max. logical packages: 1375worker2 # [ 0.003182] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]376worker2 # [ 0.005562] Zone ranges:377worker1 # [ 0.005599] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]378client # [ 0.033628] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])379lb # [ 0.034346] CPU topo: Max. logical dies: 1380worker1 # [ 0.005600] Normal empty381worker2 # [ 0.005563] DMA [mem 0x0000000000001000-0x0000000000ffffff]382lb # [ 0.034346] CPU topo: Max. dies per package: 1383worker1 # [ 0.005601] Device empty384client # [ 0.033658] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23385lb # [ 0.034349] CPU topo: Max. threads per core: 1386worker1 # [ 0.005602] Movable zone start for each node387worker2 # [ 0.005565] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]388worker2 # [ 0.005566] Normal empty389client # [ 0.033661] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)390lb # [ 0.034350] CPU topo: Num. cores per package: 1391worker1 # [ 0.005602] Early memory node ranges392worker2 # [ 0.005567] Device empty393lb # [ 0.034351] CPU topo: Num. threads per package: 1394worker1 # [ 0.005603] node 0: [mem 0x0000000000001000-0x000000000009efff]395client # [ 0.033662] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)396worker2 # [ 0.005568] Movable zone start for each node397worker2 # [ 0.005569] Early memory node ranges398lb # [ 0.034351] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs399worker1 # [ 0.005604] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]400client # [ 0.033663] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)401worker2 # [ 0.005569] node 0: [mem 0x0000000000001000-0x000000000009efff]402lb # [ 0.034376] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()403client # [ 0.033664] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)404worker1 # [ 0.005605] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]405worker2 # [ 0.005570] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]406worker1 # [ 0.005628] On node 0, zone DMA: 1 pages in unavailable ranges407lb # [ 0.034413] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]408client # [ 0.033665] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)409worker2 # [ 0.005572] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]410worker1 # [ 0.005905] On node 0, zone DMA: 97 pages in unavailable ranges411client # [ 0.033668] ACPI: Using ACPI (MADT) for SMP configuration information412lb # [ 0.034414] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]413worker2 # [ 0.005592] On node 0, zone DMA: 1 pages in unavailable ranges414worker1 # [ 0.032997] On node 0, zone DMA32: 40 pages in unavailable ranges415client # [ 0.033669] ACPI: HPET id: 0x8086a201 base: 0xfed00000416lb # [ 0.034416] [mem 0x60000000-0xafffffff] available for PCI devices417worker2 # [ 0.005871] On node 0, zone DMA: 97 pages in unavailable ranges418worker1 # [ 0.033466] ACPI: PM-Timer IO Port: 0x608419client # [ 0.033673] TSC deadline timer available420lb # [ 0.034418] Booting paravirtualized kernel on KVM421client # [ 0.033677] CPU topo: Max. logical packages: 1422worker1 # [ 0.033479] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])423worker2 # [ 0.032452] On node 0, zone DMA32: 40 pages in unavailable ranges424client # [ 0.033678] CPU topo: Max. logical dies: 1425worker2 # [ 0.032921] ACPI: PM-Timer IO Port: 0x608426worker1 # [ 0.033509] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23427lb # [ 0.034422] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns428client # [ 0.033678] CPU topo: Max. dies per package: 1429worker2 # [ 0.032934] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])430worker1 # [ 0.033511] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)431client # [ 0.033682] CPU topo: Max. threads per core: 1432lb # [ 0.038892] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1433worker2 # [ 0.032964] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23434client # [ 0.033682] CPU topo: Num. cores per package: 1435worker1 # [ 0.033513] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)436lb # [ 0.041055] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152437worker2 # [ 0.032967] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)438client # [ 0.033683] CPU topo: Num. threads per package: 1439worker1 # [ 0.033514] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)440lb # [ 0.041105] kvm-guest: PV spinlocks disabled, single CPU441client # [ 0.033683] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs442worker2 # [ 0.032969] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)443worker1 # [ 0.033515] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)444worker2 # [ 0.032970] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)445client # [ 0.033702] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()446worker1 # [ 0.033516] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)447worker2 # [ 0.032971] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)448client # [ 0.033734] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]449worker1 # [ 0.033518] ACPI: Using ACPI (MADT) for SMP configuration information450worker2 # [ 0.032972] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)451worker1 # [ 0.033519] ACPI: HPET id: 0x8086a201 base: 0xfed00000452client # [ 0.033736] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]453worker1 # [ 0.033523] TSC deadline timer available454worker2 # [ 0.032974] ACPI: Using ACPI (MADT) for SMP configuration information455client # [ 0.033738] [mem 0x60000000-0xafffffff] available for PCI devices456lb # [ 0.041106] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/8mvx6cn1l6frdr55gdj0p01an8nxnxzn-nixos-system-lb-test/init regInfo=/nix/store/0a2f35z8lm3xm6ffdv71jfscn9vyk5l1-closure-info/registration console=ttyS0,115200n8 console=tty0457worker1 # [ 0.033527] CPU topo: Max. logical packages: 1458worker2 # [ 0.032975] ACPI: HPET id: 0x8086a201 base: 0xfed00000459client # [ 0.033739] Booting paravirtualized kernel on KVM460worker1 # [ 0.033528] CPU topo: Max. logical dies: 1461worker2 # [ 0.032980] TSC deadline timer available462worker1 # [ 0.033528] CPU topo: Max. dies per package: 1463worker2 # [ 0.032984] CPU topo: Max. logical packages: 1464lb # [ 0.041208] Unknown kernel command line parameters "regInfo=/nix/store/0a2f35z8lm3xm6ffdv71jfscn9vyk5l1-closure-info/registration", will be passed to user space.465worker1 # [ 0.033532] CPU topo: Max. threads per core: 1466client # [ 0.033742] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns467worker2 # [ 0.032985] CPU topo: Max. logical dies: 1468lb # [ 0.041221] random: crng init done469worker1 # [ 0.033533] CPU topo: Num. cores per package: 1470worker2 # [ 0.032986] CPU topo: Max. dies per package: 1471client # [ 0.038195] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1472worker1 # [ 0.033533] CPU topo: Num. threads per package: 1473lb # [ 0.041222] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes474worker2 # [ 0.032989] CPU topo: Max. threads per core: 1475client # [ 0.040753] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152476worker2 # [ 0.032990] CPU topo: Num. cores per package: 1477worker1 # [ 0.033533] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs478lb # [ 0.043336] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)479worker2 # [ 0.032991] CPU topo: Num. threads per package: 1480client # [ 0.040799] kvm-guest: PV spinlocks disabled, single CPU481worker1 # [ 0.033552] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()482worker2 # [ 0.032991] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs483worker1 # [ 0.033583] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]484lb # [ 0.044608] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)485worker2 # [ 0.033014] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()486worker1 # [ 0.033585] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]487worker2 # [ 0.033050] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]488lb # [ 0.044642] Fallback order for Node 0: 0489worker1 # [ 0.033586] [mem 0x60000000-0xafffffff] available for PCI devices490worker2 # [ 0.033052] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]491worker1 # [ 0.033588] Booting paravirtualized kernel on KVM492client # [ 0.040801] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/zch17lfr063cj7f5rrjws8177ikhh5yq-nixos-system-client-test/init regInfo=/nix/store/qnavl2rn6zqsfxv612a754dx1lmnpc7g-closure-info/registration console=ttyS0,115200n8 console=tty0493worker2 # [ 0.033053] [mem 0x60000000-0xafffffff] available for PCI devices494lb # [ 0.044645] Built 1 zonelists, mobility grouping on. Total pages: 393078495lb # [ 0.044646] Policy zone: DMA32496worker2 # [ 0.033055] Booting paravirtualized kernel on KVM497worker1 # [ 0.033591] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns498lb # [ 0.047148] mem auto-init: stack:all(zero), heap alloc:on, heap free:off499client # [ 0.040896] Unknown kernel command line parameters "regInfo=/nix/store/qnavl2rn6zqsfxv612a754dx1lmnpc7g-closure-info/registration", will be passed to user space.500worker1 # [ 0.038097] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1501client # [ 0.040909] random: crng init done502worker2 # [ 0.033059] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns503lb # [ 0.050571] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1504worker1 # [ 0.040455] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152505lb # [ 0.053877] allocated 3145728 bytes of page_ext506client # [ 0.040909] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes507worker2 # [ 0.037570] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1508worker1 # [ 0.040503] kvm-guest: PV spinlocks disabled, single CPU509lb # [ 0.064164] ftrace: allocating 48731 entries in 192 pages510client # [ 0.043021] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)511worker2 # [ 0.039743] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152512lb # [ 0.064166] ftrace: allocated 192 pages with 2 groups513lb # [ 0.065220] Dynamic Preempt: lazy514worker2 # [ 0.039784] kvm-guest: PV spinlocks disabled, single CPU515client # [ 0.044081] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)516lb # [ 0.065342] rcu: Preemptible hierarchical RCU implementation.517client # [ 0.044112] Fallback order for Node 0: 0518lb # [ 0.065343] rcu: RCU event tracing is enabled.519client # [ 0.044115] Built 1 zonelists, mobility grouping on. Total pages: 393078520client # [ 0.044116] Policy zone: DMA32521lb # [ 0.065343] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.522worker1 # [ 0.040504] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/b4bykh1l473ki3g46f1yx84d1vv0a4hm-nixos-system-worker1-test/init regInfo=/nix/store/a8pnrrvlyaaij5bfwn387nrjbs9aiqwz-closure-info/registration console=ttyS0,115200n8 console=tty0523lb # [ 0.065345] Trampoline variant of Tasks RCU enabled.524client # [ 0.046899] mem auto-init: stack:all(zero), heap alloc:on, heap free:off525lb # [ 0.065346] Rude variant of Tasks RCU enabled.526client # [ 0.050153] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1527lb # [ 0.065346] Tracing variant of Tasks RCU enabled.528worker2 # [ 0.039786] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/kfz0w6wji9w61zllsspgwfdm3jmk8ba9-nixos-system-worker2-test/init regInfo=/nix/store/a2dpll57mdc301scrcj612nkyhxaivga-closure-info/registration console=ttyS0,115200n8 console=tty0529client # [ 0.053452] allocated 3145728 bytes of page_ext530worker1 # [ 0.040600] Unknown kernel command line parameters "regInfo=/nix/store/a8pnrrvlyaaij5bfwn387nrjbs9aiqwz-closure-info/registration", will be passed to user space.531worker1 # [ 0.040612] random: crng init done532lb # [ 0.065347] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.533client # [ 0.063806] ftrace: allocating 48731 entries in 192 pages534client # [ 0.063808] ftrace: allocated 192 pages with 2 groups535worker2 # [ 0.039884] Unknown kernel command line parameters "regInfo=/nix/store/a2dpll57mdc301scrcj612nkyhxaivga-closure-info/registration", will be passed to user space.536worker1 # [ 0.040614] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes537lb # [ 0.065348] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1538client # [ 0.064696] Dynamic Preempt: lazy539worker2 # [ 0.039897] random: crng init done540worker1 # [ 0.042736] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)541client # [ 0.064833] rcu: Preemptible hierarchical RCU implementation.542lb # [ 0.065358] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.543worker2 # [ 0.039898] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes544client # [ 0.064834] rcu: RCU event tracing is enabled.545worker1 # [ 0.043799] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)546lb # [ 0.065360] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.547worker2 # [ 0.042029] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)548client # [ 0.064835] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.549worker1 # [ 0.043830] Fallback order for Node 0: 0550client # [ 0.064837] Trampoline variant of Tasks RCU enabled.551worker2 # [ 0.043088] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)552lb # [ 0.065361] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.553worker1 # [ 0.043833] Built 1 zonelists, mobility grouping on. Total pages: 393078554client # [ 0.064837] Rude variant of Tasks RCU enabled.555worker1 # [ 0.043834] Policy zone: DMA32556worker2 # [ 0.043120] Fallback order for Node 0: 0557lb # [ 0.069815] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16558client # [ 0.064838] Tracing variant of Tasks RCU enabled.559worker1 # [ 0.046345] mem auto-init: stack:all(zero), heap alloc:on, heap free:off560worker2 # [ 0.043123] Built 1 zonelists, mobility grouping on. Total pages: 393078561lb # [ 0.070096] rcu: srcu_init: Setting srcu_struct sizes based on contention.562worker2 # [ 0.043124] Policy zone: DMA32563client # [ 0.064838] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.564worker1 # [ 0.049484] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1565worker2 # [ 0.045638] mem auto-init: stack:all(zero), heap alloc:on, heap free:off566worker1 # [ 0.052991] allocated 3145728 bytes of page_ext567client # [ 0.064839] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1568lb # [ 0.070103] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns569worker1 # [ 0.063249] ftrace: allocating 48731 entries in 192 pages570worker2 # [ 0.048906] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1571client # [ 0.064854] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.572worker2 # [ 0.052402] allocated 3145728 bytes of page_ext573worker1 # [ 0.063251] ftrace: allocated 192 pages with 2 groups574lb # [ 0.070215] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)575worker1 # [ 0.064107] Dynamic Preempt: lazy576worker2 # [ 0.062626] ftrace: allocating 48731 entries in 192 pages577lb # [ 0.073743] Console: colour VGA+ 80x25578client # [ 0.064856] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.579worker1 # [ 0.064252] rcu: Preemptible hierarchical RCU implementation.580worker2 # [ 0.062627] ftrace: allocated 192 pages with 2 groups581lb # [ 0.073747] printk: legacy console [tty0] enabled582worker1 # [ 0.064253] rcu: RCU event tracing is enabled.583worker2 # [ 0.063471] Dynamic Preempt: lazy584lb # [ 0.103273] printk: legacy console [ttyS0] enabled585client # [ 0.064857] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.586worker2 # [ 0.063588] rcu: Preemptible hierarchical RCU implementation.587worker1 # [ 0.064253] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.588lb # [ 0.202237] ACPI: Core revision 20250807589client # [ 0.069295] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16590worker2 # [ 0.063589] rcu: RCU event tracing is enabled.591worker1 # [ 0.064255] Trampoline variant of Tasks RCU enabled.592worker1 # [ 0.064255] Rude variant of Tasks RCU enabled.593worker2 # [ 0.063590] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.594client # [ 0.069574] rcu: srcu_init: Setting srcu_struct sizes based on contention.595worker1 # [ 0.064256] Tracing variant of Tasks RCU enabled.596worker2 # [ 0.063591] Trampoline variant of Tasks RCU enabled.597lb # [ 0.203133] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns598client # [ 0.069581] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns599worker2 # [ 0.063592] Rude variant of Tasks RCU enabled.600worker1 # [ 0.064256] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.601lb # [ 0.204618] APIC: Switch to symmetric I/O mode setup602worker2 # [ 0.063592] Tracing variant of Tasks RCU enabled.603worker1 # [ 0.064257] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1604lb # [ 0.205553] x2apic enabled605client # [ 0.069677] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)606worker2 # [ 0.063593] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.607client # [ 0.073219] Console: colour VGA+ 80x25608worker1 # [ 0.064311] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.609client # [ 0.073222] printk: legacy console [tty0] enabled610lb # [ 0.206286] APIC: Switched APIC routing to: physical x2apic611worker2 # [ 0.063593] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1612client # [ 0.102612] printk: legacy console [ttyS0] enabled613worker1 # [ 0.064313] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.614worker2 # [ 0.063614] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.615client # [ 0.198439] ACPI: Core revision 20250807616lb # [ 0.208053] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1617worker1 # [ 0.064314] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.618worker2 # [ 0.063616] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.619worker1 # [ 0.068700] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16620client # [ 0.199315] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns621worker2 # [ 0.063617] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.622lb # [ 0.208995] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns623worker1 # [ 0.068970] rcu: srcu_init: Setting srcu_struct sizes based on contention.624client # [ 0.200782] APIC: Switch to symmetric I/O mode setup625worker2 # [ 0.068033] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16626client # [ 0.201749] x2apic enabled627worker1 # [ 0.068978] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns628worker2 # [ 0.068318] rcu: srcu_init: Setting srcu_struct sizes based on contention.629lb # [ 0.210566] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)630client # [ 0.202483] APIC: Switched APIC routing to: physical x2apic631lb # [ 0.212649] x86/cpu: User Mode Instruction Prevention (UMIP) activated632worker1 # [ 0.069081] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)633worker2 # [ 0.068325] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns634worker1 # [ 0.072618] Console: colour VGA+ 80x25635worker1 # [ 0.072620] printk: legacy console [tty0] enabled636lb # [ 0.213698] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127637worker2 # [ 0.068439] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)638client # [ 0.204295] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1639worker1 # [ 0.102044] printk: legacy console [ttyS0] enabled640worker2 # [ 0.071956] Console: colour VGA+ 80x25641lb # [ 0.214563] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0642worker2 # [ 0.071959] printk: legacy console [tty0] enabled643worker1 # [ 0.197060] ACPI: Core revision 20250807644worker2 # [ 0.101300] printk: legacy console [ttyS0] enabled645client # [ 0.205249] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns646worker2 # [ 0.196481] ACPI: Core revision 20250807647lb # [ 0.215566] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto648worker1 # [ 0.197909] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns649client # [ 0.206855] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)650worker1 # [ 0.199408] APIC: Switch to symmetric I/O mode setup651lb # [ 0.216563] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl652worker2 # [ 0.197374] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns653worker1 # [ 0.200387] x2apic enabled654client # [ 0.207940] x86/cpu: User Mode Instruction Prevention (UMIP) activated655lb # [ 0.217563] Transient Scheduler Attacks: Vulnerable: No microcode656worker2 # [ 0.198939] APIC: Switch to symmetric I/O mode setup657worker1 # [ 0.201151] APIC: Switched APIC routing to: physical x2apic658client # [ 0.208976] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127659worker2 # [ 0.199890] x2apic enabled660lb # [ 0.218562] Spectre V2 : Mitigation: Enhanced / Automatic IBRS661client # [ 0.209852] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0662worker2 # [ 0.200636] APIC: Switched APIC routing to: physical x2apic663lb # [ 0.219563] Speculative Return Stack Overflow: Mitigation: Safe RET664worker1 # [ 0.202946] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1665client # [ 0.210856] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto666worker2 # [ 0.202399] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1667worker1 # [ 0.203901] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns668client # [ 0.211852] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl669worker2 # [ 0.203357] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns670worker1 # [ 0.205527] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)671client # [ 0.212852] Transient Scheduler Attacks: Vulnerable: No microcode672lb # [ 0.220562] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization673lb # [ 0.222570] Spectre V2 : Enabling IBPB for BPF674worker1 # [ 0.206613] x86/cpu: User Mode Instruction Prevention (UMIP) activated675client # [ 0.213851] Spectre V2 : Mitigation: Enhanced / Automatic IBRS676worker2 # [ 0.204958] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)677client # [ 0.214852] Speculative Return Stack Overflow: Mitigation: Safe RET678worker1 # [ 0.207659] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127679lb # [ 0.223280] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier680worker2 # [ 0.206065] x86/cpu: User Mode Instruction Prevention (UMIP) activated681worker1 # [ 0.208524] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0682lb # [ 0.224563] active return thunk: srso_alias_return_thunk683worker2 # [ 0.208048] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127684lb # [ 0.225577] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'685worker1 # [ 0.209528] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto686worker2 # [ 0.208956] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0687lb # [ 0.226562] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'688client # [ 0.215852] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization689worker1 # [ 0.210524] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl690lb # [ 0.227562] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'691client # [ 0.217859] Spectre V2 : Enabling IBPB for BPF692worker1 # [ 0.211524] Transient Scheduler Attacks: Vulnerable: No microcode693worker2 # [ 0.209961] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto694lb # [ 0.228563] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'695client # [ 0.218853] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier696worker1 # [ 0.212523] Spectre V2 : Mitigation: Enhanced / Automatic IBRS697worker2 # [ 0.210958] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl698lb # [ 0.229566] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'699client # [ 0.219852] active return thunk: srso_alias_return_thunk700worker1 # [ 0.213523] Speculative Return Stack Overflow: Mitigation: Safe RET701worker2 # [ 0.211956] Transient Scheduler Attacks: Vulnerable: No microcode702lb # [ 0.230563] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'703client # [ 0.220874] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'704worker2 # [ 0.212955] Spectre V2 : Mitigation: Enhanced / Automatic IBRS705lb # [ 0.231563] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'706client # [ 0.221852] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'707worker2 # [ 0.213955] Speculative Return Stack Overflow: Mitigation: Safe RET708lb # [ 0.232563] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256709client # [ 0.222852] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'710lb # [ 0.233563] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64711worker1 # [ 0.214523] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization712client # [ 0.223851] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'713lb # [ 0.234563] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512714worker1 # [ 0.216532] Spectre V2 : Enabling IBPB for BPF715client # [ 0.224852] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'716lb # [ 0.235563] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024717worker1 # [ 0.217524] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier718client # [ 0.225851] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'719worker2 # [ 0.214955] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization720lb # [ 0.236563] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8721worker1 # [ 0.218524] active return thunk: srso_alias_return_thunk722worker2 # [ 0.216964] Spectre V2 : Enabling IBPB for BPF723client # [ 0.226852] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'724lb # [ 0.237563] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.725worker1 # [ 0.219545] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'726client # [ 0.227852] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256727worker2 # [ 0.217956] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier728worker1 # [ 0.220523] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'729worker2 # [ 0.218955] active return thunk: srso_alias_return_thunk730client # [ 0.228852] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64731worker1 # [ 0.221523] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'732client # [ 0.229851] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512733worker2 # [ 0.219977] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'734worker1 # [ 0.222523] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'735client # [ 0.230852] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024736worker2 # [ 0.220955] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'737worker1 # [ 0.223523] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'738client # [ 0.231852] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8739worker2 # [ 0.221955] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'740worker1 # [ 0.224523] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'741worker2 # [ 0.222955] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'742client # [ 0.232852] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.743worker1 # [ 0.225523] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'744worker2 # [ 0.223958] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'745worker1 # [ 0.226524] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256746worker2 # [ 0.224956] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'747worker1 # [ 0.227523] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64748worker1 # [ 0.228523] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512749worker2 # [ 0.225956] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'750worker1 # [ 0.229523] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024751worker2 # [ 0.227956] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256752worker1 # [ 0.230524] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8753worker2 # [ 0.228956] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64754worker1 # [ 0.231523] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.755worker2 # [ 0.229956] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512756worker2 # [ 0.231956] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024757worker2 # [ 0.232956] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8758worker2 # [ 0.233956] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759lb # [ 0.273009] Freeing SMP alternatives memory: 44K760lb # [ 0.273565] pid_max: default: 32768 minimum: 301761lb # [ 0.274654] LSM: initializing lsm=capability,landlock,yama,bpf,ima762lb # [ 0.275680] landlock: Up and running.763lb # [ 0.276295] Yama: becoming mindful.764lb # [ 0.277208] LSM support for eBPF active765lb # [ 0.277678] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)766client # [ 0.268387] Freeing SMP alternatives memory: 44K767client # [ 0.268855] pid_max: default: 32768 minimum: 301768lb # [ 0.278601] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)769client # [ 0.269945] LSM: initializing lsm=capability,landlock,yama,bpf,ima770client # [ 0.270974] landlock: Up and running.771client # [ 0.271852] Yama: becoming mindful.772lb # [ 0.281483] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)773client # [ 0.272692] LSM support for eBPF active774lb # [ 0.282108] Performance Events: Fam17h+ core perfctr, AMD PMU driver.775client # [ 0.273458] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)776lb # [ 0.282567] ... version: 2777lb # [ 0.283266] ... bit width: 48778worker1 # [ 0.265850] Freeing SMP alternatives memory: 44K779client # [ 0.273889] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)780lb # [ 0.283566] ... generic counters: 6781worker1 # [ 0.266527] pid_max: default: 32768 minimum: 301782lb # [ 0.284269] ... generic bitmap: 000000000000003f783worker1 # [ 0.267357] LSM: initializing lsm=capability,landlock,yama,bpf,ima784lb # [ 0.284575] ... fixed-purpose counters: 0785worker1 # [ 0.268600] landlock: Up and running.786client # [ 0.276996] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)787lb # [ 0.285264] ... fixed-purpose bitmap: 0000000000000000788worker1 # [ 0.269202] Yama: becoming mindful.789lb # [ 0.285565] ... value mask: 0000ffffffffffff790worker1 # [ 0.269739] LSM support for eBPF active791client # [ 0.278416] Performance Events: Fam17h+ core perfctr, AMD PMU driver.792lb # [ 0.286446] ... max period: 00007fffffffffff793client # [ 0.278861] ... version: 2794worker1 # [ 0.270638] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)795client # [ 0.279553] ... bit width: 48796lb # [ 0.287231] ... global_ctrl mask: 000000000000003f797client # [ 0.279854] ... generic counters: 6798lb # [ 0.287695] signal: max sigframe size: 3376799worker1 # [ 0.271568] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)800client # [ 0.280563] ... generic bitmap: 000000000000003f801lb # [ 0.288443] rcu: Hierarchical SRCU implementation.802client # [ 0.280854] ... fixed-purpose counters: 0803lb # [ 0.289136] rcu: Max phase no-delay instances is 400.804client # [ 0.281569] ... fixed-purpose bitmap: 0000000000000000805worker1 # [ 0.274383] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)806client # [ 0.281854] ... value mask: 0000ffffffffffff807client # [ 0.282760] ... max period: 00007fffffffffff808worker1 # [ 0.275077] Performance Events: Fam17h+ core perfctr, AMD PMU driver.809worker2 # [ 0.276374] Freeing SMP alternatives memory: 44K810client # [ 0.283539] ... global_ctrl mask: 000000000000003f811worker1 # [ 0.275528] ... version: 2812worker2 # [ 0.276958] pid_max: default: 32768 minimum: 301813client # [ 0.283952] signal: max sigframe size: 3376814worker1 # [ 0.276194] ... bit width: 48815worker1 # [ 0.276526] ... generic counters: 6816client # [ 0.284734] rcu: Hierarchical SRCU implementation.817worker2 # [ 0.277986] LSM: initializing lsm=capability,landlock,yama,bpf,ima818lb # [ 0.294126] smp: Bringing up secondary CPUs ...819worker1 # [ 0.277246] ... generic bitmap: 000000000000003f820worker2 # [ 0.279068] landlock: Up and running.821client # [ 0.285432] rcu: Max phase no-delay instances is 400.822lb # [ 0.294574] smp: Brought up 1 node, 1 CPU823worker2 # [ 0.279667] Yama: becoming mindful.824worker1 # [ 0.277536] ... fixed-purpose counters: 0825lb # [ 0.295217] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)826worker2 # [ 0.280179] LSM support for eBPF active827worker1 # [ 0.278207] ... fixed-purpose bitmap: 0000000000000000828worker1 # [ 0.278527] ... value mask: 0000ffffffffffff829worker2 # [ 0.281061] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)830worker1 # [ 0.279437] ... max period: 00007fffffffffff831lb # [ 0.295814] Memory: 1454564K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)832worker2 # [ 0.281995] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)833worker1 # [ 0.280206] ... global_ctrl mask: 000000000000003f834lb # [ 0.296825] devtmpfs: initialized835worker1 # [ 0.280640] signal: max sigframe size: 3376836lb # [ 0.297634] x86/mm: Memory block size: 128MB837client # [ 0.290502] smp: Bringing up secondary CPUs ...838worker1 # [ 0.281465] rcu: Hierarchical SRCU implementation.839client # [ 0.290868] smp: Brought up 1 node, 1 CPU840worker2 # [ 0.284713] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)841worker1 # [ 0.282101] rcu: Max phase no-delay instances is 400.842client # [ 0.291520] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)843lb # [ 0.299384] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)844worker2 # [ 0.285510] Performance Events: Fam17h+ core perfctr, AMD PMU driver.845worker2 # [ 0.285959] ... version: 2846lb # [ 0.300452] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).847client # [ 0.292062] Memory: 1454560K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)848worker2 # [ 0.286644] ... bit width: 48849lb # [ 0.301650] pinctrl core: initialized pinctrl subsystem850worker2 # [ 0.286957] ... generic counters: 6851client # [ 0.293141] devtmpfs: initialized852lb # [ 0.302742] PM: RTC time: 10:44:01, date: 2026-09-22853worker2 # [ 0.287648] ... generic bitmap: 000000000000003f854client # [ 0.293954] x86/mm: Memory block size: 128MB855worker2 # [ 0.287968] ... fixed-purpose counters: 0856worker1 # [ 0.287137] smp: Bringing up secondary CPUs ...857worker2 # [ 0.288666] ... fixed-purpose bitmap: 0000000000000000858worker1 # [ 0.287541] smp: Brought up 1 node, 1 CPU859worker2 # [ 0.288958] ... value mask: 0000ffffffffffff860client # [ 0.295730] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)861worker1 # [ 0.288219] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)862worker2 # [ 0.289833] ... max period: 00007fffffffffff863lb # [ 0.306349] NET: Registered PF_NETLINK/PF_ROUTE protocol family864client # [ 0.296790] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).865worker2 # [ 0.290626] ... global_ctrl mask: 000000000000003f866client # [ 0.297968] pinctrl core: initialized pinctrl subsystem867worker2 # [ 0.291072] signal: max sigframe size: 3376868worker1 # [ 0.288768] Memory: 1454548K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)869lb # [ 0.307775] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations870worker2 # [ 0.291872] rcu: Hierarchical SRCU implementation.871client # [ 0.299086] PM: RTC time: 10:44:01, date: 2026-09-22872worker1 # [ 0.289791] devtmpfs: initialized873lb # [ 0.308594] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations874worker2 # [ 0.292524] rcu: Max phase no-delay instances is 400.875worker1 # [ 0.290571] x86/mm: Memory block size: 128MB876lb # [ 0.309860] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations877lb # [ 0.310576] audit: initializing netlink subsys (disabled)878worker1 # [ 0.292275] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)879client # [ 0.302750] NET: Registered PF_NETLINK/PF_ROUTE protocol family880lb # [ 0.311753] thermal_sys: Registered thermal governor 'fair_share'881worker1 # [ 0.293453] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).882lb # [ 0.311756] thermal_sys: Registered thermal governor 'bang_bang'883client # [ 0.304055] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations884worker1 # [ 0.294606] pinctrl core: initialized pinctrl subsystem885worker2 # [ 0.297493] smp: Bringing up secondary CPUs ...886lb # [ 0.312566] thermal_sys: Registered thermal governor 'step_wise'887worker1 # [ 0.295673] PM: RTC time: 10:44:01, date: 2026-09-22888client # [ 0.304879] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations889worker2 # [ 0.297972] smp: Brought up 1 node, 1 CPU890lb # [ 0.313493] thermal_sys: Registered thermal governor 'user_space'891worker2 # [ 0.298616] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)892client # [ 0.306137] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations893lb # [ 0.314278] audit: type=2000 audit(1790073841.942:1): state=initialized audit_enabled=0 res=1894client # [ 0.306864] audit: initializing netlink subsys (disabled)895lb # [ 0.315568] thermal_sys: Registered thermal governor 'power_allocator'896worker2 # [ 0.299104] Memory: 1454560K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)897lb # [ 0.315583] cpuidle: using governor menu898client # [ 0.308020] thermal_sys: Registered thermal governor 'fair_share'899worker1 # [ 0.299407] NET: Registered PF_NETLINK/PF_ROUTE protocol family900worker2 # [ 0.300239] devtmpfs: initialized901client # [ 0.308022] thermal_sys: Registered thermal governor 'bang_bang'902lb # [ 0.317763] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5903worker2 # [ 0.301012] x86/mm: Memory block size: 128MB904worker1 # [ 0.300717] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations905client # [ 0.308856] thermal_sys: Registered thermal governor 'step_wise'906worker1 # [ 0.301553] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations907client # [ 0.309822] thermal_sys: Registered thermal governor 'user_space'908lb # [ 0.318852] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]909worker2 # [ 0.302806] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)910lb # [ 0.319569] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry911client # [ 0.310600] audit: type=2000 audit(1790073841.951:1): state=initialized audit_enabled=0 res=1912worker1 # [ 0.302803] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations913lb # [ 0.320577] PCI: Using configuration type 1 for base access914worker2 # [ 0.303830] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).915client # [ 0.311857] thermal_sys: Registered thermal governor 'power_allocator'916worker1 # [ 0.303536] audit: initializing netlink subsys (disabled)917client # [ 0.311871] cpuidle: using governor menu918worker2 # [ 0.305025] pinctrl core: initialized pinctrl subsystem919lb # [ 0.321635] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.920worker1 # [ 0.304642] thermal_sys: Registered thermal governor 'fair_share'921worker2 # [ 0.306114] PM: RTC time: 10:44:01, date: 2026-09-22922worker1 # [ 0.304644] thermal_sys: Registered thermal governor 'bang_bang'923client # [ 0.313909] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5924worker1 # [ 0.305527] thermal_sys: Registered thermal governor 'step_wise'925client # [ 0.315104] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]926worker1 # [ 0.306462] thermal_sys: Registered thermal governor 'user_space'927client # [ 0.315857] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry928worker1 # [ 0.307281] audit: type=2000 audit(1790073841.957:1): state=initialized audit_enabled=0 res=1929worker2 # [ 0.309779] NET: Registered PF_NETLINK/PF_ROUTE protocol family930client # [ 0.316866] PCI: Using configuration type 1 for base access931worker1 # [ 0.308528] thermal_sys: Registered thermal governor 'power_allocator'932worker2 # [ 0.311139] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations933worker1 # [ 0.308542] cpuidle: using governor menu934client # [ 0.317970] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.935worker2 # [ 0.311982] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations936lb # [ 0.328889] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages937worker1 # [ 0.310736] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5938lb # [ 0.329566] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page939worker2 # [ 0.313231] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations940worker1 # [ 0.311759] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]941lb # [ 0.330566] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages942worker2 # [ 0.313969] audit: initializing netlink subsys (disabled)943worker1 # [ 0.312529] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry944worker2 # [ 0.315098] thermal_sys: Registered thermal governor 'fair_share'945worker1 # [ 0.313538] PCI: Using configuration type 1 for base access946client # [ 0.323088] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages947worker2 # [ 0.315101] thermal_sys: Registered thermal governor 'bang_bang'948client # [ 0.323856] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page949worker2 # [ 0.315958] thermal_sys: Registered thermal governor 'step_wise'950worker1 # [ 0.314638] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.951worker2 # [ 0.316891] thermal_sys: Registered thermal governor 'user_space'952worker2 # [ 0.317666] audit: type=2000 audit(1790073841.969:1): state=initialized audit_enabled=0 res=1953worker2 # [ 0.318960] thermal_sys: Registered thermal governor 'power_allocator'954lb # [ 0.337567] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page955worker2 # [ 0.318983] cpuidle: using governor menu956client # [ 0.328855] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages957worker2 # [ 0.321168] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5958client # [ 0.329854] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page959worker2 # [ 0.322236] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]960worker1 # [ 0.321773] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages961worker2 # [ 0.322961] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry962worker1 # [ 0.322527] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page963worker2 # [ 0.323970] PCI: Using configuration type 1 for base access964lb # [ 0.343985] ACPI: Added _OSI(Module Device)965lb # [ 0.344566] ACPI: Added _OSI(Processor Device)966worker2 # [ 0.325083] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.967lb # [ 0.345310] ACPI: Added _OSI(Processor Aggregator Device)968worker1 # [ 0.327527] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages969worker1 # [ 0.328527] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page970client # [ 0.340098] ACPI: Added _OSI(Module Device)971client # [ 0.340856] ACPI: Added _OSI(Processor Device)972client # [ 0.341611] ACPI: Added _OSI(Processor Aggregator Device)973worker2 # [ 0.332241] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages974worker2 # [ 0.332960] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page975lb # [ 0.352160] ACPI: 1 ACPI AML tables successfully acquired and loaded976worker1 # [ 0.337153] ACPI: Added _OSI(Module Device)977lb # [ 0.355975] ACPI: Interpreter enabled978worker1 # [ 0.337528] ACPI: Added _OSI(Processor Device)979lb # [ 0.356581] ACPI: PM: (supports S0 S3 S4 S5)980worker2 # [ 0.337959] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages981worker1 # [ 0.338285] ACPI: Added _OSI(Processor Aggregator Device)982lb # [ 0.357281] ACPI: Using IOAPIC for interrupt routing983worker2 # [ 0.338958] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page984client # [ 0.350167] ACPI: 1 ACPI AML tables successfully acquired and loaded985lb # [ 0.359629] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug986lb # [ 0.360567] PCI: Using E820 reservations for host bridge windows987client # [ 0.354266] ACPI: Interpreter enabled988client # [ 0.354870] ACPI: PM: (supports S0 S3 S4 S5)989lb # [ 0.363676] ACPI: Enabled 2 GPEs in block 00 to 3F990worker1 # [ 0.345090] ACPI: 1 ACPI AML tables successfully acquired and loaded991client # [ 0.355572] ACPI: Using IOAPIC for interrupt routing992worker2 # [ 0.347567] ACPI: Added _OSI(Module Device)993worker2 # [ 0.347959] ACPI: Added _OSI(Processor Device)994client # [ 0.355887] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug995worker2 # [ 0.348662] ACPI: Added _OSI(Processor Aggregator Device)996client # [ 0.356855] PCI: Using E820 reservations for host bridge windows997worker1 # [ 0.348483] ACPI: Interpreter enabled998worker1 # [ 0.349028] ACPI: PM: (supports S0 S3 S4 S5)999worker1 # [ 0.349527] ACPI: Using IOAPIC for interrupt routing1000client # [ 0.359984] ACPI: Enabled 2 GPEs in block 00 to 3F1001worker1 # [ 0.350407] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1002lb # [ 0.372317] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1003lb # [ 0.372571] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1004worker2 # [ 0.355643] ACPI: 1 ACPI AML tables successfully acquired and loaded1005worker1 # [ 0.355526] PCI: Using E820 reservations for host bridge windows1006lb # [ 0.373940] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1007worker1 # [ 0.356617] ACPI: Enabled 2 GPEs in block 00 to 3F1008lb # [ 0.374685] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1009lb # [ 0.375904] PCI host bridge to bus 0000:001010worker2 # [ 0.358868] ACPI: Interpreter enabled1011client # [ 0.368613] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1012worker2 # [ 0.359414] ACPI: PM: (supports S0 S3 S4 S5)1013lb # [ 0.376575] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1014worker2 # [ 0.359959] ACPI: Using IOAPIC for interrupt routing1015lb # [ 0.377565] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1016client # [ 0.368863] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1017worker2 # [ 0.360810] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1018lb # [ 0.378565] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1019client # [ 0.370932] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1020lb # [ 0.379565] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1021client # [ 0.371978] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1022lb # [ 0.380565] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1023client # [ 0.373203] PCI host bridge to bus 0000:001024lb # [ 0.381565] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1025worker1 # [ 0.365247] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1026client # [ 0.373859] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1027worker2 # [ 0.365958] PCI: Using E820 reservations for host bridge windows1028lb # [ 0.382566] pci_bus 0000:00: root bus resource [bus 00-ff]1029worker2 # [ 0.367013] ACPI: Enabled 2 GPEs in block 00 to 3F1030client # [ 0.374855] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1031worker1 # [ 0.366144] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1032lb # [ 0.383550] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1033client # [ 0.375855] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1034worker1 # [ 0.366605] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1035lb # [ 0.384965] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1036client # [ 0.376854] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1037worker1 # [ 0.367650] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1038client # [ 0.377854] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1039worker1 # [ 0.368919] PCI host bridge to bus 0000:001040client # [ 0.378862] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1041worker1 # [ 0.369532] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1042client # [ 0.379885] pci_bus 0000:00: root bus resource [bus 00-ff]1043worker1 # [ 0.370526] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1044client # [ 0.380918] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1045worker1 # [ 0.371526] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1046worker1 # [ 0.372526] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1047worker2 # [ 0.375644] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1048client # [ 0.382294] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1049worker1 # [ 0.373526] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1050worker2 # [ 0.376477] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1051worker1 # [ 0.374526] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1052lb # [ 0.387649] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1053worker1 # [ 0.375526] pci_bus 0000:00: root bus resource [bus 00-ff]1054worker2 # [ 0.377037] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1055lb # [ 0.388578] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1056worker2 # [ 0.378090] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1057lb # [ 0.389531] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1058worker1 # [ 0.376506] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1059worker2 # [ 0.379355] PCI host bridge to bus 0000:001060lb # [ 0.390808] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1061worker2 # [ 0.379962] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1062worker1 # [ 0.377963] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1063worker2 # [ 0.380958] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1064lb # [ 0.392484] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1065worker2 # [ 0.381958] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1066client # [ 0.384920] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1067worker2 # [ 0.382958] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1068client # [ 0.385868] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1069worker2 # [ 0.383958] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1070client # [ 0.386841] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1071lb # [ 0.395597] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1072worker2 # [ 0.384957] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1073lb # [ 0.396446] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1074client # [ 0.388032] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1075worker2 # [ 0.385958] pci_bus 0000:00: root bus resource [bus 00-ff]1076lb # [ 0.397321] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1077worker2 # [ 0.386904] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1078lb # [ 0.397571] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1079client # [ 0.389619] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1080worker1 # [ 0.380599] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1081worker2 # [ 0.388359] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1082worker1 # [ 0.381539] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1083lb # [ 0.399134] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1084worker1 # [ 0.382503] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1085client # [ 0.392862] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1086client # [ 0.393696] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1087worker1 # [ 0.383711] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1088lb # [ 0.400598] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1089client # [ 0.394606] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1090lb # [ 0.401443] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1091client # [ 0.394860] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1092worker1 # [ 0.385256] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1093lb # [ 0.402348] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1094client # [ 0.396626] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1095lb # [ 0.403968] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1096worker1 # [ 0.387544] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1097worker1 # [ 0.388432] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1098worker2 # [ 0.391023] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1099client # [ 0.399661] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1100worker1 # [ 0.389293] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1101worker2 # [ 0.391970] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1102client # [ 0.400484] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1103worker1 # [ 0.389532] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1104lb # [ 0.406572] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1105worker2 # [ 0.392890] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1106client # [ 0.400876] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1107lb # [ 0.408169] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1108worker2 # [ 0.394338] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1109client # [ 0.402408] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1110worker1 # [ 0.391265] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1111lb # [ 0.408587] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1112worker2 # [ 0.395686] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1113lb # [ 0.410221] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1114worker1 # [ 0.393555] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1115worker1 # [ 0.394394] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1116client # [ 0.404863] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1117worker1 # [ 0.395311] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1118lb # [ 0.411598] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1119worker2 # [ 0.398966] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1120client # [ 0.405861] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1121lb # [ 0.412458] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1122worker2 # [ 0.399801] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1123client # [ 0.406789] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1124worker1 # [ 0.396945] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1125lb # [ 0.413317] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1126worker2 # [ 0.400683] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1127client # [ 0.408231] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1128worker2 # [ 0.400964] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1129lb # [ 0.414134] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1130worker2 # [ 0.402522] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1131worker1 # [ 0.400102] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1132client # [ 0.410645] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1133client # [ 0.411518] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1134worker1 # [ 0.400532] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1135lb # [ 0.416573] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1136lb # [ 0.417411] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1137client # [ 0.411897] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1138worker2 # [ 0.403966] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1139worker1 # [ 0.401548] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1140worker2 # [ 0.404830] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1141lb # [ 0.418291] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1142worker1 # [ 0.403071] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1143worker2 # [ 0.405706] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1144client # [ 0.413423] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1145lb # [ 0.419158] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1146worker2 # [ 0.406531] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1147worker1 # [ 0.404551] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1148client # [ 0.415862] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1149worker1 # [ 0.405459] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1150client # [ 0.416707] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1151worker1 # [ 0.406301] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1152client # [ 0.417613] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1153lb # [ 0.421598] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1154worker2 # [ 0.408965] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1155lb # [ 0.422572] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1156worker2 # [ 0.409810] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1157worker1 # [ 0.407949] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1158client # [ 0.419292] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1159lb # [ 0.423555] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1160worker2 # [ 0.410656] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1161worker1 # [ 0.409554] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1162lb # [ 0.424949] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1163worker2 # [ 0.411504] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1164worker1 # [ 0.410371] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1165client # [ 0.421886] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1166worker1 # [ 0.411260] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1167lb # [ 0.426573] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1168client # [ 0.422862] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1169worker2 # [ 0.412965] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1170lb # [ 0.427410] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1171client # [ 0.423836] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1172worker1 # [ 0.412108] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1173worker2 # [ 0.413781] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1174lb # [ 0.428296] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1175lb # [ 0.428572] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1176client # [ 0.425266] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1177worker2 # [ 0.414667] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1178worker1 # [ 0.414460] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1179worker2 # [ 0.415510] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1180client # [ 0.426862] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1181lb # [ 0.430868] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1182worker1 # [ 0.415181] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1183client # [ 0.427673] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1184worker1 # [ 0.416261] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1185worker2 # [ 0.416965] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1186client # [ 0.428621] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1187lb # [ 0.432577] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1188worker2 # [ 0.417780] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1189client # [ 0.429698] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1190lb # [ 0.433523] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1191worker1 # [ 0.417149] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1192worker2 # [ 0.418662] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1193lb # [ 0.434899] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1194worker1 # [ 0.418550] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1195client # [ 0.431631] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1196worker2 # [ 0.419531] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1197worker1 # [ 0.419395] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1198worker1 # [ 0.420299] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1199client # [ 0.433856] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1200lb # [ 0.436603] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1201worker1 # [ 0.421396] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1202client # [ 0.434848] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1203lb # [ 0.437439] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1204worker2 # [ 0.422182] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1205client # [ 0.436220] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1206lb # [ 0.438576] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1207worker1 # [ 0.422827] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1208worker2 # [ 0.422965] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1209worker2 # [ 0.423889] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1210lb # [ 0.440149] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1211worker1 # [ 0.424356] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1212client # [ 0.437903] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1213worker1 # [ 0.425313] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1214worker2 # [ 0.425432] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1215client # [ 0.438854] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1216lb # [ 0.441591] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1217client # [ 0.439903] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1218worker1 # [ 0.426933] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1219lb # [ 0.442452] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1220worker2 # [ 0.426965] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1221lb # [ 0.443269] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1222worker2 # [ 0.427843] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1223client # [ 0.441406] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1224worker1 # [ 0.428535] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1225worker2 # [ 0.428691] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1226lb # [ 0.444151] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1227worker1 # [ 0.429357] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1228client # [ 0.442886] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1229worker2 # [ 0.428963] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1230worker1 # [ 0.430257] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1231client # [ 0.443757] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1232lb # [ 0.445262] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1233worker2 # [ 0.430511] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1234client # [ 0.444648] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1235worker1 # [ 0.431594] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1236lb # [ 0.446448] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1237worker2 # [ 0.431972] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1238client # [ 0.446278] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1239lb # [ 0.448306] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1240worker1 # [ 0.433556] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1241worker2 # [ 0.432927] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1242worker1 # [ 0.434410] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1243client # [ 0.447517] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1244worker1 # [ 0.435295] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1245worker2 # [ 0.434323] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1246client # [ 0.448735] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1247worker1 # [ 0.436940] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1248client # [ 0.450548] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1249worker2 # [ 0.435966] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1250worker1 # [ 0.438066] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1251worker2 # [ 0.436803] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1252worker2 # [ 0.437980] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1253worker1 # [ 0.438731] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1254worker1 # [ 0.440102] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1255worker2 # [ 0.439504] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1256worker2 # [ 0.440966] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1257lb # [ 0.448758] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1258worker2 # [ 0.441849] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1259worker2 # [ 0.442710] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1260lb # [ 0.451220] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1261worker2 # [ 0.443537] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1262lb # [ 0.451805] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1263client # [ 0.451068] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1264worker2 # [ 0.444986] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1265lb # [ 0.453152] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1266client # [ 0.453510] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1267worker2 # [ 0.446040] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1268lb # [ 0.454572] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1269client # [ 0.454088] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1270worker2 # [ 0.447587] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1271lb # [ 0.455875] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1272worker1 # [ 0.440719] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1273client # [ 0.455444] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1274lb # [ 0.456814] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1275worker1 # [ 0.443100] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1276client # [ 0.456870] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1277lb # [ 0.458582] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1278worker1 # [ 0.443754] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1279lb # [ 0.459405] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1280client # [ 0.458173] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1281worker1 # [ 0.445033] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1282client # [ 0.459108] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1283lb # [ 0.460639] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1284worker1 # [ 0.445809] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1285client # [ 0.460877] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1286lb # [ 0.462479] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1287worker1 # [ 0.446814] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1288client # [ 0.461736] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1289worker1 # [ 0.447772] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1290client # [ 0.462914] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1291lb # [ 0.466226] ACPI: PCI: Interrupt link LNKA configured for IRQ 101292worker1 # [ 0.449355] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1293worker2 # [ 0.448158] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1294client # [ 0.464515] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1295worker1 # [ 0.450172] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1296lb # [ 0.466669] ACPI: PCI: Interrupt link LNKB configured for IRQ 101297worker2 # [ 0.450562] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1298lb # [ 0.467667] ACPI: PCI: Interrupt link LNKC configured for IRQ 111299worker1 # [ 0.450842] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1300lb # [ 0.468668] ACPI: PCI: Interrupt link LNKD configured for IRQ 111301worker2 # [ 0.451195] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1302worker1 # [ 0.452079] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1303client # [ 0.467830] ACPI: PCI: Interrupt link LNKA configured for IRQ 101304lb # [ 0.469665] ACPI: PCI: Interrupt link LNKE configured for IRQ 101305worker2 # [ 0.452540] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1306client # [ 0.468698] ACPI: PCI: Interrupt link LNKB configured for IRQ 101307lb # [ 0.470660] ACPI: PCI: Interrupt link LNKF configured for IRQ 101308client # [ 0.469685] ACPI: PCI: Interrupt link LNKC configured for IRQ 111309lb # [ 0.471691] ACPI: PCI: Interrupt link LNKG configured for IRQ 111310worker2 # [ 0.453937] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1311client # [ 0.470697] ACPI: PCI: Interrupt link LNKD configured for IRQ 111312lb # [ 0.472662] ACPI: PCI: Interrupt link LNKH configured for IRQ 111313worker2 # [ 0.455222] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1314worker1 # [ 0.455914] ACPI: PCI: Interrupt link LNKA configured for IRQ 101315client # [ 0.471724] ACPI: PCI: Interrupt link LNKE configured for IRQ 101316lb # [ 0.473601] ACPI: PCI: Interrupt link GSIA configured for IRQ 161317worker1 # [ 0.456633] ACPI: PCI: Interrupt link LNKB configured for IRQ 101318lb # [ 0.474535] ACPI: PCI: Interrupt link GSIB configured for IRQ 171319client # [ 0.472660] ACPI: PCI: Interrupt link LNKF configured for IRQ 101320worker2 # [ 0.456208] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1321lb # [ 0.475288] ACPI: PCI: Interrupt link GSIC configured for IRQ 181322worker1 # [ 0.457628] ACPI: PCI: Interrupt link LNKC configured for IRQ 111323client # [ 0.473664] ACPI: PCI: Interrupt link LNKG configured for IRQ 111324lb # [ 0.475579] ACPI: PCI: Interrupt link GSID configured for IRQ 191325worker1 # [ 0.458627] ACPI: PCI: Interrupt link LNKD configured for IRQ 111326worker2 # [ 0.457963] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1327client # [ 0.474707] ACPI: PCI: Interrupt link LNKH configured for IRQ 111328lb # [ 0.476547] ACPI: PCI: Interrupt link GSIE configured for IRQ 201329worker1 # [ 0.459626] ACPI: PCI: Interrupt link LNKE configured for IRQ 101330worker2 # [ 0.458778] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1331client # [ 0.475567] ACPI: PCI: Interrupt link GSIA configured for IRQ 161332lb # [ 0.477264] ACPI: PCI: Interrupt link GSIF configured for IRQ 211333worker1 # [ 0.460623] ACPI: PCI: Interrupt link LNKF configured for IRQ 101334client # [ 0.475868] ACPI: PCI: Interrupt link GSIB configured for IRQ 171335worker2 # [ 0.459959] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1336lb # [ 0.477579] ACPI: PCI: Interrupt link GSIG configured for IRQ 221337client # [ 0.476820] ACPI: PCI: Interrupt link GSIC configured for IRQ 181338worker1 # [ 0.461627] ACPI: PCI: Interrupt link LNKG configured for IRQ 111339lb # [ 0.478502] ACPI: PCI: Interrupt link GSIH configured for IRQ 231340client # [ 0.477584] ACPI: PCI: Interrupt link GSID configured for IRQ 191341worker1 # [ 0.462624] ACPI: PCI: Interrupt link LNKH configured for IRQ 111342worker2 # [ 0.461987] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1343client # [ 0.477870] ACPI: PCI: Interrupt link GSIE configured for IRQ 201344worker1 # [ 0.463561] ACPI: PCI: Interrupt link GSIA configured for IRQ 161345lb # [ 0.480541] iommu: Default domain type: Translated1346client # [ 0.478850] ACPI: PCI: Interrupt link GSIF configured for IRQ 211347worker1 # [ 0.464544] ACPI: PCI: Interrupt link GSIB configured for IRQ 171348lb # [ 0.481141] iommu: DMA domain TLB invalidation policy: lazy mode1349client # [ 0.479813] ACPI: PCI: Interrupt link GSIG configured for IRQ 221350worker1 # [ 0.465513] ACPI: PCI: Interrupt link GSIC configured for IRQ 181351lb # [ 0.481823] ACPI: bus type USB registered1352client # [ 0.480595] ACPI: PCI: Interrupt link GSIH configured for IRQ 231353worker1 # [ 0.466256] ACPI: PCI: Interrupt link GSID configured for IRQ 191354lb # [ 0.482556] usbcore: registered new interface driver usbfs1355worker2 # [ 0.465784] ACPI: PCI: Interrupt link LNKA configured for IRQ 101356worker1 # [ 0.466548] ACPI: PCI: Interrupt link GSIE configured for IRQ 201357lb # [ 0.483461] usbcore: registered new interface driver hub1358worker2 # [ 0.466768] ACPI: PCI: Interrupt link LNKB configured for IRQ 101359client # [ 0.482016] iommu: Default domain type: Translated1360worker1 # [ 0.467478] ACPI: PCI: Interrupt link GSIF configured for IRQ 211361lb # [ 0.484264] usbcore: registered new device driver usb1362worker2 # [ 0.467760] ACPI: PCI: Interrupt link LNKC configured for IRQ 111363client # [ 0.482814] iommu: DMA domain TLB invalidation policy: lazy mode1364worker1 # [ 0.468227] ACPI: PCI: Interrupt link GSIG configured for IRQ 221365worker2 # [ 0.468756] ACPI: PCI: Interrupt link LNKD configured for IRQ 111366lb # [ 0.485469] NetLabel: Initializing1367worker1 # [ 0.468543] ACPI: PCI: Interrupt link GSIH configured for IRQ 231368client # [ 0.484010] ACPI: bus type USB registered1369lb # [ 0.485963] NetLabel: domain hash size = 1281370worker2 # [ 0.469788] ACPI: PCI: Interrupt link LNKE configured for IRQ 101371client # [ 0.484717] usbcore: registered new interface driver usbfs1372lb # [ 0.486565] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1373client # [ 0.485545] usbcore: registered new interface driver hub1374worker1 # [ 0.470657] iommu: Default domain type: Translated1375worker2 # [ 0.470830] ACPI: PCI: Interrupt link LNKF configured for IRQ 101376lb # [ 0.487592] NetLabel: unlabeled traffic allowed by default1377client # [ 0.485865] usbcore: registered new device driver usb1378worker1 # [ 0.471406] iommu: DMA domain TLB invalidation policy: lazy mode1379worker2 # [ 0.471977] ACPI: PCI: Interrupt link LNKG configured for IRQ 111380lb # [ 0.488531] PCI: Using ACPI for IRQ routing1381worker1 # [ 0.472497] ACPI: bus type USB registered1382worker2 # [ 0.472999] ACPI: PCI: Interrupt link LNKH configured for IRQ 111383client # [ 0.487573] NetLabel: Initializing1384worker1 # [ 0.473070] usbcore: registered new interface driver usbfs1385client # [ 0.487855] NetLabel: domain hash size = 1281386worker2 # [ 0.473991] ACPI: PCI: Interrupt link GSIA configured for IRQ 161387worker1 # [ 0.473541] usbcore: registered new interface driver hub1388client # [ 0.488639] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1389worker2 # [ 0.474943] ACPI: PCI: Interrupt link GSIB configured for IRQ 171390worker1 # [ 0.474423] usbcore: registered new device driver usb1391worker2 # [ 0.475675] ACPI: PCI: Interrupt link GSIC configured for IRQ 181392client # [ 0.488913] NetLabel: unlabeled traffic allowed by default1393worker2 # [ 0.475972] ACPI: PCI: Interrupt link GSID configured for IRQ 191394worker1 # [ 0.475997] NetLabel: Initializing1395worker2 # [ 0.476968] ACPI: PCI: Interrupt link GSIE configured for IRQ 201396client # [ 0.489867] PCI: Using ACPI for IRQ routing1397worker2 # [ 0.477966] ACPI: PCI: Interrupt link GSIF configured for IRQ 211398worker1 # [ 0.476527] NetLabel: domain hash size = 1281399worker2 # [ 0.478947] ACPI: PCI: Interrupt link GSIG configured for IRQ 221400worker1 # [ 0.478044] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1401worker2 # [ 0.479735] ACPI: PCI: Interrupt link GSIH configured for IRQ 231402worker1 # [ 0.478565] NetLabel: unlabeled traffic allowed by default1403worker1 # [ 0.479458] PCI: Using ACPI for IRQ routing1404worker2 # [ 0.481913] iommu: Default domain type: Translated1405worker2 # [ 0.482577] iommu: DMA domain TLB invalidation policy: lazy mode1406worker2 # [ 0.483247] ACPI: bus type USB registered1407worker2 # [ 0.484022] usbcore: registered new interface driver usbfs1408worker2 # [ 0.484975] usbcore: registered new interface driver hub1409worker2 # [ 0.485875] usbcore: registered new device driver usb1410worker2 # [ 0.487592] NetLabel: Initializing1411worker2 # [ 0.487958] NetLabel: domain hash size = 1281412worker2 # [ 0.488708] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413worker2 # [ 0.489012] NetLabel: unlabeled traffic allowed by default1414worker2 # [ 0.489973] PCI: Using ACPI for IRQ routing1415lb # [ 0.532889] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416lb # [ 0.533561] pci 0000:00:01.0: vgaarb: bridge control possible1417lb # [ 0.533561] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418lb # [ 0.533570] vgaarb: loaded1419lb # [ 0.534299] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01420client # [ 0.534804] pci 0000:00:01.0: vgaarb: setting as boot VGA device1421lb # [ 0.534567] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1422client # [ 0.534851] pci 0000:00:01.0: vgaarb: bridge control possible1423client # [ 0.534851] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1424client # [ 0.534860] vgaarb: loaded1425worker1 # [ 0.523312] pci 0000:00:01.0: vgaarb: setting as boot VGA device1426client # [ 0.535631] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01427worker1 # [ 0.523522] pci 0000:00:01.0: vgaarb: bridge control possible1428client # [ 0.535855] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1429lb # [ 0.538655] clocksource: Switched to clocksource kvm-clock1430worker1 # [ 0.523522] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1431worker1 # [ 0.523531] vgaarb: loaded1432lb # [ 0.540325] VFS: Disk quotas dquot_6.6.01433worker1 # [ 0.524314] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01434worker1 # [ 0.524528] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1435lb # [ 0.542896] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1436client # [ 0.539945] clocksource: Switched to clocksource kvm-clock1437lb # [ 0.544275] pnp: PnP ACPI init1438lb # [ 0.545126] ACPI: IRQ 4 override to edge(!), high(!)1439lb # [ 0.546135] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1440client # [ 0.543504] VFS: Disk quotas dquot_6.6.01441worker1 # [ 0.529615] clocksource: Switched to clocksource kvm-clock1442lb # [ 0.547737] pnp: PnP ACPI: found 6 devices1443client # [ 0.544200] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1444client # [ 0.545542] pnp: PnP ACPI init1445worker2 # [ 0.533335] pci 0000:00:01.0: vgaarb: setting as boot VGA device1446worker2 # [ 0.533954] pci 0000:00:01.0: vgaarb: bridge control possible1447client # [ 0.546389] ACPI: IRQ 4 override to edge(!), high(!)1448worker1 # [ 0.533178] VFS: Disk quotas dquot_6.6.01449client # [ 0.547358] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1450worker2 # [ 0.533954] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1451worker1 # [ 0.533936] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1452worker2 # [ 0.533962] vgaarb: loaded1453client # [ 0.548842] pnp: PnP ACPI: found 6 devices1454worker1 # [ 0.535392] pnp: PnP ACPI init1455worker2 # [ 0.534697] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01456worker1 # [ 0.536191] ACPI: IRQ 4 override to edge(!), high(!)1457worker2 # [ 0.534960] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1458worker1 # [ 0.537175] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1459lb # [ 0.555333] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1460worker1 # [ 0.538779] pnp: PnP ACPI: found 6 devices1461lb # [ 0.556791] clocksource: Switched to clocksource acpi_pm1462lb # [ 0.557800] NET: Registered PF_INET protocol family1463worker2 # [ 0.538052] clocksource: Switched to clocksource kvm-clock1464lb # [ 0.558992] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1465client # [ 0.556430] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1466worker2 # [ 0.541601] VFS: Disk quotas dquot_6.6.01467client # [ 0.557967] clocksource: Switched to clocksource acpi_pm1468worker2 # [ 0.542324] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1469client # [ 0.558963] NET: Registered PF_INET protocol family1470worker2 # [ 0.543684] pnp: PnP ACPI init1471client # [ 0.560166] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1472worker2 # [ 0.544531] ACPI: IRQ 4 override to edge(!), high(!)1473worker1 # [ 0.546419] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1474worker1 # [ 0.547857] clocksource: Switched to clocksource acpi_pm1475worker2 # [ 0.545451] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1476worker1 # [ 0.548828] NET: Registered PF_INET protocol family1477worker2 # [ 0.547011] pnp: PnP ACPI: found 6 devices1478worker1 # [ 0.550075] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1479worker2 # [ 0.554622] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1480worker2 # [ 0.556131] clocksource: Switched to clocksource acpi_pm1481lb # [ 0.575654] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1482worker2 # [ 0.557092] NET: Registered PF_INET protocol family1483lb # [ 0.577183] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1484worker2 # [ 0.558783] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1485lb # [ 0.578523] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1486lb # [ 0.579857] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1487lb # [ 0.581143] TCP: Hash tables configured (established 16384 bind 16384)1488client # [ 0.577565] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1489lb # [ 0.582275] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1490client # [ 0.579118] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1491lb # [ 0.583595] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1492client # [ 0.580516] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1493lb # [ 0.584696] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1494worker1 # [ 0.567393] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1495client # [ 0.581919] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1496lb # [ 0.585893] NET: Registered PF_UNIX/PF_LOCAL protocol family1497lb # [ 0.586841] NET: Registered PF_XDP protocol family1498worker1 # [ 0.568932] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1499client # [ 0.583139] TCP: Hash tables configured (established 16384 bind 16384)1500lb # [ 0.587641] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1501worker1 # [ 0.570209] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1502client # [ 0.584231] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1503lb # [ 0.588639] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1504client # [ 0.585529] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1505worker1 # [ 0.571527] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1506lb # [ 0.589619] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1507client # [ 0.586581] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1508worker1 # [ 0.572757] TCP: Hash tables configured (established 16384 bind 16384)1509lb # [ 0.590672] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1510client # [ 0.587765] NET: Registered PF_UNIX/PF_LOCAL protocol family1511worker1 # [ 0.573874] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1512lb # [ 0.591739] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1513client # [ 0.588787] NET: Registered PF_XDP protocol family1514worker1 # [ 0.575121] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1515lb # [ 0.592795] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1516client # [ 0.589651] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1517worker1 # [ 0.576205] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1518lb # [ 0.594415] ACPI: \_SB_.GSIA: Enabled at IRQ 161519client # [ 0.590652] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1520worker1 # [ 0.577409] NET: Registered PF_UNIX/PF_LOCAL protocol family1521client # [ 0.591668] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1522worker1 # [ 0.578373] NET: Registered PF_XDP protocol family1523worker2 # [ 0.575780] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1524client # [ 0.592777] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1525lb # [ 0.596655] ACPI: \_SB_.GSIB: Enabled at IRQ 171526worker1 # [ 0.579193] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1527worker2 # [ 0.577165] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1528client # [ 0.593832] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1529worker1 # [ 0.580142] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1530lb # [ 0.598668] ACPI: \_SB_.GSIC: Enabled at IRQ 181531worker2 # [ 0.578399] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1532client # [ 0.594912] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1533worker1 # [ 0.581135] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1534worker2 # [ 0.579699] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1535worker1 # [ 0.582178] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1536client # [ 0.596567] ACPI: \_SB_.GSIA: Enabled at IRQ 161537lb # [ 0.600661] ACPI: \_SB_.GSID: Enabled at IRQ 191538worker2 # [ 0.580923] TCP: Hash tables configured (established 16384 bind 16384)1539worker1 # [ 0.583266] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1540lb # [ 0.602303] PCI: CLS 0 bytes, default 641541worker1 # [ 0.584327] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1542client # [ 0.598765] ACPI: \_SB_.GSIB: Enabled at IRQ 171543worker2 # [ 0.582010] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1544worker2 # [ 0.583212] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1545worker1 # [ 0.586040] ACPI: \_SB_.GSIA: Enabled at IRQ 161546lb # [ 0.603164] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1547worker2 # [ 0.584274] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1548client # [ 0.600839] ACPI: \_SB_.GSIC: Enabled at IRQ 181549lb # [ 0.604865] Trying to unpack rootfs image as initramfs...1550worker2 # [ 0.585444] NET: Registered PF_UNIX/PF_LOCAL protocol family1551worker1 # [ 0.588428] ACPI: \_SB_.GSIB: Enabled at IRQ 171552worker2 # [ 0.586437] NET: Registered PF_XDP protocol family1553client # [ 0.602855] ACPI: \_SB_.GSID: Enabled at IRQ 191554worker2 # [ 0.587264] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1555worker1 # [ 0.590502] ACPI: \_SB_.GSIC: Enabled at IRQ 181556client # [ 0.604583] PCI: CLS 0 bytes, default 641557worker2 # [ 0.588291] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1558worker2 # [ 0.589247] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1559client # [ 0.605454] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1560worker1 # [ 0.592435] ACPI: \_SB_.GSID: Enabled at IRQ 191561worker2 # [ 0.590342] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1562client # [ 0.607168] Trying to unpack rootfs image as initramfs...1563worker2 # [ 0.591421] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1564worker1 # [ 0.594193] PCI: CLS 0 bytes, default 641565worker2 # [ 0.592517] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1566worker1 # [ 0.595133] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1567worker2 # [ 0.594235] ACPI: \_SB_.GSIA: Enabled at IRQ 161568worker1 # [ 0.596892] Trying to unpack rootfs image as initramfs...1569worker2 # [ 0.596664] ACPI: \_SB_.GSIB: Enabled at IRQ 171570worker2 # [ 0.598889] ACPI: \_SB_.GSIC: Enabled at IRQ 181571worker2 # [ 0.600952] ACPI: \_SB_.GSID: Enabled at IRQ 191572worker2 # [ 0.602836] PCI: CLS 0 bytes, default 641573worker2 # [ 0.603966] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1574worker2 # [ 0.605761] Trying to unpack rootfs image as initramfs...1575lb # [ 0.649945] Initialise system trusted keyrings1576lb # [ 0.653527] workingset: timestamp_bits=40 max_order=19 bucket_order=01577client # [ 0.651373] Initialise system trusted keyrings1578client # [ 0.654343] workingset: timestamp_bits=40 max_order=19 bucket_order=01579worker1 # [ 0.642837] Initialise system trusted keyrings1580worker1 # [ 0.645616] workingset: timestamp_bits=40 max_order=19 bucket_order=01581worker2 # [ 0.653094] Initialise system trusted keyrings1582worker2 # [ 0.654105] workingset: timestamp_bits=40 max_order=19 bucket_order=01583lb # [ 0.674703] Key type asymmetric registered1584lb # [ 0.675390] Asymmetric key parser 'x509' registered1585lb # [ 0.676224] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1586client # [ 0.675679] Key type asymmetric registered1587client # [ 0.676439] Asymmetric key parser 'x509' registered1588lb # [ 0.681783] io scheduler mq-deadline registered1589lb # [ 0.682539] io scheduler kyber registered1590client # [ 0.679355] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1591worker1 # [ 0.666925] Key type asymmetric registered1592lb # [ 0.685074] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1593client # [ 0.682365] io scheduler mq-deadline registered1594client # [ 0.683130] io scheduler kyber registered1595lb # [ 0.686325] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1596worker1 # [ 0.669588] Asymmetric key parser 'x509' registered1597worker1 # [ 0.670462] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1598client # [ 0.687015] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1599lb # [ 0.691262] Linux agpgart interface v0.1031600worker1 # [ 0.673707] io scheduler mq-deadline registered1601worker1 # [ 0.674463] io scheduler kyber registered1602lb # [ 0.692033] ACPI: bus type drm_connector registered1603client # [ 0.690299] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1604worker1 # [ 0.677155] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1605lb # [ 0.694909] usbcore: registered new interface driver usbserial_generic1606lb # [ 0.695967] usbserial: USB Serial support registered for generic1607worker1 # [ 0.678474] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1608worker2 # [ 0.677683] Key type asymmetric registered1609client # [ 0.693980] Linux agpgart interface v0.1031610worker2 # [ 0.678385] Asymmetric key parser 'x509' registered1611client # [ 0.694778] ACPI: bus type drm_connector registered1612worker2 # [ 0.679216] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1613lb # [ 0.699494] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1614lb # [ 0.699494] Please enable it if your BIOS has the CPPC option.1615worker2 # [ 0.681782] io scheduler mq-deadline registered1616client # [ 0.697755] usbcore: registered new interface driver usbserial_generic1617worker1 # [ 0.684362] Linux agpgart interface v0.1031618lb # [ 0.701679] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1619worker2 # [ 0.682515] io scheduler kyber registered1620client # [ 0.698846] usbserial: USB Serial support registered for generic1621worker1 # [ 0.685139] ACPI: bus type drm_connector registered1622lb # [ 0.704604] drop_monitor: Initializing network drop monitor service1623lb # [ 0.705806] NET: Registered PF_INET6 protocol family1624client # [ 0.702301] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1625worker2 # [ 0.686841] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1626client # [ 0.702301] Please enable it if your BIOS has the CPPC option.1627worker1 # [ 0.689674] usbcore: registered new interface driver usbserial_generic1628worker2 # [ 0.688100] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1629worker1 # [ 0.690732] usbserial: USB Serial support registered for generic1630client # [ 0.704529] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1631lb # [ 0.708950] Segment Routing with IPv61632lb # [ 0.711504] In-situ OAM (IOAM) with IPv61633worker1 # [ 0.693577] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1634client # [ 0.708437] drop_monitor: Initializing network drop monitor service1635lb # [ 0.712582] IPI shorthand broadcast: enabled1636worker1 # [ 0.693577] Please enable it if your BIOS has the CPPC option.1637worker2 # [ 0.693436] Linux agpgart interface v0.1031638client # [ 0.709612] NET: Registered PF_INET6 protocol family1639worker1 # [ 0.695800] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1640worker2 # [ 0.694200] ACPI: bus type drm_connector registered1641client # [ 0.712771] Segment Routing with IPv61642worker1 # [ 0.698729] drop_monitor: Initializing network drop monitor service1643worker2 # [ 0.697146] usbcore: registered new interface driver usbserial_generic1644worker1 # [ 0.699908] NET: Registered PF_INET6 protocol family1645worker2 # [ 0.698203] usbserial: USB Serial support registered for generic1646client # [ 0.715322] In-situ OAM (IOAM) with IPv61647client # [ 0.716394] IPI shorthand broadcast: enabled1648worker1 # [ 0.703025] Segment Routing with IPv61649lb # [ 0.720794] sched_clock: Marking stable (584014070, 136458678)->(794540267, -74067519)1650worker2 # [ 0.701668] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1651worker2 # [ 0.701668] Please enable it if your BIOS has the CPPC option.1652worker1 # [ 0.705591] In-situ OAM (IOAM) with IPv61653worker1 # [ 0.706617] IPI shorthand broadcast: enabled1654worker2 # [ 0.703877] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1655lb # [ 0.725653] registered taskstats version 11656lb # [ 0.726584] Loading compiled-in X.509 certificates1657worker2 # [ 0.706815] drop_monitor: Initializing network drop monitor service1658worker2 # [ 0.707964] NET: Registered PF_INET6 protocol family1659client # [ 0.724685] sched_clock: Marking stable (591014145, 133270074)->(792618677, -68334458)1660worker2 # [ 0.712762] Segment Routing with IPv61661worker1 # [ 0.714892] sched_clock: Marking stable (582015041, 132544977)->(788575957, -74015939)1662worker2 # [ 0.713570] In-situ OAM (IOAM) with IPv61663client # [ 0.730394] registered taskstats version 11664client # [ 0.731361] Loading compiled-in X.509 certificates1665worker2 # [ 0.716936] IPI shorthand broadcast: enabled1666worker1 # [ 0.720785] registered taskstats version 11667worker1 # [ 0.721780] Loading compiled-in X.509 certificates1668lb # [ 0.742976] Demotion targets for Node 0: null1669worker2 # [ 0.723364] sched_clock: Marking stable (590016105, 132629517)->(797129475, -74483853)1670lb # [ 0.745532] Key type .fscrypt registered1671lb # [ 0.746174] Key type fscrypt-provisioning registered1672worker2 # [ 0.726825] registered taskstats version 11673lb # [ 0.747086] ima: No TPM chip found, activating TPM-bypass!1674worker2 # [ 0.727808] Loading compiled-in X.509 certificates1675client # [ 0.747774] Demotion targets for Node 0: null1676lb # [ 0.751498] ima: Allocated hash algorithm: sha11677lb # [ 0.752274] ima: No architecture policies found1678lb # [ 0.753232] PM: Magic number: 10:416:7301679client # [ 0.751338] Key type .fscrypt registered1680client # [ 0.752021] Key type fscrypt-provisioning registered1681client # [ 0.753012] ima: No TPM chip found, activating TPM-bypass!1682lb # [ 0.756890] RAS: Correctable Errors collector initialized.1683worker1 # [ 0.740575] Demotion targets for Node 0: null1684client # [ 0.755297] ima: Allocated hash algorithm: sha11685worker1 # [ 0.741476] Key type .fscrypt registered1686client # [ 0.756088] ima: No architecture policies found1687worker1 # [ 0.742151] Key type fscrypt-provisioning registered1688client # [ 0.758298] PM: Magic number: 10:416:7301689worker1 # [ 0.744685] ima: No TPM chip found, activating TPM-bypass!1690worker1 # [ 0.745641] ima: Allocated hash algorithm: sha11691client # [ 0.759983] RAS: Correctable Errors collector initialized.1692worker1 # [ 0.746468] ima: No architecture policies found1693lb # [ 0.765526] clk: Disabling unused clocks1694lb # [ 0.766200] PM: genpd: Disabling unused power domains1695worker2 # [ 0.746663] Demotion targets for Node 0: null1696worker2 # [ 0.747546] Key type .fscrypt registered1697worker1 # [ 0.750774] PM: Magic number: 10:416:7301698worker1 # [ 0.752459] RAS: Correctable Errors collector initialized.1699worker2 # [ 0.750653] Key type fscrypt-provisioning registered1700worker2 # [ 0.751610] ima: No TPM chip found, activating TPM-bypass!1701worker2 # [ 0.752575] ima: Allocated hash algorithm: sha11702client # [ 0.769045] clk: Disabling unused clocks1703worker2 # [ 0.753409] ima: No architecture policies found1704worker2 # [ 0.755658] PM: Magic number: 10:416:7301705client # [ 0.772301] PM: genpd: Disabling unused power domains1706worker2 # [ 0.757333] RAS: Correctable Errors collector initialized.1707worker1 # [ 0.762336] clk: Disabling unused clocks1708worker1 # [ 0.764574] PM: genpd: Disabling unused power domains1709worker2 # [ 0.766419] clk: Disabling unused clocks1710worker2 # [ 0.770660] PM: genpd: Disabling unused power domains1711lb # [ 0.938347] Freeing initrd memory: 29092K1712lb # [ 0.941615] Freeing unused decrypted memory: 2028K1713client # [ 0.940850] Freeing initrd memory: 29096K1714lb # [ 0.944330] Freeing unused kernel image (initmem) memory: 3644K1715lb # [ 0.945410] Write protecting the kernel read-only data: 32768k1716lb # [ 0.947330] Freeing unused kernel image (text/rodata gap) memory: 1208K1717client # [ 0.944067] Freeing unused decrypted memory: 2028K1718lb # [ 0.948917] Freeing unused kernel image (rodata/data gap) memory: 752K1719client # [ 0.946819] Freeing unused kernel image (initmem) memory: 3644K1720client # [ 0.947940] Write protecting the kernel read-only data: 32768k1721worker1 # [ 0.935438] Freeing initrd memory: 29092K1722client # [ 0.949883] Freeing unused kernel image (text/rodata gap) memory: 1208K1723client # [ 0.951456] Freeing unused kernel image (rodata/data gap) memory: 752K1724worker1 # [ 0.938662] Freeing unused decrypted memory: 2028K1725worker1 # [ 0.941403] Freeing unused kernel image (initmem) memory: 3644K1726worker1 # [ 0.942578] Write protecting the kernel read-only data: 32768k1727worker2 # [ 0.942451] Freeing initrd memory: 29092K1728worker1 # [ 0.944530] Freeing unused kernel image (text/rodata gap) memory: 1208K1729worker1 # [ 0.946074] Freeing unused kernel image (rodata/data gap) memory: 752K1730worker2 # [ 0.945702] Freeing unused decrypted memory: 2028K1731worker2 # [ 0.948416] Freeing unused kernel image (initmem) memory: 3644K1732worker2 # [ 0.949525] Write protecting the kernel read-only data: 32768k1733worker2 # [ 0.951473] Freeing unused kernel image (text/rodata gap) memory: 1208K1734worker2 # [ 0.953051] Freeing unused kernel image (rodata/data gap) memory: 752K1735lb # [ 1.000257] x86/mm: Checked W+X mappings: passed, no W+X pages found.1736lb # [ 1.001360] Run /init as init process1737client # [ 1.002577] x86/mm: Checked W+X mappings: passed, no W+X pages found.1738client # [ 1.003697] Run /init as init process1739lb # [ 1.011607] systemd[1]: Inserted module 'autofs4'1740worker1 # [ 0.997432] x86/mm: Checked W+X mappings: passed, no W+X pages found.1741worker1 # [ 0.998576] Run /init as init process1742client # [ 1.014000] systemd[1]: Inserted module 'autofs4'1743worker2 # [ 1.004757] x86/mm: Checked W+X mappings: passed, no W+X pages found.1744worker2 # [ 1.005869] Run /init as init process1745worker1 # [ 1.009070] systemd[1]: Inserted module 'autofs4'1746lb # [ 1.027875] fuse: init (API version 7.45)1747lb # [ 1.033811] ACPI: \_SB_.GSIG: Enabled at IRQ 221748client # [ 1.030400] fuse: init (API version 7.45)1749worker2 # [ 1.016116] systemd[1]: Inserted module 'autofs4'1750lb # [ 1.036144] ACPI: \_SB_.GSIH: Enabled at IRQ 231751lb # [ 1.039064] ACPI: \_SB_.GSIE: Enabled at IRQ 201752client # [ 1.036247] ACPI: \_SB_.GSIG: Enabled at IRQ 221753lb # [ 1.041283] ACPI: \_SB_.GSIF: Enabled at IRQ 211754client # [ 1.039028] ACPI: \_SB_.GSIH: Enabled at IRQ 231755worker1 # [ 1.025584] fuse: init (API version 7.45)1756client # [ 1.042245] ACPI: \_SB_.GSIE: Enabled at IRQ 201757client # [ 1.044617] ACPI: \_SB_.GSIF: Enabled at IRQ 211758worker1 # [ 1.031398] ACPI: \_SB_.GSIG: Enabled at IRQ 221759worker1 # [ 1.033855] ACPI: \_SB_.GSIH: Enabled at IRQ 231760worker2 # [ 1.032567] fuse: init (API version 7.45)1761worker1 # [ 1.037382] ACPI: \_SB_.GSIE: Enabled at IRQ 201762worker1 # [ 1.040162] ACPI: \_SB_.GSIF: Enabled at IRQ 211763worker2 # [ 1.038549] ACPI: \_SB_.GSIG: Enabled at IRQ 221764worker2 # [ 1.041307] ACPI: \_SB_.GSIH: Enabled at IRQ 231765worker2 # [ 1.044424] ACPI: \_SB_.GSIE: Enabled at IRQ 201766worker2 # [ 1.046966] ACPI: \_SB_.GSIF: Enabled at IRQ 211767lb # [ 1.069623] systemd[1]: Successfully made /usr/ read-only.1768client # [ 1.072460] systemd[1]: Successfully made /usr/ read-only.1769worker1 # [ 1.069486] systemd[1]: Successfully made /usr/ read-only.1770worker2 # [ 1.074447] systemd[1]: Successfully made /usr/ read-only.1771lb # [ 1.405304] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1772lb # [ 1.410201] systemd[1]: Detected virtualization kvm.1773lb # [ 1.411042] systemd[1]: Detected architecture x86-64.1774lb # [ 1.411902] systemd[1]: Running in initrd.1775lb # [ 1.413089] systemd[1]: Initializing machine ID from random generator.1776lb # [ 1.414271] systemd[1]: Hostname set to <lb>.1777client # [ 1.408479] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1778client # [ 1.413287] systemd[1]: Detected virtualization kvm.1779client # [ 1.414089] systemd[1]: Detected architecture x86-64.1780client # [ 1.414937] systemd[1]: Running in initrd.1781client # [ 1.416121] systemd[1]: Initializing machine ID from random generator.1782client # [ 1.417354] systemd[1]: Hostname set to <client>.1783worker1 # [ 1.405183] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1784worker1 # [ 1.410183] systemd[1]: Detected virtualization kvm.1785worker1 # [ 1.411109] systemd[1]: Detected architecture x86-64.1786worker1 # [ 1.411956] systemd[1]: Running in initrd.1787worker1 # [ 1.413156] systemd[1]: Initializing machine ID from random generator.1788worker1 # [ 1.414360] systemd[1]: Hostname set to <worker1>.1789worker2 # [ 1.409880] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1790worker2 # [ 1.414676] systemd[1]: Detected virtualization kvm.1791worker2 # [ 1.415494] systemd[1]: Detected architecture x86-64.1792worker2 # [ 1.416315] systemd[1]: Running in initrd.1793worker2 # [ 1.417381] systemd[1]: Initializing machine ID from random generator.1794worker2 # [ 1.418510] systemd[1]: Hostname set to <worker2>.1795worker1 # [ 1.507144] systemd[1]: bpf-restrict-fs: LSM BPF program attached1796worker1 # [ 1.546259] systemd[1]: Queued start job for default target Initrd Default Target.1797worker1 # [ 1.551126] systemd[1]: Created slice Slice /system/modprobe.1798worker1 # [ 1.552404] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1799worker1 # [ 1.553776] systemd[1]: Expecting device /dev/disk/by-label/nixos...1800worker1 # [ 1.554842] systemd[1]: Reached target Path Units.1801worker1 # [ 1.555729] systemd[1]: Reached target Slice Units.1802worker1 # [ 1.556635] systemd[1]: Reached target Swaps.1803worker1 # [ 1.557454] systemd[1]: Reached target Timer Units.1804worker1 # [ 1.558446] systemd[1]: Listening on D-Bus System Message Bus Socket.1805worker1 # [ 1.559706] systemd[1]: Listening on Journal Socket (/dev/log).1806worker1 # [ 1.560930] systemd[1]: Listening on Journal Sockets.1807worker1 # [ 1.561961] systemd[1]: Listening on udev Control Socket.1808worker1 # [ 1.563007] systemd[1]: Listening on udev Kernel Socket.1809worker1 # [ 1.563998] systemd[1]: Reached target Socket Units.1810worker1 # [ 1.565820] systemd[1]: Starting Create List of Static Device Nodes...1811worker1 # [ 1.569631] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1812worker1 # [ 1.578764] systemd[1]: Starting Load Kernel Module configfs...1813worker1 # [ 1.591255] systemd[1]: Starting Journal Service...1814lb # [ 1.621446] systemd[1]: bpf-restrict-fs: LSM BPF program attached1815worker1 # [ 1.606696] systemd[1]: Starting Load Kernel Modules...1816client # [ 1.624494] systemd[1]: bpf-restrict-fs: LSM BPF program attached1817worker1 # [ 1.611650] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1818worker2 # [ 1.620915] systemd[1]: bpf-restrict-fs: LSM BPF program attached1819worker1 # [ 1.629172] netfs: FS-Cache loaded1820worker1 # [ 1.634097] 9pnet: Installing 9P2000 support1821worker1 # [ 1.648668] systemd[1]: Starting Coldplug All udev Devices...1822lb # [ 1.672898] systemd[1]: Queued start job for default target Initrd Default Target.1823lb # [ 1.676999] systemd[1]: Created slice Slice /system/modprobe.1824lb # [ 1.678234] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1825lb # [ 1.679707] systemd[1]: Expecting device /dev/disk/by-label/nixos...1826lb # [ 1.680813] systemd[1]: Reached target Path Units.1827lb # [ 1.681709] systemd[1]: Reached target Slice Units.1828worker1 # [ 1.664640] systemd[1]: Finished Create List of Static Device Nodes.1829lb # [ 1.682571] systemd[1]: Reached target Swaps.1830client # [ 1.678595] systemd[1]: Queued start job for default target Initrd Default Target.1831lb # [ 1.683339] systemd[1]: Reached target Timer Units.1832lb # [ 1.684365] systemd[1]: Listening on D-Bus System Message Bus Socket.1833lb # [ 1.685612] systemd[1]: Listening on Journal Socket (/dev/log).1834client # [ 1.682723] systemd[1]: Created slice Slice /system/modprobe.1835lb # [ 1.686784] systemd[1]: Listening on Journal Sockets.1836lb # [ 1.687809] systemd[1]: Listening on udev Control Socket.1837client # [ 1.683907] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1838worker1 # [ 1.670400] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1839lb # [ 1.688897] systemd[1]: Listening on udev Kernel Socket.1840client # [ 1.685299] systemd[1]: Expecting device /dev/disk/by-label/nixos...1841client # [ 1.686347] systemd[1]: Reached target Path Units.1842lb # [ 1.689888] systemd[1]: Reached target Socket Units.1843client # [ 1.687155] systemd[1]: Reached target Slice Units.1844worker2 # [ 1.670746] systemd[1]: Queued start job for default target Initrd Default Target.1845client # [ 1.687994] systemd[1]: Reached target Swaps.1846lb # [ 1.691874] systemd[1]: Starting Create List of Static Device Nodes...1847client # [ 1.688845] systemd[1]: Reached target Timer Units.1848client # [ 1.689848] systemd[1]: Listening on D-Bus System Message Bus Socket.1849client # [ 1.691044] systemd[1]: Listening on Journal Socket (/dev/log).1850worker2 # [ 1.674992] systemd[1]: Created slice Slice /system/modprobe.1851client # [ 1.692195] systemd[1]: Listening on Journal Sockets.1852worker2 # [ 1.676169] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1853client # [ 1.693160] systemd[1]: Listening on udev Control Socket.1854worker2 # [ 1.677511] systemd[1]: Expecting device /dev/disk/by-label/nixos...1855client # [ 1.694152] systemd[1]: Listening on udev Kernel Socket.1856lb # [ 1.697557] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1857worker2 # [ 1.678586] systemd[1]: Reached target Path Units.1858client # [ 1.695075] systemd[1]: Reached target Socket Units.1859worker1 # [ 1.681168] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1860worker2 # [ 1.679416] systemd[1]: Reached target Slice Units.1861worker2 # [ 1.680259] systemd[1]: Reached target Swaps.1862client # [ 1.696946] systemd[1]: Starting Create List of Static Device Nodes...1863worker2 # [ 1.681037] systemd[1]: Reached target Timer Units.1864worker2 # [ 1.681972] systemd[1]: Listening on D-Bus System Message Bus Socket.1865lb # [ 1.702557] systemd[1]: Starting Load Kernel Module configfs...1866worker2 # [ 1.683183] systemd[1]: Listening on Journal Socket (/dev/log).1867worker1 # [ 1.685521] systemd-journald[67]: Collecting audit messages is disabled.1868worker2 # [ 1.684308] systemd[1]: Listening on Journal Sockets.1869client # [ 1.701355] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1870worker2 # [ 1.685301] systemd[1]: Listening on udev Control Socket.1871worker2 # [ 1.686287] systemd[1]: Listening on udev Kernel Socket.1872worker2 # [ 1.687258] systemd[1]: Reached target Socket Units.1873worker1 # [ 1.690486] systemd[1]: modprobe@configfs.service: Deactivated successfully.1874worker2 # [ 1.689101] systemd[1]: Starting Create List of Static Device Nodes...1875client # [ 1.706455] systemd[1]: Starting Load Kernel Module configfs...1876worker2 # [ 1.693746] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1877worker1 # [ 1.698001] systemd[1]: Finished Load Kernel Module configfs.1878lb # [ 1.718271] systemd[1]: Starting Journal Service...1879worker1 # [ 1.700476] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1880worker2 # [ 1.701848] systemd[1]: Starting Load Kernel Module configfs...1881client # [ 1.718948] systemd[1]: Starting Journal Service...1882worker1 # [ 1.705186] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1883lb # [ 1.729669] systemd[1]: Starting Load Kernel Modules...1884worker1 # [ 1.712757] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1885worker1 # [ 1.715862] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1886client # [ 1.731427] systemd[1]: Starting Load Kernel Modules...1887worker2 # [ 1.715725] systemd[1]: Starting Journal Service...1888worker2 # [ 1.727903] systemd[1]: Starting Load Kernel Modules...1889lb # [ 1.737626] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1890client # [ 1.737372] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1891lb # [ 1.757771] netfs: FS-Cache loaded1892worker1 # [ 1.745161] systemd[1]: Finished Load Kernel Modules.1893client # [ 1.760405] netfs: FS-Cache loaded1894worker2 # [ 1.733725] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1895lb # [ 1.765786] 9pnet: Installing 9P2000 support1896client # [ 1.766927] 9pnet: Installing 9P2000 support1897worker2 # [ 1.751040] netfs: FS-Cache loaded1898worker1 # [ 1.753845] systemd[1]: Starting Apply Kernel Variables...1899worker2 # [ 1.755840] 9pnet: Installing 9P2000 support1900client # [ 1.773385] systemd[1]: Starting Coldplug All udev Devices...1901lb # [ 1.780628] systemd[1]: Starting Coldplug All udev Devices...1902worker1 # [ 1.769768] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1903worker2 # [ 1.772852] systemd[1]: Starting Coldplug All udev Devices...1904lb # [ 1.796174] systemd-journald[67]: Collecting audit messages is disabled.1905lb # [ 1.797424] systemd[1]: Finished Create List of Static Device Nodes.1906worker1 # [ 1.780887] systemd[1]: Starting Create Static Device Nodes in /dev...1907client # [ 1.796415] systemd[1]: Finished Create List of Static Device Nodes.1908lb # [ 1.800416] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1909worker2 # [ 1.788815] systemd[1]: Finished Create List of Static Device Nodes.1910lb # [ 1.808981] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1911client # [ 1.805218] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1912lb # [ 1.814385] systemd[1]: modprobe@configfs.service: Deactivated successfully.1913worker2 # [ 1.796455] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1914worker1 # [ 1.799937] systemd[1]: Finished Apply Kernel Variables.1915client # [ 1.814362] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1916lb # [ 1.820873] systemd[1]: Finished Load Kernel Module configfs.1917worker2 # [ 1.801683] systemd-journald[67]: Collecting audit messages is disabled.1918client # [ 1.818249] systemd-journald[67]: Collecting audit messages is disabled.1919worker2 # [ 1.804115] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1920client # [ 1.821799] systemd[1]: modprobe@configfs.service: Deactivated successfully.1921lb # [ 1.826932] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1922client # [ 1.826695] systemd[1]: Finished Load Kernel Module configfs.1923worker1 # [ 1.680381] systemd-modules-load[68]: Inserted module 'dm_mod'[ 1.814088] systemd[1]: Started Journal Service.1924worker2 # [ 1.812144] systemd[1]: modprobe@configfs.service: Deactivated successfully.1925worker1 # 1926lb # [ 1.831927] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1927worker1 # [ 1.683321] systemd-modules-load[68]: Inserted module 'virtio_balloon'1928worker1 # [ 1.686388] systemd-modules-load[68]: Inserted module 'virtio_gpu'1929client # [ 1.831885] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1930worker2 # [ 1.818077] systemd[1]: Finished Load Kernel Module configfs.1931lb # [ 1.838682] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1932client # [ 1.837247] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1933lb # [ 1.841497] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1934worker2 # [ 1.822154] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1935worker1 # [ 1.696216] systemd[1]: Finished Create Static Device Nodes in /dev.1936worker2 # [ 1.825986] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1937client # [ 1.844524] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1938worker1 # [ 1.700078] systemd[1]: Reached target Preparation for Local File Systems.1939worker1 # [ 1.702096] systemd[1]: Reached target Local File Systems.1940client # [ 1.848315] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1941worker2 # [ 1.832490] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1942worker1 # [ 1.705276] systemd[1]: Starting Create System Files and Directories...1943worker2 # [ 1.835985] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1944worker1 # [ 1.712408] systemd[1]: Starting Rule-based Manager for Device Events and Files...1945lb # [ 1.872048] systemd[1]: Finished Load Kernel Modules.1946lb # [ 1.878674] systemd[1]: Starting Apply Kernel Variables...1947client # [ 1.876935] systemd[1]: Finished Load Kernel Modules.1948worker2 # [ 1.867783] systemd[1]: Finished Load Kernel Modules.1949client # [ 1.885508] systemd[1]: Starting Apply Kernel Variables...1950lb # [ 1.890602] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1951worker1 # [ 1.746219] systemd[1]: Finished Create System Files and Directories.1952worker2 # [ 1.878836] systemd[1]: Starting Apply Kernel Variables...1953lb # [ 1.900681] systemd[1]: Starting Create Static Device Nodes in /dev...1954client # [ 1.898393] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1955client # [ 1.907509] systemd[1]: Starting Create Static Device Nodes in /dev...1956worker2 # [ 1.892780] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1957lb # [ 1.915628] systemd[1]: Finished Apply Kernel Variables.1958worker1 # [ 1.768900] systemd-udevd[84]: Using default interface naming scheme 'v261'.1959lb # [ 1.782555] systemd-modules-load[68]: Inserted module 'dm_mod'[ 1.920417] systemd[1]: Started Journal Service.1960lb # 1961lb # [ 1.787469] systemd-modules-load[68]: Inserted module 'virtio_balloon'1962lb # [ 1.788637] systemd-modules-load[68]: Inserted module 'virtio_gpu'1963worker2 # [ 1.906865] systemd[1]: Starting Create Static Device Nodes in /dev...1964client # [ 1.929865] systemd[1]: Finished Apply Kernel Variables.1965lb # [ 1.803115] systemd[1]: Finished Create Static Device Nodes in /dev.1966lb # [ 1.804057] systemd[1]: Reached target Preparation for Local File Systems.1967lb # [ 1.804975] systemd[1]: Reached target Local File Systems.1968client # [ 1.940188] systemd[1]: Started Journal Service.1969worker1 # [ 1.794080] systemd[1]: Started Rule-based Manager for Device Events and Files.1970worker2 # [ 1.925291] systemd[1]: Finished Apply Kernel Variables.1971lb # [ 1.809075] systemd[1]: Starting Create System Files and Directories...1972client # [ 1.810241] systemd-modules-load[68]: Inserted module 'dm_mod'1973client # [ 1.811704] systemd-modules-load[68]: Inserted module 'virtio_balloon'1974lb # [ 1.814075] systemd[1]: Starting Rule-based Manager for Device Events and Files...1975client # [ 1.814830] systemd-modules-load[68]: Inserted module 'virtio_gpu'1976worker2 # [ 1.801340] systemd-modules-load[68]: Inserted module 'dm_mod'[ 1.935109] systemd[1]: Started Journal Service.1977worker2 # 1978client # [ 1.819076] systemd[1]: Finished Create Static Device Nodes in /dev.1979worker2 # [ 1.804315] systemd-modules-load[68]: Inserted module 'virtio_balloon'1980worker2 # [ 1.807293] systemd-modules-load[68]: Inserted module 'virtio_gpu'1981client # [ 1.823918] systemd[1]: Reached target Preparation for Local File Systems.1982client # [ 1.827075] systemd[1]: Reached target Local File Systems.1983client # [ 1.830078] systemd[1]: Starting Create System Files and Directories...1984worker2 # [ 1.818251] systemd[1]: Finished Create Static Device Nodes in /dev.1985client # [ 1.834762] systemd[1]: Starting Rule-based Manager for Device Events and Files...1986worker2 # [ 1.821040] systemd[1]: Reached target Preparation for Local File Systems.1987worker2 # [ 1.824071] systemd[1]: Reached target Local File Systems.1988lb # [ 1.840434] systemd[1]: Finished Create System Files and Directories.1989worker2 # [ 1.827067] systemd[1]: Starting Create System Files and Directories...1990worker2 # [ 1.831610] systemd[1]: Starting Rule-based Manager for Device Events and Files...1991worker1 # [ 1.840278] systemd[1]: Finished Coldplug All udev Devices.1992worker1 # [ 1.842073] systemd[1]: Reached target System Initialization.1993worker1 # [ 1.842862] systemd[1]: Reached target Basic System.1994lb # [ 1.866431] systemd-udevd[83]: Using default interface naming scheme 'v261'.1995client # [ 1.869450] systemd[1]: Finished Create System Files and Directories.1996worker2 # [ 1.866155] systemd[1]: Finished Create System Files and Directories.1997client # [ 1.889377] systemd-udevd[83]: Using default interface naming scheme 'v261'.1998lb # [ 1.892522] systemd[1]: Started Rule-based Manager for Device Events and Files.1999worker2 # [ 1.890402] systemd-udevd[82]: Using default interface naming scheme 'v261'.2000client # [ 1.913068] systemd[1]: Started Rule-based Manager for Device Events and Files.
loading 2,526 hidden lines…
4527worker2 # [ 34.811719] nix-grpc-daemon[844]: ts=2026-09-22T10:44:36Z level=info event=scheduler_connected addr=lb:500514528worker2 # [ 34.813634] nix-grpc-daemon[844]: ts=2026-09-22T10:44:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4529worker1 # [ 34.868704] nix-grpc-daemon[845]: ts=2026-09-22T10:44:36Z level=info event=scheduler_connected addr=lb:500514530worker1 # [ 34.870745] nix-grpc-daemon[845]: ts=2026-09-22T10:44:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4531worker2 # [ 35.015138] nix-grpc-daemon[844]: ts=2026-09-22T10:44:36Z level=info event=scheduler_connected addr=lb:500514532worker2 # [ 35.016835] nix-grpc-daemon[844]: ts=2026-09-22T10:44:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4533worker2 # [ 35.018600] nix-grpc-daemon[844]: ts=2026-09-22T10:44:36Z level=info event=scheduler_take_over4534lb # [ 35.034830] niks3-server[1077]: 2026/09/22 10:44:36 INFO lead: acquired remote=192.168.1.4:418024535worker1 # [ 35.071618] nix-grpc-daemon[845]: ts=2026-09-22T10:44:37Z level=info event=scheduler_connected addr=lb:500514536worker1 # [ 35.073284] nix-grpc-daemon[845]: ts=2026-09-22T10:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4537lb: must succeed: curl -sf localhost:9901/clusters4538worker2 # [ 35.218623] nix-grpc-daemon[844]: ts=2026-09-22T10:44:37Z level=info event=scheduler_connected addr=lb:500514539worker2 # [ 35.222733] nix-grpc-daemon[844]: ts=2026-09-22T10:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4540lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4541worker1 # [ 35.274334] nix-grpc-daemon[845]: ts=2026-09-22T10:44:37Z level=info event=scheduler_connected addr=lb:500514542worker1 # [ 35.278165] nix-grpc-daemon[845]: ts=2026-09-22T10:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4543worker2 # [ 35.424310] nix-grpc-daemon[844]: ts=2026-09-22T10:44:37Z level=info event=scheduler_connected addr=lb:500514544worker2 # [ 35.428961] nix-grpc-daemon[844]: ts=2026-09-22T10:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4545worker1 # [ 35.479222] nix-grpc-daemon[845]: ts=2026-09-22T10:44:37Z level=info event=scheduler_connected addr=lb:500514546worker1 # [ 35.483144] nix-grpc-daemon[845]: ts=2026-09-22T10:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4547worker2 # [ 35.630371] nix-grpc-daemon[844]: ts=2026-09-22T10:44:37Z level=info event=scheduler_connected addr=lb:500514548worker2 # [ 35.634234] nix-grpc-daemon[844]: ts=2026-09-22T10:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4549worker1 # [ 35.683852] nix-grpc-daemon[845]: ts=2026-09-22T10:44:37Z level=info event=scheduler_connected addr=lb:500514550worker1 # [ 35.687656] nix-grpc-daemon[845]: ts=2026-09-22T10:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4551worker2 # [ 35.835281] nix-grpc-daemon[844]: ts=2026-09-22T10:44:37Z level=info event=scheduler_connected addr=lb:500514552worker2 # [ 35.839235] nix-grpc-daemon[844]: ts=2026-09-22T10:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4553worker1 # [ 35.888763] nix-grpc-daemon[845]: ts=2026-09-22T10:44:37Z level=info event=scheduler_connected addr=lb:500514554worker2 # [ 35.897259] nix-grpc-daemon[844]: ts=2026-09-22T10:44:37Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:340364555worker2 # [ 35.902131] nix-grpc-daemon[844]: ts=2026-09-22T10:44:37Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04556worker2 # [ 36.040596] nix-grpc-daemon[844]: ts=2026-09-22T10:44:38Z level=info event=scheduler_connected addr=lb:500514557worker2 # [ 36.052400] nix-grpc-daemon[844]: ts=2026-09-22T10:44:38Z level=info event=oidc_jwks provider=mock keys=14558worker2 # [ 36.054794] nix-grpc-daemon[844]: ts=2026-09-22T10:44:38Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:340364559worker2 # [ 36.056557] nix-grpc-daemon[844]: ts=2026-09-22T10:44:38Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04560lb: must succeed: curl -sf localhost:9901/clusters4561lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4562worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4563worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4564lb: must succeed: curl -sf localhost:9901/clusters4565lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4566lb: must succeed: curl -sf localhost:9901/clusters4567lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4568lb: must succeed: curl -sf localhost:9901/clusters4569lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4570lb: must succeed: curl -sf localhost:9901/clusters4571lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4572worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4573worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.05 seconds)4574worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4575worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4576lb: must succeed: curl -sf localhost:9901/clusters4577lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4578lb: must succeed: curl -sf localhost:9901/clusters4579lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4580worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4581worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4582worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4583worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4584worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4585worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4586worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4587worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4588(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 11.27 seconds)4589subtest: balancer auth: client cert, bearer token, nothing, unknown CN4590worker1 # [ 37.266585] nix-grpc-daemon[845]: ts=2026-09-22T10:44:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:560724591worker1 # [ 37.279510] systemd[1]: Started Nix Daemon.4592worker1 # [ 37.482806] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4593worker1 # [ 37.513777] nix-grpc-daemon[845]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 43 ms (attempt 1/5)4594worker1 # [ 37.559276] nix-grpc-daemon[845]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 189 ms (attempt 2/5)4595worker1 # [ 37.748685] nix-grpc-daemon[845]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 199 ms (attempt 3/5)4596worker1 # [ 37.953787] nix-grpc-daemon[845]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 734 ms (attempt 4/5)4597worker1 # [ 38.698704] nix-grpc-daemon[845]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4598worker1 # [ 38.700947] nix-grpc-daemon[845]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4599worker1 # [ 38.723128] nix-grpc-daemon[845]: ts=2026-09-22T10:44:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:560724600worker1 # [ 38.733074] nix-grpc-daemon[845]: ts=2026-09-22T10:44:40Z level=debug event=path_info_miss path=zch17lfr063cj7f5rrjws8177ikhh5yq-nixos-system-client-test error="error: path '/nix/store/zch17lfr063cj7f5rrjws8177ikhh5yq-nixos-system-client-test' is required, but there is no substituter that can build it"4601worker1 # [ 38.745609] nix-grpc-daemon[845]: ts=2026-09-22T10:44:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:560724602worker1 # [ 38.752648] nix-grpc-daemon[845]: ts=2026-09-22T10:44:40Z level=debug event=path_info_miss path=zch17lfr063cj7f5rrjws8177ikhh5yq-nixos-system-client-test error="error: path '/nix/store/zch17lfr063cj7f5rrjws8177ikhh5yq-nixos-system-client-test' is required, but there is no substituter that can build it"4603client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4604client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds)4605client: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4606client: (finished: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4607worker1 # [ 39.145470] nix-grpc-daemon[845]: ts=2026-09-22T10:44:41Z level=info event=oidc_jwks provider=mock keys=14608worker1 # [ 39.147781] nix-grpc-daemon[845]: ts=2026-09-22T10:44:41Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:560724609worker1 # [ 39.155613] nix-grpc-daemon[845]: ts=2026-09-22T10:44:41Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:560724610worker1 # [ 39.160805] nix-grpc-daemon[845]: ts=2026-09-22T10:44:41Z level=debug event=path_info_miss path=zch17lfr063cj7f5rrjws8177ikhh5yq-nixos-system-client-test error="error: path '/nix/store/zch17lfr063cj7f5rrjws8177ikhh5yq-nixos-system-client-test' is required, but there is no substituter that can build it"4611worker1 # [ 39.166527] nix-grpc-daemon[845]: ts=2026-09-22T10:44:41Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:560724612worker1 # [ 39.175273] nix-grpc-daemon[845]: ts=2026-09-22T10:44:41Z level=debug event=path_info_miss path=zch17lfr063cj7f5rrjws8177ikhh5yq-nixos-system-client-test error="error: path '/nix/store/zch17lfr063cj7f5rrjws8177ikhh5yq-nixos-system-client-test' is required, but there is no substituter that can build it"4613client: must succeed: rm -r /var/lib/nix-grpc-store4614client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds)4615worker2 # [ 39.450139] nix-grpc-daemon[844]: ts=2026-09-22T10:44:41Z level=info event=denied method=QueryMissing cn=- role=none4616worker2 # [ 39.699787] nix-grpc-daemon[844]: ts=2026-09-22T10:44:41Z level=info event=denied method=QueryMissing cn=stranger role=none4617(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 2.82 seconds)4618subtest: DAG build is scheduled across workers and lands in the cache4619client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24620worker2 # [ 39.954499] nix-grpc-daemon[844]: ts=2026-09-22T10:44:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:340444621worker2 # [ 39.967807] systemd[1]: Started Nix Daemon.4622worker2 # [ 40.198248] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)4623client # don't know how to build these paths:4624client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4625worker2 # [ 40.224426] nix-grpc-daemon[844]: ts=2026-09-22T10:44:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340444626worker2 # [ 40.246626] nix-grpc-daemon[844]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 64 ms (attempt 1/5)4627worker2 # [ 40.313285] nix-grpc-daemon[844]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 195 ms (attempt 2/5)4628worker2 # [ 40.514299] nix-grpc-daemon[844]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 377 ms (attempt 3/5)4629worker2 # [ 40.891696] nix-grpc-daemon[844]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 515 ms (attempt 4/5)4630worker2 # [ 41.417319] nix-grpc-daemon[844]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4631worker2 # [ 41.422191] nix-grpc-daemon[844]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4632worker2 # [ 41.456702] nix-grpc-daemon[844]: ts=2026-09-22T10:44:43Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04633worker2 # [ 41.461508] nix-grpc-daemon[844]: ts=2026-09-22T10:44:43Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14634worker1 # [ 41.524072] nix-grpc-daemon[845]: ts=2026-09-22T10:44:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560724635worker1 # [ 41.543727] nix-grpc-daemon[845]: ts=2026-09-22T10:44:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:560724636client # copying 1 paths to 'lb:50051'...4637worker1 # [ 41.557831] nix-grpc-daemon[845]: ts=2026-09-22T10:44:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:560724638client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4639worker1 # [ 41.588114] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4640worker1 # [ 41.616834] nix-grpc-daemon[845]: ts=2026-09-22T10:44:43Z level=info event=niks3_push_started pid=9244641worker1 # [ 41.853615] nix-grpc-daemon[924]: time=2026-09-22T10:44:43.852Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""4642worker1 # [ 42.023510] nix-daemon[911]: accepted connection from pid 937, user nix-grpc-daemon (trusted)4643worker1 # [ 42.038319] nix-daemon[911]: reaped child process 940, status = succeeded4644worker1 # [ 42.189910] nix-daemon[911]: accepted connection from pid 942, user nix-grpc-daemon (trusted)4645worker1 # [ 42.202107] nix-daemon[911]: reaped child process 945, status = succeeded4646lb # [ 42.233712] niks3-server[1077]: 2026/09/22 10:44:44 INFO Received uploads request method=POST path=/api/pending_closures4647worker1 # [ 42.244593] nix-grpc-daemon[924]: time=2026-09-22T10:44:44.245Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4648worker1 # [ 42.246631] nix-grpc-daemon[924]: time=2026-09-22T10:44:44.247Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4649lb # [ 42.292477] niks3-server[1077]: 2026/09/22 10:44:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4650lb # [ 42.298097] niks3-server[1077]: 2026/09/22 10:44:44 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4651lb # [ 42.310445] niks3-server[1077]: 2026/09/22 10:44:44 INFO Signed narinfos id=1 count=14652worker1 # [ 42.298722] nix-grpc-daemon[924]: time=2026-09-22T10:44:44.299Z level=INFO msg="Uploading 1 narinfos"4653lb # [ 42.320328] niks3-server[1077]: 2026/09/22 10:44:44 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4654lb # [ 42.327634] niks3-server[1077]: 2026/09/22 10:44:44 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4655lb # [ 42.336314] niks3-server[1077]: 2026/09/22 10:44:44 INFO Completed upload id=14656worker1 # [ 42.323153] nix-grpc-daemon[924]: time=2026-09-22T10:44:44.324Z level=INFO msg="Upload complete. (456ms)"4657lb # [ 42.343676] niks3-server[1077]: 2026/09/22 10:44:44 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4658worker1 # [ 42.327234] nix-grpc-daemon[845]: ts=2026-09-22T10:44:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=0 paths=1 nar_bytes_in=5044659worker1 # [ 42.334354] nix-grpc-daemon[845]: ts=2026-09-22T10:44:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560724660worker1 # [ 42.343501] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4661worker1 # [ 42.351973] nix-daemon[911]: reaped child process 922, status = succeeded4662client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4663client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4664worker1 # [ 42.357546] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4665client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4666client # farm-a> LOG-farm-a-t14667worker1 # [ 43.171029] nix-daemon[911]: reaped child process 949, status = succeeded4668worker1 # [ 43.303826] nix-daemon[911]: accepted connection from pid 966, user nix-grpc-daemon (trusted)4669worker1 # [ 43.315830] nix-daemon[911]: reaped child process 969, status = succeeded4670lb # [ 43.343567] niks3-server[1077]: 2026/09/22 10:44:45 INFO Received uploads request method=POST path=/api/pending_closures4671worker1 # [ 43.336112] nix-grpc-daemon[924]: time=2026-09-22T10:44:45.337Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4672worker1 # [ 43.337680] nix-grpc-daemon[924]: time=2026-09-22T10:44:45.338Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4673lb # [ 43.380768] niks3-server[1077]: 2026/09/22 10:44:45 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4674lb # [ 43.388159] niks3-server[1077]: 2026/09/22 10:44:45 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4675lb # [ 43.394281] niks3-server[1077]: 2026/09/22 10:44:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4676worker1 # [ 43.386711] nix-grpc-daemon[924]: time=2026-09-22T10:44:45.387Z level=INFO msg="Uploading 1 narinfos"4677lb # [ 43.399768] niks3-server[1077]: 2026/09/22 10:44:45 INFO Signed narinfos id=2 count=14678lb # [ 43.405339] niks3-server[1077]: 2026/09/22 10:44:45 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4679lb # [ 43.411137] niks3-server[1077]: 2026/09/22 10:44:45 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4680lb # [ 43.414397] niks3-server[1077]: 2026/09/22 10:44:45 INFO Completed upload id=24681worker1 # [ 43.402243] nix-grpc-daemon[924]: time=2026-09-22T10:44:45.403Z level=INFO msg="Upload complete. (237ms)"4682lb # [ 43.416325] niks3-server[1077]: 2026/09/22 10:44:45 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4683worker1 # [ 43.405611] nix-grpc-daemon[845]: ts=2026-09-22T10:44:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14684worker1 # [ 43.411774] nix-daemon[911]: reaped child process 947, status = succeeded4685worker2 # [ 43.414396] nix-grpc-daemon[844]: ts=2026-09-22T10:44:45Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04686worker2 # [ 43.416367] nix-grpc-daemon[844]: ts=2026-09-22T10:44:45Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14687worker1 # [ 43.461562] nix-grpc-daemon[845]: ts=2026-09-22T10:44:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560724688worker1 # [ 43.476269] nix-grpc-daemon[845]: ts=2026-09-22T10:44:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:560724689worker1 # [ 43.491441] nix-grpc-daemon[924]: time=2026-09-22T10:44:45.492Z level=INFO msg="All 1 paths already cached"4690client # copying 1 paths to 'lb:50051'...4691worker1 # [ 43.500107] nix-grpc-daemon[845]: ts=2026-09-22T10:44:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:560724692client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4693worker1 # [ 43.518798] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4694worker1 # [ 43.674285] nix-daemon[911]: accepted connection from pid 973, user nix-grpc-daemon (trusted)4695worker1 # [ 43.688150] nix-daemon[911]: reaped child process 976, status = succeeded4696worker1 # [ 43.822838] nix-daemon[911]: accepted connection from pid 978, user nix-grpc-daemon (trusted)4697worker1 # [ 43.835422] nix-daemon[911]: reaped child process 981, status = succeeded4698lb # [ 43.855415] niks3-server[1077]: 2026/09/22 10:44:45 INFO Received uploads request method=POST path=/api/pending_closures4699worker1 # [ 43.850721] nix-grpc-daemon[924]: time=2026-09-22T10:44:45.852Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4700worker1 # [ 43.852182] nix-grpc-daemon[924]: time=2026-09-22T10:44:45.853Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4701lb # [ 43.887058] niks3-server[1077]: 2026/09/22 10:44:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4702lb # [ 43.892316] niks3-server[1077]: 2026/09/22 10:44:45 INFO Signed narinfos id=3 count=14703worker1 # [ 43.880563] nix-grpc-daemon[924]: time=2026-09-22T10:44:45.881Z level=INFO msg="Uploading 1 narinfos"4704lb # [ 43.895246] niks3-server[1077]: 2026/09/22 10:44:45 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4705lb # [ 43.898943] niks3-server[1077]: 2026/09/22 10:44:45 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4706lb # [ 43.903554] niks3-server[1077]: 2026/09/22 10:44:45 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4707lb # [ 43.909163] niks3-server[1077]: 2026/09/22 10:44:45 INFO Completed upload id=34708worker1 # [ 43.895757] nix-grpc-daemon[924]: time=2026-09-22T10:44:45.897Z level=INFO msg="Upload complete. (362ms)"4709lb # [ 43.910853] niks3-server[1077]: 2026/09/22 10:44:45 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4710worker1 # [ 43.897907] nix-grpc-daemon[845]: ts=2026-09-22T10:44:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=0 paths=1 nar_bytes_in=6244711worker1 # [ 43.901119] nix-grpc-daemon[845]: ts=2026-09-22T10:44:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560724712worker1 # [ 43.905772] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4713worker1 # [ 43.910375] nix-daemon[911]: reaped child process 971, status = succeeded4714client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4715client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4716worker1 # [ 43.916096] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4717client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4718client # farm-b> LOG-farm-b-t14719worker1 # [ 44.728813] nix-daemon[911]: reaped child process 985, status = succeeded4720worker1 # [ 44.873189] nix-daemon[911]: accepted connection from pid 1001, user nix-grpc-daemon (trusted)4721worker1 # [ 44.886762] nix-daemon[911]: reaped child process 1004, status = succeeded4722lb # [ 44.915596] niks3-server[1077]: 2026/09/22 10:44:46 INFO Received uploads request method=POST path=/api/pending_closures4723worker1 # [ 44.910713] nix-grpc-daemon[924]: time=2026-09-22T10:44:46.911Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4724worker1 # [ 44.912549] nix-grpc-daemon[924]: time=2026-09-22T10:44:46.913Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4725lb # [ 44.954125] niks3-server[1077]: 2026/09/22 10:44:46 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4726lb # [ 44.962188] niks3-server[1077]: 2026/09/22 10:44:46 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4727lb # [ 44.973531] niks3-server[1077]: 2026/09/22 10:44:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4728lb # [ 44.979610] niks3-server[1077]: 2026/09/22 10:44:46 INFO Signed narinfos id=4 count=14729worker1 # [ 44.966586] nix-grpc-daemon[924]: time=2026-09-22T10:44:46.967Z level=INFO msg="Uploading 1 narinfos"4730lb # [ 44.983392] niks3-server[1077]: 2026/09/22 10:44:46 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4731lb # [ 44.993061] niks3-server[1077]: 2026/09/22 10:44:46 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4732lb # [ 44.998560] niks3-server[1077]: 2026/09/22 10:44:46 INFO Completed upload id=44733lb # [ 45.000292] niks3-server[1077]: 2026/09/22 10:44:46 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4734worker1 # [ 44.985431] nix-grpc-daemon[924]: time=2026-09-22T10:44:46.986Z level=INFO msg="Upload complete. (263ms)"4735worker2 # [ 44.992817] nix-grpc-daemon[844]: ts=2026-09-22T10:44:46Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04736worker1 # [ 44.992175] nix-grpc-daemon[845]: ts=2026-09-22T10:44:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14737worker2 # [ 44.994707] nix-grpc-daemon[844]: ts=2026-09-22T10:44:46Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14738worker1 # [ 45.005528] nix-daemon[911]: reaped child process 983, status = succeeded4739worker1 # [ 45.039489] nix-grpc-daemon[845]: ts=2026-09-22T10:44:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560724740worker1 # [ 45.058231] nix-grpc-daemon[845]: ts=2026-09-22T10:44:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:560724741worker1 # [ 45.071485] nix-grpc-daemon[924]: time=2026-09-22T10:44:47.072Z level=INFO msg="All 2 paths already cached"4742client # copying 1 paths to 'lb:50051'...4743worker1 # [ 45.080202] nix-grpc-daemon[845]: ts=2026-09-22T10:44:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:560724744client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4745worker1 # [ 45.100178] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4746worker1 # [ 45.245449] nix-daemon[911]: accepted connection from pid 1008, user nix-grpc-daemon (trusted)4747worker1 # [ 45.258152] nix-daemon[911]: reaped child process 1011, status = succeeded4748worker1 # [ 45.408277] nix-daemon[911]: accepted connection from pid 1013, user nix-grpc-daemon (trusted)4749worker1 # [ 45.421156] nix-daemon[911]: reaped child process 1016, status = succeeded4750lb # [ 45.449139] niks3-server[1077]: 2026/09/22 10:44:47 INFO Received uploads request method=POST path=/api/pending_closures4751worker1 # [ 45.444209] nix-grpc-daemon[924]: time=2026-09-22T10:44:47.445Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4752worker1 # [ 45.446168] nix-grpc-daemon[924]: time=2026-09-22T10:44:47.447Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4753lb # [ 45.477797] niks3-server[1077]: 2026/09/22 10:44:47 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4754lb # [ 45.488700] niks3-server[1077]: 2026/09/22 10:44:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4755worker1 # [ 45.479691] nix-grpc-daemon[924]: time=2026-09-22T10:44:47.480Z level=INFO msg="Uploading 1 narinfos"4756lb # [ 45.493117] niks3-server[1077]: 2026/09/22 10:44:47 INFO Signed narinfos id=5 count=14757lb # [ 45.499784] niks3-server[1077]: 2026/09/22 10:44:47 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4758lb # [ 45.509606] niks3-server[1077]: 2026/09/22 10:44:47 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4759lb # [ 45.520417] niks3-server[1077]: 2026/09/22 10:44:47 INFO Completed upload id=54760worker1 # [ 45.507281] nix-grpc-daemon[924]: time=2026-09-22T10:44:47.508Z level=INFO msg="Upload complete. (394ms)"4761lb # [ 45.523713] niks3-server[1077]: 2026/09/22 10:44:47 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4762worker1 # [ 45.512139] nix-grpc-daemon[845]: ts=2026-09-22T10:44:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=0 paths=1 nar_bytes_in=7524763worker1 # [ 45.516854] nix-grpc-daemon[845]: ts=2026-09-22T10:44:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560724764worker1 # [ 45.525659] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4765worker1 # [ 45.532821] nix-daemon[911]: reaped child process 1006, status = succeeded4766client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4767worker1 # [ 45.535289] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4768client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4769client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4770client # farm-top> LOG-farm-top-t14771worker1 # [ 46.560335] nix-daemon[911]: reaped child process 1020, status = succeeded4772worker1 # [ 46.710984] nix-daemon[911]: accepted connection from pid 1036, user nix-grpc-daemon (trusted)4773worker1 # [ 46.722851] nix-daemon[911]: reaped child process 1039, status = succeeded4774lb # [ 46.752201] niks3-server[1077]: 2026/09/22 10:44:48 INFO Received uploads request method=POST path=/api/pending_closures4775worker1 # [ 46.749126] nix-grpc-daemon[924]: time=2026-09-22T10:44:48.750Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4776worker1 # [ 46.752057] nix-grpc-daemon[924]: time=2026-09-22T10:44:48.752Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4777lb # [ 46.796767] niks3-server[1077]: 2026/09/22 10:44:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4778lb # [ 46.801604] niks3-server[1077]: 2026/09/22 10:44:48 INFO Signed narinfos id=6 count=14779worker1 # [ 46.791683] nix-grpc-daemon[924]: time=2026-09-22T10:44:48.792Z level=INFO msg="Uploading 1 narinfos"4780lb # [ 46.806937] niks3-server[1077]: 2026/09/22 10:44:48 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4781lb # [ 46.815413] niks3-server[1077]: 2026/09/22 10:44:48 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4782lb # [ 46.823151] niks3-server[1077]: 2026/09/22 10:44:48 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4783lb # [ 46.830876] niks3-server[1077]: 2026/09/22 10:44:48 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4784lb # [ 46.836529] niks3-server[1077]: 2026/09/22 10:44:48 INFO Completed upload id=64785lb # [ 46.838133] niks3-server[1077]: 2026/09/22 10:44:48 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4786worker1 # [ 46.823441] nix-grpc-daemon[924]: time=2026-09-22T10:44:48.824Z level=INFO msg="Upload complete. (279ms)"4787worker1 # [ 46.829636] nix-grpc-daemon[845]: ts=2026-09-22T10:44:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14788worker1 # [ 46.843513] nix-daemon[911]: reaped child process 1018, status = succeeded4789client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 7.15 seconds)4790client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4791client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.18 seconds)4792client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4793client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.02 seconds)4794client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4795client # copying 3 paths...4796client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4797client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4798client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4799client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.27 seconds)4800(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.67 seconds)4801subtest: repeat is answered Cached by the scheduler without building4802worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4803worker1 # finding garbage collector roots...4804worker1 # deleting specified paths...4805worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4806worker1 # deleting unused links...4807worker1 # note: hard linking is currently saving 0.0 KiB4808worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.21 seconds)4809worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4810worker2 # finding garbage collector roots...4811worker2 # deleting specified paths...4812worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4813worker2 # deleting unused links...4814worker2 # note: hard linking is currently saving 0.0 KiB4815worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.27 seconds)4816client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24817worker2 # [ 48.094602] nix-grpc-daemon[844]: ts=2026-09-22T10:44:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:340444818client # don't know how to build these paths:4819client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4820worker2 # [ 48.102835] nix-grpc-daemon[844]: ts=2026-09-22T10:44:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340444821worker2 # [ 48.133364] nix-grpc-daemon[844]: ts=2026-09-22T10:44:50Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14822client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.28 seconds)4823client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4824client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.16 seconds)4825lb # [ 48.393237] envoy[987]: conn peer=[2001:db8:1::1]:36920 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=715 tx=364 ms=15394826lb # [ 48.394748] envoy[987]: conn peer=[2001:db8:1::1]:36936 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=362 ms=474827lb # [ 48.396100] envoy[987]: conn peer=[2001:db8:1::1]:36946 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=199 ms=254828lb # [ 48.397402] envoy[987]: conn peer=[2001:db8:1::1]:36960 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=84829lb # [ 48.398754] envoy[987]: conn peer=[2001:db8:1::1]:36976 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9481 tx=4056 ms=69174830lb # [ 48.400139] envoy[987]: conn peer=[2001:db8:1::1]:48064 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=524831lb: must succeed: curl -sf localhost:9901/clusters4832lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4833lb: must succeed: curl -sf localhost:9901/clusters4834lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4835worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4836worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4837(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.14 seconds)4838subtest: inputs already in the cache are not sent to the scheduler4839lb: must succeed: curl -sf localhost:9901/clusters4840lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4841lb: must succeed: curl -sf localhost:9901/clusters4842lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4843worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4844worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4845client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24846client # don't know how to build these paths:4847client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4848worker1 # [ 48.898122] nix-grpc-daemon[845]: ts=2026-09-22T10:44:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:560724849worker2 # [ 48.898448] nix-grpc-daemon[844]: ts=2026-09-22T10:44:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340444850worker2 # [ 48.905472] nix-grpc-daemon[844]: ts=2026-09-22T10:44:50Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04851worker2 # [ 48.907393] nix-grpc-daemon[844]: ts=2026-09-22T10:44:50Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14852worker1 # [ 48.951320] nix-grpc-daemon[845]: ts=2026-09-22T10:44:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560724853worker1 # [ 48.958296] nix-grpc-daemon[845]: ts=2026-09-22T10:44:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:560724854worker1 # [ 48.970109] nix-grpc-daemon[924]: time=2026-09-22T10:44:50.970Z level=INFO msg="All 2 paths already cached"4855client # copying 1 paths to 'lb:50051'...4856client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4857worker1 # [ 48.978152] nix-grpc-daemon[845]: ts=2026-09-22T10:44:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:560724858worker1 # [ 48.986746] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4859worker1 # [ 49.122316] nix-daemon[911]: accepted connection from pid 1056, user nix-grpc-daemon (trusted)4860worker1 # [ 49.134973] nix-daemon[911]: reaped child process 1059, status = succeeded4861worker1 # [ 49.256927] nix-daemon[911]: accepted connection from pid 1061, user nix-grpc-daemon (trusted)4862worker1 # [ 49.269900] nix-daemon[911]: reaped child process 1064, status = succeeded4863lb # [ 49.291677] niks3-server[1077]: 2026/09/22 10:44:51 INFO Received uploads request method=POST path=/api/pending_closures4864worker1 # [ 49.283318] nix-grpc-daemon[924]: time=2026-09-22T10:44:51.284Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4865worker1 # [ 49.284713] nix-grpc-daemon[924]: time=2026-09-22T10:44:51.284Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4866lb # [ 49.307661] niks3-server[1077]: 2026/09/22 10:44:51 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4867lb # [ 49.310951] niks3-server[1077]: 2026/09/22 10:44:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4868lb # [ 49.313559] niks3-server[1077]: 2026/09/22 10:44:51 INFO Signed narinfos id=7 count=14869worker1 # [ 49.301632] nix-grpc-daemon[924]: time=2026-09-22T10:44:51.302Z level=INFO msg="Uploading 1 narinfos"4870lb # [ 49.317947] niks3-server[1077]: 2026/09/22 10:44:51 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4871lb # [ 49.322502] niks3-server[1077]: 2026/09/22 10:44:51 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4872lb # [ 49.326646] niks3-server[1077]: 2026/09/22 10:44:51 INFO Completed upload id=74873worker1 # [ 49.313249] nix-grpc-daemon[924]: time=2026-09-22T10:44:51.314Z level=INFO msg="Upload complete. (312ms)"4874lb # [ 49.327965] niks3-server[1077]: 2026/09/22 10:44:51 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4875worker1 # [ 49.315368] nix-grpc-daemon[845]: ts=2026-09-22T10:44:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=0 paths=1 nar_bytes_in=7524876worker1 # [ 49.318404] nix-grpc-daemon[845]: ts=2026-09-22T10:44:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560724877worker1 # [ 49.323323] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4878worker1 # [ 49.327960] nix-daemon[911]: reaped child process 1054, status = succeeded4879worker1 # [ 49.332471] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)4880client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4881client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4882client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4883client # farm-top> LOG-farm-top-t1b4884worker1 # [ 50.160751] nix-daemon[911]: reaped child process 1068, status = succeeded4885worker1 # [ 50.295484] nix-daemon[911]: accepted connection from pid 1084, user nix-grpc-daemon (trusted)4886worker1 # [ 50.307578] nix-daemon[911]: reaped child process 1087, status = succeeded4887lb # [ 50.330332] niks3-server[1077]: 2026/09/22 10:44:52 INFO Received uploads request method=POST path=/api/pending_closures4888worker1 # [ 50.326613] nix-grpc-daemon[924]: time=2026-09-22T10:44:52.327Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4889worker1 # [ 50.328137] nix-grpc-daemon[924]: time=2026-09-22T10:44:52.329Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4890lb # [ 50.368681] niks3-server[1077]: 2026/09/22 10:44:52 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4891lb # [ 50.385642] niks3-server[1077]: 2026/09/22 10:44:52 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4892lb # [ 50.393409] niks3-server[1077]: 2026/09/22 10:44:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4893lb # [ 50.398042] niks3-server[1077]: 2026/09/22 10:44:52 INFO Signed narinfos id=8 count=14894worker1 # [ 50.387379] nix-grpc-daemon[924]: time=2026-09-22T10:44:52.388Z level=INFO msg="Uploading 1 narinfos"4895lb # [ 50.408662] niks3-server[1077]: 2026/09/22 10:44:52 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4896lb # [ 50.411858] niks3-server[1077]: 2026/09/22 10:44:52 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4897lb # [ 50.416140] niks3-server[1077]: 2026/09/22 10:44:52 INFO Completed upload id=84898worker1 # [ 50.402780] nix-grpc-daemon[924]: time=2026-09-22T10:44:52.404Z level=INFO msg="Upload complete. (263ms)"4899lb # [ 50.417868] niks3-server[1077]: 2026/09/22 10:44:52 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4900worker1 # [ 50.409636] nix-grpc-daemon[845]: ts=2026-09-22T10:44:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14901worker1 # [ 50.422484] nix-daemon[911]: reaped child process 1066, status = succeeded4902client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.74 seconds)4903client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4904client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.15 seconds)4905lb: must succeed: curl -sf localhost:9901/clusters4906lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4907lb: must succeed: curl -sf localhost:9901/clusters4908lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4909worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4910worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4911(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.18 seconds)4912subtest: two clients wanting the same drv build it once4913client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4914client # [ 50.785942] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4915client # Running as unit: dedup1.service; invocation ID: e6a629b4318c497babe39fdabf330e9c4916client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.06 seconds)4917client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4918worker2 # [ 50.972897] nix-grpc-daemon[844]: ts=2026-09-22T10:44:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:340444919client # [ 50.991806] nix[888]: don't know how to build these paths:4920client # [ 50.993307] nix[888]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4921worker2 # [ 50.983335] nix-grpc-daemon[844]: ts=2026-09-22T10:44:52Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04922worker2 # [ 50.985988] nix-grpc-daemon[844]: ts=2026-09-22T10:44:52Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=14923worker2 # [ 50.988740] nix-grpc-daemon[844]: ts=2026-09-22T10:44:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340444924worker2 # [ 51.003062] nix-grpc-daemon[844]: ts=2026-09-22T10:44:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340444925client # [ 51.025089] nix[888]: copying 1 paths to 'lb:50051'...4926client # [ 51.026039] nix[888]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4927worker2 # [ 51.010320] nix-grpc-daemon[844]: ts=2026-09-22T10:44:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:340444928worker2 # [ 51.017707] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)4929worker2 # [ 51.031447] nix-grpc-daemon[844]: ts=2026-09-22T10:44:52Z level=info event=niks3_push_started pid=9824930worker2 # [ 51.304365] nix-grpc-daemon[982]: time=2026-09-22T10:44:53.267Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""4931worker2 # [ 51.507559] nix-daemon[931]: accepted connection from pid 995, user nix-grpc-daemon (trusted)4932worker2 # [ 51.524365] nix-daemon[931]: reaped child process 998, status = succeeded4933worker2 # [ 51.682841] nix-daemon[931]: accepted connection from pid 1000, user nix-grpc-daemon (trusted)4934worker2 # [ 51.696440] nix-daemon[931]: reaped child process 1003, status = succeeded4935lb # [ 51.724599] niks3-server[1077]: 2026/09/22 10:44:53 INFO Received uploads request method=POST path=/api/pending_closures4936worker2 # [ 51.713232] nix-grpc-daemon[982]: time=2026-09-22T10:44:53.680Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4937worker2 # [ 51.715351] nix-grpc-daemon[982]: time=2026-09-22T10:44:53.682Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4938lb # [ 51.757239] niks3-server[1077]: 2026/09/22 10:44:53 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4939lb # [ 51.768220] niks3-server[1077]: 2026/09/22 10:44:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4940lb # [ 51.773256] niks3-server[1077]: 2026/09/22 10:44:53 INFO Signed narinfos id=9 count=14941worker2 # [ 51.760254] nix-grpc-daemon[982]: time=2026-09-22T10:44:53.727Z level=INFO msg="Uploading 1 narinfos"4942lb # [ 51.781705] niks3-server[1077]: 2026/09/22 10:44:53 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4943lb # [ 51.794947] niks3-server[1077]: 2026/09/22 10:44:53 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4944lb # [ 51.798467] niks3-server[1077]: 2026/09/22 10:44:53 INFO Completed upload id=94945worker2 # [ 51.782692] nix-grpc-daemon[982]: time=2026-09-22T10:44:53.750Z level=INFO msg="Upload complete. (463ms)"4946lb # [ 51.800707] niks3-server[1077]: 2026/09/22 10:44:53 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4947worker2 # [ 51.784854] nix-grpc-daemon[844]: ts=2026-09-22T10:44:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_in=4324948worker2 # [ 51.789470] nix-grpc-daemon[844]: ts=2026-09-22T10:44:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340444949worker2 # [ 51.795270] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)4950worker2 # [ 51.797546] nix-daemon[931]: reaped child process 980, status = succeeded4951worker2 # [ 51.803491] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)4952client # [ 51.817644] nix[888]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4953client # [ 51.857420] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4954client # Running as unit: dedup2.service; invocation ID: 843c82ed814c400fb021724cea20fd8a4955client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.08 seconds)4956worker1 # [ 52.103081] nix-grpc-daemon[845]: ts=2026-09-22T10:44:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:560724957client # [ 52.120478] nix[904]: don't know how to build these paths:4958client # [ 52.121525] nix[904]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4959worker2 # [ 52.113549] nix-grpc-daemon[844]: ts=2026-09-22T10:44:54Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04960client # [ 52.135089] nix[904]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4961worker2 # [ 52.118966] nix-grpc-daemon[844]: ts=2026-09-22T10:44:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340444962client: must succeed: sleep 24963client: (finished: must succeed: sleep 2, in 2.04 seconds)4964client: waiting for success: ! systemctl is-active dedup1 dedup24965worker2 # [ 56.634809] nix-daemon[931]: reaped child process 1007, status = succeeded4966worker2 # [ 56.783537] nix-daemon[931]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)4967worker2 # [ 56.797449] nix-daemon[931]: reaped child process 1046, status = succeeded4968lb # [ 56.824425] niks3-server[1077]: 2026/09/22 10:44:58 INFO Received uploads request method=POST path=/api/pending_closures4969worker2 # [ 56.814746] nix-grpc-daemon[982]: time=2026-09-22T10:44:58.781Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4970worker2 # [ 56.816534] nix-grpc-daemon[982]: time=2026-09-22T10:44:58.781Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4971lb # [ 56.859982] niks3-server[1077]: 2026/09/22 10:44:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4972lb # [ 56.862697] niks3-server[1077]: 2026/09/22 10:44:58 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4973lb # [ 56.864831] niks3-server[1077]: 2026/09/22 10:44:58 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4974lb # [ 56.867954] niks3-server[1077]: 2026/09/22 10:44:58 INFO Signed narinfos id=10 count=14975lb # [ 56.869462] niks3-server[1077]: 2026/09/22 10:44:58 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4976worker2 # [ 56.854275] nix-grpc-daemon[982]: time=2026-09-22T10:44:58.820Z level=INFO msg="Uploading 1 narinfos"4977lb # [ 56.876252] niks3-server[1077]: 2026/09/22 10:44:58 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4978lb # [ 56.879975] niks3-server[1077]: 2026/09/22 10:44:58 INFO Completed upload id=104979worker2 # [ 56.865512] nix-grpc-daemon[982]: time=2026-09-22T10:44:58.832Z level=INFO msg="Upload complete. (247ms)"4980lb # [ 56.881662] niks3-server[1077]: 2026/09/22 10:44:58 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo4981worker2 # [ 56.870148] nix-grpc-daemon[844]: ts=2026-09-22T10:44:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14982worker2 # [ 56.872945] nix-grpc-daemon[844]: ts=2026-09-22T10:44:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14983worker2 # [ 56.879353] nix-daemon[931]: reaped child process 1005, status = succeeded4984client # [ 56.923505] systemd[1]: dedup2.service: Deactivated successfully.4985client # [ 56.927334] systemd[1]: dedup2.service: Consumed 146ms CPU time over 5.069s wall clock time, 37.2M memory peak, 5.9K incoming IP traffic, 5.7K outgoing IP traffic.4986client # [ 56.966731] systemd[1]: dedup1.service: Deactivated successfully.4987client # [ 56.968393] systemd[1]: dedup1.service: Consumed 129ms CPU time over 6.182s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.5K outgoing IP traffic.4988client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.07 seconds)4989client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4990client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.03 seconds)4991worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4992worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)4993worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4994worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.03 seconds)4995worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4996worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4997worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4998worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4999(finished: subtest: two clients wanting the same drv build it once, in 7.00 seconds)5000subtest: --timeout and --max-silent-time reach the builder5001worker1 # [ 57.960534] nix-grpc-daemon[845]: ts=2026-09-22T10:44:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:560725002worker2 # [ 57.971819] nix-grpc-daemon[844]: ts=2026-09-22T10:44:59Z level=debug event=want client=6 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv cached=05003worker2 # [ 57.973987] nix-grpc-daemon[844]: ts=2026-09-22T10:44:59Z level=debug event=assigned drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv worker=192.168.1.3:50051 clients=15004worker1 # [ 57.976773] nix-grpc-daemon[845]: ts=2026-09-22T10:44:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560725005worker1 # [ 57.985395] nix-grpc-daemon[845]: ts=2026-09-22T10:44:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:560725006worker1 # [ 57.993265] nix-grpc-daemon[845]: ts=2026-09-22T10:44:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:560725007worker1 # [ 57.998633] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5008worker1 # [ 58.155254] nix-daemon[911]: accepted connection from pid 1116, user nix-grpc-daemon (trusted)5009worker1 # [ 58.170167] nix-daemon[911]: reaped child process 1119, status = succeeded5010worker1 # [ 58.325093] nix-daemon[911]: accepted connection from pid 1121, user nix-grpc-daemon (trusted)5011worker1 # [ 58.336479] nix-daemon[911]: reaped child process 1124, status = succeeded5012lb # [ 58.367597] niks3-server[1077]: 2026/09/22 10:45:00 INFO Received uploads request method=POST path=/api/pending_closures5013worker1 # [ 58.366283] nix-grpc-daemon[924]: time=2026-09-22T10:45:00.367Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5014worker1 # [ 58.368162] nix-grpc-daemon[924]: time=2026-09-22T10:45:00.369Z level=INFO msg="Uploading qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv (424B)"5015lb # [ 58.394706] envoy[987]: conn peer=[2001:db8:1::1]:48066 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4195 tx=1781 ms=15175016lb # [ 58.399635] envoy[987]: conn peer=[2001:db8:1::1]:48080 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1440 tx=1085 ms=47955017lb # [ 58.403232] envoy[987]: conn peer=[2001:db8:1::1]:48078 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2998 tx=1451 ms=59595018lb # [ 58.412481] niks3-server[1077]: 2026/09/22 10:45:00 INFO Registered completed upload object_key=nar/1s4sciw269ksj1s7ymwz5cy4dw9n7bxv3k8wqw69h694spdh4ha2.nar.zst5019lb # [ 58.425586] niks3-server[1077]: 2026/09/22 10:45:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign5020lb # [ 58.430038] niks3-server[1077]: 2026/09/22 10:45:00 INFO Signed narinfos id=11 count=15021worker1 # [ 58.419334] nix-grpc-daemon[924]: time=2026-09-22T10:45:00.420Z level=INFO msg="Uploading 1 narinfos"5022lb # [ 58.436879] niks3-server[1077]: 2026/09/22 10:45:00 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.ls5023lb # [ 58.446385] niks3-server[1077]: 2026/09/22 10:45:00 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete5024lb # [ 58.450153] niks3-server[1077]: 2026/09/22 10:45:00 INFO Completed upload id=115025worker1 # [ 58.436966] nix-grpc-daemon[924]: time=2026-09-22T10:45:00.438Z level=INFO msg="Upload complete. (430ms)"5026lb # [ 58.452804] niks3-server[1077]: 2026/09/22 10:45:00 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.narinfo5027worker1 # [ 58.441349] nix-grpc-daemon[845]: ts=2026-09-22T10:45:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=0 paths=1 nar_bytes_in=4245028worker1 # [ 58.448220] nix-grpc-daemon[845]: ts=2026-09-22T10:45:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560725029worker1 # [ 58.460115] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5030worker1 # [ 58.467930] nix-daemon[911]: reaped child process 1114, status = succeeded5031worker1 # [ 58.476112] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5032worker1 # [ 64.891895] nix-grpc-daemon[845]: ts=2026-09-22T10:45:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=6 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv assign_id=6 outputs=05033worker1 # [ 64.920593] nix-daemon[911]: reaped child process 1126, status = succeeded5034worker1 # [ 64.923279] nix-daemon[911]: reaped child process 1128, status = succeeded5035worker1 # [ 65.117942] nix-grpc-daemon[845]: ts=2026-09-22T10:45:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:560725036worker2 # [ 65.128380] nix-grpc-daemon[844]: ts=2026-09-22T10:45:07Z level=debug event=want client=7 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv cached=05037worker2 # [ 65.132968] nix-grpc-daemon[844]: ts=2026-09-22T10:45:07Z level=debug event=assigned drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv worker=192.168.1.4:50051 clients=15038worker2 # [ 65.137835] nix-grpc-daemon[844]: ts=2026-09-22T10:45:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340445039worker2 # [ 65.149881] nix-grpc-daemon[844]: ts=2026-09-22T10:45:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340445040worker2 # [ 65.160985] nix-grpc-daemon[844]: ts=2026-09-22T10:45:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:340445041worker2 # [ 65.169898] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5042worker2 # [ 65.352747] nix-daemon[931]: accepted connection from pid 1063, user nix-grpc-daemon (trusted)5043worker2 # [ 65.367853] nix-daemon[931]: reaped child process 1066, status = succeeded5044worker2 # [ 65.533977] nix-daemon[931]: accepted connection from pid 1068, user nix-grpc-daemon (trusted)5045worker2 # [ 65.547343] nix-daemon[931]: reaped child process 1071, status = succeeded5046lb # [ 65.580133] niks3-server[1077]: 2026/09/22 10:45:07 INFO Received uploads request method=POST path=/api/pending_closures5047worker2 # [ 65.573389] nix-grpc-daemon[982]: time=2026-09-22T10:45:07.540Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5048worker2 # [ 65.575994] nix-grpc-daemon[982]: time=2026-09-22T10:45:07.542Z level=INFO msg="Uploading fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv (424B)"5049lb # [ 65.610482] niks3-server[1077]: 2026/09/22 10:45:07 INFO Registered completed upload object_key=nar/013y5sy2r6v0qdwv0dzpgvdfy08bds4mfh2qpvbgjazzrg2v71hp.nar.zst5050lb # [ 65.624797] niks3-server[1077]: 2026/09/22 10:45:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign5051lb # [ 65.631952] niks3-server[1077]: 2026/09/22 10:45:07 INFO Signed narinfos id=12 count=15052worker2 # [ 65.619609] nix-grpc-daemon[982]: time=2026-09-22T10:45:07.586Z level=INFO msg="Uploading 1 narinfos"5053lb # [ 65.637853] niks3-server[1077]: 2026/09/22 10:45:07 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.ls5054lb # [ 65.648323] niks3-server[1077]: 2026/09/22 10:45:07 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete5055lb # [ 65.654854] niks3-server[1077]: 2026/09/22 10:45:07 INFO Completed upload id=125056worker2 # [ 65.639195] nix-grpc-daemon[982]: time=2026-09-22T10:45:07.606Z level=INFO msg="Upload complete. (455ms)"5057lb # [ 65.658062] niks3-server[1077]: 2026/09/22 10:45:07 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.narinfo5058worker2 # [ 65.644129] nix-grpc-daemon[844]: ts=2026-09-22T10:45:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_in=4245059worker2 # [ 65.650061] nix-grpc-daemon[844]: ts=2026-09-22T10:45:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340445060worker2 # [ 65.658475] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5061worker2 # [ 65.668129] nix-daemon[931]: reaped child process 1061, status = succeeded5062worker2 # [ 65.672098] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5063lb # [ 68.394251] envoy[987]: conn peer=[2001:db8:1::1]:52578 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2996 tx=972 ms=69485064worker2 # [ 72.089350] nix-grpc-daemon[844]: ts=2026-09-22T10:45:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=6 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv assign_id=7 outputs=05065worker2 # [ 72.112257] nix-daemon[931]: reaped child process 1073, status = succeeded5066worker2 # [ 72.115432] nix-daemon[931]: reaped child process 1075, status = succeeded5067(finished: subtest: --timeout and --max-silent-time reach the builder, in 14.52 seconds)5068subtest: the first client leaving does not fail the second5069client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share5070client # [ 72.300931] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share.5071client # Running as unit: share1.service; invocation ID: f6dccec1e2114ef3a2ce3e156e378a6e5072client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share, in 0.07 seconds)5073client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share5074client # [ 72.509935] nix[978]: don't know how to build these paths:5075client # [ 72.510812] nix[978]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5076worker2 # [ 72.493343] nix-grpc-daemon[844]: ts=2026-09-22T10:45:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:340445077worker2 # [ 72.503753] nix-grpc-daemon[844]: ts=2026-09-22T10:45:14Z level=debug event=want client=8 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05078worker2 # [ 72.507837] nix-grpc-daemon[844]: ts=2026-09-22T10:45:14Z level=debug event=assigned drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv worker=192.168.1.3:50051 clients=15079worker1 # [ 72.517567] nix-grpc-daemon[845]: ts=2026-09-22T10:45:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560725080worker1 # [ 72.526425] nix-grpc-daemon[845]: ts=2026-09-22T10:45:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:560725081client # [ 72.546178] nix[978]: copying 1 paths to 'lb:50051'...5082client # [ 72.547048] nix[978]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' to 'lb:50051'...5083worker1 # [ 72.533634] nix-grpc-daemon[845]: ts=2026-09-22T10:45:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:560725084worker1 # [ 72.541264] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5085worker1 # [ 72.706966] nix-daemon[911]: accepted connection from pid 1151, user nix-grpc-daemon (trusted)5086worker1 # [ 72.720374] nix-daemon[911]: reaped child process 1154, status = succeeded5087worker1 # [ 72.883659] nix-daemon[911]: accepted connection from pid 1156, user nix-grpc-daemon (trusted)5088worker1 # [ 72.897578] nix-daemon[911]: reaped child process 1159, status = succeeded5089lb # [ 72.926360] niks3-server[1077]: 2026/09/22 10:45:14 INFO Received uploads request method=POST path=/api/pending_closures5090worker1 # [ 72.919934] nix-grpc-daemon[924]: time=2026-09-22T10:45:14.920Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5091worker1 # [ 72.921360] nix-grpc-daemon[924]: time=2026-09-22T10:45:14.920Z level=INFO msg="Uploading j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv (432B)"5092lb # [ 72.947888] niks3-server[1077]: 2026/09/22 10:45:14 INFO Registered completed upload object_key=nar/0s0b0gfhn36p0aylk1qa64cgw01bi7a8ca93iv4zg2q7wqcjcqhs.nar.zst5093lb # [ 72.949996] niks3-server[1077]: 2026/09/22 10:45:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign5094lb # [ 72.953576] niks3-server[1077]: 2026/09/22 10:45:14 INFO Signed narinfos id=13 count=15095worker1 # [ 72.940383] nix-grpc-daemon[924]: time=2026-09-22T10:45:14.941Z level=INFO msg="Uploading 1 narinfos"5096lb # [ 72.956993] niks3-server[1077]: 2026/09/22 10:45:14 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.ls5097lb # [ 72.962412] niks3-server[1077]: 2026/09/22 10:45:14 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete5098lb # [ 72.967343] niks3-server[1077]: 2026/09/22 10:45:14 INFO Completed upload id=135099worker1 # [ 72.955493] nix-grpc-daemon[924]: time=2026-09-22T10:45:14.956Z level=INFO msg="Upload complete. (407ms)"5100lb # [ 72.969448] niks3-server[1077]: 2026/09/22 10:45:14 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.narinfo5101worker1 # [ 72.959086] nix-grpc-daemon[845]: ts=2026-09-22T10:45:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=0 paths=1 nar_bytes_in=4325102worker1 # [ 72.961566] nix-grpc-daemon[845]: ts=2026-09-22T10:45:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560725103worker1 # [ 72.968494] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5104worker1 # [ 72.971522] nix-daemon[911]: reaped child process 1149, status = succeeded5105worker1 # [ 72.977145] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5106client # [ 72.989893] nix[978]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5107client # [ 73.361579] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share.5108client # Running as unit: share2.service; invocation ID: 0b7b1fc4147a4ef7877cdc102baa69d95109client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share, in 1.05 seconds)5110client # [ 73.542426] nix[994]: don't know how to build these paths:5111client # [ 73.543273] nix[994]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5112worker2 # [ 73.526057] nix-grpc-daemon[844]: ts=2026-09-22T10:45:15Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:340445113worker2 # [ 73.532746] nix-grpc-daemon[844]: ts=2026-09-22T10:45:15Z level=debug event=want client=9 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05114worker1 # [ 73.538462] nix-grpc-daemon[845]: ts=2026-09-22T10:45:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560725115client # [ 73.556302] nix[994]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5116client: must succeed: sleep 2; systemctl kill -s INT share15117client # [ 76.624738] systemd[1]: share1.service: Sent signal SIGINT to main process 978 (nix) on client request.5118client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.05 seconds)5119client: waiting for success: ! systemctl is-active share15120client # [ 76.632937] nix[978]: shutting downj2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5121client # [ 76.640149] nix[978]: error: interrupted by the user5122client # [ 76.685082] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE5123client # [ 76.686990] systemd[1]: share1.service: Failed with result 'exit-code'.5124client # [ 76.688821] systemd[1]: share1.service: Consumed 124ms CPU time over 4.386s wall clock time, 41.4M memory peak, 6K incoming IP traffic, 8.1K outgoing IP traffic.5125client: (finished: waiting for success: ! systemctl is-active share1, in 1.11 seconds)5126client: waiting for success: ! systemctl is-active share25127worker1 # [ 78.002771] nix-daemon[911]: reaped child process 1163, status = succeeded5128worker1 # [ 78.147329] nix-daemon[911]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5129worker1 # [ 78.160425] nix-daemon[911]: reaped child process 1200, status = succeeded5130lb # [ 78.183919] niks3-server[1077]: 2026/09/22 10:45:20 INFO Received uploads request method=POST path=/api/pending_closures5131worker1 # [ 78.176502] nix-grpc-daemon[924]: time=2026-09-22T10:45:20.177Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5132worker1 # [ 78.178127] nix-grpc-daemon[924]: time=2026-09-22T10:45:20.179Z level=INFO msg="Uploading b9lg6znd0rinl36gy6ixq1dcj6b8bhhw-slow-share (120B)"5133lb # [ 78.205078] niks3-server[1077]: 2026/09/22 10:45:20 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst5134lb # [ 78.208736] niks3-server[1077]: 2026/09/22 10:45:20 INFO Registered completed upload object_key=log/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5135lb # [ 78.210984] niks3-server[1077]: 2026/09/22 10:45:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign5136lb # [ 78.212937] niks3-server[1077]: 2026/09/22 10:45:20 INFO Signed narinfos id=14 count=15137worker1 # [ 78.200871] nix-grpc-daemon[924]: time=2026-09-22T10:45:20.202Z level=INFO msg="Uploading 1 narinfos"5138lb # [ 78.217212] niks3-server[1077]: 2026/09/22 10:45:20 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.ls5139lb # [ 78.221250] niks3-server[1077]: 2026/09/22 10:45:20 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete5140lb # [ 78.224616] niks3-server[1077]: 2026/09/22 10:45:20 INFO Completed upload id=145141worker1 # [ 78.212495] nix-grpc-daemon[924]: time=2026-09-22T10:45:20.213Z level=INFO msg="Upload complete. (224ms)"5142lb # [ 78.226610] niks3-server[1077]: 2026/09/22 10:45:20 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.narinfo5143worker1 # [ 78.216062] nix-grpc-daemon[845]: ts=2026-09-22T10:45:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=4 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=8 outputs=15144worker1 # [ 78.218310] nix-grpc-daemon[845]: ts=2026-09-22T10:45:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=5 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=8 outputs=15145worker1 # [ 78.223735] nix-daemon[911]: reaped child process 1161, status = succeeded5146client # [ 78.307471] systemd[1]: share2.service: Deactivated successfully.5147client # [ 78.311177] systemd[1]: share2.service: Consumed 109ms CPU time over 4.948s wall clock time, 35.4M memory peak, 6K incoming IP traffic, 5.7K outgoing IP traffic.5148lb # [ 78.394955] envoy[987]: conn peer=[2001:db8:1::1]:53918 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2969 tx=977 ms=69825149lb # [ 78.396531] envoy[987]: conn peer=[2001:db8:1::1]:53920 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2995 tx=786 ms=41455150lb # [ 78.397907] envoy[987]: conn peer=[2001:db8:1::1]:59710 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1453 tx=1085 ms=47385151client: (finished: waiting for success: ! systemctl is-active share2, in 1.09 seconds)5152client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }5153client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.04 seconds)5154(finished: subtest: the first client leaving does not fail the second, in 6.84 seconds)5155subtest: build hook: nix-daemon with builders = grpc://lb5156client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log5157client # [ 79.242564] systemd[1]: Started Nix Daemon.5158client # [ 79.384474] nix-daemon[1046]: accepted connection from pid 1043, user root (trusted)5159worker1 # [ 79.544984] nix-grpc-daemon[845]: ts=2026-09-22T10:45:21Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:560725160worker1 # [ 79.549599] nix-grpc-daemon[845]: ts=2026-09-22T10:45:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:560725161worker2 # [ 79.559290] nix-grpc-daemon[844]: ts=2026-09-22T10:45:21Z level=debug event=want client=10 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05162worker2 # [ 79.562388] nix-grpc-daemon[844]: ts=2026-09-22T10:45:21Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.4:50051 clients=15163worker2 # [ 79.565108] nix-grpc-daemon[844]: ts=2026-09-22T10:45:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340445164worker2 # [ 79.571637] nix-grpc-daemon[844]: ts=2026-09-22T10:45:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340445165worker2 # [ 79.577670] nix-grpc-daemon[844]: ts=2026-09-22T10:45:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:340445166worker2 # [ 79.584966] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5167worker2 # [ 79.760602] nix-daemon[931]: accepted connection from pid 1120, user nix-grpc-daemon (trusted)5168worker2 # [ 79.773415] nix-daemon[931]: reaped child process 1123, status = succeeded5169worker2 # [ 79.936873] nix-daemon[931]: accepted connection from pid 1125, user nix-grpc-daemon (trusted)5170worker2 # [ 79.950902] nix-daemon[931]: reaped child process 1128, status = succeeded5171lb # [ 79.983810] niks3-server[1077]: 2026/09/22 10:45:21 INFO Received uploads request method=POST path=/api/pending_closures5172worker2 # [ 79.976397] nix-grpc-daemon[982]: time=2026-09-22T10:45:21.943Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5173worker2 # [ 79.978259] nix-grpc-daemon[982]: time=2026-09-22T10:45:21.945Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5174lb # [ 80.014965] niks3-server[1077]: 2026/09/22 10:45:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5175worker2 # [ 80.008372] nix-grpc-daemon[982]: time=2026-09-22T10:45:21.975Z level=INFO msg="Uploading 1 narinfos"5176lb # [ 80.024287] niks3-server[1077]: 2026/09/22 10:45:21 INFO Signed narinfos id=15 count=15177lb # [ 80.028814] niks3-server[1077]: 2026/09/22 10:45:21 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5178lb # [ 80.039891] niks3-server[1077]: 2026/09/22 10:45:21 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5179lb # [ 80.048774] niks3-server[1077]: 2026/09/22 10:45:21 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5180lb # [ 80.055238] niks3-server[1077]: 2026/09/22 10:45:21 INFO Completed upload id=155181worker2 # [ 80.039386] nix-grpc-daemon[982]: time=2026-09-22T10:45:22.006Z level=INFO msg="Upload complete. (447ms)"5182lb # [ 80.060663] niks3-server[1077]: 2026/09/22 10:45:21 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo5183worker2 # [ 80.042979] nix-grpc-daemon[844]: ts=2026-09-22T10:45:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_in=5125184worker2 # [ 80.048817] nix-grpc-daemon[844]: ts=2026-09-22T10:45:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340445185worker2 # [ 80.059112] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5186worker2 # [ 80.063346] nix-daemon[931]: reaped child process 1118, status = succeeded5187worker2 # [ 80.068447] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5188worker2 # [ 81.293832] nix-daemon[931]: reaped child process 1132, status = succeeded5189worker2 # [ 81.456790] nix-daemon[931]: accepted connection from pid 1148, user nix-grpc-daemon (trusted)5190worker2 # [ 81.469038] nix-daemon[931]: reaped child process 1151, status = succeeded5191lb # [ 81.501253] niks3-server[1077]: 2026/09/22 10:45:23 INFO Received uploads request method=POST path=/api/pending_closures5192worker2 # [ 81.493613] nix-grpc-daemon[982]: time=2026-09-22T10:45:23.460Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5193worker2 # [ 81.495777] nix-grpc-daemon[982]: time=2026-09-22T10:45:23.462Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5194lb # [ 81.539060] niks3-server[1077]: 2026/09/22 10:45:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5195worker2 # [ 81.529868] nix-grpc-daemon[982]: time=2026-09-22T10:45:23.497Z level=INFO msg="Uploading 1 narinfos"5196lb # [ 81.545530] niks3-server[1077]: 2026/09/22 10:45:23 INFO Signed narinfos id=16 count=15197lb # [ 81.553815] niks3-server[1077]: 2026/09/22 10:45:23 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5198lb # [ 81.562830] niks3-server[1077]: 2026/09/22 10:45:23 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5199lb # [ 81.571479] niks3-server[1077]: 2026/09/22 10:45:23 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5200lb # [ 81.576766] niks3-server[1077]: 2026/09/22 10:45:23 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5201lb # [ 81.582831] niks3-server[1077]: 2026/09/22 10:45:23 INFO Completed upload id=165202lb # [ 81.584443] niks3-server[1077]: 2026/09/22 10:45:23 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5203worker2 # [ 81.567258] nix-grpc-daemon[982]: time=2026-09-22T10:45:23.534Z level=INFO msg="Upload complete. (290ms)"5204worker2 # [ 81.574637] nix-grpc-daemon[844]: ts=2026-09-22T10:45:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=9 outputs=15205worker1 # [ 81.587344] nix-grpc-daemon[845]: ts=2026-09-22T10:45:23Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:560725206worker2 # [ 81.587575] nix-daemon[931]: reaped child process 1130, status = succeeded5207worker1 # [ 81.598545] nix-grpc-daemon[845]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5208worker1 # [ 81.629926] nix-grpc-daemon[845]: ts=2026-09-22T10:45:23Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:560725209worker1 # [ 81.635530] nix-grpc-daemon[845]: ts=2026-09-22T10:45:23Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=0 paths=1 nar_bytes_out=1285210worker2 # [ 81.915770] nix-grpc-daemon[844]: ts=2026-09-22T10:45:23Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:340445211worker2 # [ 81.921630] nix-grpc-daemon[844]: ts=2026-09-22T10:45:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340445212worker2 # [ 81.926582] nix-grpc-daemon[982]: time=2026-09-22T10:45:23.893Z level=INFO msg="All 1 paths already cached"5213worker2 # [ 81.936733] nix-grpc-daemon[844]: ts=2026-09-22T10:45:23Z level=debug event=want client=11 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05214worker2 # [ 81.942145] nix-grpc-daemon[844]: ts=2026-09-22T10:45:23Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.4:50051 clients=15215worker2 # [ 81.980661] nix-grpc-daemon[844]: ts=2026-09-22T10:45:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340445216worker2 # [ 81.993089] nix-grpc-daemon[844]: ts=2026-09-22T10:45:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340445217worker2 # [ 82.005490] nix-grpc-daemon[982]: time=2026-09-22T10:45:23.972Z level=INFO msg="All 1 paths already cached"5218worker2 # [ 82.012749] nix-grpc-daemon[844]: ts=2026-09-22T10:45:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:340445219worker2 # [ 82.021398] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5220worker2 # [ 82.200470] nix-daemon[931]: accepted connection from pid 1157, user nix-grpc-daemon (trusted)5221worker2 # [ 82.213645] nix-daemon[931]: reaped child process 1160, status = succeeded5222worker2 # [ 82.389749] nix-daemon[931]: accepted connection from pid 1162, user nix-grpc-daemon (trusted)5223worker2 # [ 82.402516] nix-daemon[931]: reaped child process 1165, status = succeeded5224lb # [ 82.435423] niks3-server[1077]: 2026/09/22 10:45:24 INFO Received uploads request method=POST path=/api/pending_closures5225worker2 # [ 82.423694] nix-grpc-daemon[982]: time=2026-09-22T10:45:24.391Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5226worker2 # [ 82.425287] nix-grpc-daemon[982]: time=2026-09-22T10:45:24.392Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5227lb # [ 82.452569] niks3-server[1077]: 2026/09/22 10:45:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5228lb # [ 82.454448] niks3-server[1077]: 2026/09/22 10:45:24 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5229lb # [ 82.456651] niks3-server[1077]: 2026/09/22 10:45:24 INFO Signed narinfos id=17 count=15230worker2 # [ 82.441091] nix-grpc-daemon[982]: time=2026-09-22T10:45:24.408Z level=INFO msg="Uploading 1 narinfos"5231lb # [ 82.461845] niks3-server[1077]: 2026/09/22 10:45:24 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5232lb # [ 82.465645] niks3-server[1077]: 2026/09/22 10:45:24 INFO Completed upload id=175233worker2 # [ 82.449908] nix-grpc-daemon[982]: time=2026-09-22T10:45:24.417Z level=INFO msg="Upload complete. (413ms)"5234lb # [ 82.466886] niks3-server[1077]: 2026/09/22 10:45:24 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5235worker2 # [ 82.451942] nix-grpc-daemon[844]: ts=2026-09-22T10:45:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_in=6245236lb # [ 82.469691] niks3-server[1077]: 2026/09/22 10:45:24 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5237worker2 # [ 82.456315] nix-grpc-daemon[844]: ts=2026-09-22T10:45:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340445238worker2 # [ 82.462236] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5239worker2 # [ 82.464673] nix-daemon[931]: reaped child process 1155, status = succeeded5240worker2 # [ 82.470425] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5241worker2 # [ 83.482705] nix-daemon[931]: reaped child process 1169, status = succeeded5242worker2 # [ 83.645529] nix-daemon[931]: accepted connection from pid 1185, user nix-grpc-daemon (trusted)5243worker2 # [ 83.658717] nix-daemon[931]: reaped child process 1188, status = succeeded5244lb # [ 83.692708] niks3-server[1077]: 2026/09/22 10:45:25 INFO Received uploads request method=POST path=/api/pending_closures5245worker2 # [ 83.685683] nix-grpc-daemon[982]: time=2026-09-22T10:45:25.652Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5246worker2 # [ 83.688059] nix-grpc-daemon[982]: time=2026-09-22T10:45:25.654Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5247lb # [ 83.728573] niks3-server[1077]: 2026/09/22 10:45:25 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5248lb # [ 83.740143] niks3-server[1077]: 2026/09/22 10:45:25 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5249lb # [ 83.748912] niks3-server[1077]: 2026/09/22 10:45:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5250worker2 # [ 83.737960] nix-grpc-daemon[982]: time=2026-09-22T10:45:25.705Z level=INFO msg="Uploading 1 narinfos"5251lb # [ 83.755129] niks3-server[1077]: 2026/09/22 10:45:25 INFO Signed narinfos id=18 count=15252lb # [ 83.759655] niks3-server[1077]: 2026/09/22 10:45:25 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5253lb # [ 83.770182] niks3-server[1077]: 2026/09/22 10:45:25 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5254lb # [ 83.776268] niks3-server[1077]: 2026/09/22 10:45:25 INFO Completed upload id=185255lb # [ 83.778067] niks3-server[1077]: 2026/09/22 10:45:25 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5256worker2 # [ 83.760746] nix-grpc-daemon[982]: time=2026-09-22T10:45:25.727Z level=INFO msg="Upload complete. (282ms)"5257worker2 # [ 83.769228] nix-grpc-daemon[844]: ts=2026-09-22T10:45:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=10 outputs=15258worker2 # [ 83.775329] nix-grpc-daemon[844]: ts=2026-09-22T10:45:25Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:340445259worker2 # [ 83.785601] nix-grpc-daemon[844]: ts=2026-09-22T10:45:25Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:340445260worker2 # [ 83.790234] nix-grpc-daemon[844]: ts=2026-09-22T10:45:25Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_out=1765261worker2 # [ 83.794137] nix-daemon[931]: reaped child process 1167, status = succeeded5262worker1 # [ 84.048816] nix-grpc-daemon[845]: ts=2026-09-22T10:45:26Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:560725263worker1 # [ 84.054342] nix-grpc-daemon[845]: ts=2026-09-22T10:45:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:560725264worker1 # [ 84.065452] nix-grpc-daemon[924]: time=2026-09-22T10:45:26.066Z level=INFO msg="All 1 paths already cached"5265worker2 # [ 84.075813] nix-grpc-daemon[844]: ts=2026-09-22T10:45:26Z level=debug event=want client=12 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05266worker2 # [ 84.082712] nix-grpc-daemon[844]: ts=2026-09-22T10:45:26Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.4:50051 clients=15267worker2 # [ 84.087202] nix-grpc-daemon[844]: ts=2026-09-22T10:45:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340445268worker2 # [ 84.096386] nix-grpc-daemon[844]: ts=2026-09-22T10:45:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340445269worker2 # [ 84.105923] nix-grpc-daemon[982]: time=2026-09-22T10:45:26.073Z level=INFO msg="All 2 paths already cached"5270worker2 # [ 84.111906] nix-grpc-daemon[844]: ts=2026-09-22T10:45:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:340445271worker2 # [ 84.122158] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5272worker2 # [ 84.300286] nix-daemon[931]: accepted connection from pid 1194, user nix-grpc-daemon (trusted)5273worker2 # [ 84.313471] nix-daemon[931]: reaped child process 1197, status = succeeded5274worker2 # [ 84.474622] nix-daemon[931]: accepted connection from pid 1199, user nix-grpc-daemon (trusted)5275worker2 # [ 84.488197] nix-daemon[931]: reaped child process 1202, status = succeeded5276lb # [ 84.518751] niks3-server[1077]: 2026/09/22 10:45:26 INFO Received uploads request method=POST path=/api/pending_closures5277worker2 # [ 84.507217] nix-grpc-daemon[982]: time=2026-09-22T10:45:26.474Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5278worker2 # [ 84.508581] nix-grpc-daemon[982]: time=2026-09-22T10:45:26.474Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5279lb # [ 84.532054] niks3-server[1077]: 2026/09/22 10:45:26 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5280lb # [ 84.534858] niks3-server[1077]: 2026/09/22 10:45:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5281lb # [ 84.536721] niks3-server[1077]: 2026/09/22 10:45:26 INFO Signed narinfos id=19 count=15282worker2 # [ 84.520892] nix-grpc-daemon[982]: time=2026-09-22T10:45:26.488Z level=INFO msg="Uploading 1 narinfos"5283lb # [ 84.539576] niks3-server[1077]: 2026/09/22 10:45:26 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5284lb # [ 84.544328] niks3-server[1077]: 2026/09/22 10:45:26 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5285lb # [ 84.547574] niks3-server[1077]: 2026/09/22 10:45:26 INFO Completed upload id=195286worker2 # [ 84.532937] nix-grpc-daemon[982]: time=2026-09-22T10:45:26.500Z level=INFO msg="Upload complete. (398ms)"5287lb # [ 84.549181] niks3-server[1077]: 2026/09/22 10:45:26 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5288worker2 # [ 84.535048] nix-grpc-daemon[844]: ts=2026-09-22T10:45:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_in=7525289worker2 # [ 84.537539] nix-grpc-daemon[844]: ts=2026-09-22T10:45:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340445290worker2 # [ 84.543948] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5291worker2 # [ 84.547378] nix-daemon[931]: reaped child process 1192, status = succeeded5292worker2 # [ 84.552566] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5293worker2 # [ 85.978511] nix-daemon[931]: reaped child process 1206, status = succeeded5294worker2 # [ 86.129891] nix-daemon[931]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5295worker2 # [ 86.142112] nix-daemon[931]: reaped child process 1225, status = succeeded5296lb # [ 86.171129] niks3-server[1077]: 2026/09/22 10:45:28 INFO Received uploads request method=POST path=/api/pending_closures5297worker2 # [ 86.162260] nix-grpc-daemon[982]: time=2026-09-22T10:45:28.129Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5298worker2 # [ 86.163946] nix-grpc-daemon[982]: time=2026-09-22T10:45:28.131Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5299lb # [ 86.200306] niks3-server[1077]: 2026/09/22 10:45:28 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5300lb # [ 86.208567] niks3-server[1077]: 2026/09/22 10:45:28 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5301lb # [ 86.211851] niks3-server[1077]: 2026/09/22 10:45:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5302lb # [ 86.213976] niks3-server[1077]: 2026/09/22 10:45:28 INFO Signed narinfos id=20 count=15303worker2 # [ 86.198235] nix-grpc-daemon[982]: time=2026-09-22T10:45:28.165Z level=INFO msg="Uploading 1 narinfos"5304lb # [ 86.218085] niks3-server[1077]: 2026/09/22 10:45:28 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5305lb # [ 86.223117] niks3-server[1077]: 2026/09/22 10:45:28 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5306lb # [ 86.226639] niks3-server[1077]: 2026/09/22 10:45:28 INFO Completed upload id=205307worker2 # [ 86.210540] nix-grpc-daemon[982]: time=2026-09-22T10:45:28.177Z level=INFO msg="Upload complete. (250ms)"5308lb # [ 86.228171] niks3-server[1077]: 2026/09/22 10:45:28 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5309worker2 # [ 86.214280] nix-grpc-daemon[844]: ts=2026-09-22T10:45:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=11 outputs=15310worker2 # [ 86.220607] nix-daemon[931]: reaped child process 1204, status = succeeded5311worker1 # [ 86.263154] nix-grpc-daemon[845]: ts=2026-09-22T10:45:28Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:560725312worker1 # [ 86.268861] nix-grpc-daemon[845]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...5313worker1 # [ 86.275055] nix-grpc-daemon[845]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...5314worker2 # [ 86.293130] nix-grpc-daemon[844]: ts=2026-09-22T10:45:28Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:340445315worker2 # [ 86.298203] nix-grpc-daemon[844]: ts=2026-09-22T10:45:28Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_out=2325316client # [ 86.426542] nix-daemon[1046]: reaped child process 1048, status = succeeded5317client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.34 seconds)5318client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5319client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.03 seconds)5320client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5321client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.03 seconds)5322client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5323client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)5324(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.43 seconds)5325subtest: an input only one worker has still reaches the builder5326client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only5327client # this derivation will be built:5328client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5329client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5330client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.37 seconds)5331client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5332client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.17 seconds)5333worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5334worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.18 seconds)5335worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5336worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.03 seconds)5337client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&25338client # [ 87.413475] nix-daemon[1046]: accepted connection from pid 1129, user root (trusted)5339client # this derivation will be built:5340client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5341worker1 # [ 87.694392] nix-grpc-daemon[845]: ts=2026-09-22T10:45:29Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:560725342client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5343worker2 # [ 87.694180] nix-grpc-daemon[844]: ts=2026-09-22T10:45:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340445344client # copying 1 paths...5345client # copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' to 'grpc://lb:50051'...5346worker2 # [ 87.702791] nix-grpc-daemon[844]: ts=2026-09-22T10:45:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:340445347worker2 # [ 87.713770] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5348worker2 # [ 87.899128] nix-daemon[931]: accepted connection from pid 1232, user nix-grpc-daemon (trusted)5349worker2 # [ 87.913652] nix-daemon[931]: reaped child process 1235, status = succeeded5350lb # [ 87.944636] niks3-server[1077]: 2026/09/22 10:45:29 INFO Received uploads request method=POST path=/api/pending_closures5351worker2 # [ 87.933583] nix-grpc-daemon[982]: time=2026-09-22T10:45:29.900Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5352worker2 # [ 87.935001] nix-grpc-daemon[982]: time=2026-09-22T10:45:29.900Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5353lb # [ 87.959175] niks3-server[1077]: 2026/09/22 10:45:29 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5354lb # [ 87.962020] niks3-server[1077]: 2026/09/22 10:45:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5355lb # [ 87.963952] niks3-server[1077]: 2026/09/22 10:45:29 INFO Signed narinfos id=21 count=15356worker2 # [ 87.949371] nix-grpc-daemon[982]: time=2026-09-22T10:45:29.916Z level=INFO msg="Uploading 1 narinfos"5357lb # [ 87.966600] niks3-server[1077]: 2026/09/22 10:45:29 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5358lb # [ 87.971173] niks3-server[1077]: 2026/09/22 10:45:29 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5359lb # [ 87.974856] niks3-server[1077]: 2026/09/22 10:45:29 INFO Completed upload id=215360worker2 # [ 87.959055] nix-grpc-daemon[982]: time=2026-09-22T10:45:29.926Z level=INFO msg="Upload complete. (234ms)"5361worker2 # [ 87.960796] nix-grpc-daemon[844]: ts=2026-09-22T10:45:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_in=1205362lb # [ 87.977336] niks3-server[1077]: 2026/09/22 10:45:29 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5363worker2 # [ 87.966896] nix-daemon[931]: reaped child process 1230, status = succeeded5364worker2 # [ 87.971753] nix-grpc-daemon[844]: ts=2026-09-22T10:45:29Z level=debug event=want client=13 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05365worker2 # [ 87.973765] nix-grpc-daemon[844]: ts=2026-09-22T10:45:29Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=15366worker1 # [ 88.017940] nix-grpc-daemon[845]: ts=2026-09-22T10:45:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560725367worker1 # [ 88.029951] nix-grpc-daemon[845]: ts=2026-09-22T10:45:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:560725368client # copying 2 paths to 'lb:50051'...5369client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5370worker1 # [ 88.043818] nix-grpc-daemon[845]: ts=2026-09-22T10:45:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:560725371client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5372worker1 # [ 88.055893] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5373worker1 # [ 88.213980] nix-daemon[911]: accepted connection from pid 1212, user nix-grpc-daemon (trusted)5374worker1 # [ 88.227453] nix-daemon[911]: reaped child process 1215, status = succeeded5375worker1 # [ 88.350198] nix-daemon[911]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5376worker1 # [ 88.362118] nix-daemon[911]: reaped child process 1220, status = succeeded5377lb # [ 88.385651] niks3-server[1077]: 2026/09/22 10:45:30 INFO Received uploads request method=POST path=/api/pending_closures5378lb # [ 88.390628] niks3-server[1077]: 2026/09/22 10:45:30 INFO Received uploads request method=POST path=/api/pending_closures5379worker1 # [ 88.380738] nix-grpc-daemon[924]: time=2026-09-22T10:45:30.381Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5380lb # [ 88.394819] envoy[987]: conn peer=[2001:db8:1::1]:59722 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3421 tx=1830 ms=21405381lb # [ 88.396288] envoy[987]: conn peer=[2001:db8:1::1]:59732 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=333 ms=575382worker1 # [ 88.382464] nix-grpc-daemon[924]: time=2026-09-22T10:45:30.383Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5383lb # [ 88.397647] envoy[987]: conn peer=[2001:db8:1::1]:59742 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3879 tx=1958 ms=19245384worker1 # [ 88.384484] nix-grpc-daemon[924]: time=2026-09-22T10:45:30.383Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5385lb # [ 88.399157] envoy[987]: conn peer=[2001:db8:1::1]:51562 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=535386lb # [ 88.401310] envoy[987]: conn peer=[2001:db8:1::1]:51576 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=575387lb # [ 88.402741] envoy[987]: conn peer=[2001:db8:1::1]:51570 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4339 tx=2299 ms=23055388lb # [ 88.414709] niks3-server[1077]: 2026/09/22 10:45:30 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5389lb # [ 88.420191] niks3-server[1077]: 2026/09/22 10:45:30 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5390lb # [ 88.425187] niks3-server[1077]: 2026/09/22 10:45:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5391lb # [ 88.427562] niks3-server[1077]: 2026/09/22 10:45:30 INFO Signed narinfos id=22 count=15392lb # [ 88.429361] niks3-server[1077]: 2026/09/22 10:45:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5393lb # [ 88.431196] niks3-server[1077]: 2026/09/22 10:45:30 INFO Signed narinfos id=23 count=25394worker1 # [ 88.417932] nix-grpc-daemon[924]: time=2026-09-22T10:45:30.419Z level=INFO msg="Uploading 3 narinfos"5395lb # [ 88.433187] niks3-server[1077]: 2026/09/22 10:45:30 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5396lb # [ 88.435067] niks3-server[1077]: 2026/09/22 10:45:30 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5397lb # [ 88.447384] niks3-server[1077]: 2026/09/22 10:45:30 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5398lb # [ 88.451306] niks3-server[1077]: 2026/09/22 10:45:30 INFO Completed upload id=225399lb # [ 88.453595] niks3-server[1077]: 2026/09/22 10:45:30 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5400lb # [ 88.456605] niks3-server[1077]: 2026/09/22 10:45:30 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5401lb # [ 88.458528] niks3-server[1077]: 2026/09/22 10:45:30 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5402lb # [ 88.460131] niks3-server[1077]: 2026/09/22 10:45:30 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5403lb # [ 88.462118] niks3-server[1077]: 2026/09/22 10:45:30 INFO Completed upload id=235404worker1 # [ 88.449871] nix-grpc-daemon[924]: time=2026-09-22T10:45:30.451Z level=INFO msg="Upload complete. (376ms)"5405worker1 # [ 88.454127] nix-grpc-daemon[845]: ts=2026-09-22T10:45:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=0 paths=2 nar_bytes_in=9765406worker1 # [ 88.458711] nix-grpc-daemon[845]: ts=2026-09-22T10:45:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560725407worker1 # [ 88.467771] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5408worker1 # [ 88.471317] nix-daemon[911]: reaped child process 1210, status = succeeded5409worker1 # [ 88.476503] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5410client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5411client # uses-input-w1onlya> node-a: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5412client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5413worker1 # [ 89.088743] nix-daemon[911]: reaped child process 1224, status = succeeded5414worker1 # [ 89.199343] nix-daemon[911]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5415worker1 # [ 89.211414] nix-daemon[911]: reaped child process 1243, status = succeeded5416lb # [ 89.235935] niks3-server[1077]: 2026/09/22 10:45:31 INFO Received uploads request method=POST path=/api/pending_closures5417worker1 # [ 89.230555] nix-grpc-daemon[924]: time=2026-09-22T10:45:31.231Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5418lb # [ 89.264739] niks3-server[1077]: 2026/09/22 10:45:31 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5419lb # [ 89.273537] niks3-server[1077]: 2026/09/22 10:45:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5420lb # [ 89.278879] niks3-server[1077]: 2026/09/22 10:45:31 INFO Signed narinfos id=24 count=15421worker1 # [ 89.268231] nix-grpc-daemon[924]: time=2026-09-22T10:45:31.269Z level=INFO msg="Uploading 1 narinfos"5422lb # [ 89.285150] niks3-server[1077]: 2026/09/22 10:45:31 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5423lb # [ 89.298265] niks3-server[1077]: 2026/09/22 10:45:31 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5424worker1 # [ 89.289607] nix-grpc-daemon[924]: time=2026-09-22T10:45:31.290Z level=INFO msg="Upload complete. (207ms)"5425lb # [ 89.303205] niks3-server[1077]: 2026/09/22 10:45:31 INFO Completed upload id=245426worker1 # [ 89.292183] nix-grpc-daemon[845]: ts=2026-09-22T10:45:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=0 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=12 outputs=15427lb # [ 89.306593] niks3-server[1077]: 2026/09/22 10:45:31 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5428client # copying 1 paths...5429worker1 # [ 89.298890] nix-daemon[911]: reaped child process 1222, status = succeeded5430worker1 # [ 89.300597] nix-grpc-daemon[845]: ts=2026-09-22T10:45:31Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:560725431client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5432worker2 # [ 89.319963] nix-grpc-daemon[844]: ts=2026-09-22T10:45:31Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:340445433worker2 # [ 89.331669] nix-grpc-daemon[844]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5434worker2 # [ 89.346834] nix-grpc-daemon[844]: ts=2026-09-22T10:45:31Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_out=1205435client # [ 89.417583] nix-daemon[1046]: reaped child process 1132, status = succeeded5436client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.16 seconds)5437client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25438client # [ 89.553248] nix-daemon[1046]: accepted connection from pid 1151, user root (trusted)5439client # this derivation will be built:5440client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5441worker2 # [ 89.697158] nix-grpc-daemon[844]: ts=2026-09-22T10:45:31Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:340445442client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5443worker2 # [ 89.704401] nix-grpc-daemon[844]: ts=2026-09-22T10:45:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340445444worker2 # [ 89.713486] nix-grpc-daemon[982]: time=2026-09-22T10:45:31.680Z level=INFO msg="All 1 paths already cached"5445worker2 # [ 89.726397] nix-grpc-daemon[844]: ts=2026-09-22T10:45:31Z level=debug event=want client=14 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05446worker2 # [ 89.731495] nix-grpc-daemon[844]: ts=2026-09-22T10:45:31Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=15447worker2 # [ 89.770956] nix-grpc-daemon[844]: ts=2026-09-22T10:45:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340445448worker2 # [ 89.783313] nix-grpc-daemon[844]: ts=2026-09-22T10:45:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:340445449client # copying 1 paths to 'lb:50051'...5450client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5451worker2 # [ 89.801958] nix-grpc-daemon[844]: ts=2026-09-22T10:45:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:340445452worker2 # [ 89.810978] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5453worker2 # [ 89.829584] nix-grpc-daemon[844]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 30 ms (attempt 1/5)5454worker2 # [ 89.865155] nix-grpc-daemon[844]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 192 ms (attempt 2/5)5455worker2 # [ 90.059859] nix-grpc-daemon[844]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 48 ms (attempt 3/5)5456worker2 # [ 90.113746] nix-grpc-daemon[844]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 4/5)5457worker2 # [ 90.145616] nix-grpc-daemon[844]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5458worker2 # [ 90.149162] nix-grpc-daemon[844]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5459worker2 # [ 90.154636] nix-grpc-daemon[844]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5460worker2 # [ 90.361616] nix-daemon[931]: accepted connection from pid 1249, user nix-grpc-daemon (trusted)5461worker2 # [ 90.375168] nix-daemon[931]: reaped child process 1252, status = succeeded5462worker2 # [ 90.533271] nix-daemon[931]: accepted connection from pid 1254, user nix-grpc-daemon (trusted)5463worker2 # [ 90.545966] nix-daemon[931]: reaped child process 1257, status = succeeded5464lb # [ 90.579352] niks3-server[1077]: 2026/09/22 10:45:32 INFO Received uploads request method=POST path=/api/pending_closures5465worker2 # [ 90.571913] nix-grpc-daemon[982]: time=2026-09-22T10:45:32.539Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5466worker2 # [ 90.573770] nix-grpc-daemon[982]: time=2026-09-22T10:45:32.540Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5467lb # [ 90.608793] niks3-server[1077]: 2026/09/22 10:45:32 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5468lb # [ 90.618412] niks3-server[1077]: 2026/09/22 10:45:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5469worker2 # [ 90.607903] nix-grpc-daemon[982]: time=2026-09-22T10:45:32.575Z level=INFO msg="Uploading 1 narinfos"5470lb # [ 90.623607] niks3-server[1077]: 2026/09/22 10:45:32 INFO Signed narinfos id=25 count=15471lb # [ 90.628913] niks3-server[1077]: 2026/09/22 10:45:32 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5472lb # [ 90.640900] niks3-server[1077]: 2026/09/22 10:45:32 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5473lb # [ 90.647574] niks3-server[1077]: 2026/09/22 10:45:32 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5474lb # [ 90.651301] niks3-server[1077]: 2026/09/22 10:45:32 INFO Completed upload id=255475worker2 # [ 90.637732] nix-grpc-daemon[982]: time=2026-09-22T10:45:32.605Z level=INFO msg="Upload complete. (458ms)"5476worker2 # [ 90.642515] nix-grpc-daemon[844]: ts=2026-09-22T10:45:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_in=5605477worker2 # [ 90.648385] nix-grpc-daemon[844]: ts=2026-09-22T10:45:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:340445478worker2 # [ 90.657135] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5479worker2 # [ 90.665946] nix-daemon[931]: reaped child process 1242, status = succeeded5480worker2 # [ 90.671232] nix-daemon[931]: accepted connection from pid 844, user nix-grpc-daemon (trusted)5481client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5482client # uses-input-w1onlyb> worker2: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5483client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5484worker2 # [ 91.498279] nix-daemon[931]: reaped child process 1261, status = succeeded5485worker2 # [ 91.633671] nix-daemon[931]: accepted connection from pid 1277, user nix-grpc-daemon (trusted)5486worker2 # [ 91.646228] nix-daemon[931]: reaped child process 1280, status = succeeded5487lb # [ 91.678822] niks3-server[1077]: 2026/09/22 10:45:33 INFO Received uploads request method=POST path=/api/pending_closures5488worker2 # [ 91.689002] nix-grpc-daemon[982]: time=2026-09-22T10:45:33.656Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5489lb # [ 91.728368] niks3-server[1077]: 2026/09/22 10:45:33 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5490lb # [ 91.734422] niks3-server[1077]: 2026/09/22 10:45:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5491lb # [ 91.742250] niks3-server[1077]: 2026/09/22 10:45:33 INFO Signed narinfos id=26 count=15492worker2 # [ 91.729667] nix-grpc-daemon[982]: time=2026-09-22T10:45:33.696Z level=INFO msg="Uploading 1 narinfos"5493lb # [ 91.750944] niks3-server[1077]: 2026/09/22 10:45:33 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5494lb # [ 91.761180] niks3-server[1077]: 2026/09/22 10:45:33 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5495lb # [ 91.768356] niks3-server[1077]: 2026/09/22 10:45:33 INFO Completed upload id=265496worker2 # [ 91.753054] nix-grpc-daemon[982]: time=2026-09-22T10:45:33.720Z level=INFO msg="Upload complete. (274ms)"5497lb # [ 91.773233] niks3-server[1077]: 2026/09/22 10:45:33 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5498worker2 # [ 91.758848] nix-grpc-daemon[844]: ts=2026-09-22T10:45:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=13 outputs=15499worker2 # [ 91.771671] nix-daemon[931]: reaped child process 1259, status = succeeded5500client # copying 1 paths...5501worker2 # [ 91.803717] nix-grpc-daemon[844]: ts=2026-09-22T10:45:33Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:340445502client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5503worker2 # [ 91.822098] nix-grpc-daemon[844]: ts=2026-09-22T10:45:33Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:340445504worker2 # [ 91.826791] nix-grpc-daemon[844]: ts=2026-09-22T10:45:33Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_out=1205505client # [ 91.948845] nix-daemon[1046]: reaped child process 1154, status = succeeded5506client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.54 seconds)5507client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25508client # [ 92.121627] nix-daemon[1046]: accepted connection from pid 1172, user root (trusted)5509client # this derivation will be built:5510client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5511worker2 # [ 92.284205] nix-grpc-daemon[844]: ts=2026-09-22T10:45:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:340445512client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5513worker1 # [ 92.290571] nix-grpc-daemon[845]: ts=2026-09-22T10:45:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:560725514worker1 # [ 92.300795] nix-grpc-daemon[924]: time=2026-09-22T10:45:34.301Z level=INFO msg="All 1 paths already cached"5515worker2 # [ 92.314755] nix-grpc-daemon[844]: ts=2026-09-22T10:45:34Z level=debug event=want client=15 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05516worker2 # [ 92.319622] nix-grpc-daemon[844]: ts=2026-09-22T10:45:34Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=15517worker1 # [ 92.324829] nix-grpc-daemon[845]: ts=2026-09-22T10:45:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560725518worker1 # [ 92.336863] nix-grpc-daemon[845]: ts=2026-09-22T10:45:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:560725519worker1 # [ 92.348179] nix-grpc-daemon[924]: time=2026-09-22T10:45:34.349Z level=INFO msg="All 1 paths already cached"5520client # copying 1 paths to 'lb:50051'...5521client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5522worker1 # [ 92.355938] nix-grpc-daemon[845]: ts=2026-09-22T10:45:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:560725523worker1 # [ 92.364862] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5524worker1 # [ 92.523407] nix-daemon[911]: accepted connection from pid 1247, user nix-grpc-daemon (trusted)5525worker1 # [ 92.536882] nix-daemon[911]: reaped child process 1250, status = succeeded5526worker1 # [ 92.671416] nix-daemon[911]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5527worker1 # [ 92.683077] nix-daemon[911]: reaped child process 1255, status = succeeded5528lb # [ 92.710540] niks3-server[1077]: 2026/09/22 10:45:34 INFO Received uploads request method=POST path=/api/pending_closures5529worker1 # [ 92.705201] nix-grpc-daemon[924]: time=2026-09-22T10:45:34.706Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5530worker1 # [ 92.706582] nix-grpc-daemon[924]: time=2026-09-22T10:45:34.706Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5531lb # [ 92.736881] niks3-server[1077]: 2026/09/22 10:45:34 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5532lb # [ 92.746336] niks3-server[1077]: 2026/09/22 10:45:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5533lb # [ 92.751828] niks3-server[1077]: 2026/09/22 10:45:34 INFO Signed narinfos id=27 count=15534worker1 # [ 92.741438] nix-grpc-daemon[924]: time=2026-09-22T10:45:34.742Z level=INFO msg="Uploading 1 narinfos"5535lb # [ 92.760086] niks3-server[1077]: 2026/09/22 10:45:34 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5536lb # [ 92.771178] niks3-server[1077]: 2026/09/22 10:45:34 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5537worker1 # [ 92.762764] nix-grpc-daemon[924]: time=2026-09-22T10:45:34.764Z level=INFO msg="Upload complete. (388ms)"5538lb # [ 92.776488] niks3-server[1077]: 2026/09/22 10:45:34 INFO Completed upload id=275539worker1 # [ 92.764733] nix-grpc-daemon[845]: ts=2026-09-22T10:45:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=0 paths=1 nar_bytes_in=5605540lb # [ 92.779822] niks3-server[1077]: 2026/09/22 10:45:34 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5541worker1 # [ 92.770711] nix-grpc-daemon[845]: ts=2026-09-22T10:45:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:560725542worker1 # [ 92.778331] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5543worker1 # [ 92.780544] nix-daemon[911]: reaped child process 1245, status = succeeded5544client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5545worker1 # [ 92.785615] nix-daemon[911]: accepted connection from pid 845, user nix-grpc-daemon (trusted)5546client # uses-input-w1onlyc> node-a: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5547client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5548worker1 # [ 94.210646] nix-daemon[911]: reaped child process 1259, status = succeeded5549worker1 # [ 94.354771] nix-daemon[911]: accepted connection from pid 1275, user nix-grpc-daemon (trusted)5550worker1 # [ 94.366377] nix-daemon[911]: reaped child process 1278, status = succeeded5551lb # [ 94.389073] niks3-server[1077]: 2026/09/22 10:45:36 INFO Received uploads request method=POST path=/api/pending_closures5552worker1 # [ 94.380818] nix-grpc-daemon[924]: time=2026-09-22T10:45:36.381Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5553lb # [ 94.405226] niks3-server[1077]: 2026/09/22 10:45:36 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5554lb # [ 94.407053] niks3-server[1077]: 2026/09/22 10:45:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5555lb # [ 94.409396] niks3-server[1077]: 2026/09/22 10:45:36 INFO Signed narinfos id=28 count=15556worker1 # [ 94.395965] nix-grpc-daemon[924]: time=2026-09-22T10:45:36.397Z level=INFO msg="Uploading 1 narinfos"5557lb # [ 94.413150] niks3-server[1077]: 2026/09/22 10:45:36 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5558lb # [ 94.418696] niks3-server[1077]: 2026/09/22 10:45:36 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5559lb # [ 94.423091] niks3-server[1077]: 2026/09/22 10:45:36 INFO Completed upload id=285560worker1 # [ 94.409493] nix-grpc-daemon[924]: time=2026-09-22T10:45:36.410Z level=INFO msg="Upload complete. (216ms)"5561lb # [ 94.424628] niks3-server[1077]: 2026/09/22 10:45:36 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5562worker1 # [ 94.411787] nix-grpc-daemon[845]: ts=2026-09-22T10:45:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56072 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=14 outputs=15563worker1 # [ 94.418343] nix-daemon[911]: reaped child process 1257, status = succeeded5564client # copying 1 paths...5565worker2 # [ 94.455052] nix-grpc-daemon[844]: ts=2026-09-22T10:45:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:340445566worker2 # [ 94.460157] nix-grpc-daemon[844]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5567client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5568worker2 # [ 94.474942] nix-grpc-daemon[844]: ts=2026-09-22T10:45:36Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:340445569worker2 # [ 94.476881] nix-grpc-daemon[844]: ts=2026-09-22T10:45:36Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34044 duration_s=0 paths=1 nar_bytes_out=1205570client # [ 94.583566] nix-daemon[1046]: reaped child process 1175, status = succeeded5571client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.63 seconds)5572(finished: subtest: an input only one worker has still reaches the builder, in 8.07 seconds)5573subtest: upload whose reference the worker lacks is completed from the cache5574client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5575client # this derivation will be built:5576client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5577client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5578client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.33 seconds)5579client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5580client # this derivation will be built:5581client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5582client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5583client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.34 seconds)5584client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5585client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.15 seconds)5586worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5587worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.18 seconds)5588worker2 # [ 95.619827] systemd[1]: Stopping Nix worker-protocol over gRPC...5589worker2 # [ 95.828527] nix-grpc-daemon[844]: ts=2026-09-22T10:45:37Z level=info event=signal_exit5590lb # [ 96.114904] niks3-server[1077]: 2026/09/22 10:45:37 INFO lead: released remote=192.168.1.4:418025591worker2 # [ 97.099651] nix-grpc-daemon[844]: ts=2026-09-22T10:45:39Z level=info event=scheduler_restarting5592worker1 # [ 97.102952] nix-grpc-daemon[845]: ts=2026-09-22T10:45:39Z level=info event=scheduler_restarting addr=lb:500515593worker2 # [ 97.102799] nix-grpc-daemon[844]: ts=2026-09-22T10:45:39Z level=info event=scheduler_restarting addr=lb:500515594worker2 # [ 97.104393] nix-grpc-daemon[844]: ts=2026-09-22T10:45:39Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:340365595worker2 # [ 97.105735] nix-grpc-daemon[844]: ts=2026-09-22T10:45:39Z level=info event=worker_gone addr=192.168.1.4:500515596worker2 # [ 97.107078] nix-grpc-daemon[844]: ts=2026-09-22T10:45:39Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:340365597worker2 # [ 97.108606] nix-grpc-daemon[844]: ts=2026-09-22T10:45:39Z level=info event=worker_gone addr=192.168.1.3:500515598worker2 # [ 97.119883] nix-daemon[931]: reaped child process 933, status = succeeded5599worker1 # [ 97.303090] nix-grpc-daemon[845]: ts=2026-09-22T10:45:39Z level=info event=scheduler_connected addr=lb:500515600worker1 # [ 97.304861] nix-grpc-daemon[845]: ts=2026-09-22T10:45:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5601worker2 # [ 97.307204] nix-grpc-daemon[844]: ts=2026-09-22T10:45:39Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5602worker1 # [ 97.505865] nix-grpc-daemon[845]: ts=2026-09-22T10:45:39Z level=info event=scheduler_connected addr=lb:500515603worker1 # [ 97.509574] nix-grpc-daemon[845]: ts=2026-09-22T10:45:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5604worker1 # [ 97.710085] nix-grpc-daemon[845]: ts=2026-09-22T10:45:39Z level=info event=scheduler_connected addr=lb:500515605worker1 # [ 97.711878] nix-grpc-daemon[845]: ts=2026-09-22T10:45:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5606worker2 # [ 97.768366] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5607worker2 # [ 97.769362] systemd[1]: Stopped Nix worker-protocol over gRPC.5608worker2 # [ 97.770159] systemd[1]: nix-grpc-daemon.service: Consumed 2.267s CPU time over 1min 19.246s wall clock time, 57.4M memory peak, 82.8K incoming IP traffic, 237.9K outgoing IP traffic.5609worker2 # [ 97.774774] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5610worker2 # [ 97.776497] systemd[1]: Closed Nix worker-protocol over gRPC.5611worker2 # [ 97.777839] systemd[1]: nix-grpc-daemon.socket: Consumed 333us CPU time over 1min 27.984s wall clock time, 256K memory peak, 97.6K incoming IP traffic, 82.7K outgoing IP traffic.5612lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5613lb: must succeed: curl -sf localhost:9901/clusters5614lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5615lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.07 seconds)5616client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25617worker1 # [ 97.912748] nix-grpc-daemon[845]: ts=2026-09-22T10:45:39Z level=info event=scheduler_connected addr=lb:500515618worker1 # [ 97.915866] nix-grpc-daemon[845]: ts=2026-09-22T10:45:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5619worker1 # [ 98.013179] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:560725620worker1 # [ 98.019404] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:560725621client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5622client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.19 seconds)5623worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5624worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.02 seconds)5625worker2 # [ 98.114203] systemd[1]: Listening on Nix worker-protocol over gRPC.5626worker1 # [ 98.117598] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=info event=scheduler_connected addr=lb:500515627worker1 # [ 98.121119] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5628worker1 # [ 98.159805] systemd[1]: Stopping Nix worker-protocol over gRPC...5629worker1 # [ 98.321652] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=info event=scheduler_connected addr=lb:500515630worker1 # [ 98.323427] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5631lb # [ 98.395104] envoy[987]: conn peer=[2001:db8:1::1]:51580 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4159 tx=1787 ms=16675632lb # [ 98.396848] envoy[987]: conn peer=[2001:db8:1::1]:51584 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=421 tx=329 ms=445633lb # [ 98.398463] envoy[987]: conn peer=[2001:db8:1::1]:51594 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3728 tx=1902 ms=21785634lb # [ 98.401940] envoy[987]: conn peer=[2001:db8:1::1]:51610 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=575635lb # [ 98.405341] envoy[987]: conn peer=[2001:db8:1::1]:45808 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=485636lb # [ 98.408440] envoy[987]: conn peer=[2001:db8:1::1]:51618 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3755 tx=2013 ms=22445637lb # [ 98.411232] envoy[987]: conn peer=[2001:db8:1::4]:42886 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=71473 tx=3575 ms=784145638lb # [ 98.414107] envoy[987]: conn peer=[2001:db8:1::1]:45818 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=205639worker1 # [ 98.524498] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=info event=scheduler_connected addr=lb:500515640worker1 # [ 98.528329] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5641worker1 # [ 98.697065] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=info event=signal_exit5642worker1 # [ 98.728995] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=info event=scheduler_connected addr=lb:500515643worker1 # [ 98.732709] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5644worker1 # [ 98.933662] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=info event=scheduler_connected addr=lb:500515645worker1 # [ 98.937451] nix-grpc-daemon[845]: ts=2026-09-22T10:45:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5646worker2 # [ 98.981578] systemd[1]: Starting scheduler-token.service...5647worker2 # [ 99.074923] systemd[1]: scheduler-token.service: Deactivated successfully.5648worker2 # [ 99.078223] systemd[1]: Finished scheduler-token.service.5649worker2 # [ 99.080613] systemd[1]: scheduler-token.service: Consumed 37ms CPU time over 96ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5650worker2 # [ 99.087094] systemd[1]: Starting Nix worker-protocol over gRPC...5651worker1 # [ 99.139167] nix-grpc-daemon[845]: ts=2026-09-22T10:45:41Z level=info event=scheduler_connected addr=lb:500515652worker1 # [ 99.142910] nix-grpc-daemon[845]: ts=2026-09-22T10:45:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5653worker2 # [ 99.235821] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5654worker2 # [ 99.238034] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5655worker2 # [ 99.241551] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15656worker2 # [ 99.248063] nix-grpc-daemon[1307]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5657worker2 # [ 99.249391] nix-grpc-daemon[1307]: E0000 00:00:1790073941.215173 1307 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5658worker2 # [ 99.255692] nix-grpc-daemon[1307]: E0000 00:00:1790073941.220449 1307 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5659worker2 # [ 99.258916] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5660worker2 # [ 99.263135] systemd[1]: Started Nix worker-protocol over gRPC.5661lb # [ 99.286377] niks3-server[1077]: 2026/09/22 10:45:41 INFO lead: acquired remote=192.168.1.4:525825662worker2 # [ 99.271072] nix-grpc-daemon[1307]: E0000 00:00:1790073941.237755 1349 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5663worker2 # [ 99.275127] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=scheduler_take_over5664worker2 # [ 99.284385] nix-grpc-daemon[1307]: E0000 00:00:1790073941.251682 1349 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5665worker2 # [ 99.288126] nix-grpc-daemon[1307]: E0000 00:00:1790073941.254212 1349 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5666worker2 # [ 99.298864] nix-grpc-daemon[1307]: E0000 00:00:1790073941.266179 1349 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5667worker2 # [ 99.308241] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=scheduler_connected addr=lb:500515668worker2 # [ 99.309801] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5669worker1 # [ 99.343636] nix-grpc-daemon[845]: ts=2026-09-22T10:45:41Z level=info event=scheduler_connected addr=lb:500515670worker1 # [ 99.347553] nix-grpc-daemon[845]: ts=2026-09-22T10:45:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5671worker2 # [ 99.510717] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=scheduler_connected addr=lb:500515672worker2 # [ 99.512518] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5673worker1 # [ 99.548362] nix-grpc-daemon[845]: ts=2026-09-22T10:45:41Z level=info event=scheduler_connected addr=lb:500515674worker1 # [ 99.552278] nix-grpc-daemon[845]: ts=2026-09-22T10:45:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5675worker2 # [ 99.713167] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=scheduler_connected addr=lb:500515676worker2 # [ 99.714933] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5677worker1 # [ 99.752812] nix-grpc-daemon[845]: ts=2026-09-22T10:45:41Z level=info event=scheduler_connected addr=lb:500515678worker1 # [ 99.756595] nix-grpc-daemon[845]: ts=2026-09-22T10:45:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5679worker2 # [ 99.916196] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=scheduler_connected addr=lb:500515680worker2 # [ 99.920101] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5681worker1 # [ 99.957608] nix-grpc-daemon[845]: ts=2026-09-22T10:45:41Z level=info event=scheduler_connected addr=lb:500515682worker1 # [ 99.961484] nix-grpc-daemon[845]: ts=2026-09-22T10:45:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5683worker1 # [ 99.991259] nix-grpc-daemon[845]: ts=2026-09-22T10:45:41Z level=info event=scheduler_restarting5684worker1 # [ 100.014525] nix-daemon[911]: reaped child process 913, status = succeeded5685worker2 # [ 100.121327] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:42Z level=info event=scheduler_connected addr=lb:500515686worker2 # [ 100.125127] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5687worker1 # [ 100.169611] nix-grpc-daemon[845]: ts=2026-09-22T10:45:42Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5688worker2 # [ 100.325980] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:42Z level=info event=scheduler_connected addr=lb:500515689worker2 # [ 100.332811] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:42Z level=info event=oidc_jwks provider=mock keys=15690worker2 # [ 100.334186] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:42Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:597045691worker2 # [ 100.335746] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:42Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05692worker1 # [ 100.850754] nix-grpc-daemon[845]: E0000 00:00:1790073942.851928 890 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true5693worker1 # [ 100.852945] nix-grpc-daemon[845]: E0000 00:00:1790073942.852038 890 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5694worker1 # [ 101.734696] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5695worker1 # [ 101.740479] systemd[1]: Stopped Nix worker-protocol over gRPC.5696worker1 # [ 101.743333] systemd[1]: nix-grpc-daemon.service: Consumed 2.530s CPU time over 1min 23.369s wall clock time, 61.5M memory peak, 89.3K incoming IP traffic, 168.6K outgoing IP traffic.5697worker1 # [ 101.754575] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5698worker1 # [ 101.758685] systemd[1]: Closed Nix worker-protocol over gRPC.5699worker1 # [ 101.762290] systemd[1]: nix-grpc-daemon.socket: Consumed 371us CPU time over 1min 31.966s wall clock time, 256K memory peak, 89.2K incoming IP traffic, 72.1K outgoing IP traffic.5700lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5701lb: must succeed: curl -sf localhost:9901/clusters5702lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5703lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.05 seconds)5704client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25705worker2 # [ 102.001548] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:597145706worker2 # [ 102.012888] nix-daemon[931]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5707worker2 # [ 102.029095] nix-grpc-daemon[1307]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 40 ms (attempt 1/5)5708worker2 # [ 102.074896] nix-grpc-daemon[1307]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 98 ms (attempt 2/5)5709worker2 # [ 102.178898] nix-grpc-daemon[1307]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 255 ms (attempt 3/5)5710worker2 # [ 102.439963] nix-grpc-daemon[1307]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 578 ms (attempt 4/5)5711worker2 # [ 103.024995] nix-grpc-daemon[1307]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5712worker2 # [ 103.028779] nix-grpc-daemon[1307]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5713client # copying 2 paths...5714client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5715worker2 # [ 103.050124] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:597145716client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5717worker2 # [ 103.064282] nix-daemon[931]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5718worker2 # [ 103.085229] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:45Z level=info event=niks3_push_started pid=13675719worker2 # [ 103.268885] nix-grpc-daemon[1367]: time=2026-09-22T10:45:45.235Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5720worker2 # [ 103.434900] nix-daemon[931]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)5721worker2 # [ 103.445989] nix-daemon[931]: reaped child process 1383, status = succeeded5722lb # [ 103.479250] niks3-server[1077]: 2026/09/22 10:45:45 INFO Received uploads request method=POST path=/api/pending_closures5723lb # [ 103.487849] niks3-server[1077]: 2026/09/22 10:45:45 INFO Received uploads request method=POST path=/api/pending_closures5724worker2 # [ 103.478919] nix-grpc-daemon[1367]: time=2026-09-22T10:45:45.446Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5725worker2 # [ 103.480415] nix-grpc-daemon[1367]: time=2026-09-22T10:45:45.447Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5726worker2 # [ 103.482325] nix-grpc-daemon[1367]: time=2026-09-22T10:45:45.449Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5727lb # [ 103.533859] niks3-server[1077]: 2026/09/22 10:45:45 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5728lb # [ 103.544214] niks3-server[1077]: 2026/09/22 10:45:45 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5729lb # [ 103.551638] niks3-server[1077]: 2026/09/22 10:45:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5730lb # [ 103.556580] niks3-server[1077]: 2026/09/22 10:45:45 INFO Signed narinfos id=30 count=25731lb # [ 103.561447] niks3-server[1077]: 2026/09/22 10:45:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5732lb # [ 103.564530] niks3-server[1077]: 2026/09/22 10:45:45 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5733lb # [ 103.569900] niks3-server[1077]: 2026/09/22 10:45:45 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5734lb # [ 103.573992] niks3-server[1077]: 2026/09/22 10:45:45 INFO Signed narinfos id=29 count=15735worker2 # [ 103.559157] nix-grpc-daemon[1367]: time=2026-09-22T10:45:45.525Z level=INFO msg="Uploading 3 narinfos"5736lb # [ 103.594889] niks3-server[1077]: 2026/09/22 10:45:45 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5737lb # [ 103.598114] niks3-server[1077]: 2026/09/22 10:45:45 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5738lb # [ 103.600844] niks3-server[1077]: 2026/09/22 10:45:45 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5739lb # [ 103.603203] niks3-server[1077]: 2026/09/22 10:45:45 INFO Completed upload id=295740lb # [ 103.604694] niks3-server[1077]: 2026/09/22 10:45:45 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5741lb # [ 103.606766] niks3-server[1077]: 2026/09/22 10:45:45 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5742lb # [ 103.609289] niks3-server[1077]: 2026/09/22 10:45:45 INFO Completed upload id=305743worker2 # [ 103.594871] nix-grpc-daemon[1367]: time=2026-09-22T10:45:45.562Z level=INFO msg="Upload complete. (321ms)"5744worker2 # [ 103.599636] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59714 duration_s=0 paths=2 nar_bytes_in=3125745worker2 # [ 103.612263] nix-daemon[931]: reaped child process 1365, status = succeeded5746client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.82 seconds)5747worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5748worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.01 seconds)5749worker1 # [ 103.700680] systemd[1]: Listening on Nix worker-protocol over gRPC.5750worker1 # [ 103.706970] systemd[1]: Starting Nix worker-protocol over gRPC...5751worker1 # [ 103.833360] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:45Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5752worker1 # [ 103.835227] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:45Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5753worker1 # [ 103.838716] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:45Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15754worker1 # [ 103.846067] nix-grpc-daemon[1301]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5755worker1 # [ 103.847372] nix-grpc-daemon[1301]: E0000 00:00:1790073945.846794 1301 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5756worker1 # [ 103.853068] nix-grpc-daemon[1301]: E0000 00:00:1790073945.851988 1301 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5757worker1 # [ 103.857117] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:45Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5758worker1 # [ 103.861435] systemd[1]: Started Nix worker-protocol over gRPC.5759lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5760lb: must succeed: curl -sf localhost:9901/clusters5761worker1 # [ 103.868083] nix-grpc-daemon[1301]: E0000 00:00:1790073945.868880 1343 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5762worker1 # [ 103.881862] nix-grpc-daemon[1301]: E0000 00:00:1790073945.883182 1343 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5763worker1 # [ 103.887065] nix-grpc-daemon[1301]: E0000 00:00:1790073945.885526 1343 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5764worker1 # [ 103.895960] nix-grpc-daemon[1301]: E0000 00:00:1790073945.897284 1343 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5765worker1 # [ 103.905339] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:45Z level=info event=scheduler_connected addr=lb:500515766worker2 # [ 103.904607] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:45Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:597045767worker2 # [ 103.909112] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:45Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05768lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5769lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.07 seconds)5770(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 9.36 seconds)5771subtest: a niks3 restart does not cost the leader its role5772lb: must succeed: curl -sf localhost:9901/clusters5773lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5774lb: must succeed: curl -sf localhost:9901/clusters5775lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5776lb # [ 104.075758] systemd[1]: Stopping niks3 server...5777lb # [ 104.077355] niks3-server[1077]: 2026/09/22 10:45:45 INFO Shutdown signal received, draining in-flight requests timeout=10s5778lb # [ 104.082081] niks3-server[1077]: 2026/09/22 10:45:45 INFO lead: released remote=192.168.1.4:525825779worker1 # [ 104.068689] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:46Z level=info event=scheduler_lock_lost status=2005780worker2 # [ 104.067186] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:46Z level=info event=scheduler_lock_lost status=2005781lb # [ 104.097841] systemd[1]: niks3.service: Deactivated successfully.5782lb # [ 104.099526] systemd[1]: Stopped niks3 server.5783lb # [ 104.100334] systemd[1]: niks3.service: Consumed 588ms CPU time over 1min 19.476s wall clock time, 21.2M memory peak, 77.9K incoming IP traffic, 113.2K outgoing IP traffic.5784lb # [ 104.109675] systemd[1]: Starting niks3 server...5785lb # [ 104.185905] niks3-server[1215]: 2026/09/22 10:45:46 goose: no migrations to run. current version: 202609200000005786lb # [ 104.193797] niks3-server[1215]: 2026/09/22 10:45:46 OK 1_commit_pending_closure.sql (5.89ms)5787lb # [ 104.197908] niks3-server[1215]: 2026/09/22 10:45:46 OK 2_object_stats_trigger.sql (3.72ms)5788lb # [ 104.198970] niks3-server[1215]: 2026/09/22 10:45:46 goose: up to current file version: 25789lb # [ 104.201058] niks3-server[1215]: 2026/09/22 10:45:46 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key5790lb # [ 104.215175] niks3-server[1215]: 2026/09/22 10:45:46 INFO Read proxy enabled — serving cache objects from S35791lb # [ 104.216474] niks3-server[1215]: 2026/09/22 10:45:46 INFO Using socket-activated listener address=0.0.0.0:57515792lb # [ 104.218168] niks3-server[1215]: 2026/09/22 10:45:46 INFO systemd watchdog enabled interval=15s5793lb # [ 104.219925] niks3-server[1215]: 2026/09/22 10:45:46 INFO Starting HTTP server address=0.0.0.0:57515794lb # [ 104.221700] systemd[1]: Started niks3 server.5795lb: waiting for unit niks3.service5796lb: (finished: waiting for unit niks3.service, in 0.04 seconds)5797lb: must succeed: curl -sf localhost:9901/clusters5798lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5799worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5800worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5801lb: must succeed: curl -sf localhost:9901/clusters5802lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5803lb: must succeed: curl -sf localhost:9901/clusters5804lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5805worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5806worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.06 seconds)5807client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&25808worker2 # [ 104.701669] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:597145809client # don't know how to build these paths:5810client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5811worker2 # [ 104.710212] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:597145812worker2 # [ 104.736160] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:46Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=05813worker2 # [ 104.741569] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:46Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.4:50051 clients=15814worker2 # [ 104.779294] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:597145815worker2 # [ 104.787638] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:597145816client # copying 1 paths to 'lb:50051'...5817client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...5818worker2 # [ 104.795504] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:597145819worker2 # [ 104.801668] nix-daemon[931]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5820worker2 # [ 104.947065] nix-daemon[931]: accepted connection from pid 1400, user nix-grpc-daemon (trusted)5821worker2 # [ 104.959192] nix-daemon[931]: reaped child process 1403, status = succeeded5822lb # [ 105.085818] niks3-server[1215]: 2026/09/22 10:45:46 INFO lead: acquired remote=192.168.1.4:521065823worker2 # [ 105.110759] nix-daemon[931]: accepted connection from pid 1405, user nix-grpc-daemon (trusted)5824worker2 # [ 105.123049] nix-daemon[931]: reaped child process 1410, status = succeeded5825lb # [ 105.151233] niks3-server[1215]: 2026/09/22 10:45:46 INFO Received uploads request method=POST path=/api/pending_closures5826worker2 # [ 105.145795] nix-grpc-daemon[1367]: time=2026-09-22T10:45:47.112Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5827worker2 # [ 105.147200] nix-grpc-daemon[1367]: time=2026-09-22T10:45:47.112Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"5828lb # [ 105.172963] niks3-server[1215]: 2026/09/22 10:45:47 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst5829lb # [ 105.175196] niks3-server[1215]: 2026/09/22 10:45:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5830lb # [ 105.182616] niks3-server[1215]: 2026/09/22 10:45:47 INFO Signed narinfos id=31 count=15831worker2 # [ 105.167068] nix-grpc-daemon[1367]: time=2026-09-22T10:45:47.134Z level=INFO msg="Uploading 1 narinfos"5832lb # [ 105.191251] niks3-server[1215]: 2026/09/22 10:45:47 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5833lb # [ 105.197843] niks3-server[1215]: 2026/09/22 10:45:47 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls5834lb # [ 105.200544] niks3-server[1215]: 2026/09/22 10:45:47 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo5835lb # [ 105.207601] niks3-server[1215]: 2026/09/22 10:45:47 INFO Completed upload id=315836worker2 # [ 105.193043] nix-grpc-daemon[1367]: time=2026-09-22T10:45:47.160Z level=INFO msg="Upload complete. (386ms)"5837worker2 # [ 105.195245] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59714 duration_s=0 paths=1 nar_bytes_in=5365838worker2 # [ 105.197958] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:597145839worker2 # [ 105.203670] nix-daemon[931]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5840worker2 # [ 105.208112] nix-daemon[931]: reaped child process 1398, status = succeeded5841worker2 # [ 105.212611] nix-daemon[931]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5842client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...5843client # farm-a> worker2: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5844client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5845client # farm-a> LOG-farm-a-after-niks3-restart5846worker2 # [ 106.225393] nix-daemon[931]: reaped child process 1414, status = succeeded5847worker2 # [ 106.357336] nix-daemon[931]: accepted connection from pid 1430, user nix-grpc-daemon (trusted)5848worker2 # [ 106.367948] nix-daemon[931]: reaped child process 1433, status = succeeded5849lb # [ 106.396301] niks3-server[1215]: 2026/09/22 10:45:48 INFO Received uploads request method=POST path=/api/pending_closures5850worker2 # [ 106.391580] nix-grpc-daemon[1367]: time=2026-09-22T10:45:48.358Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5851worker2 # [ 106.393146] nix-grpc-daemon[1367]: time=2026-09-22T10:45:48.360Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"5852lb # [ 106.433385] niks3-server[1215]: 2026/09/22 10:45:48 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5853lb # [ 106.449364] niks3-server[1215]: 2026/09/22 10:45:48 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5854lb # [ 106.453062] niks3-server[1215]: 2026/09/22 10:45:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5855lb # [ 106.458073] niks3-server[1215]: 2026/09/22 10:45:48 INFO Signed narinfos id=32 count=15856worker2 # [ 106.445372] nix-grpc-daemon[1367]: time=2026-09-22T10:45:48.412Z level=INFO msg="Uploading 1 narinfos"5857lb # [ 106.465418] niks3-server[1215]: 2026/09/22 10:45:48 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls5858lb # [ 106.473255] niks3-server[1215]: 2026/09/22 10:45:48 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5859lb # [ 106.476359] niks3-server[1215]: 2026/09/22 10:45:48 INFO Completed upload id=325860lb # [ 106.478653] niks3-server[1215]: 2026/09/22 10:45:48 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo5861worker2 # [ 106.462226] nix-grpc-daemon[1367]: time=2026-09-22T10:45:48.429Z level=INFO msg="Upload complete. (241ms)"5862worker2 # [ 106.469800] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59714 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=15863worker2 # [ 106.481937] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:48Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=05864worker2 # [ 106.485145] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:48Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.4:50051 clients=15865worker2 # [ 106.494220] nix-daemon[931]: reaped child process 1412, status = succeeded5866worker2 # [ 106.524727] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:597145867worker2 # [ 106.537519] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:597145868worker2 # [ 106.550930] nix-grpc-daemon[1367]: time=2026-09-22T10:45:48.518Z level=INFO msg="All 1 paths already cached"5869client # copying 1 paths to 'lb:50051'...5870client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...5871worker2 # [ 106.558806] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:597145872worker2 # [ 106.569161] nix-daemon[931]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5873worker2 # [ 106.741223] nix-daemon[931]: accepted connection from pid 1439, user nix-grpc-daemon (trusted)5874worker2 # [ 106.753641] nix-daemon[931]: reaped child process 1442, status = succeeded5875worker2 # [ 106.916652] nix-daemon[931]: accepted connection from pid 1444, user nix-grpc-daemon (trusted)5876worker2 # [ 106.929098] nix-daemon[931]: reaped child process 1447, status = succeeded5877lb # [ 106.959094] niks3-server[1215]: 2026/09/22 10:45:48 INFO Received uploads request method=POST path=/api/pending_closures5878worker2 # [ 106.951108] nix-grpc-daemon[1367]: time=2026-09-22T10:45:48.918Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5879worker2 # [ 106.952915] nix-grpc-daemon[1367]: time=2026-09-22T10:45:48.919Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"5880lb # [ 106.988625] niks3-server[1215]: 2026/09/22 10:45:48 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst5881lb # [ 106.997780] niks3-server[1215]: 2026/09/22 10:45:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5882worker2 # [ 106.988868] nix-grpc-daemon[1367]: time=2026-09-22T10:45:48.956Z level=INFO msg="Uploading 1 narinfos"5883lb # [ 107.004577] niks3-server[1215]: 2026/09/22 10:45:48 INFO Signed narinfos id=33 count=15884lb # [ 107.008777] niks3-server[1215]: 2026/09/22 10:45:48 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls5885lb # [ 107.020798] niks3-server[1215]: 2026/09/22 10:45:48 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5886worker2 # [ 107.011166] nix-grpc-daemon[1367]: time=2026-09-22T10:45:48.978Z level=INFO msg="Upload complete. (431ms)"5887lb # [ 107.027046] niks3-server[1215]: 2026/09/22 10:45:48 INFO Completed upload id=335888worker2 # [ 107.013736] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59714 duration_s=0 paths=1 nar_bytes_in=6565889lb # [ 107.030925] niks3-server[1215]: 2026/09/22 10:45:48 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo5890worker2 # [ 107.019277] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:597145891worker2 # [ 107.027979] nix-daemon[931]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5892worker2 # [ 107.033060] nix-daemon[931]: reaped child process 1437, status = succeeded5893worker2 # [ 107.037921] nix-daemon[931]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5894client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...5895client # farm-b> worker2: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5896client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5897client # farm-b> LOG-farm-b-after-niks3-restart5898worker2 # [ 107.851754] nix-daemon[931]: reaped child process 1451, status = succeeded5899worker2 # [ 107.996112] nix-daemon[931]: accepted connection from pid 1467, user nix-grpc-daemon (trusted)5900worker2 # [ 108.008185] nix-daemon[931]: reaped child process 1470, status = succeeded5901lb # [ 108.040456] niks3-server[1215]: 2026/09/22 10:45:49 INFO Received uploads request method=POST path=/api/pending_closures5902worker2 # [ 108.029073] nix-grpc-daemon[1367]: time=2026-09-22T10:45:49.996Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5903worker2 # [ 108.030532] nix-grpc-daemon[1367]: time=2026-09-22T10:45:49.997Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"5904lb # [ 108.063279] niks3-server[1215]: 2026/09/22 10:45:49 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5905lb # [ 108.067440] niks3-server[1215]: 2026/09/22 10:45:49 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst5906lb # [ 108.071093] niks3-server[1215]: 2026/09/22 10:45:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5907lb # [ 108.074059] niks3-server[1215]: 2026/09/22 10:45:49 INFO Signed narinfos id=34 count=15908worker2 # [ 108.058182] nix-grpc-daemon[1367]: time=2026-09-22T10:45:50.025Z level=INFO msg="Uploading 1 narinfos"5909lb # [ 108.076849] niks3-server[1215]: 2026/09/22 10:45:49 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls5910lb # [ 108.081449] niks3-server[1215]: 2026/09/22 10:45:49 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5911lb # [ 108.084840] niks3-server[1215]: 2026/09/22 10:45:49 INFO Completed upload id=345912worker2 # [ 108.068949] nix-grpc-daemon[1367]: time=2026-09-22T10:45:50.036Z level=INFO msg="Upload complete. (225ms)"5913lb # [ 108.086356] niks3-server[1215]: 2026/09/22 10:45:49 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo5914worker2 # [ 108.071839] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59714 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=15915worker2 # [ 108.078550] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:50Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=05916worker2 # [ 108.080631] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:50Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.4:50051 clients=15917worker2 # [ 108.085640] nix-daemon[931]: reaped child process 1449, status = succeeded5918worker2 # [ 108.124321] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:597145919worker2 # [ 108.130886] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:597145920worker2 # [ 108.139076] nix-grpc-daemon[1367]: time=2026-09-22T10:45:50.106Z level=INFO msg="All 2 paths already cached"5921client # copying 1 paths to 'lb:50051'...5922worker2 # [ 108.144794] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:597145923client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...5924worker2 # [ 108.149676] nix-daemon[931]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5925worker2 # [ 108.296125] nix-daemon[931]: accepted connection from pid 1476, user nix-grpc-daemon (trusted)5926worker2 # [ 108.308223] nix-daemon[931]: reaped child process 1479, status = succeeded5927lb # [ 108.395507] envoy[987]: conn peer=192.168.1.3:49336 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=21593 tx=4155 ms=814175928lb # [ 108.399202] envoy[987]: conn peer=[2001:db8:1::1]:45826 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1005 tx=305 ms=16135929worker2 # [ 108.467030] nix-daemon[931]: accepted connection from pid 1481, user nix-grpc-daemon (trusted)5930worker2 # [ 108.478941] nix-daemon[931]: reaped child process 1484, status = succeeded5931lb # [ 108.505700] niks3-server[1215]: 2026/09/22 10:45:50 INFO Received uploads request method=POST path=/api/pending_closures5932worker2 # [ 108.498220] nix-grpc-daemon[1367]: time=2026-09-22T10:45:50.465Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5933worker2 # [ 108.499617] nix-grpc-daemon[1367]: time=2026-09-22T10:45:50.465Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"5934lb # [ 108.533657] niks3-server[1215]: 2026/09/22 10:45:50 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst5935lb # [ 108.541979] niks3-server[1215]: 2026/09/22 10:45:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5936worker2 # [ 108.533195] nix-grpc-daemon[1367]: time=2026-09-22T10:45:50.500Z level=INFO msg="Uploading 1 narinfos"5937lb # [ 108.548920] niks3-server[1215]: 2026/09/22 10:45:50 INFO Signed narinfos id=35 count=15938lb # [ 108.553574] niks3-server[1215]: 2026/09/22 10:45:50 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls5939lb # [ 108.565113] niks3-server[1215]: 2026/09/22 10:45:50 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5940worker2 # [ 108.553984] nix-grpc-daemon[1367]: time=2026-09-22T10:45:50.521Z level=INFO msg="Upload complete. (398ms)"5941lb # [ 108.569836] niks3-server[1215]: 2026/09/22 10:45:50 INFO Completed upload id=355942worker2 # [ 108.556033] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59714 duration_s=0 paths=1 nar_bytes_in=7845943worker2 # [ 108.559570] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:597145944lb # [ 108.575168] niks3-server[1215]: 2026/09/22 10:45:50 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo5945worker2 # [ 108.566108] nix-daemon[931]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5946worker2 # [ 108.569598] nix-daemon[931]: reaped child process 1474, status = succeeded5947worker2 # [ 108.573575] nix-daemon[931]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5948client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...5949client # farm-top> worker2: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5950client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5951client # farm-top> LOG-farm-top-after-niks3-restart5952worker2 # [ 109.788850] nix-daemon[931]: reaped child process 1488, status = succeeded5953worker2 # [ 109.905195] nix-daemon[931]: accepted connection from pid 1504, user nix-grpc-daemon (trusted)5954worker2 # [ 109.917916] nix-daemon[931]: reaped child process 1507, status = succeeded5955lb # [ 109.943855] niks3-server[1215]: 2026/09/22 10:45:51 INFO Received uploads request method=POST path=/api/pending_closures5956worker2 # [ 109.939291] nix-grpc-daemon[1367]: time=2026-09-22T10:45:51.906Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5957worker2 # [ 109.940804] nix-grpc-daemon[1367]: time=2026-09-22T10:45:51.908Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"5958lb # [ 109.983938] niks3-server[1215]: 2026/09/22 10:45:51 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst5959lb # [ 109.987789] niks3-server[1215]: 2026/09/22 10:45:51 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5960lb # [ 109.991095] niks3-server[1215]: 2026/09/22 10:45:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5961lb # [ 109.994040] niks3-server[1215]: 2026/09/22 10:45:51 INFO Signed narinfos id=36 count=15962worker2 # [ 109.978289] nix-grpc-daemon[1367]: time=2026-09-22T10:45:51.945Z level=INFO msg="Uploading 1 narinfos"5963lb # [ 109.999326] niks3-server[1215]: 2026/09/22 10:45:51 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls5964lb # [ 110.004992] niks3-server[1215]: 2026/09/22 10:45:51 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5965lb # [ 110.011869] niks3-server[1215]: 2026/09/22 10:45:51 INFO Completed upload id=365966worker2 # [ 109.995954] nix-grpc-daemon[1367]: time=2026-09-22T10:45:51.963Z level=INFO msg="Upload complete. (215ms)"5967lb # [ 110.013651] niks3-server[1215]: 2026/09/22 10:45:51 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo5968worker2 # [ 109.998567] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59714 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=15969worker2 # [ 110.005914] nix-daemon[931]: reaped child process 1486, status = succeeded5970client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&2, in 5.53 seconds)5971client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath5972client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath, in 0.16 seconds)5973(finished: subtest: a niks3 restart does not cost the leader its role, in 6.26 seconds)5974subtest: scheduler down: the other node takes the lock and keeps it5975lb: must succeed: curl -sf localhost:9901/clusters5976lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5977lb: must succeed: curl -sf localhost:9901/clusters5978lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5979lb: must succeed: curl -sf localhost:9901/clusters5980lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5981lb: must succeed: curl -sf localhost:9901/clusters5982lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5983worker2 # [ 110.431829] systemd[1]: Stopping Nix worker-protocol over gRPC...5984worker2 # [ 111.258807] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:53Z level=info event=signal_exit5985lb # [ 111.296573] niks3-server[1215]: 2026/09/22 10:45:53 INFO lead: released remote=192.168.1.4:521065986worker1 # [ 112.283321] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:54Z level=info event=scheduler_restarting addr=lb:500515987worker2 # [ 112.279323] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:54Z level=info event=scheduler_restarting5988worker2 # [ 112.284742] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:54Z level=info event=scheduler_restarting addr=lb:500515989worker2 # [ 112.288268] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:54Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:597045990worker2 # [ 112.292514] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:54Z level=info event=worker_gone addr=192.168.1.4:500515991worker2 # [ 112.295966] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:54Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:597045992worker2 # [ 112.299591] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:54Z level=info event=worker_gone addr=192.168.1.3:500515993worker2 # [ 112.302707] nix-grpc-daemon[1307]: E0000 00:00:1790073954.262896 1352 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5994worker2 # [ 112.307506] nix-grpc-daemon[1307]: E0000 00:00:1790073954.262989 1352 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true5995worker2 # [ 112.326329] nix-daemon[931]: reaped child process 1359, status = succeeded5996worker1 # [ 112.482693] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:54Z level=info event=scheduler_connected addr=lb:500515997worker1 # [ 112.484767] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5998worker2 # [ 112.492788] nix-grpc-daemon[1307]: ts=2026-09-22T10:45:54Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5999worker1 # [ 112.685363] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:54Z level=info event=scheduler_connected addr=lb:500516000worker1 # [ 112.687172] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6001worker1 # [ 112.887689] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:54Z level=info event=scheduler_connected addr=lb:500516002worker1 # [ 112.891410] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6003worker1 # [ 113.091800] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:55Z level=info event=scheduler_connected addr=lb:500516004worker1 # [ 113.093692] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6005worker1 # [ 113.294601] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:55Z level=info event=scheduler_connected addr=lb:500516006worker1 # [ 113.296328] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6007worker1 # [ 113.497505] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:55Z level=info event=scheduler_connected addr=lb:500516008worker1 # [ 113.501366] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6009worker1 # [ 113.701803] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:55Z level=info event=scheduler_connected addr=lb:500516010worker1 # [ 113.703501] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6011worker1 # [ 113.904264] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:55Z level=info event=scheduler_connected addr=lb:500516012worker1 # [ 113.908064] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6013worker1 # [ 114.109071] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:56Z level=info event=scheduler_connected addr=lb:500516014worker1 # [ 114.113118] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6015worker1 # [ 114.314488] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:56Z level=info event=scheduler_connected addr=lb:500516016worker1 # [ 114.317427] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6017worker2 # [ 114.383385] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6018worker2 # [ 114.385318] systemd[1]: Stopped Nix worker-protocol over gRPC.6019worker2 # [ 114.386114] systemd[1]: nix-grpc-daemon.service: Consumed 1.026s CPU time over 15.295s wall clock time, 38.2M memory peak, 46.1K incoming IP traffic, 83.2K outgoing IP traffic.6020worker2 # [ 114.390844] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.6021worker2 # [ 114.392553] systemd[1]: Closed Nix worker-protocol over gRPC.6022worker2 # [ 114.394162] systemd[1]: nix-grpc-daemon.socket: Consumed 291us CPU time over 16.282s wall clock time, 256K memory peak, 37K incoming IP traffic, 26.9K outgoing IP traffic.6023worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over6024worker1 # [ 114.518324] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:56Z level=info event=scheduler_connected addr=lb:500516025worker1 # [ 114.522265] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6026worker1 # [ 114.723243] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:56Z level=info event=scheduler_connected addr=lb:500516027worker1 # [ 114.727210] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6028worker1 # [ 114.927922] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:56Z level=info event=scheduler_connected addr=lb:500516029worker1 # [ 114.931895] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6030lb # [ 115.089050] niks3-server[1215]: 2026/09/22 10:45:56 INFO lead: acquired remote=192.168.1.3:397946031worker1 # [ 115.075922] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=scheduler_take_over6032worker1 # [ 115.132637] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=scheduler_connected addr=lb:500516033worker1 # [ 115.136413] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6034worker1 # [ 115.338739] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=scheduler_connected addr=lb:500516035worker1 # [ 115.341867] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6036worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 1.13 seconds)6037worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6038worker1 # [ 115.538772] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=scheduler_connected addr=lb:500516039worker1 # [ 115.540472] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6040worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6041worker1 # [ 115.741184] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=scheduler_connected addr=lb:500516042worker1 # [ 115.743082] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6043worker1 # [ 115.943412] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=scheduler_connected addr=lb:500516044worker1 # [ 115.946746] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:434526045worker1 # [ 115.948772] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06046worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6047worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6048client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&26049worker1 # [ 116.808646] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:434546050worker1 # [ 116.818282] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6051client # don't know how to build these paths:6052client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6053worker1 # [ 116.830626] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434546054worker1 # [ 116.841215] nix-grpc-daemon[1301]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 27 ms (attempt 1/5)6055worker1 # [ 116.868503] nix-grpc-daemon[1301]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 182 ms (attempt 2/5)6056worker1 # [ 117.056826] nix-grpc-daemon[1301]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 103 ms (attempt 3/5)6057worker1 # [ 117.166193] nix-grpc-daemon[1301]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 472 ms (attempt 4/5)6058worker1 # [ 117.644162] nix-grpc-daemon[1301]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6059worker1 # [ 117.645455] nix-grpc-daemon[1301]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6060worker1 # [ 117.659339] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:59Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=06061worker1 # [ 117.661226] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:59Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=16062worker1 # [ 117.705816] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434546063worker1 # [ 117.711814] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434546064client # copying 1 paths to 'lb:50051'...6065client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...6066worker1 # [ 117.717112] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434546067worker1 # [ 117.720980] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6068worker1 # [ 117.733473] nix-grpc-daemon[1301]: ts=2026-09-22T10:45:59Z level=info event=niks3_push_started pid=13836069worker1 # [ 117.906458] nix-grpc-daemon[1383]: time=2026-09-22T10:45:59.907Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6070worker1 # [ 118.064319] nix-daemon[911]: accepted connection from pid 1396, user nix-grpc-daemon (trusted)6071worker1 # [ 118.075952] nix-daemon[911]: reaped child process 1399, status = succeeded6072worker1 # [ 118.197208] nix-daemon[911]: accepted connection from pid 1401, user nix-grpc-daemon (trusted)6073worker1 # [ 118.209048] nix-daemon[911]: reaped child process 1404, status = succeeded6074lb # [ 118.237552] niks3-server[1215]: 2026/09/22 10:46:00 INFO Received uploads request method=POST path=/api/pending_closures6075worker1 # [ 118.229077] nix-grpc-daemon[1383]: time=2026-09-22T10:46:00.229Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6076worker1 # [ 118.230480] nix-grpc-daemon[1383]: time=2026-09-22T10:46:00.229Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"6077lb # [ 118.257691] niks3-server[1215]: 2026/09/22 10:46:00 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst6078lb # [ 118.259688] niks3-server[1215]: 2026/09/22 10:46:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6079lb # [ 118.261783] niks3-server[1215]: 2026/09/22 10:46:00 INFO Signed narinfos id=37 count=16080worker1 # [ 118.248708] nix-grpc-daemon[1383]: time=2026-09-22T10:46:00.249Z level=INFO msg="Uploading 1 narinfos"6081lb # [ 118.266920] niks3-server[1215]: 2026/09/22 10:46:00 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls6082lb # [ 118.270148] niks3-server[1215]: 2026/09/22 10:46:00 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6083lb # [ 118.273572] niks3-server[1215]: 2026/09/22 10:46:00 INFO Completed upload id=376084worker1 # [ 118.260196] nix-grpc-daemon[1383]: time=2026-09-22T10:46:00.261Z level=INFO msg="Upload complete. (349ms)"6085lb # [ 118.276316] niks3-server[1215]: 2026/09/22 10:46:00 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo6086worker1 # [ 118.262488] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43454 duration_s=0 paths=1 nar_bytes_in=5206087worker1 # [ 118.265685] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434546088worker1 # [ 118.271124] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6089worker1 # [ 118.275472] nix-daemon[911]: reaped child process 1381, status = succeeded6090client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...6091worker1 # [ 118.281330] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6092client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6093lb # [ 118.396424] envoy[987]: conn peer=[2001:db8:1::1]:59036 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9722 tx=4089 ms=53136094lb # [ 118.400154] envoy[987]: conn peer=[2001:db8:1::4]:54722 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=5880 tx=648 ms=131786095client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6096client # farm-a> LOG-farm-a-on-standby6097worker1 # [ 119.698591] nix-daemon[911]: reaped child process 1408, status = succeeded6098worker1 # [ 119.838103] nix-daemon[911]: accepted connection from pid 1424, user nix-grpc-daemon (trusted)6099worker1 # [ 119.850870] nix-daemon[911]: reaped child process 1427, status = succeeded6100lb # [ 119.875359] niks3-server[1215]: 2026/09/22 10:46:01 INFO Received uploads request method=POST path=/api/pending_closures6101worker1 # [ 119.868939] nix-grpc-daemon[1383]: time=2026-09-22T10:46:01.870Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6102worker1 # [ 119.870573] nix-grpc-daemon[1383]: time=2026-09-22T10:46:01.871Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"6103lb # [ 119.900374] niks3-server[1215]: 2026/09/22 10:46:01 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6104lb # [ 119.903550] niks3-server[1215]: 2026/09/22 10:46:01 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst6105lb # [ 119.907024] niks3-server[1215]: 2026/09/22 10:46:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6106lb # [ 119.909908] niks3-server[1215]: 2026/09/22 10:46:01 INFO Signed narinfos id=38 count=16107worker1 # [ 119.897076] nix-grpc-daemon[1383]: time=2026-09-22T10:46:01.898Z level=INFO msg="Uploading 1 narinfos"6108lb # [ 119.913075] niks3-server[1215]: 2026/09/22 10:46:01 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls6109lb # [ 119.918410] niks3-server[1215]: 2026/09/22 10:46:01 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6110lb # [ 119.922122] niks3-server[1215]: 2026/09/22 10:46:01 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo6111lb # [ 119.925145] niks3-server[1215]: 2026/09/22 10:46:01 INFO Completed upload id=386112worker1 # [ 119.911683] nix-grpc-daemon[1383]: time=2026-09-22T10:46:01.913Z level=INFO msg="Upload complete. (221ms)"6113worker1 # [ 119.914557] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43454 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=16114worker1 # [ 119.926948] nix-daemon[911]: reaped child process 1406, status = succeeded6115worker1 # [ 119.928928] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:01Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=06116worker1 # [ 119.931150] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:01Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=16117worker1 # [ 119.972499] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434546118worker1 # [ 119.981436] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434546119worker1 # [ 119.990105] nix-grpc-daemon[1383]: time=2026-09-22T10:46:01.991Z level=INFO msg="All 1 paths already cached"6120client # copying 1 paths to 'lb:50051'...6121client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...6122worker1 # [ 119.996557] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434546123worker1 # [ 120.003539] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6124worker1 # [ 120.157621] nix-daemon[911]: accepted connection from pid 1433, user nix-grpc-daemon (trusted)6125worker1 # [ 120.170528] nix-daemon[911]: reaped child process 1436, status = succeeded6126worker1 # [ 120.306149] nix-daemon[911]: accepted connection from pid 1438, user nix-grpc-daemon (trusted)6127worker1 # [ 120.317883] nix-daemon[911]: reaped child process 1441, status = succeeded6128lb # [ 120.342217] niks3-server[1215]: 2026/09/22 10:46:02 INFO Received uploads request method=POST path=/api/pending_closures6129worker1 # [ 120.339359] nix-grpc-daemon[1383]: time=2026-09-22T10:46:02.340Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6130worker1 # [ 120.340691] nix-grpc-daemon[1383]: time=2026-09-22T10:46:02.340Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"6131lb # [ 120.371558] niks3-server[1215]: 2026/09/22 10:46:02 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst6132lb # [ 120.379499] niks3-server[1215]: 2026/09/22 10:46:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6133worker1 # [ 120.372033] nix-grpc-daemon[1383]: time=2026-09-22T10:46:02.373Z level=INFO msg="Uploading 1 narinfos"6134lb # [ 120.385408] niks3-server[1215]: 2026/09/22 10:46:02 INFO Signed narinfos id=39 count=16135lb # [ 120.391914] niks3-server[1215]: 2026/09/22 10:46:02 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls6136lb # [ 120.401473] niks3-server[1215]: 2026/09/22 10:46:02 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6137lb # [ 120.406089] niks3-server[1215]: 2026/09/22 10:46:02 INFO Completed upload id=396138worker1 # [ 120.394914] nix-grpc-daemon[1383]: time=2026-09-22T10:46:02.396Z level=INFO msg="Upload complete. (382ms)"6139worker1 # [ 120.396854] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43454 duration_s=0 paths=1 nar_bytes_in=6406140lb # [ 120.411596] niks3-server[1215]: 2026/09/22 10:46:02 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo6141worker1 # [ 120.402405] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434546142worker1 # [ 120.407112] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6143worker1 # [ 120.409332] nix-daemon[911]: reaped child process 1431, status = succeeded6144client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...6145client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6146worker1 # [ 120.414415] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6147client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6148client # farm-b> LOG-farm-b-on-standby6149worker1 # [ 121.027752] nix-daemon[911]: reaped child process 1445, status = succeeded6150worker1 # [ 121.177196] nix-daemon[911]: accepted connection from pid 1461, user nix-grpc-daemon (trusted)6151worker1 # [ 121.189244] nix-daemon[911]: reaped child process 1464, status = succeeded6152lb # [ 121.215235] niks3-server[1215]: 2026/09/22 10:46:03 INFO Received uploads request method=POST path=/api/pending_closures6153worker1 # [ 121.210428] nix-grpc-daemon[1383]: time=2026-09-22T10:46:03.211Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6154worker1 # [ 121.211893] nix-grpc-daemon[1383]: time=2026-09-22T10:46:03.213Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"6155lb # [ 121.248072] niks3-server[1215]: 2026/09/22 10:46:03 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst6156lb # [ 121.263814] niks3-server[1215]: 2026/09/22 10:46:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6157worker1 # [ 121.254605] nix-grpc-daemon[1383]: time=2026-09-22T10:46:03.255Z level=INFO msg="Uploading 1 narinfos"6158lb # [ 121.268125] niks3-server[1215]: 2026/09/22 10:46:03 INFO Signed narinfos id=40 count=16159lb # [ 121.275451] niks3-server[1215]: 2026/09/22 10:46:03 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls6160lb # [ 121.282365] niks3-server[1215]: 2026/09/22 10:46:03 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6161lb # [ 121.290399] niks3-server[1215]: 2026/09/22 10:46:03 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6162lb # [ 121.293964] niks3-server[1215]: 2026/09/22 10:46:03 INFO Completed upload id=406163worker1 # [ 121.280575] nix-grpc-daemon[1383]: time=2026-09-22T10:46:03.281Z level=INFO msg="Upload complete. (260ms)"6164lb # [ 121.295530] niks3-server[1215]: 2026/09/22 10:46:03 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo6165worker1 # [ 121.283291] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43454 duration_s=0 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=16166worker1 # [ 121.290272] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:03Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=06167worker1 # [ 121.292239] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:03Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=16168worker1 # [ 121.295925] nix-daemon[911]: reaped child process 1443, status = succeeded6169worker1 # [ 121.333924] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434546170worker1 # [ 121.345243] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434546171worker1 # [ 121.364151] nix-grpc-daemon[1383]: time=2026-09-22T10:46:03.365Z level=INFO msg="All 2 paths already cached"6172client # copying 1 paths to 'lb:50051'...6173client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...6174worker1 # [ 121.373566] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434546175worker1 # [ 121.381818] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6176worker1 # [ 121.557411] nix-daemon[911]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6177worker1 # [ 121.569982] nix-daemon[911]: reaped child process 1473, status = succeeded6178worker1 # [ 121.713904] nix-daemon[911]: accepted connection from pid 1475, user nix-grpc-daemon (trusted)6179worker1 # [ 121.725795] nix-daemon[911]: reaped child process 1478, status = succeeded6180lb # [ 121.755371] niks3-server[1215]: 2026/09/22 10:46:03 INFO Received uploads request method=POST path=/api/pending_closures6181worker1 # [ 121.750250] nix-grpc-daemon[1383]: time=2026-09-22T10:46:03.751Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6182worker1 # [ 121.752208] nix-grpc-daemon[1383]: time=2026-09-22T10:46:03.753Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6183lb # [ 121.783778] niks3-server[1215]: 2026/09/22 10:46:03 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6184lb # [ 121.793190] niks3-server[1215]: 2026/09/22 10:46:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6185lb # [ 121.797506] niks3-server[1215]: 2026/09/22 10:46:03 INFO Signed narinfos id=41 count=16186worker1 # [ 121.787084] nix-grpc-daemon[1383]: time=2026-09-22T10:46:03.788Z level=INFO msg="Uploading 1 narinfos"6187lb # [ 121.810429] niks3-server[1215]: 2026/09/22 10:46:03 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6188lb # [ 121.816797] niks3-server[1215]: 2026/09/22 10:46:03 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6189lb # [ 121.823884] niks3-server[1215]: 2026/09/22 10:46:03 INFO Completed upload id=416190worker1 # [ 121.810564] nix-grpc-daemon[1383]: time=2026-09-22T10:46:03.811Z level=INFO msg="Upload complete. (415ms)"6191lb # [ 121.827262] niks3-server[1215]: 2026/09/22 10:46:03 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6192worker1 # [ 121.813643] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43454 duration_s=0 paths=1 nar_bytes_in=7686193worker1 # [ 121.818323] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434546194worker1 # [ 121.826466] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6195worker1 # [ 121.833604] nix-daemon[911]: reaped child process 1468, status = succeeded6196client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6197worker1 # [ 121.838539] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6198client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6199client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6200client # farm-top> LOG-farm-top-on-standby6201worker1 # [ 122.659840] nix-daemon[911]: reaped child process 1482, status = succeeded6202worker1 # [ 122.798537] nix-daemon[911]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6203worker1 # [ 122.811714] nix-daemon[911]: reaped child process 1501, status = succeeded6204lb # [ 122.840707] niks3-server[1215]: 2026/09/22 10:46:04 INFO Received uploads request method=POST path=/api/pending_closures6205worker1 # [ 122.835494] nix-grpc-daemon[1383]: time=2026-09-22T10:46:04.836Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6206worker1 # [ 122.837046] nix-grpc-daemon[1383]: time=2026-09-22T10:46:04.838Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6207lb # [ 122.880153] niks3-server[1215]: 2026/09/22 10:46:04 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6208lb # [ 122.889404] niks3-server[1215]: 2026/09/22 10:46:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6209lb # [ 122.897824] niks3-server[1215]: 2026/09/22 10:46:04 INFO Signed narinfos id=42 count=16210worker1 # [ 122.887608] nix-grpc-daemon[1383]: time=2026-09-22T10:46:04.888Z level=INFO msg="Uploading 1 narinfos"6211lb # [ 122.904964] niks3-server[1215]: 2026/09/22 10:46:04 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6212lb # [ 122.920944] niks3-server[1215]: 2026/09/22 10:46:04 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6213lb # [ 122.925441] niks3-server[1215]: 2026/09/22 10:46:04 INFO Completed upload id=426214worker1 # [ 122.913292] nix-grpc-daemon[1383]: time=2026-09-22T10:46:04.914Z level=INFO msg="Upload complete. (268ms)"6215lb # [ 122.927032] niks3-server[1215]: 2026/09/22 10:46:04 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6216worker1 # [ 122.916408] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43454 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16217lb # [ 122.931578] niks3-server[1215]: 2026/09/22 10:46:04 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6218worker1 # [ 122.923521] nix-daemon[911]: reaped child process 1480, status = succeeded6219client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 6.33 seconds)6220client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath6221client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.19 seconds)6222worker2 # [ 123.211582] systemd[1]: Listening on Nix worker-protocol over gRPC.6223worker2 # [ 123.218832] systemd[1]: Starting scheduler-token.service...6224worker2 # [ 123.286556] systemd[1]: scheduler-token.service: Deactivated successfully.6225worker2 # [ 123.287710] systemd[1]: Finished scheduler-token.service.6226worker2 # [ 123.289543] systemd[1]: scheduler-token.service: Consumed 20ms CPU time over 68ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6227worker2 # [ 123.294638] systemd[1]: Starting Nix worker-protocol over gRPC...6228worker2 # [ 123.451408] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:05Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6229worker2 # [ 123.453283] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:05Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6230worker2 # [ 123.456994] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:05Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16231worker2 # [ 123.464114] nix-grpc-daemon[1528]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6232worker2 # [ 123.467107] nix-grpc-daemon[1528]: E0000 00:00:1790073965.431233 1528 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6233worker2 # [ 123.469264] nix-grpc-daemon[1528]: E0000 00:00:1790073965.434388 1528 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6234worker2 # [ 123.476783] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:05Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6235worker2 # [ 123.479951] systemd[1]: Started Nix worker-protocol over gRPC.6236worker2 # [ 123.487815] nix-grpc-daemon[1528]: E0000 00:00:1790073965.455187 1570 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6237lb: must succeed: curl -sf localhost:9901/clusters6238worker2 # [ 123.499972] nix-grpc-daemon[1528]: E0000 00:00:1790073965.467343 1570 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6239worker2 # [ 123.504054] nix-grpc-daemon[1528]: E0000 00:00:1790073965.469620 1570 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6240worker2 # [ 123.514063] nix-grpc-daemon[1528]: E0000 00:00:1790073965.481341 1570 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6241worker2 # [ 123.523113] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:05Z level=info event=scheduler_connected addr=lb:500516242worker1 # [ 123.534300] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:05Z level=info event=oidc_jwks provider=mock keys=16243worker1 # [ 123.538852] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:05Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:434526244worker1 # [ 123.542625] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:05Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06245lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6246worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6247worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6248lb: must succeed: curl -sf localhost:9901/clusters6249lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6250lb: must succeed: curl -sf localhost:9901/clusters6251lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6252worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6253worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6254(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 13.61 seconds)6255subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6256client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs6257client # [ 123.876849] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.6258client # Running as unit: rs.service; invocation ID: 030011580e074c3c82e9387f571f61456259client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.07 seconds)6260worker1 # [ 124.083788] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:434546261client # [ 124.103413] nix[1299]: don't know how to build these paths:6262client # [ 124.104349] nix[1299]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6263worker1 # [ 124.098720] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:06Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06264worker1 # [ 124.103968] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:06Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=16265worker1 # [ 124.142222] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434546266worker1 # [ 124.148433] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434546267worker1 # [ 124.154203] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434546268client # [ 124.167264] nix[1299]: copying 1 paths to 'lb:50051'...6269client # [ 124.170165] nix[1299]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6270worker1 # [ 124.159428] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6271worker1 # [ 124.309126] nix-daemon[911]: accepted connection from pid 1514, user nix-grpc-daemon (trusted)6272worker1 # [ 124.320931] nix-daemon[911]: reaped child process 1517, status = succeeded6273worker1 # [ 124.463312] nix-daemon[911]: accepted connection from pid 1519, user nix-grpc-daemon (trusted)6274worker1 # [ 124.474888] nix-daemon[911]: reaped child process 1522, status = succeeded6275lb # [ 124.505074] niks3-server[1215]: 2026/09/22 10:46:06 INFO Received uploads request method=POST path=/api/pending_closures6276worker1 # [ 124.505228] nix-grpc-daemon[1383]: time=2026-09-22T10:46:06.506Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6277worker1 # [ 124.506752] nix-grpc-daemon[1383]: time=2026-09-22T10:46:06.506Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6278lb # [ 124.542730] niks3-server[1215]: 2026/09/22 10:46:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6279lb # [ 124.547877] niks3-server[1215]: 2026/09/22 10:46:06 INFO Signed narinfos id=43 count=16280worker1 # [ 124.537538] nix-grpc-daemon[1383]: time=2026-09-22T10:46:06.538Z level=INFO msg="Uploading 1 narinfos"6281lb # [ 124.553152] niks3-server[1215]: 2026/09/22 10:46:06 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6282lb # [ 124.565791] niks3-server[1215]: 2026/09/22 10:46:06 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6283lb # [ 124.568622] niks3-server[1215]: 2026/09/22 10:46:06 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6284lb # [ 124.577466] niks3-server[1215]: 2026/09/22 10:46:06 INFO Completed upload id=436285worker1 # [ 124.564107] nix-grpc-daemon[1383]: time=2026-09-22T10:46:06.565Z level=INFO msg="Upload complete. (398ms)"6286lb # [ 124.578942] niks3-server[1215]: 2026/09/22 10:46:06 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6287worker1 # [ 124.566204] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43454 duration_s=0 paths=1 nar_bytes_in=4166288worker1 # [ 124.569852] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434546289worker1 # [ 124.578792] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6290worker1 # [ 124.583769] nix-daemon[911]: reaped child process 1512, status = succeeded6291worker1 # [ 124.588820] nix-daemon[911]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)6292client # [ 124.601449] nix[1299]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6293lb: must succeed: curl -sf localhost:9901/clusters6294lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6295lb: must succeed: curl -sf localhost:9901/clusters6296lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6297lb: must succeed: curl -sf localhost:9901/clusters6298lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6299lb: must succeed: curl -sf localhost:9901/clusters6300lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6301worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6302worker2 # [ 126.731840] systemd-journald[285]: Received client request to rotate journal, rotating.6303worker2 # [ 126.750914] systemd-journald[285]: Vacuuming done, freed 0B of archived journals from /var/log/journal/8f5fe088a01f4e349c60e068e2cc2fbc.6304worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.07 seconds)6305worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service6306worker1 # [ 126.669572] systemd[1]: Stopping Nix worker-protocol over gRPC...6307worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)6308client: must succeed: sleep 5.06309worker1 # [ 126.863989] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:08Z level=info event=draining6310lb # [ 128.396965] envoy[987]: conn peer=[2001:db8:1::1]:49802 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9564 tx=4084 ms=61236311client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6312worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6313worker1 # [ 132.022459] nix-daemon[911]: reaped child process 1526, status = succeeded6314worker1 # [ 132.169144] nix-daemon[911]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6315worker1 # [ 132.180885] nix-daemon[911]: reaped child process 1570, status = succeeded6316lb # [ 132.206876] niks3-server[1215]: 2026/09/22 10:46:14 INFO Received uploads request method=POST path=/api/pending_closures6317worker1 # [ 132.203504] nix-grpc-daemon[1383]: time=2026-09-22T10:46:14.204Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6318worker1 # [ 132.205034] nix-grpc-daemon[1383]: time=2026-09-22T10:46:14.206Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6319lb # [ 132.248209] niks3-server[1215]: 2026/09/22 10:46:14 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6320lb # [ 132.256347] niks3-server[1215]: 2026/09/22 10:46:14 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6321lb # [ 132.267514] niks3-server[1215]: 2026/09/22 10:46:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6322lb # [ 132.273049] niks3-server[1215]: 2026/09/22 10:46:14 INFO Signed narinfos id=44 count=16323worker1 # [ 132.262632] nix-grpc-daemon[1383]: time=2026-09-22T10:46:14.263Z level=INFO msg="Uploading 1 narinfos"6324lb # [ 132.279185] niks3-server[1215]: 2026/09/22 10:46:14 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6325lb # [ 132.288702] niks3-server[1215]: 2026/09/22 10:46:14 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6326lb # [ 132.295130] niks3-server[1215]: 2026/09/22 10:46:14 INFO Completed upload id=446327worker1 # [ 132.281935] nix-grpc-daemon[1383]: time=2026-09-22T10:46:14.283Z level=INFO msg="Upload complete. (280ms)"6328lb # [ 132.296814] niks3-server[1215]: 2026/09/22 10:46:14 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6329worker1 # [ 132.284482] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43454 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16330worker1 # [ 132.292139] nix-daemon[911]: reaped child process 1524, status = succeeded6331client # [ 132.326528] systemd[1]: rs.service: Deactivated successfully.6332client # [ 132.328170] systemd[1]: rs.service: Consumed 120ms CPU time over 8.451s wall clock time, 39.4M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.6333worker1 # [ 132.864775] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:14Z level=info event=signal_exit6334lb # [ 133.117235] niks3-server[1215]: 2026/09/22 10:46:14 INFO lead: released remote=192.168.1.3:397946335lb # [ 133.503524] niks3-server[1215]: 2026/09/22 10:46:15 INFO lead: acquired remote=192.168.1.4:458966336worker2 # [ 133.491254] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:15Z level=info event=scheduler_take_over6337worker2 # [ 134.101339] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:16Z level=info event=scheduler_restarting addr=lb:500516338worker1 # [ 134.101989] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:16Z level=info event=scheduler_restarting6339worker1 # [ 134.111609] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:16Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:434526340worker1 # [ 134.115369] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:16Z level=info event=worker_gone addr=192.168.1.4:500516341worker1 # [ 134.118602] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:16Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:434526342worker1 # [ 134.121952] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:16Z level=info event=worker_gone addr=192.168.1.3:500516343worker1 # [ 134.125367] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:16Z level=info event=scheduler_restarting addr=lb:500516344worker1 # [ 134.141347] nix-daemon[911]: reaped child process 1373, status = succeeded6345worker2 # [ 134.302835] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:16Z level=info event=scheduler_connected addr=lb:500516346worker2 # [ 134.307287] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6347worker1 # [ 134.318475] nix-grpc-daemon[1301]: ts=2026-09-22T10:46:16Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6348worker2 # [ 134.517673] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:16Z level=info event=scheduler_connected addr=lb:500516349worker2 # [ 134.524466] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:16Z level=info event=oidc_jwks provider=mock keys=16350worker2 # [ 134.527927] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:16Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:421206351worker2 # [ 134.533092] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:16Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06352worker1 # [ 134.919384] nix-grpc-daemon[1301]: E0000 00:00:1790073976.920571 1346 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true6353worker1 # [ 134.921392] nix-grpc-daemon[1301]: E0000 00:00:1790073976.920718 1346 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6354worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.22 seconds)6355client: waiting for success: ! systemctl is-active rs6356client: (finished: waiting for success: ! systemctl is-active rs, in 0.03 seconds)6357client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6358client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)6359client: must fail: journalctl -u rs | grep -q 'reconnecting'6360client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.02 seconds)6361lb: must succeed: curl -sf localhost:9901/clusters6362lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6363worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6364worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6365worker1 # [ 135.673621] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6366worker1 # [ 135.676691] systemd[1]: Stopped Nix worker-protocol over gRPC.6367worker1 # [ 135.679482] systemd[1]: nix-grpc-daemon.service: Consumed 1.260s CPU time over 31.966s wall clock time, 35.7M memory peak, 49K incoming IP traffic, 95.6K outgoing IP traffic.6368worker1 # [ 135.694870] systemd[1]: Starting Nix worker-protocol over gRPC...6369worker1 # [ 135.850858] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:17Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6370worker1 # [ 135.853043] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:17Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6371worker1 # [ 135.856656] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:17Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16372worker1 # [ 135.864067] nix-grpc-daemon[1577]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6373worker1 # [ 135.865430] nix-grpc-daemon[1577]: E0000 00:00:1790073977.864822 1577 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6374worker1 # [ 135.870458] nix-grpc-daemon[1577]: E0000 00:00:1790073977.869816 1577 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6375worker1 # [ 135.874054] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:17Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6376worker1 # [ 135.876912] systemd[1]: Started Nix worker-protocol over gRPC.6377worker1 # [ 135.882063] nix-grpc-daemon[1577]: E0000 00:00:1790073977.882236 1619 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6378worker1 # [ 135.892953] nix-grpc-daemon[1577]: E0000 00:00:1790073977.894267 1619 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6379worker1 # [ 135.897053] nix-grpc-daemon[1577]: E0000 00:00:1790073977.896502 1619 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6380worker1 # [ 135.906882] nix-grpc-daemon[1577]: E0000 00:00:1790073977.908210 1619 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6381worker1 # [ 135.916468] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:17Z level=info event=scheduler_connected addr=lb:500516382worker2 # [ 135.916322] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:17Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:421206383worker2 # [ 135.920587] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:17Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06384lb: must succeed: curl -sf localhost:9901/clusters6385lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6386worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6387worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6388(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.66 seconds)6389subtest: scheduler restart mid-build: build finishes, no double build6390client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid6391client # [ 136.542950] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.6392client # Running as unit: mid.service; invocation ID: cbd361af806f426ba3d09c9bffe85a796393client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)6394worker1 # [ 136.729932] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:380106395worker1 # [ 136.740176] nix-daemon[911]: accepted connection from pid 1577, user nix-grpc-daemon (trusted)6396client # [ 136.765745] nix[1335]: don't know how to build these paths:6397client # [ 136.766742] nix[1335]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6398worker2 # [ 136.758666] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:18Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06399worker2 # [ 136.763915] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:18Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=16400worker2 # [ 136.810704] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421286401worker2 # [ 136.821907] nix-daemon[931]: accepted connection from pid 1528, user nix-grpc-daemon (trusted)6402worker2 # [ 136.839226] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 56 ms (attempt 1/5)6403worker2 # [ 136.901210] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 62 ms (attempt 2/5)6404worker2 # [ 136.969673] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 61 ms (attempt 3/5)6405worker2 # [ 137.036495] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 89 ms (attempt 4/5)6406worker2 # [ 137.131631] nix-grpc-daemon[1528]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6407worker2 # [ 137.135669] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6408worker2 # [ 137.153172] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421286409client # [ 137.179225] nix[1335]: copying 1 paths to 'lb:50051'...6410client # [ 137.183143] nix[1335]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6411worker2 # [ 137.167765] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421286412worker2 # [ 137.176160] nix-daemon[931]: accepted connection from pid 1528, user nix-grpc-daemon (trusted)6413worker2 # [ 137.196805] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:19Z level=info event=niks3_push_started pid=16566414worker2 # [ 137.358582] nix-grpc-daemon[1656]: time=2026-09-22T10:46:19.325Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6415worker2 # [ 137.501569] nix-daemon[931]: accepted connection from pid 1669, user nix-grpc-daemon (trusted)6416worker2 # [ 137.512195] nix-daemon[931]: reaped child process 1672, status = succeeded6417worker2 # [ 137.682565] nix-daemon[931]: accepted connection from pid 1674, user nix-grpc-daemon (trusted)6418worker2 # [ 137.694666] nix-daemon[931]: reaped child process 1677, status = succeeded6419lb # [ 137.727694] niks3-server[1215]: 2026/09/22 10:46:19 INFO Received uploads request method=POST path=/api/pending_closures6420worker2 # [ 137.721678] nix-grpc-daemon[1656]: time=2026-09-22T10:46:19.689Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6421worker2 # [ 137.723845] nix-grpc-daemon[1656]: time=2026-09-22T10:46:19.690Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6422lb # [ 137.765642] niks3-server[1215]: 2026/09/22 10:46:19 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6423lb # [ 137.770875] niks3-server[1215]: 2026/09/22 10:46:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6424lb # [ 137.775745] niks3-server[1215]: 2026/09/22 10:46:19 INFO Signed narinfos id=45 count=16425worker2 # [ 137.763093] nix-grpc-daemon[1656]: time=2026-09-22T10:46:19.730Z level=INFO msg="Uploading 1 narinfos"6426lb # [ 137.785577] niks3-server[1215]: 2026/09/22 10:46:19 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6427lb # [ 137.793978] niks3-server[1215]: 2026/09/22 10:46:19 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6428worker2 # [ 137.785141] nix-grpc-daemon[1656]: time=2026-09-22T10:46:19.752Z level=INFO msg="Upload complete. (423ms)"6429lb # [ 137.801121] niks3-server[1215]: 2026/09/22 10:46:19 INFO Completed upload id=456430worker2 # [ 137.786705] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42128 duration_s=0 paths=1 nar_bytes_in=4246431lb # [ 137.805948] niks3-server[1215]: 2026/09/22 10:46:19 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6432worker2 # [ 137.793309] nix-daemon[931]: reaped child process 1654, status = succeeded6433worker2 # [ 137.795331] nix-grpc-daemon[1528]: ts=2026-09-22T10:46:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421286434worker2 # [ 137.798814] nix-daemon[931]: accepted connection from pid 1528, user nix-grpc-daemon (trusted)6435worker2 # [ 137.805175] nix-daemon[931]: accepted connection from pid 1528, user nix-grpc-daemon (trusted)6436client # [ 137.820789] nix[1335]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6437lb # [ 138.397968] envoy[987]: conn peer=[2001:db8:1::1]:36244 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2960 tx=1425 ms=82136438lb # [ 138.401915] envoy[987]: conn peer=[2001:db8:1::3]:60362 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=4979 tx=1185 ms=304076439lb: must succeed: curl -sf localhost:9901/clusters6440lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6441lb: must succeed: curl -sf localhost:9901/clusters6442lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6443worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6444worker2 # [ 140.269324] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1528 (nix-grpc-daemon) on client request.6445lb # [ 140.287738] niks3-server[1215]: 2026/09/22 10:46:22 INFO lead: released remote=192.168.1.4:458966446worker2 # [ 140.272545] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6447worker1 # [ 140.281712] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:22Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6448client # [ 140.296160] nix[1335]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6449client # [ 140.300840] nix[1335]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6450worker2 # [ 140.289178] systemd[1]: Starting scheduler-token.service...6451worker2 # [ 140.303438] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6452worker2 # [ 140.306421] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6453worker2 # [ 140.309867] systemd[1]: nix-grpc-daemon.service: Consumed 436ms CPU time over 17.007s wall clock time, 33.7M memory peak, 15.3K incoming IP traffic, 28.4K outgoing IP traffic.6454worker2 # [ 140.314416] nix-daemon[931]: reaped child process 1683, status = succeeded6455worker2 # [ 140.315996] nix-daemon[931]: reaped child process 1648, status = succeeded6456worker2 # [ 140.325388] nix-daemon[1685]: unexpected Nix daemon error: error: interrupted by the user6457worker2 # [ 140.329380] nix-daemon[931]: reaped child process 1685, status = failed with exit code 16458worker2 # [ 140.372382] systemd[1]: scheduler-token.service: Deactivated successfully.6459worker2 # [ 140.373567] systemd[1]: Finished scheduler-token.service.6460worker2 # [ 140.375578] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 83ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6461worker2 # [ 140.377724] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6462worker2 # [ 140.382341] systemd[1]: Starting scheduler-token.service...6463worker2 # [ 140.446755] systemd[1]: scheduler-token.service: Deactivated successfully.6464worker2 # [ 140.449074] systemd[1]: Finished scheduler-token.service.6465worker2 # [ 140.449870] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 66ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6466worker2 # [ 140.454744] systemd[1]: Starting Nix worker-protocol over gRPC...6467worker1 # [ 140.482242] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:22Z level=info event=scheduler_connected addr=lb:500516468worker2 # [ 140.607357] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6469worker2 # [ 140.609800] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6470worker2 # [ 140.613429] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16471worker2 # [ 140.620064] nix-grpc-daemon[1728]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6472worker2 # [ 140.621381] nix-grpc-daemon[1728]: E0000 00:00:1790073982.586916 1728 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6473worker2 # [ 140.627059] nix-grpc-daemon[1728]: E0000 00:00:1790073982.591959 1728 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6474worker2 # [ 140.629141] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6475worker2 # [ 140.630644] systemd[1]: Started Nix worker-protocol over gRPC.6476worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.41 seconds)6477worker2 # [ 140.641674] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=info event=scheduler_take_over6478client: must succeed: sleep 5.06479lb # [ 140.657794] niks3-server[1215]: 2026/09/22 10:46:22 INFO lead: acquired remote=192.168.1.4:564286480worker2 # [ 140.644077] nix-grpc-daemon[1728]: E0000 00:00:1790073982.610941 1769 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6481worker2 # [ 140.648066] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:421386482worker2 # [ 140.651114] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06483worker2 # [ 140.662050] nix-grpc-daemon[1728]: E0000 00:00:1790073982.629349 1769 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6484worker2 # [ 140.666054] nix-grpc-daemon[1728]: E0000 00:00:1790073982.631586 1769 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6485worker2 # [ 140.675927] nix-grpc-daemon[1728]: E0000 00:00:1790073982.643239 1769 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6486worker2 # [ 140.685236] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=info event=scheduler_connected addr=lb:500516487worker2 # [ 140.691316] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=info event=oidc_jwks provider=mock keys=16488worker2 # [ 140.692607] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:421386489worker2 # [ 140.694149] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06490worker2 # [ 140.792834] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06491worker2 # [ 140.797210] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:22Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16492worker1 # [ 140.840511] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:380106493worker1 # [ 140.847120] nix-grpc-daemon[1577]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 47 ms (attempt 1/5)6494worker1 # [ 140.896457] nix-grpc-daemon[1577]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 102 ms (attempt 2/5)6495worker1 # [ 141.001179] nix-grpc-daemon[1577]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 290 ms (attempt 3/5)6496worker1 # [ 141.292569] nix-grpc-daemon[1577]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 217 ms (attempt 4/5)6497worker1 # [ 141.509887] nix-grpc-daemon[1577]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6498worker1 # [ 141.513898] nix-grpc-daemon[1577]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6499worker1 # [ 141.530382] nix-grpc-daemon[1577]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' from 'http://lb:5751'...6500worker1 # [ 141.555265] nix-daemon[911]: accepted connection from pid 1577, user nix-grpc-daemon (trusted)6501client # [ 141.582747] nix[1335]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6502worker1 # [ 141.570416] nix-daemon[911]: accepted connection from pid 1577, user nix-grpc-daemon (trusted)6503client: (finished: must succeed: sleep 5.0, in 5.04 seconds)6504client: waiting for success: ! systemctl is-active mid6505worker1 # [ 145.992929] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:27Z level=info event=niks3_push_started pid=16836506worker1 # [ 146.006075] nix-daemon[911]: reaped child process 1658, status = succeeded6507worker1 # [ 146.181128] nix-grpc-daemon[1683]: time=2026-09-22T10:46:28.181Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6508worker1 # [ 146.341481] nix-daemon[911]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)6509worker1 # [ 146.353651] nix-daemon[911]: reaped child process 1699, status = succeeded6510lb # [ 146.383809] niks3-server[1215]: 2026/09/22 10:46:28 INFO Received uploads request method=POST path=/api/pending_closures6511worker1 # [ 146.376429] nix-grpc-daemon[1683]: time=2026-09-22T10:46:28.377Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6512worker1 # [ 146.377925] nix-grpc-daemon[1683]: time=2026-09-22T10:46:28.379Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6513lb # [ 146.415583] niks3-server[1215]: 2026/09/22 10:46:28 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6514lb # [ 146.424601] niks3-server[1215]: 2026/09/22 10:46:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6515lb # [ 146.427173] niks3-server[1215]: 2026/09/22 10:46:28 INFO Signed narinfos id=46 count=16516worker1 # [ 146.415445] nix-grpc-daemon[1683]: time=2026-09-22T10:46:28.416Z level=INFO msg="Uploading 1 narinfos"6517lb # [ 146.442894] niks3-server[1215]: 2026/09/22 10:46:28 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6518lb # [ 146.448596] niks3-server[1215]: 2026/09/22 10:46:28 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6519lb # [ 146.450640] niks3-server[1215]: 2026/09/22 10:46:28 INFO Completed upload id=466520worker1 # [ 146.437165] nix-grpc-daemon[1683]: time=2026-09-22T10:46:28.438Z level=INFO msg="Upload complete. (252ms)"6521lb # [ 146.452079] niks3-server[1215]: 2026/09/22 10:46:28 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6522worker1 # [ 146.439568] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38010 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16523worker1 # [ 146.446355] nix-daemon[911]: reaped child process 1656, status = succeeded6524client # [ 146.489354] systemd[1]: mid.service: Deactivated successfully.6525client # [ 146.492988] systemd[1]: mid.service: Consumed 144ms CPU time over 9.949s wall clock time, 43.4M memory peak, 8.8K incoming IP traffic, 10.4K outgoing IP traffic.6526client: (finished: waiting for success: ! systemctl is-active mid, in 1.08 seconds)6527client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6528client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.03 seconds)6529(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.56 seconds)6530subtest: interrupting the client stops the build on the worker6531client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6532client # [ 147.081453] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6533client # Running as unit: intr.service; invocation ID: 5f5568f5127a44d38ba37e02e5b4d46c6534client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.03 seconds)6535worker1 # [ 147.252309] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:29Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:380106536client # [ 147.271521] nix[1371]: don't know how to build these paths:6537client # [ 147.272971] nix[1371]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6538worker2 # [ 147.268154] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:29Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06539worker2 # [ 147.272269] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:29Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16540worker2 # [ 147.313150] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599986541worker2 # [ 147.324988] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6542worker2 # [ 147.342241] nix-grpc-daemon[1728]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 33 ms (attempt 1/5)6543worker2 # [ 147.381092] nix-grpc-daemon[1728]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 91 ms (attempt 2/5)6544worker2 # [ 147.477774] nix-grpc-daemon[1728]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 123 ms (attempt 3/5)6545worker2 # [ 147.606784] nix-grpc-daemon[1728]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 362 ms (attempt 4/5)6546worker2 # [ 147.975184] nix-grpc-daemon[1728]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6547worker2 # [ 147.978971] nix-grpc-daemon[1728]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6548worker2 # [ 147.995131] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599986549client # [ 148.020705] nix[1371]: copying 1 paths to 'lb:50051'...6550worker2 # [ 148.009262] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:599986551client # [ 148.025133] nix[1371]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6552worker2 # [ 148.018329] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6553worker2 # [ 148.039120] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:30Z level=info event=niks3_push_started pid=18046554worker2 # [ 148.205065] nix-grpc-daemon[1804]: time=2026-09-22T10:46:30.171Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6555worker2 # [ 148.353682] nix-daemon[931]: accepted connection from pid 1817, user nix-grpc-daemon (trusted)6556worker2 # [ 148.365390] nix-daemon[931]: reaped child process 1824, status = succeeded6557lb # [ 148.398918] envoy[987]: conn peer=[2001:db8:1::4]:52902 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=2871 tx=363 ms=167606558lb # [ 148.403155] envoy[987]: conn peer=[2001:db8:1::1]:36296 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3934 tx=1853 ms=97276559worker2 # [ 148.512253] nix-daemon[931]: accepted connection from pid 1826, user nix-grpc-daemon (trusted)6560worker2 # [ 148.526340] nix-daemon[931]: reaped child process 1829, status = succeeded6561lb # [ 148.553473] niks3-server[1215]: 2026/09/22 10:46:30 INFO Received uploads request method=POST path=/api/pending_closures6562worker2 # [ 148.551083] nix-grpc-daemon[1804]: time=2026-09-22T10:46:30.517Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6563worker2 # [ 148.552672] nix-grpc-daemon[1804]: time=2026-09-22T10:46:30.517Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6564lb # [ 148.600201] niks3-server[1215]: 2026/09/22 10:46:30 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6565lb # [ 148.607810] niks3-server[1215]: 2026/09/22 10:46:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6566worker2 # [ 148.597592] nix-grpc-daemon[1804]: time=2026-09-22T10:46:30.564Z level=INFO msg="Uploading 1 narinfos"6567lb # [ 148.613123] niks3-server[1215]: 2026/09/22 10:46:30 INFO Signed narinfos id=47 count=16568lb # [ 148.618867] niks3-server[1215]: 2026/09/22 10:46:30 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6569lb # [ 148.626866] niks3-server[1215]: 2026/09/22 10:46:30 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6570lb # [ 148.636113] niks3-server[1215]: 2026/09/22 10:46:30 INFO Completed upload id=476571worker2 # [ 148.620466] nix-grpc-daemon[1804]: time=2026-09-22T10:46:30.587Z level=INFO msg="Upload complete. (410ms)"6572lb # [ 148.638154] niks3-server[1215]: 2026/09/22 10:46:30 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6573worker2 # [ 148.623296] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=0 paths=1 nar_bytes_in=4246574worker2 # [ 148.626974] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599986575worker2 # [ 148.633241] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6576worker2 # [ 148.635711] nix-daemon[931]: reaped child process 1802, status = succeeded6577worker2 # [ 148.641057] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6578client # [ 148.656556] nix[1371]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6579client: must succeed: systemctl kill -s INT intr6580worker2 # [ 150.576837] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:32Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6581client # [ 150.588979] systemd[1]: intr.service: Sent signal SIGINT to main process 1371 (nix) on client request.6582client # [ 150.596523] nix[1371]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6583client: (finished: must succeed: systemctl kill -s INT intr, in 0.05 seconds)6584client # [ 150.603077] nix[1371]: error: interrupted by the user6585client # [ 150.638397] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6586client # [ 150.639839] systemd[1]: intr.service: Failed with result 'exit-code'.6587client # [ 150.641760] systemd[1]: intr.service: Consumed 117ms CPU time over 3.559s wall clock time, 41.4M memory peak, 6.5K incoming IP traffic, 8.3K outgoing IP traffic.6588worker2 # [ 150.662279] nix-daemon[1833]: unexpected Nix daemon error: error: interrupted by the user6589worker2 # [ 150.664364] nix-daemon[931]: reaped child process 1831, status = succeeded6590worker2 # [ 150.674180] nix-daemon[931]: reaped child process 1833, status = failed with exit code 16591(finished: subtest: interrupting the client stops the build on the worker, in 3.64 seconds)6592subtest: a deploy mid-build drains: the build finishes, the new generation takes over6593client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6594client # [ 150.736447] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6595client # Running as unit: sw.service; invocation ID: 7392148c079542c2ad1a84c4a2aa28c66596client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.05 seconds)6597client # [ 150.928309] nix[1392]: don't know how to build these paths:6598client # [ 150.929697] nix[1392]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6599worker1 # [ 150.914480] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:380106600worker2 # [ 150.922781] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:32Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06601worker2 # [ 150.927648] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:32Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16602worker1 # [ 150.974153] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:380106603worker1 # [ 150.986989] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:380106604client # [ 151.010161] nix[1392]: copying 1 paths to 'lb:50051'...6605worker1 # [ 150.997887] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:380106606client # [ 151.013142] nix[1392]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6607worker1 # [ 151.006786] nix-daemon[911]: accepted connection from pid 1577, user nix-grpc-daemon (trusted)6608worker1 # [ 151.185328] nix-daemon[911]: accepted connection from pid 1727, user nix-grpc-daemon (trusted)6609worker1 # [ 151.198271] nix-daemon[911]: reaped child process 1730, status = succeeded6610worker1 # [ 151.358795] nix-daemon[911]: accepted connection from pid 1732, user nix-grpc-daemon (trusted)6611worker1 # [ 151.371425] nix-daemon[911]: reaped child process 1735, status = succeeded6612lb # [ 151.401622] niks3-server[1215]: 2026/09/22 10:46:33 INFO Received uploads request method=POST path=/api/pending_closures6613worker1 # [ 151.399911] nix-grpc-daemon[1683]: time=2026-09-22T10:46:33.400Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6614worker1 # [ 151.401419] nix-grpc-daemon[1683]: time=2026-09-22T10:46:33.400Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6615lb # [ 151.432875] niks3-server[1215]: 2026/09/22 10:46:33 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6616lb # [ 151.442834] niks3-server[1215]: 2026/09/22 10:46:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6617lb # [ 151.448581] niks3-server[1215]: 2026/09/22 10:46:33 INFO Signed narinfos id=48 count=16618worker1 # [ 151.438450] nix-grpc-daemon[1683]: time=2026-09-22T10:46:33.439Z level=INFO msg="Uploading 1 narinfos"6619lb # [ 151.455520] niks3-server[1215]: 2026/09/22 10:46:33 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6620lb # [ 151.466710] niks3-server[1215]: 2026/09/22 10:46:33 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6621lb # [ 151.471577] niks3-server[1215]: 2026/09/22 10:46:33 INFO Completed upload id=486622worker1 # [ 151.460433] nix-grpc-daemon[1683]: time=2026-09-22T10:46:33.461Z level=INFO msg="Upload complete. (440ms)"6623worker1 # [ 151.462758] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38010 duration_s=0 paths=1 nar_bytes_in=4166624lb # [ 151.475322] niks3-server[1215]: 2026/09/22 10:46:33 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6625worker1 # [ 151.469456] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:380106626worker1 # [ 151.471247] nix-daemon[911]: reaped child process 1725, status = succeeded6627worker1 # [ 151.474320] nix-daemon[911]: accepted connection from pid 1577, user nix-grpc-daemon (trusted)6628worker1 # [ 151.481078] nix-daemon[911]: accepted connection from pid 1577, user nix-grpc-daemon (trusted)6629client # [ 151.493708] nix[1392]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6630worker1: must succeed: systemctl reload nix-grpc-daemon.service6631worker1 # [ 153.216730] systemd[1]: Reloading Nix worker-protocol over gRPC...6632worker1 # [ 153.252435] systemd[1]: Reloaded Nix worker-protocol over gRPC.6633worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.08 seconds)6634worker1: must succeed: pgrep -f 'read -t [9]0 x'6635worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.04 seconds)6636client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26637client # don't know how to build these paths:6638client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6639worker2 # [ 153.514957] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:599986640worker1 # [ 153.523259] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:380106641worker2 # [ 153.539880] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:35Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06642worker2 # [ 153.544925] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:35Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16643worker2 # [ 153.584224] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599986644worker2 # [ 153.600769] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599986645client # copying 1 paths to 'lb:50051'...6646worker2 # [ 153.613621] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:599986647client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6648worker2 # [ 153.635291] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6649worker2 # [ 153.805310] nix-daemon[931]: accepted connection from pid 1884, user nix-grpc-daemon (trusted)6650worker2 # [ 153.818075] nix-daemon[931]: reaped child process 1887, status = succeeded6651worker1 # [ 153.877159] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:35Z level=info event=draining6652worker2 # [ 153.976531] nix-daemon[931]: accepted connection from pid 1889, user nix-grpc-daemon (trusted)6653worker2 # [ 153.988701] nix-daemon[931]: reaped child process 1892, status = succeeded6654lb # [ 154.020683] niks3-server[1215]: 2026/09/22 10:46:35 INFO Received uploads request method=POST path=/api/pending_closures6655worker2 # [ 154.015127] nix-grpc-daemon[1804]: time=2026-09-22T10:46:35.982Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6656worker2 # [ 154.016614] nix-grpc-daemon[1804]: time=2026-09-22T10:46:35.983Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6657lb # [ 154.054801] niks3-server[1215]: 2026/09/22 10:46:35 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6658lb # [ 154.068114] niks3-server[1215]: 2026/09/22 10:46:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6659worker2 # [ 154.060022] nix-grpc-daemon[1804]: time=2026-09-22T10:46:36.027Z level=INFO msg="Uploading 1 narinfos"6660lb # [ 154.076121] niks3-server[1215]: 2026/09/22 10:46:35 INFO Signed narinfos id=49 count=16661lb # [ 154.083086] niks3-server[1215]: 2026/09/22 10:46:35 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6662lb # [ 154.095177] niks3-server[1215]: 2026/09/22 10:46:35 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6663lb # [ 154.102443] niks3-server[1215]: 2026/09/22 10:46:35 INFO Completed upload id=496664worker2 # [ 154.086876] nix-grpc-daemon[1804]: time=2026-09-22T10:46:36.054Z level=INFO msg="Upload complete. (439ms)"6665lb # [ 154.104589] niks3-server[1215]: 2026/09/22 10:46:35 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6666worker2 # [ 154.090784] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=0 paths=1 nar_bytes_in=5286667worker2 # [ 154.096244] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599986668worker2 # [ 154.107134] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6669worker2 # [ 154.116350] nix-daemon[931]: reaped child process 1882, status = succeeded6670worker2 # [ 154.122148] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6671client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6672client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6673client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6674client # farm-a> LOG-farm-a-during-drain6675worker2 # [ 155.147149] nix-daemon[931]: reaped child process 1896, status = succeeded6676worker2 # [ 155.314783] nix-daemon[931]: accepted connection from pid 1912, user nix-grpc-daemon (trusted)6677worker2 # [ 155.325702] nix-daemon[931]: reaped child process 1915, status = succeeded6678lb # [ 155.357177] niks3-server[1215]: 2026/09/22 10:46:37 INFO Received uploads request method=POST path=/api/pending_closures6679worker2 # [ 155.351129] nix-grpc-daemon[1804]: time=2026-09-22T10:46:37.318Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6680worker2 # [ 155.352631] nix-grpc-daemon[1804]: time=2026-09-22T10:46:37.319Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6681lb # [ 155.406920] niks3-server[1215]: 2026/09/22 10:46:37 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6682lb # [ 155.417064] niks3-server[1215]: 2026/09/22 10:46:37 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6683lb # [ 155.420503] niks3-server[1215]: 2026/09/22 10:46:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6684lb # [ 155.423845] niks3-server[1215]: 2026/09/22 10:46:37 INFO Signed narinfos id=50 count=16685worker2 # [ 155.411057] nix-grpc-daemon[1804]: time=2026-09-22T10:46:37.377Z level=INFO msg="Uploading 1 narinfos"6686lb # [ 155.430824] niks3-server[1215]: 2026/09/22 10:46:37 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6687lb # [ 155.439266] niks3-server[1215]: 2026/09/22 10:46:37 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6688lb # [ 155.442201] niks3-server[1215]: 2026/09/22 10:46:37 INFO Completed upload id=506689lb # [ 155.444309] niks3-server[1215]: 2026/09/22 10:46:37 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6690worker2 # [ 155.426379] nix-grpc-daemon[1804]: time=2026-09-22T10:46:37.393Z level=INFO msg="Upload complete. (296ms)"6691worker2 # [ 155.433940] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16692worker2 # [ 155.443756] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:37Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06693worker2 # [ 155.447137] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:37Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16694worker2 # [ 155.455269] nix-daemon[931]: reaped child process 1894, status = succeeded6695worker2 # [ 155.487525] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599986696worker2 # [ 155.500070] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599986697worker2 # [ 155.513517] nix-grpc-daemon[1804]: time=2026-09-22T10:46:37.480Z level=INFO msg="All 1 paths already cached"6698client # copying 1 paths to 'lb:50051'...6699worker2 # [ 155.521760] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:599986700client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6701worker2 # [ 155.539386] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6702worker2 # [ 155.716241] nix-daemon[931]: accepted connection from pid 1921, user nix-grpc-daemon (trusted)6703worker2 # [ 155.730499] nix-daemon[931]: reaped child process 1924, status = succeeded6704worker2 # [ 155.887645] nix-daemon[931]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)6705worker2 # [ 155.899863] nix-daemon[931]: reaped child process 1929, status = succeeded6706lb # [ 155.928298] niks3-server[1215]: 2026/09/22 10:46:37 INFO Received uploads request method=POST path=/api/pending_closures6707worker2 # [ 155.921392] nix-grpc-daemon[1804]: time=2026-09-22T10:46:37.888Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6708worker2 # [ 155.922749] nix-grpc-daemon[1804]: time=2026-09-22T10:46:37.888Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6709lb # [ 155.953237] niks3-server[1215]: 2026/09/22 10:46:37 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6710lb # [ 155.965527] niks3-server[1215]: 2026/09/22 10:46:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6711lb # [ 155.972528] niks3-server[1215]: 2026/09/22 10:46:37 INFO Signed narinfos id=51 count=16712worker2 # [ 155.960125] nix-grpc-daemon[1804]: time=2026-09-22T10:46:37.927Z level=INFO msg="Uploading 1 narinfos"6713lb # [ 155.976299] niks3-server[1215]: 2026/09/22 10:46:37 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6714lb # [ 155.990229] niks3-server[1215]: 2026/09/22 10:46:37 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6715lb # [ 155.996619] niks3-server[1215]: 2026/09/22 10:46:37 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6716lb # [ 156.001530] niks3-server[1215]: 2026/09/22 10:46:37 INFO Completed upload id=516717worker2 # [ 155.985792] nix-grpc-daemon[1804]: time=2026-09-22T10:46:37.953Z level=INFO msg="Upload complete. (432ms)"6718worker2 # [ 155.988385] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=0 paths=1 nar_bytes_in=6406719worker2 # [ 155.991293] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599986720worker2 # [ 155.999065] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6721worker2 # [ 156.003116] nix-daemon[931]: reaped child process 1919, status = succeeded6722worker2 # [ 156.009147] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6723client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6724client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6725client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6726client # farm-b> LOG-farm-b-during-drain6727worker2 # [ 156.835538] nix-daemon[931]: reaped child process 1933, status = succeeded6728worker2 # [ 156.986921] nix-daemon[931]: accepted connection from pid 1949, user nix-grpc-daemon (trusted)6729worker2 # [ 156.998541] nix-daemon[931]: reaped child process 1952, status = succeeded6730lb # [ 157.026171] niks3-server[1215]: 2026/09/22 10:46:38 INFO Received uploads request method=POST path=/api/pending_closures6731worker2 # [ 157.018552] nix-grpc-daemon[1804]: time=2026-09-22T10:46:38.985Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6732worker2 # [ 157.020048] nix-grpc-daemon[1804]: time=2026-09-22T10:46:38.987Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6733lb # [ 157.064649] niks3-server[1215]: 2026/09/22 10:46:38 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6734lb # [ 157.071098] niks3-server[1215]: 2026/09/22 10:46:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6735worker2 # [ 157.061864] nix-grpc-daemon[1804]: time=2026-09-22T10:46:39.029Z level=INFO msg="Uploading 1 narinfos"6736lb # [ 157.079160] niks3-server[1215]: 2026/09/22 10:46:38 INFO Signed narinfos id=52 count=16737lb # [ 157.083676] niks3-server[1215]: 2026/09/22 10:46:38 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6738lb # [ 157.092610] niks3-server[1215]: 2026/09/22 10:46:38 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6739lb # [ 157.098160] niks3-server[1215]: 2026/09/22 10:46:38 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6740lb # [ 157.104728] niks3-server[1215]: 2026/09/22 10:46:38 INFO Completed upload id=526741worker2 # [ 157.089175] nix-grpc-daemon[1804]: time=2026-09-22T10:46:39.056Z level=INFO msg="Upload complete. (272ms)"6742lb # [ 157.107847] niks3-server[1215]: 2026/09/22 10:46:38 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6743worker2 # [ 157.093598] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16744worker2 # [ 157.103952] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:39Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06745worker2 # [ 157.107525] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:39Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16746worker2 # [ 157.116319] nix-daemon[931]: reaped child process 1931, status = succeeded6747worker2 # [ 157.147914] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599986748worker2 # [ 157.160510] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599986749worker2 # [ 157.172365] nix-grpc-daemon[1804]: time=2026-09-22T10:46:39.139Z level=INFO msg="All 2 paths already cached"6750client # copying 1 paths to 'lb:50051'...6751worker2 # [ 157.181341] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:599986752client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6753worker2 # [ 157.199233] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6754worker2 # [ 157.348721] nix-daemon[931]: accepted connection from pid 1958, user nix-grpc-daemon (trusted)6755worker2 # [ 157.361927] nix-daemon[931]: reaped child process 1961, status = succeeded6756worker2 # [ 157.493345] nix-daemon[931]: accepted connection from pid 1963, user nix-grpc-daemon (trusted)6757worker2 # [ 157.505239] nix-daemon[931]: reaped child process 1966, status = succeeded6758lb # [ 157.531888] niks3-server[1215]: 2026/09/22 10:46:39 INFO Received uploads request method=POST path=/api/pending_closures6759worker2 # [ 157.525234] nix-grpc-daemon[1804]: time=2026-09-22T10:46:39.492Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6760worker2 # [ 157.526666] nix-grpc-daemon[1804]: time=2026-09-22T10:46:39.492Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6761lb # [ 157.557562] niks3-server[1215]: 2026/09/22 10:46:39 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6762lb # [ 157.568801] niks3-server[1215]: 2026/09/22 10:46:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6763worker2 # [ 157.557782] nix-grpc-daemon[1804]: time=2026-09-22T10:46:39.524Z level=INFO msg="Uploading 1 narinfos"6764lb # [ 157.574083] niks3-server[1215]: 2026/09/22 10:46:39 INFO Signed narinfos id=53 count=16765lb # [ 157.580312] niks3-server[1215]: 2026/09/22 10:46:39 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6766lb # [ 157.590110] niks3-server[1215]: 2026/09/22 10:46:39 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6767worker2 # [ 157.581037] nix-grpc-daemon[1804]: time=2026-09-22T10:46:39.548Z level=INFO msg="Upload complete. (368ms)"6768lb # [ 157.597163] niks3-server[1215]: 2026/09/22 10:46:39 INFO Completed upload id=536769worker2 # [ 157.582926] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=0 paths=1 nar_bytes_in=7686770lb # [ 157.600938] niks3-server[1215]: 2026/09/22 10:46:39 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6771worker2 # [ 157.589435] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599986772worker2 # [ 157.594317] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6773worker2 # [ 157.596585] nix-daemon[931]: reaped child process 1956, status = succeeded6774worker2 # [ 157.601921] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)6775client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6776client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6777client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6778lb # [ 158.399420] envoy[987]: conn peer=[2001:db8:1::1]:44990 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2980 tx=851 ms=33506779client # farm-top> LOG-farm-top-during-drain6780worker2 # [ 158.415447] nix-daemon[931]: reaped child process 1970, status = succeeded6781worker2 # [ 158.555355] nix-daemon[931]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6782worker2 # [ 158.567523] nix-daemon[931]: reaped child process 1989, status = succeeded6783lb # [ 158.597872] niks3-server[1215]: 2026/09/22 10:46:40 INFO Received uploads request method=POST path=/api/pending_closures6784worker2 # [ 158.590173] nix-grpc-daemon[1804]: time=2026-09-22T10:46:40.557Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6785worker2 # [ 158.591787] nix-grpc-daemon[1804]: time=2026-09-22T10:46:40.559Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6786lb # [ 158.629636] niks3-server[1215]: 2026/09/22 10:46:40 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6787lb # [ 158.643806] niks3-server[1215]: 2026/09/22 10:46:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6788lb # [ 158.650110] niks3-server[1215]: 2026/09/22 10:46:40 INFO Signed narinfos id=54 count=16789worker2 # [ 158.637359] nix-grpc-daemon[1804]: time=2026-09-22T10:46:40.604Z level=INFO msg="Uploading 1 narinfos"6790lb # [ 158.660162] niks3-server[1215]: 2026/09/22 10:46:40 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6791lb # [ 158.663430] niks3-server[1215]: 2026/09/22 10:46:40 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6792lb # [ 158.672107] niks3-server[1215]: 2026/09/22 10:46:40 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6793lb # [ 158.678616] niks3-server[1215]: 2026/09/22 10:46:40 INFO Completed upload id=546794worker2 # [ 158.662846] nix-grpc-daemon[1804]: time=2026-09-22T10:46:40.630Z level=INFO msg="Upload complete. (254ms)"6795lb # [ 158.680930] niks3-server[1215]: 2026/09/22 10:46:40 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6796worker2 # [ 158.669633] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16797worker2 # [ 158.683664] nix-daemon[931]: reaped child process 1968, status = succeeded6798client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 5.40 seconds)6799client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6800client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.16 seconds)6801worker1: must succeed: pkill -f 'read -t [9]0 x'6802worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6803client: waiting for success: ! systemctl is-active sw6804worker1 # [ 159.102859] nix-daemon[911]: reaped child process 1739, status = succeeded6805worker1 # [ 159.245886] nix-daemon[911]: accepted connection from pid 1781, user nix-grpc-daemon (trusted)6806worker1 # [ 159.256636] nix-daemon[911]: reaped child process 1784, status = succeeded6807lb # [ 159.286455] niks3-server[1215]: 2026/09/22 10:46:41 INFO Received uploads request method=POST path=/api/pending_closures6808worker1 # [ 159.281779] nix-grpc-daemon[1683]: time=2026-09-22T10:46:41.282Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6809worker1 # [ 159.283285] nix-grpc-daemon[1683]: time=2026-09-22T10:46:41.284Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6810lb # [ 159.317299] niks3-server[1215]: 2026/09/22 10:46:41 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6811lb # [ 159.330154] niks3-server[1215]: 2026/09/22 10:46:41 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6812lb # [ 159.341394] niks3-server[1215]: 2026/09/22 10:46:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6813worker1 # [ 159.333207] nix-grpc-daemon[1683]: time=2026-09-22T10:46:41.334Z level=INFO msg="Uploading 1 narinfos"6814lb # [ 159.346713] niks3-server[1215]: 2026/09/22 10:46:41 INFO Signed narinfos id=55 count=16815lb # [ 159.351606] niks3-server[1215]: 2026/09/22 10:46:41 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6816lb # [ 159.359603] niks3-server[1215]: 2026/09/22 10:46:41 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6817lb # [ 159.366571] niks3-server[1215]: 2026/09/22 10:46:41 INFO Completed upload id=556818worker1 # [ 159.353352] nix-grpc-daemon[1683]: time=2026-09-22T10:46:41.354Z level=INFO msg="Upload complete. (258ms)"6819lb # [ 159.368589] niks3-server[1215]: 2026/09/22 10:46:41 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6820worker1 # [ 159.359409] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38010 duration_s=7 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16821worker1 # [ 159.372538] nix-daemon[911]: reaped child process 1737, status = succeeded6822client # [ 159.406893] systemd[1]: sw.service: Deactivated successfully.6823client # [ 159.410741] systemd[1]: sw.service: Consumed 124ms CPU time over 8.672s wall clock time, 41.4M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.6824worker1 # [ 159.880313] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:41Z level=info event=signal_exit6825client: (finished: waiting for success: ! systemctl is-active sw, in 1.08 seconds)6826client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6827client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.03 seconds)6828worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6829worker1 # [ 161.107101] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:43Z level=info event=scheduler_restarting6830worker2 # [ 161.108992] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:43Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:421386831worker2 # [ 161.110608] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:43Z level=info event=worker_gone addr=192.168.1.3:500516832worker1 # [ 161.115309] nix-grpc-daemon[1577]: ts=2026-09-22T10:46:43Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6833worker1 # [ 161.124578] nix-daemon[911]: reaped child process 1633, status = succeeded6834worker1 # [ 161.709023] nix-grpc-daemon[1577]: E0000 00:00:1790074003.710195 1622 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true6835worker1 # [ 161.711098] nix-grpc-daemon[1577]: E0000 00:00:1790074003.710333 1622 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6836worker1 # [ 161.905719] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6837worker1 # [ 161.907401] systemd[1]: nix-grpc-daemon.service: Consumed 620ms CPU time over 26.212s wall clock time, 32.6M memory peak, 20.8K incoming IP traffic, 27.5K outgoing IP traffic.6838worker1 # [ 161.915512] systemd[1]: Starting Nix worker-protocol over gRPC...6839worker1 # [ 162.074359] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:44Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6840worker1 # [ 162.076936] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:44Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6841worker1 # [ 162.080658] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:44Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16842worker1 # [ 162.087064] nix-grpc-daemon[1798]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6843worker1 # [ 162.088444] nix-grpc-daemon[1798]: E0000 00:00:1790074004.088073 1798 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6844worker1 # [ 162.093920] nix-grpc-daemon[1798]: E0000 00:00:1790074004.093330 1798 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6845worker1 # [ 162.098315] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:44Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6846worker1 # [ 162.100218] systemd[1]: Started Nix worker-protocol over gRPC.6847worker1 # [ 162.105675] nix-grpc-daemon[1798]: E0000 00:00:1790074004.107042 1840 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6848worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.09 seconds)6849lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6850lb: must succeed: curl -sf localhost:9901/clusters6851worker1 # [ 162.118390] nix-grpc-daemon[1798]: E0000 00:00:1790074004.119764 1840 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6852worker1 # [ 162.122053] nix-grpc-daemon[1798]: E0000 00:00:1790074004.122034 1840 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6853worker1 # [ 162.132502] nix-grpc-daemon[1798]: E0000 00:00:1790074004.133835 1840 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6854worker1 # [ 162.143825] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:44Z level=info event=scheduler_connected addr=lb:500516855worker2 # [ 162.145567] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:44Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:421386856worker2 # [ 162.150216] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:44Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06857lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6858lb: must succeed: curl -sf localhost:9901/clusters6859lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6860lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.15 seconds)6861lb: must succeed: curl -sf localhost:9901/clusters6862lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6863worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6864worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6865(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 12.69 seconds)6866subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6867client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16868worker2 # [ 163.593198] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:599986869worker2 # [ 163.600548] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599986870worker2 # [ 163.641364] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:45Z level=debug event=want client=5 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06871worker2 # [ 163.643456] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:45Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16872worker2 # [ 163.645296] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:45Z level=debug event=want client=5 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06873worker2 # [ 163.646803] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:45Z level=debug event=want client=5 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06874worker1 # [ 163.694866] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572766875worker1 # [ 163.705539] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)6876worker1 # [ 163.725196] nix-grpc-daemon[1798]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 84 ms (attempt 1/5)6877worker1 # [ 163.815198] nix-grpc-daemon[1798]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 99 ms (attempt 2/5)6878worker1 # [ 163.920303] nix-grpc-daemon[1798]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 381 ms (attempt 3/5)6879worker1 # [ 164.301516] nix-grpc-daemon[1798]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 781 ms (attempt 4/5)6880worker1 # [ 165.088110] nix-grpc-daemon[1798]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6881worker1 # [ 165.091468] nix-grpc-daemon[1798]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6882worker1 # [ 165.106461] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572766883worker1 # [ 165.116731] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572766884worker1 # [ 165.128312] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)6885worker1 # [ 165.147265] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:47Z level=info event=niks3_push_started pid=18586886worker1 # [ 165.341274] nix-grpc-daemon[1858]: time=2026-09-22T10:46:47.341Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6887worker1 # [ 165.491568] nix-daemon[911]: accepted connection from pid 1871, user nix-grpc-daemon (trusted)6888worker1 # [ 165.504228] nix-daemon[911]: reaped child process 1874, status = succeeded6889worker1 # [ 165.660688] nix-daemon[911]: accepted connection from pid 1876, user nix-grpc-daemon (trusted)6890worker1 # [ 165.672962] nix-daemon[911]: reaped child process 1879, status = succeeded6891lb # [ 165.703260] niks3-server[1215]: 2026/09/22 10:46:47 INFO Received uploads request method=POST path=/api/pending_closures6892worker1 # [ 165.698640] nix-grpc-daemon[1858]: time=2026-09-22T10:46:47.700Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6893worker1 # [ 165.701621] nix-grpc-daemon[1858]: time=2026-09-22T10:46:47.701Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6894lb # [ 165.734925] niks3-server[1215]: 2026/09/22 10:46:47 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6895lb # [ 165.747612] niks3-server[1215]: 2026/09/22 10:46:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6896lb # [ 165.755584] niks3-server[1215]: 2026/09/22 10:46:47 INFO Signed narinfos id=56 count=16897worker1 # [ 165.745265] nix-grpc-daemon[1858]: time=2026-09-22T10:46:47.746Z level=INFO msg="Uploading 1 narinfos"6898lb # [ 165.766904] niks3-server[1215]: 2026/09/22 10:46:47 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6899lb # [ 165.774086] niks3-server[1215]: 2026/09/22 10:46:47 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6900lb # [ 165.779628] niks3-server[1215]: 2026/09/22 10:46:47 INFO Completed upload id=566901worker1 # [ 165.766413] nix-grpc-daemon[1858]: time=2026-09-22T10:46:47.767Z level=INFO msg="Upload complete. (421ms)"6902lb # [ 165.783030] niks3-server[1215]: 2026/09/22 10:46:47 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6903worker1 # [ 165.770276] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=0 paths=1 nar_bytes_in=5046904worker1 # [ 165.776300] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572766905worker1 # [ 165.784139] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)6906worker1 # [ 165.789249] nix-daemon[911]: reaped child process 1856, status = succeeded6907worker1 # [ 165.795066] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)6908worker1 # [ 166.818209] nix-daemon[911]: reaped child process 1883, status = succeeded6909worker1 # [ 166.989070] nix-daemon[911]: accepted connection from pid 1899, user nix-grpc-daemon (trusted)6910worker1 # [ 167.001627] nix-daemon[911]: reaped child process 1902, status = succeeded6911lb # [ 167.025718] niks3-server[1215]: 2026/09/22 10:46:48 INFO Received uploads request method=POST path=/api/pending_closures6912worker1 # [ 167.019984] nix-grpc-daemon[1858]: time=2026-09-22T10:46:49.021Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6913worker1 # [ 167.021487] nix-grpc-daemon[1858]: time=2026-09-22T10:46:49.022Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6914lb # [ 167.054625] niks3-server[1215]: 2026/09/22 10:46:48 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6915lb # [ 167.070091] niks3-server[1215]: 2026/09/22 10:46:48 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6916lb # [ 167.079415] niks3-server[1215]: 2026/09/22 10:46:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6917lb # [ 167.083788] niks3-server[1215]: 2026/09/22 10:46:48 INFO Signed narinfos id=57 count=16918worker1 # [ 167.071617] nix-grpc-daemon[1858]: time=2026-09-22T10:46:49.072Z level=INFO msg="Uploading 1 narinfos"6919lb # [ 167.089328] niks3-server[1215]: 2026/09/22 10:46:48 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6920lb # [ 167.094599] niks3-server[1215]: 2026/09/22 10:46:48 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6921lb # [ 167.097371] niks3-server[1215]: 2026/09/22 10:46:48 INFO Completed upload id=576922worker1 # [ 167.084079] nix-grpc-daemon[1858]: time=2026-09-22T10:46:49.085Z level=INFO msg="Upload complete. (283ms)"6923lb # [ 167.099604] niks3-server[1215]: 2026/09/22 10:46:48 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6924worker1 # [ 167.088510] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=7 outputs=16925worker2 # [ 167.088187] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:49Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16926worker1 # [ 167.100684] nix-daemon[911]: reaped child process 1881, status = succeeded6927worker2 # [ 167.097678] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:49Z level=debug event=want client=5 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06928worker1 # [ 167.141333] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572766929worker1 # [ 167.153690] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572766930worker1 # [ 167.164698] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572766931worker1 # [ 167.172533] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)6932worker1 # [ 167.321490] nix-daemon[911]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6933worker1 # [ 167.333627] nix-daemon[911]: reaped child process 1909, status = succeeded6934worker1 # [ 167.462561] nix-daemon[911]: accepted connection from pid 1911, user nix-grpc-daemon (trusted)6935worker1 # [ 167.474239] nix-daemon[911]: reaped child process 1914, status = succeeded6936lb # [ 167.498892] niks3-server[1215]: 2026/09/22 10:46:49 INFO Received uploads request method=POST path=/api/pending_closures6937worker1 # [ 167.493178] nix-grpc-daemon[1858]: time=2026-09-22T10:46:49.494Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6938worker1 # [ 167.494678] nix-grpc-daemon[1858]: time=2026-09-22T10:46:49.496Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6939lb # [ 167.528322] niks3-server[1215]: 2026/09/22 10:46:49 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6940lb # [ 167.537935] niks3-server[1215]: 2026/09/22 10:46:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6941lb # [ 167.544556] niks3-server[1215]: 2026/09/22 10:46:49 INFO Signed narinfos id=58 count=16942worker1 # [ 167.534389] nix-grpc-daemon[1858]: time=2026-09-22T10:46:49.535Z level=INFO msg="Uploading 1 narinfos"6943lb # [ 167.549291] niks3-server[1215]: 2026/09/22 10:46:49 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6944lb # [ 167.561525] niks3-server[1215]: 2026/09/22 10:46:49 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6945lb # [ 167.567303] niks3-server[1215]: 2026/09/22 10:46:49 INFO Completed upload id=586946worker1 # [ 167.554062] nix-grpc-daemon[1858]: time=2026-09-22T10:46:49.555Z level=INFO msg="Upload complete. (368ms)"6947lb # [ 167.569638] niks3-server[1215]: 2026/09/22 10:46:49 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6948worker1 # [ 167.557076] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=0 paths=1 nar_bytes_in=5046949worker1 # [ 167.562714] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572766950worker1 # [ 167.567343] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)6951worker1 # [ 167.569548] nix-daemon[911]: reaped child process 1904, status = succeeded6952worker1 # [ 167.574734] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)6953lb # [ 168.399833] envoy[987]: conn peer=[2001:db8:1::1]:57720 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9555 tx=4046 ms=51676954lb # [ 168.403621] envoy[987]: conn peer=[2001:db8:1::1]:44994 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2975 tx=1401 ms=84556955lb # [ 168.407217] envoy[987]: conn peer=[2001:db8:1::3]:42844 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1114 tx=448 ms=251996956worker1 # [ 168.587298] nix-daemon[911]: reaped child process 1918, status = succeeded6957worker1 # [ 168.708448] nix-daemon[911]: accepted connection from pid 1934, user nix-grpc-daemon (trusted)6958worker1 # [ 168.719630] nix-daemon[911]: reaped child process 1937, status = succeeded6959lb # [ 168.744732] niks3-server[1215]: 2026/09/22 10:46:50 INFO Received uploads request method=POST path=/api/pending_closures6960worker1 # [ 168.739041] nix-grpc-daemon[1858]: time=2026-09-22T10:46:50.740Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6961worker1 # [ 168.740818] nix-grpc-daemon[1858]: time=2026-09-22T10:46:50.742Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6962lb # [ 168.784146] niks3-server[1215]: 2026/09/22 10:46:50 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6963lb # [ 168.791310] niks3-server[1215]: 2026/09/22 10:46:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6964lb # [ 168.796198] niks3-server[1215]: 2026/09/22 10:46:50 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6965lb # [ 168.803590] niks3-server[1215]: 2026/09/22 10:46:50 INFO Signed narinfos id=59 count=16966worker1 # [ 168.793194] nix-grpc-daemon[1858]: time=2026-09-22T10:46:50.794Z level=INFO msg="Uploading 1 narinfos"6967lb # [ 168.809052] niks3-server[1215]: 2026/09/22 10:46:50 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6968lb # [ 168.818617] niks3-server[1215]: 2026/09/22 10:46:50 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6969lb # [ 168.823845] niks3-server[1215]: 2026/09/22 10:46:50 INFO Completed upload id=596970worker1 # [ 168.812740] nix-grpc-daemon[1858]: time=2026-09-22T10:46:50.814Z level=INFO msg="Upload complete. (231ms)"6971lb # [ 168.827513] niks3-server[1215]: 2026/09/22 10:46:50 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6972worker1 # [ 168.819325] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=8 outputs=16973worker2 # [ 168.818459] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:50Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16974worker1 # [ 168.824770] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572766975worker2 # [ 168.824693] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:50Z level=debug event=want client=5 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06976worker1 # [ 168.834298] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572766977worker1 # [ 168.837257] nix-daemon[911]: reaped child process 1916, status = succeeded6978worker1 # [ 168.840888] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572766979worker1 # [ 168.844531] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)6980worker1 # [ 168.980493] nix-daemon[911]: accepted connection from pid 1941, user nix-grpc-daemon (trusted)6981worker1 # [ 168.992684] nix-daemon[911]: reaped child process 1944, status = succeeded6982worker1 # [ 169.119406] nix-daemon[911]: accepted connection from pid 1946, user nix-grpc-daemon (trusted)6983worker1 # [ 169.131080] nix-daemon[911]: reaped child process 1949, status = succeeded6984lb # [ 169.158808] niks3-server[1215]: 2026/09/22 10:46:50 INFO Received uploads request method=POST path=/api/pending_closures6985worker1 # [ 169.149307] nix-grpc-daemon[1858]: time=2026-09-22T10:46:51.150Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6986worker1 # [ 169.150795] nix-grpc-daemon[1858]: time=2026-09-22T10:46:51.152Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6987lb # [ 169.176927] niks3-server[1215]: 2026/09/22 10:46:51 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6988lb # [ 169.180141] niks3-server[1215]: 2026/09/22 10:46:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6989lb # [ 169.182323] niks3-server[1215]: 2026/09/22 10:46:51 INFO Signed narinfos id=60 count=16990worker1 # [ 169.168902] nix-grpc-daemon[1858]: time=2026-09-22T10:46:51.170Z level=INFO msg="Uploading 1 narinfos"6991lb # [ 169.187509] niks3-server[1215]: 2026/09/22 10:46:51 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6992lb # [ 169.195112] niks3-server[1215]: 2026/09/22 10:46:51 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6993lb # [ 169.199122] niks3-server[1215]: 2026/09/22 10:46:51 INFO Completed upload id=606994worker1 # [ 169.185727] nix-grpc-daemon[1858]: time=2026-09-22T10:46:51.187Z level=INFO msg="Upload complete. (335ms)"6995lb # [ 169.201644] niks3-server[1215]: 2026/09/22 10:46:51 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6996worker1 # [ 169.188183] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=0 paths=1 nar_bytes_in=5046997worker1 # [ 169.190490] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572766998worker1 # [ 169.194553] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)6999worker1 # [ 169.199566] nix-daemon[911]: reaped child process 1939, status = succeeded7000worker1 # [ 169.204219] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7001worker1 # [ 170.419220] nix-daemon[911]: reaped child process 1953, status = succeeded7002worker1 # [ 170.558412] nix-daemon[911]: accepted connection from pid 1969, user nix-grpc-daemon (trusted)7003worker1 # [ 170.570351] nix-daemon[911]: reaped child process 1972, status = succeeded7004lb # [ 170.599804] niks3-server[1215]: 2026/09/22 10:46:52 INFO Received uploads request method=POST path=/api/pending_closures7005worker1 # [ 170.595774] nix-grpc-daemon[1858]: time=2026-09-22T10:46:52.596Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7006worker1 # [ 170.597363] nix-grpc-daemon[1858]: time=2026-09-22T10:46:52.598Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"7007lb # [ 170.637132] niks3-server[1215]: 2026/09/22 10:46:52 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst7008lb # [ 170.645195] niks3-server[1215]: 2026/09/22 10:46:52 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv7009lb # [ 170.657118] niks3-server[1215]: 2026/09/22 10:46:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign7010worker1 # [ 170.649291] nix-grpc-daemon[1858]: time=2026-09-22T10:46:52.650Z level=INFO msg="Uploading 1 narinfos"7011lb # [ 170.662836] niks3-server[1215]: 2026/09/22 10:46:52 INFO Signed narinfos id=61 count=17012lb # [ 170.666585] niks3-server[1215]: 2026/09/22 10:46:52 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls7013lb # [ 170.675108] niks3-server[1215]: 2026/09/22 10:46:52 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete7014lb # [ 170.681746] niks3-server[1215]: 2026/09/22 10:46:52 INFO Completed upload id=617015worker1 # [ 170.668396] nix-grpc-daemon[1858]: time=2026-09-22T10:46:52.669Z level=INFO msg="Upload complete. (257ms)"7016lb # [ 170.685089] niks3-server[1215]: 2026/09/22 10:46:52 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo7017worker1 # [ 170.673816] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=9 outputs=17018worker2 # [ 170.675821] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:52Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=17019worker1 # [ 170.682923] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767020worker2 # [ 170.681947] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:52Z level=debug event=want client=5 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=07021worker1 # [ 170.692570] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572767022worker1 # [ 170.700816] nix-daemon[911]: reaped child process 1951, status = succeeded7023worker1 # [ 170.703510] nix-grpc-daemon[1858]: time=2026-09-22T10:46:52.704Z level=INFO msg="All 1 paths already cached"7024worker1 # [ 170.707346] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572767025worker1 # [ 170.711402] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7026worker1 # [ 170.865197] nix-daemon[911]: accepted connection from pid 1976, user nix-grpc-daemon (trusted)7027worker1 # [ 170.877376] nix-daemon[911]: reaped child process 1979, status = succeeded7028worker1 # [ 171.009711] nix-daemon[911]: accepted connection from pid 1981, user nix-grpc-daemon (trusted)7029worker1 # [ 171.022124] nix-daemon[911]: reaped child process 1984, status = succeeded7030lb # [ 171.049253] niks3-server[1215]: 2026/09/22 10:46:52 INFO Received uploads request method=POST path=/api/pending_closures7031worker1 # [ 171.043570] nix-grpc-daemon[1858]: time=2026-09-22T10:46:53.044Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7032worker1 # [ 171.044965] nix-grpc-daemon[1858]: time=2026-09-22T10:46:53.044Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"7033lb # [ 171.076829] niks3-server[1215]: 2026/09/22 10:46:52 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst7034lb # [ 171.084855] niks3-server[1215]: 2026/09/22 10:46:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign7035lb # [ 171.091609] niks3-server[1215]: 2026/09/22 10:46:52 INFO Signed narinfos id=62 count=17036worker1 # [ 171.081193] nix-grpc-daemon[1858]: time=2026-09-22T10:46:53.082Z level=INFO msg="Uploading 1 narinfos"7037lb # [ 171.095152] niks3-server[1215]: 2026/09/22 10:46:52 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls7038lb # [ 171.108158] niks3-server[1215]: 2026/09/22 10:46:52 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete7039worker1 # [ 171.099288] nix-grpc-daemon[1858]: time=2026-09-22T10:46:53.100Z level=INFO msg="Upload complete. (382ms)"7040lb # [ 171.112971] niks3-server[1215]: 2026/09/22 10:46:52 INFO Completed upload id=627041worker1 # [ 171.101117] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=0 paths=1 nar_bytes_in=6247042lb # [ 171.116311] niks3-server[1215]: 2026/09/22 10:46:52 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo7043worker1 # [ 171.107384] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767044worker1 # [ 171.111475] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7045worker1 # [ 171.113722] nix-daemon[911]: reaped child process 1974, status = succeeded7046worker1 # [ 171.118964] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7047worker1 # [ 171.931626] nix-daemon[911]: reaped child process 1988, status = succeeded7048worker1 # [ 172.065727] nix-daemon[911]: accepted connection from pid 2004, user nix-grpc-daemon (trusted)7049worker1 # [ 172.077812] nix-daemon[911]: reaped child process 2007, status = succeeded7050lb # [ 172.106387] niks3-server[1215]: 2026/09/22 10:46:53 INFO Received uploads request method=POST path=/api/pending_closures7051worker1 # [ 172.105549] nix-grpc-daemon[1858]: time=2026-09-22T10:46:54.106Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7052worker1 # [ 172.107164] nix-grpc-daemon[1858]: time=2026-09-22T10:46:54.108Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"7053lb # [ 172.151652] niks3-server[1215]: 2026/09/22 10:46:53 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst7054lb # [ 172.159515] niks3-server[1215]: 2026/09/22 10:46:53 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv7055lb # [ 172.165905] niks3-server[1215]: 2026/09/22 10:46:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign7056worker1 # [ 172.157392] nix-grpc-daemon[1858]: time=2026-09-22T10:46:54.158Z level=INFO msg="Uploading 1 narinfos"7057lb # [ 172.171194] niks3-server[1215]: 2026/09/22 10:46:54 INFO Signed narinfos id=63 count=17058lb # [ 172.176357] niks3-server[1215]: 2026/09/22 10:46:54 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls7059lb # [ 172.181061] niks3-server[1215]: 2026/09/22 10:46:54 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete7060lb # [ 172.183950] niks3-server[1215]: 2026/09/22 10:46:54 INFO Completed upload id=637061worker1 # [ 172.170628] nix-grpc-daemon[1858]: time=2026-09-22T10:46:54.171Z level=INFO msg="Upload complete. (245ms)"7062lb # [ 172.186400] niks3-server[1215]: 2026/09/22 10:46:54 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo7063worker1 # [ 172.176922] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=10 outputs=17064worker2 # [ 172.176238] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:54Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=17065worker1 # [ 172.182386] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767066worker2 # [ 172.183850] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:54Z level=debug event=want client=5 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=07067worker1 # [ 172.191093] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572767068worker1 # [ 172.195843] nix-daemon[911]: reaped child process 1986, status = succeeded7069worker1 # [ 172.197762] nix-grpc-daemon[1858]: time=2026-09-22T10:46:54.199Z level=INFO msg="All 1 paths already cached"7070worker1 # [ 172.201479] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572767071worker1 # [ 172.206345] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7072worker1 # [ 172.354469] nix-daemon[911]: accepted connection from pid 2011, user nix-grpc-daemon (trusted)7073worker1 # [ 172.367878] nix-daemon[911]: reaped child process 2014, status = succeeded7074worker1 # [ 172.493809] nix-daemon[911]: accepted connection from pid 2016, user nix-grpc-daemon (trusted)7075worker1 # [ 172.505738] nix-daemon[911]: reaped child process 2019, status = succeeded7076lb # [ 172.528524] niks3-server[1215]: 2026/09/22 10:46:54 INFO Received uploads request method=POST path=/api/pending_closures7077worker1 # [ 172.519279] nix-grpc-daemon[1858]: time=2026-09-22T10:46:54.520Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7078worker1 # [ 172.520723] nix-grpc-daemon[1858]: time=2026-09-22T10:46:54.522Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"7079lb # [ 172.545615] niks3-server[1215]: 2026/09/22 10:46:54 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst7080lb # [ 172.549241] niks3-server[1215]: 2026/09/22 10:46:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7081worker1 # [ 172.540157] nix-grpc-daemon[1858]: time=2026-09-22T10:46:54.541Z level=INFO msg="Uploading 1 narinfos"7082lb # [ 172.553500] niks3-server[1215]: 2026/09/22 10:46:54 INFO Signed narinfos id=64 count=17083lb # [ 172.558682] niks3-server[1215]: 2026/09/22 10:46:54 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls7084lb # [ 172.564058] niks3-server[1215]: 2026/09/22 10:46:54 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7085lb # [ 172.567146] niks3-server[1215]: 2026/09/22 10:46:54 INFO Completed upload id=647086worker1 # [ 172.553757] nix-grpc-daemon[1858]: time=2026-09-22T10:46:54.555Z level=INFO msg="Upload complete. (342ms)"7087worker1 # [ 172.555647] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=0 paths=1 nar_bytes_in=6247088lb # [ 172.569721] niks3-server[1215]: 2026/09/22 10:46:54 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo7089worker1 # [ 172.558401] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767090worker1 # [ 172.564059] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7091worker1 # [ 172.568539] nix-daemon[911]: reaped child process 2009, status = succeeded7092worker1 # [ 172.572789] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7093worker1 # [ 173.787593] nix-daemon[911]: reaped child process 2023, status = succeeded7094worker1 # [ 173.900563] nix-daemon[911]: accepted connection from pid 2039, user nix-grpc-daemon (trusted)7095worker1 # [ 173.911952] nix-daemon[911]: reaped child process 2042, status = succeeded7096lb # [ 173.933323] niks3-server[1215]: 2026/09/22 10:46:55 INFO Received uploads request method=POST path=/api/pending_closures7097worker1 # [ 173.923593] nix-grpc-daemon[1858]: time=2026-09-22T10:46:55.924Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7098worker1 # [ 173.925792] nix-grpc-daemon[1858]: time=2026-09-22T10:46:55.926Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"7099lb # [ 173.953666] niks3-server[1215]: 2026/09/22 10:46:55 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv7100lb # [ 173.959880] niks3-server[1215]: 2026/09/22 10:46:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7101lb # [ 173.962422] niks3-server[1215]: 2026/09/22 10:46:55 INFO Signed narinfos id=65 count=17102worker1 # [ 173.949085] nix-grpc-daemon[1858]: time=2026-09-22T10:46:55.950Z level=INFO msg="Uploading 1 narinfos"7103lb # [ 173.967149] niks3-server[1215]: 2026/09/22 10:46:55 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst7104lb # [ 173.969458] niks3-server[1215]: 2026/09/22 10:46:55 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls7105lb # [ 173.973325] niks3-server[1215]: 2026/09/22 10:46:55 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7106lb # [ 173.976494] niks3-server[1215]: 2026/09/22 10:46:55 INFO Completed upload id=657107worker1 # [ 173.963031] nix-grpc-daemon[1858]: time=2026-09-22T10:46:55.964Z level=INFO msg="Upload complete. (184ms)"7108lb # [ 173.979147] niks3-server[1215]: 2026/09/22 10:46:55 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo7109worker1 # [ 173.965248] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=11 outputs=17110worker1 # [ 173.969308] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767111worker2 # [ 173.964630] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:55Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=17112worker1 # [ 173.975297] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572767113worker1 # [ 173.977843] nix-daemon[911]: reaped child process 2021, status = succeeded7114worker2 # [ 173.974642] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:55Z level=debug event=want client=5 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=07115worker1 # [ 173.981218] nix-grpc-daemon[1858]: time=2026-09-22T10:46:55.982Z level=INFO msg="All 1 paths already cached"7116worker1 # [ 173.984190] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572767117worker1 # [ 173.987762] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7118worker1 # [ 174.117469] nix-daemon[911]: accepted connection from pid 2046, user nix-grpc-daemon (trusted)7119worker1 # [ 174.129564] nix-daemon[911]: reaped child process 2049, status = succeeded7120worker1 # [ 174.254373] nix-daemon[911]: accepted connection from pid 2051, user nix-grpc-daemon (trusted)7121worker1 # [ 174.266529] nix-daemon[911]: reaped child process 2054, status = succeeded7122lb # [ 174.288785] niks3-server[1215]: 2026/09/22 10:46:56 INFO Received uploads request method=POST path=/api/pending_closures7123worker1 # [ 174.282650] nix-grpc-daemon[1858]: time=2026-09-22T10:46:56.284Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7124worker1 # [ 174.284523] nix-grpc-daemon[1858]: time=2026-09-22T10:46:56.285Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"7125lb # [ 174.306070] niks3-server[1215]: 2026/09/22 10:46:56 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst7126lb # [ 174.309684] niks3-server[1215]: 2026/09/22 10:46:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7127lb # [ 174.311422] niks3-server[1215]: 2026/09/22 10:46:56 INFO Signed narinfos id=66 count=17128worker1 # [ 174.299283] nix-grpc-daemon[1858]: time=2026-09-22T10:46:56.300Z level=INFO msg="Uploading 1 narinfos"7129lb # [ 174.316107] niks3-server[1215]: 2026/09/22 10:46:56 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls7130lb # [ 174.319248] niks3-server[1215]: 2026/09/22 10:46:56 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7131lb # [ 174.323140] niks3-server[1215]: 2026/09/22 10:46:56 INFO Completed upload id=667132worker1 # [ 174.309498] nix-grpc-daemon[1858]: time=2026-09-22T10:46:56.310Z level=INFO msg="Upload complete. (316ms)"7133lb # [ 174.324639] niks3-server[1215]: 2026/09/22 10:46:56 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo7134worker1 # [ 174.311395] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=0 paths=1 nar_bytes_in=6247135worker1 # [ 174.315180] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767136worker1 # [ 174.321185] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7137worker1 # [ 174.323421] nix-daemon[911]: reaped child process 2044, status = succeeded7138worker1 # [ 174.328702] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7139worker1 # [ 175.340841] nix-daemon[911]: reaped child process 2058, status = succeeded7140worker1 # [ 175.461097] nix-daemon[911]: accepted connection from pid 2074, user nix-grpc-daemon (trusted)7141worker1 # [ 175.473290] nix-daemon[911]: reaped child process 2077, status = succeeded7142lb # [ 175.496957] niks3-server[1215]: 2026/09/22 10:46:57 INFO Received uploads request method=POST path=/api/pending_closures7143worker1 # [ 175.491625] nix-grpc-daemon[1858]: time=2026-09-22T10:46:57.492Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7144worker1 # [ 175.493868] nix-grpc-daemon[1858]: time=2026-09-22T10:46:57.494Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"7145lb # [ 175.539085] niks3-server[1215]: 2026/09/22 10:46:57 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv7146lb # [ 175.544146] niks3-server[1215]: 2026/09/22 10:46:57 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst7147lb # [ 175.552222] niks3-server[1215]: 2026/09/22 10:46:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7148worker1 # [ 175.544872] nix-grpc-daemon[1858]: time=2026-09-22T10:46:57.545Z level=INFO msg="Uploading 1 narinfos"7149lb # [ 175.557738] niks3-server[1215]: 2026/09/22 10:46:57 INFO Signed narinfos id=67 count=17150lb # [ 175.564505] niks3-server[1215]: 2026/09/22 10:46:57 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls7151lb # [ 175.573616] niks3-server[1215]: 2026/09/22 10:46:57 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7152lb # [ 175.578855] niks3-server[1215]: 2026/09/22 10:46:57 INFO Completed upload id=677153worker1 # [ 175.567351] nix-grpc-daemon[1858]: time=2026-09-22T10:46:57.568Z level=INFO msg="Upload complete. (232ms)"7154lb # [ 175.582048] niks3-server[1215]: 2026/09/22 10:46:57 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo7155worker1 # [ 175.569807] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=12 outputs=17156worker1 # [ 175.576810] nix-daemon[911]: reaped child process 2056, status = succeeded7157worker2 # [ 175.569871] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:57Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=17158worker2 # [ 175.580432] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:57Z level=debug event=want client=5 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=07159worker1 # [ 175.620927] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767160worker1 # [ 175.629516] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572767161worker1 # [ 175.654461] nix-grpc-daemon[1858]: time=2026-09-22T10:46:57.655Z level=INFO msg="All 2 paths already cached"7162worker1 # [ 175.661491] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572767163worker1 # [ 175.665258] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7164worker1 # [ 175.803125] nix-daemon[911]: accepted connection from pid 2081, user nix-grpc-daemon (trusted)7165worker1 # [ 175.816045] nix-daemon[911]: reaped child process 2084, status = succeeded7166worker1 # [ 175.956109] nix-daemon[911]: accepted connection from pid 2086, user nix-grpc-daemon (trusted)7167worker1 # [ 175.968312] nix-daemon[911]: reaped child process 2089, status = succeeded7168lb # [ 175.991420] niks3-server[1215]: 2026/09/22 10:46:57 INFO Received uploads request method=POST path=/api/pending_closures7169worker1 # [ 175.985550] nix-grpc-daemon[1858]: time=2026-09-22T10:46:57.986Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7170worker1 # [ 175.987392] nix-grpc-daemon[1858]: time=2026-09-22T10:46:57.988Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7171lb # [ 176.016926] niks3-server[1215]: 2026/09/22 10:46:57 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7172lb # [ 176.028088] niks3-server[1215]: 2026/09/22 10:46:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7173lb # [ 176.031776] niks3-server[1215]: 2026/09/22 10:46:57 INFO Signed narinfos id=68 count=17174worker1 # [ 176.020888] nix-grpc-daemon[1858]: time=2026-09-22T10:46:58.022Z level=INFO msg="Uploading 1 narinfos"7175lb # [ 176.036995] niks3-server[1215]: 2026/09/22 10:46:57 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7176lb # [ 176.045058] niks3-server[1215]: 2026/09/22 10:46:57 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7177lb # [ 176.048187] niks3-server[1215]: 2026/09/22 10:46:57 INFO Completed upload id=687178worker1 # [ 176.034847] nix-grpc-daemon[1858]: time=2026-09-22T10:46:58.036Z level=INFO msg="Upload complete. (363ms)"7179worker1 # [ 176.036829] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=0 paths=1 nar_bytes_in=7527180lb # [ 176.050908] niks3-server[1215]: 2026/09/22 10:46:57 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7181worker1 # [ 176.039790] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767182worker1 # [ 176.044962] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7183worker1 # [ 176.049490] nix-daemon[911]: reaped child process 2079, status = succeeded7184worker1 # [ 176.054420] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7185worker1 # [ 176.867082] nix-daemon[911]: reaped child process 2093, status = succeeded7186worker1 # [ 177.002301] nix-daemon[911]: accepted connection from pid 2109, user nix-grpc-daemon (trusted)7187worker1 # [ 177.015164] nix-daemon[911]: reaped child process 2112, status = succeeded7188lb # [ 177.041805] niks3-server[1215]: 2026/09/22 10:46:58 INFO Received uploads request method=POST path=/api/pending_closures7189worker1 # [ 177.032629] nix-grpc-daemon[1858]: time=2026-09-22T10:46:59.033Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7190worker1 # [ 177.035060] nix-grpc-daemon[1858]: time=2026-09-22T10:46:59.035Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7191lb # [ 177.064905] niks3-server[1215]: 2026/09/22 10:46:58 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7192lb # [ 177.067735] niks3-server[1215]: 2026/09/22 10:46:58 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7193lb # [ 177.071815] niks3-server[1215]: 2026/09/22 10:46:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7194lb # [ 177.073708] niks3-server[1215]: 2026/09/22 10:46:58 INFO Signed narinfos id=69 count=17195worker1 # [ 177.061606] nix-grpc-daemon[1858]: time=2026-09-22T10:46:59.062Z level=INFO msg="Uploading 1 narinfos"7196lb # [ 177.077523] niks3-server[1215]: 2026/09/22 10:46:58 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7197lb # [ 177.082047] niks3-server[1215]: 2026/09/22 10:46:58 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7198lb # [ 177.085585] niks3-server[1215]: 2026/09/22 10:46:58 INFO Completed upload id=697199worker1 # [ 177.072051] nix-grpc-daemon[1858]: time=2026-09-22T10:46:59.073Z level=INFO msg="Upload complete. (210ms)"7200lb # [ 177.087133] niks3-server[1215]: 2026/09/22 10:46:58 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7201worker1 # [ 177.074405] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=13 outputs=17202worker2 # [ 177.074316] nix-grpc-daemon[1728]: ts=2026-09-22T10:46:59Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17203worker1 # [ 177.081353] nix-daemon[911]: reaped child process 2091, status = succeeded7204worker1 # [ 177.118524] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767205worker1 # [ 177.124586] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572767206worker1 # [ 177.131050] nix-grpc-daemon[1858]: time=2026-09-22T10:46:59.132Z level=INFO msg="All 2 paths already cached"7207worker1 # [ 177.134421] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572767208worker1 # [ 177.139318] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7209worker1 # [ 177.285490] nix-daemon[911]: accepted connection from pid 2116, user nix-grpc-daemon (trusted)7210worker1 # [ 177.298578] nix-daemon[911]: reaped child process 2119, status = succeeded7211worker1 # [ 177.446247] nix-daemon[911]: accepted connection from pid 2121, user nix-grpc-daemon (trusted)7212worker1 # [ 177.458352] nix-daemon[911]: reaped child process 2124, status = succeeded7213lb # [ 177.485554] niks3-server[1215]: 2026/09/22 10:46:59 INFO Received uploads request method=POST path=/api/pending_closures7214worker1 # [ 177.479033] nix-grpc-daemon[1858]: time=2026-09-22T10:46:59.480Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7215worker1 # [ 177.480503] nix-grpc-daemon[1858]: time=2026-09-22T10:46:59.481Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7216lb # [ 177.503491] niks3-server[1215]: 2026/09/22 10:46:59 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7217lb # [ 177.507480] niks3-server[1215]: 2026/09/22 10:46:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7218lb # [ 177.509092] niks3-server[1215]: 2026/09/22 10:46:59 INFO Signed narinfos id=70 count=17219worker1 # [ 177.496756] nix-grpc-daemon[1858]: time=2026-09-22T10:46:59.498Z level=INFO msg="Uploading 1 narinfos"7220lb # [ 177.512932] niks3-server[1215]: 2026/09/22 10:46:59 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7221lb # [ 177.517060] niks3-server[1215]: 2026/09/22 10:46:59 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7222lb # [ 177.520492] niks3-server[1215]: 2026/09/22 10:46:59 INFO Completed upload id=707223worker1 # [ 177.506780] nix-grpc-daemon[1858]: time=2026-09-22T10:46:59.508Z level=INFO msg="Upload complete. (362ms)"7224worker1 # [ 177.508185] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=0 paths=1 nar_bytes_in=7527225lb # [ 177.522260] niks3-server[1215]: 2026/09/22 10:46:59 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7226worker1 # [ 177.513194] nix-grpc-daemon[1798]: ts=2026-09-22T10:46:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767227worker1 # [ 177.518363] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7228worker1 # [ 177.520597] nix-daemon[911]: reaped child process 2114, status = succeeded7229worker1 # [ 177.525666] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7230worker1 # [ 178.940737] nix-daemon[911]: reaped child process 2128, status = succeeded7231worker1 # [ 179.081429] nix-daemon[911]: accepted connection from pid 2144, user nix-grpc-daemon (trusted)7232worker1 # [ 179.092680] nix-daemon[911]: reaped child process 2147, status = succeeded7233lb # [ 179.122520] niks3-server[1215]: 2026/09/22 10:47:00 INFO Received uploads request method=POST path=/api/pending_closures7234worker1 # [ 179.117503] nix-grpc-daemon[1858]: time=2026-09-22T10:47:01.118Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7235worker1 # [ 179.119367] nix-grpc-daemon[1858]: time=2026-09-22T10:47:01.120Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7236lb # [ 179.154130] niks3-server[1215]: 2026/09/22 10:47:00 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7237lb # [ 179.170160] niks3-server[1215]: 2026/09/22 10:47:01 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7238lb # [ 179.178739] niks3-server[1215]: 2026/09/22 10:47:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7239lb # [ 179.183590] niks3-server[1215]: 2026/09/22 10:47:01 INFO Signed narinfos id=71 count=17240worker1 # [ 179.172875] nix-grpc-daemon[1858]: time=2026-09-22T10:47:01.173Z level=INFO msg="Uploading 1 narinfos"7241lb # [ 179.188145] niks3-server[1215]: 2026/09/22 10:47:01 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7242lb # [ 179.197943] niks3-server[1215]: 2026/09/22 10:47:01 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7243lb # [ 179.204100] niks3-server[1215]: 2026/09/22 10:47:01 INFO Completed upload id=717244worker1 # [ 179.190786] nix-grpc-daemon[1858]: time=2026-09-22T10:47:01.192Z level=INFO msg="Upload complete. (257ms)"7245lb # [ 179.206381] niks3-server[1215]: 2026/09/22 10:47:01 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7246worker2 # [ 179.193352] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:01Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17247worker1 # [ 179.198443] nix-grpc-daemon[1798]: ts=2026-09-22T10:47:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=14 outputs=17248worker1 # [ 179.210343] nix-daemon[911]: reaped child process 2126, status = succeeded7249worker1 # [ 179.238425] nix-grpc-daemon[1798]: ts=2026-09-22T10:47:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767250worker1 # [ 179.248719] nix-grpc-daemon[1798]: ts=2026-09-22T10:47:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572767251worker1 # [ 179.258687] nix-grpc-daemon[1858]: time=2026-09-22T10:47:01.259Z level=INFO msg="All 2 paths already cached"7252worker1 # [ 179.264691] nix-grpc-daemon[1798]: ts=2026-09-22T10:47:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572767253worker1 # [ 179.274162] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7254worker1 # [ 179.410068] nix-daemon[911]: accepted connection from pid 2151, user nix-grpc-daemon (trusted)7255worker1 # [ 179.424569] nix-daemon[911]: reaped child process 2154, status = succeeded7256worker1 # [ 179.552817] nix-daemon[911]: accepted connection from pid 2156, user nix-grpc-daemon (trusted)7257worker1 # [ 179.565230] nix-daemon[911]: reaped child process 2159, status = succeeded7258lb # [ 179.588831] niks3-server[1215]: 2026/09/22 10:47:01 INFO Received uploads request method=POST path=/api/pending_closures7259worker1 # [ 179.583511] nix-grpc-daemon[1858]: time=2026-09-22T10:47:01.584Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7260worker1 # [ 179.585136] nix-grpc-daemon[1858]: time=2026-09-22T10:47:01.586Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7261lb # [ 179.622815] niks3-server[1215]: 2026/09/22 10:47:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7262lb # [ 179.627752] niks3-server[1215]: 2026/09/22 10:47:01 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7263lb # [ 179.632366] niks3-server[1215]: 2026/09/22 10:47:01 INFO Signed narinfos id=72 count=17264worker1 # [ 179.621836] nix-grpc-daemon[1858]: time=2026-09-22T10:47:01.623Z level=INFO msg="Uploading 1 narinfos"7265lb # [ 179.639060] niks3-server[1215]: 2026/09/22 10:47:01 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7266lb # [ 179.651263] niks3-server[1215]: 2026/09/22 10:47:01 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7267worker1 # [ 179.644420] nix-grpc-daemon[1858]: time=2026-09-22T10:47:01.645Z level=INFO msg="Upload complete. (360ms)"7268lb # [ 179.658173] niks3-server[1215]: 2026/09/22 10:47:01 INFO Completed upload id=727269worker1 # [ 179.646439] nix-grpc-daemon[1798]: ts=2026-09-22T10:47:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=0 paths=1 nar_bytes_in=7527270lb # [ 179.661623] niks3-server[1215]: 2026/09/22 10:47:01 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7271worker1 # [ 179.652497] nix-grpc-daemon[1798]: ts=2026-09-22T10:47:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572767272worker1 # [ 179.661354] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7273worker1 # [ 179.663698] nix-daemon[911]: reaped child process 2149, status = succeeded7274worker1 # [ 179.669176] nix-daemon[911]: accepted connection from pid 1798, user nix-grpc-daemon (trusted)7275worker1 # [ 180.683558] nix-daemon[911]: reaped child process 2163, status = succeeded7276worker1 # [ 180.829093] nix-daemon[911]: accepted connection from pid 2179, user nix-grpc-daemon (trusted)7277worker1 # [ 180.840818] nix-daemon[911]: reaped child process 2182, status = succeeded7278lb # [ 180.870206] niks3-server[1215]: 2026/09/22 10:47:02 INFO Received uploads request method=POST path=/api/pending_closures7279worker1 # [ 180.865545] nix-grpc-daemon[1858]: time=2026-09-22T10:47:02.866Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7280worker1 # [ 180.868055] nix-grpc-daemon[1858]: time=2026-09-22T10:47:02.868Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7281lb # [ 180.906591] niks3-server[1215]: 2026/09/22 10:47:02 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7282lb # [ 180.918856] niks3-server[1215]: 2026/09/22 10:47:02 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7283lb # [ 180.926602] niks3-server[1215]: 2026/09/22 10:47:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7284worker1 # [ 180.919360] nix-grpc-daemon[1858]: time=2026-09-22T10:47:02.920Z level=INFO msg="Uploading 1 narinfos"7285lb # [ 180.933229] niks3-server[1215]: 2026/09/22 10:47:02 INFO Signed narinfos id=73 count=17286lb # [ 180.937475] niks3-server[1215]: 2026/09/22 10:47:02 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7287lb # [ 180.949323] niks3-server[1215]: 2026/09/22 10:47:02 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7288lb # [ 180.954041] niks3-server[1215]: 2026/09/22 10:47:02 INFO Completed upload id=737289worker1 # [ 180.942078] nix-grpc-daemon[1858]: time=2026-09-22T10:47:02.943Z level=INFO msg="Upload complete. (266ms)"7290lb # [ 180.956128] niks3-server[1215]: 2026/09/22 10:47:02 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7291worker1 # [ 180.946073] nix-grpc-daemon[1798]: ts=2026-09-22T10:47:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57276 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=15 outputs=17292worker1 # [ 180.952815] nix-daemon[911]: reaped child process 2161, status = succeeded7293client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 17.61 seconds)7294worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7295worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7296client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'7297client # [ 181.089092] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7298client # Running as unit: gpu.service; invocation ID: e445ebb703d74d0384fa83eba6b1778e7299client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.07 seconds)7300lb: must succeed: curl -sf localhost:9901/clusters7301lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7302lb: must succeed: curl -sf localhost:9901/clusters7303lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7304worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7305worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)7306client # [ 181.320787] nix[1472]: don't know how to build these paths:7307client # [ 181.322295] nix[1472]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7308worker2 # [ 181.302970] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:599987309worker1 # [ 181.310452] nix-grpc-daemon[1798]: ts=2026-09-22T10:47:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572767310client # [ 181.339386] nix[1472]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker7311worker2 # [ 181.323288] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:03Z level=debug event=want client=6 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07312lb: must succeed: curl -sf localhost:9901/clusters7313lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7314lb: must succeed: curl -sf localhost:9901/clusters7315lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)7316worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7317worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)7318client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7319lb # [ 188.400783] envoy[987]: conn peer=[2001:db8:1::1]:40358 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27141 tx=11419 ms=173677320client # [ 211.339925] nix[1472]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}7321client # [ 211.345452] nix[1472]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7322client # [ 211.373452] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7323client # [ 211.376301] systemd[1]: gpu.service: Failed with result 'exit-code'.7324client # [ 211.379718] systemd[1]: gpu.service: Consumed 149ms CPU time over 30.287s wall clock time, 37.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.7325client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.46 seconds)7326client: must succeed: journalctl -u gpu -o cat7327client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)7328lb: must succeed: curl -sf localhost:9901/clusters7329lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7330lb: must succeed: curl -sf localhost:9901/clusters7331lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)7332worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7333worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)7334(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 48.70 seconds)7335subtest: low disk drains a worker and builds go to the other7336worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7337worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.02 seconds)7338worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7339worker1 # [ 213.114028] nix-grpc-daemon[1798]: ts=2026-09-22T10:47:35Z level=info event=unhealthy reason=min_free7340worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.05 seconds)7341client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&27342client # don't know how to build these paths:7343worker2 # [ 213.308031] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:599987344client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7345worker2 # [ 213.311339] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599987346worker2 # [ 213.326039] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:35Z level=debug event=want client=7 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07347worker2 # [ 213.327963] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:35Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17348worker2 # [ 213.368836] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599987349worker2 # [ 213.376237] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599987350client # copying 1 paths to 'lb:50051'...7351client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7352worker2 # [ 213.383745] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:599987353worker2 # [ 213.389154] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)7354worker2 # [ 213.535059] nix-daemon[931]: accepted connection from pid 2023, user nix-grpc-daemon (trusted)7355worker2 # [ 213.547207] nix-daemon[931]: reaped child process 2026, status = succeeded7356worker2 # [ 213.717832] nix-daemon[931]: accepted connection from pid 2028, user nix-grpc-daemon (trusted)7357worker2 # [ 213.729771] nix-daemon[931]: reaped child process 2031, status = succeeded7358lb # [ 213.762720] niks3-server[1215]: 2026/09/22 10:47:35 INFO Received uploads request method=POST path=/api/pending_closures7359worker2 # [ 213.755400] nix-grpc-daemon[1804]: time=2026-09-22T10:47:35.722Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7360worker2 # [ 213.756886] nix-grpc-daemon[1804]: time=2026-09-22T10:47:35.724Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7361lb # [ 213.800656] niks3-server[1215]: 2026/09/22 10:47:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7362worker2 # [ 213.791042] nix-grpc-daemon[1804]: time=2026-09-22T10:47:35.758Z level=INFO msg="Uploading 1 narinfos"7363lb # [ 213.807415] niks3-server[1215]: 2026/09/22 10:47:35 INFO Signed narinfos id=74 count=17364lb # [ 213.812697] niks3-server[1215]: 2026/09/22 10:47:35 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7365lb # [ 213.818160] niks3-server[1215]: 2026/09/22 10:47:35 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7366lb # [ 213.832397] niks3-server[1215]: 2026/09/22 10:47:35 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7367worker2 # [ 213.822332] nix-grpc-daemon[1804]: time=2026-09-22T10:47:35.789Z level=INFO msg="Upload complete. (428ms)"7368lb # [ 213.838413] niks3-server[1215]: 2026/09/22 10:47:35 INFO Completed upload id=747369worker2 # [ 213.825075] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=0 paths=1 nar_bytes_in=5127370lb # [ 213.842085] niks3-server[1215]: 2026/09/22 10:47:35 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7371worker2 # [ 213.830504] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599987372worker2 # [ 213.837188] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)7373worker2 # [ 213.839823] nix-daemon[931]: reaped child process 2021, status = succeeded7374worker2 # [ 213.845875] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)7375client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7376client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7377client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7378client # farm-a> LOG-farm-a-drain7379worker2 # [ 215.262398] nix-daemon[931]: reaped child process 2035, status = succeeded7380worker2 # [ 215.401424] nix-daemon[931]: accepted connection from pid 2051, user nix-grpc-daemon (trusted)7381worker2 # [ 215.412122] nix-daemon[931]: reaped child process 2054, status = succeeded7382lb # [ 215.446470] niks3-server[1215]: 2026/09/22 10:47:37 INFO Received uploads request method=POST path=/api/pending_closures7383worker2 # [ 215.440873] nix-grpc-daemon[1804]: time=2026-09-22T10:47:37.408Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7384worker2 # [ 215.442437] nix-grpc-daemon[1804]: time=2026-09-22T10:47:37.409Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7385lb # [ 215.490060] niks3-server[1215]: 2026/09/22 10:47:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7386lb # [ 215.493824] niks3-server[1215]: 2026/09/22 10:47:37 INFO Signed narinfos id=75 count=17387worker2 # [ 215.481417] nix-grpc-daemon[1804]: time=2026-09-22T10:47:37.448Z level=INFO msg="Uploading 1 narinfos"7388lb # [ 215.505148] niks3-server[1215]: 2026/09/22 10:47:37 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7389lb # [ 215.510087] niks3-server[1215]: 2026/09/22 10:47:37 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7390lb # [ 215.517762] niks3-server[1215]: 2026/09/22 10:47:37 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7391lb # [ 215.522840] niks3-server[1215]: 2026/09/22 10:47:37 INFO Completed upload id=757392worker2 # [ 215.508275] nix-grpc-daemon[1804]: time=2026-09-22T10:47:37.475Z level=INFO msg="Upload complete. (254ms)"7393lb # [ 215.524702] niks3-server[1215]: 2026/09/22 10:47:37 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7394lb # [ 215.527131] niks3-server[1215]: 2026/09/22 10:47:37 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7395worker2 # [ 215.512329] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=16 outputs=17396worker2 # [ 215.518240] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:37Z level=debug event=want client=7 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07397worker2 # [ 215.520214] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:37Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17398worker2 # [ 215.523883] nix-daemon[931]: reaped child process 2033, status = succeeded7399worker2 # [ 215.566133] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599987400worker2 # [ 215.577398] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599987401worker2 # [ 215.600491] nix-grpc-daemon[1804]: time=2026-09-22T10:47:37.567Z level=INFO msg="All 1 paths already cached"7402client # copying 1 paths to 'lb:50051'...7403client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7404worker2 # [ 215.604664] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:599987405worker2 # [ 215.608378] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)7406worker2 # [ 215.745449] nix-daemon[931]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7407worker2 # [ 215.758272] nix-daemon[931]: reaped child process 2063, status = succeeded7408worker2 # [ 215.912818] nix-daemon[931]: accepted connection from pid 2065, user nix-grpc-daemon (trusted)7409worker2 # [ 215.923389] nix-daemon[931]: reaped child process 2068, status = succeeded7410lb # [ 215.952837] niks3-server[1215]: 2026/09/22 10:47:37 INFO Received uploads request method=POST path=/api/pending_closures7411worker2 # [ 215.945087] nix-grpc-daemon[1804]: time=2026-09-22T10:47:37.912Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7412worker2 # [ 215.946575] nix-grpc-daemon[1804]: time=2026-09-22T10:47:37.913Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7413lb # [ 215.980660] niks3-server[1215]: 2026/09/22 10:47:37 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7414lb # [ 215.992069] niks3-server[1215]: 2026/09/22 10:47:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign7415lb # [ 215.996619] niks3-server[1215]: 2026/09/22 10:47:37 INFO Signed narinfos id=76 count=17416worker2 # [ 215.983552] nix-grpc-daemon[1804]: time=2026-09-22T10:47:37.950Z level=INFO msg="Uploading 1 narinfos"7417lb # [ 216.001542] niks3-server[1215]: 2026/09/22 10:47:37 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7418lb # [ 216.013992] niks3-server[1215]: 2026/09/22 10:47:37 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete7419worker2 # [ 216.005462] nix-grpc-daemon[1804]: time=2026-09-22T10:47:37.972Z level=INFO msg="Upload complete. (390ms)"7420lb # [ 216.021556] niks3-server[1215]: 2026/09/22 10:47:37 INFO Completed upload id=767421worker2 # [ 216.007436] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=0 paths=1 nar_bytes_in=6247422lb # [ 216.024754] niks3-server[1215]: 2026/09/22 10:47:37 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7423worker2 # [ 216.013454] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599987424worker2 # [ 216.017752] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)7425worker2 # [ 216.019964] nix-daemon[931]: reaped child process 2058, status = succeeded7426worker2 # [ 216.025282] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)7427client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7428client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7429client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7430client # farm-b> LOG-farm-b-drain7431worker2 # [ 216.839282] nix-daemon[931]: reaped child process 2072, status = succeeded7432worker2 # [ 216.987375] nix-daemon[931]: accepted connection from pid 2088, user nix-grpc-daemon (trusted)7433worker2 # [ 216.998119] nix-daemon[931]: reaped child process 2091, status = succeeded7434lb # [ 217.030674] niks3-server[1215]: 2026/09/22 10:47:38 INFO Received uploads request method=POST path=/api/pending_closures7435worker2 # [ 217.022902] nix-grpc-daemon[1804]: time=2026-09-22T10:47:38.990Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7436worker2 # [ 217.024378] nix-grpc-daemon[1804]: time=2026-09-22T10:47:38.991Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7437lb # [ 217.071093] niks3-server[1215]: 2026/09/22 10:47:38 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7438lb # [ 217.076450] niks3-server[1215]: 2026/09/22 10:47:38 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7439lb # [ 217.088917] niks3-server[1215]: 2026/09/22 10:47:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign7440worker2 # [ 217.078112] nix-grpc-daemon[1804]: time=2026-09-22T10:47:39.045Z level=INFO msg="Uploading 1 narinfos"7441lb # [ 217.093887] niks3-server[1215]: 2026/09/22 10:47:38 INFO Signed narinfos id=77 count=17442lb # [ 217.098764] niks3-server[1215]: 2026/09/22 10:47:38 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7443lb # [ 217.108108] niks3-server[1215]: 2026/09/22 10:47:38 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete7444lb # [ 217.113693] niks3-server[1215]: 2026/09/22 10:47:38 INFO Completed upload id=777445worker2 # [ 217.097924] nix-grpc-daemon[1804]: time=2026-09-22T10:47:39.065Z level=INFO msg="Upload complete. (266ms)"7446lb # [ 217.115266] niks3-server[1215]: 2026/09/22 10:47:38 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7447worker2 # [ 217.100301] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=17 outputs=17448worker2 # [ 217.106792] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:39Z level=debug event=want client=7 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07449worker2 # [ 217.109247] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:39Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17450worker2 # [ 217.111167] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599987451worker2 # [ 217.115795] nix-daemon[931]: reaped child process 2070, status = succeeded7452worker2 # [ 217.119238] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599987453worker2 # [ 217.125497] nix-grpc-daemon[1804]: time=2026-09-22T10:47:39.092Z level=INFO msg="All 2 paths already cached"7454client # copying 1 paths to 'lb:50051'...7455worker2 # [ 217.130742] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:599987456client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7457worker2 # [ 217.136382] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)7458worker2 # [ 217.303577] nix-daemon[931]: accepted connection from pid 2097, user nix-grpc-daemon (trusted)7459worker2 # [ 217.316963] nix-daemon[931]: reaped child process 2100, status = succeeded7460worker2 # [ 217.476744] nix-daemon[931]: accepted connection from pid 2102, user nix-grpc-daemon (trusted)7461worker2 # [ 217.488678] nix-daemon[931]: reaped child process 2105, status = succeeded7462lb # [ 217.516091] niks3-server[1215]: 2026/09/22 10:47:39 INFO Received uploads request method=POST path=/api/pending_closures7463worker2 # [ 217.504518] nix-grpc-daemon[1804]: time=2026-09-22T10:47:39.471Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7464worker2 # [ 217.505945] nix-grpc-daemon[1804]: time=2026-09-22T10:47:39.471Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7465lb # [ 217.528506] niks3-server[1215]: 2026/09/22 10:47:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign7466lb # [ 217.530854] niks3-server[1215]: 2026/09/22 10:47:39 INFO Signed narinfos id=78 count=17467worker2 # [ 217.515173] nix-grpc-daemon[1804]: time=2026-09-22T10:47:39.482Z level=INFO msg="Uploading 1 narinfos"7468lb # [ 217.534225] niks3-server[1215]: 2026/09/22 10:47:39 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7469lb # [ 217.538302] niks3-server[1215]: 2026/09/22 10:47:39 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7470lb # [ 217.542159] niks3-server[1215]: 2026/09/22 10:47:39 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete7471lb # [ 217.545088] niks3-server[1215]: 2026/09/22 10:47:39 INFO Completed upload id=787472worker2 # [ 217.530527] nix-grpc-daemon[1804]: time=2026-09-22T10:47:39.497Z level=INFO msg="Upload complete. (387ms)"7473lb # [ 217.547029] niks3-server[1215]: 2026/09/22 10:47:39 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7474worker2 # [ 217.532440] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=0 paths=1 nar_bytes_in=7527475worker2 # [ 217.535169] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599987476worker2 # [ 217.541056] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)7477worker2 # [ 217.544545] nix-daemon[931]: reaped child process 2095, status = succeeded7478worker2 # [ 217.549905] nix-daemon[931]: accepted connection from pid 1728, user nix-grpc-daemon (trusted)7479client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7480client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7481lb # [ 218.403597] envoy[987]: conn peer=[2001:db8:1::1]:43224 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=497 ms=300347482client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7483worker2 # [ 218.762983] nix-daemon[931]: reaped child process 2109, status = succeeded7484client # farm-top> LOG-farm-top-drain7485worker2 # [ 218.903080] nix-daemon[931]: accepted connection from pid 2125, user nix-grpc-daemon (trusted)7486worker2 # [ 218.915237] nix-daemon[931]: reaped child process 2128, status = succeeded7487lb # [ 218.947610] niks3-server[1215]: 2026/09/22 10:47:40 INFO Received uploads request method=POST path=/api/pending_closures7488worker2 # [ 218.937218] nix-grpc-daemon[1804]: time=2026-09-22T10:47:40.903Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7489worker2 # [ 218.938605] nix-grpc-daemon[1804]: time=2026-09-22T10:47:40.903Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7490lb # [ 218.965389] niks3-server[1215]: 2026/09/22 10:47:40 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7491lb # [ 218.968293] niks3-server[1215]: 2026/09/22 10:47:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign7492lb # [ 218.970042] niks3-server[1215]: 2026/09/22 10:47:40 INFO Signed narinfos id=79 count=17493worker2 # [ 218.955608] nix-grpc-daemon[1804]: time=2026-09-22T10:47:40.922Z level=INFO msg="Uploading 1 narinfos"7494lb # [ 218.971405] niks3-server[1215]: 2026/09/22 10:47:40 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7495lb # [ 218.978056] niks3-server[1215]: 2026/09/22 10:47:40 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7496lb # [ 218.980769] niks3-server[1215]: 2026/09/22 10:47:40 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete7497lb # [ 218.983480] niks3-server[1215]: 2026/09/22 10:47:40 INFO Completed upload id=797498worker2 # [ 218.967688] nix-grpc-daemon[1804]: time=2026-09-22T10:47:40.935Z level=INFO msg="Upload complete. (212ms)"7499lb # [ 218.985359] niks3-server[1215]: 2026/09/22 10:47:40 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7500worker2 # [ 218.969962] nix-grpc-daemon[1728]: ts=2026-09-22T10:47:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59998 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=18 outputs=17501worker2 # [ 218.977156] nix-daemon[931]: reaped child process 2107, status = succeeded7502client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 5.88 seconds)7503client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7504client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.19 seconds)7505worker1: must succeed: rm /nix/.rw-store/fill7506worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.12 seconds)7507worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7508worker1 # [ 220.118730] nix-grpc-daemon[1798]: ts=2026-09-22T10:47:42Z level=info event=healthy reason=min_free7509worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.10 seconds)7510(finished: subtest: low disk drains a worker and builds go to the other, in 8.42 seconds)7511(finished: run the VM test script, in 221.43 seconds)7512test script finished in 221.65s7513cleanup7514kill QemuMachine (pid 49)7515client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7516kill QemuMachine (pid 50)7517lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7518kill QemuMachine (pid 52)7519worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7520kill QemuMachine (pid 51)7521worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7522(finished: cleanup, in 1.02 seconds)7523additionally exposed symbols:7524 client, lb, worker1, worker2,7525 vlan1,7526 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh