vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #489
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker1 # Disk image does not exist, creating the virtualisation disk image...13worker1: QEMU running (pid 50)14worker1 # Formatting '/build/vm-state-worker1/tmp.70XPRWOFLj', fmt=raw size=107374182415client # Disk image does not exist, creating the virtualisation disk image...16worker1 # mke2fs 1.47.4 (6-Mar-2025)17client # Formatting '/build/vm-state-client/tmp.8oInvWY94u', fmt=raw size=107374182418worker1 # Discarding device blocks: 0/262144 done19client: QEMU running (pid 49)20client # mke2fs 1.47.4 (6-Mar-2025)21worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes22worker2 # Disk image does not exist, creating the virtualisation disk image...23worker2: QEMU running (pid 52)24client # Discarding device blocks: 0/262144 done25worker1 # Filesystem UUID: e3cdf389-40e5-4787-9f82-90b620ab700026client # Creating filesystem with 262144 4k blocks and 65536 inodes27worker1 # Superblock backups stored on blocks:28client # Filesystem UUID: b43e5ad4-3a5d-4dba-9231-114682aa5aac29worker1 # 32768, 98304, 163840, 22937630client # Superblock backups stored on blocks:31worker1 # 32client # 32768, 98304, 163840, 22937633worker1 # Allocating group tables: 0/8 done34client # 35worker1 # Writing inode tables: 0/8 done36client # Allocating group tables: 0/8 done37worker1 # Creating journal (8192 blocks): done38client # Writing inode tables: 0/8 done39worker1 # Writing superblocks and filesystem accounting information: 0/8 done40client # Creating journal (8192 blocks): done41worker1 # 42client # Writing superblocks and filesystem accounting information: 0/8 done43worker1 # Virtualisation disk image created.44client # 45lb: QEMU running (pid 51)46client # Virtualisation disk image created.47lb # Disk image does not exist, creating the virtualisation disk image...48worker2 # Formatting '/build/vm-state-worker2/tmp.9eiaDPplir', fmt=raw size=107374182449lb # Formatting '/build/vm-state-lb/tmp.PaBz5gKktb', fmt=raw size=107374182450worker2 # mke2fs 1.47.4 (6-Mar-2025)51lb # mke2fs 1.47.4 (6-Mar-2025)52worker2 # Discarding device blocks: 0/262144 done53lb # Discarding device blocks: 0/262144 done54worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes55lb # Creating filesystem with 262144 4k blocks and 65536 inodes56worker2 # Filesystem UUID: 38641851-c8e9-424a-be26-c4cf0da97f8157lb # Filesystem UUID: e488cc7c-bb68-4c65-915e-0666e145429558worker2 # Superblock backups stored on blocks:59lb # Superblock backups stored on blocks:60worker2 # 32768, 98304, 163840, 22937661lb # 32768, 98304, 163840, 22937662worker2 # 63lb # 64worker2 # Allocating group tables: 0/8 done65lb # Allocating group tables: 0/8 done66worker2 # Writing inode tables: 0/8 done67lb # Writing inode tables: 0/8 done68worker2 # Creating journal (8192 blocks): done69lb # Creating journal (8192 blocks): done70worker2 # Writing superblocks and filesystem accounting information: 0/8 done71lb # Writing superblocks and filesystem accounting information: 0/8 done72worker2 # 73lb # 74worker2 # Virtualisation disk image created.75lb # Virtualisation disk image created.76(finished: start all VMs, in 0.20 seconds)77client: waiting for unit network-addresses-eth1.service78client: waiting for the VM to finish booting79worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83worker1 # 84worker1 # 85worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0086worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...87worker1 # 88worker1 # 89worker1 # 90worker1 # 91client # 92client # 93worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB0094lb # 95client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0096lb # 97lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0098worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...99worker1 # 100worker1 # 101client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...102client # 103client # 104client # 105client # 106lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...107lb # 108lb # 109lb # 110lb # 111client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00112lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00113client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...114client # 115client # 116lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...117lb # 118lb # 119worker2 # 120worker2 # 121worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00122worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...123worker2 # 124worker2 # 125worker2 # 126worker2 # 127worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00128worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129worker2 # 130worker2 # 131worker1 # Booting from ROM...132client # Booting from ROM...133lb # Booting from ROM...134worker2 # Booting from ROM...135worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/kn8lkkabvm6j2z1h4bz436yvy0sl8h8s-nixos-system-worker1-test/init regInfo=/nix/store/fi7n7rcxwmjd9xsqbdsf0axfx61p1yqr-closure-info/registration console=ttyS0,115200n8 console=tty0137worker1 # [ 0.000000] BIOS-provided physical RAM map:138worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved141worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable142worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved143client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026144worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved145worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved146worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved147worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved148client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qa4bpmmrzxnmnwz5m9pc3b9s796ifdak-nixos-system-client-test/init regInfo=/nix/store/fn7g8nqplkbnzmdkpggk8d349iiq0ydb-closure-info/registration console=ttyS0,115200n8 console=tty0149worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved150client # [ 0.000000] BIOS-provided physical RAM map:151worker1 # [ 0.000000] NX (Execute Disable) protection: active152worker1 # [ 0.000000] APIC: Static calls initialized153client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable154worker1 # [ 0.000000] SMBIOS 2.8 present.155client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved156worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014157client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved158worker1 # [ 0.000000] DMI: Memory slots populated: 1/1159worker1 # [ 0.000000] Hypervisor detected: KVM160client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable161worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000162client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved163worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00164client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved165worker1 # [ 0.000001] kvm-clock: using sched offset of 480200280 cycles166client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved167worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns168client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved169worker1 # [ 0.000004] tsc: Detected 2400.012 MHz processor170worker1 # [ 0.000811] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000171client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved172lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026173client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved174worker1 # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs175client # [ 0.000000] NX (Execute Disable) protection: active176worker1 # [ 0.000840] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT177client # [ 0.000000] APIC: Static calls initialized178client # [ 0.000000] SMBIOS 2.8 present.179worker1 # [ 0.002730] found SMP MP-table at [mem 0x000f5450-0x000f545f]180worker1 # [ 0.002741] Using GB pages for direct mapping181client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014182worker1 # [ 0.002852] RAMDISK: [mem 0x5e367000-0x5ffcffff]183lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/j1nn9bbhrn5xz154akfhw453f8i07i4d-nixos-system-lb-test/init regInfo=/nix/store/hq8v7nws787a9kwd3h009z1n9dqal4fl-closure-info/registration console=ttyS0,115200n8 console=tty0184client # [ 0.000000] DMI: Memory slots populated: 1/1185worker1 # [ 0.002860] ACPI: Early table checksum verification disabled186client # [ 0.000000] Hypervisor detected: KVM187worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026188lb # [ 0.000000] BIOS-provided physical RAM map:189worker1 # [ 0.002862] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )190client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000191lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable192client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00193worker1 # [ 0.002866] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)194lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved195client # [ 0.000000] kvm-clock: using sched offset of 483561163 cycles196worker1 # [ 0.002869] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)197lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved198client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns199lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable200worker1 # [ 0.002876] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)201client # [ 0.000005] tsc: Detected 2400.012 MHz processor202worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/hhh9ld89fpwxsi59mykz4xra0di8j4h8-nixos-system-worker2-test/init regInfo=/nix/store/scs6ihx0jbj5aw1gbnl2wv8q1ggxqmhd-closure-info/registration console=ttyS0,115200n8 console=tty0203worker1 # [ 0.002878] ACPI: FACS 0x000000005FFE0000 000040204lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved205worker2 # [ 0.000000] BIOS-provided physical RAM map:206client # [ 0.000804] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000207worker1 # [ 0.002880] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)208lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved209worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable210client # [ 0.000831] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs211lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved212worker1 # [ 0.002881] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)213worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved214client # [ 0.000834] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT215worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved216lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved217client # [ 0.002734] found SMP MP-table at [mem 0x000f5450-0x000f545f]218worker1 # [ 0.002883] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)219client # [ 0.002746] Using GB pages for direct mapping220worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable221lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved222client # [ 0.002825] RAMDISK: [mem 0x5e366000-0x5ffcffff]223worker1 # [ 0.002884] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)224lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved225worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved226client # [ 0.002832] ACPI: Early table checksum verification disabled227worker1 # [ 0.002885] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]228lb # [ 0.000000] NX (Execute Disable) protection: active229worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved230lb # [ 0.000000] APIC: Static calls initialized231client # [ 0.002835] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )232worker1 # [ 0.002887] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]233lb # [ 0.000000] SMBIOS 2.8 present.234worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved235client # [ 0.002839] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)236worker1 # [ 0.002887] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]237worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved238lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014239client # [ 0.002842] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)240worker1 # [ 0.002888] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]241lb # [ 0.000000] DMI: Memory slots populated: 1/1242worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved243lb # [ 0.000000] Hypervisor detected: KVM244worker1 # [ 0.002888] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]245client # [ 0.002857] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)246worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved247lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000248client # [ 0.002859] ACPI: FACS 0x000000005FFE0000 000040249worker1 # [ 0.002888] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]250worker2 # [ 0.000000] NX (Execute Disable) protection: active251lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00252worker2 # [ 0.000000] APIC: Static calls initialized253worker1 # [ 0.002889] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]254lb # [ 0.000000] kvm-clock: using sched offset of 466731286 cycles255client # [ 0.002860] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)256worker2 # [ 0.000000] SMBIOS 2.8 present.257worker1 # [ 0.003109] No NUMA configuration found258client # [ 0.002862] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)259worker1 # [ 0.003110] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]260worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014261lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns262worker2 # [ 0.000000] DMI: Memory slots populated: 1/1263lb # [ 0.000005] tsc: Detected 2400.012 MHz processor264worker1 # [ 0.003112] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]265client # [ 0.002863] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)266worker1 # [ 0.005397] Zone ranges:267worker2 # [ 0.000000] Hypervisor detected: KVM268lb # [ 0.000804] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000269worker1 # [ 0.005397] DMA [mem 0x0000000000001000-0x0000000000ffffff]270worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000271client # [ 0.002865] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)272worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00273lb # [ 0.000831] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs274worker1 # [ 0.005399] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]275worker1 # [ 0.005400] Normal empty276client # [ 0.002866] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]277worker1 # [ 0.005400] Device empty278worker2 # [ 0.000000] kvm-clock: using sched offset of 480042970 cycles279lb # [ 0.000834] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT280client # [ 0.002867] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]281worker1 # [ 0.005401] Movable zone start for each node282lb # [ 0.002722] found SMP MP-table at [mem 0x000f5450-0x000f545f]283worker1 # [ 0.005402] Early memory node ranges284client # [ 0.002868] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]285worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns286lb # [ 0.002734] Using GB pages for direct mapping287worker1 # [ 0.005402] node 0: [mem 0x0000000000001000-0x000000000009efff]288worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor289lb # [ 0.002818] RAMDISK: [mem 0x5e367000-0x5ffcffff]290client # [ 0.002868] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]291worker1 # [ 0.005403] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]292worker2 # [ 0.000814] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000293lb # [ 0.002827] ACPI: Early table checksum verification disabled294client # [ 0.002869] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]295lb # [ 0.002830] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )296worker1 # [ 0.005404] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]297worker2 # [ 0.000842] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs298client # [ 0.002869] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]299worker1 # [ 0.005423] On node 0, zone DMA: 1 pages in unavailable ranges300lb # [ 0.002834] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)301worker2 # [ 0.000845] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT302worker1 # [ 0.005689] On node 0, zone DMA: 97 pages in unavailable ranges303client # [ 0.002870] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]304worker2 # [ 0.002738] found SMP MP-table at [mem 0x000f5450-0x000f545f]305client # [ 0.003094] No NUMA configuration found306lb # [ 0.002838] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)307worker1 # [ 0.032858] On node 0, zone DMA32: 40 pages in unavailable ranges308worker2 # [ 0.002750] Using GB pages for direct mapping309worker1 # [ 0.033315] ACPI: PM-Timer IO Port: 0x608310client # [ 0.003094] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]311worker2 # [ 0.002872] RAMDISK: [mem 0x5e367000-0x5ffcffff]312lb # [ 0.002845] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)313worker1 # [ 0.033326] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])314client # [ 0.003097] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]315worker2 # [ 0.002880] ACPI: Early table checksum verification disabled316lb # [ 0.002847] ACPI: FACS 0x000000005FFE0000 000040317client # [ 0.005730] Zone ranges:318worker1 # [ 0.033353] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23319worker2 # [ 0.002882] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )320client # [ 0.005731] DMA [mem 0x0000000000001000-0x0000000000ffffff]321lb # [ 0.002849] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)322worker1 # [ 0.033355] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)323worker2 # [ 0.002886] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)324client # [ 0.005732] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]325worker1 # [ 0.033357] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)326lb # [ 0.002850] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)327client # [ 0.005733] Normal empty328client # [ 0.005734] Device empty329worker2 # [ 0.002890] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)330worker1 # [ 0.033358] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)331client # [ 0.005735] Movable zone start for each node332lb # [ 0.002852] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)333client # [ 0.005736] Early memory node ranges334worker1 # [ 0.033359] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)335worker2 # [ 0.002897] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)336lb # [ 0.002853] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)337client # [ 0.005736] node 0: [mem 0x0000000000001000-0x000000000009efff]338worker2 # [ 0.002899] ACPI: FACS 0x000000005FFE0000 000040339worker1 # [ 0.033360] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)340client # [ 0.005737] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]341lb # [ 0.002855] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]342worker1 # [ 0.033362] ACPI: Using ACPI (MADT) for SMP configuration information343worker2 # [ 0.002900] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)344client # [ 0.005738] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]345lb # [ 0.002856] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]346worker1 # [ 0.033363] ACPI: HPET id: 0x8086a201 base: 0xfed00000347worker1 # [ 0.033366] TSC deadline timer available348worker2 # [ 0.002902] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)349client # [ 0.005757] On node 0, zone DMA: 1 pages in unavailable ranges350lb # [ 0.002856] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]351worker1 # [ 0.033370] CPU topo: Max. logical packages: 1352client # [ 0.006029] On node 0, zone DMA: 97 pages in unavailable ranges353worker2 # [ 0.002903] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)354lb # [ 0.002857] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]355worker1 # [ 0.033370] CPU topo: Max. logical dies: 1356client # [ 0.033293] On node 0, zone DMA32: 40 pages in unavailable ranges357worker1 # [ 0.033371] CPU topo: Max. dies per package: 1358lb # [ 0.002857] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]359client # [ 0.033744] ACPI: PM-Timer IO Port: 0x608360worker2 # [ 0.002905] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)361worker1 # [ 0.033374] CPU topo: Max. threads per core: 1362client # [ 0.033756] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])363worker1 # [ 0.033375] CPU topo: Num. cores per package: 1364lb # [ 0.002858] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]365worker2 # [ 0.002906] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]366worker1 # [ 0.033375] CPU topo: Num. threads per package: 1367client # [ 0.033782] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23368lb # [ 0.002858] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]369worker2 # [ 0.002907] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]370worker1 # [ 0.033375] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs371lb # [ 0.003092] No NUMA configuration found372client # [ 0.033785] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)373worker2 # [ 0.002908] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]374worker1 # [ 0.033393] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()375lb # [ 0.003093] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]376client # [ 0.033787] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)377worker2 # [ 0.002908] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]378lb # [ 0.003095] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]379client # [ 0.033788] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)380worker1 # [ 0.033423] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]381lb # [ 0.005474] Zone ranges:382worker2 # [ 0.002909] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]383client # [ 0.033789] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)384worker1 # [ 0.033425] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]385lb # [ 0.005475] DMA [mem 0x0000000000001000-0x0000000000ffffff]386worker2 # [ 0.002909] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]387worker1 # [ 0.033426] [mem 0x60000000-0xafffffff] available for PCI devices388client # [ 0.033789] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)389lb # [ 0.005476] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]390lb # [ 0.005477] Normal empty391worker2 # [ 0.002909] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]392lb # [ 0.005478] Device empty393client # [ 0.033792] ACPI: Using ACPI (MADT) for SMP configuration information394worker1 # [ 0.033428] Booting paravirtualized kernel on KVM395worker2 # [ 0.003131] No NUMA configuration found396lb # [ 0.005479] Movable zone start for each node397client # [ 0.033792] ACPI: HPET id: 0x8086a201 base: 0xfed00000398lb # [ 0.005480] Early memory node ranges399worker1 # [ 0.033431] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns400worker2 # [ 0.003132] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]401client # [ 0.033796] TSC deadline timer available402client # [ 0.033800] CPU topo: Max. logical packages: 1403lb # [ 0.005480] node 0: [mem 0x0000000000001000-0x000000000009efff]404worker2 # [ 0.003134] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]405worker1 # [ 0.037866] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1406client # [ 0.033801] CPU topo: Max. logical dies: 1407worker2 # [ 0.005782] Zone ranges:408lb # [ 0.005481] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]409worker1 # [ 0.040387] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152410client # [ 0.033802] CPU topo: Max. dies per package: 1411worker2 # [ 0.005783] DMA [mem 0x0000000000001000-0x0000000000ffffff]412client # [ 0.033805] CPU topo: Max. threads per core: 1413lb # [ 0.005482] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]414worker1 # [ 0.040430] kvm-guest: PV spinlocks disabled, single CPU415worker2 # [ 0.005784] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]416client # [ 0.033806] CPU topo: Num. cores per package: 1417worker2 # [ 0.005785] Normal empty418lb # [ 0.005504] On node 0, zone DMA: 1 pages in unavailable ranges419worker2 # [ 0.005786] Device empty420client # [ 0.033806] CPU topo: Num. threads per package: 1421lb # [ 0.005784] On node 0, zone DMA: 97 pages in unavailable ranges422worker2 # [ 0.005787] Movable zone start for each node423client # [ 0.033806] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs424worker2 # [ 0.005787] Early memory node ranges425lb # [ 0.033914] On node 0, zone DMA32: 40 pages in unavailable ranges426lb # [ 0.034376] ACPI: PM-Timer IO Port: 0x608427client # [ 0.033823] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()428worker2 # [ 0.005788] node 0: [mem 0x0000000000001000-0x000000000009efff]429lb # [ 0.034388] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])430worker2 # [ 0.005789] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]431client # [ 0.033859] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]432worker1 # [ 0.040431] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/kn8lkkabvm6j2z1h4bz436yvy0sl8h8s-nixos-system-worker1-test/init regInfo=/nix/store/fi7n7rcxwmjd9xsqbdsf0axfx61p1yqr-closure-info/registration console=ttyS0,115200n8 console=tty0433lb # [ 0.034416] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23434worker2 # [ 0.005790] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]435client # [ 0.033861] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]436lb # [ 0.034419] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)437worker2 # [ 0.005809] On node 0, zone DMA: 1 pages in unavailable ranges438client # [ 0.033862] [mem 0x60000000-0xafffffff] available for PCI devices439worker1 # [ 0.040523] Unknown kernel command line parameters "regInfo=/nix/store/fi7n7rcxwmjd9xsqbdsf0axfx61p1yqr-closure-info/registration", will be passed to user space.440lb # [ 0.034421] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)441worker2 # [ 0.006078] On node 0, zone DMA: 97 pages in unavailable ranges442worker1 # [ 0.040536] random: crng init done443client # [ 0.033864] Booting paravirtualized kernel on KVM444worker2 # [ 0.033274] On node 0, zone DMA32: 40 pages in unavailable ranges445lb # [ 0.034422] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)446worker1 # [ 0.040537] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes447worker2 # [ 0.033723] ACPI: PM-Timer IO Port: 0x608448client # [ 0.033866] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns449lb # [ 0.034423] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)450worker2 # [ 0.033734] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])451worker1 # [ 0.042706] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)452client # [ 0.038306] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1453lb # [ 0.034424] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)454worker2 # [ 0.033758] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23455worker1 # [ 0.043785] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)456client # [ 0.040503] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152457lb # [ 0.034426] ACPI: Using ACPI (MADT) for SMP configuration information458worker1 # [ 0.043816] Fallback order for Node 0: 0459worker2 # [ 0.033761] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)460client # [ 0.040547] kvm-guest: PV spinlocks disabled, single CPU461lb # [ 0.034427] ACPI: HPET id: 0x8086a201 base: 0xfed00000462worker2 # [ 0.033763] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)463worker1 # [ 0.043819] Built 1 zonelists, mobility grouping on. Total pages: 393078464lb # [ 0.034431] TSC deadline timer available465worker1 # [ 0.043820] Policy zone: DMA32466lb # [ 0.034435] CPU topo: Max. logical packages: 1467worker2 # [ 0.033764] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)468lb # [ 0.034436] CPU topo: Max. logical dies: 1469worker1 # [ 0.046633] mem auto-init: stack:all(zero), heap alloc:on, heap free:off470worker2 # [ 0.033765] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)471lb # [ 0.034436] CPU topo: Max. dies per package: 1472worker1 # [ 0.049804] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1473lb # [ 0.034440] CPU topo: Max. threads per core: 1474worker2 # [ 0.033766] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)475worker1 # [ 0.053472] allocated 3145728 bytes of page_ext476lb # [ 0.034440] CPU topo: Num. cores per package: 1477worker1 # [ 0.063463] ftrace: allocating 48731 entries in 192 pages478worker2 # [ 0.033768] ACPI: Using ACPI (MADT) for SMP configuration information479client # [ 0.040549] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qa4bpmmrzxnmnwz5m9pc3b9s796ifdak-nixos-system-client-test/init regInfo=/nix/store/fn7g8nqplkbnzmdkpggk8d349iiq0ydb-closure-info/registration console=ttyS0,115200n8 console=tty0480lb # [ 0.034441] CPU topo: Num. threads per package: 1481worker1 # [ 0.063465] ftrace: allocated 192 pages with 2 groups482worker2 # [ 0.033769] ACPI: HPET id: 0x8086a201 base: 0xfed00000483worker1 # [ 0.064294] Dynamic Preempt: lazy484worker2 # [ 0.033772] TSC deadline timer available485lb # [ 0.034441] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs486worker1 # [ 0.064474] rcu: Preemptible hierarchical RCU implementation.487client # [ 0.040641] Unknown kernel command line parameters "regInfo=/nix/store/fn7g8nqplkbnzmdkpggk8d349iiq0ydb-closure-info/registration", will be passed to user space.488worker2 # [ 0.033776] CPU topo: Max. logical packages: 1489lb # [ 0.034460] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()490client # [ 0.040654] random: crng init done491worker2 # [ 0.033777] CPU topo: Max. logical dies: 1492worker1 # [ 0.064474] rcu: RCU event tracing is enabled.493worker2 # [ 0.033777] CPU topo: Max. dies per package: 1494lb # [ 0.034493] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]495worker1 # [ 0.064475] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.496client # [ 0.040655] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes497worker2 # [ 0.033781] CPU topo: Max. threads per core: 1498worker1 # [ 0.064477] Trampoline variant of Tasks RCU enabled.499lb # [ 0.034495] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]500worker2 # [ 0.033782] CPU topo: Num. cores per package: 1501client # [ 0.042815] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)502worker1 # [ 0.064477] Rude variant of Tasks RCU enabled.503worker2 # [ 0.033782] CPU topo: Num. threads per package: 1504lb # [ 0.034496] [mem 0x60000000-0xafffffff] available for PCI devices505worker1 # [ 0.064477] Tracing variant of Tasks RCU enabled.506client # [ 0.043908] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)507lb # [ 0.034498] Booting paravirtualized kernel on KVM508worker2 # [ 0.033782] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs509client # [ 0.043938] Fallback order for Node 0: 0510worker1 # [ 0.064478] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.511worker2 # [ 0.033799] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()512client # [ 0.043941] Built 1 zonelists, mobility grouping on. Total pages: 393078513lb # [ 0.034501] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns514worker1 # [ 0.064479] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1515client # [ 0.043942] Policy zone: DMA32516worker2 # [ 0.033828] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]517lb # [ 0.038945] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1518client # [ 0.046749] mem auto-init: stack:all(zero), heap alloc:on, heap free:off519worker1 # [ 0.064494] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.520worker2 # [ 0.033829] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]521lb # [ 0.041246] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152522client # [ 0.049856] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1523worker2 # [ 0.033830] [mem 0x60000000-0xafffffff] available for PCI devices524lb # [ 0.041290] kvm-guest: PV spinlocks disabled, single CPU525worker1 # [ 0.064496] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.526client # [ 0.053217] allocated 3145728 bytes of page_ext527client # [ 0.063176] ftrace: allocating 48731 entries in 192 pages528worker2 # [ 0.033832] Booting paravirtualized kernel on KVM529worker1 # [ 0.064496] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.530client # [ 0.063178] ftrace: allocated 192 pages with 2 groups531worker1 # [ 0.068806] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16532client # [ 0.064026] Dynamic Preempt: lazy533worker2 # [ 0.033834] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns534client # [ 0.064186] rcu: Preemptible hierarchical RCU implementation.535worker1 # [ 0.069073] rcu: srcu_init: Setting srcu_struct sizes based on contention.536worker2 # [ 0.038293] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1537client # [ 0.064186] rcu: RCU event tracing is enabled.538worker2 # [ 0.040485] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152539worker1 # [ 0.069079] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns540client # [ 0.064187] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.541lb # [ 0.041291] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/j1nn9bbhrn5xz154akfhw453f8i07i4d-nixos-system-lb-test/init regInfo=/nix/store/hq8v7nws787a9kwd3h009z1n9dqal4fl-closure-info/registration console=ttyS0,115200n8 console=tty0542worker2 # [ 0.040527] kvm-guest: PV spinlocks disabled, single CPU543client # [ 0.064188] Trampoline variant of Tasks RCU enabled.544client # [ 0.064189] Rude variant of Tasks RCU enabled.545worker1 # [ 0.069179] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)546client # [ 0.064189] Tracing variant of Tasks RCU enabled.547worker1 # [ 0.072731] Console: colour VGA+ 80x25548lb # [ 0.041388] Unknown kernel command line parameters "regInfo=/nix/store/hq8v7nws787a9kwd3h009z1n9dqal4fl-closure-info/registration", will be passed to user space.549worker1 # [ 0.072734] printk: legacy console [tty0] enabled550lb # [ 0.041400] random: crng init done551client # [ 0.064190] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.552worker1 # [ 0.102286] printk: legacy console [ttyS0] enabled553lb # [ 0.041401] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes554client # [ 0.064190] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1555worker1 # [ 0.197433] ACPI: Core revision 20250807556lb # [ 0.043641] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)557worker2 # [ 0.040528] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/hhh9ld89fpwxsi59mykz4xra0di8j4h8-nixos-system-worker2-test/init regInfo=/nix/store/scs6ihx0jbj5aw1gbnl2wv8q1ggxqmhd-closure-info/registration console=ttyS0,115200n8 console=tty0558client # [ 0.064205] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.559worker1 # [ 0.198278] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns560lb # [ 0.045048] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)561lb # [ 0.045082] Fallback order for Node 0: 0562client # [ 0.064207] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.563worker2 # [ 0.040619] Unknown kernel command line parameters "regInfo=/nix/store/scs6ihx0jbj5aw1gbnl2wv8q1ggxqmhd-closure-info/registration", will be passed to user space.564worker1 # [ 0.199748] APIC: Switch to symmetric I/O mode setup565lb # [ 0.045085] Built 1 zonelists, mobility grouping on. Total pages: 393078566worker2 # [ 0.040632] random: crng init done567client # [ 0.064208] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.568worker1 # [ 0.200712] x2apic enabled569lb # [ 0.045086] Policy zone: DMA32570client # [ 0.068670] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16571worker2 # [ 0.040632] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes572lb # [ 0.047695] mem auto-init: stack:all(zero), heap alloc:on, heap free:off573worker1 # [ 0.201445] APIC: Switched APIC routing to: physical x2apic574client # [ 0.068934] rcu: srcu_init: Setting srcu_struct sizes based on contention.575worker2 # [ 0.042801] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)576lb # [ 0.051057] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1577lb # [ 0.054541] allocated 3145728 bytes of page_ext578worker2 # [ 0.043895] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)579client # [ 0.068940] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns580lb # [ 0.064594] ftrace: allocating 48731 entries in 192 pages581worker2 # [ 0.043925] Fallback order for Node 0: 0582worker1 # [ 0.203218] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1583lb # [ 0.064596] ftrace: allocated 192 pages with 2 groups584worker2 # [ 0.043927] Built 1 zonelists, mobility grouping on. Total pages: 393078585lb # [ 0.065474] Dynamic Preempt: lazy586client # [ 0.069041] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)587worker2 # [ 0.043928] Policy zone: DMA32588client # [ 0.072567] Console: colour VGA+ 80x25589lb # [ 0.065596] rcu: Preemptible hierarchical RCU implementation.590worker1 # [ 0.204168] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns591client # [ 0.072570] printk: legacy console [tty0] enabled592worker2 # [ 0.046763] mem auto-init: stack:all(zero), heap alloc:on, heap free:off593lb # [ 0.065597] rcu: RCU event tracing is enabled.594client # [ 0.101945] printk: legacy console [ttyS0] enabled595worker2 # [ 0.049840] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1596lb # [ 0.065598] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.597worker1 # [ 0.205782] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)598client # [ 0.198102] ACPI: Core revision 20250807599worker2 # [ 0.053204] allocated 3145728 bytes of page_ext600lb # [ 0.065599] Trampoline variant of Tasks RCU enabled.601worker2 # [ 0.063148] ftrace: allocating 48731 entries in 192 pages602lb # [ 0.065600] Rude variant of Tasks RCU enabled.603worker1 # [ 0.206866] x86/cpu: User Mode Instruction Prevention (UMIP) activated604worker2 # [ 0.063150] ftrace: allocated 192 pages with 2 groups605lb # [ 0.065600] Tracing variant of Tasks RCU enabled.606worker2 # [ 0.064026] Dynamic Preempt: lazy607client # [ 0.198958] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns608worker1 # [ 0.207899] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127609worker2 # [ 0.064162] rcu: Preemptible hierarchical RCU implementation.610lb # [ 0.065601] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.611client # [ 0.200434] APIC: Switch to symmetric I/O mode setup612worker2 # [ 0.064163] rcu: RCU event tracing is enabled.613worker1 # [ 0.208779] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0614lb # [ 0.065601] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1615client # [ 0.201352] x2apic enabled616worker2 # [ 0.064164] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.617lb # [ 0.065616] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.618worker2 # [ 0.064165] Trampoline variant of Tasks RCU enabled.619client # [ 0.202082] APIC: Switched APIC routing to: physical x2apic620worker1 # [ 0.209783] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto621worker2 # [ 0.064166] Rude variant of Tasks RCU enabled.622lb # [ 0.065618] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.623worker2 # [ 0.064166] Tracing variant of Tasks RCU enabled.624worker1 # [ 0.210779] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl625client # [ 0.203851] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1626worker2 # [ 0.064167] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.627lb # [ 0.065619] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.628lb # [ 0.070328] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16629worker2 # [ 0.064167] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1630worker1 # [ 0.211779] Transient Scheduler Attacks: Vulnerable: No microcode631client # [ 0.204807] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns632worker1 # [ 0.212779] Spectre V2 : Mitigation: Enhanced / Automatic IBRS633lb # [ 0.070613] rcu: srcu_init: Setting srcu_struct sizes based on contention.634worker2 # [ 0.064232] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.635worker1 # [ 0.213779] Speculative Return Stack Overflow: Mitigation: Safe RET636lb # [ 0.070619] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns637client # [ 0.206404] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)638worker2 # [ 0.064234] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.639lb # [ 0.070731] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)640client # [ 0.208486] x86/cpu: User Mode Instruction Prevention (UMIP) activated641worker2 # [ 0.064235] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.642lb # [ 0.074316] Console: colour VGA+ 80x25643worker2 # [ 0.068591] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16644lb # [ 0.074319] printk: legacy console [tty0] enabled645client # [ 0.209552] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127646lb # [ 0.103628] printk: legacy console [ttyS0] enabled647worker2 # [ 0.068873] rcu: srcu_init: Setting srcu_struct sizes based on contention.648client # [ 0.210402] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0649lb # [ 0.199576] ACPI: Core revision 20250807650worker1 # [ 0.214779] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization651worker2 # [ 0.068878] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns652worker1 # [ 0.216786] Spectre V2 : Enabling IBPB for BPF653client # [ 0.211405] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto654worker2 # [ 0.068983] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)655lb # [ 0.200512] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns656worker2 # [ 0.072529] Console: colour VGA+ 80x25657worker1 # [ 0.218290] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier658worker2 # [ 0.072532] printk: legacy console [tty0] enabled659lb # [ 0.202000] APIC: Switch to symmetric I/O mode setup660client # [ 0.212401] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl661worker1 # [ 0.218780] active return thunk: srso_alias_return_thunk662worker2 # [ 0.101922] printk: legacy console [ttyS0] enabled663lb # [ 0.202931] x2apic enabled664client # [ 0.213401] Transient Scheduler Attacks: Vulnerable: No microcode665worker2 # [ 0.197650] ACPI: Core revision 20250807666worker1 # [ 0.220405] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'667lb # [ 0.203671] APIC: Switched APIC routing to: physical x2apic668client # [ 0.214401] Spectre V2 : Mitigation: Enhanced / Automatic IBRS669worker1 # [ 0.221642] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'670worker2 # [ 0.198494] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns671client # [ 0.215401] Speculative Return Stack Overflow: Mitigation: Safe RET672worker1 # [ 0.222503] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'673lb # [ 0.205457] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1674worker2 # [ 0.200027] APIC: Switch to symmetric I/O mode setup675worker1 # [ 0.223498] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'676worker2 # [ 0.200969] x2apic enabled677lb # [ 0.206413] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns678worker1 # [ 0.224508] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'679worker2 # [ 0.201704] APIC: Switched APIC routing to: physical x2apic680worker1 # [ 0.224779] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'681lb # [ 0.208053] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)682client # [ 0.216401] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization683worker2 # [ 0.203514] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1684worker1 # [ 0.225779] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'685lb # [ 0.209138] x86/cpu: User Mode Instruction Prevention (UMIP) activated686client # [ 0.218408] Spectre V2 : Enabling IBPB for BPF687worker1 # [ 0.227653] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256688lb # [ 0.210186] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127689worker2 # [ 0.204473] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns690client # [ 0.219402] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier691worker1 # [ 0.228476] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64692lb # [ 0.211051] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0693client # [ 0.220402] active return thunk: srso_alias_return_thunk694worker1 # [ 0.229476] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512695worker2 # [ 0.206115] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)696client # [ 0.221422] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'697worker1 # [ 0.229779] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024698lb # [ 0.212054] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto699worker2 # [ 0.207199] x86/cpu: User Mode Instruction Prevention (UMIP) activated700worker1 # [ 0.230779] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8701client # [ 0.222401] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'702worker2 # [ 0.208264] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127703lb # [ 0.213051] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl704client # [ 0.223401] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'705worker2 # [ 0.209113] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0706lb # [ 0.214051] Transient Scheduler Attacks: Vulnerable: No microcode707worker1 # [ 0.231779] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.708client # [ 0.224401] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'709lb # [ 0.215050] Spectre V2 : Mitigation: Enhanced / Automatic IBRS710client # [ 0.225401] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'711worker2 # [ 0.210116] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto712lb # [ 0.216051] Speculative Return Stack Overflow: Mitigation: Safe RET713client # [ 0.226401] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'714worker2 # [ 0.211112] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl715client # [ 0.227401] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'716worker2 # [ 0.212113] Transient Scheduler Attacks: Vulnerable: No microcode717client # [ 0.228402] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256718worker2 # [ 0.213112] Spectre V2 : Mitigation: Enhanced / Automatic IBRS719lb # [ 0.217050] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization720client # [ 0.230157] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64721worker2 # [ 0.214112] Speculative Return Stack Overflow: Mitigation: Safe RET722lb # [ 0.219058] Spectre V2 : Enabling IBPB for BPF723client # [ 0.231132] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512724lb # [ 0.219773] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier725client # [ 0.232183] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024726lb # [ 0.221050] active return thunk: srso_alias_return_thunk727client # [ 0.233163] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8728lb # [ 0.222071] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'729worker2 # [ 0.215112] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization730client # [ 0.234173] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.731worker2 # [ 0.217120] Spectre V2 : Enabling IBPB for BPF732lb # [ 0.223050] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'733lb # [ 0.224050] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'734worker2 # [ 0.218113] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier735lb # [ 0.225050] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'736worker2 # [ 0.219113] active return thunk: srso_alias_return_thunk737lb # [ 0.226050] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'738worker2 # [ 0.220134] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'739lb # [ 0.227050] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'740worker2 # [ 0.222112] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'741lb # [ 0.228050] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'742worker2 # [ 0.223112] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'743lb # [ 0.229050] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256744worker2 # [ 0.224112] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'745lb # [ 0.230050] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64746worker2 # [ 0.225112] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'747lb # [ 0.231050] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512748worker2 # [ 0.226112] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'749lb # [ 0.232050] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024750lb # [ 0.233050] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8751worker2 # [ 0.227112] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'752worker2 # [ 0.228113] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256753lb # [ 0.234050] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.754worker2 # [ 0.229112] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64755worker2 # [ 0.230112] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512756worker2 # [ 0.231113] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024757worker2 # [ 0.232112] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8758worker2 # [ 0.233112] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759worker1 # [ 0.266251] Freeing SMP alternatives memory: 44K760worker1 # [ 0.266782] pid_max: default: 32768 minimum: 301761worker1 # [ 0.267859] LSM: initializing lsm=capability,landlock,yama,bpf,ima762worker1 # [ 0.268893] landlock: Up and running.763worker1 # [ 0.269499] Yama: becoming mindful.764worker1 # [ 0.270000] LSM support for eBPF active765worker1 # [ 0.270921] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)766worker1 # [ 0.271819] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)767worker1 # [ 0.274742] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)768client # [ 0.268020] Freeing SMP alternatives memory: 44K769worker1 # [ 0.275331] Performance Events: Fam17h+ core perfctr, AMD PMU driver.770client # [ 0.268404] pid_max: default: 32768 minimum: 301771worker1 # [ 0.275784] ... version: 2772worker1 # [ 0.276479] ... bit width: 48773client # [ 0.269492] LSM: initializing lsm=capability,landlock,yama,bpf,ima774worker1 # [ 0.276782] ... generic counters: 6775client # [ 0.270506] landlock: Up and running.776client # [ 0.271140] Yama: becoming mindful.777worker1 # [ 0.277485] ... generic bitmap: 000000000000003f778worker1 # [ 0.277793] ... fixed-purpose counters: 0779client # [ 0.272029] LSM support for eBPF active780worker1 # [ 0.278497] ... fixed-purpose bitmap: 0000000000000000781client # [ 0.272997] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)782worker1 # [ 0.278782] ... value mask: 0000ffffffffffff783worker1 # [ 0.279649] ... max period: 00007fffffffffff784client # [ 0.273444] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)785worker1 # [ 0.280441] ... global_ctrl mask: 000000000000003f786worker1 # [ 0.280899] signal: max sigframe size: 3376787worker1 # [ 0.281724] rcu: Hierarchical SRCU implementation.788client # [ 0.276706] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)789worker1 # [ 0.282340] rcu: Max phase no-delay instances is 400.790client # [ 0.277957] Performance Events: Fam17h+ core perfctr, AMD PMU driver.791client # [ 0.278410] ... version: 2792client # [ 0.279151] ... bit width: 48793lb # [ 0.269079] Freeing SMP alternatives memory: 44K794client # [ 0.279403] ... generic counters: 6795lb # [ 0.269872] pid_max: default: 32768 minimum: 301796client # [ 0.280134] ... generic bitmap: 000000000000003f797lb # [ 0.270704] LSM: initializing lsm=capability,landlock,yama,bpf,ima798client # [ 0.280403] ... fixed-purpose counters: 0799worker1 # [ 0.287339] smp: Bringing up secondary CPUs ...800lb # [ 0.271167] landlock: Up and running.801client # [ 0.281144] ... fixed-purpose bitmap: 0000000000000000802lb # [ 0.272482] Yama: becoming mindful.803worker1 # [ 0.287796] smp: Brought up 1 node, 1 CPU804client # [ 0.281403] ... value mask: 0000ffffffffffff805lb # [ 0.273264] LSM support for eBPF active806worker1 # [ 0.288434] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)807worker2 # [ 0.268127] Freeing SMP alternatives memory: 44K808client # [ 0.282307] ... max period: 00007fffffffffff809lb # [ 0.274062] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)810worker2 # [ 0.268888] pid_max: default: 32768 minimum: 301811client # [ 0.283063] ... global_ctrl mask: 000000000000003f812client # [ 0.283511] signal: max sigframe size: 3376813worker1 # [ 0.288935] Memory: 1454576K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)814worker2 # [ 0.269207] LSM: initializing lsm=capability,landlock,yama,bpf,ima815lb # [ 0.275091] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)816worker1 # [ 0.290001] devtmpfs: initialized817client # [ 0.284256] rcu: Hierarchical SRCU implementation.818worker2 # [ 0.270227] landlock: Up and running.819worker2 # [ 0.271113] Yama: becoming mindful.820worker1 # [ 0.290787] x86/mm: Memory block size: 128MB821client # [ 0.285009] rcu: Max phase no-delay instances is 400.822worker2 # [ 0.272297] LSM support for eBPF active823lb # [ 0.277544] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)824worker2 # [ 0.273053] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)825worker1 # [ 0.292528] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)826lb # [ 0.278622] Performance Events: Fam17h+ core perfctr, AMD PMU driver.827worker2 # [ 0.274157] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)828lb # [ 0.279056] ... version: 2829worker1 # [ 0.293662] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).830lb # [ 0.279768] ... bit width: 48831worker1 # [ 0.294888] pinctrl core: initialized pinctrl subsystem832lb # [ 0.280053] ... generic counters: 6833client # [ 0.289839] smp: Bringing up secondary CPUs ...834lb # [ 0.280745] ... generic bitmap: 000000000000003f835worker1 # [ 0.296049] PM: RTC time: 10:46:16, date: 2026-09-22836client # [ 0.290417] smp: Brought up 1 node, 1 CPU837worker2 # [ 0.277002] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)838lb # [ 0.281059] ... fixed-purpose counters: 0839client # [ 0.291090] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)840lb # [ 0.281770] ... fixed-purpose bitmap: 0000000000000000841worker2 # [ 0.277663] Performance Events: Fam17h+ core perfctr, AMD PMU driver.842lb # [ 0.282052] ... value mask: 0000ffffffffffff843worker2 # [ 0.278117] ... version: 2844lb # [ 0.282933] ... max period: 00007fffffffffff845worker2 # [ 0.278802] ... bit width: 48846client # [ 0.291620] Memory: 1454540K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)847worker1 # [ 0.299533] NET: Registered PF_NETLINK/PF_ROUTE protocol family848worker2 # [ 0.279115] ... generic counters: 6849lb # [ 0.283725] ... global_ctrl mask: 000000000000003f850client # [ 0.292623] devtmpfs: initialized851worker2 # [ 0.279792] ... generic bitmap: 000000000000003f852lb # [ 0.284159] signal: max sigframe size: 3376853client # [ 0.293382] x86/mm: Memory block size: 128MB854worker1 # [ 0.301246] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations855worker2 # [ 0.280125] ... fixed-purpose counters: 0856lb # [ 0.284958] rcu: Hierarchical SRCU implementation.857worker2 # [ 0.280809] ... fixed-purpose bitmap: 0000000000000000858lb # [ 0.285618] rcu: Max phase no-delay instances is 400.859worker1 # [ 0.301804] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations860worker2 # [ 0.281115] ... value mask: 0000ffffffffffff861client # [ 0.295007] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)862worker1 # [ 0.303063] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations863worker2 # [ 0.282021] ... max period: 00007fffffffffff864client # [ 0.295432] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).865worker1 # [ 0.303796] audit: initializing netlink subsys (disabled)866worker2 # [ 0.282799] ... global_ctrl mask: 000000000000003f867client # [ 0.296486] pinctrl core: initialized pinctrl subsystem868worker2 # [ 0.283247] signal: max sigframe size: 3376869worker1 # [ 0.304935] thermal_sys: Registered thermal governor 'fair_share'870worker2 # [ 0.284010] rcu: Hierarchical SRCU implementation.871client # [ 0.297606] PM: RTC time: 10:46:16, date: 2026-09-22872worker1 # [ 0.304938] thermal_sys: Registered thermal governor 'bang_bang'873worker2 # [ 0.284676] rcu: Max phase no-delay instances is 400.874lb # [ 0.290745] smp: Bringing up secondary CPUs ...875worker1 # [ 0.305783] thermal_sys: Registered thermal governor 'step_wise'876lb # [ 0.291066] smp: Brought up 1 node, 1 CPU877worker1 # [ 0.306715] thermal_sys: Registered thermal governor 'user_space'878lb # [ 0.291731] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)879worker1 # [ 0.307520] audit: type=2000 audit(1790073976.670:1): state=initialized audit_enabled=0 res=1880client # [ 0.301287] NET: Registered PF_NETLINK/PF_ROUTE protocol family881worker1 # [ 0.308784] thermal_sys: Registered thermal governor 'power_allocator'882lb # [ 0.292225] Memory: 1454580K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)883worker1 # [ 0.308798] cpuidle: using governor menu884client # [ 0.302607] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations885lb # [ 0.293372] devtmpfs: initialized886worker2 # [ 0.289670] smp: Bringing up secondary CPUs ...887client # [ 0.303426] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations888worker1 # [ 0.310775] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5889worker2 # [ 0.290128] smp: Brought up 1 node, 1 CPU890lb # [ 0.294143] x86/mm: Memory block size: 128MB891worker2 # [ 0.290775] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)892client # [ 0.304682] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations893worker1 # [ 0.312002] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]894client # [ 0.305413] audit: initializing netlink subsys (disabled)895lb # [ 0.295955] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)896worker1 # [ 0.312784] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry897worker2 # [ 0.291319] Memory: 1454556K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)898worker1 # [ 0.313792] PCI: Using configuration type 1 for base access899client # [ 0.306520] thermal_sys: Registered thermal governor 'fair_share'900lb # [ 0.296898] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).901worker2 # [ 0.292344] devtmpfs: initialized902client # [ 0.306522] thermal_sys: Registered thermal governor 'bang_bang'903lb # [ 0.298139] pinctrl core: initialized pinctrl subsystem904worker2 # [ 0.293073] x86/mm: Memory block size: 128MB905worker1 # [ 0.314886] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.906client # [ 0.307405] thermal_sys: Registered thermal governor 'step_wise'907lb # [ 0.299227] PM: RTC time: 10:46:16, date: 2026-09-22908client # [ 0.308335] thermal_sys: Registered thermal governor 'user_space'909worker2 # [ 0.294709] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)910client # [ 0.309135] audit: type=2000 audit(1790073976.679:1): state=initialized audit_enabled=0 res=1911worker2 # [ 0.295141] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).912client # [ 0.310407] thermal_sys: Registered thermal governor 'power_allocator'913client # [ 0.310420] cpuidle: using governor menu914worker2 # [ 0.296205] pinctrl core: initialized pinctrl subsystem915lb # [ 0.302958] NET: Registered PF_NETLINK/PF_ROUTE protocol family916worker2 # [ 0.297319] PM: RTC time: 10:46:16, date: 2026-09-22917client # [ 0.312671] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5918lb # [ 0.304285] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations919worker1 # [ 0.322071] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages920lb # [ 0.305076] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations921client # [ 0.313626] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]922worker1 # [ 0.322783] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page923client # [ 0.314406] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry924lb # [ 0.306342] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations925worker2 # [ 0.300929] NET: Registered PF_NETLINK/PF_ROUTE protocol family926client # [ 0.315416] PCI: Using configuration type 1 for base access927lb # [ 0.307064] audit: initializing netlink subsys (disabled)928worker2 # [ 0.302335] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations929lb # [ 0.308208] thermal_sys: Registered thermal governor 'fair_share'930client # [ 0.316498] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.931lb # [ 0.308210] thermal_sys: Registered thermal governor 'bang_bang'932worker2 # [ 0.303135] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations933lb # [ 0.309054] thermal_sys: Registered thermal governor 'step_wise'934worker1 # [ 0.327783] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages935worker2 # [ 0.304402] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations936lb # [ 0.309990] thermal_sys: Registered thermal governor 'user_space'937worker1 # [ 0.328782] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page938worker2 # [ 0.305125] audit: initializing netlink subsys (disabled)939lb # [ 0.310788] audit: type=2000 audit(1790073976.692:1): state=initialized audit_enabled=0 res=1940worker2 # [ 0.306235] thermal_sys: Registered thermal governor 'fair_share'941lb # [ 0.312055] thermal_sys: Registered thermal governor 'power_allocator'942client # [ 0.321661] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages943worker2 # [ 0.306237] thermal_sys: Registered thermal governor 'bang_bang'944lb # [ 0.312069] cpuidle: using governor menu945client # [ 0.322405] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page946worker2 # [ 0.307116] thermal_sys: Registered thermal governor 'step_wise'947lb # [ 0.314248] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5948worker2 # [ 0.308053] thermal_sys: Registered thermal governor 'user_space'949lb # [ 0.315349] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]950worker2 # [ 0.308883] audit: type=2000 audit(1790073976.696:1): state=initialized audit_enabled=0 res=1951lb # [ 0.316056] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry952worker2 # [ 0.310118] thermal_sys: Registered thermal governor 'power_allocator'953worker2 # [ 0.310133] cpuidle: using governor menu954lb # [ 0.317065] PCI: Using configuration type 1 for base access955client # [ 0.327405] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages956worker1 # [ 0.337378] ACPI: Added _OSI(Module Device)957worker1 # [ 0.337783] ACPI: Added _OSI(Processor Device)958worker2 # [ 0.312337] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5959client # [ 0.328404] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page960lb # [ 0.318131] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.961worker1 # [ 0.338475] ACPI: Added _OSI(Processor Aggregator Device)962worker2 # [ 0.313348] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]963worker2 # [ 0.314117] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry964worker2 # [ 0.315127] PCI: Using configuration type 1 for base access965worker2 # [ 0.316288] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.966lb # [ 0.325252] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages967worker1 # [ 0.345388] ACPI: 1 ACPI AML tables successfully acquired and loaded968lb # [ 0.326054] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page969client # [ 0.338960] ACPI: Added _OSI(Module Device)970client # [ 0.339406] ACPI: Added _OSI(Processor Device)971worker1 # [ 0.348725] ACPI: Interpreter enabled972client # [ 0.340153] ACPI: Added _OSI(Processor Aggregator Device)973worker1 # [ 0.349248] ACPI: PM: (supports S0 S3 S4 S5)974worker2 # [ 0.323361] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages975lb # [ 0.331054] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages976worker1 # [ 0.349782] ACPI: Using IOAPIC for interrupt routing977worker2 # [ 0.324116] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page978lb # [ 0.332053] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page979worker1 # [ 0.350626] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug980worker2 # [ 0.329116] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages981worker2 # [ 0.330116] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page982worker1 # [ 0.355782] PCI: Using E820 reservations for host bridge windows983worker1 # [ 0.356852] ACPI: Enabled 2 GPEs in block 00 to 3F984client # [ 0.348424] ACPI: 1 ACPI AML tables successfully acquired and loaded985lb # [ 0.342498] ACPI: Added _OSI(Module Device)986client # [ 0.352834] ACPI: Interpreter enabled987lb # [ 0.343054] ACPI: Added _OSI(Processor Device)988client # [ 0.353427] ACPI: PM: (supports S0 S3 S4 S5)989lb # [ 0.343777] ACPI: Added _OSI(Processor Aggregator Device)990client # [ 0.354141] ACPI: Using IOAPIC for interrupt routing991worker2 # [ 0.338719] ACPI: Added _OSI(Module Device)992worker2 # [ 0.339117] ACPI: Added _OSI(Processor Device)993worker1 # [ 0.365437] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])994worker2 # [ 0.339906] ACPI: Added _OSI(Processor Aggregator Device)995client # [ 0.356457] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug996client # [ 0.357404] PCI: Using E820 reservations for host bridge windows997worker1 # [ 0.366265] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]998worker1 # [ 0.366859] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]999lb # [ 0.350626] ACPI: 1 ACPI AML tables successfully acquired and loaded1000client # [ 0.360530] ACPI: Enabled 2 GPEs in block 00 to 3F1001worker1 # [ 0.367904] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1002worker1 # [ 0.369131] PCI host bridge to bus 0000:001003worker1 # [ 0.369787] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1004worker2 # [ 0.346729] ACPI: 1 ACPI AML tables successfully acquired and loaded1005worker1 # [ 0.370782] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1006lb # [ 0.354469] ACPI: Interpreter enabled1007lb # [ 0.355068] ACPI: PM: (supports S0 S3 S4 S5)1008worker1 # [ 0.371782] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1009lb # [ 0.355763] ACPI: Using IOAPIC for interrupt routing1010worker1 # [ 0.372781] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1011worker2 # [ 0.350067] ACPI: Interpreter enabled1012worker1 # [ 0.373782] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1013client # [ 0.367958] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1014worker2 # [ 0.350568] ACPI: PM: (supports S0 S3 S4 S5)1015worker1 # [ 0.374782] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1016worker2 # [ 0.351115] ACPI: Using IOAPIC for interrupt routing1017lb # [ 0.358103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1018client # [ 0.368410] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1019worker1 # [ 0.375783] pci_bus 0000:00: root bus resource [bus 00-ff]1020lb # [ 0.359053] PCI: Using E820 reservations for host bridge windows1021worker2 # [ 0.351948] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1022client # [ 0.369482] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1023worker1 # [ 0.376709] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1024client # [ 0.370525] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1025lb # [ 0.362110] ACPI: Enabled 2 GPEs in block 00 to 3F1026worker1 # [ 0.378244] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1027client # [ 0.371769] PCI host bridge to bus 0000:001028client # [ 0.372408] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1029client # [ 0.373404] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1030worker2 # [ 0.357116] PCI: Using E820 reservations for host bridge windows1031worker2 # [ 0.358213] ACPI: Enabled 2 GPEs in block 00 to 3F1032client # [ 0.374404] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1033client # [ 0.375404] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1034client # [ 0.376404] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1035client # [ 0.377404] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1036lb # [ 0.369218] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1037client # [ 0.378410] pci_bus 0000:00: root bus resource [bus 00-ff]1038worker1 # [ 0.380845] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1039lb # [ 0.370059] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1040worker1 # [ 0.381794] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1041client # [ 0.379369] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1042lb # [ 0.371130] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1043worker1 # [ 0.382765] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1044lb # [ 0.372175] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1045client # [ 0.380862] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1046lb # [ 0.373391] PCI host bridge to bus 0000:001047worker2 # [ 0.366828] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1048worker1 # [ 0.383990] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1049lb # [ 0.374057] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1050lb # [ 0.375054] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1051worker1 # [ 0.385494] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1052worker2 # [ 0.367746] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1053lb # [ 0.376053] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1054worker2 # [ 0.368208] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1055lb # [ 0.377054] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1056worker2 # [ 0.369240] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1057worker1 # [ 0.388813] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1058worker2 # [ 0.370461] PCI host bridge to bus 0000:001059lb # [ 0.378053] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1060worker1 # [ 0.389708] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1061worker2 # [ 0.371129] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1062lb # [ 0.379053] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1063worker1 # [ 0.390528] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1064lb # [ 0.380058] pci_bus 0000:00: root bus resource [bus 00-ff]1065worker2 # [ 0.372115] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1066client # [ 0.383466] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1067worker1 # [ 0.390788] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1068worker2 # [ 0.373115] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1069client # [ 0.384415] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1070lb # [ 0.381012] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1071worker2 # [ 0.374115] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1072client # [ 0.385345] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1073worker1 # [ 0.392884] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1074lb # [ 0.382506] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1075worker2 # [ 0.375115] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1076client # [ 0.386533] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1077worker2 # [ 0.376115] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1078worker1 # [ 0.394814] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1079worker2 # [ 0.377115] pci_bus 0000:00: root bus resource [bus 00-ff]1080client # [ 0.388104] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1081worker1 # [ 0.395685] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1082worker2 # [ 0.378094] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1083worker1 # [ 0.396556] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1084worker2 # [ 0.379554] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1085client # [ 0.391421] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1086worker1 # [ 0.398302] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1087client # [ 0.392270] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1088client # [ 0.393170] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1089client # [ 0.393409] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1090lb # [ 0.385137] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1091lb # [ 0.386066] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1092worker1 # [ 0.401008] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1093client # [ 0.395287] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1094worker1 # [ 0.402435] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1095lb # [ 0.387036] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1096worker1 # [ 0.402804] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1097lb # [ 0.388053] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1098client # [ 0.397412] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1099worker1 # [ 0.404353] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1100worker2 # [ 0.382179] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1101client # [ 0.398331] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1102lb # [ 0.389775] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1103worker2 # [ 0.383127] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1104client # [ 0.399114] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1105worker2 # [ 0.384079] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1106worker1 # [ 0.405813] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1107client # [ 0.399965] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1108worker1 # [ 0.406684] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1109worker2 # [ 0.385362] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1110lb # [ 0.393081] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1111worker1 # [ 0.407521] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1112lb # [ 0.393943] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1113worker2 # [ 0.386903] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1114lb # [ 0.394815] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1115worker1 # [ 0.408327] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1116client # [ 0.402411] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1117lb # [ 0.395179] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1118client # [ 0.403411] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1119worker2 # [ 0.390123] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1120client # [ 0.404392] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1121worker1 # [ 0.410812] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1122lb # [ 0.396663] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1123worker2 # [ 0.391025] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1124worker1 # [ 0.411654] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1125client # [ 0.405760] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1126worker2 # [ 0.391875] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1127worker1 # [ 0.412557] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1128lb # [ 0.399084] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1129worker2 # [ 0.392121] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1130worker1 # [ 0.414224] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1131lb # [ 0.399956] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1132client # [ 0.407412] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1133lb # [ 0.400755] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1134client # [ 0.408246] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1135worker2 # [ 0.393881] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1136client # [ 0.409145] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1137lb # [ 0.401636] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1138worker1 # [ 0.416815] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1139worker2 # [ 0.396142] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1140worker1 # [ 0.417806] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1141client # [ 0.409973] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1142worker2 # [ 0.397012] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1143worker1 # [ 0.418723] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1144worker2 # [ 0.397880] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1145client # [ 0.411439] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1146lb # [ 0.405012] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1147worker1 # [ 0.420188] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1148client # [ 0.412280] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1149worker2 # [ 0.398679] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1150lb # [ 0.405706] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1151client # [ 0.413127] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1152lb # [ 0.406075] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1153worker1 # [ 0.421811] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1154client # [ 0.413966] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1155worker1 # [ 0.422685] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1156lb # [ 0.407618] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1157worker1 # [ 0.423515] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1158worker1 # [ 0.423788] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1159worker2 # [ 0.402156] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1160worker2 # [ 0.403122] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1161lb # [ 0.410084] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1162client # [ 0.416434] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1163worker2 # [ 0.404055] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1164worker1 # [ 0.425782] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1165client # [ 0.417411] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1166lb # [ 0.410982] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1167client # [ 0.418340] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1168worker2 # [ 0.405586] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1169worker1 # [ 0.427794] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1170lb # [ 0.411782] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1171worker1 # [ 0.428740] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1172client # [ 0.419832] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1173lb # [ 0.412595] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1174worker2 # [ 0.407147] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1175worker1 # [ 0.430168] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1176client # [ 0.421436] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1177worker2 # [ 0.407975] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1178client # [ 0.422288] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1179lb # [ 0.414824] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1180worker2 # [ 0.409137] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1181worker1 # [ 0.431820] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1182client # [ 0.423134] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1183lb # [ 0.415060] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1184worker1 # [ 0.432657] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1185client # [ 0.423410] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1186lb # [ 0.415993] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1187worker2 # [ 0.410658] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1188worker1 # [ 0.433482] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1189lb # [ 0.417459] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1190client # [ 0.424949] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1191worker2 # [ 0.412143] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1192worker1 # [ 0.435153] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1193worker2 # [ 0.413049] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1194client # [ 0.427417] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1195worker2 # [ 0.413894] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1196client # [ 0.428382] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1197worker1 # [ 0.436811] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1198lb # [ 0.421089] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1199worker2 # [ 0.415612] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1200worker1 # [ 0.437631] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1201lb # [ 0.421917] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1202client # [ 0.429764] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1203worker1 # [ 0.438485] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1204lb # [ 0.422754] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1205worker2 # [ 0.418065] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1206worker1 # [ 0.439367] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1207client # [ 0.431439] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1208lb # [ 0.423683] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1209worker2 # [ 0.418763] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1210client # [ 0.432282] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1211worker1 # [ 0.440417] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1212worker2 # [ 0.419137] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1213client # [ 0.433131] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1214lb # [ 0.425084] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1215worker1 # [ 0.441579] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1216worker2 # [ 0.420722] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1217lb # [ 0.425928] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1218client # [ 0.433973] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1219worker1 # [ 0.443395] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1220worker2 # [ 0.422122] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1221lb # [ 0.426847] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1222worker2 # [ 0.423015] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1223client # [ 0.435434] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1224lb # [ 0.428059] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1225client # [ 0.436248] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1226worker2 # [ 0.423858] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1227client # [ 0.437116] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1228lb # [ 0.429655] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1229worker2 # [ 0.424120] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1230client # [ 0.437995] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1231lb # [ 0.431066] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1232worker2 # [ 0.425693] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1233lb # [ 0.432000] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1234client # [ 0.439026] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1235worker2 # [ 0.426853] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1236worker2 # [ 0.427138] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1237lb # [ 0.433442] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1238client # [ 0.439613] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1239client # [ 0.441078] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1240worker2 # [ 0.428674] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1241worker1 # [ 0.443972] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1242worker2 # [ 0.430154] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1243lb # [ 0.436033] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1244worker1 # [ 0.446395] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1245lb # [ 0.436766] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1246worker2 # [ 0.430995] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1247worker1 # [ 0.447074] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1248lb # [ 0.437077] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1249worker2 # [ 0.432137] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1250worker1 # [ 0.448376] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1251lb # [ 0.438818] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1252worker2 # [ 0.433693] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1253worker1 # [ 0.449021] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1254worker2 # [ 0.435144] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1255worker1 # [ 0.450069] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1256lb # [ 0.441061] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1257worker2 # [ 0.435980] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1258lb # [ 0.441889] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1259worker1 # [ 0.451028] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1260worker2 # [ 0.436841] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1261lb # [ 0.442772] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1262client # [ 0.441624] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1263worker1 # [ 0.452810] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1264worker2 # [ 0.437874] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1265worker1 # [ 0.453631] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1266lb # [ 0.444312] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1267client # [ 0.444054] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1268worker2 # [ 0.440168] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1269worker1 # [ 0.454798] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1270lb # [ 0.445674] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1271client # [ 0.444628] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1272worker2 # [ 0.441188] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1273lb # [ 0.446277] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1274worker1 # [ 0.456696] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1275client # [ 0.446019] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1276worker2 # [ 0.442609] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1277lb # [ 0.447733] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1278client # [ 0.447000] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1279client # [ 0.447714] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1280worker1 # [ 0.460353] ACPI: PCI: Interrupt link LNKA configured for IRQ 101281worker1 # [ 0.460890] ACPI: PCI: Interrupt link LNKB configured for IRQ 101282client # [ 0.448649] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1283worker1 # [ 0.461907] ACPI: PCI: Interrupt link LNKC configured for IRQ 111284client # [ 0.450456] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1285worker1 # [ 0.462880] ACPI: PCI: Interrupt link LNKD configured for IRQ 111286client # [ 0.451278] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1287worker1 # [ 0.463880] ACPI: PCI: Interrupt link LNKE configured for IRQ 101288worker1 # [ 0.464877] ACPI: PCI: Interrupt link LNKF configured for IRQ 101289client # [ 0.452431] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1290worker1 # [ 0.466005] ACPI: PCI: Interrupt link LNKG configured for IRQ 111291client # [ 0.454016] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1292worker1 # [ 0.466903] ACPI: PCI: Interrupt link LNKH configured for IRQ 111293worker1 # [ 0.467827] ACPI: PCI: Interrupt link GSIA configured for IRQ 161294worker1 # [ 0.468773] ACPI: PCI: Interrupt link GSIB configured for IRQ 171295worker2 # [ 0.443315] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1296worker1 # [ 0.469514] ACPI: PCI: Interrupt link GSIC configured for IRQ 181297lb # [ 0.448295] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1298worker2 # [ 0.445668] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1299client # [ 0.457369] ACPI: PCI: Interrupt link LNKA configured for IRQ 101300worker1 # [ 0.469808] ACPI: PCI: Interrupt link GSID configured for IRQ 191301lb # [ 0.450676] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1302worker1 # [ 0.470748] ACPI: PCI: Interrupt link GSIE configured for IRQ 201303client # [ 0.458224] ACPI: PCI: Interrupt link LNKB configured for IRQ 101304worker2 # [ 0.446383] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1305worker1 # [ 0.471503] ACPI: PCI: Interrupt link GSIF configured for IRQ 211306client # [ 0.459204] ACPI: PCI: Interrupt link LNKC configured for IRQ 111307lb # [ 0.451312] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1308worker1 # [ 0.471797] ACPI: PCI: Interrupt link GSIG configured for IRQ 221309client # [ 0.460224] ACPI: PCI: Interrupt link LNKD configured for IRQ 111310worker2 # [ 0.447617] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1311worker1 # [ 0.472736] ACPI: PCI: Interrupt link GSIH configured for IRQ 231312lb # [ 0.452599] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1313client # [ 0.461220] ACPI: PCI: Interrupt link LNKE configured for IRQ 101314worker2 # [ 0.448368] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1315client # [ 0.462216] ACPI: PCI: Interrupt link LNKF configured for IRQ 101316worker1 # [ 0.474525] iommu: Default domain type: Translated1317lb # [ 0.453345] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1318client # [ 0.463217] ACPI: PCI: Interrupt link LNKG configured for IRQ 111319worker2 # [ 0.449428] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1320worker1 # [ 0.474789] iommu: DMA domain TLB invalidation policy: lazy mode1321lb # [ 0.454359] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1322worker1 # [ 0.475950] ACPI: bus type USB registered1323client # [ 0.464430] ACPI: PCI: Interrupt link LNKH configured for IRQ 111324worker2 # [ 0.450371] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1325worker1 # [ 0.476662] usbcore: registered new interface driver usbfs1326client # [ 0.465446] ACPI: PCI: Interrupt link GSIA configured for IRQ 161327lb # [ 0.455335] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1328worker1 # [ 0.477422] usbcore: registered new interface driver hub1329client # [ 0.466410] ACPI: PCI: Interrupt link GSIB configured for IRQ 171330worker2 # [ 0.451872] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1331worker1 # [ 0.477792] usbcore: registered new device driver usb1332client # [ 0.467393] ACPI: PCI: Interrupt link GSIC configured for IRQ 181333worker2 # [ 0.452121] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1334lb # [ 0.457082] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1335client # [ 0.468138] ACPI: PCI: Interrupt link GSID configured for IRQ 191336worker1 # [ 0.479489] NetLabel: Initializing1337lb # [ 0.458061] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1338worker1 # [ 0.479783] NetLabel: domain hash size = 1281339worker2 # [ 0.453409] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1340client # [ 0.468420] ACPI: PCI: Interrupt link GSIE configured for IRQ 201341worker1 # [ 0.480466] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1342client # [ 0.469356] ACPI: PCI: Interrupt link GSIF configured for IRQ 211343lb # [ 0.459351] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1344worker2 # [ 0.454967] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1345worker1 # [ 0.480817] NetLabel: unlabeled traffic allowed by default1346client # [ 0.470151] ACPI: PCI: Interrupt link GSIG configured for IRQ 221347worker1 # [ 0.481696] PCI: Using ACPI for IRQ routing1348client # [ 0.470418] ACPI: PCI: Interrupt link GSIH configured for IRQ 231349lb # [ 0.460663] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1350client # [ 0.472423] iommu: Default domain type: Translated1351worker2 # [ 0.458708] ACPI: PCI: Interrupt link LNKA configured for IRQ 101352client # [ 0.473232] iommu: DMA domain TLB invalidation policy: lazy mode1353client # [ 0.474385] ACPI: bus type USB registered1354worker2 # [ 0.459231] ACPI: PCI: Interrupt link LNKB configured for IRQ 101355lb # [ 0.464338] ACPI: PCI: Interrupt link LNKA configured for IRQ 101356client # [ 0.474950] usbcore: registered new interface driver usbfs1357worker2 # [ 0.460213] ACPI: PCI: Interrupt link LNKC configured for IRQ 111358lb # [ 0.465160] ACPI: PCI: Interrupt link LNKB configured for IRQ 101359client # [ 0.475416] usbcore: registered new interface driver hub1360worker2 # [ 0.461219] ACPI: PCI: Interrupt link LNKD configured for IRQ 111361client # [ 0.476241] usbcore: registered new device driver usb1362lb # [ 0.466153] ACPI: PCI: Interrupt link LNKC configured for IRQ 111363worker2 # [ 0.462216] ACPI: PCI: Interrupt link LNKE configured for IRQ 101364lb # [ 0.467158] ACPI: PCI: Interrupt link LNKD configured for IRQ 111365client # [ 0.477898] NetLabel: Initializing1366worker2 # [ 0.463211] ACPI: PCI: Interrupt link LNKF configured for IRQ 101367client # [ 0.478405] NetLabel: domain hash size = 1281368lb # [ 0.468161] ACPI: PCI: Interrupt link LNKE configured for IRQ 101369worker2 # [ 0.464215] ACPI: PCI: Interrupt link LNKG configured for IRQ 111370client # [ 0.479127] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1371lb # [ 0.469156] ACPI: PCI: Interrupt link LNKF configured for IRQ 101372client # [ 0.479442] NetLabel: unlabeled traffic allowed by default1373worker2 # [ 0.465214] ACPI: PCI: Interrupt link LNKH configured for IRQ 111374client # [ 0.480304] PCI: Using ACPI for IRQ routing1375lb # [ 0.470189] ACPI: PCI: Interrupt link LNKG configured for IRQ 111376worker2 # [ 0.466151] ACPI: PCI: Interrupt link GSIA configured for IRQ 161377lb # [ 0.471165] ACPI: PCI: Interrupt link LNKH configured for IRQ 111378worker2 # [ 0.467130] ACPI: PCI: Interrupt link GSIB configured for IRQ 171379lb # [ 0.472096] ACPI: PCI: Interrupt link GSIA configured for IRQ 161380worker2 # [ 0.468121] ACPI: PCI: Interrupt link GSIC configured for IRQ 181381lb # [ 0.473069] ACPI: PCI: Interrupt link GSIB configured for IRQ 171382worker2 # [ 0.469108] ACPI: PCI: Interrupt link GSID configured for IRQ 191383lb # [ 0.474066] ACPI: PCI: Interrupt link GSIC configured for IRQ 181384worker2 # [ 0.470083] ACPI: PCI: Interrupt link GSIE configured for IRQ 201385worker2 # [ 0.470911] ACPI: PCI: Interrupt link GSIF configured for IRQ 211386lb # [ 0.475066] ACPI: PCI: Interrupt link GSID configured for IRQ 191387worker2 # [ 0.471890] ACPI: PCI: Interrupt link GSIG configured for IRQ 221388lb # [ 0.476067] ACPI: PCI: Interrupt link GSIE configured for IRQ 201389worker2 # [ 0.472130] ACPI: PCI: Interrupt link GSIH configured for IRQ 231390lb # [ 0.477065] ACPI: PCI: Interrupt link GSIF configured for IRQ 211391lb # [ 0.478067] ACPI: PCI: Interrupt link GSIG configured for IRQ 221392worker2 # [ 0.474147] iommu: Default domain type: Translated1393lb # [ 0.479068] ACPI: PCI: Interrupt link GSIH configured for IRQ 231394worker2 # [ 0.474975] iommu: DMA domain TLB invalidation policy: lazy mode1395worker2 # [ 0.476159] ACPI: bus type USB registered1396lb # [ 0.481469] iommu: Default domain type: Translated1397worker2 # [ 0.476942] usbcore: registered new interface driver usbfs1398lb # [ 0.482070] iommu: DMA domain TLB invalidation policy: lazy mode1399worker2 # [ 0.477835] usbcore: registered new interface driver hub1400lb # [ 0.483359] ACPI: bus type USB registered1401worker2 # [ 0.478126] usbcore: registered new device driver usb1402lb # [ 0.484129] usbcore: registered new interface driver usbfs1403worker2 # [ 0.479844] NetLabel: Initializing1404lb # [ 0.485236] usbcore: registered new interface driver hub1405worker2 # [ 0.480121] NetLabel: domain hash size = 1281406lb # [ 0.486071] usbcore: registered new device driver usb1407worker2 # [ 0.480859] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1408worker2 # [ 0.481164] NetLabel: unlabeled traffic allowed by default1409lb # [ 0.488017] NetLabel: Initializing1410worker2 # [ 0.482127] PCI: Using ACPI for IRQ routing1411lb # [ 0.488534] NetLabel: domain hash size = 1281412lb # [ 0.489053] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413lb # [ 0.490059] NetLabel: unlabeled traffic allowed by default1414lb # [ 0.491026] PCI: Using ACPI for IRQ routing1415worker1 # [ 0.525545] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416worker1 # [ 0.525778] pci 0000:00:01.0: vgaarb: bridge control possible1417worker1 # [ 0.525778] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418worker1 # [ 0.525786] vgaarb: loaded1419worker1 # [ 0.526492] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01420worker1 # [ 0.526783] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1421worker1 # [ 0.530870] clocksource: Switched to clocksource kvm-clock1422client # [ 0.524872] pci 0000:00:01.0: vgaarb: setting as boot VGA device1423client # [ 0.525400] pci 0000:00:01.0: vgaarb: bridge control possible1424worker1 # [ 0.534390] VFS: Disk quotas dquot_6.6.01425client # [ 0.525400] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1426worker1 # [ 0.535089] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1427client # [ 0.525408] vgaarb: loaded1428worker1 # [ 0.536362] pnp: PnP ACPI init1429client # [ 0.526106] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01430worker1 # [ 0.537117] ACPI: IRQ 4 override to edge(!), high(!)1431client # [ 0.526405] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1432worker1 # [ 0.538058] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1433worker1 # [ 0.539562] pnp: PnP ACPI: found 6 devices1434client # [ 0.529489] clocksource: Switched to clocksource kvm-clock1435client # [ 0.532937] VFS: Disk quotas dquot_6.6.01436client # [ 0.533642] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1437client # [ 0.534957] pnp: PnP ACPI init1438client # [ 0.535749] ACPI: IRQ 4 override to edge(!), high(!)1439worker1 # [ 0.547114] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1440worker1 # [ 0.548564] clocksource: Switched to clocksource acpi_pm1441client # [ 0.536702] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1442worker1 # [ 0.549579] NET: Registered PF_INET protocol family1443client # [ 0.538190] pnp: PnP ACPI: found 6 devices1444worker2 # [ 0.525712] pci 0000:00:01.0: vgaarb: setting as boot VGA device1445worker1 # [ 0.550832] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1446worker2 # [ 0.526111] pci 0000:00:01.0: vgaarb: bridge control possible1447worker2 # [ 0.526111] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1448worker2 # [ 0.526119] vgaarb: loaded1449worker2 # [ 0.526871] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01450worker2 # [ 0.527117] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1451lb # [ 0.537110] pci 0000:00:01.0: vgaarb: setting as boot VGA device1452lb # [ 0.538049] pci 0000:00:01.0: vgaarb: bridge control possible1453client # [ 0.545765] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1454client # [ 0.547171] clocksource: Switched to clocksource acpi_pm1455lb # [ 0.538049] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1456worker2 # [ 0.530209] clocksource: Switched to clocksource kvm-clock1457lb # [ 0.538063] vgaarb: loaded1458client # [ 0.548134] NET: Registered PF_INET protocol family1459lb # [ 0.538854] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01460client # [ 0.549282] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1461lb # [ 0.539702] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1462worker2 # [ 0.533650] VFS: Disk quotas dquot_6.6.01463worker2 # [ 0.534392] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1464worker2 # [ 0.535755] pnp: PnP ACPI init1465lb # [ 0.543171] clocksource: Switched to clocksource kvm-clock1466worker2 # [ 0.536560] ACPI: IRQ 4 override to edge(!), high(!)1467worker2 # [ 0.537545] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1468worker1 # [ 0.567246] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1469worker2 # [ 0.539072] pnp: PnP ACPI: found 6 devices1470worker1 # [ 0.568825] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1471lb # [ 0.547126] VFS: Disk quotas dquot_6.6.01472worker1 # [ 0.570168] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1473lb # [ 0.547882] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1474worker1 # [ 0.571528] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1475lb # [ 0.549590] pnp: PnP ACPI init1476worker1 # [ 0.572836] TCP: Hash tables configured (established 16384 bind 16384)1477lb # [ 0.550531] ACPI: IRQ 4 override to edge(!), high(!)1478worker1 # [ 0.574007] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1479lb # [ 0.551584] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1480worker1 # [ 0.575311] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1481lb # [ 0.553213] pnp: PnP ACPI: found 6 devices1482worker1 # [ 0.576436] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1483worker2 # [ 0.546624] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1484worker1 # [ 0.577638] NET: Registered PF_UNIX/PF_LOCAL protocol family1485worker2 # [ 0.548058] clocksource: Switched to clocksource acpi_pm1486client # [ 0.565788] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1487worker1 # [ 0.578628] NET: Registered PF_XDP protocol family1488worker2 # [ 0.549024] NET: Registered PF_INET protocol family1489worker1 # [ 0.579486] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1490client # [ 0.567281] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1491worker2 # [ 0.550191] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1492worker1 # [ 0.580516] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1493client # [ 0.568565] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1494worker1 # [ 0.581527] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1495client # [ 0.569863] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1496worker1 # [ 0.582660] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1497client # [ 0.571091] TCP: Hash tables configured (established 16384 bind 16384)1498worker1 # [ 0.583772] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1499client # [ 0.572183] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1500lb # [ 0.560981] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1501worker1 # [ 0.584873] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1502lb # [ 0.562418] clocksource: Switched to clocksource acpi_pm1503client # [ 0.573364] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1504worker1 # [ 0.586504] ACPI: \_SB_.GSIA: Enabled at IRQ 161505lb # [ 0.563424] NET: Registered PF_INET protocol family1506client # [ 0.574386] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1507client # [ 0.575595] NET: Registered PF_UNIX/PF_LOCAL protocol family1508lb # [ 0.564641] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1509client # [ 0.576554] NET: Registered PF_XDP protocol family1510worker1 # [ 0.588737] ACPI: \_SB_.GSIB: Enabled at IRQ 171511client # [ 0.577335] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1512client # [ 0.578292] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1513worker1 # [ 0.590792] ACPI: \_SB_.GSIC: Enabled at IRQ 181514client # [ 0.579246] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1515client # [ 0.580289] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1516worker1 # [ 0.592752] ACPI: \_SB_.GSID: Enabled at IRQ 191517client # [ 0.581324] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1518worker1 # [ 0.594373] PCI: CLS 0 bytes, default 641519client # [ 0.582364] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1520client # [ 0.584089] ACPI: \_SB_.GSIA: Enabled at IRQ 161521worker1 # [ 0.595277] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1522worker2 # [ 0.566800] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1523worker1 # [ 0.597467] Trying to unpack rootfs image as initramfs...1524client # [ 0.586267] ACPI: \_SB_.GSIB: Enabled at IRQ 171525worker2 # [ 0.568360] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1526worker2 # [ 0.569674] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1527client # [ 0.588639] ACPI: \_SB_.GSIC: Enabled at IRQ 181528worker2 # [ 0.571070] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1529worker2 # [ 0.572364] TCP: Hash tables configured (established 16384 bind 16384)1530client # [ 0.590608] ACPI: \_SB_.GSID: Enabled at IRQ 191531worker2 # [ 0.573510] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1532client # [ 0.592306] PCI: CLS 0 bytes, default 641533worker2 # [ 0.574818] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1534lb # [ 0.582001] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1535client # [ 0.593226] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1536worker2 # [ 0.575921] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1537client # [ 0.594886] Trying to unpack rootfs image as initramfs...1538worker2 # [ 0.577116] NET: Registered PF_UNIX/PF_LOCAL protocol family1539lb # [ 0.583553] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1540worker2 # [ 0.578065] NET: Registered PF_XDP protocol family1541lb # [ 0.584927] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1542worker2 # [ 0.578924] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1543lb # [ 0.586307] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1544worker2 # [ 0.579964] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1545lb # [ 0.587621] TCP: Hash tables configured (established 16384 bind 16384)1546worker2 # [ 0.580987] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1547worker2 # [ 0.582117] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1548lb # [ 0.588776] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1549worker2 # [ 0.583257] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1550lb # [ 0.590188] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1551worker2 # [ 0.584372] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1552lb # [ 0.591306] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1553worker2 # [ 0.586059] ACPI: \_SB_.GSIA: Enabled at IRQ 161554lb # [ 0.592616] NET: Registered PF_UNIX/PF_LOCAL protocol family1555lb # [ 0.593665] NET: Registered PF_XDP protocol family1556lb # [ 0.594545] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1557worker2 # [ 0.588312] ACPI: \_SB_.GSIB: Enabled at IRQ 171558lb # [ 0.595593] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1559lb # [ 0.596646] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1560worker2 # [ 0.590456] ACPI: \_SB_.GSIC: Enabled at IRQ 181561lb # [ 0.597768] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1562worker2 # [ 0.592393] ACPI: \_SB_.GSID: Enabled at IRQ 191563lb # [ 0.598974] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1564lb # [ 0.600111] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1565worker2 # [ 0.593987] PCI: CLS 0 bytes, default 641566lb # [ 0.601928] ACPI: \_SB_.GSIA: Enabled at IRQ 161567worker2 # [ 0.594901] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1568worker2 # [ 0.596709] Trying to unpack rootfs image as initramfs...1569lb # [ 0.605247] ACPI: \_SB_.GSIB: Enabled at IRQ 171570lb # [ 0.607263] ACPI: \_SB_.GSIC: Enabled at IRQ 181571lb # [ 0.609243] ACPI: \_SB_.GSID: Enabled at IRQ 191572lb # [ 0.610952] PCI: CLS 0 bytes, default 641573lb # [ 0.611863] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1574lb # [ 0.613676] Trying to unpack rootfs image as initramfs...1575worker1 # [ 0.640233] Initialise system trusted keyrings1576worker1 # [ 0.643783] workingset: timestamp_bits=40 max_order=19 bucket_order=01577client # [ 0.640876] Initialise system trusted keyrings1578client # [ 0.641814] workingset: timestamp_bits=40 max_order=19 bucket_order=01579worker1 # [ 0.663086] Key type asymmetric registered1580worker1 # [ 0.666725] Asymmetric key parser 'x509' registered1581worker2 # [ 0.637413] Initialise system trusted keyrings1582worker1 # [ 0.667636] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1583worker2 # [ 0.640299] workingset: timestamp_bits=40 max_order=19 bucket_order=01584worker1 # [ 0.670954] io scheduler mq-deadline registered1585client # [ 0.661425] Key type asymmetric registered1586worker1 # [ 0.673740] io scheduler kyber registered1587worker1 # [ 0.676767] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1588client # [ 0.666521] Asymmetric key parser 'x509' registered1589worker1 # [ 0.678130] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1590client # [ 0.667328] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1591client # [ 0.669619] io scheduler mq-deadline registered1592client # [ 0.670342] io scheduler kyber registered1593worker1 # [ 0.683920] Linux agpgart interface v0.1031594lb # [ 0.660942] Initialise system trusted keyrings1595worker1 # [ 0.684738] ACPI: bus type drm_connector registered1596client # [ 0.674007] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1597client # [ 0.675252] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1598lb # [ 0.664510] workingset: timestamp_bits=40 max_order=19 bucket_order=01599worker1 # [ 0.688057] usbcore: registered new interface driver usbserial_generic1600worker1 # [ 0.689107] usbserial: USB Serial support registered for generic1601worker2 # [ 0.661214] Key type asymmetric registered1602client # [ 0.679260] Linux agpgart interface v0.1031603client # [ 0.680002] ACPI: bus type drm_connector registered1604worker1 # [ 0.692730] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1605worker2 # [ 0.664268] Asymmetric key parser 'x509' registered1606worker1 # [ 0.692730] Please enable it if your BIOS has the CPPC option.1607client # [ 0.682944] usbcore: registered new interface driver usbserial_generic1608worker2 # [ 0.665114] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1609worker1 # [ 0.694928] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1610client # [ 0.683970] usbserial: USB Serial support registered for generic1611worker2 # [ 0.668376] io scheduler mq-deadline registered1612worker2 # [ 0.669122] io scheduler kyber registered1613worker1 # [ 0.698941] drop_monitor: Initializing network drop monitor service1614client # [ 0.687485] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1615worker1 # [ 0.700133] NET: Registered PF_INET6 protocol family1616client # [ 0.687485] Please enable it if your BIOS has the CPPC option.1617worker2 # [ 0.671778] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1618client # [ 0.689614] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1619worker1 # [ 0.702874] Segment Routing with IPv61620worker2 # [ 0.673048] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1621worker1 # [ 0.703531] In-situ OAM (IOAM) with IPv61622client # [ 0.692607] drop_monitor: Initializing network drop monitor service1623client # [ 0.693740] NET: Registered PF_INET6 protocol family1624lb # [ 0.683847] Key type asymmetric registered1625worker1 # [ 0.707141] IPI shorthand broadcast: enabled1626worker2 # [ 0.678668] Linux agpgart interface v0.1031627client # [ 0.696942] Segment Routing with IPv61628worker2 # [ 0.679486] ACPI: bus type drm_connector registered1629client # [ 0.699507] In-situ OAM (IOAM) with IPv61630lb # [ 0.688488] Asymmetric key parser 'x509' registered1631client # [ 0.700512] IPI shorthand broadcast: enabled1632worker2 # [ 0.682702] usbcore: registered new interface driver usbserial_generic1633lb # [ 0.689417] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1634worker2 # [ 0.683724] usbserial: USB Serial support registered for generic1635lb # [ 0.692658] io scheduler mq-deadline registered1636worker1 # [ 0.715443] sched_clock: Marking stable (582014651, 132699047)->(786201128, -71487430)1637lb # [ 0.693417] io scheduler kyber registered1638worker2 # [ 0.687270] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1639worker2 # [ 0.687270] Please enable it if your BIOS has the CPPC option.1640worker1 # [ 0.718943] registered taskstats version 11641worker2 # [ 0.689526] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1642worker1 # [ 0.719939] Loading compiled-in X.509 certificates1643lb # [ 0.697109] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1644client # [ 0.708716] sched_clock: Marking stable (575013935, 133460903)->(783486112, -75011274)1645lb # [ 0.698509] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1646worker2 # [ 0.692386] drop_monitor: Initializing network drop monitor service1647worker2 # [ 0.693558] NET: Registered PF_INET6 protocol family1648lb # [ 0.702137] Linux agpgart interface v0.1031649client # [ 0.713726] registered taskstats version 11650lb # [ 0.702973] ACPI: bus type drm_connector registered1651worker2 # [ 0.696694] Segment Routing with IPv61652client # [ 0.714675] Loading compiled-in X.509 certificates1653worker2 # [ 0.699302] In-situ OAM (IOAM) with IPv61654lb # [ 0.705936] usbcore: registered new interface driver usbserial_generic1655worker2 # [ 0.700344] IPI shorthand broadcast: enabled1656lb # [ 0.707025] usbserial: USB Serial support registered for generic1657lb # [ 0.710469] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1658lb # [ 0.710469] Please enable it if your BIOS has the CPPC option.1659lb # [ 0.712772] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1660worker1 # [ 0.738732] Demotion targets for Node 0: null1661worker2 # [ 0.708604] sched_clock: Marking stable (575014346, 133233044)->(784385798, -76138408)1662worker1 # [ 0.739629] Key type .fscrypt registered1663lb # [ 0.716667] drop_monitor: Initializing network drop monitor service1664lb # [ 0.717877] NET: Registered PF_INET6 protocol family1665worker1 # [ 0.741728] Key type fscrypt-provisioning registered1666worker2 # [ 0.712339] registered taskstats version 11667worker1 # [ 0.742654] ima: No TPM chip found, activating TPM-bypass!1668client # [ 0.731034] Demotion targets for Node 0: null1669worker2 # [ 0.713309] Loading compiled-in X.509 certificates1670worker1 # [ 0.743551] ima: Allocated hash algorithm: sha11671lb # [ 0.722576] Segment Routing with IPv61672worker1 # [ 0.745759] ima: No architecture policies found1673lb # [ 0.723278] In-situ OAM (IOAM) with IPv61674client # [ 0.734537] Key type .fscrypt registered1675client # [ 0.735181] Key type fscrypt-provisioning registered1676client # [ 0.736103] ima: No TPM chip found, activating TPM-bypass!1677worker1 # [ 0.748735] PM: Magic number: 10:966:7801678lb # [ 0.725870] IPI shorthand broadcast: enabled1679client # [ 0.738488] ima: Allocated hash algorithm: sha11680worker1 # [ 0.750507] RAS: Correctable Errors collector initialized.1681client # [ 0.739239] ima: No architecture policies found1682client # [ 0.741493] PM: Magic number: 10:966:7801683client # [ 0.743070] RAS: Correctable Errors collector initialized.1684lb # [ 0.734208] sched_clock: Marking stable (600013733, 133430063)->(808611797, -75168001)1685worker1 # [ 0.760460] clk: Disabling unused clocks1686lb # [ 0.737669] registered taskstats version 11687lb # [ 0.738668] Loading compiled-in X.509 certificates1688worker1 # [ 0.762736] PM: genpd: Disabling unused power domains1689worker2 # [ 0.733262] Demotion targets for Node 0: null1690worker2 # [ 0.734122] Key type .fscrypt registered1691client # [ 0.752145] clk: Disabling unused clocks1692worker2 # [ 0.737260] Key type fscrypt-provisioning registered1693worker2 # [ 0.738217] ima: No TPM chip found, activating TPM-bypass!1694client # [ 0.756494] PM: genpd: Disabling unused power domains1695worker2 # [ 0.739159] ima: Allocated hash algorithm: sha11696worker2 # [ 0.739985] ima: No architecture policies found1697worker2 # [ 0.742259] PM: Magic number: 10:966:7801698worker2 # [ 0.743869] RAS: Correctable Errors collector initialized.1699lb # [ 0.758460] Demotion targets for Node 0: null1700lb # [ 0.759420] Key type .fscrypt registered1701lb # [ 0.760194] Key type fscrypt-provisioning registered1702worker2 # [ 0.753900] clk: Disabling unused clocks1703lb # [ 0.761209] ima: No TPM chip found, activating TPM-bypass!1704worker2 # [ 0.756260] PM: genpd: Disabling unused power domains1705lb # [ 0.763465] ima: Allocated hash algorithm: sha11706lb # [ 0.764322] ima: No architecture policies found1707lb # [ 0.768469] PM: Magic number: 10:966:7801708lb # [ 0.772473] RAS: Correctable Errors collector initialized.1709lb # [ 0.781532] clk: Disabling unused clocks1710lb # [ 0.782287] PM: genpd: Disabling unused power domains1711client # [ 0.916586] Freeing initrd memory: 29096K1712client # [ 0.919679] Freeing unused decrypted memory: 2028K1713client # [ 0.922250] Freeing unused kernel image (initmem) memory: 3644K1714client # [ 0.923303] Write protecting the kernel read-only data: 32768k1715client # [ 0.925121] Freeing unused kernel image (text/rodata gap) memory: 1208K1716client # [ 0.926615] Freeing unused kernel image (rodata/data gap) memory: 752K1717worker1 # [ 0.941380] Freeing initrd memory: 29092K1718worker1 # [ 0.944963] Freeing unused decrypted memory: 2028K1719worker1 # [ 0.948054] Freeing unused kernel image (initmem) memory: 3644K1720worker1 # [ 0.949136] Write protecting the kernel read-only data: 32768k1721worker1 # [ 0.951172] Freeing unused kernel image (text/rodata gap) memory: 1208K1722worker1 # [ 0.952822] Freeing unused kernel image (rodata/data gap) memory: 752K1723worker2 # [ 0.925741] Freeing initrd memory: 29092K1724worker2 # [ 0.928856] Freeing unused decrypted memory: 2028K1725worker2 # [ 0.931516] Freeing unused kernel image (initmem) memory: 3644K1726worker2 # [ 0.932625] Write protecting the kernel read-only data: 32768k1727worker2 # [ 0.934531] Freeing unused kernel image (text/rodata gap) memory: 1208K1728worker2 # [ 0.936060] Freeing unused kernel image (rodata/data gap) memory: 752K1729lb # [ 0.957055] Freeing initrd memory: 29092K1730lb # [ 0.960567] Freeing unused decrypted memory: 2028K1731lb # [ 0.963564] Freeing unused kernel image (initmem) memory: 3644K1732lb # [ 0.964744] Write protecting the kernel read-only data: 32768k1733client # [ 0.977735] x86/mm: Checked W+X mappings: passed, no W+X pages found.1734lb # [ 0.966941] Freeing unused kernel image (text/rodata gap) memory: 1208K1735client # [ 0.978802] Run /init as init process1736lb # [ 0.968560] Freeing unused kernel image (rodata/data gap) memory: 752K1737client # [ 0.988754] systemd[1]: Inserted module 'autofs4'1738worker1 # [ 1.003949] x86/mm: Checked W+X mappings: passed, no W+X pages found.1739worker1 # [ 1.005012] Run /init as init process1740worker1 # [ 1.016034] systemd[1]: Inserted module 'autofs4'1741client # [ 1.005034] fuse: init (API version 7.45)1742worker2 # [ 0.987047] x86/mm: Checked W+X mappings: passed, no W+X pages found.1743worker2 # [ 0.988187] Run /init as init process1744client # [ 1.010823] ACPI: \_SB_.GSIG: Enabled at IRQ 221745client # [ 1.013200] ACPI: \_SB_.GSIH: Enabled at IRQ 231746worker2 # [ 0.998086] systemd[1]: Inserted module 'autofs4'1747client # [ 1.016061] ACPI: \_SB_.GSIE: Enabled at IRQ 201748client # [ 1.018249] ACPI: \_SB_.GSIF: Enabled at IRQ 211749worker1 # [ 1.033019] fuse: init (API version 7.45)1750worker1 # [ 1.038976] ACPI: \_SB_.GSIG: Enabled at IRQ 221751worker1 # [ 1.041815] ACPI: \_SB_.GSIH: Enabled at IRQ 231752worker2 # [ 1.014265] fuse: init (API version 7.45)1753lb # [ 1.020610] x86/mm: Checked W+X mappings: passed, no W+X pages found.1754lb # [ 1.021781] Run /init as init process1755worker1 # [ 1.045091] ACPI: \_SB_.GSIE: Enabled at IRQ 201756worker1 # [ 1.047516] ACPI: \_SB_.GSIF: Enabled at IRQ 211757worker2 # [ 1.019989] ACPI: \_SB_.GSIG: Enabled at IRQ 221758worker2 # [ 1.022331] ACPI: \_SB_.GSIH: Enabled at IRQ 231759worker2 # [ 1.025206] ACPI: \_SB_.GSIE: Enabled at IRQ 201760lb # [ 1.032811] systemd[1]: Inserted module 'autofs4'1761worker2 # [ 1.027443] ACPI: \_SB_.GSIF: Enabled at IRQ 211762client # [ 1.046685] systemd[1]: Successfully made /usr/ read-only.1763lb # [ 1.051598] fuse: init (API version 7.45)1764worker1 # [ 1.077228] systemd[1]: Successfully made /usr/ read-only.1765lb # [ 1.058281] ACPI: \_SB_.GSIG: Enabled at IRQ 221766worker2 # [ 1.051948] systemd[1]: Successfully made /usr/ read-only.1767lb # [ 1.060920] ACPI: \_SB_.GSIH: Enabled at IRQ 231768lb # [ 1.064130] ACPI: \_SB_.GSIE: Enabled at IRQ 201769lb # [ 1.066619] ACPI: \_SB_.GSIF: Enabled at IRQ 211770lb # [ 1.094325] systemd[1]: Successfully made /usr/ read-only.1771client # [ 1.383383] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1772client # [ 1.395284] systemd[1]: Detected virtualization kvm.1773client # [ 1.397419] systemd[1]: Detected architecture x86-64.1774client # [ 1.399606] systemd[1]: Running in initrd.1775client # [ 1.402249] systemd[1]: Initializing machine ID from random generator.1776worker1 # [ 1.413090] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1777worker1 # [ 1.418129] systemd[1]: Detected virtualization kvm.1778client # [ 1.405142] systemd[1]: Hostname set to <client>.1779worker1 # [ 1.418965] systemd[1]: Detected architecture x86-64.1780worker1 # [ 1.419865] systemd[1]: Running in initrd.1781worker1 # [ 1.421013] systemd[1]: Initializing machine ID from random generator.1782worker1 # [ 1.422255] systemd[1]: Hostname set to <worker1>.1783worker2 # [ 1.387366] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1784worker2 # [ 1.396556] systemd[1]: Detected virtualization kvm.1785worker2 # [ 1.398735] systemd[1]: Detected architecture x86-64.1786worker2 # [ 1.400938] systemd[1]: Running in initrd.1787worker2 # [ 1.403585] systemd[1]: Initializing machine ID from random generator.1788worker2 # [ 1.406594] systemd[1]: Hostname set to <worker2>.1789lb # [ 1.430888] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1790lb # [ 1.442701] systemd[1]: Detected virtualization kvm.1791lb # [ 1.444755] systemd[1]: Detected architecture x86-64.1792lb # [ 1.446836] systemd[1]: Running in initrd.1793lb # [ 1.449322] systemd[1]: Initializing machine ID from random generator.1794lb # [ 1.452175] systemd[1]: Hostname set to <lb>.1795client # [ 1.613164] systemd[1]: bpf-restrict-fs: LSM BPF program attached1796worker1 # [ 1.627514] systemd[1]: bpf-restrict-fs: LSM BPF program attached1797worker2 # [ 1.634131] systemd[1]: bpf-restrict-fs: LSM BPF program attached1798client # [ 1.653382] systemd[1]: Queued start job for default target Initrd Default Target.1799client # [ 1.657868] systemd[1]: Created slice Slice /system/modprobe.1800client # [ 1.659077] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1801client # [ 1.660561] systemd[1]: Expecting device /dev/disk/by-label/nixos...1802client # [ 1.661706] systemd[1]: Reached target Path Units.1803client # [ 1.662617] systemd[1]: Reached target Slice Units.1804client # [ 1.663540] systemd[1]: Reached target Swaps.1805client # [ 1.664333] systemd[1]: Reached target Timer Units.1806client # [ 1.665345] systemd[1]: Listening on D-Bus System Message Bus Socket.1807client # [ 1.666666] systemd[1]: Listening on Journal Socket (/dev/log).1808client # [ 1.667852] systemd[1]: Listening on Journal Sockets.1809client # [ 1.668896] systemd[1]: Listening on udev Control Socket.1810worker1 # [ 1.680357] systemd[1]: Queued start job for default target Initrd Default Target.1811client # [ 1.669915] systemd[1]: Listening on udev Kernel Socket.1812client # [ 1.670906] systemd[1]: Reached target Socket Units.1813client # [ 1.672806] systemd[1]: Starting Create List of Static Device Nodes...1814worker1 # [ 1.685107] systemd[1]: Created slice Slice /system/modprobe.1815worker1 # [ 1.686292] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1816worker1 # [ 1.687737] systemd[1]: Expecting device /dev/disk/by-label/nixos...1817client # [ 1.676547] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1818worker1 # [ 1.688874] systemd[1]: Reached target Path Units.1819worker1 # [ 1.689692] systemd[1]: Reached target Slice Units.1820worker1 # [ 1.690594] systemd[1]: Reached target Swaps.1821worker1 # [ 1.691562] systemd[1]: Reached target Timer Units.1822worker1 # [ 1.692676] systemd[1]: Listening on D-Bus System Message Bus Socket.1823worker1 # [ 1.694012] systemd[1]: Listening on Journal Socket (/dev/log).1824worker1 # [ 1.695200] systemd[1]: Listening on Journal Sockets.1825worker1 # [ 1.696214] systemd[1]: Listening on udev Control Socket.1826client # [ 1.684550] systemd[1]: Starting Load Kernel Module configfs...1827worker1 # [ 1.697261] systemd[1]: Listening on udev Kernel Socket.1828worker1 # [ 1.698211] systemd[1]: Reached target Socket Units.1829worker1 # [ 1.700214] systemd[1]: Starting Create List of Static Device Nodes...1830lb # [ 1.677933] systemd[1]: bpf-restrict-fs: LSM BPF program attached1831worker1 # [ 1.705831] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1832client # [ 1.698660] systemd[1]: Starting Journal Service...1833worker1 # [ 1.713988] systemd[1]: Starting Load Kernel Module configfs...1834worker2 # [ 1.690159] systemd[1]: Queued start job for default target Initrd Default Target.1835client # [ 1.711804] systemd[1]: Starting Load Kernel Modules...1836worker2 # [ 1.694551] systemd[1]: Created slice Slice /system/modprobe.1837worker2 # [ 1.695712] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1838worker2 # [ 1.697153] systemd[1]: Expecting device /dev/disk/by-label/nixos...1839worker2 # [ 1.698308] systemd[1]: Reached target Path Units.1840worker2 # [ 1.699181] systemd[1]: Reached target Slice Units.1841worker2 # [ 1.700099] systemd[1]: Reached target Swaps.1842worker2 # [ 1.700944] systemd[1]: Reached target Timer Units.1843worker1 # [ 1.730816] systemd[1]: Starting Journal Service...1844worker2 # [ 1.701918] systemd[1]: Listening on D-Bus System Message Bus Socket.1845worker2 # [ 1.703164] systemd[1]: Listening on Journal Socket (/dev/log).1846worker2 # [ 1.704385] systemd[1]: Listening on Journal Sockets.1847worker2 # [ 1.705421] systemd[1]: Listening on udev Control Socket.1848worker2 # [ 1.706523] systemd[1]: Listening on udev Kernel Socket.1849worker2 # [ 1.707514] systemd[1]: Reached target Socket Units.1850worker2 # [ 1.709309] systemd[1]: Starting Create List of Static Device Nodes...1851client # [ 1.718573] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1852worker2 # [ 1.713056] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1853worker1 # [ 1.746224] systemd[1]: Starting Load Kernel Modules...1854client # [ 1.736451] netfs: FS-Cache loaded1855worker2 # [ 1.719927] systemd[1]: Starting Load Kernel Module configfs...1856client # [ 1.741385] 9pnet: Installing 9P2000 support1857lb # [ 1.733962] systemd[1]: Queued start job for default target Initrd Default Target.1858lb # [ 1.737899] systemd[1]: Created slice Slice /system/modprobe.1859lb # [ 1.739141] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1860client # [ 1.751240] systemd[1]: Starting Coldplug All udev Devices...1861worker1 # [ 1.751440] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1862lb # [ 1.740597] systemd[1]: Expecting device /dev/disk/by-label/nixos...1863worker2 # [ 1.734325] systemd[1]: Starting Journal Service...1864lb # [ 1.741674] systemd[1]: Reached target Path Units.1865lb # [ 1.742578] systemd[1]: Reached target Slice Units.1866lb # [ 1.743497] systemd[1]: Reached target Swaps.1867worker1 # [ 1.767350] netfs: FS-Cache loaded1868lb # [ 1.744313] systemd[1]: Reached target Timer Units.1869lb # [ 1.745326] systemd[1]: Listening on D-Bus System Message Bus Socket.1870lb # [ 1.746664] systemd[1]: Listening on Journal Socket (/dev/log).1871lb # [ 1.747858] systemd[1]: Listening on Journal Sockets.1872lb # [ 1.748892] systemd[1]: Listening on udev Control Socket.1873worker1 # [ 1.772466] 9pnet: Installing 9P2000 support1874lb # [ 1.749972] systemd[1]: Listening on udev Kernel Socket.1875lb # [ 1.750960] systemd[1]: Reached target Socket Units.1876worker2 # [ 1.745338] systemd[1]: Starting Load Kernel Modules...1877lb # [ 1.752887] systemd[1]: Starting Create List of Static Device Nodes...1878client # [ 1.767594] systemd[1]: Finished Create List of Static Device Nodes.1879lb # [ 1.758320] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1880client # [ 1.773174] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1881worker1 # [ 1.785917] systemd[1]: Starting Coldplug All udev Devices...1882lb # [ 1.766636] systemd[1]: Starting Load Kernel Module configfs...1883client # [ 1.781926] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1884worker2 # [ 1.753324] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1885worker1 # [ 1.796649] systemd-journald[67]: Collecting audit messages is disabled.1886client # [ 1.785435] systemd-journald[67]: Collecting audit messages is disabled.1887client # [ 1.787928] systemd[1]: modprobe@configfs.service: Deactivated successfully.1888worker2 # [ 1.770906] netfs: FS-Cache loaded1889worker1 # [ 1.801061] systemd[1]: Finished Create List of Static Device Nodes.1890lb # [ 1.781641] systemd[1]: Starting Journal Service...1891worker2 # [ 1.775640] 9pnet: Installing 9P2000 support1892worker1 # [ 1.807371] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1893client # [ 1.795850] systemd[1]: Finished Load Kernel Module configfs.1894client # [ 1.800073] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1895worker2 # [ 1.785318] systemd[1]: Starting Coldplug All udev Devices...1896worker1 # [ 1.816145] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1897lb # [ 1.794529] systemd[1]: Starting Load Kernel Modules...1898client # [ 1.805191] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1899worker1 # [ 1.821123] systemd[1]: modprobe@configfs.service: Deactivated successfully.1900client # [ 1.812706] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1901client # [ 1.815503] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1902worker1 # [ 1.828072] systemd[1]: Finished Load Kernel Module configfs.1903worker2 # [ 1.798430] systemd[1]: Finished Create List of Static Device Nodes.1904worker1 # [ 1.834158] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1905worker2 # [ 1.805839] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1906lb # [ 1.801537] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1907worker1 # [ 1.838756] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1908lb # [ 1.817774] netfs: FS-Cache loaded1909worker2 # [ 1.813639] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1910lb # [ 1.822643] 9pnet: Installing 9P2000 support1911worker1 # [ 1.845789] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1912worker2 # [ 1.817904] systemd-journald[67]: Collecting audit messages is disabled.1913worker1 # [ 1.848731] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1914worker2 # [ 1.819469] systemd[1]: modprobe@configfs.service: Deactivated successfully.1915worker2 # [ 1.824305] systemd[1]: Finished Load Kernel Module configfs.1916lb # [ 1.836587] systemd[1]: Starting Coldplug All udev Devices...1917worker2 # [ 1.830705] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1918client # [ 1.849614] systemd[1]: Finished Load Kernel Modules.1919worker2 # [ 1.835084] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1920client # [ 1.857735] systemd[1]: Starting Apply Kernel Variables...1921worker2 # [ 1.842350] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1922lb # [ 1.851756] systemd[1]: Finished Create List of Static Device Nodes.1923worker2 # [ 1.845391] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1924worker1 # [ 1.878243] systemd[1]: Finished Load Kernel Modules.1925lb # [ 1.856320] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1926client # [ 1.869571] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1927worker1 # [ 1.884933] systemd[1]: Starting Apply Kernel Variables...1928lb # [ 1.864880] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1929client # [ 1.877840] systemd[1]: Starting Create Static Device Nodes in /dev...1930lb # [ 1.866883] systemd-journald[67]: Collecting audit messages is disabled.1931lb # [ 1.871895] systemd[1]: modprobe@configfs.service: Deactivated successfully.1932worker1 # [ 1.895278] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1933lb # [ 1.877829] systemd[1]: Finished Load Kernel Module configfs.1934worker2 # [ 1.874472] systemd[1]: Finished Load Kernel Modules.1935worker1 # [ 1.905376] systemd[1]: Starting Create Static Device Nodes in /dev...1936lb # [ 1.882218] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1937lb # [ 1.884722] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1938client # [ 1.899106] systemd[1]: Finished Apply Kernel Variables.1939worker2 # [ 1.882143] systemd[1]: Starting Apply Kernel Variables...1940lb # [ 1.891246] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1941worker1 # [ 1.918167] systemd[1]: Started Journal Service.1942lb # [ 1.895524] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1943client # [ 1.908715] systemd[1]: Started Journal Service.1944worker1 # [ 1.784829] systemd-modules-load[68]: Inserted module 'dm_mod'1945worker1 # [ 1.788919] systemd-modules-load[68]: Inserted module 'virtio_balloon'1946client # [ 1.774774] systemd-modules-load[68]: Inserted module 'dm_mod'1947worker1 # [ 1.789963] systemd-modules-load[68]: Inserted module 'virtio_gpu'1948client # [ 1.777926] systemd-modules-load[68]: Inserted module 'virtio_balloon'1949worker2 # [ 1.893369] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1950client # [ 1.780053] systemd-modules-load[68]: Inserted module 'virtio_gpu'1951worker1 # [ 1.795961] systemd[1]: Finished Apply Kernel Variables.1952worker2 # [ 1.901689] systemd[1]: Starting Create Static Device Nodes in /dev...1953client # [ 1.786854] systemd[1]: Finished Create Static Device Nodes in /dev.1954client # [ 1.792044] systemd[1]: Reached target Preparation for Local File Systems.1955client # [ 1.795071] systemd[1]: Reached target Local File Systems.1956client # [ 1.797940] systemd[1]: Starting Create System Files and Directories...1957worker1 # [ 1.811079] systemd[1]: Finished Create Static Device Nodes in /dev.1958worker1 # [ 1.811993] systemd[1]: Reached target Preparation for Local File Systems.1959worker1 # [ 1.812903] systemd[1]: Reached target Local File Systems.1960lb # [ 1.922522] systemd[1]: Finished Load Kernel Modules.1961worker1 # [ 1.816061] systemd[1]: Starting Create System Files and Directories...1962client # [ 1.805076] systemd[1]: Starting Rule-based Manager for Device Events and Files...1963worker2 # [ 1.921325] systemd[1]: Finished Apply Kernel Variables.1964lb # [ 1.931653] systemd[1]: Starting Apply Kernel Variables...1965worker1 # [ 1.823067] systemd[1]: Starting Rule-based Manager for Device Events and Files...1966worker2 # [ 1.936788] systemd[1]: Finished Create Static Device Nodes in /dev.1967lb # [ 1.945520] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1968worker2 # [ 1.805511] systemd-modules-load[68]: Inserted module 'dm_mod'1969worker2 # [ 1.807369] systemd-modules-load[68]: Inserted module 'virtio_balloon'1970worker2 # [ 1.941596] systemd[1]: Started Journal Service.1971worker2 # [ 1.810330] systemd-modules-load[68]: Inserted module 'virtio_gpu'1972client # [ 1.830129] systemd[1]: Finished Create System Files and Directories.1973lb # [ 1.954690] systemd[1]: Starting Create Static Device Nodes in /dev...1974worker2 # [ 1.819150] systemd[1]: Reached target Preparation for Local File Systems.1975worker1 # [ 1.849471] systemd[1]: Finished Create System Files and Directories.1976worker2 # [ 1.820191] systemd[1]: Reached target Local File Systems.1977worker2 # [ 1.823146] systemd[1]: Starting Create System Files and Directories...1978lb # [ 1.968558] systemd[1]: Finished Apply Kernel Variables.1979worker2 # [ 1.831070] systemd[1]: Starting Rule-based Manager for Device Events and Files...1980client # [ 1.856285] systemd-udevd[84]: Using default interface naming scheme 'v261'.1981worker1 # [ 1.869273] systemd-udevd[83]: Using default interface naming scheme 'v261'.1982lb # [ 1.985077] systemd[1]: Started Journal Service.1983lb # [ 1.852609] systemd-modules-load[68]: Inserted module 'dm_mod'1984lb # [ 1.855381] systemd-modules-load[68]: Inserted module 'virtio_balloon'1985lb # [ 1.856404] systemd-modules-load[68]: Inserted module 'virtio_gpu'1986lb # [ 1.859358] systemd[1]: Finished Create Static Device Nodes in /dev.1987worker2 # [ 1.855087] systemd[1]: Finished Create System Files and Directories.1988lb # [ 1.868117] systemd[1]: Reached target Preparation for Local File Systems.1989lb # [ 1.869114] systemd[1]: Reached target Local File Systems.1990worker1 # [ 1.892851] systemd[1]: Started Rule-based Manager for Device Events and Files.1991client # [ 1.881541] systemd[1]: Started Rule-based Manager for Device Events and Files.1992lb # [ 1.871254] systemd[1]: Starting Create System Files and Directories...1993lb # [ 1.878134] systemd[1]: Starting Rule-based Manager for Device Events and Files...1994worker2 # [ 1.878769] systemd-udevd[83]: Using default interface naming scheme 'v261'.1995lb # [ 1.904959] systemd[1]: Finished Create System Files and Directories.1996worker2 # [ 1.906080] systemd[1]: Started Rule-based Manager for Device Events and Files.1997client # [ 1.926103] systemd[1]: Finished Coldplug All udev Devices.1998client # [ 1.927146] systemd[1]: Reached target System Initialization.1999client # [ 1.927951] systemd[1]: Reached target Basic System.2000worker1 # [ 1.947815] systemd[1]: Finished Coldplug All udev Devices.
loading 2,564 hidden lines…
4565worker2 # [ 38.521283] nix-grpc-daemon[786]: ts=2026-09-22T10:46:55Z level=info event=scheduler_connected addr=lb:500514566worker1 # [ 38.551650] nix-grpc-daemon[784]: ts=2026-09-22T10:46:55Z level=info event=scheduler_connected addr=lb:500514567worker2 # [ 38.525269] nix-grpc-daemon[786]: ts=2026-09-22T10:46:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4568worker1 # [ 38.555838] nix-grpc-daemon[784]: ts=2026-09-22T10:46:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4569worker2 # [ 38.726067] nix-grpc-daemon[786]: ts=2026-09-22T10:46:55Z level=info event=scheduler_connected addr=lb:500514570worker2 # [ 38.727832] nix-grpc-daemon[786]: ts=2026-09-22T10:46:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4571worker1 # [ 38.758100] nix-grpc-daemon[784]: ts=2026-09-22T10:46:55Z level=info event=scheduler_connected addr=lb:500514572worker1 # [ 38.761322] nix-grpc-daemon[784]: ts=2026-09-22T10:46:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4573worker1 # [ 38.958641] nix-grpc-daemon[784]: ts=2026-09-22T10:46:56Z level=info event=scheduler_connected addr=lb:500514574worker2 # [ 38.928908] nix-grpc-daemon[786]: ts=2026-09-22T10:46:56Z level=info event=scheduler_connected addr=lb:500514575worker1 # [ 38.960679] nix-grpc-daemon[784]: ts=2026-09-22T10:46:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4576worker2 # [ 38.930883] nix-grpc-daemon[786]: ts=2026-09-22T10:46:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4577worker1 # [ 39.161226] nix-grpc-daemon[784]: ts=2026-09-22T10:46:56Z level=info event=scheduler_connected addr=lb:500514578worker1 # [ 39.162920] nix-grpc-daemon[784]: ts=2026-09-22T10:46:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4579worker2 # [ 39.132162] nix-grpc-daemon[786]: ts=2026-09-22T10:46:56Z level=info event=scheduler_connected addr=lb:500514580worker2 # [ 39.136240] nix-grpc-daemon[786]: ts=2026-09-22T10:46:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4581worker1 # [ 39.363488] nix-grpc-daemon[784]: ts=2026-09-22T10:46:56Z level=info event=scheduler_connected addr=lb:500514582worker1 # [ 39.365275] nix-grpc-daemon[784]: ts=2026-09-22T10:46:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4583worker2 # [ 39.338642] nix-grpc-daemon[786]: ts=2026-09-22T10:46:56Z level=info event=scheduler_connected addr=lb:500514584worker2 # [ 39.341777] nix-grpc-daemon[786]: ts=2026-09-22T10:46:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4585lb: must succeed: curl -sf localhost:9901/clusters4586lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4587lb # [ 39.521673] niks3-server[1053]: 2026/09/22 10:46:56 INFO lead: acquired remote=192.168.1.4:551144588worker2 # [ 39.515396] nix-grpc-daemon[786]: ts=2026-09-22T10:46:56Z level=info event=scheduler_take_over4589worker1 # [ 39.566568] nix-grpc-daemon[784]: ts=2026-09-22T10:46:56Z level=info event=scheduler_connected addr=lb:500514590worker2 # [ 39.539332] nix-grpc-daemon[786]: ts=2026-09-22T10:46:56Z level=info event=scheduler_connected addr=lb:500514591worker1 # [ 39.570902] nix-grpc-daemon[784]: ts=2026-09-22T10:46:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4592worker2 # [ 39.543682] nix-grpc-daemon[786]: ts=2026-09-22T10:46:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4593worker1 # [ 39.771938] nix-grpc-daemon[784]: ts=2026-09-22T10:46:56Z level=info event=scheduler_connected addr=lb:500514594worker2 # [ 39.744352] nix-grpc-daemon[786]: ts=2026-09-22T10:46:56Z level=info event=scheduler_connected addr=lb:500514595worker1 # [ 39.774134] nix-grpc-daemon[784]: ts=2026-09-22T10:46:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4596worker2 # [ 39.746152] nix-grpc-daemon[786]: ts=2026-09-22T10:46:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4597worker1 # [ 39.974872] nix-grpc-daemon[784]: ts=2026-09-22T10:46:57Z level=info event=scheduler_connected addr=lb:500514598worker1 # [ 39.976972] nix-grpc-daemon[784]: ts=2026-09-22T10:46:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4599worker2 # [ 39.947213] nix-grpc-daemon[786]: ts=2026-09-22T10:46:57Z level=info event=scheduler_connected addr=lb:500514600worker2 # [ 39.950412] nix-grpc-daemon[786]: ts=2026-09-22T10:46:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4601worker1 # [ 40.177681] nix-grpc-daemon[784]: ts=2026-09-22T10:46:57Z level=info event=scheduler_connected addr=lb:500514602worker2 # [ 40.148494] nix-grpc-daemon[786]: ts=2026-09-22T10:46:57Z level=info event=scheduler_connected addr=lb:500514603worker2 # [ 40.158756] nix-grpc-daemon[786]: ts=2026-09-22T10:46:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:349444604worker2 # [ 40.163671] nix-grpc-daemon[786]: ts=2026-09-22T10:46:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04605worker2 # [ 40.181177] nix-grpc-daemon[786]: ts=2026-09-22T10:46:57Z level=info event=oidc_jwks provider=mock keys=14606worker2 # [ 40.185398] nix-grpc-daemon[786]: ts=2026-09-22T10:46:57Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:349444607worker2 # [ 40.189329] nix-grpc-daemon[786]: ts=2026-09-22T10:46:57Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04608lb: must succeed: curl -sf localhost:9901/clusters4609lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4610worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4611worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4612lb: must succeed: curl -sf localhost:9901/clusters4613lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4614lb: must succeed: curl -sf localhost:9901/clusters4615lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4616lb: must succeed: curl -sf localhost:9901/clusters4617lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4618lb: must succeed: curl -sf localhost:9901/clusters4619lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4620worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4621worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.03 seconds)4622worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4623worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)4624lb: must succeed: curl -sf localhost:9901/clusters4625lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4626lb: must succeed: curl -sf localhost:9901/clusters4627lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4628worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4629worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4630worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4631worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4632worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4633worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)4634worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4635worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)4636(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 10.12 seconds)4637subtest: balancer auth: client cert, bearer token, nothing, unknown CN4638worker2 # [ 41.336217] nix-grpc-daemon[786]: ts=2026-09-22T10:46:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:349524639worker2 # [ 41.346788] systemd[1]: Started Nix Daemon.4640worker2 # [ 41.555266] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4641worker2 # [ 41.579810] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 85 ms (attempt 1/5)4642worker2 # [ 41.668115] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 187 ms (attempt 2/5)4643worker2 # [ 41.860515] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 243 ms (attempt 3/5)4644worker2 # [ 42.109443] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 208 ms (attempt 4/5)4645worker2 # [ 42.323723] nix-grpc-daemon[786]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4646worker2 # [ 42.327742] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4647worker2 # [ 42.363664] nix-grpc-daemon[786]: ts=2026-09-22T10:46:59Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:349524648worker2 # [ 42.375108] nix-grpc-daemon[786]: ts=2026-09-22T10:46:59Z level=debug event=path_info_miss path=qa4bpmmrzxnmnwz5m9pc3b9s796ifdak-nixos-system-client-test error="error: path '/nix/store/qa4bpmmrzxnmnwz5m9pc3b9s796ifdak-nixos-system-client-test' is required, but there is no substituter that can build it"4649worker2 # [ 42.380351] nix-grpc-daemon[786]: ts=2026-09-22T10:46:59Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:349524650worker2 # [ 42.386067] nix-grpc-daemon[786]: ts=2026-09-22T10:46:59Z level=debug event=path_info_miss path=qa4bpmmrzxnmnwz5m9pc3b9s796ifdak-nixos-system-client-test error="error: path '/nix/store/qa4bpmmrzxnmnwz5m9pc3b9s796ifdak-nixos-system-client-test' is required, but there is no substituter that can build it"4651client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4652client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds)4653client: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4654client: (finished: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.03 seconds)4655worker1 # [ 42.766890] nix-grpc-daemon[784]: ts=2026-09-22T10:46:59Z level=info event=oidc_jwks provider=mock keys=14656worker1 # [ 42.772206] nix-grpc-daemon[784]: ts=2026-09-22T10:46:59Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:357844657worker1 # [ 42.784723] systemd[1]: Started Nix Daemon.4658worker1 # [ 42.975728] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4659worker1 # [ 42.998494] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 97 ms (attempt 1/5)4660worker1 # [ 43.098293] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 194 ms (attempt 2/5)4661worker1 # [ 43.292454] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 275 ms (attempt 3/5)4662worker1 # [ 43.570745] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 682 ms (attempt 4/5)4663worker1 # [ 44.262287] nix-grpc-daemon[784]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4664worker1 # [ 44.265958] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4665worker2 # [ 44.258728] nix-grpc-daemon[786]: ts=2026-09-22T10:47:01Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:349524666worker1 # [ 44.304560] nix-grpc-daemon[784]: ts=2026-09-22T10:47:01Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:357844667worker2 # [ 44.269507] nix-grpc-daemon[786]: ts=2026-09-22T10:47:01Z level=debug event=path_info_miss path=qa4bpmmrzxnmnwz5m9pc3b9s796ifdak-nixos-system-client-test error="error: path '/nix/store/qa4bpmmrzxnmnwz5m9pc3b9s796ifdak-nixos-system-client-test' is required, but there is no substituter that can build it"4668worker1 # [ 44.315649] nix-grpc-daemon[784]: ts=2026-09-22T10:47:01Z level=debug event=path_info_miss path=qa4bpmmrzxnmnwz5m9pc3b9s796ifdak-nixos-system-client-test error="error: path '/nix/store/qa4bpmmrzxnmnwz5m9pc3b9s796ifdak-nixos-system-client-test' is required, but there is no substituter that can build it"4669client: must succeed: rm -r /var/lib/nix-grpc-store4670client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.04 seconds)4671worker1 # [ 44.572816] nix-grpc-daemon[784]: ts=2026-09-22T10:47:01Z level=info event=denied method=QueryMissing cn=- role=none4672worker1 # [ 44.772989] nix-grpc-daemon[784]: ts=2026-09-22T10:47:01Z level=info event=denied method=QueryMissing cn=stranger role=none4673(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.69 seconds)4674subtest: DAG build is scheduled across workers and lands in the cache4675client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24676client # don't know how to build these paths:4677client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4678worker1 # [ 44.999958] nix-grpc-daemon[784]: ts=2026-09-22T10:47:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357844679worker1 # [ 45.009860] nix-grpc-daemon[784]: ts=2026-09-22T10:47:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357844680worker2 # [ 45.015094] nix-grpc-daemon[786]: ts=2026-09-22T10:47:02Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04681worker2 # [ 45.020682] nix-grpc-daemon[786]: ts=2026-09-22T10:47:02Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14682worker1 # [ 45.103638] nix-grpc-daemon[784]: ts=2026-09-22T10:47:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357844683worker1 # [ 45.120154] nix-grpc-daemon[784]: ts=2026-09-22T10:47:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357844684client # copying 1 paths to 'lb:50051'...4685worker1 # [ 45.129787] nix-grpc-daemon[784]: ts=2026-09-22T10:47:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357844686client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4687worker1 # [ 45.155920] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4688worker1 # [ 45.174139] nix-grpc-daemon[784]: ts=2026-09-22T10:47:02Z level=info event=niks3_push_started pid=8644689worker1 # [ 45.467824] nix-grpc-daemon[864]: time=2026-09-22T10:47:02.586Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""4690worker1 # [ 45.651315] nix-daemon[852]: accepted connection from pid 876, user nix-grpc-daemon (trusted)4691worker1 # [ 45.668158] nix-daemon[852]: reaped child process 879, status = succeeded4692worker1 # [ 45.796373] nix-daemon[852]: accepted connection from pid 882, user nix-grpc-daemon (trusted)4693worker1 # [ 45.810565] nix-daemon[852]: reaped child process 885, status = succeeded4694lb # [ 45.794501] niks3-server[1053]: 2026/09/22 10:47:02 INFO Received uploads request method=POST path=/api/pending_closures4695worker1 # [ 45.831641] nix-grpc-daemon[864]: time=2026-09-22T10:47:02.953Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4696worker1 # [ 45.834294] nix-grpc-daemon[864]: time=2026-09-22T10:47:02.956Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4697lb # [ 45.850762] niks3-server[1053]: 2026/09/22 10:47:02 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4698lb # [ 45.852594] niks3-server[1053]: 2026/09/22 10:47:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4699lb # [ 45.861245] niks3-server[1053]: 2026/09/22 10:47:02 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4700lb # [ 45.869613] niks3-server[1053]: 2026/09/22 10:47:02 INFO Signed narinfos id=1 count=14701worker1 # [ 45.895521] nix-grpc-daemon[864]: time=2026-09-22T10:47:03.017Z level=INFO msg="Uploading 1 narinfos"4702lb # [ 45.884806] niks3-server[1053]: 2026/09/22 10:47:02 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4703lb # [ 45.891369] niks3-server[1053]: 2026/09/22 10:47:02 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4704lb # [ 45.902190] niks3-server[1053]: 2026/09/22 10:47:02 INFO Completed upload id=14705worker1 # [ 45.926608] nix-grpc-daemon[864]: time=2026-09-22T10:47:03.048Z level=INFO msg="Upload complete. (432ms)"4706worker1 # [ 45.932905] nix-grpc-daemon[784]: ts=2026-09-22T10:47:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=0 paths=1 nar_bytes_in=5044707worker1 # [ 45.939592] nix-grpc-daemon[784]: ts=2026-09-22T10:47:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357844708worker1 # [ 45.951264] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4709worker1 # [ 45.960213] nix-daemon[852]: reaped child process 862, status = succeeded4710client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4711worker1 # [ 45.965801] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4712client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4713client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4714client # farm-a> LOG-farm-a-t14715worker1 # [ 46.782599] nix-daemon[852]: reaped child process 889, status = succeeded4716worker1 # [ 46.906238] nix-daemon[852]: accepted connection from pid 905, user nix-grpc-daemon (trusted)4717worker1 # [ 46.920720] nix-daemon[852]: reaped child process 908, status = succeeded4718lb # [ 46.906873] niks3-server[1053]: 2026/09/22 10:47:03 INFO Received uploads request method=POST path=/api/pending_closures4719worker1 # [ 46.978384] nix-grpc-daemon[864]: time=2026-09-22T10:47:04.100Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4720worker1 # [ 46.981082] nix-grpc-daemon[864]: time=2026-09-22T10:47:04.103Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4721lb # [ 46.994632] niks3-server[1053]: 2026/09/22 10:47:03 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4722lb # [ 47.003727] niks3-server[1053]: 2026/09/22 10:47:03 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4723lb # [ 47.009113] niks3-server[1053]: 2026/09/22 10:47:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4724lb # [ 47.018401] niks3-server[1053]: 2026/09/22 10:47:03 INFO Signed narinfos id=2 count=14725worker1 # [ 47.044137] nix-grpc-daemon[864]: time=2026-09-22T10:47:04.164Z level=INFO msg="Uploading 1 narinfos"4726lb # [ 47.022173] niks3-server[1053]: 2026/09/22 10:47:03 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4727lb # [ 47.032488] niks3-server[1053]: 2026/09/22 10:47:03 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4728lb # [ 47.037813] niks3-server[1053]: 2026/09/22 10:47:03 INFO Completed upload id=24729lb # [ 47.040208] niks3-server[1053]: 2026/09/22 10:47:03 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4730worker1 # [ 47.061959] nix-grpc-daemon[864]: time=2026-09-22T10:47:04.184Z level=INFO msg="Upload complete. (287ms)"4731worker1 # [ 47.069848] nix-grpc-daemon[784]: ts=2026-09-22T10:47:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14732worker1 # [ 47.085827] nix-daemon[852]: reaped child process 887, status = succeeded4733worker2 # [ 47.054545] nix-grpc-daemon[786]: ts=2026-09-22T10:47:04Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04734worker2 # [ 47.058508] nix-grpc-daemon[786]: ts=2026-09-22T10:47:04Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14735worker1 # [ 47.130245] nix-grpc-daemon[784]: ts=2026-09-22T10:47:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357844736worker1 # [ 47.143624] nix-grpc-daemon[784]: ts=2026-09-22T10:47:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357844737worker1 # [ 47.159108] nix-grpc-daemon[864]: time=2026-09-22T10:47:04.280Z level=INFO msg="All 1 paths already cached"4738client # copying 1 paths to 'lb:50051'...4739client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4740worker1 # [ 47.167868] nix-grpc-daemon[784]: ts=2026-09-22T10:47:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357844741worker1 # [ 47.184464] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4742worker1 # [ 47.364378] nix-daemon[852]: accepted connection from pid 912, user nix-grpc-daemon (trusted)4743worker1 # [ 47.380670] nix-daemon[852]: reaped child process 915, status = succeeded4744worker1 # [ 47.531744] nix-daemon[852]: accepted connection from pid 917, user nix-grpc-daemon (trusted)4745worker1 # [ 47.546876] nix-daemon[852]: reaped child process 920, status = succeeded4746lb # [ 47.536534] niks3-server[1053]: 2026/09/22 10:47:04 INFO Received uploads request method=POST path=/api/pending_closures4747worker1 # [ 47.573099] nix-grpc-daemon[864]: time=2026-09-22T10:47:04.695Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4748worker1 # [ 47.574765] nix-grpc-daemon[864]: time=2026-09-22T10:47:04.697Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4749lb # [ 47.577456] niks3-server[1053]: 2026/09/22 10:47:04 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4750lb # [ 47.585266] niks3-server[1053]: 2026/09/22 10:47:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4751lb # [ 47.590496] niks3-server[1053]: 2026/09/22 10:47:04 INFO Signed narinfos id=3 count=14752worker1 # [ 47.617923] nix-grpc-daemon[864]: time=2026-09-22T10:47:04.739Z level=INFO msg="Uploading 1 narinfos"4753lb # [ 47.605806] niks3-server[1053]: 2026/09/22 10:47:04 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4754lb # [ 47.608753] niks3-server[1053]: 2026/09/22 10:47:04 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4755lb # [ 47.616198] niks3-server[1053]: 2026/09/22 10:47:04 INFO Completed upload id=34756worker1 # [ 47.640544] nix-grpc-daemon[864]: time=2026-09-22T10:47:04.762Z level=INFO msg="Upload complete. (439ms)"4757lb # [ 47.619304] niks3-server[1053]: 2026/09/22 10:47:04 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4758worker1 # [ 47.645298] nix-grpc-daemon[784]: ts=2026-09-22T10:47:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=0 paths=1 nar_bytes_in=6244759worker1 # [ 47.651355] nix-grpc-daemon[784]: ts=2026-09-22T10:47:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357844760worker1 # [ 47.659528] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4761client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4762worker1 # [ 47.674086] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4763worker1 # [ 47.675542] nix-daemon[852]: reaped child process 910, status = succeeded4764client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4765lb # [ 48.679248] envoy[932]: conn peer=[2001:db8:1::1]:56630 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=715 tx=364 ms=11014766lb # [ 48.680970] envoy[932]: conn peer=[2001:db8:1::1]:56634 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=365 ms=15784767lb # [ 48.682821] envoy[932]: conn peer=[2001:db8:1::1]:37998 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=74768lb # [ 48.684481] envoy[932]: conn peer=[2001:db8:1::1]:38002 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=84769client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4770worker1 # [ 48.970617] nix-grpc-daemon[784]: E0000 00:00:1790074026.090591 824 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4771worker1 # [ 48.999134] nix-grpc-daemon[784]: E0000 00:00:1790074026.118731 824 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4772worker1 # [ 49.002278] nix-grpc-daemon[784]: E0000 00:00:1790074026.119832 824 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4773worker1 # [ 49.021664] nix-grpc-daemon[784]: E0000 00:00:1790074026.143248 824 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4774client # farm-b> LOG-farm-b-t14775worker1 # [ 49.294624] nix-daemon[852]: reaped child process 924, status = succeeded4776worker1 # [ 49.414285] nix-daemon[852]: accepted connection from pid 940, user nix-grpc-daemon (trusted)4777worker1 # [ 49.428256] nix-daemon[852]: reaped child process 943, status = succeeded4778lb # [ 49.415563] niks3-server[1053]: 2026/09/22 10:47:06 INFO Received uploads request method=POST path=/api/pending_closures4779worker1 # [ 49.447497] nix-grpc-daemon[864]: time=2026-09-22T10:47:06.569Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4780worker1 # [ 49.449802] nix-grpc-daemon[864]: time=2026-09-22T10:47:06.571Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4781lb # [ 49.443039] niks3-server[1053]: 2026/09/22 10:47:06 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4782lb # [ 49.449442] niks3-server[1053]: 2026/09/22 10:47:06 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4783lb # [ 49.454315] niks3-server[1053]: 2026/09/22 10:47:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4784lb # [ 49.456468] niks3-server[1053]: 2026/09/22 10:47:06 INFO Signed narinfos id=4 count=14785worker1 # [ 49.480550] nix-grpc-daemon[864]: time=2026-09-22T10:47:06.602Z level=INFO msg="Uploading 1 narinfos"4786lb # [ 49.459601] niks3-server[1053]: 2026/09/22 10:47:06 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4787lb # [ 49.464055] niks3-server[1053]: 2026/09/22 10:47:06 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4788worker1 # [ 49.492621] nix-grpc-daemon[864]: time=2026-09-22T10:47:06.614Z level=INFO msg="Upload complete. (210ms)"4789lb # [ 49.469121] niks3-server[1053]: 2026/09/22 10:47:06 INFO Completed upload id=44790lb # [ 49.470779] niks3-server[1053]: 2026/09/22 10:47:06 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4791worker1 # [ 49.495422] nix-grpc-daemon[784]: ts=2026-09-22T10:47:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14792worker1 # [ 49.502883] nix-daemon[852]: reaped child process 922, status = succeeded4793worker2 # [ 49.475481] nix-grpc-daemon[786]: ts=2026-09-22T10:47:06Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04794worker2 # [ 49.479535] nix-grpc-daemon[786]: ts=2026-09-22T10:47:06Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14795worker1 # [ 49.551673] nix-grpc-daemon[784]: ts=2026-09-22T10:47:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357844796worker1 # [ 49.564502] nix-grpc-daemon[784]: ts=2026-09-22T10:47:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357844797worker1 # [ 49.579564] nix-grpc-daemon[864]: time=2026-09-22T10:47:06.701Z level=INFO msg="All 2 paths already cached"4798client # copying 1 paths to 'lb:50051'...4799client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4800worker1 # [ 49.590062] nix-grpc-daemon[784]: ts=2026-09-22T10:47:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357844801worker1 # [ 49.598728] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4802worker1 # [ 49.775264] nix-daemon[852]: accepted connection from pid 947, user nix-grpc-daemon (trusted)4803worker1 # [ 49.790321] nix-daemon[852]: reaped child process 950, status = succeeded4804worker1 # [ 49.941906] nix-daemon[852]: accepted connection from pid 952, user nix-grpc-daemon (trusted)4805worker1 # [ 49.957700] nix-daemon[852]: reaped child process 955, status = succeeded4806lb # [ 49.942497] niks3-server[1053]: 2026/09/22 10:47:06 INFO Received uploads request method=POST path=/api/pending_closures4807worker1 # [ 49.971565] nix-grpc-daemon[864]: time=2026-09-22T10:47:07.093Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4808worker1 # [ 49.973134] nix-grpc-daemon[864]: time=2026-09-22T10:47:07.093Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4809lb # [ 49.957921] niks3-server[1053]: 2026/09/22 10:47:06 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4810lb # [ 49.963078] niks3-server[1053]: 2026/09/22 10:47:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4811lb # [ 49.966860] niks3-server[1053]: 2026/09/22 10:47:06 INFO Signed narinfos id=5 count=14812worker1 # [ 49.990685] nix-grpc-daemon[864]: time=2026-09-22T10:47:07.112Z level=INFO msg="Uploading 1 narinfos"4813lb # [ 49.970633] niks3-server[1053]: 2026/09/22 10:47:06 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4814lb # [ 49.976069] niks3-server[1053]: 2026/09/22 10:47:06 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4815lb # [ 49.984176] niks3-server[1053]: 2026/09/22 10:47:06 INFO Completed upload id=54816worker1 # [ 50.008327] nix-grpc-daemon[864]: time=2026-09-22T10:47:07.130Z level=INFO msg="Upload complete. (396ms)"4817lb # [ 49.985848] niks3-server[1053]: 2026/09/22 10:47:06 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4818worker1 # [ 50.011642] nix-grpc-daemon[784]: ts=2026-09-22T10:47:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=0 paths=1 nar_bytes_in=7524819worker1 # [ 50.015810] nix-grpc-daemon[784]: ts=2026-09-22T10:47:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357844820worker1 # [ 50.023081] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4821worker1 # [ 50.029139] nix-daemon[852]: reaped child process 945, status = succeeded4822worker1 # [ 50.033325] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4823client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4824client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4825client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4826client # farm-top> LOG-farm-top-t14827worker1 # [ 51.066297] nix-daemon[852]: reaped child process 959, status = succeeded4828worker1 # [ 51.194472] nix-daemon[852]: accepted connection from pid 975, user nix-grpc-daemon (trusted)4829worker1 # [ 51.210701] nix-daemon[852]: reaped child process 978, status = succeeded4830lb # [ 51.197183] niks3-server[1053]: 2026/09/22 10:47:08 INFO Received uploads request method=POST path=/api/pending_closures4831worker1 # [ 51.232496] nix-grpc-daemon[864]: time=2026-09-22T10:47:08.354Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4832worker1 # [ 51.236380] nix-grpc-daemon[864]: time=2026-09-22T10:47:08.356Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4833lb # [ 51.228184] niks3-server[1053]: 2026/09/22 10:47:08 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4834lb # [ 51.234743] niks3-server[1053]: 2026/09/22 10:47:08 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4835lb # [ 51.238734] niks3-server[1053]: 2026/09/22 10:47:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4836lb # [ 51.241240] niks3-server[1053]: 2026/09/22 10:47:08 INFO Signed narinfos id=6 count=14837worker1 # [ 51.267148] nix-grpc-daemon[864]: time=2026-09-22T10:47:08.389Z level=INFO msg="Uploading 1 narinfos"4838lb # [ 51.247877] niks3-server[1053]: 2026/09/22 10:47:08 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4839lb # [ 51.256335] niks3-server[1053]: 2026/09/22 10:47:08 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4840lb # [ 51.260205] niks3-server[1053]: 2026/09/22 10:47:08 INFO Completed upload id=64841worker1 # [ 51.284328] nix-grpc-daemon[864]: time=2026-09-22T10:47:08.406Z level=INFO msg="Upload complete. (241ms)"4842lb # [ 51.262421] niks3-server[1053]: 2026/09/22 10:47:08 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4843worker1 # [ 51.288493] nix-grpc-daemon[784]: ts=2026-09-22T10:47:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14844worker1 # [ 51.296357] nix-daemon[852]: reaped child process 957, status = succeeded4845client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 6.53 seconds)4846client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4847client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.20 seconds)4848client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4849client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.02 seconds)4850client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4851client # copying 3 paths...4852client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4853client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4854client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4855client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.26 seconds)4856(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.07 seconds)4857subtest: repeat is answered Cached by the scheduler without building4858worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4859worker1 # finding garbage collector roots...4860worker1 # deleting specified paths...4861worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4862worker1 # deleting unused links...4863worker1 # note: hard linking is currently saving 0.0 KiB4864worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.23 seconds)4865worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4866worker2 # finding garbage collector roots...4867worker2 # deleting specified paths...4868worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4869worker2 # deleting unused links...4870worker2 # note: hard linking is currently saving 0.0 KiB4871worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.25 seconds)4872client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24873worker2 # [ 52.506790] nix-grpc-daemon[786]: ts=2026-09-22T10:47:09Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:349524874client # don't know how to build these paths:4875client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4876worker2 # [ 52.514615] nix-grpc-daemon[786]: ts=2026-09-22T10:47:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349524877worker2 # [ 52.538719] nix-grpc-daemon[786]: ts=2026-09-22T10:47:09Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14878client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.26 seconds)4879client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4880client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.20 seconds)4881lb: must succeed: curl -sf localhost:9901/clusters4882lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4883lb: must succeed: curl -sf localhost:9901/clusters4884lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4885worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4886worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4887(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.14 seconds)4888subtest: inputs already in the cache are not sent to the scheduler4889lb: must succeed: curl -sf localhost:9901/clusters4890lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4891lb: must succeed: curl -sf localhost:9901/clusters4892lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4893worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4894worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.03 seconds)4895client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24896worker2 # [ 53.283884] nix-grpc-daemon[786]: ts=2026-09-22T10:47:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:349524897client # don't know how to build these paths:4898client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4899worker1 # [ 53.322666] nix-grpc-daemon[784]: ts=2026-09-22T10:47:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357844900worker1 # [ 53.332525] nix-grpc-daemon[864]: time=2026-09-22T10:47:10.454Z level=INFO msg="All 2 paths already cached"4901worker2 # [ 53.314269] nix-grpc-daemon[786]: ts=2026-09-22T10:47:10Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04902worker2 # [ 53.316286] nix-grpc-daemon[786]: ts=2026-09-22T10:47:10Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14903worker1 # [ 53.389472] nix-grpc-daemon[784]: ts=2026-09-22T10:47:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357844904worker1 # [ 53.403321] nix-grpc-daemon[784]: ts=2026-09-22T10:47:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357844905worker1 # [ 53.417203] nix-grpc-daemon[864]: time=2026-09-22T10:47:10.539Z level=INFO msg="All 2 paths already cached"4906client # copying 1 paths to 'lb:50051'...4907client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4908worker1 # [ 53.426680] nix-grpc-daemon[784]: ts=2026-09-22T10:47:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357844909worker1 # [ 53.439553] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4910worker1 # [ 53.622796] nix-daemon[852]: accepted connection from pid 995, user nix-grpc-daemon (trusted)4911worker1 # [ 53.637399] nix-daemon[852]: reaped child process 998, status = succeeded4912worker1 # [ 53.779791] nix-daemon[852]: accepted connection from pid 1000, user nix-grpc-daemon (trusted)4913worker1 # [ 53.794603] nix-daemon[852]: reaped child process 1003, status = succeeded4914lb # [ 53.782615] niks3-server[1053]: 2026/09/22 10:47:10 INFO Received uploads request method=POST path=/api/pending_closures4915worker1 # [ 53.815382] nix-grpc-daemon[864]: time=2026-09-22T10:47:10.937Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4916worker1 # [ 53.817568] nix-grpc-daemon[864]: time=2026-09-22T10:47:10.939Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4917lb # [ 53.810996] niks3-server[1053]: 2026/09/22 10:47:10 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4918lb # [ 53.822970] niks3-server[1053]: 2026/09/22 10:47:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4919worker1 # [ 53.852459] nix-grpc-daemon[864]: time=2026-09-22T10:47:10.974Z level=INFO msg="Uploading 1 narinfos"4920lb # [ 53.828783] niks3-server[1053]: 2026/09/22 10:47:10 INFO Signed narinfos id=7 count=14921lb # [ 53.834538] niks3-server[1053]: 2026/09/22 10:47:10 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4922lb # [ 53.845516] niks3-server[1053]: 2026/09/22 10:47:10 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4923lb # [ 53.850594] niks3-server[1053]: 2026/09/22 10:47:10 INFO Completed upload id=74924worker1 # [ 53.877558] nix-grpc-daemon[864]: time=2026-09-22T10:47:10.999Z level=INFO msg="Upload complete. (424ms)"4925lb # [ 53.856075] niks3-server[1053]: 2026/09/22 10:47:10 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4926worker1 # [ 53.880573] nix-grpc-daemon[784]: ts=2026-09-22T10:47:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=0 paths=1 nar_bytes_in=7524927worker1 # [ 53.885432] nix-grpc-daemon[784]: ts=2026-09-22T10:47:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357844928worker1 # [ 53.894340] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4929worker1 # [ 53.899872] nix-daemon[852]: reaped child process 993, status = succeeded4930client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4931worker1 # [ 53.906559] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4932client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4933client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4934client # farm-top> LOG-farm-top-t1b4935worker1 # [ 54.519443] nix-daemon[852]: reaped child process 1007, status = succeeded4936worker1 # [ 54.669627] nix-daemon[852]: accepted connection from pid 1023, user nix-grpc-daemon (trusted)4937worker1 # [ 54.685089] nix-daemon[852]: reaped child process 1026, status = succeeded4938lb # [ 54.674258] niks3-server[1053]: 2026/09/22 10:47:11 INFO Received uploads request method=POST path=/api/pending_closures4939worker1 # [ 54.707628] nix-grpc-daemon[864]: time=2026-09-22T10:47:11.829Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4940worker1 # [ 54.710295] nix-grpc-daemon[864]: time=2026-09-22T10:47:11.831Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4941lb # [ 54.714704] niks3-server[1053]: 2026/09/22 10:47:11 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4942lb # [ 54.722801] niks3-server[1053]: 2026/09/22 10:47:11 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4943lb # [ 54.735170] niks3-server[1053]: 2026/09/22 10:47:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4944lb # [ 54.740768] niks3-server[1053]: 2026/09/22 10:47:11 INFO Signed narinfos id=8 count=14945worker1 # [ 54.768041] nix-grpc-daemon[864]: time=2026-09-22T10:47:11.890Z level=INFO msg="Uploading 1 narinfos"4946lb # [ 54.747568] niks3-server[1053]: 2026/09/22 10:47:11 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4947lb # [ 54.758913] niks3-server[1053]: 2026/09/22 10:47:11 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4948lb # [ 54.764340] niks3-server[1053]: 2026/09/22 10:47:11 INFO Completed upload id=84949worker1 # [ 54.789773] nix-grpc-daemon[864]: time=2026-09-22T10:47:11.912Z level=INFO msg="Upload complete. (278ms)"4950lb # [ 54.766463] niks3-server[1053]: 2026/09/22 10:47:11 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4951worker1 # [ 54.792801] nix-grpc-daemon[784]: ts=2026-09-22T10:47:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=0 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14952worker1 # [ 54.800453] nix-daemon[852]: reaped child process 1005, status = succeeded4953client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.71 seconds)4954client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4955client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.17 seconds)4956lb: must succeed: curl -sf localhost:9901/clusters4957lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4958lb: must succeed: curl -sf localhost:9901/clusters4959lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4960worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4961worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.03 seconds)4962(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.12 seconds)4963subtest: outputs read back through any worker carry the cache signature4964client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4965worker1 # [ 55.321689] nix-grpc-daemon[784]: ts=2026-09-22T10:47:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:364204966client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4967client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4968worker1 # [ 55.365546] nix-grpc-daemon[784]: ts=2026-09-22T10:47:12Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:364204969client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.26 seconds)4970client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4971worker2 # [ 55.493551] nix-grpc-daemon[786]: ts=2026-09-22T10:47:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:535104972client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):4973client # /nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a4974client # /nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b4975client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4976worker2 # [ 55.537324] nix-grpc-daemon[786]: ts=2026-09-22T10:47:12Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:535104977worker2 # [ 55.542970] nix-grpc-daemon[786]: copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4978worker2 # [ 55.560794] nix-grpc-daemon[786]: copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4979worker2 # [ 55.569680] nix-grpc-daemon[786]: copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4980client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&ca-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ca.pem' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.25 seconds)4981client: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4982client # finding garbage collector roots...4983client # removing stale temporary roots file "/nix/var/nix/temproots/887"4984client # removing stale temporary roots file "/nix/var/nix/temproots/866"4985client # removing stale temporary roots file "/nix/var/nix/temproots/818"4986client # removing stale temporary roots file "/nix/var/nix/temproots/854"4987client # removing stale temporary roots file "/nix/var/nix/temproots/833"4988client # removing stale temporary roots file "/nix/var/nix/temproots/874"4989client # deleting specified paths...4990client # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4991client # deleting unused links...4992client # note: hard linking is currently saving 0.0 KiB4993worker2 # [ 56.000612] nix-grpc-daemon[786]: ts=2026-09-22T10:47:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:349524994worker1 # [ 56.034619] nix-grpc-daemon[784]: ts=2026-09-22T10:47:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357844995worker1 # [ 56.042243] nix-grpc-daemon[784]: copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4996worker1 # [ 56.063171] nix-grpc-daemon[864]: time=2026-09-22T10:47:13.184Z level=INFO msg="All 1 paths already cached"4997worker1 # [ 56.069951] nix-grpc-daemon[784]: ts=2026-09-22T10:47:13Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:357844998worker2 # [ 56.046210] nix-grpc-daemon[786]: ts=2026-09-22T10:47:13Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:349524999client # copying 1 paths...5000worker1 # [ 56.087753] nix-grpc-daemon[784]: ts=2026-09-22T10:47:13Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:357845001client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'grpc://lb:50051'...5002worker2 # [ 56.072600] nix-grpc-daemon[786]: ts=2026-09-22T10:47:13Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:349525003worker2 # [ 56.076642] nix-grpc-daemon[786]: ts=2026-09-22T10:47:13Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=0 paths=1 nar_bytes_out=2325004client: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.55 seconds)5005(finished: subtest: outputs read back through any worker carry the cache signature, in 1.07 seconds)5006subtest: two clients wanting the same drv build it once5007client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup5008client # [ 56.215551] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.5009client # Running as unit: dedup1.service; invocation ID: 73072807f88946daa331a42e32085deb5010client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.04 seconds)5011client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup5012worker1 # [ 56.410894] nix-grpc-daemon[784]: ts=2026-09-22T10:47:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357845013client # [ 56.402303] nix[937]: don't know how to build these paths:5014client # [ 56.403185] nix[937]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5015worker2 # [ 56.393743] nix-grpc-daemon[786]: ts=2026-09-22T10:47:13Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05016worker2 # [ 56.396608] nix-grpc-daemon[786]: ts=2026-09-22T10:47:13Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=15017worker2 # [ 56.438945] nix-grpc-daemon[786]: ts=2026-09-22T10:47:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349525018worker2 # [ 56.453107] nix-grpc-daemon[786]: ts=2026-09-22T10:47:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349525019client # [ 56.479321] nix[937]: copying 1 paths to 'lb:50051'...5020worker2 # [ 56.462622] nix-grpc-daemon[786]: ts=2026-09-22T10:47:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349525021client # [ 56.482141] nix[937]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...5022worker2 # [ 56.473657] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5023worker2 # [ 56.485637] nix-grpc-daemon[786]: ts=2026-09-22T10:47:13Z level=info event=niks3_push_started pid=9275024worker2 # [ 56.728149] nix-grpc-daemon[927]: time=2026-09-22T10:47:13.902Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5025worker2 # [ 56.903803] nix-daemon[873]: accepted connection from pid 939, user nix-grpc-daemon (trusted)5026worker2 # [ 56.919702] nix-daemon[873]: reaped child process 942, status = succeeded5027worker2 # [ 57.069266] nix-daemon[873]: accepted connection from pid 945, user nix-grpc-daemon (trusted)5028worker2 # [ 57.080418] nix-daemon[873]: reaped child process 948, status = succeeded5029lb # [ 57.100593] niks3-server[1053]: 2026/09/22 10:47:13 INFO Received uploads request method=POST path=/api/pending_closures5030worker2 # [ 57.116899] nix-grpc-daemon[927]: time=2026-09-22T10:47:14.294Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5031worker2 # [ 57.119686] nix-grpc-daemon[927]: time=2026-09-22T10:47:14.297Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"5032lb # [ 57.170566] niks3-server[1053]: 2026/09/22 10:47:14 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst5033lb # [ 57.175636] niks3-server[1053]: 2026/09/22 10:47:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign5034worker2 # [ 57.174080] nix-grpc-daemon[927]: time=2026-09-22T10:47:14.351Z level=INFO msg="Uploading 1 narinfos"5035lb # [ 57.180298] niks3-server[1053]: 2026/09/22 10:47:14 INFO Signed narinfos id=9 count=15036lb # [ 57.189976] niks3-server[1053]: 2026/09/22 10:47:14 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls5037lb # [ 57.196111] niks3-server[1053]: 2026/09/22 10:47:14 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete5038lb # [ 57.202491] niks3-server[1053]: 2026/09/22 10:47:14 INFO Completed upload id=95039worker2 # [ 57.195983] nix-grpc-daemon[927]: time=2026-09-22T10:47:14.373Z level=INFO msg="Upload complete. (441ms)"5040lb # [ 57.203668] niks3-server[1053]: 2026/09/22 10:47:14 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo5041worker2 # [ 57.198917] nix-grpc-daemon[786]: ts=2026-09-22T10:47:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=0 paths=1 nar_bytes_in=4325042worker2 # [ 57.202219] nix-grpc-daemon[786]: ts=2026-09-22T10:47:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349525043worker2 # [ 57.208069] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5044worker2 # [ 57.214979] nix-daemon[873]: reaped child process 925, status = succeeded5045client # [ 57.234838] nix[937]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5046worker2 # [ 57.218497] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5047client # [ 57.270812] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.5048client # Running as unit: dedup2.service; invocation ID: 29d60e79a0534818a87299d9afa8b8475049client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.06 seconds)5050worker1 # [ 57.510850] nix-grpc-daemon[784]: ts=2026-09-22T10:47:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357845051client # [ 57.505428] nix[953]: don't know how to build these paths:5052client # [ 57.506280] nix[953]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5053worker2 # [ 57.501726] nix-grpc-daemon[786]: ts=2026-09-22T10:47:14Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05054worker2 # [ 57.508841] nix-grpc-daemon[786]: ts=2026-09-22T10:47:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349525055client # [ 57.531452] nix[953]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5056client: must succeed: sleep 25057lb # [ 58.679575] envoy[932]: conn peer=[2001:db8:1::1]:38014 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9494 tx=4393 ms=63335058lb # [ 58.681151] envoy[932]: conn peer=[2001:db8:1::1]:38020 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=485059lb # [ 58.682583] envoy[932]: conn peer=[2001:db8:1::1]:34788 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4182 tx=1895 ms=14915060lb # [ 58.683966] envoy[932]: conn peer=[2001:db8:1::1]:34790 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=969 tx=2139 ms=1335061lb # [ 58.685351] envoy[932]: conn peer=[2001:db8:1::1]:34800 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=409 ms=625062client: (finished: must succeed: sleep 2, in 2.03 seconds)5063client: waiting for success: ! systemctl is-active dedup1 dedup25064worker2 # [ 60.634500] nix-daemon[873]: reaped child process 952, status = succeeded5065worker2 # [ 60.778214] nix-daemon[873]: accepted connection from pid 984, user nix-grpc-daemon (trusted)5066worker2 # [ 60.790714] nix-daemon[873]: reaped child process 987, status = succeeded5067lb # [ 60.813177] niks3-server[1053]: 2026/09/22 10:47:17 INFO Received uploads request method=POST path=/api/pending_closures5068worker2 # [ 60.955837] nix-grpc-daemon[927]: time=2026-09-22T10:47:18.132Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5069worker2 # [ 60.957805] nix-grpc-daemon[927]: time=2026-09-22T10:47:18.132Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"5070lb # [ 60.986036] niks3-server[1053]: 2026/09/22 10:47:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign5071lb # [ 60.989030] niks3-server[1053]: 2026/09/22 10:47:17 INFO Signed narinfos id=10 count=15072worker2 # [ 60.982781] nix-grpc-daemon[927]: time=2026-09-22T10:47:18.159Z level=INFO msg="Uploading 1 narinfos"5073lb # [ 60.994593] niks3-server[1053]: 2026/09/22 10:47:17 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5074lb # [ 61.000986] niks3-server[1053]: 2026/09/22 10:47:17 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete5075lb # [ 61.008931] niks3-server[1053]: 2026/09/22 10:47:17 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst5076lb # [ 61.026075] niks3-server[1053]: 2026/09/22 10:47:17 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls5077lb # [ 61.039325] niks3-server[1053]: 2026/09/22 10:47:17 INFO Completed upload id=105078worker2 # [ 61.033370] nix-grpc-daemon[927]: time=2026-09-22T10:47:18.210Z level=INFO msg="Upload complete. (403ms)"5079lb # [ 61.050101] niks3-server[1053]: 2026/09/22 10:47:17 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo5080worker2 # [ 61.043315] nix-grpc-daemon[786]: ts=2026-09-22T10:47:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=15081worker2 # [ 61.051226] nix-grpc-daemon[786]: ts=2026-09-22T10:47:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=15082worker2 # [ 61.063782] nix-daemon[873]: reaped child process 950, status = succeeded5083client # [ 61.109613] systemd[1]: dedup1.service: Deactivated successfully.5084client # [ 61.114171] systemd[1]: dedup1.service: Consumed 122ms CPU time over 4.896s wall clock time, 41.5M memory peak, 7.2K incoming IP traffic, 8.5K outgoing IP traffic.5085client # [ 61.119763] systemd[1]: dedup2.service: Deactivated successfully.5086client # [ 61.122877] systemd[1]: dedup2.service: Consumed 141ms CPU time over 3.851s wall clock time, 35.2M memory peak, 6K incoming IP traffic, 5.7K outgoing IP traffic.5087client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.10 seconds)5088client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }5089client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)5090worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5091worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5092worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5093worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5094worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5095worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5096worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5097worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5098(finished: subtest: two clients wanting the same drv build it once, in 5.82 seconds)5099subtest: the first client leaving does not fail the second5100client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share5101client # [ 62.045966] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share.5102client # Running as unit: share1.service; invocation ID: 15afaff7918143b082044079448cf4c55103client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share, in 0.07 seconds)5104client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share5105client # [ 62.243717] nix[993]: don't know how to build these paths:5106client # [ 62.244546] nix[993]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5107worker2 # [ 62.225796] nix-grpc-daemon[786]: ts=2026-09-22T10:47:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:349525108worker2 # [ 62.236539] nix-grpc-daemon[786]: ts=2026-09-22T10:47:19Z level=debug event=want client=6 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05109worker2 # [ 62.240797] nix-grpc-daemon[786]: ts=2026-09-22T10:47:19Z level=debug event=assigned drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv worker=192.168.1.3:50051 clients=15110worker1 # [ 62.310323] nix-grpc-daemon[784]: ts=2026-09-22T10:47:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357845111worker1 # [ 62.317599] nix-grpc-daemon[784]: ts=2026-09-22T10:47:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357845112client # [ 62.310197] nix[993]: copying 1 paths to 'lb:50051'...5113client # [ 62.310943] nix[993]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' to 'lb:50051'...5114worker1 # [ 62.323385] nix-grpc-daemon[784]: ts=2026-09-22T10:47:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357845115worker1 # [ 62.328493] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5116worker1 # [ 62.483046] nix-daemon[852]: accepted connection from pid 1052, user nix-grpc-daemon (trusted)5117worker1 # [ 62.497265] nix-daemon[852]: reaped child process 1055, status = succeeded5118worker1 # [ 62.636602] nix-daemon[852]: accepted connection from pid 1057, user nix-grpc-daemon (trusted)5119worker1 # [ 62.650426] nix-daemon[852]: reaped child process 1060, status = succeeded5120lb # [ 62.637562] niks3-server[1053]: 2026/09/22 10:47:19 INFO Received uploads request method=POST path=/api/pending_closures5121worker1 # [ 62.666886] nix-grpc-daemon[864]: time=2026-09-22T10:47:19.788Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5122worker1 # [ 62.668201] nix-grpc-daemon[864]: time=2026-09-22T10:47:19.788Z level=INFO msg="Uploading j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv (432B)"5123lb # [ 62.653178] niks3-server[1053]: 2026/09/22 10:47:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign5124worker1 # [ 62.679751] nix-grpc-daemon[864]: time=2026-09-22T10:47:19.801Z level=INFO msg="Uploading 1 narinfos"5125lb # [ 62.656192] niks3-server[1053]: 2026/09/22 10:47:19 INFO Signed narinfos id=11 count=15126lb # [ 62.658504] niks3-server[1053]: 2026/09/22 10:47:19 INFO Registered completed upload object_key=nar/0s0b0gfhn36p0aylk1qa64cgw01bi7a8ca93iv4zg2q7wqcjcqhs.nar.zst5127lb # [ 62.661807] niks3-server[1053]: 2026/09/22 10:47:19 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.ls5128lb # [ 62.665900] niks3-server[1053]: 2026/09/22 10:47:19 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete5129lb # [ 62.669059] niks3-server[1053]: 2026/09/22 10:47:19 INFO Completed upload id=115130worker1 # [ 62.692976] nix-grpc-daemon[864]: time=2026-09-22T10:47:19.815Z level=INFO msg="Upload complete. (357ms)"5131worker1 # [ 62.695194] nix-grpc-daemon[784]: ts=2026-09-22T10:47:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=0 paths=1 nar_bytes_in=4325132lb # [ 62.672362] niks3-server[1053]: 2026/09/22 10:47:19 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.narinfo5133worker1 # [ 62.698807] nix-grpc-daemon[784]: ts=2026-09-22T10:47:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357845134worker1 # [ 62.706189] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5135worker1 # [ 62.708436] nix-daemon[852]: reaped child process 1050, status = succeeded5136worker1 # [ 62.713500] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5137client # [ 62.700413] nix[993]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5138client # [ 63.120670] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share.5139client # Running as unit: share2.service; invocation ID: 4a3508a9ac7b4ed3ae59268306b2bcff5140client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag share, in 1.08 seconds)5141client # [ 63.342363] nix[1009]: don't know how to build these paths:5142client # [ 63.343265] nix[1009]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5143worker2 # [ 63.323358] nix-grpc-daemon[786]: ts=2026-09-22T10:47:20Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:349525144worker2 # [ 63.334420] nix-grpc-daemon[786]: ts=2026-09-22T10:47:20Z level=debug event=want client=7 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05145worker1 # [ 63.368818] nix-grpc-daemon[784]: ts=2026-09-22T10:47:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357845146client # [ 63.361969] nix[1009]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5147client: must succeed: sleep 2; systemctl kill -s INT share15148client # [ 66.410713] systemd[1]: share1.service: Sent signal SIGINT to main process 993 (nix) on client request.5149client # [ 66.417878] nix[993]: shutting downj2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5150client # [ 66.423351] nix[993]: [mutex.cc : 2506] RAW: thread should hold at least a read lock on Mutex 0x7ce4d4dfccb85151client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.06 seconds)5152client: waiting for success: ! systemctl is-active share15153client # [ 66.446468] systemd-coredump[1024]: Process 993 (nix) of user 0 terminated abnormally with signal 6/ABRT, processing...5154client # [ 66.464097] systemd[1]: Created slice Slice /system/systemd-coredump.5155client # [ 66.469512] systemd[1]: Started Process Core Dump (PID 1024/UID 0).5156client # [ 68.291043] systemd-coredump[1029]: Cannot store coredump of 993 (nix): No space left on device5157client # [ 68.384109] systemd-coredump[1029]: Process 993 (nix) of user 0 terminated abnormally without generating a coredump.5158client # [ 68.398816] systemd[1]: systemd-coredump@0-1-1024_1025-0.service: Deactivated successfully.5159client # [ 68.405359] systemd[1]: systemd-coredump@0-1-1024_1025-0.service: Consumed 1.556s CPU time over 1.923s wall clock time, 969M memory peak, 931M written to disk.5160client # [ 68.417578] systemd[1]: share1.service: Main process exited, code=dumped, status=6/ABRT5161client # [ 68.418973] systemd[1]: share1.service: Failed with result 'core-dump'.5162client # [ 68.420792] systemd[1]: share1.service: Consumed 265ms CPU time over 6.371s wall clock time, 43.5M memory peak, 4K written to disk, 6.2K incoming IP traffic, 8.1K outgoing IP traffic.5163client: (finished: waiting for success: ! systemctl is-active share1, in 2.15 seconds)5164lb # [ 68.679503] envoy[932]: conn peer=[2001:db8:1::1]:34810 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3011 tx=1563 ms=46755165lb # [ 68.683257] envoy[932]: conn peer=[2001:db8:1::1]:34826 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1440 tx=1197 ms=35825166lb # [ 68.686754] envoy[932]: conn peer=[2001:db8:1::1]:34834 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2982 tx=787 ms=61585167client: waiting for success: ! systemctl is-active share25168worker1 # [ 68.934335] nix-daemon[852]: reaped child process 1064, status = succeeded5169worker1 # [ 69.086325] nix-daemon[852]: accepted connection from pid 1098, user nix-grpc-daemon (trusted)5170worker1 # [ 69.100953] nix-daemon[852]: reaped child process 1101, status = succeeded5171lb # [ 69.091864] niks3-server[1053]: 2026/09/22 10:47:25 INFO Received uploads request method=POST path=/api/pending_closures5172worker1 # [ 69.124220] nix-grpc-daemon[864]: time=2026-09-22T10:47:26.246Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5173worker1 # [ 69.127067] nix-grpc-daemon[864]: time=2026-09-22T10:47:26.248Z level=INFO msg="Uploading b9lg6znd0rinl36gy6ixq1dcj6b8bhhw-slow-share (120B)"5174lb # [ 69.132219] niks3-server[1053]: 2026/09/22 10:47:25 INFO Registered completed upload object_key=log/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5175lb # [ 69.139633] niks3-server[1053]: 2026/09/22 10:47:25 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst5176lb # [ 69.153744] niks3-server[1053]: 2026/09/22 10:47:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign5177lb # [ 69.161413] niks3-server[1053]: 2026/09/22 10:47:26 INFO Signed narinfos id=12 count=15178worker1 # [ 69.188473] nix-grpc-daemon[864]: time=2026-09-22T10:47:26.310Z level=INFO msg="Uploading 1 narinfos"5179lb # [ 69.164383] niks3-server[1053]: 2026/09/22 10:47:26 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.ls5180lb # [ 69.174680] niks3-server[1053]: 2026/09/22 10:47:26 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete5181lb # [ 69.178450] niks3-server[1053]: 2026/09/22 10:47:26 INFO Completed upload id=125182worker1 # [ 69.202465] nix-grpc-daemon[864]: time=2026-09-22T10:47:26.324Z level=INFO msg="Upload complete. (276ms)"5183lb # [ 69.181125] niks3-server[1053]: 2026/09/22 10:47:26 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.narinfo5184worker1 # [ 69.207645] nix-grpc-daemon[784]: ts=2026-09-22T10:47:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=5 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=6 outputs=15185worker1 # [ 69.213530] nix-grpc-daemon[784]: ts=2026-09-22T10:47:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=6 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=6 outputs=15186worker1 # [ 69.226212] nix-daemon[852]: reaped child process 1062, status = succeeded5187client # [ 69.277164] systemd[1]: share2.service: Deactivated successfully.5188client # [ 69.281335] systemd[1]: share2.service: Consumed 123ms CPU time over 6.159s wall clock time, 35.4M memory peak, 4K written to disk, 6.1K incoming IP traffic, 5.7K outgoing IP traffic.5189client: (finished: waiting for success: ! systemctl is-active share2, in 1.11 seconds)5190client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }5191client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.04 seconds)5192(finished: subtest: the first client leaving does not fail the second, in 7.98 seconds)5193subtest: build hook: nix-daemon with builders = grpc://lb5194client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log5195client # [ 70.133221] systemd[1]: Started Nix Daemon.5196client # [ 70.284779] nix-daemon[1070]: accepted connection from pid 1067, user root (trusted)5197worker1 # [ 70.484526] nix-grpc-daemon[784]: ts=2026-09-22T10:47:27Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:357845198worker2 # [ 70.458670] nix-grpc-daemon[786]: ts=2026-09-22T10:47:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349525199worker2 # [ 70.475124] nix-grpc-daemon[786]: ts=2026-09-22T10:47:27Z level=debug event=want client=8 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05200worker2 # [ 70.480644] nix-grpc-daemon[786]: ts=2026-09-22T10:47:27Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.4:50051 clients=15201worker2 # [ 70.520195] nix-grpc-daemon[786]: ts=2026-09-22T10:47:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349525202worker2 # [ 70.533837] nix-grpc-daemon[786]: ts=2026-09-22T10:47:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349525203worker2 # [ 70.545970] nix-grpc-daemon[786]: ts=2026-09-22T10:47:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349525204worker2 # [ 70.558582] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5205worker2 # [ 70.761154] nix-daemon[873]: accepted connection from pid 1022, user nix-grpc-daemon (trusted)5206worker2 # [ 70.776443] nix-daemon[873]: reaped child process 1025, status = succeeded5207worker2 # [ 70.937102] nix-daemon[873]: accepted connection from pid 1027, user nix-grpc-daemon (trusted)5208worker2 # [ 70.952223] nix-daemon[873]: reaped child process 1030, status = succeeded5209lb # [ 70.975711] niks3-server[1053]: 2026/09/22 10:47:27 INFO Received uploads request method=POST path=/api/pending_closures5210worker2 # [ 70.979968] nix-grpc-daemon[927]: time=2026-09-22T10:47:28.157Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5211worker2 # [ 70.982172] nix-grpc-daemon[927]: time=2026-09-22T10:47:28.158Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5212lb # [ 71.009158] niks3-server[1053]: 2026/09/22 10:47:27 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5213lb # [ 71.018560] niks3-server[1053]: 2026/09/22 10:47:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign5214worker2 # [ 71.017652] nix-grpc-daemon[927]: time=2026-09-22T10:47:28.194Z level=INFO msg="Uploading 1 narinfos"5215lb # [ 71.023828] niks3-server[1053]: 2026/09/22 10:47:27 INFO Signed narinfos id=13 count=15216lb # [ 71.030320] niks3-server[1053]: 2026/09/22 10:47:27 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5217lb # [ 71.039660] niks3-server[1053]: 2026/09/22 10:47:27 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete5218worker2 # [ 71.039628] nix-grpc-daemon[927]: time=2026-09-22T10:47:28.217Z level=INFO msg="Upload complete. (468ms)"5219lb # [ 71.046537] niks3-server[1053]: 2026/09/22 10:47:27 INFO Completed upload id=135220worker2 # [ 71.042401] nix-grpc-daemon[786]: ts=2026-09-22T10:47:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=0 paths=1 nar_bytes_in=5125221lb # [ 71.051386] niks3-server[1053]: 2026/09/22 10:47:27 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo5222worker2 # [ 71.047945] nix-grpc-daemon[786]: ts=2026-09-22T10:47:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349525223worker2 # [ 71.058066] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5224worker2 # [ 71.064162] nix-daemon[873]: reaped child process 1020, status = succeeded5225worker2 # [ 71.068529] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5226worker2 # [ 71.894917] nix-daemon[873]: reaped child process 1034, status = succeeded5227worker2 # [ 72.052971] nix-daemon[873]: accepted connection from pid 1050, user nix-grpc-daemon (trusted)5228worker2 # [ 72.065907] nix-daemon[873]: reaped child process 1053, status = succeeded5229lb # [ 72.088859] niks3-server[1053]: 2026/09/22 10:47:28 INFO Received uploads request method=POST path=/api/pending_closures5230worker2 # [ 72.090560] nix-grpc-daemon[927]: time=2026-09-22T10:47:29.267Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5231worker2 # [ 72.092884] nix-grpc-daemon[927]: time=2026-09-22T10:47:29.269Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5232lb # [ 72.117298] niks3-server[1053]: 2026/09/22 10:47:28 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5233lb # [ 72.133144] niks3-server[1053]: 2026/09/22 10:47:28 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5234lb # [ 72.144880] niks3-server[1053]: 2026/09/22 10:47:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign5235lb # [ 72.149418] niks3-server[1053]: 2026/09/22 10:47:28 INFO Signed narinfos id=14 count=15236worker2 # [ 72.146292] nix-grpc-daemon[927]: time=2026-09-22T10:47:29.323Z level=INFO msg="Uploading 1 narinfos"5237lb # [ 72.154320] niks3-server[1053]: 2026/09/22 10:47:28 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5238lb # [ 72.164932] niks3-server[1053]: 2026/09/22 10:47:29 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete5239lb # [ 72.172334] niks3-server[1053]: 2026/09/22 10:47:29 INFO Completed upload id=145240worker2 # [ 72.166298] nix-grpc-daemon[927]: time=2026-09-22T10:47:29.343Z level=INFO msg="Upload complete. (290ms)"5241lb # [ 72.174643] niks3-server[1053]: 2026/09/22 10:47:29 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5242worker2 # [ 72.172980] nix-grpc-daemon[786]: ts=2026-09-22T10:47:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=7 outputs=15243worker1 # [ 72.208856] nix-grpc-daemon[784]: ts=2026-09-22T10:47:29Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:357845244worker2 # [ 72.187281] nix-daemon[873]: reaped child process 1032, status = succeeded5245worker1 # [ 72.217756] nix-grpc-daemon[784]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5246worker1 # [ 72.240373] nix-grpc-daemon[784]: ts=2026-09-22T10:47:29Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:357845247worker1 # [ 72.246094] nix-grpc-daemon[784]: ts=2026-09-22T10:47:29Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=0 paths=1 nar_bytes_out=1285248worker2 # [ 72.489687] nix-grpc-daemon[786]: ts=2026-09-22T10:47:29Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:349525249worker2 # [ 72.492228] nix-grpc-daemon[786]: ts=2026-09-22T10:47:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349525250worker2 # [ 72.498255] nix-grpc-daemon[927]: time=2026-09-22T10:47:29.675Z level=INFO msg="All 1 paths already cached"5251worker2 # [ 72.508834] nix-grpc-daemon[786]: ts=2026-09-22T10:47:29Z level=debug event=want client=9 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05252worker2 # [ 72.510811] nix-grpc-daemon[786]: ts=2026-09-22T10:47:29Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.4:50051 clients=15253worker2 # [ 72.513772] nix-grpc-daemon[786]: ts=2026-09-22T10:47:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349525254worker2 # [ 72.521562] nix-grpc-daemon[786]: ts=2026-09-22T10:47:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349525255worker2 # [ 72.530445] nix-grpc-daemon[927]: time=2026-09-22T10:47:29.707Z level=INFO msg="All 1 paths already cached"5256worker2 # [ 72.534870] nix-grpc-daemon[786]: ts=2026-09-22T10:47:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349525257worker2 # [ 72.538483] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5258worker2 # [ 72.683737] nix-daemon[873]: accepted connection from pid 1059, user nix-grpc-daemon (trusted)5259worker2 # [ 72.697189] nix-daemon[873]: reaped child process 1062, status = succeeded5260worker2 # [ 72.846698] nix-daemon[873]: accepted connection from pid 1064, user nix-grpc-daemon (trusted)5261worker2 # [ 72.859451] nix-daemon[873]: reaped child process 1067, status = succeeded5262lb # [ 72.880822] niks3-server[1053]: 2026/09/22 10:47:29 INFO Received uploads request method=POST path=/api/pending_closures5263worker2 # [ 72.879918] nix-grpc-daemon[927]: time=2026-09-22T10:47:30.057Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5264worker2 # [ 72.881277] nix-grpc-daemon[927]: time=2026-09-22T10:47:30.057Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5265lb # [ 72.905883] niks3-server[1053]: 2026/09/22 10:47:29 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5266lb # [ 72.915138] niks3-server[1053]: 2026/09/22 10:47:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5267lb # [ 72.921236] niks3-server[1053]: 2026/09/22 10:47:29 INFO Signed narinfos id=15 count=15268worker2 # [ 72.918608] nix-grpc-daemon[927]: time=2026-09-22T10:47:30.095Z level=INFO msg="Uploading 1 narinfos"5269lb # [ 72.926947] niks3-server[1053]: 2026/09/22 10:47:29 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5270lb # [ 72.937712] niks3-server[1053]: 2026/09/22 10:47:29 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5271worker2 # [ 72.935975] nix-grpc-daemon[927]: time=2026-09-22T10:47:30.113Z level=INFO msg="Upload complete. (390ms)"5272lb # [ 72.942780] niks3-server[1053]: 2026/09/22 10:47:29 INFO Completed upload id=155273worker2 # [ 72.937954] nix-grpc-daemon[786]: ts=2026-09-22T10:47:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=0 paths=1 nar_bytes_in=6245274lb # [ 72.947560] niks3-server[1053]: 2026/09/22 10:47:29 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5275worker2 # [ 72.944446] nix-grpc-daemon[786]: ts=2026-09-22T10:47:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349525276worker2 # [ 72.948628] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5277worker2 # [ 72.950974] nix-daemon[873]: reaped child process 1057, status = succeeded5278worker2 # [ 72.956384] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5279worker2 # [ 73.778503] nix-daemon[873]: reaped child process 1071, status = succeeded5280worker2 # [ 73.934670] nix-daemon[873]: accepted connection from pid 1087, user nix-grpc-daemon (trusted)5281worker2 # [ 73.945378] nix-daemon[873]: reaped child process 1090, status = succeeded5282lb # [ 73.969397] niks3-server[1053]: 2026/09/22 10:47:30 INFO Received uploads request method=POST path=/api/pending_closures5283worker2 # [ 73.969897] nix-grpc-daemon[927]: time=2026-09-22T10:47:31.147Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5284worker2 # [ 73.971725] nix-grpc-daemon[927]: time=2026-09-22T10:47:31.148Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5285lb # [ 73.998320] niks3-server[1053]: 2026/09/22 10:47:30 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5286lb # [ 74.011927] niks3-server[1053]: 2026/09/22 10:47:30 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5287lb # [ 74.022860] niks3-server[1053]: 2026/09/22 10:47:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5288worker2 # [ 74.020234] nix-grpc-daemon[927]: time=2026-09-22T10:47:31.197Z level=INFO msg="Uploading 1 narinfos"5289lb # [ 74.026730] niks3-server[1053]: 2026/09/22 10:47:30 INFO Signed narinfos id=16 count=15290lb # [ 74.033147] niks3-server[1053]: 2026/09/22 10:47:30 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5291lb # [ 74.044127] niks3-server[1053]: 2026/09/22 10:47:30 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5292lb # [ 74.048945] niks3-server[1053]: 2026/09/22 10:47:30 INFO Completed upload id=165293lb # [ 74.051250] niks3-server[1053]: 2026/09/22 10:47:30 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5294worker2 # [ 74.043858] nix-grpc-daemon[927]: time=2026-09-22T10:47:31.220Z level=INFO msg="Upload complete. (277ms)"5295worker2 # [ 74.049685] nix-grpc-daemon[786]: ts=2026-09-22T10:47:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=8 outputs=15296worker2 # [ 74.062404] nix-daemon[873]: reaped child process 1069, status = succeeded5297worker2 # [ 74.094341] nix-grpc-daemon[786]: ts=2026-09-22T10:47:31Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:349525298worker2 # [ 74.112129] nix-grpc-daemon[786]: ts=2026-09-22T10:47:31Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:349525299worker2 # [ 74.116581] nix-grpc-daemon[786]: ts=2026-09-22T10:47:31Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=0 paths=1 nar_bytes_out=1765300worker1 # [ 74.412046] nix-grpc-daemon[784]: ts=2026-09-22T10:47:31Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:357845301worker1 # [ 74.415302] nix-grpc-daemon[784]: ts=2026-09-22T10:47:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357845302worker1 # [ 74.421481] nix-grpc-daemon[864]: time=2026-09-22T10:47:31.543Z level=INFO msg="All 1 paths already cached"5303worker2 # [ 74.403334] nix-grpc-daemon[786]: ts=2026-09-22T10:47:31Z level=debug event=want client=10 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05304worker2 # [ 74.408298] nix-grpc-daemon[786]: ts=2026-09-22T10:47:31Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.4:50051 clients=15305worker2 # [ 74.413244] nix-grpc-daemon[786]: ts=2026-09-22T10:47:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349525306worker2 # [ 74.421782] nix-grpc-daemon[786]: ts=2026-09-22T10:47:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349525307worker2 # [ 74.431160] nix-grpc-daemon[927]: time=2026-09-22T10:47:31.608Z level=INFO msg="All 2 paths already cached"5308worker2 # [ 74.437609] nix-grpc-daemon[786]: ts=2026-09-22T10:47:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349525309worker2 # [ 74.447157] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5310worker2 # [ 74.604586] nix-daemon[873]: accepted connection from pid 1096, user nix-grpc-daemon (trusted)5311worker2 # [ 74.617503] nix-daemon[873]: reaped child process 1099, status = succeeded5312worker2 # [ 74.767534] nix-daemon[873]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5313worker2 # [ 74.780430] nix-daemon[873]: reaped child process 1104, status = succeeded5314lb # [ 74.800189] niks3-server[1053]: 2026/09/22 10:47:31 INFO Received uploads request method=POST path=/api/pending_closures5315worker2 # [ 74.802550] nix-grpc-daemon[927]: time=2026-09-22T10:47:31.979Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5316worker2 # [ 74.803964] nix-grpc-daemon[927]: time=2026-09-22T10:47:31.979Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5317lb # [ 74.823542] niks3-server[1053]: 2026/09/22 10:47:31 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5318lb # [ 74.835873] niks3-server[1053]: 2026/09/22 10:47:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5319lb # [ 74.842447] niks3-server[1053]: 2026/09/22 10:47:31 INFO Signed narinfos id=17 count=15320worker2 # [ 74.839299] nix-grpc-daemon[927]: time=2026-09-22T10:47:32.016Z level=INFO msg="Uploading 1 narinfos"5321lb # [ 74.847145] niks3-server[1053]: 2026/09/22 10:47:31 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5322lb # [ 74.858304] niks3-server[1053]: 2026/09/22 10:47:31 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5323worker2 # [ 74.857931] nix-grpc-daemon[927]: time=2026-09-22T10:47:32.035Z level=INFO msg="Upload complete. (399ms)"5324lb # [ 74.864312] niks3-server[1053]: 2026/09/22 10:47:31 INFO Completed upload id=175325worker2 # [ 74.859948] nix-grpc-daemon[786]: ts=2026-09-22T10:47:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=0 paths=1 nar_bytes_in=7525326lb # [ 74.868947] niks3-server[1053]: 2026/09/22 10:47:31 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5327worker2 # [ 74.866070] nix-daemon[873]: reaped child process 1094, status = succeeded5328worker2 # [ 74.867761] nix-grpc-daemon[786]: ts=2026-09-22T10:47:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349525329worker2 # [ 74.871572] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5330worker2 # [ 74.876992] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5331worker2 # [ 76.100547] nix-daemon[873]: reaped child process 1108, status = succeeded5332worker2 # [ 76.227286] nix-daemon[873]: accepted connection from pid 1124, user nix-grpc-daemon (trusted)5333worker2 # [ 76.239334] nix-daemon[873]: reaped child process 1127, status = succeeded5334lb # [ 76.256965] niks3-server[1053]: 2026/09/22 10:47:33 INFO Received uploads request method=POST path=/api/pending_closures5335worker2 # [ 76.259986] nix-grpc-daemon[927]: time=2026-09-22T10:47:33.437Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5336worker2 # [ 76.261547] nix-grpc-daemon[927]: time=2026-09-22T10:47:33.438Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5337lb # [ 76.297145] niks3-server[1053]: 2026/09/22 10:47:33 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5338lb # [ 76.302230] niks3-server[1053]: 2026/09/22 10:47:33 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5339lb # [ 76.316473] niks3-server[1053]: 2026/09/22 10:47:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5340lb # [ 76.322467] niks3-server[1053]: 2026/09/22 10:47:33 INFO Signed narinfos id=18 count=15341worker2 # [ 76.318727] nix-grpc-daemon[927]: time=2026-09-22T10:47:33.495Z level=INFO msg="Uploading 1 narinfos"5342lb # [ 76.326551] niks3-server[1053]: 2026/09/22 10:47:33 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5343lb # [ 76.336796] niks3-server[1053]: 2026/09/22 10:47:33 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5344lb # [ 76.342784] niks3-server[1053]: 2026/09/22 10:47:33 INFO Completed upload id=185345worker2 # [ 76.336319] nix-grpc-daemon[927]: time=2026-09-22T10:47:33.513Z level=INFO msg="Upload complete. (250ms)"5346lb # [ 76.345361] niks3-server[1053]: 2026/09/22 10:47:33 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5347worker2 # [ 76.343720] nix-grpc-daemon[786]: ts=2026-09-22T10:47:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=9 outputs=15348worker2 # [ 76.350806] nix-grpc-daemon[786]: ts=2026-09-22T10:47:33Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:349525349worker2 # [ 76.360270] nix-grpc-daemon[786]: ts=2026-09-22T10:47:33Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:349525350worker2 # [ 76.363274] nix-grpc-daemon[786]: ts=2026-09-22T10:47:33Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=0 paths=1 nar_bytes_out=2325351worker2 # [ 76.368513] nix-daemon[873]: reaped child process 1106, status = succeeded5352client # [ 76.469487] nix-daemon[1070]: reaped child process 1072, status = succeeded5353client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.50 seconds)5354client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5355client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.02 seconds)5356client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5357client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.03 seconds)5358client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5359client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.02 seconds)5360(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 6.58 seconds)5361subtest: an input only one worker has still reaches the builder5362client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only5363client # this derivation will be built:5364client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5365client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5366client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.36 seconds)5367client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5368client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.16 seconds)5369worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5370worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.19 seconds)5371worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5372worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)5373client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&25374client # [ 77.456262] nix-daemon[1070]: accepted connection from pid 1152, user root (trusted)5375client # this derivation will be built:5376client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5377worker2 # [ 77.623105] nix-grpc-daemon[786]: ts=2026-09-22T10:47:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:349525378client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5379worker1 # [ 77.659332] nix-grpc-daemon[784]: ts=2026-09-22T10:47:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357845380worker1 # [ 77.813338] nix-daemon[852]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5381worker1 # [ 77.824806] nix-daemon[852]: reaped child process 1112, status = succeeded5382lb # [ 77.811655] niks3-server[1053]: 2026/09/22 10:47:34 INFO Received uploads request method=POST path=/api/pending_closures5383worker1 # [ 77.843549] nix-grpc-daemon[864]: time=2026-09-22T10:47:34.965Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5384worker1 # [ 77.844919] nix-grpc-daemon[864]: time=2026-09-22T10:47:34.965Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5385lb # [ 77.835602] niks3-server[1053]: 2026/09/22 10:47:34 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5386lb # [ 77.846893] niks3-server[1053]: 2026/09/22 10:47:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5387worker1 # [ 77.875355] nix-grpc-daemon[864]: time=2026-09-22T10:47:34.997Z level=INFO msg="Uploading 1 narinfos"5388lb # [ 77.851459] niks3-server[1053]: 2026/09/22 10:47:34 INFO Signed narinfos id=19 count=15389lb # [ 77.862090] niks3-server[1053]: 2026/09/22 10:47:34 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5390worker1 # [ 77.894282] nix-grpc-daemon[864]: time=2026-09-22T10:47:35.016Z level=INFO msg="Upload complete. (236ms)"5391lb # [ 77.870890] niks3-server[1053]: 2026/09/22 10:47:34 INFO Completed upload id=195392lb # [ 77.874450] niks3-server[1053]: 2026/09/22 10:47:34 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5393lb # [ 77.878460] niks3-server[1053]: 2026/09/22 10:47:34 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5394worker2 # [ 77.881312] nix-grpc-daemon[786]: ts=2026-09-22T10:47:35Z level=debug event=want client=11 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05395worker2 # [ 77.883394] nix-grpc-daemon[786]: ts=2026-09-22T10:47:35Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=15396worker1 # [ 77.955414] nix-grpc-daemon[784]: ts=2026-09-22T10:47:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357845397worker1 # [ 77.967794] nix-grpc-daemon[784]: ts=2026-09-22T10:47:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357845398client # copying 2 paths to 'lb:50051'...5399client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5400worker1 # [ 77.983578] nix-grpc-daemon[784]: ts=2026-09-22T10:47:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357845401client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5402worker1 # [ 77.993343] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5403worker1 # [ 78.164968] nix-daemon[852]: accepted connection from pid 1116, user nix-grpc-daemon (trusted)5404worker1 # [ 78.178261] nix-daemon[852]: reaped child process 1119, status = succeeded5405worker1 # [ 78.342323] nix-daemon[852]: accepted connection from pid 1121, user nix-grpc-daemon (trusted)5406worker1 # [ 78.355146] nix-daemon[852]: reaped child process 1124, status = succeeded5407lb # [ 78.346740] niks3-server[1053]: 2026/09/22 10:47:35 INFO Received uploads request method=POST path=/api/pending_closures5408lb # [ 78.358708] niks3-server[1053]: 2026/09/22 10:47:35 INFO Received uploads request method=POST path=/api/pending_closures5409worker1 # [ 78.390801] nix-grpc-daemon[864]: time=2026-09-22T10:47:35.512Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5410worker1 # [ 78.392232] nix-grpc-daemon[864]: time=2026-09-22T10:47:35.512Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5411worker1 # [ 78.393809] nix-grpc-daemon[864]: time=2026-09-22T10:47:35.512Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5412lb # [ 78.387404] niks3-server[1053]: 2026/09/22 10:47:35 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5413lb # [ 78.398907] niks3-server[1053]: 2026/09/22 10:47:35 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5414lb # [ 78.414764] niks3-server[1053]: 2026/09/22 10:47:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5415lb # [ 78.418885] niks3-server[1053]: 2026/09/22 10:47:35 INFO Signed narinfos id=20 count=15416lb # [ 78.423309] niks3-server[1053]: 2026/09/22 10:47:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5417lb # [ 78.426352] niks3-server[1053]: 2026/09/22 10:47:35 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5418lb # [ 78.428826] niks3-server[1053]: 2026/09/22 10:47:35 INFO Signed narinfos id=21 count=25419lb # [ 78.430650] niks3-server[1053]: 2026/09/22 10:47:35 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5420worker1 # [ 78.453831] nix-grpc-daemon[864]: time=2026-09-22T10:47:35.575Z level=INFO msg="Uploading 3 narinfos"5421lb # [ 78.441068] niks3-server[1053]: 2026/09/22 10:47:35 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5422lb # [ 78.446591] niks3-server[1053]: 2026/09/22 10:47:35 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5423lb # [ 78.448044] niks3-server[1053]: 2026/09/22 10:47:35 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5424lb # [ 78.450994] niks3-server[1053]: 2026/09/22 10:47:35 INFO Completed upload id=205425lb # [ 78.452777] niks3-server[1053]: 2026/09/22 10:47:35 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5426lb # [ 78.455651] niks3-server[1053]: 2026/09/22 10:47:35 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5427lb # [ 78.458637] niks3-server[1053]: 2026/09/22 10:47:35 INFO Completed upload id=215428worker1 # [ 78.482709] nix-grpc-daemon[864]: time=2026-09-22T10:47:35.605Z level=INFO msg="Upload complete. (475ms)"5429worker1 # [ 78.487771] nix-grpc-daemon[784]: ts=2026-09-22T10:47:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=0 paths=2 nar_bytes_in=9765430worker1 # [ 78.493359] nix-grpc-daemon[784]: ts=2026-09-22T10:47:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357845431worker1 # [ 78.505175] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5432worker1 # [ 78.517870] nix-daemon[852]: reaped child process 1114, status = succeeded5433client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5434worker1 # [ 78.523556] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5435client # uses-input-w1onlya> node-a: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5436lb # [ 78.680968] envoy[932]: conn peer=[2001:db8:1::1]:59666 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1453 tx=1197 ms=59085437lb # [ 78.684590] envoy[932]: conn peer=[2001:db8:1::1]:59684 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=333 ms=575438lb # [ 78.687933] envoy[932]: conn peer=[2001:db8:1::1]:59680 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3397 tx=1939 ms=18165439lb # [ 78.691423] envoy[932]: conn peer=[2001:db8:1::1]:59700 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3866 tx=2154 ms=16785440lb # [ 78.694885] envoy[932]: conn peer=[2001:db8:1::1]:40372 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=555441lb # [ 78.698420] envoy[932]: conn peer=[2001:db8:1::1]:40388 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4339 tx=2389 ms=20265442lb # [ 78.701809] envoy[932]: conn peer=[2001:db8:1::1]:40402 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=485443client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5444worker1 # [ 79.546226] nix-daemon[852]: reaped child process 1128, status = succeeded5445worker1 # [ 79.700581] nix-daemon[852]: accepted connection from pid 1144, user nix-grpc-daemon (trusted)5446worker1 # [ 79.712075] nix-daemon[852]: reaped child process 1147, status = succeeded5447lb # [ 79.699354] niks3-server[1053]: 2026/09/22 10:47:36 INFO Received uploads request method=POST path=/api/pending_closures5448worker1 # [ 79.730176] nix-grpc-daemon[864]: time=2026-09-22T10:47:36.852Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5449lb # [ 79.727238] niks3-server[1053]: 2026/09/22 10:47:36 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5450lb # [ 79.736583] niks3-server[1053]: 2026/09/22 10:47:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5451lb # [ 79.741694] niks3-server[1053]: 2026/09/22 10:47:36 INFO Signed narinfos id=22 count=15452worker1 # [ 79.768450] nix-grpc-daemon[864]: time=2026-09-22T10:47:36.890Z level=INFO msg="Uploading 1 narinfos"5453lb # [ 79.747861] niks3-server[1053]: 2026/09/22 10:47:36 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5454lb # [ 79.759556] niks3-server[1053]: 2026/09/22 10:47:36 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5455worker1 # [ 79.791155] nix-grpc-daemon[864]: time=2026-09-22T10:47:36.913Z level=INFO msg="Upload complete. (258ms)"5456lb # [ 79.767293] niks3-server[1053]: 2026/09/22 10:47:36 INFO Completed upload id=225457lb # [ 79.769452] niks3-server[1053]: 2026/09/22 10:47:36 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5458worker1 # [ 79.793941] nix-grpc-daemon[784]: ts=2026-09-22T10:47:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=10 outputs=15459worker1 # [ 79.801614] nix-daemon[852]: reaped child process 1126, status = succeeded5460client # copying 1 paths...5461worker2 # [ 79.816144] nix-grpc-daemon[786]: ts=2026-09-22T10:47:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:349525462worker2 # [ 79.824887] nix-grpc-daemon[786]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5463client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5464worker1 # [ 79.877051] nix-grpc-daemon[784]: ts=2026-09-22T10:47:36Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:357845465worker1 # [ 79.880573] nix-grpc-daemon[784]: ts=2026-09-22T10:47:36Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=0 paths=1 nar_bytes_out=1205466client # [ 79.969569] nix-daemon[1070]: reaped child process 1155, status = succeeded5467client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.69 seconds)5468client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25469client # [ 80.156048] nix-daemon[1070]: accepted connection from pid 1174, user root (trusted)5470client # this derivation will be built:5471client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5472worker2 # [ 80.322081] nix-grpc-daemon[786]: ts=2026-09-22T10:47:37Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:349525473worker2 # [ 80.324299] nix-grpc-daemon[786]: ts=2026-09-22T10:47:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349525474client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5475worker2 # [ 80.341488] nix-grpc-daemon[786]: ts=2026-09-22T10:47:37Z level=debug event=want client=12 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05476worker2 # [ 80.343822] nix-grpc-daemon[786]: ts=2026-09-22T10:47:37Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=15477worker2 # [ 80.385802] nix-grpc-daemon[786]: ts=2026-09-22T10:47:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349525478worker2 # [ 80.393921] nix-grpc-daemon[786]: ts=2026-09-22T10:47:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349525479client # copying 1 paths to 'lb:50051'...5480worker2 # [ 80.406591] nix-grpc-daemon[786]: ts=2026-09-22T10:47:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349525481client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5482worker2 # [ 80.414443] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5483worker2 # [ 80.422537] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 3 ms (attempt 1/5)5484worker2 # [ 80.428061] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 130 ms (attempt 2/5)5485worker2 # [ 80.560726] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 366 ms (attempt 3/5)5486worker2 # [ 80.930421] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 314 ms (attempt 4/5)5487worker2 # [ 81.250681] nix-grpc-daemon[786]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5488worker2 # [ 81.254394] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5489worker2 # [ 81.259935] nix-grpc-daemon[786]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5490worker2 # [ 81.443527] nix-daemon[873]: accepted connection from pid 1144, user nix-grpc-daemon (trusted)5491worker2 # [ 81.455373] nix-daemon[873]: reaped child process 1147, status = succeeded5492worker2 # [ 81.597062] nix-daemon[873]: accepted connection from pid 1149, user nix-grpc-daemon (trusted)5493worker2 # [ 81.609255] nix-daemon[873]: reaped child process 1152, status = succeeded5494lb # [ 81.628440] niks3-server[1053]: 2026/09/22 10:47:38 INFO Received uploads request method=POST path=/api/pending_closures5495worker2 # [ 81.633476] nix-grpc-daemon[927]: time=2026-09-22T10:47:38.810Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5496worker2 # [ 81.634949] nix-grpc-daemon[927]: time=2026-09-22T10:47:38.810Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5497lb # [ 81.658089] niks3-server[1053]: 2026/09/22 10:47:38 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5498lb # [ 81.670086] niks3-server[1053]: 2026/09/22 10:47:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5499worker2 # [ 81.668703] nix-grpc-daemon[927]: time=2026-09-22T10:47:38.846Z level=INFO msg="Uploading 1 narinfos"5500lb # [ 81.675080] niks3-server[1053]: 2026/09/22 10:47:38 INFO Signed narinfos id=23 count=15501lb # [ 81.681991] niks3-server[1053]: 2026/09/22 10:47:38 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5502lb # [ 81.692091] niks3-server[1053]: 2026/09/22 10:47:38 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5503worker2 # [ 81.691687] nix-grpc-daemon[927]: time=2026-09-22T10:47:38.869Z level=INFO msg="Upload complete. (413ms)"5504lb # [ 81.698128] niks3-server[1053]: 2026/09/22 10:47:38 INFO Completed upload id=235505worker2 # [ 81.694228] nix-grpc-daemon[786]: ts=2026-09-22T10:47:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=1 paths=1 nar_bytes_in=5605506lb # [ 81.700366] niks3-server[1053]: 2026/09/22 10:47:38 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5507worker2 # [ 81.698429] nix-grpc-daemon[786]: ts=2026-09-22T10:47:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349525508worker2 # [ 81.704505] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5509worker2 # [ 81.707194] nix-daemon[873]: reaped child process 1137, status = succeeded5510worker2 # [ 81.710804] nix-grpc-daemon[786]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5511client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5512worker2 # [ 81.719704] nix-daemon[873]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5513client # uses-input-w1onlyb> worker2: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5514client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5515worker2 # [ 83.334947] nix-daemon[873]: reaped child process 1157, status = succeeded5516worker2 # [ 83.444084] nix-daemon[873]: accepted connection from pid 1173, user nix-grpc-daemon (trusted)5517worker2 # [ 83.454949] nix-daemon[873]: reaped child process 1176, status = succeeded5518lb # [ 83.470231] niks3-server[1053]: 2026/09/22 10:47:40 INFO Received uploads request method=POST path=/api/pending_closures5519worker2 # [ 83.471190] nix-grpc-daemon[927]: time=2026-09-22T10:47:40.648Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5520lb # [ 83.488844] niks3-server[1053]: 2026/09/22 10:47:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5521worker2 # [ 83.485131] nix-grpc-daemon[927]: time=2026-09-22T10:47:40.662Z level=INFO msg="Uploading 1 narinfos"5522lb # [ 83.491790] niks3-server[1053]: 2026/09/22 10:47:40 INFO Signed narinfos id=24 count=15523lb # [ 83.494447] niks3-server[1053]: 2026/09/22 10:47:40 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5524lb # [ 83.496839] niks3-server[1053]: 2026/09/22 10:47:40 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5525lb # [ 83.501055] niks3-server[1053]: 2026/09/22 10:47:40 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5526lb # [ 83.504194] niks3-server[1053]: 2026/09/22 10:47:40 INFO Completed upload id=245527worker2 # [ 83.497855] nix-grpc-daemon[927]: time=2026-09-22T10:47:40.675Z level=INFO msg="Upload complete. (170ms)"5528lb # [ 83.506508] niks3-server[1053]: 2026/09/22 10:47:40 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5529worker2 # [ 83.500554] nix-grpc-daemon[786]: ts=2026-09-22T10:47:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=11 outputs=15530client # copying 1 paths...5531worker2 # [ 83.508962] nix-daemon[873]: reaped child process 1154, status = succeeded5532worker1 # [ 83.537550] nix-grpc-daemon[784]: ts=2026-09-22T10:47:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:357845533worker1 # [ 83.546641] nix-grpc-daemon[784]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5534client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5535worker2 # [ 83.537448] nix-grpc-daemon[786]: ts=2026-09-22T10:47:40Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:349525536worker2 # [ 83.539388] nix-grpc-daemon[786]: ts=2026-09-22T10:47:40Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34952 duration_s=0 paths=1 nar_bytes_out=1205537client # [ 83.645629] nix-daemon[1070]: reaped child process 1177, status = succeeded5538client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.67 seconds)5539client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25540client # [ 83.799074] nix-daemon[1070]: accepted connection from pid 1196, user root (trusted)5541client # this derivation will be built:5542client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5543worker1 # [ 84.005588] nix-grpc-daemon[784]: ts=2026-09-22T10:47:41Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:357845544client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5545worker2 # [ 83.978263] nix-grpc-daemon[786]: ts=2026-09-22T10:47:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349525546worker2 # [ 83.987355] nix-grpc-daemon[927]: time=2026-09-22T10:47:41.164Z level=INFO msg="All 1 paths already cached"5547worker2 # [ 84.001165] nix-grpc-daemon[786]: ts=2026-09-22T10:47:41Z level=debug event=want client=13 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05548worker1 # [ 84.037843] nix-grpc-daemon[784]: ts=2026-09-22T10:47:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357845549worker2 # [ 84.005885] nix-grpc-daemon[786]: ts=2026-09-22T10:47:41Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=15550worker1 # [ 84.046262] nix-grpc-daemon[784]: ts=2026-09-22T10:47:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357845551worker1 # [ 84.055705] nix-grpc-daemon[864]: time=2026-09-22T10:47:41.177Z level=INFO msg="All 1 paths already cached"5552client # copying 1 paths to 'lb:50051'...5553client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5554worker1 # [ 84.063144] nix-grpc-daemon[784]: ts=2026-09-22T10:47:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357845555worker1 # [ 84.074390] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5556worker1 # [ 84.258298] nix-daemon[852]: accepted connection from pid 1152, user nix-grpc-daemon (trusted)5557worker1 # [ 84.273277] nix-daemon[852]: reaped child process 1155, status = succeeded5558worker1 # [ 84.407239] nix-daemon[852]: accepted connection from pid 1157, user nix-grpc-daemon (trusted)5559worker1 # [ 84.421063] nix-daemon[852]: reaped child process 1160, status = succeeded5560lb # [ 84.408821] niks3-server[1053]: 2026/09/22 10:47:41 INFO Received uploads request method=POST path=/api/pending_closures5561worker1 # [ 84.441427] nix-grpc-daemon[864]: time=2026-09-22T10:47:41.563Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5562worker1 # [ 84.443279] nix-grpc-daemon[864]: time=2026-09-22T10:47:41.565Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5563lb # [ 84.442155] niks3-server[1053]: 2026/09/22 10:47:41 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5564lb # [ 84.450696] niks3-server[1053]: 2026/09/22 10:47:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5565lb # [ 84.457272] niks3-server[1053]: 2026/09/22 10:47:41 INFO Signed narinfos id=25 count=15566worker1 # [ 84.484280] nix-grpc-daemon[864]: time=2026-09-22T10:47:41.606Z level=INFO msg="Uploading 1 narinfos"5567lb # [ 84.460712] niks3-server[1053]: 2026/09/22 10:47:41 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5568lb # [ 84.474686] niks3-server[1053]: 2026/09/22 10:47:41 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5569worker1 # [ 84.505255] nix-grpc-daemon[864]: time=2026-09-22T10:47:41.627Z level=INFO msg="Upload complete. (418ms)"5570lb # [ 84.481225] niks3-server[1053]: 2026/09/22 10:47:41 INFO Completed upload id=255571worker1 # [ 84.507717] nix-grpc-daemon[784]: ts=2026-09-22T10:47:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=0 paths=1 nar_bytes_in=5605572lb # [ 84.485102] niks3-server[1053]: 2026/09/22 10:47:41 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5573worker1 # [ 84.513688] nix-grpc-daemon[784]: ts=2026-09-22T10:47:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357845574worker1 # [ 84.522915] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5575worker1 # [ 84.526723] nix-daemon[852]: reaped child process 1150, status = succeeded5576worker1 # [ 84.531324] nix-daemon[852]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5577client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5578client # uses-input-w1onlyc> node-a: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5579client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5580worker1 # [ 85.746303] nix-daemon[852]: reaped child process 1164, status = succeeded5581worker1 # [ 85.868881] nix-daemon[852]: accepted connection from pid 1180, user nix-grpc-daemon (trusted)5582worker1 # [ 85.883936] nix-daemon[852]: reaped child process 1183, status = succeeded5583lb # [ 85.868930] niks3-server[1053]: 2026/09/22 10:47:42 INFO Received uploads request method=POST path=/api/pending_closures5584worker1 # [ 85.898353] nix-grpc-daemon[864]: time=2026-09-22T10:47:43.019Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5585lb # [ 85.886199] niks3-server[1053]: 2026/09/22 10:47:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5586lb # [ 85.888227] niks3-server[1053]: 2026/09/22 10:47:42 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5587lb # [ 85.890165] niks3-server[1053]: 2026/09/22 10:47:42 INFO Signed narinfos id=26 count=15588worker1 # [ 85.915401] nix-grpc-daemon[864]: time=2026-09-22T10:47:43.037Z level=INFO msg="Uploading 1 narinfos"5589lb # [ 85.893927] niks3-server[1053]: 2026/09/22 10:47:42 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5590lb # [ 85.898054] niks3-server[1053]: 2026/09/22 10:47:42 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5591lb # [ 85.901333] niks3-server[1053]: 2026/09/22 10:47:42 INFO Completed upload id=265592worker1 # [ 85.925251] nix-grpc-daemon[864]: time=2026-09-22T10:47:43.047Z level=INFO msg="Upload complete. (187ms)"5593lb # [ 85.904638] niks3-server[1053]: 2026/09/22 10:47:42 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5594worker1 # [ 85.927994] nix-grpc-daemon[784]: ts=2026-09-22T10:47:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=12 outputs=15595worker1 # [ 85.935187] nix-daemon[852]: reaped child process 1162, status = succeeded5596client # copying 1 paths...5597worker2 # [ 85.946116] nix-grpc-daemon[786]: ts=2026-09-22T10:47:43Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:349525598worker2 # [ 85.957519] nix-grpc-daemon[786]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5599client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5600worker1 # [ 86.015321] nix-grpc-daemon[784]: ts=2026-09-22T10:47:43Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:357845601worker1 # [ 86.019833] nix-grpc-daemon[784]: ts=2026-09-22T10:47:43Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:35784 duration_s=0 paths=1 nar_bytes_out=1205602client # [ 86.053946] nix-daemon[1070]: reaped child process 1199, status = succeeded5603client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.42 seconds)5604(finished: subtest: an input only one worker has still reaches the builder, in 9.51 seconds)5605subtest: upload whose reference the worker lacks is completed from the cache5606client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5607client # this derivation will be built:5608client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5609client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5610client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.39 seconds)5611client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5612client # this derivation will be built:5613client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5614client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5615client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.34 seconds)5616client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5617client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.16 seconds)5618worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5619worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.16 seconds)5620worker2 # [ 87.145597] systemd[1]: Stopping Nix worker-protocol over gRPC...5621worker2 # [ 88.049542] nix-grpc-daemon[786]: ts=2026-09-22T10:47:45Z level=info event=signal_exit5622lb # [ 88.334832] niks3-server[1053]: 2026/09/22 10:47:45 INFO lead: released remote=192.168.1.4:551145623lb # [ 88.680586] envoy[932]: conn peer=[2001:db8:1::1]:40410 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3868 tx=2083 ms=22755624lb # [ 88.684545] envoy[932]: conn peer=[2001:db8:1::1]:40418 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=505625lb # [ 88.688478] envoy[932]: conn peer=[2001:db8:1::1]:40432 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3755 tx=2122 ms=32655626lb # [ 88.692211] envoy[932]: conn peer=[2001:db8:1::1]:48374 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=485627lb # [ 88.695602] envoy[932]: conn peer=[2001:db8:1::1]:48388 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3743 tx=2121 ms=20205628lb # [ 88.699301] envoy[932]: conn peer=[2001:db8:1::1]:48398 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=95629worker1 # [ 89.360213] nix-grpc-daemon[784]: ts=2026-09-22T10:47:46Z level=info event=scheduler_restarting addr=lb:500515630worker2 # [ 89.333893] nix-grpc-daemon[786]: ts=2026-09-22T10:47:46Z level=info event=scheduler_restarting5631worker2 # [ 89.337513] nix-grpc-daemon[786]: ts=2026-09-22T10:47:46Z level=info event=scheduler_restarting addr=lb:500515632worker2 # [ 89.341233] nix-grpc-daemon[786]: ts=2026-09-22T10:47:46Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:349445633worker2 # [ 89.344544] nix-grpc-daemon[786]: ts=2026-09-22T10:47:46Z level=info event=worker_gone addr=192.168.1.4:500515634worker2 # [ 89.347689] nix-grpc-daemon[786]: ts=2026-09-22T10:47:46Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:349445635worker2 # [ 89.351385] nix-grpc-daemon[786]: ts=2026-09-22T10:47:46Z level=info event=worker_gone addr=192.168.1.3:500515636worker2 # [ 89.413896] nix-daemon[873]: reaped child process 875, status = succeeded5637lb # [ 89.523650] niks3-server[1053]: 2026/09/22 10:47:46 INFO lead: acquired remote=192.168.1.3:436245638worker1 # [ 89.547904] nix-grpc-daemon[784]: ts=2026-09-22T10:47:46Z level=info event=scheduler_take_over5639worker1 # [ 89.560595] nix-grpc-daemon[784]: ts=2026-09-22T10:47:46Z level=info event=scheduler_connected addr=lb:500515640worker1 # [ 89.564408] nix-grpc-daemon[784]: ts=2026-09-22T10:47:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5641worker2 # [ 89.538281] nix-grpc-daemon[786]: ts=2026-09-22T10:47:46Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5642worker1 # [ 89.765185] nix-grpc-daemon[784]: ts=2026-09-22T10:47:46Z level=info event=scheduler_connected addr=lb:500515643worker1 # [ 89.767206] nix-grpc-daemon[784]: ts=2026-09-22T10:47:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5644worker1 # [ 89.967597] nix-grpc-daemon[784]: ts=2026-09-22T10:47:47Z level=info event=scheduler_connected addr=lb:500515645worker1 # [ 89.969535] nix-grpc-daemon[784]: ts=2026-09-22T10:47:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5646worker2 # [ 90.026306] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5647worker2 # [ 90.029247] systemd[1]: Stopped Nix worker-protocol over gRPC.5648worker2 # [ 90.030267] systemd[1]: nix-grpc-daemon.service: Consumed 2.022s CPU time over 1min 11.243s wall clock time, 57.1M memory peak, 77.4K incoming IP traffic, 244.9K outgoing IP traffic.5649worker2 # [ 90.034798] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5650worker2 # [ 90.036550] systemd[1]: Closed Nix worker-protocol over gRPC.5651worker2 # [ 90.038122] systemd[1]: nix-grpc-daemon.socket: Consumed 278us CPU time over 1min 19.790s wall clock time, 256K memory peak, 94.2K incoming IP traffic, 83K outgoing IP traffic.5652lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5653lb: must succeed: curl -sf localhost:9901/clusters5654lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5655lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.04 seconds)5656client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25657worker1 # [ 90.170077] nix-grpc-daemon[784]: ts=2026-09-22T10:47:47Z level=info event=scheduler_connected addr=lb:500515658worker1 # [ 90.180951] nix-grpc-daemon[784]: ts=2026-09-22T10:47:47Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:343725659worker1 # [ 90.185962] nix-grpc-daemon[784]: ts=2026-09-22T10:47:47Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05660worker1 # [ 90.299481] nix-grpc-daemon[784]: ts=2026-09-22T10:47:47Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357845661worker1 # [ 90.301759] nix-grpc-daemon[784]: ts=2026-09-22T10:47:47Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:357845662client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5663client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.21 seconds)5664worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5665worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.03 seconds)5666worker2 # [ 90.377459] systemd[1]: Listening on Nix worker-protocol over gRPC.5667worker1 # [ 90.442516] systemd[1]: Stopping Nix worker-protocol over gRPC...5668worker1 # [ 90.966653] nix-grpc-daemon[784]: ts=2026-09-22T10:47:48Z level=info event=signal_exit5669worker2 # [ 91.192933] systemd[1]: Starting scheduler-token.service...5670lb # [ 91.227643] niks3-server[1053]: 2026/09/22 10:47:48 INFO lead: released remote=192.168.1.3:436245671worker2 # [ 91.248200] systemd[1]: scheduler-token.service: Deactivated successfully.5672worker2 # [ 91.249418] systemd[1]: Finished scheduler-token.service.5673worker2 # [ 91.250868] systemd[1]: scheduler-token.service: Consumed 23ms CPU time over 55ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5674worker2 # [ 91.256747] systemd[1]: Starting Nix worker-protocol over gRPC...5675worker2 # [ 91.398713] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:48Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5676worker2 # [ 91.401184] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:48Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5677worker2 # [ 91.404314] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:48Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15678worker2 # [ 91.410068] nix-grpc-daemon[1203]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5679worker2 # [ 91.412171] nix-grpc-daemon[1203]: E0000 00:00:1790074068.587056 1203 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5680worker2 # [ 91.417069] nix-grpc-daemon[1203]: E0000 00:00:1790074068.592149 1203 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5681worker2 # [ 91.421161] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:48Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5682worker2 # [ 91.422762] systemd[1]: Started Nix worker-protocol over gRPC.5683worker2 # [ 91.430335] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:48Z level=info event=scheduler_take_over5684lb # [ 91.436714] niks3-server[1053]: 2026/09/22 10:47:48 INFO lead: acquired remote=192.168.1.4:514345685worker2 # [ 91.433067] nix-grpc-daemon[1203]: E0000 00:00:1790074068.609664 1243 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5686worker2 # [ 91.444411] nix-grpc-daemon[1203]: E0000 00:00:1790074068.621751 1243 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5687worker2 # [ 91.449131] nix-grpc-daemon[1203]: E0000 00:00:1790074068.624064 1243 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5688worker2 # [ 91.458466] nix-grpc-daemon[1203]: E0000 00:00:1790074068.635865 1243 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5689worker2 # [ 91.466247] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:48Z level=info event=scheduler_connected addr=lb:500515690worker2 # [ 91.467715] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5691worker2 # [ 91.668698] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:48Z level=info event=scheduler_connected addr=lb:500515692worker2 # [ 91.670528] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5693worker2 # [ 91.871451] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:49Z level=info event=scheduler_connected addr=lb:500515694worker2 # [ 91.873382] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5695worker2 # [ 92.074261] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:49Z level=info event=scheduler_connected addr=lb:500515696worker2 # [ 92.076165] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5697worker1 # [ 92.254050] nix-grpc-daemon[784]: ts=2026-09-22T10:47:49Z level=info event=scheduler_restarting5698worker1 # [ 92.255716] nix-grpc-daemon[784]: ts=2026-09-22T10:47:49Z level=info event=scheduler_restarting addr=lb:500515699worker1 # [ 92.257213] nix-grpc-daemon[784]: ts=2026-09-22T10:47:49Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:343725700worker1 # [ 92.258543] nix-grpc-daemon[784]: ts=2026-09-22T10:47:49Z level=info event=worker_gone addr=192.168.1.3:500515701worker1 # [ 92.273385] nix-daemon[852]: reaped child process 854, status = succeeded5702worker2 # [ 92.277171] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:49Z level=info event=scheduler_connected addr=lb:500515703worker2 # [ 92.279600] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5704worker1 # [ 92.463777] nix-grpc-daemon[784]: ts=2026-09-22T10:47:49Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5705worker2 # [ 92.480910] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:49Z level=info event=scheduler_connected addr=lb:500515706worker2 # [ 92.487977] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:49Z level=info event=oidc_jwks provider=mock keys=15707worker2 # [ 92.489473] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:49Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:330485708worker2 # [ 92.491155] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:49Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05709worker1 # [ 93.121126] nix-grpc-daemon[784]: E0000 00:00:1790074070.243285 829 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true5710worker1 # [ 93.123247] nix-grpc-daemon[784]: E0000 00:00:1790074070.243395 829 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5711worker1 # [ 93.995228] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5712worker1 # [ 93.996532] systemd[1]: Stopped Nix worker-protocol over gRPC.5713worker1 # [ 93.998404] systemd[1]: nix-grpc-daemon.service: Consumed 2.676s CPU time over 1min 15.299s wall clock time, 62.1M memory peak, 92.6K incoming IP traffic, 175.6K outgoing IP traffic.5714worker1 # [ 94.003211] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5715worker1 # [ 94.005282] systemd[1]: Closed Nix worker-protocol over gRPC.5716worker1 # [ 94.006123] systemd[1]: nix-grpc-daemon.socket: Consumed 330us CPU time over 1min 24.651s wall clock time, 256K memory peak, 92.1K incoming IP traffic, 81.4K outgoing IP traffic.5717lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5718lb: must succeed: curl -sf localhost:9901/clusters5719lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5720lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.04 seconds)5721client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25722worker2 # [ 94.236509] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:424665723worker2 # [ 94.247771] nix-daemon[873]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5724worker2 # [ 94.260411] nix-grpc-daemon[1203]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 22 ms (attempt 1/5)5725worker2 # [ 94.285092] nix-grpc-daemon[1203]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 12 ms (attempt 2/5)5726worker2 # [ 94.300135] nix-grpc-daemon[1203]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 66 ms (attempt 3/5)5727worker2 # [ 94.369573] nix-grpc-daemon[1203]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 340 ms (attempt 4/5)5728worker2 # [ 94.710517] nix-grpc-daemon[1203]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5729worker2 # [ 94.711902] nix-grpc-daemon[1203]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5730client # copying 2 paths...5731client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5732worker2 # [ 94.725729] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:424665733client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5734worker2 # [ 94.732192] nix-daemon[873]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5735worker2 # [ 94.743109] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:51Z level=info event=niks3_push_started pid=12635736worker2 # [ 94.872413] nix-grpc-daemon[1263]: time=2026-09-22T10:47:52.049Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5737worker2 # [ 95.003374] nix-daemon[873]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)5738worker2 # [ 95.019687] nix-daemon[873]: reaped child process 1279, status = succeeded5739lb # [ 95.038640] niks3-server[1053]: 2026/09/22 10:47:51 INFO Received uploads request method=POST path=/api/pending_closures5740lb # [ 95.048186] niks3-server[1053]: 2026/09/22 10:47:51 INFO Received uploads request method=POST path=/api/pending_closures5741worker2 # [ 95.048366] nix-grpc-daemon[1263]: time=2026-09-22T10:47:52.225Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5742worker2 # [ 95.049998] nix-grpc-daemon[1263]: time=2026-09-22T10:47:52.227Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5743worker2 # [ 95.051885] nix-grpc-daemon[1263]: time=2026-09-22T10:47:52.229Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5744lb # [ 95.077958] niks3-server[1053]: 2026/09/22 10:47:51 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5745lb # [ 95.088595] niks3-server[1053]: 2026/09/22 10:47:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5746lb # [ 95.090438] niks3-server[1053]: 2026/09/22 10:47:51 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5747lb # [ 95.098265] niks3-server[1053]: 2026/09/22 10:47:51 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5748lb # [ 95.104882] niks3-server[1053]: 2026/09/22 10:47:51 INFO Signed narinfos id=28 count=25749lb # [ 95.106963] niks3-server[1053]: 2026/09/22 10:47:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5750lb # [ 95.109122] niks3-server[1053]: 2026/09/22 10:47:51 INFO Signed narinfos id=27 count=15751worker2 # [ 95.102883] nix-grpc-daemon[1263]: time=2026-09-22T10:47:52.280Z level=INFO msg="Uploading 3 narinfos"5752lb # [ 95.114531] niks3-server[1053]: 2026/09/22 10:47:51 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5753lb # [ 95.130674] niks3-server[1053]: 2026/09/22 10:47:51 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5754lb # [ 95.133441] niks3-server[1053]: 2026/09/22 10:47:51 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5755lb # [ 95.136679] niks3-server[1053]: 2026/09/22 10:47:51 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5756lb # [ 95.138943] niks3-server[1053]: 2026/09/22 10:47:51 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5757lb # [ 95.140618] niks3-server[1053]: 2026/09/22 10:47:51 INFO Completed upload id=275758lb # [ 95.142285] niks3-server[1053]: 2026/09/22 10:47:51 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5759lb # [ 95.144883] niks3-server[1053]: 2026/09/22 10:47:51 INFO Completed upload id=285760worker2 # [ 95.139709] nix-grpc-daemon[1263]: time=2026-09-22T10:47:52.317Z level=INFO msg="Upload complete. (263ms)"5761worker2 # [ 95.142357] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42466 duration_s=0 paths=2 nar_bytes_in=3125762worker2 # [ 95.149658] nix-daemon[873]: reaped child process 1261, status = succeeded5763client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.14 seconds)5764worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5765worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.01 seconds)5766worker1 # [ 95.250220] systemd[1]: Listening on Nix worker-protocol over gRPC.5767worker1 # [ 95.254995] systemd[1]: Starting Nix worker-protocol over gRPC...5768worker1 # [ 95.378941] nix-grpc-daemon[1206]: ts=2026-09-22T10:47:52Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5769worker1 # [ 95.382369] nix-grpc-daemon[1206]: ts=2026-09-22T10:47:52Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5770worker1 # [ 95.384516] nix-grpc-daemon[1206]: ts=2026-09-22T10:47:52Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15771worker1 # [ 95.390084] nix-grpc-daemon[1206]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5772worker1 # [ 95.391471] nix-grpc-daemon[1206]: E0000 00:00:1790074072.511641 1206 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5773worker1 # [ 95.398276] nix-grpc-daemon[1206]: E0000 00:00:1790074072.516870 1206 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5774worker1 # [ 95.402610] nix-grpc-daemon[1206]: ts=2026-09-22T10:47:52Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5775worker1 # [ 95.404823] systemd[1]: Started Nix worker-protocol over gRPC.5776worker1 # [ 95.408671] nix-grpc-daemon[1206]: E0000 00:00:1790074072.531028 1247 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5777lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5778lb: must succeed: curl -sf localhost:9901/clusters5779worker1 # [ 95.423088] nix-grpc-daemon[1206]: E0000 00:00:1790074072.544313 1247 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5780worker1 # [ 95.426228] nix-grpc-daemon[1206]: E0000 00:00:1790074072.544499 1247 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5781worker1 # [ 95.436258] nix-grpc-daemon[1206]: E0000 00:00:1790074072.558563 1247 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5782worker1 # [ 95.443431] nix-grpc-daemon[1206]: ts=2026-09-22T10:47:52Z level=info event=scheduler_connected addr=lb:500515783worker2 # [ 95.414982] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:52Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:330485784worker2 # [ 95.417303] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:52Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05785lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5786lb: must succeed: curl -sf localhost:9901/clusters5787lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5788lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.09 seconds)5789(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.42 seconds)5790subtest: a niks3 restart does not cost the leader its role5791lb: must succeed: curl -sf localhost:9901/clusters5792lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5793lb: must succeed: curl -sf localhost:9901/clusters5794lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5795lb # [ 96.572363] niks3-server[1053]: 2026/09/22 10:47:53 INFO Shutdown signal received, draining in-flight requests timeout=10s5796lb # [ 96.576231] systemd[1]: Stopping niks3 server...5797worker1 # [ 96.601099] nix-grpc-daemon[1206]: ts=2026-09-22T10:47:53Z level=info event=scheduler_lock_lost status=2005798lb # [ 96.579542] niks3-server[1053]: 2026/09/22 10:47:53 INFO lead: released remote=192.168.1.4:514345799worker2 # [ 96.576827] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:53Z level=info event=scheduler_lock_lost status=2005800lb # [ 96.597612] systemd[1]: niks3.service: Deactivated successfully.5801lb # [ 96.599857] systemd[1]: Stopped niks3 server.5802lb # [ 96.601268] systemd[1]: niks3.service: Consumed 599ms CPU time over 1min 7.500s wall clock time, 21.1M memory peak, 80.4K incoming IP traffic, 113.5K outgoing IP traffic.5803lb # [ 96.610541] systemd[1]: Starting niks3 server...5804lb # [ 96.681485] niks3-server[1196]: 2026/09/22 10:47:53 goose: no migrations to run. current version: 202609200000005805lb # [ 96.687959] niks3-server[1196]: 2026/09/22 10:47:53 OK 1_commit_pending_closure.sql (4.79ms)5806lb # [ 96.691567] niks3-server[1196]: 2026/09/22 10:47:53 OK 2_object_stats_trigger.sql (1.77ms)5807lb # [ 96.692662] niks3-server[1196]: 2026/09/22 10:47:53 goose: up to current file version: 25808lb # [ 96.694224] niks3-server[1196]: 2026/09/22 10:47:53 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key5809lb # [ 96.706047] niks3-server[1196]: 2026/09/22 10:47:53 INFO Read proxy enabled — serving cache objects from S35810lb # [ 96.707709] niks3-server[1196]: 2026/09/22 10:47:53 INFO Using socket-activated listener address=0.0.0.0:57515811lb # [ 96.709123] niks3-server[1196]: 2026/09/22 10:47:53 INFO systemd watchdog enabled interval=15s5812lb # [ 96.711168] systemd[1]: Started niks3 server.5813lb # [ 96.711892] niks3-server[1196]: 2026/09/22 10:47:53 INFO Starting HTTP server address=0.0.0.0:57515814lb: waiting for unit niks3.service5815lb: (finished: waiting for unit niks3.service, in 0.04 seconds)5816lb: must succeed: curl -sf localhost:9901/clusters5817lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5818worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5819worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5820lb: must succeed: curl -sf localhost:9901/clusters5821lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5822lb: must succeed: curl -sf localhost:9901/clusters5823lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5824worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5825worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.03 seconds)5826client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&25827worker1 # [ 97.184153] nix-grpc-daemon[1206]: ts=2026-09-22T10:47:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:496445828worker1 # [ 97.192663] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)5829client # don't know how to build these paths:5830client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5831worker1 # [ 97.202154] nix-grpc-daemon[1206]: ts=2026-09-22T10:47:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:496445832worker1 # [ 97.210301] nix-grpc-daemon[1206]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 1/5)5833worker1 # [ 97.239082] nix-grpc-daemon[1206]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 124 ms (attempt 2/5)5834worker1 # [ 97.366449] nix-grpc-daemon[1206]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 9 ms (attempt 3/5)5835worker1 # [ 97.375613] nix-grpc-daemon[1206]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 432 ms (attempt 4/5)5836lb # [ 97.589256] niks3-server[1196]: 2026/09/22 10:47:54 INFO lead: acquired remote=192.168.1.4:580205837worker1 # [ 97.811203] nix-grpc-daemon[1206]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5838worker1 # [ 97.813691] nix-grpc-daemon[1206]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5839worker2 # [ 97.808685] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:54Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=05840worker2 # [ 97.810649] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:54Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.4:50051 clients=15841worker2 # [ 97.850950] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:424665842worker2 # [ 97.858782] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:424665843client # copying 1 paths to 'lb:50051'...5844client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...5845worker2 # [ 97.865353] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:424665846worker2 # [ 97.870655] nix-daemon[873]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5847worker2 # [ 98.018170] nix-daemon[873]: accepted connection from pid 1298, user nix-grpc-daemon (trusted)5848worker2 # [ 98.034295] nix-daemon[873]: reaped child process 1301, status = succeeded5849worker2 # [ 98.173638] nix-daemon[873]: accepted connection from pid 1303, user nix-grpc-daemon (trusted)5850worker2 # [ 98.186756] nix-daemon[873]: reaped child process 1306, status = succeeded5851lb # [ 98.203197] niks3-server[1196]: 2026/09/22 10:47:55 INFO Received uploads request method=POST path=/api/pending_closures5852worker2 # [ 98.203060] nix-grpc-daemon[1263]: time=2026-09-22T10:47:55.380Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5853worker2 # [ 98.205442] nix-grpc-daemon[1263]: time=2026-09-22T10:47:55.382Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"5854lb # [ 98.226199] niks3-server[1196]: 2026/09/22 10:47:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5855lb # [ 98.230564] niks3-server[1196]: 2026/09/22 10:47:55 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst5856lb # [ 98.241783] niks3-server[1196]: 2026/09/22 10:47:55 INFO Signed narinfos id=29 count=15857worker2 # [ 98.237096] nix-grpc-daemon[1263]: time=2026-09-22T10:47:55.414Z level=INFO msg="Uploading 1 narinfos"5858lb # [ 98.252101] niks3-server[1196]: 2026/09/22 10:47:55 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5859lb # [ 98.255529] niks3-server[1196]: 2026/09/22 10:47:55 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls5860lb # [ 98.259632] niks3-server[1196]: 2026/09/22 10:47:55 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo5861lb # [ 98.264039] niks3-server[1196]: 2026/09/22 10:47:55 INFO Completed upload id=295862worker2 # [ 98.257623] nix-grpc-daemon[1263]: time=2026-09-22T10:47:55.435Z level=INFO msg="Upload complete. (381ms)"5863worker2 # [ 98.259616] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42466 duration_s=0 paths=1 nar_bytes_in=5365864worker2 # [ 98.263228] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:424665865worker2 # [ 98.270073] nix-daemon[873]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5866worker2 # [ 98.275912] nix-daemon[873]: reaped child process 1296, status = succeeded5867client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...5868client # farm-a> worker2: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5869worker2 # [ 98.280720] nix-daemon[873]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5870lb # [ 98.680858] envoy[932]: conn peer=[2001:db8:1::4]:52736 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=87143 tx=4032 ms=704255871lb # [ 98.683136] envoy[932]: conn peer=[2001:db8:1::1]:48412 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=115872lb # [ 98.684856] envoy[932]: conn peer=[2001:db8:1::3]:48528 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=22107 tx=4175 ms=726315873lb # [ 98.686692] envoy[932]: conn peer=[2001:db8:1::1]:40220 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1006 tx=282 ms=9185874client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5875client # farm-a> LOG-farm-a-after-niks3-restart5876worker2 # [ 99.094361] nix-daemon[873]: reaped child process 1310, status = succeeded5877worker2 # [ 99.220874] nix-daemon[873]: accepted connection from pid 1326, user nix-grpc-daemon (trusted)5878worker2 # [ 99.232666] nix-daemon[873]: reaped child process 1329, status = succeeded5879lb # [ 99.249984] niks3-server[1196]: 2026/09/22 10:47:56 INFO Received uploads request method=POST path=/api/pending_closures5880worker2 # [ 99.250097] nix-grpc-daemon[1263]: time=2026-09-22T10:47:56.427Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5881worker2 # [ 99.253085] nix-grpc-daemon[1263]: time=2026-09-22T10:47:56.429Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"5882lb # [ 99.274110] niks3-server[1196]: 2026/09/22 10:47:56 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5883lb # [ 99.276763] niks3-server[1196]: 2026/09/22 10:47:56 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5884lb # [ 99.281058] niks3-server[1196]: 2026/09/22 10:47:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5885lb # [ 99.283219] niks3-server[1196]: 2026/09/22 10:47:56 INFO Signed narinfos id=30 count=15886worker2 # [ 99.278490] nix-grpc-daemon[1263]: time=2026-09-22T10:47:56.455Z level=INFO msg="Uploading 1 narinfos"5887lb # [ 99.287830] niks3-server[1196]: 2026/09/22 10:47:56 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls5888lb # [ 99.292530] niks3-server[1196]: 2026/09/22 10:47:56 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5889lb # [ 99.295302] niks3-server[1196]: 2026/09/22 10:47:56 INFO Completed upload id=305890worker2 # [ 99.288987] nix-grpc-daemon[1263]: time=2026-09-22T10:47:56.466Z level=INFO msg="Upload complete. (200ms)"5891lb # [ 99.298102] niks3-server[1196]: 2026/09/22 10:47:56 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo5892worker2 # [ 99.292987] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42466 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=15893worker2 # [ 99.299466] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:56Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=05894worker2 # [ 99.301637] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:56Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.4:50051 clients=15895worker2 # [ 99.307087] nix-daemon[873]: reaped child process 1308, status = succeeded5896worker2 # [ 99.342142] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:424665897worker2 # [ 99.349477] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:424665898worker2 # [ 99.355731] nix-grpc-daemon[1263]: time=2026-09-22T10:47:56.533Z level=INFO msg="All 1 paths already cached"5899client # copying 1 paths to 'lb:50051'...5900client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...5901worker2 # [ 99.360235] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:424665902worker2 # [ 99.365678] nix-daemon[873]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5903worker2 # [ 99.502300] nix-daemon[873]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)5904worker2 # [ 99.515964] nix-daemon[873]: reaped child process 1338, status = succeeded5905worker2 # [ 99.649563] nix-daemon[873]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5906worker2 # [ 99.661924] nix-daemon[873]: reaped child process 1343, status = succeeded5907lb # [ 99.677892] niks3-server[1196]: 2026/09/22 10:47:56 INFO Received uploads request method=POST path=/api/pending_closures5908worker2 # [ 99.676387] nix-grpc-daemon[1263]: time=2026-09-22T10:47:56.853Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5909worker2 # [ 99.677750] nix-grpc-daemon[1263]: time=2026-09-22T10:47:56.853Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"5910lb # [ 99.691773] niks3-server[1196]: 2026/09/22 10:47:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5911lb # [ 99.694219] niks3-server[1196]: 2026/09/22 10:47:56 INFO Signed narinfos id=31 count=15912worker2 # [ 99.689290] nix-grpc-daemon[1263]: time=2026-09-22T10:47:56.866Z level=INFO msg="Uploading 1 narinfos"5913lb # [ 99.695855] niks3-server[1196]: 2026/09/22 10:47:56 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst5914lb # [ 99.699943] niks3-server[1196]: 2026/09/22 10:47:56 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls5915lb # [ 99.704286] niks3-server[1196]: 2026/09/22 10:47:56 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5916lb # [ 99.707052] niks3-server[1196]: 2026/09/22 10:47:56 INFO Completed upload id=315917worker2 # [ 99.702151] nix-grpc-daemon[1263]: time=2026-09-22T10:47:56.879Z level=INFO msg="Upload complete. (331ms)"5918lb # [ 99.709057] niks3-server[1196]: 2026/09/22 10:47:56 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo5919worker2 # [ 99.704371] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42466 duration_s=0 paths=1 nar_bytes_in=6565920worker2 # [ 99.706778] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:424665921worker2 # [ 99.713078] nix-daemon[873]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5922worker2 # [ 99.720336] nix-daemon[873]: reaped child process 1333, status = succeeded5923client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...5924client # farm-b> worker2: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5925worker2 # [ 99.723646] nix-daemon[873]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5926client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5927client # farm-b> LOG-farm-b-after-niks3-restart5928worker2 # [ 101.137747] nix-daemon[873]: reaped child process 1347, status = succeeded5929worker2 # [ 101.262564] nix-daemon[873]: accepted connection from pid 1363, user nix-grpc-daemon (trusted)5930worker2 # [ 101.275641] nix-daemon[873]: reaped child process 1366, status = succeeded5931lb # [ 101.295331] niks3-server[1196]: 2026/09/22 10:47:58 INFO Received uploads request method=POST path=/api/pending_closures5932worker2 # [ 101.298575] nix-grpc-daemon[1263]: time=2026-09-22T10:47:58.475Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5933worker2 # [ 101.300181] nix-grpc-daemon[1263]: time=2026-09-22T10:47:58.477Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"5934lb # [ 101.336947] niks3-server[1196]: 2026/09/22 10:47:58 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst5935lb # [ 101.345224] niks3-server[1196]: 2026/09/22 10:47:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5936worker2 # [ 101.344460] nix-grpc-daemon[1263]: time=2026-09-22T10:47:58.521Z level=INFO msg="Uploading 1 narinfos"5937lb # [ 101.351157] niks3-server[1196]: 2026/09/22 10:47:58 INFO Signed narinfos id=32 count=15938lb # [ 101.355969] niks3-server[1196]: 2026/09/22 10:47:58 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5939lb # [ 101.362053] niks3-server[1196]: 2026/09/22 10:47:58 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls5940lb # [ 101.366926] niks3-server[1196]: 2026/09/22 10:47:58 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5941lb # [ 101.372390] niks3-server[1196]: 2026/09/22 10:47:58 INFO Completed upload id=325942worker2 # [ 101.366028] nix-grpc-daemon[1263]: time=2026-09-22T10:47:58.543Z level=INFO msg="Upload complete. (235ms)"5943lb # [ 101.374401] niks3-server[1196]: 2026/09/22 10:47:58 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo5944worker2 # [ 101.368704] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42466 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=15945worker2 # [ 101.375405] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:58Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=05946worker2 # [ 101.377417] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:58Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.4:50051 clients=15947worker2 # [ 101.381643] nix-daemon[873]: reaped child process 1345, status = succeeded5948worker2 # [ 101.419456] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:424665949worker2 # [ 101.430337] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:424665950worker2 # [ 101.440985] nix-grpc-daemon[1263]: time=2026-09-22T10:47:58.618Z level=INFO msg="All 2 paths already cached"5951client # copying 1 paths to 'lb:50051'...5952client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...5953worker2 # [ 101.448438] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:424665954worker2 # [ 101.453342] nix-daemon[873]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5955worker2 # [ 101.584799] nix-daemon[873]: accepted connection from pid 1372, user nix-grpc-daemon (trusted)5956worker2 # [ 101.597307] nix-daemon[873]: reaped child process 1375, status = succeeded5957worker2 # [ 101.744618] nix-daemon[873]: accepted connection from pid 1377, user nix-grpc-daemon (trusted)5958worker2 # [ 101.757946] nix-daemon[873]: reaped child process 1380, status = succeeded5959lb # [ 101.778328] niks3-server[1196]: 2026/09/22 10:47:58 INFO Received uploads request method=POST path=/api/pending_closures5960worker2 # [ 101.794136] nix-grpc-daemon[1263]: time=2026-09-22T10:47:58.971Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5961worker2 # [ 101.795765] nix-grpc-daemon[1263]: time=2026-09-22T10:47:58.971Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"5962lb # [ 101.820079] niks3-server[1196]: 2026/09/22 10:47:58 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst5963lb # [ 101.828594] niks3-server[1196]: 2026/09/22 10:47:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5964worker2 # [ 101.826332] nix-grpc-daemon[1263]: time=2026-09-22T10:47:59.003Z level=INFO msg="Uploading 1 narinfos"5965lb # [ 101.832628] niks3-server[1196]: 2026/09/22 10:47:58 INFO Signed narinfos id=33 count=15966lb # [ 101.839254] niks3-server[1196]: 2026/09/22 10:47:58 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls5967lb # [ 101.844412] niks3-server[1196]: 2026/09/22 10:47:58 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5968lb # [ 101.848524] niks3-server[1196]: 2026/09/22 10:47:58 INFO Completed upload id=335969worker2 # [ 101.841982] nix-grpc-daemon[1263]: time=2026-09-22T10:47:59.019Z level=INFO msg="Upload complete. (383ms)"5970worker2 # [ 101.844054] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42466 duration_s=0 paths=1 nar_bytes_in=7845971lb # [ 101.851230] niks3-server[1196]: 2026/09/22 10:47:58 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo5972worker2 # [ 101.847776] nix-grpc-daemon[1203]: ts=2026-09-22T10:47:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:424665973worker2 # [ 101.853535] nix-daemon[873]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5974worker2 # [ 101.856599] nix-daemon[873]: reaped child process 1370, status = succeeded5975client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...5976client # farm-top> worker2: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5977worker2 # [ 101.862592] nix-daemon[873]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5978client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5979client # farm-top> LOG-farm-top-after-niks3-restart5980worker2 # [ 102.676201] nix-daemon[873]: reaped child process 1384, status = succeeded5981worker2 # [ 102.800822] nix-daemon[873]: accepted connection from pid 1400, user nix-grpc-daemon (trusted)5982worker2 # [ 102.812354] nix-daemon[873]: reaped child process 1403, status = succeeded5983lb # [ 102.834783] niks3-server[1196]: 2026/09/22 10:47:59 INFO Received uploads request method=POST path=/api/pending_closures5984worker2 # [ 102.836758] nix-grpc-daemon[1263]: time=2026-09-22T10:48:00.013Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5985worker2 # [ 102.839057] nix-grpc-daemon[1263]: time=2026-09-22T10:48:00.015Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"5986lb # [ 102.867290] niks3-server[1196]: 2026/09/22 10:47:59 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5987lb # [ 102.878636] niks3-server[1196]: 2026/09/22 10:47:59 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst5988lb # [ 102.889368] niks3-server[1196]: 2026/09/22 10:47:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5989worker2 # [ 102.889922] nix-grpc-daemon[1263]: time=2026-09-22T10:48:00.067Z level=INFO msg="Uploading 1 narinfos"5990lb # [ 102.896702] niks3-server[1196]: 2026/09/22 10:47:59 INFO Signed narinfos id=34 count=15991lb # [ 102.901472] niks3-server[1196]: 2026/09/22 10:47:59 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls5992lb # [ 102.911405] niks3-server[1196]: 2026/09/22 10:47:59 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5993lb # [ 102.914459] niks3-server[1196]: 2026/09/22 10:47:59 INFO Completed upload id=345994worker2 # [ 102.908349] nix-grpc-daemon[1263]: time=2026-09-22T10:48:00.085Z level=INFO msg="Upload complete. (240ms)"5995lb # [ 102.916209] niks3-server[1196]: 2026/09/22 10:47:59 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo5996worker2 # [ 102.913827] nix-grpc-daemon[1203]: ts=2026-09-22T10:48:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42466 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=15997worker2 # [ 102.926091] nix-daemon[873]: reaped child process 1382, status = succeeded5998client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&2, in 5.98 seconds)5999client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath6000client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath, in 0.16 seconds)6001(finished: subtest: a niks3 restart does not cost the leader its role, in 6.63 seconds)6002subtest: scheduler down: the other node takes the lock and keeps it6003lb: must succeed: curl -sf localhost:9901/clusters6004lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6005lb: must succeed: curl -sf localhost:9901/clusters6006lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6007lb: must succeed: curl -sf localhost:9901/clusters6008lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6009lb: must succeed: curl -sf localhost:9901/clusters6010lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6011worker2 # [ 103.314427] systemd[1]: Stopping Nix worker-protocol over gRPC...6012worker2 # [ 103.424410] nix-grpc-daemon[1203]: ts=2026-09-22T10:48:00Z level=info event=signal_exit6013lb # [ 103.796136] niks3-server[1196]: 2026/09/22 10:48:00 INFO lead: released remote=192.168.1.4:580206014worker2 # [ 104.789315] nix-grpc-daemon[1203]: ts=2026-09-22T10:48:01Z level=info event=scheduler_restarting6015worker2 # [ 104.792168] nix-grpc-daemon[1203]: ts=2026-09-22T10:48:01Z level=info event=scheduler_restarting addr=lb:500516016worker1 # [ 104.820865] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:01Z level=info event=scheduler_restarting addr=lb:500516017worker2 # [ 104.793408] nix-grpc-daemon[1203]: ts=2026-09-22T10:48:01Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:330486018worker2 # [ 104.794855] nix-grpc-daemon[1203]: ts=2026-09-22T10:48:01Z level=info event=worker_gone addr=192.168.1.4:500516019worker2 # [ 104.796106] nix-grpc-daemon[1203]: ts=2026-09-22T10:48:01Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:330486020worker2 # [ 104.797441] nix-grpc-daemon[1203]: ts=2026-09-22T10:48:01Z level=info event=worker_gone addr=192.168.1.3:500516021worker2 # [ 104.805061] nix-daemon[873]: reaped child process 1255, status = succeeded6022worker1 # [ 105.020542] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:02Z level=info event=scheduler_connected addr=lb:500516023worker1 # [ 105.022516] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6024worker2 # [ 105.000869] nix-grpc-daemon[1203]: ts=2026-09-22T10:48:02Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6025worker1 # [ 105.223378] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:02Z level=info event=scheduler_connected addr=lb:500516026worker1 # [ 105.227372] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6027worker1 # [ 105.427940] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:02Z level=info event=scheduler_connected addr=lb:500516028worker1 # [ 105.432098] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6029worker2 # [ 105.454824] nix-grpc-daemon[1203]: E0000 00:00:1790074082.632189 1248 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true6030worker2 # [ 105.457127] nix-grpc-daemon[1203]: E0000 00:00:1790074082.634563 1248 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6031worker1 # [ 105.632504] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:02Z level=info event=scheduler_connected addr=lb:500516032worker1 # [ 105.634504] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6033worker1 # [ 105.836529] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:02Z level=info event=scheduler_connected addr=lb:500516034worker1 # [ 105.840333] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6035worker1 # [ 106.040860] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:03Z level=info event=scheduler_connected addr=lb:500516036worker1 # [ 106.044820] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6037worker1 # [ 106.245730] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:03Z level=info event=scheduler_connected addr=lb:500516038worker1 # [ 106.250594] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6039worker1 # [ 106.450742] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:03Z level=info event=scheduler_connected addr=lb:500516040worker1 # [ 106.452700] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6041worker1 # [ 106.653412] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:03Z level=info event=scheduler_connected addr=lb:500516042worker1 # [ 106.656382] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6043worker1 # [ 106.859240] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:03Z level=info event=scheduler_connected addr=lb:500516044worker1 # [ 106.862482] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6045worker2 # [ 106.866756] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6046worker2 # [ 106.869727] systemd[1]: Stopped Nix worker-protocol over gRPC.6047worker2 # [ 106.874678] systemd[1]: nix-grpc-daemon.service: Consumed 1.053s CPU time over 15.608s wall clock time, 35.2M memory peak, 45.7K incoming IP traffic, 83.5K outgoing IP traffic.6048worker2 # [ 106.884410] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.6049worker2 # [ 106.888348] systemd[1]: Closed Nix worker-protocol over gRPC.6050worker2 # [ 106.891138] systemd[1]: nix-grpc-daemon.socket: Consumed 351us CPU time over 16.514s wall clock time, 256K memory peak, 37.1K incoming IP traffic, 27.7K outgoing IP traffic.6051worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over6052worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.06 seconds)6053worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6054worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)6055worker1 # [ 107.059218] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:04Z level=info event=scheduler_connected addr=lb:500516056worker1 # [ 107.060908] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6057worker1 # [ 107.262684] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:04Z level=info event=scheduler_connected addr=lb:500516058worker1 # [ 107.266732] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6059worker1 # [ 107.467548] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:04Z level=info event=scheduler_connected addr=lb:500516060worker1 # [ 107.471518] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6061worker1 # [ 107.608944] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:04Z level=info event=scheduler_take_over6062lb # [ 107.585655] niks3-server[1196]: 2026/09/22 10:48:04 INFO lead: acquired remote=192.168.1.3:452126063worker1 # [ 107.672482] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:04Z level=info event=scheduler_connected addr=lb:500516064worker1 # [ 107.676417] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6065worker1 # [ 107.877512] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:04Z level=info event=scheduler_connected addr=lb:500516066worker1 # [ 107.879174] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6067worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6068worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6069worker1 # [ 108.079586] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:05Z level=info event=scheduler_connected addr=lb:500516070worker1 # [ 108.082900] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:05Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:428406071worker1 # [ 108.084359] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:05Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06072lb # [ 108.680662] envoy[932]: conn peer=[2001:db8:1::1]:40226 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9653 tx=4434 ms=57796073lb # [ 108.682249] envoy[932]: conn peer=[2001:db8:1::4]:35502 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=5880 tx=648 ms=135286074worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6075worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6076client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&26077worker1 # [ 109.313610] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:496446078client # don't know how to build these paths:6079client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6080worker1 # [ 109.322202] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:496446081worker1 # [ 109.345877] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:06Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=06082worker1 # [ 109.350786] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:06Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=16083worker1 # [ 109.394399] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496446084worker1 # [ 109.404883] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:496446085client # copying 1 paths to 'lb:50051'...6086client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...6087worker1 # [ 109.410650] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:496446088worker1 # [ 109.416327] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6089worker1 # [ 109.425981] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:06Z level=info event=niks3_push_started pid=12856090worker1 # [ 109.600229] nix-grpc-daemon[1285]: time=2026-09-22T10:48:06.722Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6091worker1 # [ 109.748097] nix-daemon[852]: accepted connection from pid 1298, user nix-grpc-daemon (trusted)6092worker1 # [ 109.759859] nix-daemon[852]: reaped child process 1301, status = succeeded6093worker1 # [ 109.915361] nix-daemon[852]: accepted connection from pid 1303, user nix-grpc-daemon (trusted)6094worker1 # [ 109.927960] nix-daemon[852]: reaped child process 1306, status = succeeded6095lb # [ 109.918789] niks3-server[1196]: 2026/09/22 10:48:06 INFO Received uploads request method=POST path=/api/pending_closures6096worker1 # [ 109.950555] nix-grpc-daemon[1285]: time=2026-09-22T10:48:07.072Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6097worker1 # [ 109.953629] nix-grpc-daemon[1285]: time=2026-09-22T10:48:07.074Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"6098lb # [ 109.951596] niks3-server[1196]: 2026/09/22 10:48:06 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst6099lb # [ 109.962118] niks3-server[1196]: 2026/09/22 10:48:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign6100worker1 # [ 109.992335] nix-grpc-daemon[1285]: time=2026-09-22T10:48:07.114Z level=INFO msg="Uploading 1 narinfos"6101lb # [ 109.968076] niks3-server[1196]: 2026/09/22 10:48:06 INFO Signed narinfos id=35 count=16102lb # [ 109.980892] niks3-server[1196]: 2026/09/22 10:48:06 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls6103lb # [ 109.985912] niks3-server[1196]: 2026/09/22 10:48:06 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete6104lb # [ 109.996622] niks3-server[1196]: 2026/09/22 10:48:06 INFO Completed upload id=356105worker1 # [ 110.020601] nix-grpc-daemon[1285]: time=2026-09-22T10:48:07.142Z level=INFO msg="Upload complete. (416ms)"6106lb # [ 109.998077] niks3-server[1196]: 2026/09/22 10:48:06 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo6107worker1 # [ 110.022802] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49644 duration_s=0 paths=1 nar_bytes_in=5206108worker1 # [ 110.026965] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496446109worker1 # [ 110.033232] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6110worker1 # [ 110.037072] nix-daemon[852]: reaped child process 1283, status = succeeded6111client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...6112client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6113worker1 # [ 110.041645] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6114client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6115client # farm-a> LOG-farm-a-on-standby6116worker1 # [ 110.869780] nix-daemon[852]: reaped child process 1310, status = succeeded6117worker1 # [ 110.994927] nix-daemon[852]: accepted connection from pid 1326, user nix-grpc-daemon (trusted)6118worker1 # [ 111.006786] nix-daemon[852]: reaped child process 1329, status = succeeded6119lb # [ 110.997149] niks3-server[1196]: 2026/09/22 10:48:07 INFO Received uploads request method=POST path=/api/pending_closures6120worker1 # [ 111.028699] nix-grpc-daemon[1285]: time=2026-09-22T10:48:08.150Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6121worker1 # [ 111.031066] nix-grpc-daemon[1285]: time=2026-09-22T10:48:08.152Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"6122lb # [ 111.036478] niks3-server[1196]: 2026/09/22 10:48:07 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst6123lb # [ 111.045881] niks3-server[1196]: 2026/09/22 10:48:07 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6124lb # [ 111.053777] niks3-server[1196]: 2026/09/22 10:48:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign6125lb # [ 111.060709] niks3-server[1196]: 2026/09/22 10:48:07 INFO Signed narinfos id=36 count=16126worker1 # [ 111.085247] nix-grpc-daemon[1285]: time=2026-09-22T10:48:08.206Z level=INFO msg="Uploading 1 narinfos"6127lb # [ 111.067642] niks3-server[1196]: 2026/09/22 10:48:07 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls6128lb # [ 111.076425] niks3-server[1196]: 2026/09/22 10:48:07 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete6129lb # [ 111.080570] niks3-server[1196]: 2026/09/22 10:48:07 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo6130lb # [ 111.083510] niks3-server[1196]: 2026/09/22 10:48:07 INFO Completed upload id=366131worker1 # [ 111.107477] nix-grpc-daemon[1285]: time=2026-09-22T10:48:08.229Z level=INFO msg="Upload complete. (259ms)"6132worker1 # [ 111.111187] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49644 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=16133worker1 # [ 111.123548] nix-daemon[852]: reaped child process 1308, status = succeeded6134worker1 # [ 111.124819] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:08Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=06135worker1 # [ 111.126596] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:08Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=16136worker1 # [ 111.168328] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496446137worker1 # [ 111.181082] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:496446138worker1 # [ 111.194557] nix-grpc-daemon[1285]: time=2026-09-22T10:48:08.316Z level=INFO msg="All 1 paths already cached"6139client # copying 1 paths to 'lb:50051'...6140client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...6141worker1 # [ 111.203253] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:496446142worker1 # [ 111.211539] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6143worker1 # [ 111.373669] nix-daemon[852]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6144worker1 # [ 111.386682] nix-daemon[852]: reaped child process 1338, status = succeeded6145worker1 # [ 111.547211] nix-daemon[852]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)6146worker1 # [ 111.560102] nix-daemon[852]: reaped child process 1343, status = succeeded6147lb # [ 111.549368] niks3-server[1196]: 2026/09/22 10:48:08 INFO Received uploads request method=POST path=/api/pending_closures6148worker1 # [ 111.581553] nix-grpc-daemon[1285]: time=2026-09-22T10:48:08.703Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6149worker1 # [ 111.584151] nix-grpc-daemon[1285]: time=2026-09-22T10:48:08.705Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"6150lb # [ 111.577512] niks3-server[1196]: 2026/09/22 10:48:08 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst6151lb # [ 111.589221] niks3-server[1196]: 2026/09/22 10:48:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6152worker1 # [ 111.620731] nix-grpc-daemon[1285]: time=2026-09-22T10:48:08.742Z level=INFO msg="Uploading 1 narinfos"6153lb # [ 111.596524] niks3-server[1196]: 2026/09/22 10:48:08 INFO Signed narinfos id=37 count=16154lb # [ 111.601266] niks3-server[1196]: 2026/09/22 10:48:08 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls6155lb # [ 111.613384] niks3-server[1196]: 2026/09/22 10:48:08 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6156worker1 # [ 111.643562] nix-grpc-daemon[1285]: time=2026-09-22T10:48:08.765Z level=INFO msg="Upload complete. (419ms)"6157lb # [ 111.619548] niks3-server[1196]: 2026/09/22 10:48:08 INFO Completed upload id=376158worker1 # [ 111.645881] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49644 duration_s=0 paths=1 nar_bytes_in=6406159lb # [ 111.623118] niks3-server[1196]: 2026/09/22 10:48:08 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo6160worker1 # [ 111.651130] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496446161worker1 # [ 111.660567] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6162worker1 # [ 111.664076] nix-daemon[852]: reaped child process 1333, status = succeeded6163client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...6164worker1 # [ 111.668106] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6165client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6166client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6167client # farm-b> LOG-farm-b-on-standby6168worker1 # [ 113.297690] nix-daemon[852]: reaped child process 1347, status = succeeded6169worker1 # [ 113.441097] nix-daemon[852]: accepted connection from pid 1363, user nix-grpc-daemon (trusted)6170worker1 # [ 113.453098] nix-daemon[852]: reaped child process 1366, status = succeeded6171lb # [ 113.443571] niks3-server[1196]: 2026/09/22 10:48:10 INFO Received uploads request method=POST path=/api/pending_closures6172worker1 # [ 113.477427] nix-grpc-daemon[1285]: time=2026-09-22T10:48:10.599Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6173worker1 # [ 113.479686] nix-grpc-daemon[1285]: time=2026-09-22T10:48:10.601Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"6174lb # [ 113.479857] niks3-server[1196]: 2026/09/22 10:48:10 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6175lb # [ 113.484830] niks3-server[1196]: 2026/09/22 10:48:10 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst6176lb # [ 113.496188] niks3-server[1196]: 2026/09/22 10:48:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6177lb # [ 113.500232] niks3-server[1196]: 2026/09/22 10:48:10 INFO Signed narinfos id=38 count=16178worker1 # [ 113.527169] nix-grpc-daemon[1285]: time=2026-09-22T10:48:10.649Z level=INFO msg="Uploading 1 narinfos"6179lb # [ 113.511664] niks3-server[1196]: 2026/09/22 10:48:10 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls6180lb # [ 113.516695] niks3-server[1196]: 2026/09/22 10:48:10 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6181lb # [ 113.522366] niks3-server[1196]: 2026/09/22 10:48:10 INFO Completed upload id=386182worker1 # [ 113.546427] nix-grpc-daemon[1285]: time=2026-09-22T10:48:10.668Z level=INFO msg="Upload complete. (269ms)"6183lb # [ 113.525725] niks3-server[1196]: 2026/09/22 10:48:10 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo6184worker1 # [ 113.557705] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49644 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=16185worker1 # [ 113.564554] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:10Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=06186worker1 # [ 113.569222] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:10Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=16187worker1 # [ 113.578172] nix-daemon[852]: reaped child process 1345, status = succeeded6188worker1 # [ 113.608374] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496446189worker1 # [ 113.621049] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:496446190worker1 # [ 113.639939] nix-grpc-daemon[1285]: time=2026-09-22T10:48:10.762Z level=INFO msg="All 2 paths already cached"6191client # copying 1 paths to 'lb:50051'...6192client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...6193worker1 # [ 113.648571] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:496446194worker1 # [ 113.657351] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6195worker1 # [ 113.810098] nix-daemon[852]: accepted connection from pid 1372, user nix-grpc-daemon (trusted)6196worker1 # [ 113.823288] nix-daemon[852]: reaped child process 1375, status = succeeded6197worker1 # [ 113.978388] nix-daemon[852]: accepted connection from pid 1377, user nix-grpc-daemon (trusted)6198worker1 # [ 113.991262] nix-daemon[852]: reaped child process 1380, status = succeeded6199lb # [ 113.981295] niks3-server[1196]: 2026/09/22 10:48:10 INFO Received uploads request method=POST path=/api/pending_closures6200worker1 # [ 114.013052] nix-grpc-daemon[1285]: time=2026-09-22T10:48:11.135Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6201worker1 # [ 114.014777] nix-grpc-daemon[1285]: time=2026-09-22T10:48:11.136Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6202lb # [ 114.015149] niks3-server[1196]: 2026/09/22 10:48:10 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6203lb # [ 114.020331] niks3-server[1196]: 2026/09/22 10:48:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6204lb # [ 114.025473] niks3-server[1196]: 2026/09/22 10:48:10 INFO Signed narinfos id=39 count=16205worker1 # [ 114.053654] nix-grpc-daemon[1285]: time=2026-09-22T10:48:11.175Z level=INFO msg="Uploading 1 narinfos"6206lb # [ 114.037299] niks3-server[1196]: 2026/09/22 10:48:10 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6207lb # [ 114.044794] niks3-server[1196]: 2026/09/22 10:48:10 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6208lb # [ 114.058792] niks3-server[1196]: 2026/09/22 10:48:10 INFO Completed upload id=396209worker1 # [ 114.082873] nix-grpc-daemon[1285]: time=2026-09-22T10:48:11.204Z level=INFO msg="Upload complete. (412ms)"6210worker1 # [ 114.087748] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49644 duration_s=0 paths=1 nar_bytes_in=7686211worker1 # [ 114.093293] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496446212lb # [ 114.071177] niks3-server[1196]: 2026/09/22 10:48:10 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6213worker1 # [ 114.102305] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6214worker1 # [ 114.109860] nix-daemon[852]: reaped child process 1370, status = succeeded6215worker1 # [ 114.113828] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6216client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6217client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6218client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6219client # farm-top> LOG-farm-top-on-standby6220worker1 # [ 115.528772] nix-daemon[852]: reaped child process 1384, status = succeeded6221worker1 # [ 115.637645] nix-daemon[852]: accepted connection from pid 1400, user nix-grpc-daemon (trusted)6222worker1 # [ 115.649719] nix-daemon[852]: reaped child process 1403, status = succeeded6223lb # [ 115.639063] niks3-server[1196]: 2026/09/22 10:48:12 INFO Received uploads request method=POST path=/api/pending_closures6224worker1 # [ 115.675493] nix-grpc-daemon[1285]: time=2026-09-22T10:48:12.797Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6225worker1 # [ 115.678054] nix-grpc-daemon[1285]: time=2026-09-22T10:48:12.799Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6226lb # [ 115.679751] niks3-server[1196]: 2026/09/22 10:48:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6227lb # [ 115.683870] niks3-server[1196]: 2026/09/22 10:48:12 INFO Signed narinfos id=40 count=16228worker1 # [ 115.707610] nix-grpc-daemon[1285]: time=2026-09-22T10:48:12.829Z level=INFO msg="Uploading 1 narinfos"6229lb # [ 115.684950] niks3-server[1196]: 2026/09/22 10:48:12 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6230lb # [ 115.689801] niks3-server[1196]: 2026/09/22 10:48:12 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6231lb # [ 115.697782] niks3-server[1196]: 2026/09/22 10:48:12 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6232lb # [ 115.703291] niks3-server[1196]: 2026/09/22 10:48:12 INFO Completed upload id=406233worker1 # [ 115.727255] nix-grpc-daemon[1285]: time=2026-09-22T10:48:12.849Z level=INFO msg="Upload complete. (206ms)"6234worker1 # [ 115.729972] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49644 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16235lb # [ 115.707276] niks3-server[1196]: 2026/09/22 10:48:12 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6236lb # [ 115.710756] niks3-server[1196]: 2026/09/22 10:48:12 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6237worker1 # [ 115.738431] nix-daemon[852]: reaped child process 1382, status = succeeded6238client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 6.67 seconds)6239client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath6240client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.19 seconds)6241worker2 # [ 115.989621] systemd[1]: Listening on Nix worker-protocol over gRPC.6242worker2 # [ 115.995762] systemd[1]: Starting scheduler-token.service...6243worker2 # [ 116.061459] systemd[1]: scheduler-token.service: Deactivated successfully.6244worker2 # [ 116.063068] systemd[1]: Finished scheduler-token.service.6245worker2 # [ 116.064513] systemd[1]: scheduler-token.service: Consumed 20ms CPU time over 67ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6246worker2 # [ 116.069647] systemd[1]: Starting Nix worker-protocol over gRPC...6247worker2 # [ 116.215342] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:13Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6248worker2 # [ 116.217823] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:13Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6249worker2 # [ 116.221526] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:13Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16250worker2 # [ 116.228068] nix-grpc-daemon[1424]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6251worker2 # [ 116.229386] nix-grpc-daemon[1424]: E0000 00:00:1790074093.404965 1424 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6252worker2 # [ 116.234942] nix-grpc-daemon[1424]: E0000 00:00:1790074093.410082 1424 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6253worker2 # [ 116.238886] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:13Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6254worker2 # [ 116.241350] systemd[1]: Started Nix worker-protocol over gRPC.6255lb: must succeed: curl -sf localhost:9901/clusters6256worker2 # [ 116.248090] nix-grpc-daemon[1424]: E0000 00:00:1790074093.423155 1465 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6257worker2 # [ 116.258075] nix-grpc-daemon[1424]: E0000 00:00:1790074093.435529 1465 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6258worker2 # [ 116.262053] nix-grpc-daemon[1424]: E0000 00:00:1790074093.437766 1465 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6259worker2 # [ 116.272050] nix-grpc-daemon[1424]: E0000 00:00:1790074093.449480 1465 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6260worker2 # [ 116.280139] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:13Z level=info event=scheduler_connected addr=lb:500516261worker1 # [ 116.314873] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:13Z level=info event=oidc_jwks provider=mock keys=16262worker1 # [ 116.316493] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:13Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:428406263worker1 # [ 116.318090] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:13Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06264lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6265worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6266worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6267lb: must succeed: curl -sf localhost:9901/clusters6268lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6269lb: must succeed: curl -sf localhost:9901/clusters6270lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6271worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6272worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6273(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 13.40 seconds)6274subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6275client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs6276client # [ 116.555254] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.6277client # Running as unit: rs.service; invocation ID: 47f44b0b391c4520af9e211bac0a7dd36278client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.04 seconds)6279worker1 # [ 116.777399] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:496446280client # [ 116.771157] nix[1322]: don't know how to build these paths:6281client # [ 116.772260] nix[1322]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6282worker1 # [ 116.791164] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:13Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06283worker1 # [ 116.796398] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:13Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=16284worker1 # [ 116.834959] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496446285worker1 # [ 116.843662] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:496446286client # [ 116.836334] nix[1322]: copying 1 paths to 'lb:50051'...6287worker1 # [ 116.849479] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:496446288client # [ 116.839238] nix[1322]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6289worker1 # [ 116.858941] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6290worker1 # [ 117.004604] nix-daemon[852]: accepted connection from pid 1416, user nix-grpc-daemon (trusted)6291worker1 # [ 117.017495] nix-daemon[852]: reaped child process 1419, status = succeeded6292worker1 # [ 117.181062] nix-daemon[852]: accepted connection from pid 1421, user nix-grpc-daemon (trusted)6293worker1 # [ 117.194383] nix-daemon[852]: reaped child process 1424, status = succeeded6294lb # [ 117.182891] niks3-server[1196]: 2026/09/22 10:48:14 INFO Received uploads request method=POST path=/api/pending_closures6295worker1 # [ 117.217043] nix-grpc-daemon[1285]: time=2026-09-22T10:48:14.339Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6296worker1 # [ 117.218601] nix-grpc-daemon[1285]: time=2026-09-22T10:48:14.340Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6297lb # [ 117.214437] niks3-server[1196]: 2026/09/22 10:48:14 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6298lb # [ 117.224993] niks3-server[1196]: 2026/09/22 10:48:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6299lb # [ 117.232094] niks3-server[1196]: 2026/09/22 10:48:14 INFO Signed narinfos id=41 count=16300worker1 # [ 117.259181] nix-grpc-daemon[1285]: time=2026-09-22T10:48:14.381Z level=INFO msg="Uploading 1 narinfos"6301lb # [ 117.237104] niks3-server[1196]: 2026/09/22 10:48:14 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6302lb # [ 117.248298] niks3-server[1196]: 2026/09/22 10:48:14 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6303lb # [ 117.254686] niks3-server[1196]: 2026/09/22 10:48:14 INFO Completed upload id=416304worker1 # [ 117.278670] nix-grpc-daemon[1285]: time=2026-09-22T10:48:14.400Z level=INFO msg="Upload complete. (413ms)"6305lb # [ 117.258269] niks3-server[1196]: 2026/09/22 10:48:14 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6306worker1 # [ 117.282779] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49644 duration_s=0 paths=1 nar_bytes_in=4166307worker1 # [ 117.288142] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496446308worker1 # [ 117.297260] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6309worker1 # [ 117.300438] nix-daemon[852]: reaped child process 1414, status = succeeded6310client # [ 117.291777] nix[1322]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6311worker1 # [ 117.305093] nix-daemon[852]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6312lb # [ 118.681157] envoy[932]: conn peer=[2001:db8:1::1]:59852 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9577 tx=4392 ms=64316313lb: must succeed: curl -sf localhost:9901/clusters6314lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6315lb: must succeed: curl -sf localhost:9901/clusters6316lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6317lb: must succeed: curl -sf localhost:9901/clusters6318lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6319lb: must succeed: curl -sf localhost:9901/clusters6320lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6321worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6322worker2 # [ 119.265416] systemd-journald[286]: Received client request to rotate journal, rotating.6323worker2 # [ 119.276765] systemd-journald[286]: Vacuuming done, freed 0B of archived journals from /var/log/journal/22889ca121274cfa846369b3932ca906.6324worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.04 seconds)6325worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service6326worker1 # [ 119.220905] systemd[1]: Stopping Nix worker-protocol over gRPC...6327worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)6328client: must succeed: sleep 5.06329worker1 # [ 119.406044] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:16Z level=info event=draining6330client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6331worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6332worker1 # [ 124.526781] nix-daemon[852]: reaped child process 1428, status = succeeded6333worker1 # [ 124.654026] nix-daemon[852]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6334worker1 # [ 124.666297] nix-daemon[852]: reaped child process 1472, status = succeeded6335lb # [ 124.655572] niks3-server[1196]: 2026/09/22 10:48:21 INFO Received uploads request method=POST path=/api/pending_closures6336worker1 # [ 124.684275] nix-grpc-daemon[1285]: time=2026-09-22T10:48:21.806Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6337worker1 # [ 124.687252] nix-grpc-daemon[1285]: time=2026-09-22T10:48:21.808Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6338lb # [ 124.678743] niks3-server[1196]: 2026/09/22 10:48:21 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6339lb # [ 124.681823] niks3-server[1196]: 2026/09/22 10:48:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6340lb # [ 124.684327] niks3-server[1196]: 2026/09/22 10:48:21 INFO Signed narinfos id=42 count=16341worker1 # [ 124.708587] nix-grpc-daemon[1285]: time=2026-09-22T10:48:21.830Z level=INFO msg="Uploading 1 narinfos"6342lb # [ 124.687779] niks3-server[1196]: 2026/09/22 10:48:21 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6343lb # [ 124.690583] niks3-server[1196]: 2026/09/22 10:48:21 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6344lb # [ 124.694132] niks3-server[1196]: 2026/09/22 10:48:21 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6345lb # [ 124.696968] niks3-server[1196]: 2026/09/22 10:48:21 INFO Completed upload id=426346worker1 # [ 124.722069] nix-grpc-daemon[1285]: time=2026-09-22T10:48:21.844Z level=INFO msg="Upload complete. (204ms)"6347lb # [ 124.698922] niks3-server[1196]: 2026/09/22 10:48:21 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6348worker1 # [ 124.725712] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49644 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16349worker1 # [ 124.732948] nix-daemon[852]: reaped child process 1426, status = succeeded6350client # [ 124.737708] systemd[1]: rs.service: Deactivated successfully.6351client # [ 124.739568] systemd[1]: rs.service: Consumed 123ms CPU time over 8.183s wall clock time, 41.4M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.6352worker1 # [ 125.406419] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:22Z level=info event=signal_exit6353lb # [ 125.813265] niks3-server[1196]: 2026/09/22 10:48:22 INFO lead: released remote=192.168.1.3:452126354lb # [ 126.251186] niks3-server[1196]: 2026/09/22 10:48:23 INFO lead: acquired remote=192.168.1.4:573006355worker2 # [ 126.247029] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:23Z level=info event=scheduler_take_over6356worker2 # [ 126.806160] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:23Z level=info event=scheduler_restarting addr=lb:500516357worker1 # [ 126.840943] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:23Z level=info event=scheduler_restarting6358worker1 # [ 126.845484] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:23Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:428406359worker1 # [ 126.849160] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:23Z level=info event=worker_gone addr=192.168.1.4:500516360worker1 # [ 126.852596] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:23Z level=info event=scheduler_restarting addr=lb:500516361worker1 # [ 126.856343] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:23Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:428406362worker1 # [ 126.860087] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:23Z level=info event=worker_gone addr=192.168.1.3:500516363worker1 # [ 126.873432] nix-daemon[852]: reaped child process 1258, status = succeeded6364worker2 # [ 127.010865] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:24Z level=info event=scheduler_connected addr=lb:500516365worker2 # [ 127.014779] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6366worker1 # [ 127.048382] nix-grpc-daemon[1206]: E0000 00:00:1790074104.170599 1251 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true6367worker1 # [ 127.058132] nix-grpc-daemon[1206]: E0000 00:00:1790074104.176389 1251 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6368worker1 # [ 127.062887] nix-grpc-daemon[1206]: ts=2026-09-22T10:48:24Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6369worker2 # [ 127.212877] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:24Z level=info event=scheduler_connected addr=lb:500516370worker2 # [ 127.216715] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6371worker2 # [ 127.419140] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:24Z level=info event=scheduler_connected addr=lb:500516372worker2 # [ 127.422276] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6373worker2 # [ 127.620509] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:24Z level=info event=scheduler_connected addr=lb:500516374worker2 # [ 127.624620] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6375worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.18 seconds)6376client: waiting for success: ! systemctl is-active rs6377client: (finished: waiting for success: ! systemctl is-active rs, in 0.03 seconds)6378client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6379client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)6380client: must fail: journalctl -u rs | grep -q 'reconnecting'6381client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.02 seconds)6382lb: must succeed: curl -sf localhost:9901/clusters6383lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6384worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6385worker2 # [ 127.825481] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:25Z level=info event=scheduler_connected addr=lb:500516386worker2 # [ 127.829470] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6387worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6388worker2 # [ 128.029635] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:25Z level=info event=scheduler_connected addr=lb:500516389worker2 # [ 128.039068] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:25Z level=info event=oidc_jwks provider=mock keys=16390worker2 # [ 128.040586] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:25Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:501346391worker2 # [ 128.042151] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:25Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06392worker1 # [ 128.436696] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6393worker1 # [ 128.438688] systemd[1]: Stopped Nix worker-protocol over gRPC.6394worker1 # [ 128.441689] systemd[1]: nix-grpc-daemon.service: Consumed 1.314s CPU time over 33.180s wall clock time, 35.8M memory peak, 48.6K incoming IP traffic, 105.3K outgoing IP traffic.6395worker1 # [ 128.451715] systemd[1]: Starting Nix worker-protocol over gRPC...6396worker1 # [ 128.597935] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:25Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6397worker1 # [ 128.599834] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:25Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6398worker1 # [ 128.603536] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:25Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16399worker1 # [ 128.610072] nix-grpc-daemon[1479]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6400worker1 # [ 128.611430] nix-grpc-daemon[1479]: E0000 00:00:1790074105.731867 1479 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6401worker1 # [ 128.616712] nix-grpc-daemon[1479]: E0000 00:00:1790074105.737227 1479 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6402worker1 # [ 128.619762] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:25Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6403worker1 # [ 128.621312] systemd[1]: Started Nix worker-protocol over gRPC.6404worker1 # [ 128.628070] nix-grpc-daemon[1479]: E0000 00:00:1790074105.749294 1521 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6405worker1 # [ 128.639511] nix-grpc-daemon[1479]: E0000 00:00:1790074105.761822 1521 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6406worker1 # [ 128.645057] nix-grpc-daemon[1479]: E0000 00:00:1790074105.764520 1521 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6407worker1 # [ 128.654395] nix-grpc-daemon[1479]: E0000 00:00:1790074105.776713 1521 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6408worker1 # [ 128.662293] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:25Z level=info event=scheduler_connected addr=lb:500516409worker2 # [ 128.633931] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:25Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:501346410worker2 # [ 128.638586] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:25Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06411lb # [ 128.681208] envoy[932]: conn peer=[2001:db8:1::1]:49888 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2973 tx=1536 ms=79606412lb # [ 128.682796] envoy[932]: conn peer=[2001:db8:1::3]:49364 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=4581 tx=1106 ms=316006413lb: must succeed: curl -sf localhost:9901/clusters6414lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6415worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6416worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6417(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.49 seconds)6418subtest: scheduler restart mid-build: build finishes, no double build6419client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid6420client # [ 129.069721] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.6421client # Running as unit: mid.service; invocation ID: a2306b1e8a0046128c58cbbfa359ea8e6422client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)6423worker2 # [ 129.267145] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:501426424worker2 # [ 129.278431] nix-daemon[873]: accepted connection from pid 1424, user nix-grpc-daemon (trusted)6425client # [ 129.307348] nix[1358]: don't know how to build these paths:6426client # [ 129.308202] nix[1358]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6427worker2 # [ 129.297888] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:26Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06428worker2 # [ 129.302643] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:26Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=16429worker2 # [ 129.346535] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:501426430worker2 # [ 129.358230] nix-grpc-daemon[1424]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 18 ms (attempt 1/5)6431worker2 # [ 129.381946] nix-grpc-daemon[1424]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 35 ms (attempt 2/5)6432worker2 # [ 129.422742] nix-grpc-daemon[1424]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 145 ms (attempt 3/5)6433worker2 # [ 129.573431] nix-grpc-daemon[1424]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 51 ms (attempt 4/5)6434worker2 # [ 129.624996] nix-grpc-daemon[1424]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6435worker2 # [ 129.628291] nix-grpc-daemon[1424]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6436worker2 # [ 129.647196] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:501426437client # [ 129.673881] nix[1358]: copying 1 paths to 'lb:50051'...6438client # [ 129.674770] nix[1358]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6439worker2 # [ 129.657326] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:501426440worker2 # [ 129.667129] nix-daemon[873]: accepted connection from pid 1424, user nix-grpc-daemon (trusted)6441worker2 # [ 129.686159] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:26Z level=info event=niks3_push_started pid=15526442worker2 # [ 129.838471] nix-grpc-daemon[1552]: time=2026-09-22T10:48:27.015Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6443worker2 # [ 129.996066] nix-daemon[873]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6444worker2 # [ 130.006853] nix-daemon[873]: reaped child process 1568, status = succeeded6445worker2 # [ 130.160502] nix-daemon[873]: accepted connection from pid 1570, user nix-grpc-daemon (trusted)6446worker2 # [ 130.172234] nix-daemon[873]: reaped child process 1573, status = succeeded6447lb # [ 130.192838] niks3-server[1196]: 2026/09/22 10:48:27 INFO Received uploads request method=POST path=/api/pending_closures6448worker2 # [ 130.197420] nix-grpc-daemon[1552]: time=2026-09-22T10:48:27.374Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6449worker2 # [ 130.201702] nix-grpc-daemon[1552]: time=2026-09-22T10:48:27.376Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6450lb # [ 130.234259] niks3-server[1196]: 2026/09/22 10:48:27 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6451lb # [ 130.247129] niks3-server[1196]: 2026/09/22 10:48:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6452worker2 # [ 130.249961] nix-grpc-daemon[1552]: time=2026-09-22T10:48:27.426Z level=INFO msg="Uploading 1 narinfos"6453lb # [ 130.256656] niks3-server[1196]: 2026/09/22 10:48:27 INFO Signed narinfos id=43 count=16454lb # [ 130.271198] niks3-server[1196]: 2026/09/22 10:48:27 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6455lb # [ 130.274802] niks3-server[1196]: 2026/09/22 10:48:27 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6456lb # [ 130.281342] niks3-server[1196]: 2026/09/22 10:48:27 INFO Completed upload id=436457worker2 # [ 130.275709] nix-grpc-daemon[1552]: time=2026-09-22T10:48:27.452Z level=INFO msg="Upload complete. (433ms)"6458worker2 # [ 130.278951] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50142 duration_s=0 paths=1 nar_bytes_in=4246459worker2 # [ 130.282590] nix-grpc-daemon[1424]: ts=2026-09-22T10:48:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:501426460worker2 # [ 130.289306] nix-daemon[873]: accepted connection from pid 1424, user nix-grpc-daemon (trusted)6461worker2 # [ 130.299724] nix-daemon[873]: reaped child process 1550, status = succeeded6462client # [ 130.321668] nix[1358]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6463worker2 # [ 130.306077] nix-daemon[873]: accepted connection from pid 1424, user nix-grpc-daemon (trusted)6464lb: must succeed: curl -sf localhost:9901/clusters6465lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6466lb: must succeed: curl -sf localhost:9901/clusters6467lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6468worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6469worker2 # [ 131.694285] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1424 (nix-grpc-daemon) on client request.6470worker2 # [ 131.696516] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6471lb # [ 131.705815] niks3-server[1196]: 2026/09/22 10:48:28 INFO lead: released remote=192.168.1.4:573006472client # [ 131.725650] nix[1358]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6473worker1 # [ 131.737827] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:28Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6474worker2 # [ 131.714791] systemd[1]: Starting scheduler-token.service...6475client # [ 131.730303] nix[1358]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6476worker2 # [ 131.726607] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6477worker2 # [ 131.728177] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6478worker2 # [ 131.732720] systemd[1]: nix-grpc-daemon.service: Consumed 415ms CPU time over 15.655s wall clock time, 33.7M memory peak, 15.7K incoming IP traffic, 30K outgoing IP traffic.6479worker2 # [ 131.738495] nix-daemon[873]: reaped child process 1579, status = succeeded6480worker2 # [ 131.740251] nix-daemon[873]: reaped child process 1542, status = succeeded6481worker2 # [ 131.750252] nix-daemon[1581]: unexpected Nix daemon error: error: interrupted by the user6482worker2 # [ 131.755173] nix-daemon[873]: reaped child process 1581, status = failed with exit code 16483worker2 # [ 131.791982] systemd[1]: scheduler-token.service: Deactivated successfully.6484worker2 # [ 131.793549] systemd[1]: Finished scheduler-token.service.6485worker2 # [ 131.794824] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 77ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6486worker2 # [ 131.797771] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6487worker2 # [ 131.802091] systemd[1]: Starting scheduler-token.service...6488worker2 # [ 131.856686] systemd[1]: scheduler-token.service: Deactivated successfully.6489worker2 # [ 131.857784] systemd[1]: Finished scheduler-token.service.6490worker2 # [ 131.859068] systemd[1]: scheduler-token.service: Consumed 16ms CPU time over 54ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6491worker2 # [ 131.864203] systemd[1]: Starting Nix worker-protocol over gRPC...6492worker1 # [ 131.937781] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:29Z level=info event=scheduler_connected addr=lb:500516493worker2 # [ 132.008896] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6494worker2 # [ 132.011165] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6495worker2 # [ 132.014620] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16496worker2 # [ 132.020060] nix-grpc-daemon[1620]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6497worker2 # [ 132.023098] nix-grpc-daemon[1620]: E0000 00:00:1790074109.197434 1620 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6498worker2 # [ 132.025247] nix-grpc-daemon[1620]: E0000 00:00:1790074109.200452 1620 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6499worker2 # [ 132.032078] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6500worker2 # [ 132.033733] systemd[1]: Started Nix worker-protocol over gRPC.6501worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.38 seconds)6502client: must succeed: sleep 5.06503worker2 # [ 132.042081] nix-grpc-daemon[1620]: E0000 00:00:1790074109.219166 1662 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6504lb # [ 132.048921] niks3-server[1196]: 2026/09/22 10:48:28 INFO lead: acquired remote=192.168.1.4:598286505worker2 # [ 132.045901] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=scheduler_take_over6506worker2 # [ 132.048823] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:501906507worker2 # [ 132.051591] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06508worker2 # [ 132.062127] nix-grpc-daemon[1620]: E0000 00:00:1790074109.239569 1662 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6509worker2 # [ 132.066056] nix-grpc-daemon[1620]: E0000 00:00:1790074109.241802 1662 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6510worker2 # [ 132.076446] nix-grpc-daemon[1620]: E0000 00:00:1790074109.253851 1662 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6511worker2 # [ 132.085544] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=scheduler_connected addr=lb:500516512worker2 # [ 132.087178] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6513client # [ 132.231939] nix[1358]: scheduler at lb:50051: no healthy upstream, reconnecting6514worker2 # [ 132.288680] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=scheduler_connected addr=lb:500516515worker2 # [ 132.292461] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6516worker2 # [ 132.493182] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=scheduler_connected addr=lb:500516517worker2 # [ 132.495242] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6518worker2 # [ 132.696125] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=scheduler_connected addr=lb:500516519worker2 # [ 132.698173] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6520worker2 # [ 132.898937] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:30Z level=info event=scheduler_connected addr=lb:500516521worker2 # [ 132.900947] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6522worker2 # [ 133.104234] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:30Z level=info event=scheduler_connected addr=lb:500516523worker2 # [ 133.114136] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:30Z level=info event=oidc_jwks provider=mock keys=16524worker2 # [ 133.117926] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:30Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:447966525worker2 # [ 133.121874] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:30Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06526worker2 # [ 133.224163] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:30Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06527worker2 # [ 133.228165] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:30Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16528worker1 # [ 133.274762] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:439546529worker1 # [ 133.286284] nix-daemon[852]: accepted connection from pid 1479, user nix-grpc-daemon (trusted)6530worker1 # [ 133.303736] nix-grpc-daemon[1479]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 69 ms (attempt 1/5)6531worker1 # [ 133.378793] nix-grpc-daemon[1479]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 145 ms (attempt 2/5)6532worker1 # [ 133.526885] nix-grpc-daemon[1479]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 146 ms (attempt 3/5)6533worker1 # [ 133.673632] nix-grpc-daemon[1479]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 461 ms (attempt 4/5)6534worker1 # [ 134.141056] nix-grpc-daemon[1479]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6535worker1 # [ 134.145239] nix-grpc-daemon[1479]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6536worker1 # [ 134.160240] nix-grpc-daemon[1479]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' from 'http://lb:5751'...6537worker1 # [ 134.179754] nix-daemon[852]: accepted connection from pid 1479, user nix-grpc-daemon (trusted)6538worker1 # [ 134.195374] nix-daemon[852]: accepted connection from pid 1479, user nix-grpc-daemon (trusted)6539client # [ 134.182458] nix[1358]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6540client: (finished: must succeed: sleep 5.0, in 5.04 seconds)6541client: waiting for success: ! systemctl is-active mid6542worker1 # [ 137.419298] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:34Z level=info event=niks3_push_started pid=15816543worker1 # [ 137.450083] nix-daemon[852]: reaped child process 1556, status = succeeded6544worker1 # [ 137.618754] nix-grpc-daemon[1581]: time=2026-09-22T10:48:34.740Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6545worker1 # [ 137.782260] nix-daemon[852]: accepted connection from pid 1594, user nix-grpc-daemon (trusted)6546worker1 # [ 137.795142] nix-daemon[852]: reaped child process 1597, status = succeeded6547lb # [ 137.784764] niks3-server[1196]: 2026/09/22 10:48:34 INFO Received uploads request method=POST path=/api/pending_closures6548worker1 # [ 137.816930] nix-grpc-daemon[1581]: time=2026-09-22T10:48:34.939Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6549worker1 # [ 137.818469] nix-grpc-daemon[1581]: time=2026-09-22T10:48:34.940Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6550lb # [ 137.837214] niks3-server[1196]: 2026/09/22 10:48:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6551lb # [ 137.843820] niks3-server[1196]: 2026/09/22 10:48:34 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6552lb # [ 137.854506] niks3-server[1196]: 2026/09/22 10:48:34 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6553lb # [ 137.866944] niks3-server[1196]: 2026/09/22 10:48:34 INFO Signed narinfos id=44 count=16554worker1 # [ 137.890793] nix-grpc-daemon[1581]: time=2026-09-22T10:48:35.013Z level=INFO msg="Uploading 1 narinfos"6555lb # [ 137.873792] niks3-server[1196]: 2026/09/22 10:48:34 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6556lb # [ 137.878069] niks3-server[1196]: 2026/09/22 10:48:34 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6557lb # [ 137.885432] niks3-server[1196]: 2026/09/22 10:48:34 INFO Completed upload id=446558worker1 # [ 137.909390] nix-grpc-daemon[1581]: time=2026-09-22T10:48:35.031Z level=INFO msg="Upload complete. (287ms)"6559worker1 # [ 137.912052] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43954 duration_s=4 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16560lb # [ 137.891609] niks3-server[1196]: 2026/09/22 10:48:34 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6561worker1 # [ 137.919622] nix-daemon[852]: reaped child process 1554, status = succeeded6562client # [ 137.943606] systemd[1]: mid.service: Deactivated successfully.6563client # [ 137.947851] systemd[1]: mid.service: Consumed 138ms CPU time over 8.877s wall clock time, 41.4M memory peak, 9K incoming IP traffic, 11.1K outgoing IP traffic.6564client: (finished: waiting for success: ! systemctl is-active mid, in 1.11 seconds)6565client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6566client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds)6567(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 9.48 seconds)6568subtest: interrupting the client stops the build on the worker6569client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6570client # [ 138.523171] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6571client # Running as unit: intr.service; invocation ID: 14f9d0646b9f472099b2297a5966bbc96572client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.04 seconds)6573lb # [ 138.682389] envoy[932]: conn peer=[2001:db8:1::4]:42444 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=6171 tx=511 ms=154306574lb # [ 138.687205] envoy[932]: conn peer=[2001:db8:1::1]:50826 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4119 tx=2036 ms=86316575worker2 # [ 138.716238] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:448006576worker2 # [ 138.727233] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6577client # [ 138.757364] nix[1394]: don't know how to build these paths:6578client # [ 138.758878] nix[1394]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6579worker2 # [ 138.752462] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:35Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06580worker2 # [ 138.757634] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:35Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16581worker2 # [ 138.802318] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448006582worker2 # [ 138.814307] nix-grpc-daemon[1620]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 32 ms (attempt 1/5)6583worker2 # [ 138.852733] nix-grpc-daemon[1620]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 167 ms (attempt 2/5)6584worker2 # [ 139.025613] nix-grpc-daemon[1620]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 368 ms (attempt 3/5)6585worker2 # [ 139.400579] nix-grpc-daemon[1620]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 793 ms (attempt 4/5)6586worker2 # [ 140.200890] nix-grpc-daemon[1620]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6587worker2 # [ 140.205180] nix-grpc-daemon[1620]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6588worker2 # [ 140.223420] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:448006589client # [ 140.251331] nix[1394]: copying 1 paths to 'lb:50051'...6590client # [ 140.252220] nix[1394]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6591worker2 # [ 140.234393] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:448006592worker2 # [ 140.246152] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6593worker2 # [ 140.265460] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:37Z level=info event=niks3_push_started pid=17006594worker2 # [ 140.440828] nix-grpc-daemon[1700]: time=2026-09-22T10:48:37.617Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6595worker2 # [ 140.599372] nix-daemon[873]: accepted connection from pid 1713, user nix-grpc-daemon (trusted)6596worker2 # [ 140.614530] nix-daemon[873]: reaped child process 1716, status = succeeded6597worker2 # [ 140.772758] nix-daemon[873]: accepted connection from pid 1718, user nix-grpc-daemon (trusted)6598worker2 # [ 140.787196] nix-daemon[873]: reaped child process 1721, status = succeeded6599lb # [ 140.807368] niks3-server[1196]: 2026/09/22 10:48:37 INFO Received uploads request method=POST path=/api/pending_closures6600worker2 # [ 140.807814] nix-grpc-daemon[1700]: time=2026-09-22T10:48:37.985Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6601worker2 # [ 140.811169] nix-grpc-daemon[1700]: time=2026-09-22T10:48:37.986Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6602lb # [ 140.838625] niks3-server[1196]: 2026/09/22 10:48:37 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6603lb # [ 140.848497] niks3-server[1196]: 2026/09/22 10:48:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6604lb # [ 140.853256] niks3-server[1196]: 2026/09/22 10:48:37 INFO Signed narinfos id=45 count=16605worker2 # [ 140.850520] nix-grpc-daemon[1700]: time=2026-09-22T10:48:38.027Z level=INFO msg="Uploading 1 narinfos"6606lb # [ 140.866885] niks3-server[1196]: 2026/09/22 10:48:37 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6607lb # [ 140.871219] niks3-server[1196]: 2026/09/22 10:48:37 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6608lb # [ 140.880678] niks3-server[1196]: 2026/09/22 10:48:37 INFO Completed upload id=456609worker2 # [ 140.874424] nix-grpc-daemon[1700]: time=2026-09-22T10:48:38.051Z level=INFO msg="Upload complete. (430ms)"6610lb # [ 140.882347] niks3-server[1196]: 2026/09/22 10:48:37 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6611worker2 # [ 140.876816] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=0 paths=1 nar_bytes_in=4246612worker2 # [ 140.879747] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448006613worker2 # [ 140.885279] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6614worker2 # [ 140.892743] nix-daemon[873]: reaped child process 1698, status = succeeded6615client # [ 140.912455] nix[1394]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6616worker2 # [ 140.895778] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6617client: must succeed: systemctl kill -s INT intr6618client # [ 142.042704] systemd[1]: intr.service: Sent signal SIGINT to main process 1394 (nix) on client request.6619worker2 # [ 142.027180] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:39Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6620client # [ 142.049656] nix[1394]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6621client: (finished: must succeed: systemctl kill -s INT intr, in 0.05 seconds)6622client # [ 142.054732] nix[1394]: error: interrupted by the user6623client # [ 142.075285] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6624client # [ 142.076677] systemd[1]: intr.service: Failed with result 'exit-code'.6625client # [ 142.078953] systemd[1]: intr.service: Consumed 108ms CPU time over 3.555s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.2K outgoing IP traffic.6626worker2 # [ 142.125303] nix-daemon[1729]: unexpected Nix daemon error: error: interrupted by the user6627worker2 # [ 142.126873] nix-daemon[873]: reaped child process 1727, status = succeeded6628worker2 # [ 142.131542] nix-daemon[873]: reaped child process 1729, status = failed with exit code 16629(finished: subtest: interrupting the client stops the build on the worker, in 3.66 seconds)6630subtest: a deploy mid-build drains: the build finishes, the new generation takes over6631client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6632client # [ 142.181711] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6633client # Running as unit: sw.service; invocation ID: b2b30940707a42f79901f0ac4c9265406634client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.04 seconds)6635worker2 # [ 142.363816] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:448006636client # [ 142.387879] nix[1415]: don't know how to build these paths:6637client # [ 142.388750] nix[1415]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6638worker2 # [ 142.381168] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:39Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06639worker2 # [ 142.385095] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:39Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16640worker1 # [ 142.424759] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:439546641worker1 # [ 142.436783] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:439546642client # [ 142.434520] nix[1415]: copying 1 paths to 'lb:50051'...6643client # [ 142.435395] nix[1415]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6644worker1 # [ 142.448448] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:439546645worker1 # [ 142.462981] nix-daemon[852]: accepted connection from pid 1479, user nix-grpc-daemon (trusted)6646worker1 # [ 142.624593] nix-daemon[852]: accepted connection from pid 1625, user nix-grpc-daemon (trusted)6647worker1 # [ 142.638644] nix-daemon[852]: reaped child process 1628, status = succeeded6648worker1 # [ 142.784953] nix-daemon[852]: accepted connection from pid 1630, user nix-grpc-daemon (trusted)6649worker1 # [ 142.796858] nix-daemon[852]: reaped child process 1633, status = succeeded6650lb # [ 142.788745] niks3-server[1196]: 2026/09/22 10:48:39 INFO Received uploads request method=POST path=/api/pending_closures6651worker1 # [ 142.821783] nix-grpc-daemon[1581]: time=2026-09-22T10:48:39.943Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6652worker1 # [ 142.823855] nix-grpc-daemon[1581]: time=2026-09-22T10:48:39.945Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6653lb # [ 142.815564] niks3-server[1196]: 2026/09/22 10:48:39 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6654lb # [ 142.827875] niks3-server[1196]: 2026/09/22 10:48:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6655lb # [ 142.833051] niks3-server[1196]: 2026/09/22 10:48:39 INFO Signed narinfos id=46 count=16656worker1 # [ 142.859927] nix-grpc-daemon[1581]: time=2026-09-22T10:48:39.982Z level=INFO msg="Uploading 1 narinfos"6657lb # [ 142.839796] niks3-server[1196]: 2026/09/22 10:48:39 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6658lb # [ 142.852747] niks3-server[1196]: 2026/09/22 10:48:39 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6659worker1 # [ 142.882761] nix-grpc-daemon[1581]: time=2026-09-22T10:48:40.005Z level=INFO msg="Upload complete. (407ms)"6660lb # [ 142.858882] niks3-server[1196]: 2026/09/22 10:48:39 INFO Completed upload id=466661worker1 # [ 142.884978] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43954 duration_s=0 paths=1 nar_bytes_in=4166662lb # [ 142.862317] niks3-server[1196]: 2026/09/22 10:48:39 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6663worker1 # [ 142.890670] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:439546664worker1 # [ 142.900914] nix-daemon[852]: accepted connection from pid 1479, user nix-grpc-daemon (trusted)6665worker1 # [ 142.906775] nix-daemon[852]: reaped child process 1623, status = succeeded6666worker1 # [ 142.909791] nix-daemon[852]: accepted connection from pid 1479, user nix-grpc-daemon (trusted)6667client # [ 142.896697] nix[1415]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6668worker1: must succeed: systemctl reload nix-grpc-daemon.service6669worker1 # [ 144.744257] systemd[1]: Reloading Nix worker-protocol over gRPC...6670worker1 # [ 144.783268] systemd[1]: Reloaded Nix worker-protocol over gRPC.6671worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.08 seconds)6672worker1: must succeed: pgrep -f 'read -t [9]0 x'6673worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.04 seconds)6674client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26675client # don't know how to build these paths:6676client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6677worker2 # [ 145.004439] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:42Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:448006678worker1 # [ 145.038064] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:439546679worker2 # [ 145.028588] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:42Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06680worker2 # [ 145.033642] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:42Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16681worker2 # [ 145.073376] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448006682worker2 # [ 145.088132] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:448006683client # copying 1 paths to 'lb:50051'...6684client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6685worker2 # [ 145.100682] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:448006686worker2 # [ 145.112189] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6687worker2 # [ 145.281725] nix-daemon[873]: accepted connection from pid 1776, user nix-grpc-daemon (trusted)6688worker2 # [ 145.296785] nix-daemon[873]: reaped child process 1779, status = succeeded6689worker2 # [ 145.450794] nix-daemon[873]: accepted connection from pid 1781, user nix-grpc-daemon (trusted)6690worker2 # [ 145.464459] nix-daemon[873]: reaped child process 1784, status = succeeded6691lb # [ 145.483957] niks3-server[1196]: 2026/09/22 10:48:42 INFO Received uploads request method=POST path=/api/pending_closures6692worker2 # [ 145.482343] nix-grpc-daemon[1700]: time=2026-09-22T10:48:42.659Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6693worker2 # [ 145.484288] nix-grpc-daemon[1700]: time=2026-09-22T10:48:42.661Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6694lb # [ 145.501061] niks3-server[1196]: 2026/09/22 10:48:42 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6695lb # [ 145.505057] niks3-server[1196]: 2026/09/22 10:48:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6696lb # [ 145.506943] niks3-server[1196]: 2026/09/22 10:48:42 INFO Signed narinfos id=47 count=16697worker2 # [ 145.500895] nix-grpc-daemon[1700]: time=2026-09-22T10:48:42.678Z level=INFO msg="Uploading 1 narinfos"6698lb # [ 145.515100] niks3-server[1196]: 2026/09/22 10:48:42 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6699lb # [ 145.517650] niks3-server[1196]: 2026/09/22 10:48:42 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6700lb # [ 145.521664] niks3-server[1196]: 2026/09/22 10:48:42 INFO Completed upload id=476701worker2 # [ 145.515235] nix-grpc-daemon[1700]: time=2026-09-22T10:48:42.692Z level=INFO msg="Upload complete. (390ms)"6702lb # [ 145.523868] niks3-server[1196]: 2026/09/22 10:48:42 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6703worker2 # [ 145.517502] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=0 paths=1 nar_bytes_in=5286704worker2 # [ 145.522039] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448006705worker2 # [ 145.528457] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6706worker2 # [ 145.530863] nix-daemon[873]: reaped child process 1774, status = succeeded6707client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6708client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6709worker2 # [ 145.536092] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6710worker1 # [ 145.620738] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:42Z level=info event=draining6711client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6712client # farm-a> LOG-farm-a-during-drain6713worker2 # [ 146.360988] nix-daemon[873]: reaped child process 1788, status = succeeded6714worker2 # [ 146.474949] nix-daemon[873]: accepted connection from pid 1804, user nix-grpc-daemon (trusted)6715worker2 # [ 146.489026] nix-daemon[873]: reaped child process 1807, status = succeeded6716lb # [ 146.507660] niks3-server[1196]: 2026/09/22 10:48:43 INFO Received uploads request method=POST path=/api/pending_closures6717worker2 # [ 146.510371] nix-grpc-daemon[1700]: time=2026-09-22T10:48:43.687Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6718worker2 # [ 146.511857] nix-grpc-daemon[1700]: time=2026-09-22T10:48:43.689Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6719lb # [ 146.547576] niks3-server[1196]: 2026/09/22 10:48:43 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6720lb # [ 146.550109] niks3-server[1196]: 2026/09/22 10:48:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6721lb # [ 146.553573] niks3-server[1196]: 2026/09/22 10:48:43 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6722lb # [ 146.556267] niks3-server[1196]: 2026/09/22 10:48:43 INFO Signed narinfos id=48 count=16723lb # [ 146.558115] niks3-server[1196]: 2026/09/22 10:48:43 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6724worker2 # [ 146.551739] nix-grpc-daemon[1700]: time=2026-09-22T10:48:43.728Z level=INFO msg="Uploading 1 narinfos"6725lb # [ 146.564065] niks3-server[1196]: 2026/09/22 10:48:43 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6726lb # [ 146.567578] niks3-server[1196]: 2026/09/22 10:48:43 INFO Completed upload id=486727worker2 # [ 146.561231] nix-grpc-daemon[1700]: time=2026-09-22T10:48:43.738Z level=INFO msg="Upload complete. (218ms)"6728lb # [ 146.570223] niks3-server[1196]: 2026/09/22 10:48:43 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6729worker2 # [ 146.564245] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16730worker2 # [ 146.570874] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:43Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06731worker2 # [ 146.573245] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:43Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16732worker2 # [ 146.578697] nix-daemon[873]: reaped child process 1786, status = succeeded6733worker2 # [ 146.614918] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448006734worker2 # [ 146.625308] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:448006735worker2 # [ 146.637893] nix-grpc-daemon[1700]: time=2026-09-22T10:48:43.815Z level=INFO msg="All 1 paths already cached"6736client # copying 1 paths to 'lb:50051'...6737client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6738worker2 # [ 146.645683] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:448006739worker2 # [ 146.652703] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6740worker2 # [ 146.812176] nix-daemon[873]: accepted connection from pid 1813, user nix-grpc-daemon (trusted)6741worker2 # [ 146.829071] nix-daemon[873]: reaped child process 1816, status = succeeded6742worker2 # [ 146.974948] nix-daemon[873]: accepted connection from pid 1818, user nix-grpc-daemon (trusted)6743worker2 # [ 146.990191] nix-daemon[873]: reaped child process 1821, status = succeeded6744lb # [ 147.007707] niks3-server[1196]: 2026/09/22 10:48:43 INFO Received uploads request method=POST path=/api/pending_closures6745worker2 # [ 147.005439] nix-grpc-daemon[1700]: time=2026-09-22T10:48:44.182Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6746worker2 # [ 147.006981] nix-grpc-daemon[1700]: time=2026-09-22T10:48:44.184Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6747lb # [ 147.024229] niks3-server[1196]: 2026/09/22 10:48:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6748lb # [ 147.026748] niks3-server[1196]: 2026/09/22 10:48:43 INFO Signed narinfos id=49 count=16749worker2 # [ 147.020422] nix-grpc-daemon[1700]: time=2026-09-22T10:48:44.197Z level=INFO msg="Uploading 1 narinfos"6750lb # [ 147.029739] niks3-server[1196]: 2026/09/22 10:48:43 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6751lb # [ 147.032348] niks3-server[1196]: 2026/09/22 10:48:43 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6752lb # [ 147.036839] niks3-server[1196]: 2026/09/22 10:48:43 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6753lb # [ 147.039470] niks3-server[1196]: 2026/09/22 10:48:43 INFO Completed upload id=496754worker2 # [ 147.033103] nix-grpc-daemon[1700]: time=2026-09-22T10:48:44.210Z level=INFO msg="Upload complete. (369ms)"6755worker2 # [ 147.035408] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=0 paths=1 nar_bytes_in=6406756lb # [ 147.042544] niks3-server[1196]: 2026/09/22 10:48:43 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6757worker2 # [ 147.039567] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448006758worker2 # [ 147.046291] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6759worker2 # [ 147.049751] nix-daemon[873]: reaped child process 1811, status = succeeded6760worker2 # [ 147.055179] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6761client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6762client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6763client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6764client # farm-b> LOG-farm-b-during-drain6765worker2 # [ 147.885573] nix-daemon[873]: reaped child process 1825, status = succeeded6766worker2 # [ 148.021531] nix-daemon[873]: accepted connection from pid 1841, user nix-grpc-daemon (trusted)6767worker2 # [ 148.037455] nix-daemon[873]: reaped child process 1844, status = succeeded6768lb # [ 148.053054] niks3-server[1196]: 2026/09/22 10:48:44 INFO Received uploads request method=POST path=/api/pending_closures6769worker2 # [ 148.053061] nix-grpc-daemon[1700]: time=2026-09-22T10:48:45.230Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6770worker2 # [ 148.055127] nix-grpc-daemon[1700]: time=2026-09-22T10:48:45.231Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6771lb # [ 148.071447] niks3-server[1196]: 2026/09/22 10:48:44 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6772lb # [ 148.076647] niks3-server[1196]: 2026/09/22 10:48:44 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6773lb # [ 148.078781] niks3-server[1196]: 2026/09/22 10:48:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6774lb # [ 148.080925] niks3-server[1196]: 2026/09/22 10:48:44 INFO Signed narinfos id=50 count=16775worker2 # [ 148.074500] nix-grpc-daemon[1700]: time=2026-09-22T10:48:45.251Z level=INFO msg="Uploading 1 narinfos"6776lb # [ 148.084641] niks3-server[1196]: 2026/09/22 10:48:44 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6777lb # [ 148.088824] niks3-server[1196]: 2026/09/22 10:48:44 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6778lb # [ 148.091892] niks3-server[1196]: 2026/09/22 10:48:44 INFO Completed upload id=506779worker2 # [ 148.086734] nix-grpc-daemon[1700]: time=2026-09-22T10:48:45.264Z level=INFO msg="Upload complete. (224ms)"6780lb # [ 148.093509] niks3-server[1196]: 2026/09/22 10:48:44 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6781worker2 # [ 148.090261] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16782worker2 # [ 148.096482] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:45Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06783worker2 # [ 148.098548] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:45Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16784worker2 # [ 148.103827] nix-daemon[873]: reaped child process 1823, status = succeeded6785worker2 # [ 148.139398] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448006786worker2 # [ 148.147168] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:448006787worker2 # [ 148.156229] nix-grpc-daemon[1700]: time=2026-09-22T10:48:45.333Z level=INFO msg="All 2 paths already cached"6788client # copying 1 paths to 'lb:50051'...6789client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6790worker2 # [ 148.163234] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:448006791worker2 # [ 148.169957] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6792worker2 # [ 148.304411] nix-daemon[873]: accepted connection from pid 1850, user nix-grpc-daemon (trusted)6793worker2 # [ 148.319127] nix-daemon[873]: reaped child process 1853, status = succeeded6794worker2 # [ 148.465172] nix-daemon[873]: accepted connection from pid 1855, user nix-grpc-daemon (trusted)6795worker2 # [ 148.479172] nix-daemon[873]: reaped child process 1858, status = succeeded6796lb # [ 148.496798] niks3-server[1196]: 2026/09/22 10:48:45 INFO Received uploads request method=POST path=/api/pending_closures6797worker2 # [ 148.499480] nix-grpc-daemon[1700]: time=2026-09-22T10:48:45.676Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6798worker2 # [ 148.500968] nix-grpc-daemon[1700]: time=2026-09-22T10:48:45.676Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6799lb # [ 148.521536] niks3-server[1196]: 2026/09/22 10:48:45 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6800lb # [ 148.533843] niks3-server[1196]: 2026/09/22 10:48:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6801worker2 # [ 148.535323] nix-grpc-daemon[1700]: time=2026-09-22T10:48:45.712Z level=INFO msg="Uploading 1 narinfos"6802lb # [ 148.541589] niks3-server[1196]: 2026/09/22 10:48:45 INFO Signed narinfos id=51 count=16803lb # [ 148.546319] niks3-server[1196]: 2026/09/22 10:48:45 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6804lb # [ 148.558557] niks3-server[1196]: 2026/09/22 10:48:45 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6805worker2 # [ 148.559394] nix-grpc-daemon[1700]: time=2026-09-22T10:48:45.736Z level=INFO msg="Upload complete. (382ms)"6806lb # [ 148.565944] niks3-server[1196]: 2026/09/22 10:48:45 INFO Completed upload id=516807worker2 # [ 148.562129] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=0 paths=1 nar_bytes_in=7686808lb # [ 148.569518] niks3-server[1196]: 2026/09/22 10:48:45 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6809worker2 # [ 148.567753] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448006810worker2 # [ 148.578050] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6811worker2 # [ 148.583686] nix-daemon[873]: reaped child process 1848, status = succeeded6812client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6813client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6814worker2 # [ 148.586674] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)6815lb # [ 148.682728] envoy[932]: conn peer=[2001:db8:1::1]:49172 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2967 tx=854 ms=33396816client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6817client # farm-top> LOG-farm-top-during-drain6818worker2 # [ 149.215925] nix-daemon[873]: reaped child process 1862, status = succeeded6819worker2 # [ 149.361510] nix-daemon[873]: accepted connection from pid 1878, user nix-grpc-daemon (trusted)6820worker2 # [ 149.376164] nix-daemon[873]: reaped child process 1881, status = succeeded6821lb # [ 149.397265] niks3-server[1196]: 2026/09/22 10:48:46 INFO Received uploads request method=POST path=/api/pending_closures6822worker2 # [ 149.398266] nix-grpc-daemon[1700]: time=2026-09-22T10:48:46.575Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6823worker2 # [ 149.400434] nix-grpc-daemon[1700]: time=2026-09-22T10:48:46.577Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6824lb # [ 149.430301] niks3-server[1196]: 2026/09/22 10:48:46 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6825lb # [ 149.444597] niks3-server[1196]: 2026/09/22 10:48:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6826worker2 # [ 149.443268] nix-grpc-daemon[1700]: time=2026-09-22T10:48:46.620Z level=INFO msg="Uploading 1 narinfos"6827lb # [ 149.451125] niks3-server[1196]: 2026/09/22 10:48:46 INFO Signed narinfos id=52 count=16828lb # [ 149.458344] niks3-server[1196]: 2026/09/22 10:48:46 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6829lb # [ 149.465881] niks3-server[1196]: 2026/09/22 10:48:46 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6830lb # [ 149.473587] niks3-server[1196]: 2026/09/22 10:48:46 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6831lb # [ 149.478576] niks3-server[1196]: 2026/09/22 10:48:46 INFO Completed upload id=526832worker2 # [ 149.472596] nix-grpc-daemon[1700]: time=2026-09-22T10:48:46.649Z level=INFO msg="Upload complete. (278ms)"6833lb # [ 149.480914] niks3-server[1196]: 2026/09/22 10:48:46 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6834worker2 # [ 149.480168] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=0 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16835worker2 # [ 149.494496] nix-daemon[873]: reaped child process 1860, status = succeeded6836client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 4.76 seconds)6837client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6838client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.21 seconds)6839worker1: must succeed: pkill -f 'read -t [9]0 x'6840worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6841client: waiting for success: ! systemctl is-active sw6842worker1 # [ 149.932352] nix-daemon[852]: reaped child process 1637, status = succeeded6843worker1 # [ 150.072482] nix-daemon[852]: accepted connection from pid 1679, user nix-grpc-daemon (trusted)6844worker1 # [ 150.085248] nix-daemon[852]: reaped child process 1682, status = succeeded6845lb # [ 150.074753] niks3-server[1196]: 2026/09/22 10:48:46 INFO Received uploads request method=POST path=/api/pending_closures6846worker1 # [ 150.104839] nix-grpc-daemon[1581]: time=2026-09-22T10:48:47.226Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6847worker1 # [ 150.107029] nix-grpc-daemon[1581]: time=2026-09-22T10:48:47.229Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6848lb # [ 150.096096] niks3-server[1196]: 2026/09/22 10:48:46 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6849lb # [ 150.098915] niks3-server[1196]: 2026/09/22 10:48:46 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6850lb # [ 150.103237] niks3-server[1196]: 2026/09/22 10:48:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6851lb # [ 150.105469] niks3-server[1196]: 2026/09/22 10:48:46 INFO Signed narinfos id=53 count=16852worker1 # [ 150.131193] nix-grpc-daemon[1581]: time=2026-09-22T10:48:47.252Z level=INFO msg="Uploading 1 narinfos"6853lb # [ 150.109280] niks3-server[1196]: 2026/09/22 10:48:46 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6854lb # [ 150.113055] niks3-server[1196]: 2026/09/22 10:48:46 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6855lb # [ 150.116582] niks3-server[1196]: 2026/09/22 10:48:46 INFO Completed upload id=536856worker1 # [ 150.140330] nix-grpc-daemon[1581]: time=2026-09-22T10:48:47.262Z level=INFO msg="Upload complete. (217ms)"6857lb # [ 150.118195] niks3-server[1196]: 2026/09/22 10:48:46 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6858worker1 # [ 150.143335] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43954 duration_s=7 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16859worker1 # [ 150.151233] nix-daemon[852]: reaped child process 1635, status = succeeded6860client # [ 150.173838] systemd[1]: sw.service: Deactivated successfully.6861client # [ 150.178516] systemd[1]: sw.service: Consumed 121ms CPU time over 7.994s wall clock time, 41.4M memory peak, 7.3K incoming IP traffic, 8.4K outgoing IP traffic.6862worker1 # [ 150.624979] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:47Z level=info event=signal_exit6863client: (finished: waiting for success: ! systemctl is-active sw, in 1.07 seconds)6864client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6865client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.03 seconds)6866worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6867worker2 # [ 151.827286] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:49Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:501906868worker1 # [ 151.851840] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:48Z level=info event=scheduler_restarting6869worker2 # [ 151.831174] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:49Z level=info event=worker_gone addr=192.168.1.3:500516870worker1 # [ 151.868305] nix-grpc-daemon[1479]: ts=2026-09-22T10:48:48Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6871worker1 # [ 151.885299] nix-daemon[852]: reaped child process 1547, status = succeeded6872worker1 # [ 152.471475] nix-grpc-daemon[1479]: E0000 00:00:1790074129.593595 1524 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true6873worker1 # [ 152.476337] nix-grpc-daemon[1479]: E0000 00:00:1790074129.593740 1524 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6874worker1 # [ 152.658551] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6875worker1 # [ 152.663325] systemd[1]: nix-grpc-daemon.service: Consumed 646ms CPU time over 24.206s wall clock time, 32.5M memory peak, 20.9K incoming IP traffic, 27.6K outgoing IP traffic.6876worker1 # [ 152.678806] systemd[1]: Starting Nix worker-protocol over gRPC...6877worker1 # [ 152.864467] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:49Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6878worker1 # [ 152.866908] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:49Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6879worker1 # [ 152.871396] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:49Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16880worker1 # [ 152.878165] nix-grpc-daemon[1696]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6881worker1 # [ 152.881101] nix-grpc-daemon[1696]: E0000 00:00:1790074130.000222 1696 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6882worker1 # [ 152.886615] nix-grpc-daemon[1696]: E0000 00:00:1790074130.005717 1696 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6883worker1 # [ 152.890761] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:50Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6884worker1 # [ 152.892759] systemd[1]: Started Nix worker-protocol over gRPC.6885worker1 # [ 152.898102] nix-grpc-daemon[1696]: E0000 00:00:1790074130.020025 1737 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6886worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 1.95 seconds)6887lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6888lb: must succeed: curl -sf localhost:9901/clusters6889worker1 # [ 152.910685] nix-grpc-daemon[1696]: E0000 00:00:1790074130.033054 1737 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6890worker1 # [ 152.913990] nix-grpc-daemon[1696]: E0000 00:00:1790074130.035218 1737 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6891worker1 # [ 152.924767] nix-grpc-daemon[1696]: E0000 00:00:1790074130.047079 1737 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6892worker1 # [ 152.936940] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:50Z level=info event=scheduler_connected addr=lb:500516893worker2 # [ 152.909271] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:50Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:447966894worker2 # [ 152.910870] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:50Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06895lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6896lb: must succeed: curl -sf localhost:9901/clusters6897lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6898lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.14 seconds)6899lb: must succeed: curl -sf localhost:9901/clusters6900lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6901worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6902worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6903(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 11.97 seconds)6904subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6905client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16906worker1 # [ 154.370398] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:442786907worker1 # [ 154.382380] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)6908worker2 # [ 154.367951] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:448006909worker2 # [ 154.411962] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:51Z level=debug event=want client=5 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06910worker2 # [ 154.413648] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:51Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16911worker2 # [ 154.415425] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:51Z level=debug event=want client=5 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06912worker2 # [ 154.416987] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:51Z level=debug event=want client=5 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06913worker1 # [ 154.487105] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442786914worker1 # [ 154.499651] nix-grpc-daemon[1696]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 60 ms (attempt 1/5)6915worker1 # [ 154.565498] nix-grpc-daemon[1696]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 161 ms (attempt 2/5)6916worker1 # [ 154.732602] nix-grpc-daemon[1696]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 55 ms (attempt 3/5)6917worker1 # [ 154.793743] nix-grpc-daemon[1696]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 263 ms (attempt 4/5)6918worker1 # [ 155.063128] nix-grpc-daemon[1696]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6919worker1 # [ 155.067091] nix-grpc-daemon[1696]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6920worker1 # [ 155.084967] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:442786921worker1 # [ 155.096890] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:442786922worker1 # [ 155.107200] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)6923worker1 # [ 155.128387] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:52Z level=info event=niks3_push_started pid=17566924worker1 # [ 155.330620] nix-grpc-daemon[1756]: time=2026-09-22T10:48:52.452Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6925worker1 # [ 155.505881] nix-daemon[852]: accepted connection from pid 1769, user nix-grpc-daemon (trusted)6926worker1 # [ 155.518066] nix-daemon[852]: reaped child process 1772, status = succeeded6927worker1 # [ 155.689765] nix-daemon[852]: accepted connection from pid 1774, user nix-grpc-daemon (trusted)6928worker1 # [ 155.702410] nix-daemon[852]: reaped child process 1777, status = succeeded6929lb # [ 155.696078] niks3-server[1196]: 2026/09/22 10:48:52 INFO Received uploads request method=POST path=/api/pending_closures6930worker1 # [ 155.729688] nix-grpc-daemon[1756]: time=2026-09-22T10:48:52.852Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6931worker1 # [ 155.732654] nix-grpc-daemon[1756]: time=2026-09-22T10:48:52.853Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6932lb # [ 155.737975] niks3-server[1196]: 2026/09/22 10:48:52 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6933lb # [ 155.751062] niks3-server[1196]: 2026/09/22 10:48:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6934lb # [ 155.757263] niks3-server[1196]: 2026/09/22 10:48:52 INFO Signed narinfos id=54 count=16935worker1 # [ 155.785032] nix-grpc-daemon[1756]: time=2026-09-22T10:48:52.907Z level=INFO msg="Uploading 1 narinfos"6936lb # [ 155.771056] niks3-server[1196]: 2026/09/22 10:48:52 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6937lb # [ 155.775485] niks3-server[1196]: 2026/09/22 10:48:52 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6938lb # [ 155.779162] niks3-server[1196]: 2026/09/22 10:48:52 INFO Completed upload id=546939lb # [ 155.781452] niks3-server[1196]: 2026/09/22 10:48:52 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6940worker1 # [ 155.804460] nix-grpc-daemon[1756]: time=2026-09-22T10:48:52.926Z level=INFO msg="Upload complete. (469ms)"6941worker1 # [ 155.811278] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=0 paths=1 nar_bytes_in=5046942worker1 # [ 155.816280] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442786943worker1 # [ 155.824257] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)6944worker1 # [ 155.831905] nix-daemon[852]: reaped child process 1754, status = succeeded6945worker1 # [ 155.838148] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)6946worker1 # [ 156.660871] nix-daemon[852]: reaped child process 1781, status = succeeded6947worker1 # [ 156.823386] nix-daemon[852]: accepted connection from pid 1797, user nix-grpc-daemon (trusted)6948worker1 # [ 156.836888] nix-daemon[852]: reaped child process 1800, status = succeeded6949lb # [ 156.826745] niks3-server[1196]: 2026/09/22 10:48:53 INFO Received uploads request method=POST path=/api/pending_closures6950worker1 # [ 156.860824] nix-grpc-daemon[1756]: time=2026-09-22T10:48:53.982Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6951worker1 # [ 156.862585] nix-grpc-daemon[1756]: time=2026-09-22T10:48:53.984Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6952lb # [ 156.865415] niks3-server[1196]: 2026/09/22 10:48:53 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6953lb # [ 156.873145] niks3-server[1196]: 2026/09/22 10:48:53 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6954lb # [ 156.882620] niks3-server[1196]: 2026/09/22 10:48:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6955lb # [ 156.886738] niks3-server[1196]: 2026/09/22 10:48:53 INFO Signed narinfos id=55 count=16956worker1 # [ 156.911417] nix-grpc-daemon[1756]: time=2026-09-22T10:48:54.033Z level=INFO msg="Uploading 1 narinfos"6957lb # [ 156.891752] niks3-server[1196]: 2026/09/22 10:48:53 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6958lb # [ 156.896232] niks3-server[1196]: 2026/09/22 10:48:53 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6959lb # [ 156.901310] niks3-server[1196]: 2026/09/22 10:48:53 INFO Completed upload id=556960worker1 # [ 156.925435] nix-grpc-daemon[1756]: time=2026-09-22T10:48:54.047Z level=INFO msg="Upload complete. (281ms)"6961lb # [ 156.903659] niks3-server[1196]: 2026/09/22 10:48:53 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6962worker1 # [ 156.930646] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=7 outputs=16963worker2 # [ 156.902283] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:54Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16964worker1 # [ 156.942971] nix-daemon[852]: reaped child process 1779, status = succeeded6965worker2 # [ 156.912277] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:54Z level=debug event=want client=5 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06966worker1 # [ 156.974742] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442786967worker1 # [ 156.985405] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:442786968worker1 # [ 156.995518] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:442786969worker1 # [ 157.005164] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)6970worker1 # [ 157.178906] nix-daemon[852]: accepted connection from pid 1804, user nix-grpc-daemon (trusted)6971worker1 # [ 157.191644] nix-daemon[852]: reaped child process 1807, status = succeeded6972worker1 # [ 157.356438] nix-daemon[852]: accepted connection from pid 1809, user nix-grpc-daemon (trusted)6973worker1 # [ 157.369085] nix-daemon[852]: reaped child process 1812, status = succeeded6974lb # [ 157.360836] niks3-server[1196]: 2026/09/22 10:48:54 INFO Received uploads request method=POST path=/api/pending_closures6975worker1 # [ 157.393217] nix-grpc-daemon[1756]: time=2026-09-22T10:48:54.515Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6976worker1 # [ 157.394724] nix-grpc-daemon[1756]: time=2026-09-22T10:48:54.515Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6977lb # [ 157.386314] niks3-server[1196]: 2026/09/22 10:48:54 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6978lb # [ 157.393717] niks3-server[1196]: 2026/09/22 10:48:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6979lb # [ 157.398727] niks3-server[1196]: 2026/09/22 10:48:54 INFO Signed narinfos id=56 count=16980worker1 # [ 157.425082] nix-grpc-daemon[1756]: time=2026-09-22T10:48:54.547Z level=INFO msg="Uploading 1 narinfos"6981lb # [ 157.402587] niks3-server[1196]: 2026/09/22 10:48:54 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6982lb # [ 157.408526] niks3-server[1196]: 2026/09/22 10:48:54 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6983lb # [ 157.412027] niks3-server[1196]: 2026/09/22 10:48:54 INFO Completed upload id=566984worker1 # [ 157.437227] nix-grpc-daemon[1756]: time=2026-09-22T10:48:54.559Z level=INFO msg="Upload complete. (421ms)"6985lb # [ 157.413843] niks3-server[1196]: 2026/09/22 10:48:54 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6986worker1 # [ 157.439511] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=0 paths=1 nar_bytes_in=5046987worker1 # [ 157.442071] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442786988worker1 # [ 157.450443] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)6989worker1 # [ 157.452664] nix-daemon[852]: reaped child process 1802, status = succeeded6990worker1 # [ 157.457860] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)6991worker1 # [ 158.079559] nix-daemon[852]: reaped child process 1816, status = succeeded6992worker1 # [ 158.233485] nix-daemon[852]: accepted connection from pid 1832, user nix-grpc-daemon (trusted)6993worker1 # [ 158.245488] nix-daemon[852]: reaped child process 1835, status = succeeded6994lb # [ 158.239864] niks3-server[1196]: 2026/09/22 10:48:55 INFO Received uploads request method=POST path=/api/pending_closures6995worker1 # [ 158.270249] nix-grpc-daemon[1756]: time=2026-09-22T10:48:55.392Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6996worker1 # [ 158.272446] nix-grpc-daemon[1756]: time=2026-09-22T10:48:55.394Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6997lb # [ 158.272979] niks3-server[1196]: 2026/09/22 10:48:55 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6998lb # [ 158.281975] niks3-server[1196]: 2026/09/22 10:48:55 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6999lb # [ 158.293194] niks3-server[1196]: 2026/09/22 10:48:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign7000lb # [ 158.297461] niks3-server[1196]: 2026/09/22 10:48:55 INFO Signed narinfos id=57 count=17001worker1 # [ 158.323859] nix-grpc-daemon[1756]: time=2026-09-22T10:48:55.445Z level=INFO msg="Uploading 1 narinfos"7002lb # [ 158.303155] niks3-server[1196]: 2026/09/22 10:48:55 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls7003lb # [ 158.311773] niks3-server[1196]: 2026/09/22 10:48:55 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete7004lb # [ 158.318311] niks3-server[1196]: 2026/09/22 10:48:55 INFO Completed upload id=577005lb # [ 158.319686] niks3-server[1196]: 2026/09/22 10:48:55 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo7006worker1 # [ 158.342646] nix-grpc-daemon[1756]: time=2026-09-22T10:48:55.464Z level=INFO msg="Upload complete. (278ms)"7007worker2 # [ 158.318544] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:55Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=17008worker2 # [ 158.324319] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:55Z level=debug event=want client=5 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=07009worker1 # [ 158.351234] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=0 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=8 outputs=17010worker1 # [ 158.357702] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787011worker1 # [ 158.366783] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:442787012worker1 # [ 158.371407] nix-daemon[852]: reaped child process 1814, status = succeeded7013worker1 # [ 158.374781] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:442787014worker1 # [ 158.379848] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7015worker1 # [ 158.534642] nix-daemon[852]: accepted connection from pid 1839, user nix-grpc-daemon (trusted)7016worker1 # [ 158.547721] nix-daemon[852]: reaped child process 1842, status = succeeded7017lb # [ 158.682994] envoy[932]: conn peer=[2001:db8:1::1]:44822 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9611 tx=4423 ms=45337018worker1 # [ 158.710844] nix-daemon[852]: accepted connection from pid 1844, user nix-grpc-daemon (trusted)7019lb # [ 158.686753] envoy[932]: conn peer=[2001:db8:1::1]:49188 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2962 tx=1516 ms=77667020lb # [ 158.690267] envoy[932]: conn peer=[2001:db8:1::3]:51372 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1114 tx=448 ms=231977021worker1 # [ 158.722259] nix-daemon[852]: reaped child process 1847, status = succeeded7022lb # [ 158.713729] niks3-server[1196]: 2026/09/22 10:48:55 INFO Received uploads request method=POST path=/api/pending_closures7023worker1 # [ 158.745549] nix-grpc-daemon[1756]: time=2026-09-22T10:48:55.867Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7024worker1 # [ 158.747090] nix-grpc-daemon[1756]: time=2026-09-22T10:48:55.869Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"7025lb # [ 158.741587] niks3-server[1196]: 2026/09/22 10:48:55 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst7026lb # [ 158.752599] niks3-server[1196]: 2026/09/22 10:48:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign7027lb # [ 158.759314] niks3-server[1196]: 2026/09/22 10:48:55 INFO Signed narinfos id=58 count=17028worker1 # [ 158.786088] nix-grpc-daemon[1756]: time=2026-09-22T10:48:55.908Z level=INFO msg="Uploading 1 narinfos"7029lb # [ 158.763112] niks3-server[1196]: 2026/09/22 10:48:55 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls7030lb # [ 158.773612] niks3-server[1196]: 2026/09/22 10:48:55 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete7031worker1 # [ 158.802550] nix-grpc-daemon[1756]: time=2026-09-22T10:48:55.924Z level=INFO msg="Upload complete. (417ms)"7032lb # [ 158.779138] niks3-server[1196]: 2026/09/22 10:48:55 INFO Completed upload id=587033worker1 # [ 158.804658] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=0 paths=1 nar_bytes_in=5047034lb # [ 158.783598] niks3-server[1196]: 2026/09/22 10:48:55 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo7035worker1 # [ 158.807568] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787036worker1 # [ 158.814064] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7037worker1 # [ 158.820130] nix-daemon[852]: reaped child process 1837, status = succeeded7038worker1 # [ 158.823073] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7039worker1 # [ 159.850800] nix-daemon[852]: reaped child process 1851, status = succeeded7040worker1 # [ 160.007821] nix-daemon[852]: accepted connection from pid 1867, user nix-grpc-daemon (trusted)7041worker1 # [ 160.020955] nix-daemon[852]: reaped child process 1870, status = succeeded7042lb # [ 160.011431] niks3-server[1196]: 2026/09/22 10:48:56 INFO Received uploads request method=POST path=/api/pending_closures7043worker1 # [ 160.040047] nix-grpc-daemon[1756]: time=2026-09-22T10:48:57.162Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7044worker1 # [ 160.043280] nix-grpc-daemon[1756]: time=2026-09-22T10:48:57.163Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"7045lb # [ 160.031211] niks3-server[1196]: 2026/09/22 10:48:56 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst7046lb # [ 160.035328] niks3-server[1196]: 2026/09/22 10:48:56 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv7047lb # [ 160.039237] niks3-server[1196]: 2026/09/22 10:48:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign7048lb # [ 160.041969] niks3-server[1196]: 2026/09/22 10:48:56 INFO Signed narinfos id=59 count=17049worker1 # [ 160.065746] nix-grpc-daemon[1756]: time=2026-09-22T10:48:57.187Z level=INFO msg="Uploading 1 narinfos"7050lb # [ 160.044762] niks3-server[1196]: 2026/09/22 10:48:56 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls7051lb # [ 160.049975] niks3-server[1196]: 2026/09/22 10:48:56 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete7052lb # [ 160.053062] niks3-server[1196]: 2026/09/22 10:48:56 INFO Completed upload id=597053worker1 # [ 160.078132] nix-grpc-daemon[1756]: time=2026-09-22T10:48:57.200Z level=INFO msg="Upload complete. (247ms)"7054lb # [ 160.054779] niks3-server[1196]: 2026/09/22 10:48:56 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo7055worker1 # [ 160.080873] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=9 outputs=17056worker1 # [ 160.085162] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787057worker2 # [ 160.052607] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:57Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=17058worker1 # [ 160.092166] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:442787059worker1 # [ 160.094121] nix-daemon[852]: reaped child process 1849, status = succeeded7060worker2 # [ 160.062177] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:57Z level=debug event=want client=5 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=07061worker1 # [ 160.099228] nix-grpc-daemon[1756]: time=2026-09-22T10:48:57.221Z level=INFO msg="All 1 paths already cached"7062worker1 # [ 160.102926] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:442787063worker1 # [ 160.107948] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7064worker1 # [ 160.261189] nix-daemon[852]: accepted connection from pid 1874, user nix-grpc-daemon (trusted)7065worker1 # [ 160.274632] nix-daemon[852]: reaped child process 1877, status = succeeded7066worker1 # [ 160.441375] nix-daemon[852]: accepted connection from pid 1879, user nix-grpc-daemon (trusted)7067worker1 # [ 160.454242] nix-daemon[852]: reaped child process 1882, status = succeeded7068lb # [ 160.446386] niks3-server[1196]: 2026/09/22 10:48:57 INFO Received uploads request method=POST path=/api/pending_closures7069worker1 # [ 160.474839] nix-grpc-daemon[1756]: time=2026-09-22T10:48:57.597Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7070worker1 # [ 160.476349] nix-grpc-daemon[1756]: time=2026-09-22T10:48:57.598Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"7071lb # [ 160.463234] niks3-server[1196]: 2026/09/22 10:48:57 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst7072lb # [ 160.473286] niks3-server[1196]: 2026/09/22 10:48:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign7073lb # [ 160.480606] niks3-server[1196]: 2026/09/22 10:48:57 INFO Signed narinfos id=60 count=17074worker1 # [ 160.507826] nix-grpc-daemon[1756]: time=2026-09-22T10:48:57.630Z level=INFO msg="Uploading 1 narinfos"7075lb # [ 160.484827] niks3-server[1196]: 2026/09/22 10:48:57 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls7076lb # [ 160.497788] niks3-server[1196]: 2026/09/22 10:48:57 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete7077worker1 # [ 160.528353] nix-grpc-daemon[1756]: time=2026-09-22T10:48:57.650Z level=INFO msg="Upload complete. (415ms)"7078lb # [ 160.504538] niks3-server[1196]: 2026/09/22 10:48:57 INFO Completed upload id=607079worker1 # [ 160.530516] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=0 paths=1 nar_bytes_in=6247080lb # [ 160.508264] niks3-server[1196]: 2026/09/22 10:48:57 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo7081worker1 # [ 160.534565] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787082worker1 # [ 160.541472] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7083worker1 # [ 160.544338] nix-daemon[852]: reaped child process 1872, status = succeeded7084worker1 # [ 160.549266] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7085worker1 # [ 161.764868] nix-daemon[852]: reaped child process 1886, status = succeeded7086worker1 # [ 161.902102] nix-daemon[852]: accepted connection from pid 1902, user nix-grpc-daemon (trusted)7087worker1 # [ 161.914329] nix-daemon[852]: reaped child process 1905, status = succeeded7088lb # [ 161.904975] niks3-server[1196]: 2026/09/22 10:48:58 INFO Received uploads request method=POST path=/api/pending_closures7089worker1 # [ 161.938885] nix-grpc-daemon[1756]: time=2026-09-22T10:48:59.060Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7090worker1 # [ 161.944178] nix-grpc-daemon[1756]: time=2026-09-22T10:48:59.062Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"7091lb # [ 161.940850] niks3-server[1196]: 2026/09/22 10:48:58 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv7092lb # [ 161.950392] niks3-server[1196]: 2026/09/22 10:48:58 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst7093lb # [ 161.959155] niks3-server[1196]: 2026/09/22 10:48:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign7094worker1 # [ 161.988882] nix-grpc-daemon[1756]: time=2026-09-22T10:48:59.111Z level=INFO msg="Uploading 1 narinfos"7095lb # [ 161.965101] niks3-server[1196]: 2026/09/22 10:48:58 INFO Signed narinfos id=61 count=17096lb # [ 161.970210] niks3-server[1196]: 2026/09/22 10:48:58 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls7097lb # [ 161.976398] niks3-server[1196]: 2026/09/22 10:48:58 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete7098lb # [ 161.980466] niks3-server[1196]: 2026/09/22 10:48:58 INFO Completed upload id=617099lb # [ 161.982337] niks3-server[1196]: 2026/09/22 10:48:58 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo7100worker1 # [ 162.005720] nix-grpc-daemon[1756]: time=2026-09-22T10:48:59.128Z level=INFO msg="Upload complete. (249ms)"7101worker1 # [ 162.010797] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=10 outputs=17102worker2 # [ 161.982302] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:59Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=17103worker1 # [ 162.023743] nix-daemon[852]: reaped child process 1884, status = succeeded7104worker2 # [ 161.991916] nix-grpc-daemon[1620]: ts=2026-09-22T10:48:59Z level=debug event=want client=5 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=07105worker1 # [ 162.056181] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787106worker1 # [ 162.067264] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:442787107worker1 # [ 162.080433] nix-grpc-daemon[1756]: time=2026-09-22T10:48:59.202Z level=INFO msg="All 1 paths already cached"7108worker1 # [ 162.088492] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:442787109worker1 # [ 162.097170] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7110worker1 # [ 162.241283] nix-daemon[852]: accepted connection from pid 1909, user nix-grpc-daemon (trusted)7111worker1 # [ 162.254306] nix-daemon[852]: reaped child process 1912, status = succeeded7112worker1 # [ 162.374566] nix-daemon[852]: accepted connection from pid 1914, user nix-grpc-daemon (trusted)7113worker1 # [ 162.387703] nix-daemon[852]: reaped child process 1917, status = succeeded7114lb # [ 162.377081] niks3-server[1196]: 2026/09/22 10:48:59 INFO Received uploads request method=POST path=/api/pending_closures7115worker1 # [ 162.409751] nix-grpc-daemon[1756]: time=2026-09-22T10:48:59.532Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7116worker1 # [ 162.411370] nix-grpc-daemon[1756]: time=2026-09-22T10:48:59.533Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"7117lb # [ 162.405400] niks3-server[1196]: 2026/09/22 10:48:59 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst7118lb # [ 162.416644] niks3-server[1196]: 2026/09/22 10:48:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign7119lb # [ 162.420616] niks3-server[1196]: 2026/09/22 10:48:59 INFO Signed narinfos id=62 count=17120worker1 # [ 162.446837] nix-grpc-daemon[1756]: time=2026-09-22T10:48:59.569Z level=INFO msg="Uploading 1 narinfos"7121lb # [ 162.428739] niks3-server[1196]: 2026/09/22 10:48:59 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls7122lb # [ 162.438885] niks3-server[1196]: 2026/09/22 10:48:59 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete7123lb # [ 162.442660] niks3-server[1196]: 2026/09/22 10:48:59 INFO Completed upload id=627124worker1 # [ 162.466594] nix-grpc-daemon[1756]: time=2026-09-22T10:48:59.588Z level=INFO msg="Upload complete. (358ms)"7125worker1 # [ 162.468739] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=0 paths=1 nar_bytes_in=6247126lb # [ 162.445712] niks3-server[1196]: 2026/09/22 10:48:59 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo7127worker1 # [ 162.472043] nix-grpc-daemon[1696]: ts=2026-09-22T10:48:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787128worker1 # [ 162.478595] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7129worker1 # [ 162.482192] nix-daemon[852]: reaped child process 1907, status = succeeded7130worker1 # [ 162.487219] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7131worker1 # [ 163.303430] nix-daemon[852]: reaped child process 1921, status = succeeded7132worker1 # [ 163.429102] nix-daemon[852]: accepted connection from pid 1937, user nix-grpc-daemon (trusted)7133worker1 # [ 163.441651] nix-daemon[852]: reaped child process 1940, status = succeeded7134lb # [ 163.427804] niks3-server[1196]: 2026/09/22 10:49:00 INFO Received uploads request method=POST path=/api/pending_closures7135worker1 # [ 163.460073] nix-grpc-daemon[1756]: time=2026-09-22T10:49:00.582Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7136worker1 # [ 163.462344] nix-grpc-daemon[1756]: time=2026-09-22T10:49:00.583Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"7137lb # [ 163.464150] niks3-server[1196]: 2026/09/22 10:49:00 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst7138lb # [ 163.478619] niks3-server[1196]: 2026/09/22 10:49:00 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv7139lb # [ 163.486350] niks3-server[1196]: 2026/09/22 10:49:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign7140lb # [ 163.491635] niks3-server[1196]: 2026/09/22 10:49:00 INFO Signed narinfos id=63 count=17141worker1 # [ 163.518580] nix-grpc-daemon[1756]: time=2026-09-22T10:49:00.640Z level=INFO msg="Uploading 1 narinfos"7142lb # [ 163.498233] niks3-server[1196]: 2026/09/22 10:49:00 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls7143lb # [ 163.507384] niks3-server[1196]: 2026/09/22 10:49:00 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete7144lb # [ 163.512303] niks3-server[1196]: 2026/09/22 10:49:00 INFO Completed upload id=637145worker1 # [ 163.536314] nix-grpc-daemon[1756]: time=2026-09-22T10:49:00.658Z level=INFO msg="Upload complete. (242ms)"7146lb # [ 163.517071] niks3-server[1196]: 2026/09/22 10:49:00 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo7147worker1 # [ 163.541700] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=11 outputs=17148worker2 # [ 163.514487] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:00Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=17149worker1 # [ 163.549264] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787150worker2 # [ 163.521813] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:00Z level=debug event=want client=5 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=07151worker1 # [ 163.559358] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:442787152worker1 # [ 163.562119] nix-daemon[852]: reaped child process 1919, status = succeeded7153worker1 # [ 163.565746] nix-grpc-daemon[1756]: time=2026-09-22T10:49:00.688Z level=INFO msg="All 1 paths already cached"7154worker1 # [ 163.569341] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:442787155worker1 # [ 163.573539] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7156worker1 # [ 163.703756] nix-daemon[852]: accepted connection from pid 1944, user nix-grpc-daemon (trusted)7157worker1 # [ 163.718264] nix-daemon[852]: reaped child process 1947, status = succeeded7158worker1 # [ 163.850038] nix-daemon[852]: accepted connection from pid 1949, user nix-grpc-daemon (trusted)7159worker1 # [ 163.862917] nix-daemon[852]: reaped child process 1952, status = succeeded7160lb # [ 163.849739] niks3-server[1196]: 2026/09/22 10:49:00 INFO Received uploads request method=POST path=/api/pending_closures7161worker1 # [ 163.882352] nix-grpc-daemon[1756]: time=2026-09-22T10:49:01.004Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7162worker1 # [ 163.883775] nix-grpc-daemon[1756]: time=2026-09-22T10:49:01.004Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"7163lb # [ 163.874146] niks3-server[1196]: 2026/09/22 10:49:00 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst7164lb # [ 163.886566] niks3-server[1196]: 2026/09/22 10:49:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7165worker1 # [ 163.917548] nix-grpc-daemon[1756]: time=2026-09-22T10:49:01.039Z level=INFO msg="Uploading 1 narinfos"7166lb # [ 163.893535] niks3-server[1196]: 2026/09/22 10:49:00 INFO Signed narinfos id=64 count=17167lb # [ 163.897647] niks3-server[1196]: 2026/09/22 10:49:00 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls7168lb # [ 163.910097] niks3-server[1196]: 2026/09/22 10:49:00 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7169worker1 # [ 163.940445] nix-grpc-daemon[1756]: time=2026-09-22T10:49:01.062Z level=INFO msg="Upload complete. (361ms)"7170lb # [ 163.916798] niks3-server[1196]: 2026/09/22 10:49:00 INFO Completed upload id=647171worker1 # [ 163.942618] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=0 paths=1 nar_bytes_in=6247172lb # [ 163.920075] niks3-server[1196]: 2026/09/22 10:49:00 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo7173worker1 # [ 163.948435] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787174worker1 # [ 163.956060] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7175worker1 # [ 163.957370] nix-daemon[852]: reaped child process 1942, status = succeeded7176worker1 # [ 163.962711] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7177worker1 # [ 164.778422] nix-daemon[852]: reaped child process 1956, status = succeeded7178worker1 # [ 164.919603] nix-daemon[852]: accepted connection from pid 1972, user nix-grpc-daemon (trusted)7179worker1 # [ 164.932653] nix-daemon[852]: reaped child process 1975, status = succeeded7180lb # [ 164.925457] niks3-server[1196]: 2026/09/22 10:49:01 INFO Received uploads request method=POST path=/api/pending_closures7181worker1 # [ 164.957465] nix-grpc-daemon[1756]: time=2026-09-22T10:49:02.079Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7182worker1 # [ 164.960042] nix-grpc-daemon[1756]: time=2026-09-22T10:49:02.081Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"7183lb # [ 164.953361] niks3-server[1196]: 2026/09/22 10:49:01 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv7184lb # [ 164.971412] niks3-server[1196]: 2026/09/22 10:49:01 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst7185lb # [ 164.980422] niks3-server[1196]: 2026/09/22 10:49:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7186lb # [ 164.986567] niks3-server[1196]: 2026/09/22 10:49:01 INFO Signed narinfos id=65 count=17187worker1 # [ 165.010847] nix-grpc-daemon[1756]: time=2026-09-22T10:49:02.132Z level=INFO msg="Uploading 1 narinfos"7188lb # [ 164.989921] niks3-server[1196]: 2026/09/22 10:49:01 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls7189lb # [ 164.999307] niks3-server[1196]: 2026/09/22 10:49:01 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7190lb # [ 165.004412] niks3-server[1196]: 2026/09/22 10:49:01 INFO Completed upload id=657191lb # [ 165.008076] niks3-server[1196]: 2026/09/22 10:49:01 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo7192worker1 # [ 165.030788] nix-grpc-daemon[1756]: time=2026-09-22T10:49:02.152Z level=INFO msg="Upload complete. (261ms)"7193worker2 # [ 165.007690] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:02Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=17194worker1 # [ 165.036734] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=12 outputs=17195worker1 # [ 165.051326] nix-daemon[852]: reaped child process 1954, status = succeeded7196worker1 # [ 165.080773] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787197worker2 # [ 165.053807] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:02Z level=debug event=want client=5 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=07198worker1 # [ 165.093070] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:442787199worker1 # [ 165.105817] nix-grpc-daemon[1756]: time=2026-09-22T10:49:02.228Z level=INFO msg="All 2 paths already cached"7200worker1 # [ 165.113870] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:442787201worker1 # [ 165.124181] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7202worker1 # [ 165.286739] nix-daemon[852]: accepted connection from pid 1979, user nix-grpc-daemon (trusted)7203worker1 # [ 165.300149] nix-daemon[852]: reaped child process 1982, status = succeeded7204worker1 # [ 165.446087] nix-daemon[852]: accepted connection from pid 1984, user nix-grpc-daemon (trusted)7205worker1 # [ 165.459172] nix-daemon[852]: reaped child process 1987, status = succeeded7206lb # [ 165.448612] niks3-server[1196]: 2026/09/22 10:49:02 INFO Received uploads request method=POST path=/api/pending_closures7207worker1 # [ 165.477203] nix-grpc-daemon[1756]: time=2026-09-22T10:49:02.599Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7208worker1 # [ 165.478714] nix-grpc-daemon[1756]: time=2026-09-22T10:49:02.601Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7209lb # [ 165.466065] niks3-server[1196]: 2026/09/22 10:49:02 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7210lb # [ 165.470105] niks3-server[1196]: 2026/09/22 10:49:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7211lb # [ 165.472481] niks3-server[1196]: 2026/09/22 10:49:02 INFO Signed narinfos id=66 count=17212worker1 # [ 165.497776] nix-grpc-daemon[1756]: time=2026-09-22T10:49:02.620Z level=INFO msg="Uploading 1 narinfos"7213lb # [ 165.475270] niks3-server[1196]: 2026/09/22 10:49:02 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7214lb # [ 165.481876] niks3-server[1196]: 2026/09/22 10:49:02 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7215lb # [ 165.485529] niks3-server[1196]: 2026/09/22 10:49:02 INFO Completed upload id=667216worker1 # [ 165.509301] nix-grpc-daemon[1756]: time=2026-09-22T10:49:02.631Z level=INFO msg="Upload complete. (374ms)"7217lb # [ 165.487197] niks3-server[1196]: 2026/09/22 10:49:02 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7218worker1 # [ 165.511490] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=0 paths=1 nar_bytes_in=7527219worker1 # [ 165.515970] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787220worker1 # [ 165.521978] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7221worker1 # [ 165.524346] nix-daemon[852]: reaped child process 1977, status = succeeded7222worker1 # [ 165.529770] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7223worker1 # [ 166.551571] nix-daemon[852]: reaped child process 1991, status = succeeded7224worker1 # [ 166.686251] nix-daemon[852]: accepted connection from pid 2007, user nix-grpc-daemon (trusted)7225worker1 # [ 166.699469] nix-daemon[852]: reaped child process 2010, status = succeeded7226lb # [ 166.691664] niks3-server[1196]: 2026/09/22 10:49:03 INFO Received uploads request method=POST path=/api/pending_closures7227worker1 # [ 166.724600] nix-grpc-daemon[1756]: time=2026-09-22T10:49:03.846Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7228worker1 # [ 166.726345] nix-grpc-daemon[1756]: time=2026-09-22T10:49:03.848Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7229lb # [ 166.732412] niks3-server[1196]: 2026/09/22 10:49:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7230worker1 # [ 166.761133] nix-grpc-daemon[1756]: time=2026-09-22T10:49:03.883Z level=INFO msg="Uploading 1 narinfos"7231lb # [ 166.738096] niks3-server[1196]: 2026/09/22 10:49:03 INFO Signed narinfos id=67 count=17232lb # [ 166.747584] niks3-server[1196]: 2026/09/22 10:49:03 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7233lb # [ 166.755139] niks3-server[1196]: 2026/09/22 10:49:03 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7234lb # [ 166.761879] niks3-server[1196]: 2026/09/22 10:49:03 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7235lb # [ 166.770625] niks3-server[1196]: 2026/09/22 10:49:03 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7236lb # [ 166.774842] niks3-server[1196]: 2026/09/22 10:49:03 INFO Completed upload id=677237lb # [ 166.776644] niks3-server[1196]: 2026/09/22 10:49:03 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7238worker1 # [ 166.800278] nix-grpc-daemon[1756]: time=2026-09-22T10:49:03.922Z level=INFO msg="Upload complete. (263ms)"7239worker2 # [ 166.776554] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:03Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17240worker1 # [ 166.805712] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=13 outputs=17241worker1 # [ 166.818444] nix-daemon[852]: reaped child process 1989, status = succeeded7242worker1 # [ 166.850450] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787243worker1 # [ 166.862668] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:442787244worker1 # [ 166.875165] nix-grpc-daemon[1756]: time=2026-09-22T10:49:03.997Z level=INFO msg="All 2 paths already cached"7245worker1 # [ 166.882869] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:442787246worker1 # [ 166.893168] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7247worker1 # [ 167.061204] nix-daemon[852]: accepted connection from pid 2014, user nix-grpc-daemon (trusted)7248worker1 # [ 167.075270] nix-daemon[852]: reaped child process 2017, status = succeeded7249worker1 # [ 167.224209] nix-daemon[852]: accepted connection from pid 2019, user nix-grpc-daemon (trusted)7250worker1 # [ 167.236897] nix-daemon[852]: reaped child process 2022, status = succeeded7251lb # [ 167.228795] niks3-server[1196]: 2026/09/22 10:49:04 INFO Received uploads request method=POST path=/api/pending_closures7252worker1 # [ 167.260589] nix-grpc-daemon[1756]: time=2026-09-22T10:49:04.382Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7253worker1 # [ 167.262158] nix-grpc-daemon[1756]: time=2026-09-22T10:49:04.384Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7254lb # [ 167.256110] niks3-server[1196]: 2026/09/22 10:49:04 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7255lb # [ 167.267512] niks3-server[1196]: 2026/09/22 10:49:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7256lb # [ 167.273891] niks3-server[1196]: 2026/09/22 10:49:04 INFO Signed narinfos id=68 count=17257worker1 # [ 167.300846] nix-grpc-daemon[1756]: time=2026-09-22T10:49:04.423Z level=INFO msg="Uploading 1 narinfos"7258lb # [ 167.277470] niks3-server[1196]: 2026/09/22 10:49:04 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7259lb # [ 167.289983] niks3-server[1196]: 2026/09/22 10:49:04 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7260worker1 # [ 167.319952] nix-grpc-daemon[1756]: time=2026-09-22T10:49:04.442Z level=INFO msg="Upload complete. (415ms)"7261lb # [ 167.296267] niks3-server[1196]: 2026/09/22 10:49:04 INFO Completed upload id=687262worker1 # [ 167.322642] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=0 paths=1 nar_bytes_in=7527263lb # [ 167.299900] niks3-server[1196]: 2026/09/22 10:49:04 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7264worker1 # [ 167.328128] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787265worker1 # [ 167.337284] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7266worker1 # [ 167.341726] nix-daemon[852]: reaped child process 2012, status = succeeded7267worker1 # [ 167.346933] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7268worker1 # [ 168.174832] nix-daemon[852]: reaped child process 2026, status = succeeded7269worker1 # [ 168.325097] nix-daemon[852]: accepted connection from pid 2042, user nix-grpc-daemon (trusted)7270worker1 # [ 168.336788] nix-daemon[852]: reaped child process 2045, status = succeeded7271lb # [ 168.330409] niks3-server[1196]: 2026/09/22 10:49:05 INFO Received uploads request method=POST path=/api/pending_closures7272worker1 # [ 168.363401] nix-grpc-daemon[1756]: time=2026-09-22T10:49:05.485Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7273worker1 # [ 168.365647] nix-grpc-daemon[1756]: time=2026-09-22T10:49:05.487Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7274lb # [ 168.366136] niks3-server[1196]: 2026/09/22 10:49:05 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7275lb # [ 168.373347] niks3-server[1196]: 2026/09/22 10:49:05 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7276lb # [ 168.386332] niks3-server[1196]: 2026/09/22 10:49:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7277lb # [ 168.392053] niks3-server[1196]: 2026/09/22 10:49:05 INFO Signed narinfos id=69 count=17278worker1 # [ 168.418922] nix-grpc-daemon[1756]: time=2026-09-22T10:49:05.541Z level=INFO msg="Uploading 1 narinfos"7279lb # [ 168.397411] niks3-server[1196]: 2026/09/22 10:49:05 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7280lb # [ 168.406831] niks3-server[1196]: 2026/09/22 10:49:05 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7281lb # [ 168.412201] niks3-server[1196]: 2026/09/22 10:49:05 INFO Completed upload id=697282lb # [ 168.415520] niks3-server[1196]: 2026/09/22 10:49:05 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7283worker1 # [ 168.438170] nix-grpc-daemon[1756]: time=2026-09-22T10:49:05.560Z level=INFO msg="Upload complete. (284ms)"7284worker2 # [ 168.415200] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:05Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17285worker1 # [ 168.446653] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=14 outputs=17286worker1 # [ 168.459846] nix-daemon[852]: reaped child process 2024, status = succeeded7287worker1 # [ 168.490255] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787288worker1 # [ 168.503686] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:442787289worker1 # [ 168.519365] nix-grpc-daemon[1756]: time=2026-09-22T10:49:05.641Z level=INFO msg="All 2 paths already cached"7290worker1 # [ 168.527584] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:442787291worker1 # [ 168.535861] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7292worker1 # [ 168.706651] nix-daemon[852]: accepted connection from pid 2049, user nix-grpc-daemon (trusted)7293worker1 # [ 168.719913] nix-daemon[852]: reaped child process 2052, status = succeeded7294worker1 # [ 168.874258] nix-daemon[852]: accepted connection from pid 2054, user nix-grpc-daemon (trusted)7295worker1 # [ 168.886877] nix-daemon[852]: reaped child process 2057, status = succeeded7296lb # [ 168.878857] niks3-server[1196]: 2026/09/22 10:49:05 INFO Received uploads request method=POST path=/api/pending_closures7297worker1 # [ 168.911108] nix-grpc-daemon[1756]: time=2026-09-22T10:49:06.033Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7298worker1 # [ 168.912551] nix-grpc-daemon[1756]: time=2026-09-22T10:49:06.033Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7299lb # [ 168.902145] niks3-server[1196]: 2026/09/22 10:49:05 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7300lb # [ 168.914985] niks3-server[1196]: 2026/09/22 10:49:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7301lb # [ 168.920327] niks3-server[1196]: 2026/09/22 10:49:05 INFO Signed narinfos id=70 count=17302worker1 # [ 168.947343] nix-grpc-daemon[1756]: time=2026-09-22T10:49:06.069Z level=INFO msg="Uploading 1 narinfos"7303lb # [ 168.927391] niks3-server[1196]: 2026/09/22 10:49:05 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7304lb # [ 168.939093] niks3-server[1196]: 2026/09/22 10:49:05 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7305worker1 # [ 168.969250] nix-grpc-daemon[1756]: time=2026-09-22T10:49:06.091Z level=INFO msg="Upload complete. (420ms)"7306lb # [ 168.945548] niks3-server[1196]: 2026/09/22 10:49:05 INFO Completed upload id=707307lb # [ 168.948351] niks3-server[1196]: 2026/09/22 10:49:05 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7308worker1 # [ 168.972639] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=0 paths=1 nar_bytes_in=7527309worker1 # [ 168.978839] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:442787310worker1 # [ 168.988278] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7311worker1 # [ 168.991485] nix-daemon[852]: reaped child process 2047, status = succeeded7312worker1 # [ 168.997508] nix-daemon[852]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)7313worker1 # [ 170.213350] nix-daemon[852]: reaped child process 2061, status = succeeded7314worker1 # [ 170.351458] nix-daemon[852]: accepted connection from pid 2077, user nix-grpc-daemon (trusted)7315worker1 # [ 170.365568] nix-daemon[852]: reaped child process 2080, status = succeeded7316lb # [ 170.357585] niks3-server[1196]: 2026/09/22 10:49:07 INFO Received uploads request method=POST path=/api/pending_closures7317worker1 # [ 170.389650] nix-grpc-daemon[1756]: time=2026-09-22T10:49:07.511Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7318worker1 # [ 170.391464] nix-grpc-daemon[1756]: time=2026-09-22T10:49:07.513Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7319lb # [ 170.399372] niks3-server[1196]: 2026/09/22 10:49:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7320lb # [ 170.407364] niks3-server[1196]: 2026/09/22 10:49:07 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7321worker1 # [ 170.439726] nix-grpc-daemon[1756]: time=2026-09-22T10:49:07.561Z level=INFO msg="Uploading 1 narinfos"7322lb # [ 170.415273] niks3-server[1196]: 2026/09/22 10:49:07 INFO Signed narinfos id=71 count=17323lb # [ 170.420269] niks3-server[1196]: 2026/09/22 10:49:07 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7324lb # [ 170.424827] niks3-server[1196]: 2026/09/22 10:49:07 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7325lb # [ 170.436498] niks3-server[1196]: 2026/09/22 10:49:07 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7326lb # [ 170.442597] niks3-server[1196]: 2026/09/22 10:49:07 INFO Completed upload id=717327lb # [ 170.444561] niks3-server[1196]: 2026/09/22 10:49:07 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7328worker1 # [ 170.468481] nix-grpc-daemon[1756]: time=2026-09-22T10:49:07.590Z level=INFO msg="Upload complete. (264ms)"7329worker1 # [ 170.476383] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44278 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=15 outputs=17330worker1 # [ 170.489616] nix-daemon[852]: reaped child process 2059, status = succeeded7331client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 16.40 seconds)7332worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7333worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)7334client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'7335client # [ 170.615317] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7336client # Running as unit: gpu.service; invocation ID: 8a317e263d3b4f5d8b7994e260f60c347337client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.04 seconds)7338lb: must succeed: curl -sf localhost:9901/clusters7339lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)7340lb: must succeed: curl -sf localhost:9901/clusters7341lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7342worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7343worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7344client # [ 170.840967] nix[1495]: don't know how to build these paths:7345client # [ 170.841894] nix[1495]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7346worker2 # [ 170.822511] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:448007347worker2 # [ 170.828074] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:448007348client # [ 170.867726] nix[1495]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker7349worker2 # [ 170.849602] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:08Z level=debug event=want client=6 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07350lb: must succeed: curl -sf localhost:9901/clusters7351lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7352lb: must succeed: curl -sf localhost:9901/clusters7353lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7354worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7355worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7356client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7357lb # [ 178.686245] envoy[932]: conn peer=[2001:db8:1::1]:52754 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27098 tx=12410 ms=161277358client # [ 200.870360] nix[1495]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}7359client # [ 200.876787] nix[1495]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7360client # [ 200.917434] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7361client # [ 200.920892] systemd[1]: gpu.service: Failed with result 'exit-code'.7362client # [ 200.925764] systemd[1]: gpu.service: Consumed 143ms CPU time over 30.309s wall clock time, 37.6M memory peak, 5.1K incoming IP traffic, 5.1K outgoing IP traffic.7363client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.74 seconds)7364client: must succeed: journalctl -u gpu -o cat7365client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)7366lb: must succeed: curl -sf localhost:9901/clusters7367lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7368lb: must succeed: curl -sf localhost:9901/clusters7369lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7370worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7371worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7372(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 47.83 seconds)7373subtest: low disk drains a worker and builds go to the other7374worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7375worker1 # [ 202.904355] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:40Z level=info event=unhealthy reason=min_free7376worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.98 seconds)7377worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7378worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.05 seconds)7379client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&27380client # don't know how to build these paths:7381client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7382worker2 # [ 203.175144] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:448007383worker2 # [ 203.180551] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:448007384worker2 # [ 203.197305] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:40Z level=debug event=want client=7 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07385worker2 # [ 203.201575] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:40Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17386worker2 # [ 203.244710] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448007387worker2 # [ 203.255487] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:448007388client # copying 1 paths to 'lb:50051'...7389client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7390worker2 # [ 203.266712] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:448007391worker2 # [ 203.277712] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)7392worker2 # [ 203.441985] nix-daemon[873]: accepted connection from pid 1915, user nix-grpc-daemon (trusted)7393worker2 # [ 203.453721] nix-daemon[873]: reaped child process 1918, status = succeeded7394worker2 # [ 203.607783] nix-daemon[873]: accepted connection from pid 1920, user nix-grpc-daemon (trusted)7395worker2 # [ 203.619300] nix-daemon[873]: reaped child process 1923, status = succeeded7396lb # [ 203.640359] niks3-server[1196]: 2026/09/22 10:49:40 INFO Received uploads request method=POST path=/api/pending_closures7397worker2 # [ 203.639139] nix-grpc-daemon[1700]: time=2026-09-22T10:49:40.816Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7398worker2 # [ 203.640491] nix-grpc-daemon[1700]: time=2026-09-22T10:49:40.816Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7399lb # [ 203.656061] niks3-server[1196]: 2026/09/22 10:49:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7400worker2 # [ 203.652371] nix-grpc-daemon[1700]: time=2026-09-22T10:49:40.829Z level=INFO msg="Uploading 1 narinfos"7401lb # [ 203.659212] niks3-server[1196]: 2026/09/22 10:49:40 INFO Signed narinfos id=72 count=17402lb # [ 203.661563] niks3-server[1196]: 2026/09/22 10:49:40 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7403lb # [ 203.665302] niks3-server[1196]: 2026/09/22 10:49:40 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7404lb # [ 203.669714] niks3-server[1196]: 2026/09/22 10:49:40 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7405lb # [ 203.673683] niks3-server[1196]: 2026/09/22 10:49:40 INFO Completed upload id=727406worker2 # [ 203.667371] nix-grpc-daemon[1700]: time=2026-09-22T10:49:40.844Z level=INFO msg="Upload complete. (378ms)"7407lb # [ 203.675471] niks3-server[1196]: 2026/09/22 10:49:40 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7408worker2 # [ 203.669538] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=0 paths=1 nar_bytes_in=5127409worker2 # [ 203.673838] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448007410worker2 # [ 203.679295] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)7411worker2 # [ 203.681449] nix-daemon[873]: reaped child process 1913, status = succeeded7412worker2 # [ 203.686570] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)7413client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7414client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7415client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7416client # farm-a> LOG-farm-a-drain7417worker2 # [ 204.910863] nix-daemon[873]: reaped child process 1927, status = succeeded7418worker2 # [ 205.070652] nix-daemon[873]: accepted connection from pid 1943, user nix-grpc-daemon (trusted)7419worker2 # [ 205.081087] nix-daemon[873]: reaped child process 1946, status = succeeded7420lb # [ 205.103816] niks3-server[1196]: 2026/09/22 10:49:41 INFO Received uploads request method=POST path=/api/pending_closures7421worker2 # [ 205.107610] nix-grpc-daemon[1700]: time=2026-09-22T10:49:42.284Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7422worker2 # [ 205.109813] nix-grpc-daemon[1700]: time=2026-09-22T10:49:42.286Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7423lb # [ 205.139977] niks3-server[1196]: 2026/09/22 10:49:41 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7424lb # [ 205.150695] niks3-server[1196]: 2026/09/22 10:49:41 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7425lb # [ 205.160721] niks3-server[1196]: 2026/09/22 10:49:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7426lb # [ 205.165474] niks3-server[1196]: 2026/09/22 10:49:42 INFO Signed narinfos id=73 count=17427worker2 # [ 205.159504] nix-grpc-daemon[1700]: time=2026-09-22T10:49:42.336Z level=INFO msg="Uploading 1 narinfos"7428lb # [ 205.170956] niks3-server[1196]: 2026/09/22 10:49:42 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7429lb # [ 205.179578] niks3-server[1196]: 2026/09/22 10:49:42 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7430lb # [ 205.183084] niks3-server[1196]: 2026/09/22 10:49:42 INFO Completed upload id=737431lb # [ 205.185243] niks3-server[1196]: 2026/09/22 10:49:42 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7432worker2 # [ 205.178192] nix-grpc-daemon[1700]: time=2026-09-22T10:49:42.355Z level=INFO msg="Upload complete. (283ms)"7433worker2 # [ 205.184386] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=16 outputs=17434worker2 # [ 205.196807] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:42Z level=debug event=want client=7 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07435worker2 # [ 205.200806] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:42Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17436worker2 # [ 205.209253] nix-daemon[873]: reaped child process 1925, status = succeeded7437worker2 # [ 205.240769] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448007438worker2 # [ 205.250467] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:448007439worker2 # [ 205.260426] nix-grpc-daemon[1700]: time=2026-09-22T10:49:42.437Z level=INFO msg="All 1 paths already cached"7440client # copying 1 paths to 'lb:50051'...7441client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7442worker2 # [ 205.268158] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:448007443worker2 # [ 205.277153] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)7444worker2 # [ 205.441299] nix-daemon[873]: accepted connection from pid 1952, user nix-grpc-daemon (trusted)7445worker2 # [ 205.453907] nix-daemon[873]: reaped child process 1955, status = succeeded7446worker2 # [ 205.607774] nix-daemon[873]: accepted connection from pid 1957, user nix-grpc-daemon (trusted)7447worker2 # [ 205.619909] nix-daemon[873]: reaped child process 1960, status = succeeded7448lb # [ 205.634823] niks3-server[1196]: 2026/09/22 10:49:42 INFO Received uploads request method=POST path=/api/pending_closures7449worker2 # [ 205.632969] nix-grpc-daemon[1700]: time=2026-09-22T10:49:42.810Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7450worker2 # [ 205.634459] nix-grpc-daemon[1700]: time=2026-09-22T10:49:42.811Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7451lb # [ 205.651724] niks3-server[1196]: 2026/09/22 10:49:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7452lb # [ 205.653780] niks3-server[1196]: 2026/09/22 10:49:42 INFO Signed narinfos id=74 count=17453worker2 # [ 205.647487] nix-grpc-daemon[1700]: time=2026-09-22T10:49:42.824Z level=INFO msg="Uploading 1 narinfos"7454lb # [ 205.657068] niks3-server[1196]: 2026/09/22 10:49:42 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7455lb # [ 205.661601] niks3-server[1196]: 2026/09/22 10:49:42 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7456lb # [ 205.664027] niks3-server[1196]: 2026/09/22 10:49:42 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7457lb # [ 205.667696] niks3-server[1196]: 2026/09/22 10:49:42 INFO Completed upload id=747458worker2 # [ 205.662508] nix-grpc-daemon[1700]: time=2026-09-22T10:49:42.839Z level=INFO msg="Upload complete. (374ms)"7459lb # [ 205.669587] niks3-server[1196]: 2026/09/22 10:49:42 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7460worker2 # [ 205.664660] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=0 paths=1 nar_bytes_in=6247461worker2 # [ 205.668179] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448007462worker2 # [ 205.673983] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)7463worker2 # [ 205.676222] nix-daemon[873]: reaped child process 1950, status = succeeded7464worker2 # [ 205.681050] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)7465client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7466client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7467client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7468client # farm-b> LOG-farm-b-drain7469worker2 # [ 207.105968] nix-daemon[873]: reaped child process 1964, status = succeeded7470worker2 # [ 207.255664] nix-daemon[873]: accepted connection from pid 1980, user nix-grpc-daemon (trusted)7471worker2 # [ 207.266115] nix-daemon[873]: reaped child process 1983, status = succeeded7472lb # [ 207.288164] niks3-server[1196]: 2026/09/22 10:49:44 INFO Received uploads request method=POST path=/api/pending_closures7473worker2 # [ 207.289762] nix-grpc-daemon[1700]: time=2026-09-22T10:49:44.466Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7474worker2 # [ 207.291526] nix-grpc-daemon[1700]: time=2026-09-22T10:49:44.468Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7475lb # [ 207.331133] niks3-server[1196]: 2026/09/22 10:49:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7476worker2 # [ 207.333328] nix-grpc-daemon[1700]: time=2026-09-22T10:49:44.510Z level=INFO msg="Uploading 1 narinfos"7477lb # [ 207.339650] niks3-server[1196]: 2026/09/22 10:49:44 INFO Signed narinfos id=75 count=17478lb # [ 207.349689] niks3-server[1196]: 2026/09/22 10:49:44 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7479lb # [ 207.354362] niks3-server[1196]: 2026/09/22 10:49:44 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7480lb # [ 207.360983] niks3-server[1196]: 2026/09/22 10:49:44 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7481lb # [ 207.369270] niks3-server[1196]: 2026/09/22 10:49:44 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7482lb # [ 207.372123] niks3-server[1196]: 2026/09/22 10:49:44 INFO Completed upload id=757483lb # [ 207.373975] niks3-server[1196]: 2026/09/22 10:49:44 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7484worker2 # [ 207.367319] nix-grpc-daemon[1700]: time=2026-09-22T10:49:44.544Z level=INFO msg="Upload complete. (278ms)"7485worker2 # [ 207.373636] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=17 outputs=17486worker2 # [ 207.383196] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:44Z level=debug event=want client=7 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07487worker2 # [ 207.387211] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:44Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17488worker2 # [ 207.394626] nix-daemon[873]: reaped child process 1962, status = succeeded7489worker2 # [ 207.427475] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448007490worker2 # [ 207.440579] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:448007491worker2 # [ 207.454385] nix-grpc-daemon[1700]: time=2026-09-22T10:49:44.631Z level=INFO msg="All 2 paths already cached"7492client # copying 1 paths to 'lb:50051'...7493client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7494worker2 # [ 207.463421] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:448007495worker2 # [ 207.471300] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)7496worker2 # [ 207.639104] nix-daemon[873]: accepted connection from pid 1989, user nix-grpc-daemon (trusted)7497worker2 # [ 207.651885] nix-daemon[873]: reaped child process 1992, status = succeeded7498worker2 # [ 207.787462] nix-daemon[873]: accepted connection from pid 1994, user nix-grpc-daemon (trusted)7499worker2 # [ 207.798900] nix-daemon[873]: reaped child process 1997, status = succeeded7500lb # [ 207.816149] niks3-server[1196]: 2026/09/22 10:49:44 INFO Received uploads request method=POST path=/api/pending_closures7501worker2 # [ 207.817898] nix-grpc-daemon[1700]: time=2026-09-22T10:49:44.995Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7502worker2 # [ 207.819221] nix-grpc-daemon[1700]: time=2026-09-22T10:49:44.995Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7503lb # [ 207.841828] niks3-server[1196]: 2026/09/22 10:49:44 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7504lb # [ 207.852879] niks3-server[1196]: 2026/09/22 10:49:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign7505lb # [ 207.858801] niks3-server[1196]: 2026/09/22 10:49:44 INFO Signed narinfos id=76 count=17506worker2 # [ 207.855615] nix-grpc-daemon[1700]: time=2026-09-22T10:49:45.032Z level=INFO msg="Uploading 1 narinfos"7507lb # [ 207.863781] niks3-server[1196]: 2026/09/22 10:49:44 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7508lb # [ 207.876732] niks3-server[1196]: 2026/09/22 10:49:44 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete7509worker2 # [ 207.876291] nix-grpc-daemon[1700]: time=2026-09-22T10:49:45.053Z level=INFO msg="Upload complete. (393ms)"7510lb # [ 207.882490] niks3-server[1196]: 2026/09/22 10:49:44 INFO Completed upload id=767511worker2 # [ 207.878346] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=0 paths=1 nar_bytes_in=7527512lb # [ 207.886147] niks3-server[1196]: 2026/09/22 10:49:44 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7513worker2 # [ 207.883841] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:448007514worker2 # [ 207.891768] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)7515worker2 # [ 207.894104] nix-daemon[873]: reaped child process 1987, status = succeeded7516worker2 # [ 207.899210] nix-daemon[873]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)7517client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7518client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7519client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7520client # farm-top> LOG-farm-top-drain7521lb # [ 208.687737] envoy[932]: conn peer=[2001:db8:1::1]:34044 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300517522worker2 # [ 208.719296] nix-daemon[873]: reaped child process 2001, status = succeeded7523worker2 # [ 208.864089] nix-daemon[873]: accepted connection from pid 2017, user nix-grpc-daemon (trusted)7524worker2 # [ 208.874753] nix-daemon[873]: reaped child process 2020, status = succeeded7525lb # [ 208.896805] niks3-server[1196]: 2026/09/22 10:49:45 INFO Received uploads request method=POST path=/api/pending_closures7526worker2 # [ 208.898502] nix-grpc-daemon[1700]: time=2026-09-22T10:49:46.075Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7527worker2 # [ 208.900415] nix-grpc-daemon[1700]: time=2026-09-22T10:49:46.077Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7528lb # [ 208.926658] niks3-server[1196]: 2026/09/22 10:49:45 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7529lb # [ 208.941150] niks3-server[1196]: 2026/09/22 10:49:45 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7530lb # [ 208.953085] niks3-server[1196]: 2026/09/22 10:49:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign7531worker2 # [ 208.950639] nix-grpc-daemon[1700]: time=2026-09-22T10:49:46.127Z level=INFO msg="Uploading 1 narinfos"7532lb # [ 208.957100] niks3-server[1196]: 2026/09/22 10:49:45 INFO Signed narinfos id=77 count=17533lb # [ 208.962698] niks3-server[1196]: 2026/09/22 10:49:45 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7534lb # [ 208.970680] niks3-server[1196]: 2026/09/22 10:49:45 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete7535lb # [ 208.977189] niks3-server[1196]: 2026/09/22 10:49:45 INFO Completed upload id=777536lb # [ 208.978934] niks3-server[1196]: 2026/09/22 10:49:45 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7537worker2 # [ 208.971109] nix-grpc-daemon[1700]: time=2026-09-22T10:49:46.148Z level=INFO msg="Upload complete. (265ms)"7538worker2 # [ 208.977568] nix-grpc-daemon[1620]: ts=2026-09-22T10:49:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44800 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=18 outputs=17539worker2 # [ 208.990945] nix-daemon[873]: reaped child process 1999, status = succeeded7540client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 6.06 seconds)7541client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7542client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.18 seconds)7543worker1: must succeed: rm /nix/.rw-store/fill7544worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.13 seconds)7545worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7546worker1 # [ 209.908449] nix-grpc-daemon[1696]: ts=2026-09-22T10:49:47Z level=info event=healthy reason=min_free7547worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.12 seconds)7548(finished: subtest: low disk drains a worker and builds go to the other, in 8.60 seconds)7549(finished: run the VM test script, in 211.42 seconds)7550test script finished in 211.64s7551cleanup7552kill QemuMachine (pid 49)7553client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7554kill QemuMachine (pid 51)7555lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7556kill QemuMachine (pid 50)7557worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7558kill QemuMachine (pid 52)7559worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7560(finished: cleanup, in 1.19 seconds)7561additionally exposed symbols:7562 client, lb, worker1, worker2,7563 vlan1,7564 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh