vm-test-run-nix-grpc-farm log

succeeded `vm-test-run-nix-grpc-farm` `checks.x86_64-linux.farm` · build #419 · raw

1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker2 # Disk image does not exist, creating the virtualisation disk image...13client # Disk image does not exist, creating the virtualisation disk image...14worker2: QEMU running (pid 52)15worker2 # Formatting '/build/vm-state-worker2/tmp.F9nXh6wVeo', fmt=raw size=107374182416client: QEMU running (pid 49)17worker2 # mke2fs 1.47.4 (6-Mar-2025)18client # Formatting '/build/vm-state-client/tmp.R9XCmYIQNT', fmt=raw size=107374182419worker2 # Discarding device blocks: 0/262144 done20worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes21worker1 # Disk image does not exist, creating the virtualisation disk image...22client # mke2fs 1.47.4 (6-Mar-2025)23worker1: QEMU running (pid 51)24worker2 # Filesystem UUID: bd3eb464-6f09-4836-b084-1e970dd8a12325worker1 # Formatting '/build/vm-state-worker1/tmp.zCtWvqeXer', fmt=raw size=107374182426worker2 # Superblock backups stored on blocks:27worker1 # mke2fs 1.47.4 (6-Mar-2025)28worker2 # 32768, 98304, 163840, 22937629worker1 # Discarding device blocks: 0/262144 done30worker2 # 31worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes32worker1 # Filesystem UUID: 30e5ab67-6b9b-4631-b0be-d6d3157b2cfd33worker2 # Allocating group tables: 0/8 done34client # Discarding device blocks: 0/262144 done35worker2 # Writing inode tables: 0/8 done36client # Creating filesystem with 262144 4k blocks and 65536 inodes37worker2 # Creating journal (8192 blocks): done38client # Filesystem UUID: ede2a6d6-bcc4-47cc-8160-325050fffc0839worker2 # Writing superblocks and filesystem accounting information: 0/8 done40client # Superblock backups stored on blocks:41client # 32768, 98304, 163840, 22937642worker2 # 43worker2 # Virtualisation disk image created.44client # 45worker1 # Superblock backups stored on blocks:46client # Allocating group tables: 0/8 done47worker1 # 32768, 98304, 163840, 22937648client # Writing inode tables: 0/8 done49worker1 # 50client # Creating journal (8192 blocks): done51worker1 # Allocating group tables: 0/8 done52client # Writing superblocks and filesystem accounting information: 0/8 done53worker1 # Writing inode tables: 0/8 done54worker1 # Creating journal (8192 blocks): done55client # 56client # Virtualisation disk image created.57worker1 # Writing superblocks and filesystem accounting information: 0/8 done58worker1 # 59worker1 # Virtualisation disk image created.60lb # Disk image does not exist, creating the virtualisation disk image...61lb: QEMU running (pid 50)62lb # Formatting '/build/vm-state-lb/tmp.qcR6vPJ7aj', fmt=raw size=107374182463lb # mke2fs 1.47.4 (6-Mar-2025)64lb # Discarding device blocks: 0/262144 done65(finished: start all VMs, in 0.56 seconds)66lb # Creating filesystem with 262144 4k blocks and 65536 inodes67client: waiting for unit network-addresses-eth1.service68lb # Filesystem UUID: 110084dc-0a9c-4682-a44c-d9549fc09c4969client: waiting for the VM to finish booting70lb # Superblock backups stored on blocks:71lb # 32768, 98304, 163840, 22937672lb # 73lb # Allocating group tables: 0/8 done74lb # Writing inode tables: 0/8 done75lb # Creating journal (8192 blocks): done76lb # Writing superblocks and filesystem accounting information: 0/8 done77lb # 78lb # Virtualisation disk image created.79client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83client # 84client # 85client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0086lb # 87lb # 88lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0089client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...90client # 91client # 92client # 93client # 94client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB0095lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...96lb # 97lb # 98lb # 99lb # 100worker2 # 101worker2 # 102lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00103worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00104client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...105client # 106client # 107worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...108worker2 # 109worker2 # 110worker2 # 111worker2 # 112lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...113lb # 114lb # 115worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00116worker1 # 117worker1 # 118worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00119worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...120worker2 # 121worker2 # 122worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...123worker1 # 124worker1 # 125worker1 # 126worker1 # 127worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00128worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129worker1 # 130worker1 # 131client # Booting from ROM...132lb # Booting from ROM...133worker2 # Booting from ROM...134lb # Probing EDD (edd=off to disable)... ok135worker1 # Booting from ROM...136client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026137client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/jwg6hsvasqr1bfb4p1nn6ayia3202nfr-nixos-system-client-test/init regInfo=/nix/store/ycwmfabcj5bpk7yq23lg5g1pj9l5i2yl-closure-info/registration console=ttyS0,115200n8 console=tty0138client # [ 0.000000] BIOS-provided physical RAM map:139client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable140client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved141client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved142client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable143client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved144client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved145client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved146client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved147client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved148client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved149lb # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026150client # [ 0.000000] NX (Execute Disable) protection: active151client # [ 0.000000] APIC: Static calls initialized152client # [ 0.000000] SMBIOS 2.8 present.153client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014154client # [ 0.000000] DMI: Memory slots populated: 1/1155client # [ 0.000000] Hypervisor detected: KVM156lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/0qln5mjy767qjp5wbmxjg8jay8k05dsm-nixos-system-lb-test/init regInfo=/nix/store/nwvvzfrgncwhnws6wdjigc6jviavrdxi-closure-info/registration console=ttyS0,115200n8 console=tty0157client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000158lb # [ 0.000000] BIOS-provided physical RAM map:159client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00160lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable161client # [ 0.000000] kvm-clock: using sched offset of 464686806 cycles162lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved163lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved164client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns165client # [ 0.000005] tsc: Detected 2400.012 MHz processor166lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable167client # [ 0.000815] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000168lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved169worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026170client # [ 0.000842] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs171lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved172client # [ 0.000845] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT173lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved174client # [ 0.002748] found SMP MP-table at [mem 0x000f5450-0x000f545f]175lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved176client # [ 0.002761] Using GB pages for direct mapping177worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026178client # [ 0.002830] RAMDISK: [mem 0x5e366000-0x5ffcffff]179lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved180client # [ 0.002840] ACPI: Early table checksum verification disabled181worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qmcgfpcw2i7m6pmgb5saqy8wfr82vfq3-nixos-system-worker1-test/init regInfo=/nix/store/q5s318i3gaswdpaw2h8ilxrpz9w65pb1-closure-info/registration console=ttyS0,115200n8 console=tty0182lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved183client # [ 0.002843] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )184worker1 # [ 0.000000] BIOS-provided physical RAM map:185lb # [ 0.000000] NX (Execute Disable) protection: active186lb # [ 0.000000] APIC: Static calls initialized187worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable188client # [ 0.002848] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)189lb # [ 0.000000] SMBIOS 2.8 present.190worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved191client # [ 0.002853] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)192worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/lapab15gsml370yzals8ffgdzxiibh3q-nixos-system-worker2-test/init regInfo=/nix/store/x39icyk4l2nfvms09gpv9vdscv458nsv-closure-info/registration console=ttyS0,115200n8 console=tty0193lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014194worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved195worker2 # [ 0.000000] BIOS-provided physical RAM map:196lb # [ 0.000000] DMI: Memory slots populated: 1/1197client # [ 0.002861] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)198lb # [ 0.000000] Hypervisor detected: KVM199worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable200client # [ 0.002863] ACPI: FACS 0x000000005FFE0000 000040201worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable202lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000203worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved204worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved205client # [ 0.002865] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)206lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00207worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved208worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved209lb # [ 0.000001] kvm-clock: using sched offset of 467612903 cycles210client # [ 0.002866] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)211worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved212worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable213client # [ 0.002868] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)214worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved215worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved216lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns217lb # [ 0.000005] tsc: Detected 2400.012 MHz processor218worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved219client # [ 0.002869] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)220worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved221lb # [ 0.000819] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000222worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved223worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved224client # [ 0.002870] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]225worker1 # [ 0.000000] NX (Execute Disable) protection: active226lb # [ 0.000846] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs227worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved228client # [ 0.002872] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]229worker1 # [ 0.000000] APIC: Static calls initialized230lb # [ 0.000849] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT231worker1 # [ 0.000000] SMBIOS 2.8 present.232worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved233client # [ 0.002872] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]234lb # [ 0.002751] found SMP MP-table at [mem 0x000f5450-0x000f545f]235worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved236client # [ 0.002873] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]237worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014238lb # [ 0.002762] Using GB pages for direct mapping239worker2 # [ 0.000000] NX (Execute Disable) protection: active240worker1 # [ 0.000000] DMI: Memory slots populated: 1/1241lb # [ 0.002912] RAMDISK: [mem 0x5e367000-0x5ffcffff]242client # [ 0.002873] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]243worker2 # [ 0.000000] APIC: Static calls initialized244worker1 # [ 0.000000] Hypervisor detected: KVM245worker2 # [ 0.000000] SMBIOS 2.8 present.246lb # [ 0.002921] ACPI: Early table checksum verification disabled247client # [ 0.002874] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]248worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000249lb # [ 0.002924] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )250worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00251client # [ 0.002874] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]252worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014253client # [ 0.003120] No NUMA configuration found254worker1 # [ 0.000000] kvm-clock: using sched offset of 485105314 cycles255worker2 # [ 0.000000] DMI: Memory slots populated: 1/1256lb # [ 0.002928] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)257worker2 # [ 0.000000] Hypervisor detected: KVM258client # [ 0.003121] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]259lb # [ 0.002932] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)260worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000261worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns262client # [ 0.003123] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]263client # [ 0.005531] Zone ranges:264worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00265worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor266lb # [ 0.002940] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)267worker1 # [ 0.000824] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000268worker2 # [ 0.000000] kvm-clock: using sched offset of 487913430 cycles269client # [ 0.005532] DMA [mem 0x0000000000001000-0x0000000000ffffff]270lb # [ 0.002942] ACPI: FACS 0x000000005FFE0000 000040271client # [ 0.005534] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]272worker1 # [ 0.000852] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs273client # [ 0.005535] Normal empty274worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns275lb # [ 0.002943] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)276client # [ 0.005536] Device empty277worker2 # [ 0.000006] tsc: Detected 2400.012 MHz processor278worker1 # [ 0.000855] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT279client # [ 0.005536] Movable zone start for each node280lb # [ 0.002945] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)281client # [ 0.005537] Early memory node ranges282worker2 # [ 0.000819] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000283worker1 # [ 0.002771] found SMP MP-table at [mem 0x000f5450-0x000f545f]284worker1 # [ 0.002784] Using GB pages for direct mapping285client # [ 0.005538] node 0: [mem 0x0000000000001000-0x000000000009efff]286lb # [ 0.002946] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)287worker2 # [ 0.000847] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs288worker1 # [ 0.002852] RAMDISK: [mem 0x5e367000-0x5ffcffff]289client # [ 0.005539] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]290worker2 # [ 0.000850] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT291worker1 # [ 0.002861] ACPI: Early table checksum verification disabled292lb # [ 0.002948] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)293client # [ 0.005540] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]294worker1 # [ 0.002865] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )295worker2 # [ 0.002757] found SMP MP-table at [mem 0x000f5450-0x000f545f]296lb # [ 0.002949] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]297worker2 # [ 0.002772] Using GB pages for direct mapping298client # [ 0.005560] On node 0, zone DMA: 1 pages in unavailable ranges299worker1 # [ 0.002871] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)300worker2 # [ 0.002859] RAMDISK: [mem 0x5e367000-0x5ffcffff]301client # [ 0.005819] On node 0, zone DMA: 97 pages in unavailable ranges302lb # [ 0.002950] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]303worker2 # [ 0.002870] ACPI: Early table checksum verification disabled304client # [ 0.032620] On node 0, zone DMA32: 40 pages in unavailable ranges305lb # [ 0.002951] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]306worker1 # [ 0.002876] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)307client # [ 0.033100] ACPI: PM-Timer IO Port: 0x608308worker2 # [ 0.002875] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )309lb # [ 0.002951] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]310client # [ 0.033114] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])311worker1 # [ 0.002884] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)312worker2 # [ 0.002879] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)313worker1 # [ 0.002886] ACPI: FACS 0x000000005FFE0000 000040314lb # [ 0.002952] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]315client # [ 0.033152] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23316worker2 # [ 0.002884] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)317lb # [ 0.002952] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]318worker1 # [ 0.002887] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)319client # [ 0.033155] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)320worker2 # [ 0.002892] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)321client # [ 0.033156] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)322lb # [ 0.002953] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]323worker1 # [ 0.002889] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)324worker2 # [ 0.002894] ACPI: FACS 0x000000005FFE0000 000040325lb # [ 0.003178] No NUMA configuration found326client # [ 0.033158] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)327worker1 # [ 0.002891] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)328lb # [ 0.003179] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]329worker2 # [ 0.002896] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)330client # [ 0.033159] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)331lb # [ 0.003182] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]332worker1 # [ 0.002892] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)333lb # [ 0.005589] Zone ranges:334client # [ 0.033160] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)335worker2 # [ 0.002897] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)336lb # [ 0.005590] DMA [mem 0x0000000000001000-0x0000000000ffffff]337client # [ 0.033162] ACPI: Using ACPI (MADT) for SMP configuration information338worker1 # [ 0.002893] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]339worker2 # [ 0.002899] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)340client # [ 0.033164] ACPI: HPET id: 0x8086a201 base: 0xfed00000341lb # [ 0.005591] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]342worker1 # [ 0.002895] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]343client # [ 0.033169] TSC deadline timer available344lb # [ 0.005592] Normal empty345lb # [ 0.005593] Device empty346worker2 # [ 0.002900] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)347client # [ 0.033173] CPU topo: Max. logical packages: 1348worker1 # [ 0.002895] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]349lb # [ 0.005594] Movable zone start for each node350client # [ 0.033174] CPU topo: Max. logical dies: 1351worker2 # [ 0.002902] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]352lb # [ 0.005595] Early memory node ranges353worker1 # [ 0.002896] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]354client # [ 0.033174] CPU topo: Max. dies per package: 1355worker2 # [ 0.002903] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]356lb # [ 0.005595] node 0: [mem 0x0000000000001000-0x000000000009efff]357client # [ 0.033178] CPU topo: Max. threads per core: 1358worker1 # [ 0.002896] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]359client # [ 0.033179] CPU topo: Num. cores per package: 1360lb # [ 0.005596] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]361worker2 # [ 0.002903] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]362worker1 # [ 0.002897] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]363client # [ 0.033179] CPU topo: Num. threads per package: 1364lb # [ 0.005597] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]365worker2 # [ 0.002904] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]366worker1 # [ 0.002897] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]367client # [ 0.033179] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs368worker1 # [ 0.003126] No NUMA configuration found369lb # [ 0.005617] On node 0, zone DMA: 1 pages in unavailable ranges370worker2 # [ 0.002904] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]371client # [ 0.033205] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()372lb # [ 0.005897] On node 0, zone DMA: 97 pages in unavailable ranges373worker1 # [ 0.003127] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]374worker2 # [ 0.002905] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]375worker1 # [ 0.003130] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]376lb # [ 0.033095] On node 0, zone DMA32: 40 pages in unavailable ranges377client # [ 0.033242] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]378worker1 # [ 0.005377] Zone ranges:379worker2 # [ 0.002905] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]380lb # [ 0.033592] ACPI: PM-Timer IO Port: 0x608381worker2 # [ 0.003151] No NUMA configuration found382client # [ 0.033244] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]383worker1 # [ 0.005378] DMA [mem 0x0000000000001000-0x0000000000ffffff]384lb # [ 0.033604] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])385client # [ 0.033246] [mem 0x60000000-0xafffffff] available for PCI devices386worker1 # [ 0.005380] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]387worker2 # [ 0.003152] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]388lb # [ 0.033636] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23389worker1 # [ 0.005381] Normal empty390client # [ 0.033248] Booting paravirtualized kernel on KVM391worker2 # [ 0.003155] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]392worker1 # [ 0.005382] Device empty393worker2 # [ 0.005688] Zone ranges:394lb # [ 0.033639] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)395worker1 # [ 0.005382] Movable zone start for each node396worker2 # [ 0.005690] DMA [mem 0x0000000000001000-0x0000000000ffffff]397client # [ 0.033252] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns398worker1 # [ 0.005383] Early memory node ranges399lb # [ 0.033640] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)400worker2 # [ 0.005691] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]401worker1 # [ 0.005383] node 0: [mem 0x0000000000001000-0x000000000009efff]402client # [ 0.037711] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1403worker2 # [ 0.005693] Normal empty404lb # [ 0.033642] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)405worker2 # [ 0.005694] Device empty406worker1 # [ 0.005384] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]407client # [ 0.039916] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152408lb # [ 0.033643] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)409worker2 # [ 0.005695] Movable zone start for each node410worker2 # [ 0.005696] Early memory node ranges411client # [ 0.039967] kvm-guest: PV spinlocks disabled, single CPU412worker1 # [ 0.005385] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]413lb # [ 0.033643] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)414worker2 # [ 0.005696] node 0: [mem 0x0000000000001000-0x000000000009efff]415worker1 # [ 0.005405] On node 0, zone DMA: 1 pages in unavailable ranges416lb # [ 0.033646] ACPI: Using ACPI (MADT) for SMP configuration information417worker2 # [ 0.005697] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]418worker1 # [ 0.005671] On node 0, zone DMA: 97 pages in unavailable ranges419lb # [ 0.033647] ACPI: HPET id: 0x8086a201 base: 0xfed00000420worker1 # [ 0.032446] On node 0, zone DMA32: 40 pages in unavailable ranges421lb # [ 0.033651] TSC deadline timer available422worker2 # [ 0.005698] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]423worker1 # [ 0.032923] ACPI: PM-Timer IO Port: 0x608424lb # [ 0.033655] CPU topo: Max. logical packages: 1425worker2 # [ 0.005721] On node 0, zone DMA: 1 pages in unavailable ranges426lb # [ 0.033656] CPU topo: Max. logical dies: 1427worker1 # [ 0.032937] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])428worker2 # [ 0.005982] On node 0, zone DMA: 97 pages in unavailable ranges429client # [ 0.039969] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/jwg6hsvasqr1bfb4p1nn6ayia3202nfr-nixos-system-client-test/init regInfo=/nix/store/ycwmfabcj5bpk7yq23lg5g1pj9l5i2yl-closure-info/registration console=ttyS0,115200n8 console=tty0430lb # [ 0.033656] CPU topo: Max. dies per package: 1431worker1 # [ 0.032964] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23432worker2 # [ 0.032347] On node 0, zone DMA32: 40 pages in unavailable ranges433lb # [ 0.033662] CPU topo: Max. threads per core: 1434worker1 # [ 0.032967] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)435worker2 # [ 0.032838] ACPI: PM-Timer IO Port: 0x608436lb # [ 0.033662] CPU topo: Num. cores per package: 1437client # [ 0.040082] Unknown kernel command line parameters "regInfo=/nix/store/ycwmfabcj5bpk7yq23lg5g1pj9l5i2yl-closure-info/registration", will be passed to user space.438worker2 # [ 0.032855] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])439lb # [ 0.033663] CPU topo: Num. threads per package: 1440worker1 # [ 0.032968] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)441client # [ 0.040096] random: crng init done442lb # [ 0.033663] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs443worker2 # [ 0.032893] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23444worker1 # [ 0.032970] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)445client # [ 0.040097] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes446worker2 # [ 0.032896] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)447lb # [ 0.033683] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()448worker1 # [ 0.032971] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)449client # [ 0.042188] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)450worker2 # [ 0.032898] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)451worker1 # [ 0.032971] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)452lb # [ 0.033716] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]453client # [ 0.043233] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)454worker2 # [ 0.032899] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)455worker1 # [ 0.032974] ACPI: Using ACPI (MADT) for SMP configuration information456client # [ 0.043273] Fallback order for Node 0: 0457lb # [ 0.033718] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]458worker1 # [ 0.032975] ACPI: HPET id: 0x8086a201 base: 0xfed00000459worker2 # [ 0.032900] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)460lb # [ 0.033719] [mem 0x60000000-0xafffffff] available for PCI devices461client # [ 0.043277] Built 1 zonelists, mobility grouping on. Total pages: 393078462worker1 # [ 0.032980] TSC deadline timer available463client # [ 0.043278] Policy zone: DMA32464worker2 # [ 0.032901] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)465lb # [ 0.033721] Booting paravirtualized kernel on KVM466worker1 # [ 0.032984] CPU topo: Max. logical packages: 1467worker2 # [ 0.032904] ACPI: Using ACPI (MADT) for SMP configuration information468client # [ 0.045904] mem auto-init: stack:all(zero), heap alloc:on, heap free:off469worker1 # [ 0.032985] CPU topo: Max. logical dies: 1470worker1 # [ 0.032985] CPU topo: Max. dies per package: 1471lb # [ 0.033724] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns472worker2 # [ 0.032905] ACPI: HPET id: 0x8086a201 base: 0xfed00000473client # [ 0.049274] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1474worker1 # [ 0.032989] CPU topo: Max. threads per core: 1475worker2 # [ 0.032912] TSC deadline timer available476client # [ 0.052514] allocated 3145728 bytes of page_ext477lb # [ 0.038179] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1478worker1 # [ 0.032990] CPU topo: Num. cores per package: 1479worker2 # [ 0.032916] CPU topo: Max. logical packages: 1480client # [ 0.062898] ftrace: allocating 48731 entries in 192 pages481worker2 # [ 0.032917] CPU topo: Max. logical dies: 1482worker1 # [ 0.032990] CPU topo: Num. threads per package: 1483lb # [ 0.040309] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152484client # [ 0.062902] ftrace: allocated 192 pages with 2 groups485worker2 # [ 0.032917] CPU topo: Max. dies per package: 1486lb # [ 0.040355] kvm-guest: PV spinlocks disabled, single CPU487client # [ 0.063826] Dynamic Preempt: lazy488worker1 # [ 0.032990] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs489worker2 # [ 0.032921] CPU topo: Max. threads per core: 1490client # [ 0.063993] rcu: Preemptible hierarchical RCU implementation.491worker1 # [ 0.033014] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()492worker2 # [ 0.032922] CPU topo: Num. cores per package: 1493client # [ 0.063994] rcu: RCU event tracing is enabled.494worker2 # [ 0.032922] CPU topo: Num. threads per package: 1495worker1 # [ 0.033054] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]496client # [ 0.063995] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.497worker2 # [ 0.032922] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs498client # [ 0.063996] Trampoline variant of Tasks RCU enabled.499worker1 # [ 0.033055] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]500worker2 # [ 0.032948] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()501client # [ 0.063997] Rude variant of Tasks RCU enabled.502worker1 # [ 0.033057] [mem 0x60000000-0xafffffff] available for PCI devices503client # [ 0.063997] Tracing variant of Tasks RCU enabled.504lb # [ 0.040357] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/0qln5mjy767qjp5wbmxjg8jay8k05dsm-nixos-system-lb-test/init regInfo=/nix/store/nwvvzfrgncwhnws6wdjigc6jviavrdxi-closure-info/registration console=ttyS0,115200n8 console=tty0505worker1 # [ 0.033058] Booting paravirtualized kernel on KVM506worker2 # [ 0.032987] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]507client # [ 0.063998] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.508worker2 # [ 0.032989] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]509worker1 # [ 0.033062] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns510client # [ 0.063999] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1511lb # [ 0.040456] Unknown kernel command line parameters "regInfo=/nix/store/nwvvzfrgncwhnws6wdjigc6jviavrdxi-closure-info/registration", will be passed to user space.512worker2 # [ 0.032990] [mem 0x60000000-0xafffffff] available for PCI devices513lb # [ 0.040469] random: crng init done514worker1 # [ 0.037535] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1515worker2 # [ 0.032992] Booting paravirtualized kernel on KVM516client # [ 0.064021] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.517lb # [ 0.040470] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes518worker1 # [ 0.039838] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152519worker2 # [ 0.032996] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns520worker1 # [ 0.039881] kvm-guest: PV spinlocks disabled, single CPU521client # [ 0.064023] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.522lb # [ 0.042901] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)523worker2 # [ 0.037500] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1524client # [ 0.064024] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.525lb # [ 0.044022] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)526worker2 # [ 0.039612] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152527lb # [ 0.044055] Fallback order for Node 0: 0528client # [ 0.068413] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16529worker2 # [ 0.039664] kvm-guest: PV spinlocks disabled, single CPU530lb # [ 0.044058] Built 1 zonelists, mobility grouping on. Total pages: 393078531client # [ 0.068708] rcu: srcu_init: Setting srcu_struct sizes based on contention.532lb # [ 0.044059] Policy zone: DMA32533worker1 # [ 0.039883] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qmcgfpcw2i7m6pmgb5saqy8wfr82vfq3-nixos-system-worker1-test/init regInfo=/nix/store/q5s318i3gaswdpaw2h8ilxrpz9w65pb1-closure-info/registration console=ttyS0,115200n8 console=tty0534client # [ 0.068719] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns535lb # [ 0.046484] mem auto-init: stack:all(zero), heap alloc:on, heap free:off536lb # [ 0.049772] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1537client # [ 0.068832] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)538lb # [ 0.053043] allocated 3145728 bytes of page_ext539client # [ 0.072410] Console: colour VGA+ 80x25540worker1 # [ 0.039979] Unknown kernel command line parameters "regInfo=/nix/store/q5s318i3gaswdpaw2h8ilxrpz9w65pb1-closure-info/registration", will be passed to user space.541lb # [ 0.063584] ftrace: allocating 48731 entries in 192 pages542worker2 # [ 0.039665] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/lapab15gsml370yzals8ffgdzxiibh3q-nixos-system-worker2-test/init regInfo=/nix/store/x39icyk4l2nfvms09gpv9vdscv458nsv-closure-info/registration console=ttyS0,115200n8 console=tty0543client # [ 0.072413] printk: legacy console [tty0] enabled544worker1 # [ 0.039993] random: crng init done545lb # [ 0.063586] ftrace: allocated 192 pages with 2 groups546client # [ 0.101929] printk: legacy console [ttyS0] enabled547lb # [ 0.064516] Dynamic Preempt: lazy548worker1 # [ 0.039994] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes549client # [ 0.197831] ACPI: Core revision 20250807550lb # [ 0.064699] rcu: Preemptible hierarchical RCU implementation.551worker2 # [ 0.039766] Unknown kernel command line parameters "regInfo=/nix/store/x39icyk4l2nfvms09gpv9vdscv458nsv-closure-info/registration", will be passed to user space.552worker1 # [ 0.042066] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)553lb # [ 0.064700] rcu: RCU event tracing is enabled.554worker2 # [ 0.039780] random: crng init done555worker1 # [ 0.043110] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)556lb # [ 0.064701] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.557client # [ 0.198674] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns558worker2 # [ 0.039781] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes559worker1 # [ 0.043143] Fallback order for Node 0: 0560lb # [ 0.064702] Trampoline variant of Tasks RCU enabled.561worker2 # [ 0.042044] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)562client # [ 0.200164] APIC: Switch to symmetric I/O mode setup563lb # [ 0.064703] Rude variant of Tasks RCU enabled.564worker1 # [ 0.043146] Built 1 zonelists, mobility grouping on. Total pages: 393078565worker1 # [ 0.043147] Policy zone: DMA32566client # [ 0.201105] x2apic enabled567lb # [ 0.064703] Tracing variant of Tasks RCU enabled.568worker2 # [ 0.043073] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)569worker2 # [ 0.043110] Fallback order for Node 0: 0570worker1 # [ 0.045571] mem auto-init: stack:all(zero), heap alloc:on, heap free:off571lb # [ 0.064704] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.572client # [ 0.201835] APIC: Switched APIC routing to: physical x2apic573worker2 # [ 0.043114] Built 1 zonelists, mobility grouping on. Total pages: 393078574worker1 # [ 0.048773] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1575lb # [ 0.064705] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1576worker2 # [ 0.043116] Policy zone: DMA32577worker1 # [ 0.052280] allocated 3145728 bytes of page_ext578worker2 # [ 0.045557] mem auto-init: stack:all(zero), heap alloc:on, heap free:off579worker1 # [ 0.062543] ftrace: allocating 48731 entries in 192 pages580lb # [ 0.064731] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.581client # [ 0.203626] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1582worker1 # [ 0.062545] ftrace: allocated 192 pages with 2 groups583worker2 # [ 0.048867] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1584worker1 # [ 0.063377] Dynamic Preempt: lazy585worker2 # [ 0.052284] allocated 3145728 bytes of page_ext586worker1 # [ 0.063545] rcu: Preemptible hierarchical RCU implementation.587worker2 # [ 0.062856] ftrace: allocating 48731 entries in 192 pages588lb # [ 0.064734] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.589client # [ 0.204562] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns590worker1 # [ 0.063546] rcu: RCU event tracing is enabled.591lb # [ 0.064735] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.592worker2 # [ 0.062859] ftrace: allocated 192 pages with 2 groups593worker2 # [ 0.063784] Dynamic Preempt: lazy594worker1 # [ 0.063547] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.595lb # [ 0.069177] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16596client # [ 0.206165] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)597worker2 # [ 0.063961] rcu: Preemptible hierarchical RCU implementation.598worker1 # [ 0.063548] Trampoline variant of Tasks RCU enabled.599lb # [ 0.069467] rcu: srcu_init: Setting srcu_struct sizes based on contention.600worker2 # [ 0.063962] rcu: RCU event tracing is enabled.601worker1 # [ 0.063549] Rude variant of Tasks RCU enabled.602client # [ 0.207251] x86/cpu: User Mode Instruction Prevention (UMIP) activated603worker1 # [ 0.063549] Tracing variant of Tasks RCU enabled.604worker2 # [ 0.063963] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.605lb # [ 0.069475] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns606worker2 # [ 0.063964] Trampoline variant of Tasks RCU enabled.607client # [ 0.208329] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127608worker1 # [ 0.063550] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.609worker2 # [ 0.063965] Rude variant of Tasks RCU enabled.610lb # [ 0.069581] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)611worker1 # [ 0.063551] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1612client # [ 0.209163] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0613worker2 # [ 0.063965] Tracing variant of Tasks RCU enabled.614lb # [ 0.073342] Console: colour VGA+ 80x25615lb # [ 0.073345] printk: legacy console [tty0] enabled616worker2 # [ 0.063966] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.617worker1 # [ 0.063565] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.618lb # [ 0.103456] printk: legacy console [ttyS0] enabled619client # [ 0.210166] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto620worker2 # [ 0.063966] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1621worker1 # [ 0.063567] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.622lb # [ 0.201231] ACPI: Core revision 20250807623worker2 # [ 0.063992] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.624client # [ 0.211162] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl625worker1 # [ 0.063568] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.626worker1 # [ 0.067977] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16627worker2 # [ 0.063994] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.628lb # [ 0.202167] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns629client # [ 0.212162] Transient Scheduler Attacks: Vulnerable: No microcode630worker1 # [ 0.068257] rcu: srcu_init: Setting srcu_struct sizes based on contention.631client # [ 0.213162] Spectre V2 : Mitigation: Enhanced / Automatic IBRS632lb # [ 0.203760] APIC: Switch to symmetric I/O mode setup633worker2 # [ 0.063995] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.634lb # [ 0.204714] x2apic enabled635worker2 # [ 0.068393] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16636client # [ 0.214162] Speculative Return Stack Overflow: Mitigation: Safe RET637worker1 # [ 0.068263] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns638worker2 # [ 0.068675] rcu: srcu_init: Setting srcu_struct sizes based on contention.639lb # [ 0.205484] APIC: Switched APIC routing to: physical x2apic640worker1 # [ 0.068371] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)641worker1 # [ 0.071917] Console: colour VGA+ 80x25642worker2 # [ 0.068685] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns643worker1 # [ 0.071921] printk: legacy console [tty0] enabled644worker1 # [ 0.101529] printk: legacy console [ttyS0] enabled645lb # [ 0.207298] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1646worker2 # [ 0.068798] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)647worker2 # [ 0.072381] Console: colour VGA+ 80x25648worker1 # [ 0.197570] ACPI: Core revision 20250807649client # [ 0.215162] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization650worker2 # [ 0.072385] printk: legacy console [tty0] enabled651lb # [ 0.208264] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns652worker2 # [ 0.101844] printk: legacy console [ttyS0] enabled653client # [ 0.217170] Spectre V2 : Enabling IBPB for BPF654worker1 # [ 0.198430] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns655worker2 # [ 0.197055] ACPI: Core revision 20250807656lb # [ 0.209874] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)657client # [ 0.217865] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier658worker1 # [ 0.199987] APIC: Switch to symmetric I/O mode setup659client # [ 0.219163] active return thunk: srso_alias_return_thunk660worker1 # [ 0.200927] x2apic enabled661lb # [ 0.210958] x86/cpu: User Mode Instruction Prevention (UMIP) activated662worker2 # [ 0.197920] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns663worker2 # [ 0.199484] APIC: Switch to symmetric I/O mode setup664worker1 # [ 0.201677] APIC: Switched APIC routing to: physical x2apic665client # [ 0.219995] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'666lb # [ 0.212022] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127667worker2 # [ 0.200416] x2apic enabled668client # [ 0.221161] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'669lb # [ 0.212871] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0670worker2 # [ 0.201148] APIC: Switched APIC routing to: physical x2apic671worker1 # [ 0.203478] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1672client # [ 0.222161] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'673lb # [ 0.213875] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto674client # [ 0.223161] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'675worker1 # [ 0.204426] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns676worker2 # [ 0.202956] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1677client # [ 0.224107] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'678lb # [ 0.215871] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl679worker1 # [ 0.206062] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)680client # [ 0.224875] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'681lb # [ 0.216871] Transient Scheduler Attacks: Vulnerable: No microcode682worker2 # [ 0.203905] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns683worker1 # [ 0.207141] x86/cpu: User Mode Instruction Prevention (UMIP) activated684lb # [ 0.217871] Spectre V2 : Mitigation: Enhanced / Automatic IBRS685client # [ 0.225907] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'686worker2 # [ 0.205512] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)687worker1 # [ 0.208175] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127688client # [ 0.227162] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256689lb # [ 0.218871] Speculative Return Stack Overflow: Mitigation: Safe RET690worker2 # [ 0.206598] x86/cpu: User Mode Instruction Prevention (UMIP) activated691client # [ 0.228063] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64692worker1 # [ 0.209056] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0693worker2 # [ 0.207674] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127694client # [ 0.228862] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512695worker1 # [ 0.210059] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto696worker2 # [ 0.208509] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0697client # [ 0.229866] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024698client # [ 0.230865] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8699lb # [ 0.219870] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization700worker1 # [ 0.211056] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl701worker2 # [ 0.209513] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto702lb # [ 0.221878] Spectre V2 : Enabling IBPB for BPF703worker1 # [ 0.212056] Transient Scheduler Attacks: Vulnerable: No microcode704client # [ 0.231161] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.705worker2 # [ 0.210509] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl706worker1 # [ 0.213055] Spectre V2 : Mitigation: Enhanced / Automatic IBRS707lb # [ 0.222872] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier708worker2 # [ 0.211509] Transient Scheduler Attacks: Vulnerable: No microcode709worker1 # [ 0.214056] Speculative Return Stack Overflow: Mitigation: Safe RET710lb # [ 0.223871] active return thunk: srso_alias_return_thunk711worker2 # [ 0.212508] Spectre V2 : Mitigation: Enhanced / Automatic IBRS712lb # [ 0.224893] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'713worker2 # [ 0.213508] Speculative Return Stack Overflow: Mitigation: Safe RET714lb # [ 0.225870] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'715lb # [ 0.226870] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'716worker1 # [ 0.215055] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization717lb # [ 0.227871] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'718worker1 # [ 0.217062] Spectre V2 : Enabling IBPB for BPF719lb # [ 0.228871] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'720worker2 # [ 0.214508] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization721worker1 # [ 0.217778] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier722worker2 # [ 0.216516] Spectre V2 : Enabling IBPB for BPF723lb # [ 0.229870] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'724worker1 # [ 0.219056] active return thunk: srso_alias_return_thunk725worker2 # [ 0.217226] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier726lb # [ 0.230870] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'727worker1 # [ 0.220076] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'728worker2 # [ 0.218509] active return thunk: srso_alias_return_thunk729lb # [ 0.231871] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256730worker1 # [ 0.221055] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'731lb # [ 0.233600] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64732worker2 # [ 0.219389] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'733worker1 # [ 0.222055] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'734lb # [ 0.234606] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512735worker2 # [ 0.220508] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'736worker1 # [ 0.223055] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'737lb # [ 0.235610] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024738worker2 # [ 0.221508] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'739worker1 # [ 0.224055] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'740lb # [ 0.236598] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8741worker2 # [ 0.222508] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'742worker1 # [ 0.225055] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'743worker2 # [ 0.223508] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'744lb # [ 0.236871] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.745worker1 # [ 0.226055] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'746worker2 # [ 0.224508] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'747worker1 # [ 0.227056] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256748worker2 # [ 0.225508] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'749worker1 # [ 0.228055] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64750worker2 # [ 0.226509] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256751worker1 # [ 0.229055] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512752worker2 # [ 0.227508] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64753worker1 # [ 0.230055] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024754worker2 # [ 0.228508] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512755worker1 # [ 0.231055] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8756worker2 # [ 0.229508] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024757worker2 # [ 0.230508] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8758worker1 # [ 0.232055] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759worker2 # [ 0.231435] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.760client # [ 0.266562] Freeing SMP alternatives memory: 44K761client # [ 0.267165] pid_max: default: 32768 minimum: 301762client # [ 0.268262] LSM: initializing lsm=capability,landlock,yama,bpf,ima763client # [ 0.269265] landlock: Up and running.764client # [ 0.269881] Yama: becoming mindful.765client # [ 0.270790] LSM support for eBPF active766client # [ 0.271294] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)767client # [ 0.272197] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)768client # [ 0.275229] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)769client # [ 0.276813] Performance Events: Fam17h+ core perfctr, AMD PMU driver.770client # [ 0.277170] ... version: 2771client # [ 0.277937] ... bit width: 48772client # [ 0.278164] ... generic counters: 6773client # [ 0.278913] ... generic bitmap: 000000000000003f774client # [ 0.279165] ... fixed-purpose counters: 0775client # [ 0.279917] ... fixed-purpose bitmap: 0000000000000000776lb # [ 0.272944] Freeing SMP alternatives memory: 44K777lb # [ 0.273725] pid_max: default: 32768 minimum: 301778client # [ 0.280164] ... value mask: 0000ffffffffffff779client # [ 0.281068] ... max period: 00007fffffffffff780lb # [ 0.274514] LSM: initializing lsm=capability,landlock,yama,bpf,ima781lb # [ 0.274969] landlock: Up and running.782client # [ 0.281851] ... global_ctrl mask: 000000000000003f783lb # [ 0.275871] Yama: becoming mindful.784client # [ 0.282262] signal: max sigframe size: 3376785lb # [ 0.277085] LSM support for eBPF active786client # [ 0.283054] rcu: Hierarchical SRCU implementation.787client # [ 0.283760] rcu: Max phase no-delay instances is 400.788lb # [ 0.277823] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)789lb # [ 0.278909] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)790worker1 # [ 0.268016] Freeing SMP alternatives memory: 44K791worker1 # [ 0.268616] pid_max: default: 32768 minimum: 301792worker1 # [ 0.269147] LSM: initializing lsm=capability,landlock,yama,bpf,ima793worker2 # [ 0.267817] Freeing SMP alternatives memory: 44K794lb # [ 0.281452] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)795worker1 # [ 0.270153] landlock: Up and running.796worker2 # [ 0.268513] pid_max: default: 32768 minimum: 301797worker1 # [ 0.271057] Yama: becoming mindful.798client # [ 0.288887] smp: Bringing up secondary CPUs ...799lb # [ 0.282419] Performance Events: Fam17h+ core perfctr, AMD PMU driver.800worker1 # [ 0.272189] LSM support for eBPF active801worker2 # [ 0.269355] LSM: initializing lsm=capability,landlock,yama,bpf,ima802client # [ 0.289179] smp: Brought up 1 node, 1 CPU803worker2 # [ 0.270561] landlock: Up and running.804lb # [ 0.282876] ... version: 2805worker2 # [ 0.271144] Yama: becoming mindful.806worker1 # [ 0.272938] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)807lb # [ 0.283566] ... bit width: 48808client # [ 0.289825] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)809worker2 # [ 0.271728] LSM support for eBPF active810lb # [ 0.283873] ... generic counters: 6811worker1 # [ 0.274091] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)812lb # [ 0.284580] ... generic bitmap: 000000000000003f813worker2 # [ 0.272612] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)814lb # [ 0.284920] ... fixed-purpose counters: 0815client # [ 0.290365] Memory: 1454548K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)816worker2 # [ 0.273543] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)817lb # [ 0.285620] ... fixed-purpose bitmap: 0000000000000000818client # [ 0.291421] devtmpfs: initialized819worker1 # [ 0.276377] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)820lb # [ 0.285873] ... value mask: 0000ffffffffffff821client # [ 0.292158] x86/mm: Memory block size: 128MB822lb # [ 0.286763] ... max period: 00007fffffffffff823worker1 # [ 0.277645] Performance Events: Fam17h+ core perfctr, AMD PMU driver.824worker2 # [ 0.276055] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)825lb # [ 0.287543] ... global_ctrl mask: 000000000000003f826worker1 # [ 0.278065] ... version: 2827client # [ 0.293765] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)828worker1 # [ 0.278749] ... bit width: 48829lb # [ 0.287987] signal: max sigframe size: 3376830worker2 # [ 0.277044] Performance Events: Fam17h+ core perfctr, AMD PMU driver.831worker1 # [ 0.279058] ... generic counters: 6832lb # [ 0.288762] rcu: Hierarchical SRCU implementation.833client # [ 0.294194] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).834worker2 # [ 0.277513] ... version: 2835worker1 # [ 0.279743] ... generic bitmap: 000000000000003f836lb # [ 0.289433] rcu: Max phase no-delay instances is 400.837worker2 # [ 0.278194] ... bit width: 48838client # [ 0.295237] pinctrl core: initialized pinctrl subsystem839worker1 # [ 0.280058] ... fixed-purpose counters: 0840worker2 # [ 0.278511] ... generic counters: 6841client # [ 0.296316] PM: RTC time: 13:33:03, date: 2026-09-20842worker1 # [ 0.280727] ... fixed-purpose bitmap: 0000000000000000843worker2 # [ 0.279210] ... generic bitmap: 000000000000003f844worker1 # [ 0.281058] ... value mask: 0000ffffffffffff845worker2 # [ 0.279522] ... fixed-purpose counters: 0846worker1 # [ 0.281939] ... max period: 00007fffffffffff847worker2 # [ 0.280277] ... fixed-purpose bitmap: 0000000000000000848worker1 # [ 0.282725] ... global_ctrl mask: 000000000000003f849worker2 # [ 0.280511] ... value mask: 0000ffffffffffff850worker1 # [ 0.283168] signal: max sigframe size: 3376851client # [ 0.299722] NET: Registered PF_NETLINK/PF_ROUTE protocol family852worker2 # [ 0.281413] ... max period: 00007fffffffffff853lb # [ 0.294437] smp: Bringing up secondary CPUs ...854worker1 # [ 0.283929] rcu: Hierarchical SRCU implementation.855lb # [ 0.294887] smp: Brought up 1 node, 1 CPU856worker2 # [ 0.282153] ... global_ctrl mask: 000000000000003f857worker1 # [ 0.284618] rcu: Max phase no-delay instances is 400.858client # [ 0.300644] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations859worker2 # [ 0.282630] signal: max sigframe size: 3376860lb # [ 0.295548] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)861client # [ 0.301191] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations862worker2 # [ 0.283402] rcu: Hierarchical SRCU implementation.863worker2 # [ 0.284112] rcu: Max phase no-delay instances is 400.864lb # [ 0.296035] Memory: 1454540K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)865client # [ 0.302491] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations866lb # [ 0.297144] devtmpfs: initialized867client # [ 0.303175] audit: initializing netlink subsys (disabled)868lb # [ 0.298054] x86/mm: Memory block size: 128MB869client # [ 0.304315] thermal_sys: Registered thermal governor 'fair_share'870worker1 # [ 0.289633] smp: Bringing up secondary CPUs ...871worker1 # [ 0.290073] smp: Brought up 1 node, 1 CPU872client # [ 0.304317] thermal_sys: Registered thermal governor 'bang_bang'873client # [ 0.305165] thermal_sys: Registered thermal governor 'step_wise'874worker1 # [ 0.290723] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)875lb # [ 0.299880] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)876worker2 # [ 0.289227] smp: Bringing up secondary CPUs ...877client # [ 0.306147] thermal_sys: Registered thermal governor 'user_space'878worker2 # [ 0.289527] smp: Brought up 1 node, 1 CPU879lb # [ 0.300896] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).880worker1 # [ 0.291259] Memory: 1454576K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)881client # [ 0.306885] audit: type=2000 audit(1789911184.584:1): state=initialized audit_enabled=0 res=1882lb # [ 0.301960] pinctrl core: initialized pinctrl subsystem883worker2 # [ 0.290158] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)884worker1 # [ 0.292288] devtmpfs: initialized885client # [ 0.308167] thermal_sys: Registered thermal governor 'power_allocator'886lb # [ 0.303063] PM: RTC time: 13:33:03, date: 2026-09-20887client # [ 0.308182] cpuidle: using governor menu888worker1 # [ 0.293110] x86/mm: Memory block size: 128MB889worker2 # [ 0.290736] Memory: 1454552K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)890worker2 # [ 0.291834] devtmpfs: initialized891client # [ 0.310388] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5892worker1 # [ 0.294884] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)893worker2 # [ 0.292608] x86/mm: Memory block size: 128MB894client # [ 0.311386] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]895lb # [ 0.306636] NET: Registered PF_NETLINK/PF_ROUTE protocol family896worker1 # [ 0.295943] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).897client # [ 0.312168] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry898worker1 # [ 0.297137] pinctrl core: initialized pinctrl subsystem899worker2 # [ 0.294374] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)900lb # [ 0.307379] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations901client # [ 0.313177] PCI: Using configuration type 1 for base access902worker1 # [ 0.298211] PM: RTC time: 13:33:03, date: 2026-09-20903lb # [ 0.307900] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations904worker2 # [ 0.295365] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).905client # [ 0.314300] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.906worker2 # [ 0.296603] pinctrl core: initialized pinctrl subsystem907lb # [ 0.309152] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations908worker2 # [ 0.297734] PM: RTC time: 13:33:03, date: 2026-09-20909lb # [ 0.309884] audit: initializing netlink subsys (disabled)910worker1 # [ 0.301983] NET: Registered PF_NETLINK/PF_ROUTE protocol family911lb # [ 0.310875] audit: type=2000 audit(1789911184.598:1): state=initialized audit_enabled=0 res=1912lb # [ 0.311995] thermal_sys: Registered thermal governor 'fair_share'913worker1 # [ 0.303305] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations914worker2 # [ 0.301215] NET: Registered PF_NETLINK/PF_ROUTE protocol family915lb # [ 0.311997] thermal_sys: Registered thermal governor 'bang_bang'916client # [ 0.319442] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages917worker1 # [ 0.304085] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations918lb # [ 0.312874] thermal_sys: Registered thermal governor 'step_wise'919client # [ 0.320166] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page920worker2 # [ 0.302029] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations921lb # [ 0.313806] thermal_sys: Registered thermal governor 'user_space'922worker1 # [ 0.305346] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations923worker2 # [ 0.302538] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations924worker1 # [ 0.306069] audit: initializing netlink subsys (disabled)925lb # [ 0.314597] thermal_sys: Registered thermal governor 'power_allocator'926lb # [ 0.314897] cpuidle: using governor menu927worker1 # [ 0.307200] thermal_sys: Registered thermal governor 'fair_share'928worker2 # [ 0.303780] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations929worker1 # [ 0.307202] thermal_sys: Registered thermal governor 'bang_bang'930worker2 # [ 0.304524] audit: initializing netlink subsys (disabled)931lb # [ 0.317119] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5932worker1 # [ 0.308059] thermal_sys: Registered thermal governor 'step_wise'933client # [ 0.325166] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages934worker2 # [ 0.305659] thermal_sys: Registered thermal governor 'fair_share'935lb # [ 0.318111] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]936worker1 # [ 0.308975] thermal_sys: Registered thermal governor 'user_space'937client # [ 0.326164] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page938worker2 # [ 0.305661] thermal_sys: Registered thermal governor 'bang_bang'939lb # [ 0.318876] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry940worker1 # [ 0.309749] audit: type=2000 audit(1789911184.608:1): state=initialized audit_enabled=0 res=1941worker2 # [ 0.306511] thermal_sys: Registered thermal governor 'step_wise'942lb # [ 0.319887] PCI: Using configuration type 1 for base access943worker2 # [ 0.307418] thermal_sys: Registered thermal governor 'user_space'944worker1 # [ 0.311061] thermal_sys: Registered thermal governor 'power_allocator'945worker1 # [ 0.311090] cpuidle: using governor menu946lb # [ 0.320966] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.947worker2 # [ 0.308233] audit: type=2000 audit(1789911184.609:1): state=initialized audit_enabled=0 res=1948worker1 # [ 0.313265] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5949worker2 # [ 0.309514] thermal_sys: Registered thermal governor 'power_allocator'950worker2 # [ 0.309532] cpuidle: using governor menu951worker1 # [ 0.314312] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]952worker2 # [ 0.311773] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5953worker1 # [ 0.315062] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry954worker1 # [ 0.316071] PCI: Using configuration type 1 for base access955worker2 # [ 0.312757] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]956worker1 # [ 0.317136] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.957worker2 # [ 0.313515] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry958client # [ 0.336588] ACPI: Added _OSI(Module Device)959lb # [ 0.327902] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages960worker2 # [ 0.314524] PCI: Using configuration type 1 for base access961client # [ 0.337166] ACPI: Added _OSI(Processor Device)962lb # [ 0.328874] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page963client # [ 0.337901] ACPI: Added _OSI(Processor Aggregator Device)964worker2 # [ 0.315615] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.965worker1 # [ 0.322330] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages966worker1 # [ 0.323059] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page967lb # [ 0.333874] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages968lb # [ 0.334874] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page969worker2 # [ 0.321753] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages970worker2 # [ 0.322513] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page971worker1 # [ 0.328059] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages972client # [ 0.346714] ACPI: 1 ACPI AML tables successfully acquired and loaded973worker1 # [ 0.329058] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page974worker2 # [ 0.327520] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages975worker2 # [ 0.328511] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page976client # [ 0.350603] ACPI: Interpreter enabled977client # [ 0.351181] ACPI: PM: (supports S0 S3 S4 S5)978client # [ 0.351919] ACPI: Using IOAPIC for interrupt routing979lb # [ 0.346158] ACPI: Added _OSI(Module Device)980client # [ 0.352214] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug981lb # [ 0.346850] ACPI: Added _OSI(Processor Device)982client # [ 0.353165] PCI: Using E820 reservations for host bridge windows983lb # [ 0.347414] ACPI: Added _OSI(Processor Aggregator Device)984worker1 # [ 0.339401] ACPI: Added _OSI(Module Device)985worker1 # [ 0.340060] ACPI: Added _OSI(Processor Device)986client # [ 0.356298] ACPI: Enabled 2 GPEs in block 00 to 3F987worker1 # [ 0.340772] ACPI: Added _OSI(Processor Aggregator Device)988worker2 # [ 0.338968] ACPI: Added _OSI(Module Device)989worker2 # [ 0.339512] ACPI: Added _OSI(Processor Device)990lb # [ 0.353596] ACPI: 1 ACPI AML tables successfully acquired and loaded991worker2 # [ 0.340233] ACPI: Added _OSI(Processor Aggregator Device)992lb # [ 0.357275] ACPI: Interpreter enabled993lb # [ 0.357889] ACPI: PM: (supports S0 S3 S4 S5)994client # [ 0.365047] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])995lb # [ 0.358594] ACPI: Using IOAPIC for interrupt routing996worker1 # [ 0.349465] ACPI: 1 ACPI AML tables successfully acquired and loaded997client # [ 0.365964] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]998client # [ 0.367593] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]999lb # [ 0.360945] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1000worker2 # [ 0.348063] ACPI: 1 ACPI AML tables successfully acquired and loaded1001client # [ 0.368288] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1002lb # [ 0.361874] PCI: Using E820 reservations for host bridge windows1003worker1 # [ 0.353471] ACPI: Interpreter enabled1004client # [ 0.369545] PCI host bridge to bus 0000:001005worker1 # [ 0.354073] ACPI: PM: (supports S0 S3 S4 S5)1006client # [ 0.370168] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1007worker1 # [ 0.354787] ACPI: Using IOAPIC for interrupt routing1008client # [ 0.371165] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1009lb # [ 0.364955] ACPI: Enabled 2 GPEs in block 00 to 3F1010worker2 # [ 0.351935] ACPI: Interpreter enabled1011client # [ 0.372164] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1012worker2 # [ 0.352528] ACPI: PM: (supports S0 S3 S4 S5)1013worker2 # [ 0.353234] ACPI: Using IOAPIC for interrupt routing1014client # [ 0.373165] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1015worker1 # [ 0.357122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1016worker1 # [ 0.358059] PCI: Using E820 reservations for host bridge windows1017client # [ 0.374164] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1018worker2 # [ 0.353598] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1019worker2 # [ 0.354513] PCI: Using E820 reservations for host bridge windows1020client # [ 0.375164] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1021client # [ 0.376166] pci_bus 0000:00: root bus resource [bus 00-ff]1022worker1 # [ 0.361123] ACPI: Enabled 2 GPEs in block 00 to 3F1023client # [ 0.377167] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1024worker2 # [ 0.357659] ACPI: Enabled 2 GPEs in block 00 to 3F1025lb # [ 0.372372] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1026client # [ 0.378608] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1027lb # [ 0.372881] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1028lb # [ 0.373953] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1029lb # [ 0.374996] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1030lb # [ 0.376223] PCI host bridge to bus 0000:001031lb # [ 0.376878] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1032worker1 # [ 0.369075] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1033lb # [ 0.377873] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1034worker1 # [ 0.370065] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1035lb # [ 0.378873] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1036worker2 # [ 0.366399] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1037client # [ 0.381198] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1038worker1 # [ 0.371147] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1039lb # [ 0.379874] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1040client # [ 0.382176] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1041worker2 # [ 0.367328] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1042worker1 # [ 0.372182] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1043lb # [ 0.380874] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1044client # [ 0.383113] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1045worker1 # [ 0.373428] PCI host bridge to bus 0000:001046lb # [ 0.381873] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1047worker2 # [ 0.369264] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1048worker1 # [ 0.374063] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1049lb # [ 0.382874] pci_bus 0000:00: root bus resource [bus 00-ff]1050client # [ 0.384329] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1051worker2 # [ 0.370490] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1052worker1 # [ 0.375059] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1053lb # [ 0.383809] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1054worker2 # [ 0.371689] PCI host bridge to bus 0000:001055worker1 # [ 0.376059] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1056client # [ 0.385936] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1057worker2 # [ 0.372368] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1058worker1 # [ 0.377058] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1059lb # [ 0.385340] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1060worker2 # [ 0.373301] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1061worker1 # [ 0.378064] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1062worker2 # [ 0.374353] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1063client # [ 0.389172] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1064worker1 # [ 0.379059] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1065worker2 # [ 0.375480] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1066worker1 # [ 0.380084] pci_bus 0000:00: root bus resource [bus 00-ff]1067client # [ 0.390057] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1068worker2 # [ 0.376425] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1069client # [ 0.390890] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1070worker1 # [ 0.381019] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1071client # [ 0.391170] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1072worker2 # [ 0.377424] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1073worker1 # [ 0.382487] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1074worker2 # [ 0.378443] pci_bus 0000:00: root bus resource [bus 00-ff]1075client # [ 0.392717] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1076worker2 # [ 0.379295] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1077lb # [ 0.387949] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1078lb # [ 0.388886] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1079worker2 # [ 0.380969] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1080client # [ 0.395172] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1081lb # [ 0.389834] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1082client # [ 0.396030] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1083client # [ 0.396932] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1084lb # [ 0.391055] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1085client # [ 0.398581] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1086worker1 # [ 0.385177] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1087lb # [ 0.392838] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1088worker1 # [ 0.386071] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1089worker1 # [ 0.387014] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1090lb # [ 0.395894] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1091worker2 # [ 0.383583] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1092worker1 # [ 0.387972] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1093client # [ 0.401174] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1094lb # [ 0.396739] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1095worker2 # [ 0.384523] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1096client # [ 0.402171] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1097lb # [ 0.397599] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1098worker2 # [ 0.385495] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1099worker1 # [ 0.389858] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1100client # [ 0.403142] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1101lb # [ 0.397879] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1102worker2 # [ 0.386786] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1103client # [ 0.404523] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1104lb # [ 0.399803] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1105worker1 # [ 0.393077] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1106worker2 # [ 0.388434] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1107worker1 # [ 0.394026] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1108client # [ 0.406172] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1109lb # [ 0.401896] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1110client # [ 0.407017] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1111worker1 # [ 0.394915] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1112lb # [ 0.402816] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1113client # [ 0.407857] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1114worker1 # [ 0.396014] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1115worker2 # [ 0.391521] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1116lb # [ 0.403603] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1117worker2 # [ 0.392382] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1118worker2 # [ 0.393275] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1119client # [ 0.408728] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1120lb # [ 0.404444] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1121worker1 # [ 0.397973] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1122worker2 # [ 0.394341] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1123client # [ 0.411172] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1124worker2 # [ 0.396161] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1125worker1 # [ 0.400946] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1126client # [ 0.412018] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1127worker1 # [ 0.401768] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1128lb # [ 0.407514] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1129client # [ 0.412886] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1130lb # [ 0.407884] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1131worker1 # [ 0.402087] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1132worker2 # [ 0.397520] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1133lb # [ 0.408825] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1134client # [ 0.413779] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1135worker2 # [ 0.398344] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1136worker1 # [ 0.403634] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1137worker2 # [ 0.399217] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1138lb # [ 0.410302] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1139worker2 # [ 0.400949] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1140client # [ 0.416178] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1141lb # [ 0.411883] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1142client # [ 0.417196] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1143worker1 # [ 0.406069] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1144lb # [ 0.412739] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1145client # [ 0.418146] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1146worker1 # [ 0.406911] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1147lb # [ 0.413600] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1148worker1 # [ 0.407794] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1149client # [ 0.419770] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1150worker2 # [ 0.403752] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1151lb # [ 0.414469] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1152worker2 # [ 0.405146] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1153worker1 # [ 0.408613] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1154worker2 # [ 0.405534] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1155client # [ 0.421172] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1156client # [ 0.422032] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1157lb # [ 0.416882] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1158worker1 # [ 0.410078] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1159worker2 # [ 0.407077] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1160lb # [ 0.417734] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1161client # [ 0.422888] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1162worker1 # [ 0.410922] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1163client # [ 0.423171] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1164lb # [ 0.418605] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1165worker2 # [ 0.408520] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1166worker1 # [ 0.411785] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1167worker2 # [ 0.409403] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1168client # [ 0.425076] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1169lb # [ 0.420005] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1170worker2 # [ 0.410204] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1171worker1 # [ 0.412664] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1172client # [ 0.426977] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1173client # [ 0.427868] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1174worker2 # [ 0.411150] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1175worker1 # [ 0.414904] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1176lb # [ 0.423092] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1177client # [ 0.428742] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1178worker1 # [ 0.415803] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1179lb # [ 0.423881] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1180worker2 # [ 0.413520] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1181lb # [ 0.424823] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1182worker1 # [ 0.416087] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1183client # [ 0.430206] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1184worker2 # [ 0.414363] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1185client # [ 0.431037] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1186worker2 # [ 0.415244] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1187lb # [ 0.426234] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1188worker1 # [ 0.417695] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1189client # [ 0.431849] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1190worker2 # [ 0.416081] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1191lb # [ 0.427881] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1192client # [ 0.433711] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1193lb # [ 0.428739] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1194worker1 # [ 0.421079] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1195lb # [ 0.429635] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1196client # [ 0.435173] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1197worker1 # [ 0.421911] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1198worker2 # [ 0.418672] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1199lb # [ 0.429880] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1200client # [ 0.436004] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1201worker1 # [ 0.422821] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1202worker2 # [ 0.419518] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1203client # [ 0.436884] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1204worker2 # [ 0.420460] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1205lb # [ 0.431458] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1206worker1 # [ 0.423679] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1207client # [ 0.437716] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1208worker2 # [ 0.421888] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1209lb # [ 0.432886] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1210client # [ 0.438744] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1211worker1 # [ 0.425067] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1212lb # [ 0.433865] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1213worker1 # [ 0.425901] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1214client # [ 0.439551] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1215worker2 # [ 0.423520] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1216lb # [ 0.435433] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1217worker2 # [ 0.424372] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1218client # [ 0.440700] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1219worker1 # [ 0.426779] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1220worker2 # [ 0.425269] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1221worker1 # [ 0.427065] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1222lb # [ 0.436892] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1223worker2 # [ 0.425517] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1224worker1 # [ 0.428566] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1225lb # [ 0.437736] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1226worker2 # [ 0.427079] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1227lb # [ 0.438895] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1228worker1 # [ 0.430085] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1229worker1 # [ 0.431041] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1230lb # [ 0.440455] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1231worker2 # [ 0.428535] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1232worker2 # [ 0.429487] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1233worker1 # [ 0.432409] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1234lb # [ 0.441882] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1235lb # [ 0.442726] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1236worker2 # [ 0.430948] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1237lb # [ 0.443590] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1238worker1 # [ 0.435429] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1239worker2 # [ 0.432520] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1240client # [ 0.441360] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1241lb # [ 0.444471] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1242worker1 # [ 0.436690] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1243client # [ 0.443679] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1244worker2 # [ 0.433329] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1245worker1 # [ 0.437081] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1246worker2 # [ 0.434249] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1247lb # [ 0.446423] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1248client # [ 0.444410] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1249worker1 # [ 0.438625] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1250lb # [ 0.447092] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1251worker2 # [ 0.435071] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1252client # [ 0.445768] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1253lb # [ 0.448552] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1254client # [ 0.446422] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1255worker1 # [ 0.440950] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1256worker2 # [ 0.436531] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1257worker1 # [ 0.441676] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1258worker2 # [ 0.437389] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1259client # [ 0.447465] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1260worker1 # [ 0.442080] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1261worker2 # [ 0.438237] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1262client # [ 0.448458] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1263worker1 # [ 0.444115] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1264client # [ 0.450004] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1265worker2 # [ 0.439086] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1266client # [ 0.450806] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1267worker1 # [ 0.445907] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1268worker2 # [ 0.440409] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1269client # [ 0.451501] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1270worker1 # [ 0.446895] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1271worker2 # [ 0.441366] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1272client # [ 0.452751] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1273worker1 # [ 0.448797] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1274worker2 # [ 0.443174] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1275lb # [ 0.449134] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1276client # [ 0.456318] ACPI: PCI: Interrupt link LNKA configured for IRQ 101277lb # [ 0.451520] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1278client # [ 0.457278] ACPI: PCI: Interrupt link LNKB configured for IRQ 101279client # [ 0.458267] ACPI: PCI: Interrupt link LNKC configured for IRQ 111280lb # [ 0.452147] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1281client # [ 0.459264] ACPI: PCI: Interrupt link LNKD configured for IRQ 111282client # [ 0.460265] ACPI: PCI: Interrupt link LNKE configured for IRQ 101283lb # [ 0.453468] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1284client # [ 0.461262] ACPI: PCI: Interrupt link LNKF configured for IRQ 101285lb # [ 0.454839] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1286client # [ 0.462311] ACPI: PCI: Interrupt link LNKG configured for IRQ 111287lb # [ 0.456147] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1288client # [ 0.463299] ACPI: PCI: Interrupt link LNKH configured for IRQ 111289client # [ 0.464220] ACPI: PCI: Interrupt link GSIA configured for IRQ 161290lb # [ 0.457143] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1291client # [ 0.465153] ACPI: PCI: Interrupt link GSIB configured for IRQ 171292worker2 # [ 0.443760] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1293worker1 # [ 0.449267] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1294client # [ 0.465869] ACPI: PCI: Interrupt link GSIC configured for IRQ 181295lb # [ 0.458836] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1296client # [ 0.466179] ACPI: PCI: Interrupt link GSID configured for IRQ 191297worker2 # [ 0.446200] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1298worker1 # [ 0.451847] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1299lb # [ 0.459507] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1300client # [ 0.467109] ACPI: PCI: Interrupt link GSIE configured for IRQ 201301worker2 # [ 0.446774] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1302client # [ 0.467894] ACPI: PCI: Interrupt link GSIF configured for IRQ 211303lb # [ 0.460196] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1304worker1 # [ 0.453097] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1305client # [ 0.468177] ACPI: PCI: Interrupt link GSIG configured for IRQ 221306worker2 # [ 0.448160] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1307worker1 # [ 0.454720] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1308client # [ 0.469116] ACPI: PCI: Interrupt link GSIH configured for IRQ 231309lb # [ 0.461903] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1310worker2 # [ 0.448755] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1311worker1 # [ 0.455331] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1312client # [ 0.470965] iommu: Default domain type: Translated1313client # [ 0.471757] iommu: DMA domain TLB invalidation policy: lazy mode1314worker2 # [ 0.449806] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1315worker1 # [ 0.456384] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1316client # [ 0.472422] ACPI: bus type USB registered1317lb # [ 0.465794] ACPI: PCI: Interrupt link LNKA configured for IRQ 101318worker2 # [ 0.450775] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1319worker1 # [ 0.457334] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1320client # [ 0.473159] usbcore: registered new interface driver usbfs1321lb # [ 0.466700] ACPI: PCI: Interrupt link LNKB configured for IRQ 101322client # [ 0.474030] usbcore: registered new interface driver hub1323client # [ 0.474798] usbcore: registered new device driver usb1324worker2 # [ 0.452528] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1325lb # [ 0.467692] ACPI: PCI: Interrupt link LNKC configured for IRQ 111326worker1 # [ 0.459086] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1327worker2 # [ 0.453370] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1328lb # [ 0.468677] ACPI: PCI: Interrupt link LNKD configured for IRQ 111329worker1 # [ 0.460064] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1330client # [ 0.476103] NetLabel: Initializing1331lb # [ 0.469684] ACPI: PCI: Interrupt link LNKE configured for IRQ 101332client # [ 0.476590] NetLabel: domain hash size = 1281333worker1 # [ 0.461325] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1334worker2 # [ 0.454571] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1335client # [ 0.477165] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1336lb # [ 0.470695] ACPI: PCI: Interrupt link LNKF configured for IRQ 101337client # [ 0.478065] NetLabel: unlabeled traffic allowed by default1338worker1 # [ 0.462647] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1339lb # [ 0.471894] ACPI: PCI: Interrupt link LNKG configured for IRQ 111340client # [ 0.478821] PCI: Using ACPI for IRQ routing1341worker2 # [ 0.456265] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1342lb # [ 0.472948] ACPI: PCI: Interrupt link LNKH configured for IRQ 111343lb # [ 0.473910] ACPI: PCI: Interrupt link GSIA configured for IRQ 161344lb # [ 0.474866] ACPI: PCI: Interrupt link GSIB configured for IRQ 171345lb # [ 0.475830] ACPI: PCI: Interrupt link GSIC configured for IRQ 181346worker1 # [ 0.466190] ACPI: PCI: Interrupt link LNKA configured for IRQ 101347worker2 # [ 0.459615] ACPI: PCI: Interrupt link LNKA configured for IRQ 101348lb # [ 0.476619] ACPI: PCI: Interrupt link GSID configured for IRQ 191349worker1 # [ 0.467170] ACPI: PCI: Interrupt link LNKB configured for IRQ 101350worker2 # [ 0.460617] ACPI: PCI: Interrupt link LNKB configured for IRQ 101351lb # [ 0.476886] ACPI: PCI: Interrupt link GSIE configured for IRQ 201352worker1 # [ 0.468155] ACPI: PCI: Interrupt link LNKC configured for IRQ 111353worker2 # [ 0.461612] ACPI: PCI: Interrupt link LNKC configured for IRQ 111354lb # [ 0.477841] ACPI: PCI: Interrupt link GSIF configured for IRQ 211355worker1 # [ 0.469156] ACPI: PCI: Interrupt link LNKD configured for IRQ 111356worker2 # [ 0.462607] ACPI: PCI: Interrupt link LNKD configured for IRQ 111357lb # [ 0.478626] ACPI: PCI: Interrupt link GSIG configured for IRQ 221358worker1 # [ 0.470157] ACPI: PCI: Interrupt link LNKE configured for IRQ 101359worker2 # [ 0.463787] ACPI: PCI: Interrupt link LNKE configured for IRQ 101360lb # [ 0.478889] ACPI: PCI: Interrupt link GSIH configured for IRQ 231361worker1 # [ 0.471171] ACPI: PCI: Interrupt link LNKF configured for IRQ 101362worker2 # [ 0.464618] ACPI: PCI: Interrupt link LNKF configured for IRQ 101363worker1 # [ 0.472163] ACPI: PCI: Interrupt link LNKG configured for IRQ 111364lb # [ 0.480922] iommu: Default domain type: Translated1365worker2 # [ 0.465741] ACPI: PCI: Interrupt link LNKG configured for IRQ 111366worker1 # [ 0.473157] ACPI: PCI: Interrupt link LNKH configured for IRQ 111367lb # [ 0.481709] iommu: DMA domain TLB invalidation policy: lazy mode1368worker2 # [ 0.466640] ACPI: PCI: Interrupt link LNKH configured for IRQ 111369worker1 # [ 0.474097] ACPI: PCI: Interrupt link GSIA configured for IRQ 161370lb # [ 0.482859] ACPI: bus type USB registered1371worker2 # [ 0.467562] ACPI: PCI: Interrupt link GSIA configured for IRQ 161372worker1 # [ 0.475045] ACPI: PCI: Interrupt link GSIB configured for IRQ 171373lb # [ 0.483420] usbcore: registered new interface driver usbfs1374worker1 # [ 0.475789] ACPI: PCI: Interrupt link GSIC configured for IRQ 181375worker2 # [ 0.468527] ACPI: PCI: Interrupt link GSIB configured for IRQ 171376lb # [ 0.483893] usbcore: registered new interface driver hub1377worker1 # [ 0.476072] ACPI: PCI: Interrupt link GSID configured for IRQ 191378worker2 # [ 0.469475] ACPI: PCI: Interrupt link GSIC configured for IRQ 181379lb # [ 0.484756] usbcore: registered new device driver usb1380worker2 # [ 0.470238] ACPI: PCI: Interrupt link GSID configured for IRQ 191381worker1 # [ 0.477008] ACPI: PCI: Interrupt link GSIE configured for IRQ 201382lb # [ 0.486552] NetLabel: Initializing1383worker2 # [ 0.470524] ACPI: PCI: Interrupt link GSIE configured for IRQ 201384worker1 # [ 0.477828] ACPI: PCI: Interrupt link GSIF configured for IRQ 211385lb # [ 0.486874] NetLabel: domain hash size = 1281386worker1 # [ 0.478071] ACPI: PCI: Interrupt link GSIG configured for IRQ 221387worker2 # [ 0.471451] ACPI: PCI: Interrupt link GSIF configured for IRQ 211388lb # [ 0.487592] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1389worker1 # [ 0.478992] ACPI: PCI: Interrupt link GSIH configured for IRQ 231390worker2 # [ 0.472323] ACPI: PCI: Interrupt link GSIG configured for IRQ 221391lb # [ 0.487907] NetLabel: unlabeled traffic allowed by default1392worker2 # [ 0.473354] ACPI: PCI: Interrupt link GSIH configured for IRQ 231393lb # [ 0.488881] PCI: Using ACPI for IRQ routing1394worker1 # [ 0.481073] iommu: Default domain type: Translated1395worker2 # [ 0.475297] iommu: Default domain type: Translated1396worker1 # [ 0.481843] iommu: DMA domain TLB invalidation policy: lazy mode1397worker2 # [ 0.475518] iommu: DMA domain TLB invalidation policy: lazy mode1398worker1 # [ 0.483218] ACPI: bus type USB registered1399worker1 # [ 0.483944] usbcore: registered new interface driver usbfs1400worker1 # [ 0.484714] usbcore: registered new interface driver hub1401worker2 # [ 0.476681] ACPI: bus type USB registered1402worker1 # [ 0.485078] usbcore: registered new device driver usb1403worker2 # [ 0.477588] usbcore: registered new interface driver usbfs1404worker2 # [ 0.478492] usbcore: registered new interface driver hub1405worker1 # [ 0.486875] NetLabel: Initializing1406worker2 # [ 0.479165] usbcore: registered new device driver usb1407worker1 # [ 0.487491] NetLabel: domain hash size = 1281408worker2 # [ 0.480400] NetLabel: Initializing1409worker2 # [ 0.480934] NetLabel: domain hash size = 1281410worker1 # [ 0.488058] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1411worker2 # [ 0.481511] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1412worker1 # [ 0.489110] NetLabel: unlabeled traffic allowed by default1413worker1 # [ 0.490030] PCI: Using ACPI for IRQ routing1414worker2 # [ 0.482443] NetLabel: unlabeled traffic allowed by default1415worker2 # [ 0.483183] PCI: Using ACPI for IRQ routing1416client # [ 0.522819] pci 0000:00:01.0: vgaarb: setting as boot VGA device1417client # [ 0.523161] pci 0000:00:01.0: vgaarb: bridge control possible1418client # [ 0.523161] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1419client # [ 0.523169] vgaarb: loaded1420client # [ 0.523889] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01421client # [ 0.524166] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1422client # [ 0.527252] clocksource: Switched to clocksource kvm-clock1423client # [ 0.530844] VFS: Disk quotas dquot_6.6.01424client # [ 0.531605] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1425client # [ 0.532953] pnp: PnP ACPI init1426client # [ 0.533798] ACPI: IRQ 4 override to edge(!), high(!)1427client # [ 0.534871] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1428client # [ 0.536385] pnp: PnP ACPI: found 6 devices1429lb # [ 0.536335] pci 0000:00:01.0: vgaarb: setting as boot VGA device1430lb # [ 0.536870] pci 0000:00:01.0: vgaarb: bridge control possible1431lb # [ 0.536870] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1432lb # [ 0.536877] vgaarb: loaded1433client # [ 0.544082] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1434lb # [ 0.537604] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01435client # [ 0.545488] clocksource: Switched to clocksource acpi_pm1436lb # [ 0.537873] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1437client # [ 0.546399] NET: Registered PF_INET protocol family1438client # [ 0.547545] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1439worker2 # [ 0.526329] pci 0000:00:01.0: vgaarb: setting as boot VGA device1440worker1 # [ 0.534227] pci 0000:00:01.0: vgaarb: setting as boot VGA device1441worker2 # [ 0.526507] pci 0000:00:01.0: vgaarb: bridge control possible1442worker1 # [ 0.535054] pci 0000:00:01.0: vgaarb: bridge control possible1443lb # [ 0.541962] clocksource: Switched to clocksource kvm-clock1444worker2 # [ 0.526507] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1445worker1 # [ 0.535054] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1446worker2 # [ 0.526516] vgaarb: loaded1447worker1 # [ 0.535065] vgaarb: loaded1448worker2 # [ 0.527251] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01449worker1 # [ 0.535874] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01450worker2 # [ 0.527512] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1451worker1 # [ 0.536666] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1452lb # [ 0.545457] VFS: Disk quotas dquot_6.6.01453lb # [ 0.546169] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1454lb # [ 0.547518] pnp: PnP ACPI init1455lb # [ 0.548305] ACPI: IRQ 4 override to edge(!), high(!)1456worker1 # [ 0.540198] clocksource: Switched to clocksource kvm-clock1457lb # [ 0.549317] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1458worker2 # [ 0.531594] clocksource: Switched to clocksource kvm-clock1459lb # [ 0.550850] pnp: PnP ACPI: found 6 devices1460worker1 # [ 0.544150] VFS: Disk quotas dquot_6.6.01461worker2 # [ 0.535146] VFS: Disk quotas dquot_6.6.01462worker1 # [ 0.544866] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1463worker2 # [ 0.535851] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1464worker1 # [ 0.546272] pnp: PnP ACPI init1465worker2 # [ 0.537130] pnp: PnP ACPI init1466client # [ 0.564840] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1467worker1 # [ 0.547113] ACPI: IRQ 4 override to edge(!), high(!)1468worker2 # [ 0.537906] ACPI: IRQ 4 override to edge(!), high(!)1469worker2 # [ 0.538817] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1470client # [ 0.566333] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1471worker1 # [ 0.548126] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1472worker2 # [ 0.540312] pnp: PnP ACPI: found 6 devices1473worker1 # [ 0.549656] pnp: PnP ACPI: found 6 devices1474client # [ 0.567610] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1475lb # [ 0.558468] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1476lb # [ 0.559867] clocksource: Switched to clocksource acpi_pm1477client # [ 0.568898] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1478lb # [ 0.560812] NET: Registered PF_INET protocol family1479client # [ 0.570103] TCP: Hash tables configured (established 16384 bind 16384)1480lb # [ 0.561951] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1481client # [ 0.571187] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1482client # [ 0.572483] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1483client # [ 0.573529] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1484client # [ 0.574709] NET: Registered PF_UNIX/PF_LOCAL protocol family1485client # [ 0.575642] NET: Registered PF_XDP protocol family1486worker2 # [ 0.547900] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1487worker1 # [ 0.557436] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1488client # [ 0.576466] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1489worker2 # [ 0.549289] clocksource: Switched to clocksource acpi_pm1490worker1 # [ 0.558844] clocksource: Switched to clocksource acpi_pm1491client # [ 0.577410] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1492worker2 # [ 0.550223] NET: Registered PF_INET protocol family1493worker1 # [ 0.559856] NET: Registered PF_INET protocol family1494client # [ 0.578373] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1495worker2 # [ 0.551368] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1496client # [ 0.579462] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1497worker1 # [ 0.561064] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1498client # [ 0.580534] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1499client # [ 0.581595] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1500client # [ 0.583251] ACPI: \_SB_.GSIA: Enabled at IRQ 161501client # [ 0.585504] ACPI: \_SB_.GSIB: Enabled at IRQ 171502client # [ 0.587604] ACPI: \_SB_.GSIC: Enabled at IRQ 181503lb # [ 0.578644] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1504client # [ 0.589557] ACPI: \_SB_.GSID: Enabled at IRQ 191505lb # [ 0.580075] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1506lb # [ 0.581420] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1507client # [ 0.591189] PCI: CLS 0 bytes, default 641508lb # [ 0.582761] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1509client # [ 0.592073] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1510lb # [ 0.584014] TCP: Hash tables configured (established 16384 bind 16384)1511client # [ 0.593892] Trying to unpack rootfs image as initramfs...1512lb # [ 0.585178] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1513lb # [ 0.586510] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1514worker2 # [ 0.568051] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1515lb # [ 0.587603] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1516worker2 # [ 0.569683] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1517lb # [ 0.588820] NET: Registered PF_UNIX/PF_LOCAL protocol family1518worker1 # [ 0.579137] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1519lb # [ 0.589791] NET: Registered PF_XDP protocol family1520worker2 # [ 0.570989] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1521worker1 # [ 0.580740] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1522lb # [ 0.590648] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1523worker2 # [ 0.572317] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1524worker1 # [ 0.582068] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1525lb # [ 0.591691] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1526worker2 # [ 0.573572] TCP: Hash tables configured (established 16384 bind 16384)1527lb # [ 0.592739] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1528worker1 # [ 0.583421] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1529worker2 # [ 0.574699] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1530worker1 # [ 0.584743] TCP: Hash tables configured (established 16384 bind 16384)1531lb # [ 0.593945] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1532worker2 # [ 0.575984] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1533lb # [ 0.595098] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1534worker1 # [ 0.585923] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1535worker2 # [ 0.577077] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1536lb # [ 0.596282] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1537worker1 # [ 0.587246] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1538worker2 # [ 0.578274] NET: Registered PF_UNIX/PF_LOCAL protocol family1539worker2 # [ 0.579240] NET: Registered PF_XDP protocol family1540worker1 # [ 0.588403] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1541lb # [ 0.598011] ACPI: \_SB_.GSIA: Enabled at IRQ 161542worker2 # [ 0.580076] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1543worker1 # [ 0.589597] NET: Registered PF_UNIX/PF_LOCAL protocol family1544worker2 # [ 0.581115] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1545worker1 # [ 0.590609] NET: Registered PF_XDP protocol family1546lb # [ 0.600473] ACPI: \_SB_.GSIB: Enabled at IRQ 171547worker2 # [ 0.582076] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1548worker1 # [ 0.591449] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1549worker1 # [ 0.592417] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1550worker2 # [ 0.583132] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1551lb # [ 0.602520] ACPI: \_SB_.GSIC: Enabled at IRQ 181552worker1 # [ 0.593383] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1553worker2 # [ 0.584191] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1554worker1 # [ 0.594429] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1555worker2 # [ 0.585251] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1556lb # [ 0.604479] ACPI: \_SB_.GSID: Enabled at IRQ 191557worker1 # [ 0.595480] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1558worker2 # [ 0.586917] ACPI: \_SB_.GSIA: Enabled at IRQ 161559lb # [ 0.606107] PCI: CLS 0 bytes, default 641560worker1 # [ 0.596568] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1561worker2 # [ 0.589030] ACPI: \_SB_.GSIB: Enabled at IRQ 171562worker1 # [ 0.598400] ACPI: \_SB_.GSIA: Enabled at IRQ 161563lb # [ 0.607000] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1564lb # [ 0.608696] Trying to unpack rootfs image as initramfs...1565worker2 # [ 0.590989] ACPI: \_SB_.GSIC: Enabled at IRQ 181566worker1 # [ 0.601107] ACPI: \_SB_.GSIB: Enabled at IRQ 171567worker2 # [ 0.592949] ACPI: \_SB_.GSID: Enabled at IRQ 191568worker1 # [ 0.603067] ACPI: \_SB_.GSIC: Enabled at IRQ 181569worker2 # [ 0.594669] PCI: CLS 0 bytes, default 641570worker1 # [ 0.605052] ACPI: \_SB_.GSID: Enabled at IRQ 191571worker2 # [ 0.595586] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1572worker2 # [ 0.597278] Trying to unpack rootfs image as initramfs...1573worker1 # [ 0.606696] PCI: CLS 0 bytes, default 641574worker1 # [ 0.607601] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1575worker1 # [ 0.609332] Trying to unpack rootfs image as initramfs...1576client # [ 0.639905] Initialise system trusted keyrings1577client # [ 0.643520] workingset: timestamp_bits=40 max_order=19 bucket_order=01578client # [ 0.662960] Key type asymmetric registered1579lb # [ 0.655954] Initialise system trusted keyrings1580lb # [ 0.656963] workingset: timestamp_bits=40 max_order=19 bucket_order=01581client # [ 0.668496] Asymmetric key parser 'x509' registered1582client # [ 0.669348] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1583client # [ 0.670732] io scheduler mq-deadline registered1584worker2 # [ 0.645296] Initialise system trusted keyrings1585client # [ 0.673494] io scheduler kyber registered1586worker2 # [ 0.646257] workingset: timestamp_bits=40 max_order=19 bucket_order=01587worker1 # [ 0.656438] Initialise system trusted keyrings1588worker1 # [ 0.657417] workingset: timestamp_bits=40 max_order=19 bucket_order=01589client # [ 0.676611] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1590client # [ 0.677858] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1591client # [ 0.682891] Linux agpgart interface v0.1031592client # [ 0.683660] ACPI: bus type drm_connector registered1593client # [ 0.687243] usbcore: registered new interface driver usbserial_generic1594client # [ 0.688271] usbserial: USB Serial support registered for generic1595lb # [ 0.680485] Key type asymmetric registered1596lb # [ 0.681233] Asymmetric key parser 'x509' registered1597lb # [ 0.682115] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1598client # [ 0.691474] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1599client # [ 0.691474] Please enable it if your BIOS has the CPPC option.1600client # [ 0.693616] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1601lb # [ 0.685596] io scheduler mq-deadline registered1602lb # [ 0.686350] io scheduler kyber registered1603worker2 # [ 0.667963] Key type asymmetric registered1604worker2 # [ 0.668656] Asymmetric key parser 'x509' registered1605worker2 # [ 0.669519] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1606client # [ 0.697622] drop_monitor: Initializing network drop monitor service1607client # [ 0.698757] NET: Registered PF_INET6 protocol family1608worker1 # [ 0.681129] Key type asymmetric registered1609lb # [ 0.690624] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1610worker1 # [ 0.681882] Asymmetric key parser 'x509' registered1611client # [ 0.701558] Segment Routing with IPv61612lb # [ 0.691892] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1613worker1 # [ 0.682725] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1614client # [ 0.702195] In-situ OAM (IOAM) with IPv61615worker2 # [ 0.675049] io scheduler mq-deadline registered1616worker2 # [ 0.675806] io scheduler kyber registered1617client # [ 0.704749] IPI shorthand broadcast: enabled1618worker2 # [ 0.678459] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1619worker1 # [ 0.688027] io scheduler mq-deadline registered1620lb # [ 0.697489] Linux agpgart interface v0.1031621worker1 # [ 0.688758] io scheduler kyber registered1622lb # [ 0.698237] ACPI: bus type drm_connector registered1623worker2 # [ 0.679731] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1624lb # [ 0.700562] usbcore: registered new interface driver usbserial_generic1625worker1 # [ 0.691599] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1626lb # [ 0.701611] usbserial: USB Serial support registered for generic1627client # [ 0.712150] sched_clock: Marking stable (578015010, 133442623)->(785214114, -73756481)1628worker2 # [ 0.685704] Linux agpgart interface v0.1031629worker1 # [ 0.694899] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1630worker2 # [ 0.686514] ACPI: bus type drm_connector registered1631lb # [ 0.704444] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1632lb # [ 0.704444] Please enable it if your BIOS has the CPPC option.1633client # [ 0.715702] registered taskstats version 11634lb # [ 0.706667] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1635worker1 # [ 0.698503] Linux agpgart interface v0.1031636client # [ 0.716807] Loading compiled-in X.509 certificates1637worker2 # [ 0.689403] usbcore: registered new interface driver usbserial_generic1638worker1 # [ 0.699265] ACPI: bus type drm_connector registered1639worker2 # [ 0.690449] usbserial: USB Serial support registered for generic1640lb # [ 0.710586] drop_monitor: Initializing network drop monitor service1641lb # [ 0.711726] NET: Registered PF_INET6 protocol family1642worker1 # [ 0.702332] usbcore: registered new interface driver usbserial_generic1643worker1 # [ 0.703447] usbserial: USB Serial support registered for generic1644worker2 # [ 0.693867] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1645worker2 # [ 0.693867] Please enable it if your BIOS has the CPPC option.1646worker2 # [ 0.696073] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1647lb # [ 0.716523] Segment Routing with IPv61648worker1 # [ 0.706833] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1649lb # [ 0.717174] In-situ OAM (IOAM) with IPv61650worker1 # [ 0.706833] Please enable it if your BIOS has the CPPC option.1651lb # [ 0.718255] IPI shorthand broadcast: enabled1652worker2 # [ 0.700075] drop_monitor: Initializing network drop monitor service1653worker1 # [ 0.709076] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1654worker2 # [ 0.701277] NET: Registered PF_INET6 protocol family1655worker1 # [ 0.713017] drop_monitor: Initializing network drop monitor service1656worker1 # [ 0.714238] NET: Registered PF_INET6 protocol family1657worker2 # [ 0.705303] Segment Routing with IPv61658worker2 # [ 0.707885] In-situ OAM (IOAM) with IPv61659client # [ 0.735482] Demotion targets for Node 0: null1660lb # [ 0.726540] sched_clock: Marking stable (590013257, 136395308)->(798385468, -71976903)1661worker2 # [ 0.708967] IPI shorthand broadcast: enabled1662worker1 # [ 0.718380] Segment Routing with IPv61663worker1 # [ 0.719094] In-situ OAM (IOAM) with IPv61664client # [ 0.737521] Key type .fscrypt registered1665client # [ 0.738156] Key type fscrypt-provisioning registered1666client # [ 0.739097] ima: No TPM chip found, activating TPM-bypass!1667worker1 # [ 0.722161] IPI shorthand broadcast: enabled1668lb # [ 0.731503] registered taskstats version 11669lb # [ 0.732504] Loading compiled-in X.509 certificates1670client # [ 0.743482] ima: Allocated hash algorithm: sha11671client # [ 0.744277] ima: No architecture policies found1672worker2 # [ 0.716342] sched_clock: Marking stable (583014596, 132822936)->(793316703, -77479171)1673client # [ 0.746483] PM: Magic number: 10:928:5841674client # [ 0.748184] RAS: Correctable Errors collector initialized.1675worker1 # [ 0.730533] sched_clock: Marking stable (596015180, 133791531)->(800250598, -70443887)1676worker2 # [ 0.723095] registered taskstats version 11677worker2 # [ 0.724031] Loading compiled-in X.509 certificates1678worker1 # [ 0.734087] registered taskstats version 11679worker1 # [ 0.735207] Loading compiled-in X.509 certificates1680client # [ 0.758421] clk: Disabling unused clocks1681client # [ 0.760473] PM: genpd: Disabling unused power domains1682lb # [ 0.751425] Demotion targets for Node 0: null1683lb # [ 0.752417] Key type .fscrypt registered1684lb # [ 0.753152] Key type fscrypt-provisioning registered1685lb # [ 0.754180] ima: No TPM chip found, activating TPM-bypass!1686lb # [ 0.757423] ima: Allocated hash algorithm: sha11687lb # [ 0.758274] ima: No architecture policies found1688worker2 # [ 0.742905] Demotion targets for Node 0: null1689worker2 # [ 0.743886] Key type .fscrypt registered1690lb # [ 0.762429] PM: Magic number: 10:928:5841691worker2 # [ 0.744561] Key type fscrypt-provisioning registered1692worker1 # [ 0.754822] Demotion targets for Node 0: null1693worker2 # [ 0.745554] ima: No TPM chip found, activating TPM-bypass!1694lb # [ 0.764100] RAS: Correctable Errors collector initialized.1695worker1 # [ 0.755851] Key type .fscrypt registered1696worker1 # [ 0.756566] Key type fscrypt-provisioning registered1697worker1 # [ 0.757544] ima: No TPM chip found, activating TPM-bypass!1698worker2 # [ 0.748858] ima: Allocated hash algorithm: sha11699worker2 # [ 0.749644] ima: No architecture policies found1700worker1 # [ 0.760829] ima: Allocated hash algorithm: sha11701worker1 # [ 0.761673] ima: No architecture policies found1702worker2 # [ 0.753864] PM: Magic number: 10:928:5841703lb # [ 0.773122] clk: Disabling unused clocks1704worker2 # [ 0.755530] RAS: Correctable Errors collector initialized.1705worker1 # [ 0.765828] PM: Magic number: 10:928:5841706lb # [ 0.775427] PM: genpd: Disabling unused power domains1707worker1 # [ 0.767691] RAS: Correctable Errors collector initialized.1708worker2 # [ 0.763584] clk: Disabling unused clocks1709worker1 # [ 0.776577] clk: Disabling unused clocks1710worker2 # [ 0.767870] PM: genpd: Disabling unused power domains1711worker1 # [ 0.778821] PM: genpd: Disabling unused power domains1712client # [ 0.930235] Freeing initrd memory: 29096K1713client # [ 0.933460] Freeing unused decrypted memory: 2028K1714client # [ 0.936135] Freeing unused kernel image (initmem) memory: 3644K1715client # [ 0.937349] Write protecting the kernel read-only data: 32768k1716client # [ 0.939313] Freeing unused kernel image (text/rodata gap) memory: 1208K1717client # [ 0.940869] Freeing unused kernel image (rodata/data gap) memory: 752K1718lb # [ 0.944508] Freeing initrd memory: 29092K1719lb # [ 0.947658] Freeing unused decrypted memory: 2028K1720lb # [ 0.950295] Freeing unused kernel image (initmem) memory: 3644K1721lb # [ 0.951382] Write protecting the kernel read-only data: 32768k1722lb # [ 0.953298] Freeing unused kernel image (text/rodata gap) memory: 1208K1723lb # [ 0.954819] Freeing unused kernel image (rodata/data gap) memory: 752K1724worker2 # [ 0.940461] Freeing initrd memory: 29092K1725worker1 # [ 0.952867] Freeing initrd memory: 29092K1726worker2 # [ 0.943743] Freeing unused decrypted memory: 2028K1727worker2 # [ 0.946392] Freeing unused kernel image (initmem) memory: 3644K1728worker1 # [ 0.956292] Freeing unused decrypted memory: 2028K1729worker2 # [ 0.947538] Write protecting the kernel read-only data: 32768k1730worker2 # [ 0.949464] Freeing unused kernel image (text/rodata gap) memory: 1208K1731worker1 # [ 0.959222] Freeing unused kernel image (initmem) memory: 3644K1732worker2 # [ 0.951010] Freeing unused kernel image (rodata/data gap) memory: 752K1733worker1 # [ 0.960350] Write protecting the kernel read-only data: 32768k1734worker1 # [ 0.962349] Freeing unused kernel image (text/rodata gap) memory: 1208K1735worker1 # [ 0.963954] Freeing unused kernel image (rodata/data gap) memory: 752K1736client # [ 0.992263] x86/mm: Checked W+X mappings: passed, no W+X pages found.1737client # [ 0.993470] Run /init as init process1738client # [ 1.003934] systemd[1]: Inserted module 'autofs4'1739lb # [ 1.007013] x86/mm: Checked W+X mappings: passed, no W+X pages found.1740lb # [ 1.008229] Run /init as init process1741client # [ 1.020606] fuse: init (API version 7.45)1742client # [ 1.026405] ACPI: \_SB_.GSIG: Enabled at IRQ 221743client # [ 1.029140] ACPI: \_SB_.GSIH: Enabled at IRQ 231744lb # [ 1.020752] systemd[1]: Inserted module 'autofs4'1745worker2 # [ 1.002710] x86/mm: Checked W+X mappings: passed, no W+X pages found.1746worker2 # [ 1.003806] Run /init as init process1747client # [ 1.032144] ACPI: \_SB_.GSIE: Enabled at IRQ 201748worker1 # [ 1.015296] x86/mm: Checked W+X mappings: passed, no W+X pages found.1749client # [ 1.034544] ACPI: \_SB_.GSIF: Enabled at IRQ 211750worker1 # [ 1.016452] Run /init as init process1751worker2 # [ 1.015389] systemd[1]: Inserted module 'autofs4'1752worker1 # [ 1.027467] systemd[1]: Inserted module 'autofs4'1753lb # [ 1.038017] fuse: init (API version 7.45)1754lb # [ 1.044175] ACPI: \_SB_.GSIG: Enabled at IRQ 221755lb # [ 1.047024] ACPI: \_SB_.GSIH: Enabled at IRQ 231756worker2 # [ 1.031903] fuse: init (API version 7.45)1757lb # [ 1.050465] ACPI: \_SB_.GSIE: Enabled at IRQ 201758lb # [ 1.053062] ACPI: \_SB_.GSIF: Enabled at IRQ 211759worker1 # [ 1.044538] fuse: init (API version 7.45)1760worker2 # [ 1.037730] ACPI: \_SB_.GSIG: Enabled at IRQ 221761client # [ 1.067087] systemd[1]: Successfully made /usr/ read-only.1762worker2 # [ 1.040234] ACPI: \_SB_.GSIH: Enabled at IRQ 231763worker1 # [ 1.050516] ACPI: \_SB_.GSIG: Enabled at IRQ 221764worker2 # [ 1.043128] ACPI: \_SB_.GSIE: Enabled at IRQ 201765worker1 # [ 1.053022] ACPI: \_SB_.GSIH: Enabled at IRQ 231766worker2 # [ 1.045338] ACPI: \_SB_.GSIF: Enabled at IRQ 211767worker1 # [ 1.055888] ACPI: \_SB_.GSIE: Enabled at IRQ 201768worker1 # [ 1.058067] ACPI: \_SB_.GSIF: Enabled at IRQ 211769lb # [ 1.082778] systemd[1]: Successfully made /usr/ read-only.1770worker2 # [ 1.072810] systemd[1]: Successfully made /usr/ read-only.1771worker1 # [ 1.090851] systemd[1]: Successfully made /usr/ read-only.1772client # [ 1.403432] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1773client # [ 1.409973] systemd[1]: Detected virtualization kvm.1774client # [ 1.411099] systemd[1]: Detected architecture x86-64.1775client # [ 1.412231] systemd[1]: Running in initrd.1776client # [ 1.413726] systemd[1]: Initializing machine ID from random generator.1777client # [ 1.415262] systemd[1]: Hostname set to <client>.1778lb # [ 1.419854] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1779lb # [ 1.432056] systemd[1]: Detected virtualization kvm.1780lb # [ 1.434271] systemd[1]: Detected architecture x86-64.1781lb # [ 1.436452] systemd[1]: Running in initrd.1782worker2 # [ 1.409450] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1783lb # [ 1.439076] systemd[1]: Initializing machine ID from random generator.1784worker2 # [ 1.421573] systemd[1]: Detected virtualization kvm.1785lb # [ 1.442023] systemd[1]: Hostname set to <lb>.1786worker2 # [ 1.423737] systemd[1]: Detected architecture x86-64.1787worker2 # [ 1.425970] systemd[1]: Running in initrd.1788worker1 # [ 1.427487] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1789worker2 # [ 1.428557] systemd[1]: Initializing machine ID from random generator.1790worker1 # [ 1.439432] systemd[1]: Detected virtualization kvm.1791worker2 # [ 1.431484] systemd[1]: Hostname set to <worker2>.1792worker1 # [ 1.441627] systemd[1]: Detected architecture x86-64.1793worker1 # [ 1.443772] systemd[1]: Running in initrd.1794worker1 # [ 1.446459] systemd[1]: Initializing machine ID from random generator.1795worker1 # [ 1.449380] systemd[1]: Hostname set to <worker1>.1796client # [ 1.620832] systemd[1]: bpf-restrict-fs: LSM BPF program attached1797lb # [ 1.649373] systemd[1]: bpf-restrict-fs: LSM BPF program attached1798client # [ 1.659696] systemd[1]: Queued start job for default target Initrd Default Target.1799client # [ 1.663747] systemd[1]: Created slice Slice /system/modprobe.1800client # [ 1.664958] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1801client # [ 1.666401] systemd[1]: Expecting device /dev/disk/by-label/nixos...1802client # [ 1.667642] systemd[1]: Reached target Path Units.1803client # [ 1.668563] systemd[1]: Reached target Slice Units.1804client # [ 1.669496] systemd[1]: Reached target Swaps.1805client # [ 1.670285] systemd[1]: Reached target Timer Units.1806client # [ 1.671271] systemd[1]: Listening on D-Bus System Message Bus Socket.1807client # [ 1.672576] systemd[1]: Listening on Journal Socket (/dev/log).1808client # [ 1.673771] systemd[1]: Listening on Journal Sockets.1809client # [ 1.674844] systemd[1]: Listening on udev Control Socket.1810worker2 # [ 1.647277] systemd[1]: bpf-restrict-fs: LSM BPF program attached1811client # [ 1.675892] systemd[1]: Listening on udev Kernel Socket.1812client # [ 1.676898] systemd[1]: Reached target Socket Units.1813client # [ 1.678799] systemd[1]: Starting Create List of Static Device Nodes...1814client # [ 1.682529] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1815worker1 # [ 1.667211] systemd[1]: bpf-restrict-fs: LSM BPF program attached1816client # [ 1.691653] systemd[1]: Starting Load Kernel Module configfs...1817lb # [ 1.690550] systemd[1]: Queued start job for default target Initrd Default Target.1818lb # [ 1.694894] systemd[1]: Created slice Slice /system/modprobe.1819client # [ 1.704263] systemd[1]: Starting Journal Service...1820lb # [ 1.696186] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1821lb # [ 1.697706] systemd[1]: Expecting device /dev/disk/by-label/nixos...1822lb # [ 1.698891] systemd[1]: Reached target Path Units.1823lb # [ 1.699835] systemd[1]: Reached target Slice Units.1824lb # [ 1.700788] systemd[1]: Reached target Swaps.1825lb # [ 1.701720] systemd[1]: Reached target Timer Units.1826lb # [ 1.702837] systemd[1]: Listening on D-Bus System Message Bus Socket.1827lb # [ 1.704144] systemd[1]: Listening on Journal Socket (/dev/log).1828lb # [ 1.705369] systemd[1]: Listening on Journal Sockets.1829lb # [ 1.706483] systemd[1]: Listening on udev Control Socket.1830lb # [ 1.707539] systemd[1]: Listening on udev Kernel Socket.1831lb # [ 1.708533] systemd[1]: Reached target Socket Units.1832client # [ 1.718565] systemd[1]: Starting Load Kernel Modules...1833lb # [ 1.710482] systemd[1]: Starting Create List of Static Device Nodes...1834worker2 # [ 1.695795] systemd[1]: Queued start job for default target Initrd Default Target.1835lb # [ 1.716500] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1836worker2 # [ 1.700529] systemd[1]: Created slice Slice /system/modprobe.1837worker2 # [ 1.702054] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1838lb # [ 1.722185] systemd[1]: Starting Load Kernel Module configfs...1839worker2 # [ 1.703862] systemd[1]: Expecting device /dev/disk/by-label/nixos...1840worker2 # [ 1.705235] systemd[1]: Reached target Path Units.1841worker2 # [ 1.706320] systemd[1]: Reached target Slice Units.1842worker2 # [ 1.707430] systemd[1]: Reached target Swaps.1843worker2 # [ 1.708435] systemd[1]: Reached target Timer Units.1844worker2 # [ 1.709661] systemd[1]: Listening on D-Bus System Message Bus Socket.1845client # [ 1.726536] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1846worker2 # [ 1.711172] systemd[1]: Listening on Journal Socket (/dev/log).1847worker2 # [ 1.712650] systemd[1]: Listening on Journal Sockets.1848worker1 # [ 1.721910] systemd[1]: Queued start job for default target Initrd Default Target.1849worker2 # [ 1.714049] systemd[1]: Listening on udev Control Socket.1850client # [ 1.742092] netfs: FS-Cache loaded1851worker2 # [ 1.715349] systemd[1]: Listening on udev Kernel Socket.1852worker2 # [ 1.716566] systemd[1]: Reached target Socket Units.1853worker1 # [ 1.726249] systemd[1]: Created slice Slice /system/modprobe.1854worker1 # [ 1.727469] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1855worker2 # [ 1.718916] systemd[1]: Starting Create List of Static Device Nodes...1856client # [ 1.747017] 9pnet: Installing 9P2000 support1857worker1 # [ 1.728924] systemd[1]: Expecting device /dev/disk/by-label/nixos...1858worker1 # [ 1.730066] systemd[1]: Reached target Path Units.1859worker1 # [ 1.731042] systemd[1]: Reached target Slice Units.1860worker1 # [ 1.731975] systemd[1]: Reached target Swaps.1861lb # [ 1.741512] systemd[1]: Starting Journal Service...1862worker1 # [ 1.732764] systemd[1]: Reached target Timer Units.1863worker1 # [ 1.733815] systemd[1]: Listening on D-Bus System Message Bus Socket.1864worker2 # [ 1.724933] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1865worker1 # [ 1.735095] systemd[1]: Listening on Journal Socket (/dev/log).1866worker1 # [ 1.736356] systemd[1]: Listening on Journal Sockets.1867worker1 # [ 1.737392] systemd[1]: Listening on udev Control Socket.1868client # [ 1.756359] systemd[1]: Starting Coldplug All udev Devices...1869worker1 # [ 1.738427] systemd[1]: Listening on udev Kernel Socket.1870worker1 # [ 1.739363] systemd[1]: Reached target Socket Units.1871worker1 # [ 1.741338] systemd[1]: Starting Create List of Static Device Nodes...1872worker2 # [ 1.734025] systemd[1]: Starting Load Kernel Module configfs...1873worker1 # [ 1.745878] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1874lb # [ 1.756553] systemd[1]: Starting Load Kernel Modules...1875worker1 # [ 1.752882] systemd[1]: Starting Load Kernel Module configfs...1876client # [ 1.772615] systemd[1]: Finished Create List of Static Device Nodes.1877worker2 # [ 1.751493] systemd[1]: Starting Journal Service...1878client # [ 1.779051] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1879lb # [ 1.760515] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1880worker1 # [ 1.768007] systemd[1]: Starting Journal Service...1881lb # [ 1.777584] netfs: FS-Cache loaded1882client # [ 1.786872] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1883lb # [ 1.782716] 9pnet: Installing 9P2000 support1884client # [ 1.791979] systemd-journald[67]: Collecting audit messages is disabled.1885worker2 # [ 1.764928] systemd[1]: Starting Load Kernel Modules...1886client # [ 1.794870] systemd[1]: modprobe@configfs.service: Deactivated successfully.1887worker1 # [ 1.781917] systemd[1]: Starting Load Kernel Modules...1888client # [ 1.800564] systemd[1]: Finished Load Kernel Module configfs.1889lb # [ 1.794638] systemd[1]: Starting Coldplug All udev Devices...1890client # [ 1.803801] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1891client # [ 1.807953] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1892worker2 # [ 1.771922] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1893worker2 # [ 1.787953] netfs: FS-Cache loaded1894client # [ 1.815231] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1895client # [ 1.819531] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1896worker2 # [ 1.792877] 9pnet: Installing 9P2000 support1897worker1 # [ 1.789900] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1898lb # [ 1.811495] systemd[1]: Finished Create List of Static Device Nodes.1899worker1 # [ 1.805659] netfs: FS-Cache loaded1900lb # [ 1.815094] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1901worker1 # [ 1.810757] 9pnet: Installing 9P2000 support1902lb # [ 1.822863] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1903worker2 # [ 1.805140] systemd[1]: Starting Coldplug All udev Devices...1904lb # [ 1.827362] systemd-journald[67]: Collecting audit messages is disabled.1905worker1 # [ 1.819890] systemd[1]: Starting Coldplug All udev Devices...1906lb # [ 1.829842] systemd[1]: modprobe@configfs.service: Deactivated successfully.1907client # [ 1.845472] systemd[1]: Finished Load Kernel Modules.1908lb # [ 1.836808] systemd[1]: Finished Load Kernel Module configfs.1909worker2 # [ 1.821304] systemd[1]: Finished Create List of Static Device Nodes.1910lb # [ 1.839973] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1911client # [ 1.853546] systemd[1]: Starting Apply Kernel Variables...1912worker1 # [ 1.836019] systemd[1]: Finished Create List of Static Device Nodes.1913lb # [ 1.845124] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1914worker2 # [ 1.829998] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1915worker1 # [ 1.842496] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1916lb # [ 1.852630] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1917worker2 # [ 1.835916] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1918lb # [ 1.854617] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1919client # [ 1.865567] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1920worker2 # [ 1.841531] systemd[1]: modprobe@configfs.service: Deactivated successfully.1921worker1 # [ 1.851286] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1922worker2 # [ 1.845320] systemd-journald[67]: Collecting audit messages is disabled.1923worker1 # [ 1.855533] systemd-journald[67]: Collecting audit messages is disabled.1924client # [ 1.874880] systemd[1]: Starting Create Static Device Nodes in /dev...1925worker2 # [ 1.848284] systemd[1]: Finished Load Kernel Module configfs.1926worker1 # [ 1.858296] systemd[1]: modprobe@configfs.service: Deactivated successfully.1927worker2 # [ 1.854406] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1928worker1 # [ 1.865265] systemd[1]: Finished Load Kernel Module configfs.1929worker2 # [ 1.859520] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1930worker1 # [ 1.869735] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1931worker1 # [ 1.872340] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1932lb # [ 1.885023] systemd[1]: Finished Load Kernel Modules.1933client # [ 1.895567] systemd[1]: Finished Apply Kernel Variables.1934worker2 # [ 1.868089] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1935worker2 # [ 1.870041] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1936worker1 # [ 1.882003] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1937lb # [ 1.892633] systemd[1]: Starting Apply Kernel Variables...1938worker1 # [ 1.884000] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1939client # [ 1.908373] systemd[1]: Started Journal Service.1940client # [ 1.777209] systemd-modules-load[68]: Inserted module 'dm_mod'1941client # [ 1.778575] systemd-modules-load[68]: Inserted module 'virtio_balloon'1942client # [ 1.782365] systemd-modules-load[68]: Inserted module 'virtio_gpu'1943lb # [ 1.906490] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1944client # [ 1.783377] systemd[1]: Finished Create Static Device Nodes in /dev.1945client # [ 1.784491] systemd[1]: Reached target Preparation for Local File Systems.1946client # [ 1.786315] systemd[1]: Reached target Local File Systems.1947client # [ 1.789096] systemd[1]: Starting Create System Files and Directories...1948lb # [ 1.915612] systemd[1]: Starting Create Static Device Nodes in /dev...1949client # [ 1.794070] systemd[1]: Starting Rule-based Manager for Device Events and Files...1950worker2 # [ 1.903466] systemd[1]: Finished Load Kernel Modules.1951worker1 # [ 1.916414] systemd[1]: Finished Load Kernel Modules.1952lb # [ 1.930576] systemd[1]: Finished Apply Kernel Variables.1953worker2 # [ 1.912058] systemd[1]: Starting Apply Kernel Variables...1954worker1 # [ 1.925053] systemd[1]: Starting Apply Kernel Variables...1955worker2 # [ 1.923471] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1956lb # [ 1.948063] systemd[1]: Started Journal Service.1957worker1 # [ 1.939310] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1958lb # [ 1.814216] systemd-modules-load[68]: Inserted module 'dm_mod'1959lb # [ 1.816155] systemd-modules-load[68]: Inserted module 'virtio_balloon'1960client # [ 1.828454] systemd[1]: Finished Create System Files and Directories.1961lb # [ 1.818323] systemd-modules-load[68]: Inserted module 'virtio_gpu'1962worker2 # [ 1.936350] systemd[1]: Starting Create Static Device Nodes in /dev...1963lb # [ 1.819315] systemd[1]: Finished Create Static Device Nodes in /dev.1964lb # [ 1.821361] systemd[1]: Reached target Preparation for Local File Systems.1965lb # [ 1.823329] systemd[1]: Reached target Local File Systems.1966worker1 # [ 1.950051] systemd[1]: Starting Create Static Device Nodes in /dev...1967lb # [ 1.826151] systemd[1]: Starting Create System Files and Directories...1968lb # [ 1.832948] systemd[1]: Starting Rule-based Manager for Device Events and Files...1969client # [ 1.851267] systemd-udevd[77]: Using default interface naming scheme 'v261'.1970worker2 # [ 1.959976] systemd[1]: Finished Apply Kernel Variables.1971worker1 # [ 1.970950] systemd[1]: Finished Apply Kernel Variables.1972worker2 # [ 1.965080] systemd[1]: Started Journal Service.1973worker2 # [ 1.834176] systemd-modules-load[68]: Inserted module 'dm_mod'1974worker2 # [ 1.836307] systemd-modules-load[68]: Inserted module 'virtio_balloon'1975worker1 # [ 1.979549] systemd[1]: Started Journal Service.1976worker2 # [ 1.838814] systemd-modules-load[68]: Inserted module 'virtio_gpu'1977worker1 # [ 1.848244] systemd-modules-load[68]: Inserted module 'dm_mod'1978worker1 # [ 1.850681] systemd-modules-load[68]: Inserted module 'virtio_balloon'1979worker1 # [ 1.853330] systemd-modules-load[68]: Inserted module 'virtio_gpu'1980client # [ 1.875139] systemd[1]: Started Rule-based Manager for Device Events and Files.1981worker1 # [ 1.858767] systemd[1]: Finished Create Static Device Nodes in /dev.1982lb # [ 1.865429] systemd[1]: Finished Create System Files and Directories.1983worker2 # [ 1.852131] systemd[1]: Finished Create Static Device Nodes in /dev.1984worker2 # [ 1.853129] systemd[1]: Reached target Preparation for Local File Systems.1985worker2 # [ 1.854096] systemd[1]: Reached target Local File Systems.1986worker1 # [ 1.863824] systemd[1]: Reached target Preparation for Local File Systems.1987worker2 # [ 1.856951] systemd[1]: Starting Create System Files and Directories...1988worker1 # [ 1.866989] systemd[1]: Reached target Local File Systems.1989worker1 # [ 1.870038] systemd[1]: Starting Create System Files and Directories...1990worker2 # [ 1.865074] systemd[1]: Starting Rule-based Manager for Device Events and Files...1991worker1 # [ 1.877077] systemd[1]: Starting Rule-based Manager for Device Events and Files...1992lb # [ 1.888253] systemd-udevd[76]: Using default interface naming scheme 'v261'.1993client # [ 1.917904] systemd[1]: Finished Coldplug All udev Devices.1994client # [ 1.918807] systemd[1]: Reached target System Initialization.1995client # [ 1.919700] systemd[1]: Reached target Basic System.1996worker2 # [ 1.893723] systemd[1]: Finished Create System Files and Directories.1997lb # [ 1.911648] systemd[1]: Started Rule-based Manager for Device Events and Files.1998worker1 # [ 1.907607] systemd[1]: Finished Create System Files and Directories.1999worker2 # [ 1.917918] systemd-udevd[83]: Using default interface naming scheme 'v261'.2000worker1 # [ 1.930144] systemd-udevd[83]: Using default interface naming scheme 'v261'.

loading 1,901 hidden lines…

3902worker1: (connecting took 0.00 seconds)3903worker1: (finished: waiting for the VM to finish booting, in 0.00 seconds)3904worker1: (finished: waiting for unit network-addresses-eth1.service, in 0.08 seconds)3905worker1: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }3906worker1: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }, in 0.03 seconds)3907worker2: waiting for unit network-addresses-eth1.service3908worker2: waiting for the VM to finish booting3909worker2: Guest shell says: b'Spawning backdoor root shell...\n'3910worker2: connected to guest root shell3911worker2: (connecting took 0.00 seconds)3912worker2: (finished: waiting for the VM to finish booting, in 0.00 seconds)3913worker2: (finished: waiting for unit network-addresses-eth1.service, in 0.07 seconds)3914worker2: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }3915worker2: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }, in 0.02 seconds)3916lb: waiting for unit niks3.service3917worker1 # [ 12.890515] dhcpcd[689]: eth0: soliciting an IPv6 router3918worker1 # [ 12.893511] dhcpcd[689]: eth0: Router Advertisement from fe80::23919worker1 # [ 12.895583] dhcpcd[689]: eth0: adding address fec0::5054:ff:fe12:3456/643920worker1 # [ 12.897764] dhcpcd[689]: eth0: adding route to fec0::/643921worker1 # [ 12.899519] dhcpcd[689]: eth0: adding default route via fe80::23922worker1 # [ 13.299325] dhcpcd[689]: eth0: soliciting a DHCP lease3923worker1 # [ 13.450105] NET: Registered PF_PACKET protocol family3924worker1 # [ 13.321443] dhcpcd[689]: eth0: offered 10.0.2.15 from 10.0.2.23925worker1 # [ 13.323168] dhcpcd[689]: eth0: probing address 10.0.2.15/243926lb # [ 13.479295] dhcpcd[781]: eth0: soliciting a DHCP lease3927lb # [ 13.630699] NET: Registered PF_PACKET protocol family3928lb # [ 13.499412] dhcpcd[781]: eth0: offered 10.0.2.15 from 10.0.2.23929lb # [ 13.502172] dhcpcd[781]: eth0: probing address 10.0.2.15/243930lb # [ 13.922224] dhcpcd[781]: eth0: soliciting an IPv6 router3931lb # [ 13.924928] dhcpcd[781]: eth0: Router Advertisement from fe80::23932lb # [ 13.925810] dhcpcd[781]: eth0: adding address fec0::5054:ff:fe12:3456/643933lb # [ 13.926757] dhcpcd[781]: eth0: adding route to fec0::/643934lb # [ 13.927563] dhcpcd[781]: eth0: adding default route via fe80::23935lb # [ 13.965347] postgresql-pre-start[798]: selecting default time zone ... UTC3936lb # [ 13.969569] postgresql-pre-start[798]: creating configuration files ... ok3937lb # [ 14.209721] postgresql-pre-start[798]: running bootstrap script ... ok3938lb # [ 14.747789] postgresql-pre-start[798]: performing post-bootstrap initialization ... ok3939lb # [ 14.879716] postgresql-pre-start[798]: syncing data to disk ... ok3940lb # [ 14.881715] postgresql-pre-start[798]: initdb: warning: enabling "trust" authentication for local connections3941lb # [ 14.882976] postgresql-pre-start[798]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.3942lb # [ 14.884866] postgresql-pre-start[798]: Success. You can now start the database server using:3943lb # [ 14.885946] postgresql-pre-start[798]: pg_ctl -D /var/lib/postgresql/18 -l logfile start3944lb # [ 15.025948] postgres[858]: [858] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit3945lb # [ 15.030446] postgres[858]: [858] LOG: listening on IPv6 address "::1", port 54323946lb # [ 15.031751] postgres[858]: [858] LOG: listening on IPv4 address "127.0.0.1", port 54323947lb # [ 15.034454] postgres[858]: [858] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432"3948lb # [ 15.057200] postgres[873]: [873] LOG: database system was shut down at 2026-09-20 13:33:18 GMT3949lb # [ 15.065177] postgres[858]: [858] LOG: database system is ready to accept connections3950lb # [ 15.071526] systemd[1]: Started PostgreSQL Server.3951lb # [ 15.077691] systemd[1]: Starting PostgreSQL Setup Scripts...3952lb # [ 15.250408] postgresql-setup-start[885]: CREATE DATABASE3953lb # [ 15.297173] postgresql-setup-start[890]: CREATE ROLE3954lb # [ 15.319414] postgresql-setup-start[892]: ALTER DATABASE3955lb # [ 15.325303] systemd[1]: Finished PostgreSQL Setup Scripts.3956lb # [ 15.326894] systemd[1]: Reached target PostgreSQL.3957worker2 # [ 15.949851] dhcpcd[692]: eth0: leased 10.0.2.15 for 86400 seconds3958worker2 # [ 15.954642] dhcpcd[692]: eth0: adding route to 10.0.2.0/243959worker2 # [ 15.958395] dhcpcd[692]: eth0: adding default route via 10.0.2.23960worker2 # [ 16.070504] systemd[1]: Started DHCP Client.3961worker2 # [ 16.071809] systemd[1]: Reached target Multi-User System.3962worker2 # [ 16.073194] systemd[1]: Startup finished in 874ms (kernel) + 4.014s (initrd) + 11.183s (userspace) = 16.072s.3963client # [ 17.652456] dhcpcd[565]: eth0: leased 10.0.2.15 for 86400 seconds3964client # [ 17.655457] dhcpcd[565]: eth0: adding route to 10.0.2.0/243965client # [ 17.657520] dhcpcd[565]: eth0: adding default route via 10.0.2.23966client # [ 17.763792] systemd[1]: Started DHCP Client.3967client # [ 17.765756] systemd[1]: Reached target Multi-User System.3968client # [ 17.766722] systemd[1]: Startup finished in 863ms (kernel) + 5.056s (initrd) + 11.846s (userspace) = 17.766s.3969worker1 # [ 18.118987] dhcpcd[689]: eth0: leased 10.0.2.15 for 86400 seconds3970worker1 # [ 18.120294] dhcpcd[689]: eth0: adding route to 10.0.2.0/243971worker1 # [ 18.121348] dhcpcd[689]: eth0: adding default route via 10.0.2.23972worker1 # [ 18.210765] systemd[1]: Started DHCP Client.3973worker1 # [ 18.212804] systemd[1]: Reached target Multi-User System.3974worker1 # [ 18.213754] systemd[1]: Startup finished in 886ms (kernel) + 5.058s (initrd) + 12.267s (userspace) = 18.212s.3975lb # [ 18.338394] dhcpcd[781]: eth0: leased 10.0.2.15 for 86400 seconds3976lb # [ 18.339700] dhcpcd[781]: eth0: adding route to 10.0.2.0/243977lb # [ 18.340720] dhcpcd[781]: eth0: adding default route via 10.0.2.23978lb # [ 18.443429] systemd[1]: Started DHCP Client.3979lb # [ 18.445962] systemd[1]: Reached target Network is Online.3980lb # [ 18.450428] systemd[1]: Started Envoy reverse proxy.3981lb # [ 18.454358] systemd[1]: Started mock-oidc.service.3982lb # [ 18.542389] mock-oidc-server[984]: Mock OIDC Server running3983lb # [ 18.543389] mock-oidc-server[984]: OIDC Address: 192.168.1.2:80803984lb # [ 18.544761] mock-oidc-server[984]: Issue Address: 0.0.0.0:80813985lb # [ 18.545808] mock-oidc-server[984]: Issuer: http://192.168.1.2:8080/oidc3986lb # [ 18.546927] mock-oidc-server[984]: JWKS: http://192.168.1.2:8080/oidc/.well-known/jwks.json3987lb # [ 18.548304] mock-oidc-server[984]: Discovery: http://192.168.1.2:8080/oidc/.well-known/openid-configuration3988lb # [ 18.549922] mock-oidc-server[984]: Issue tokens: http://0.0.0.0:8081/issue?sub=...3989lb # [ 18.926086] envoy[983]: [2026-09-20 13:33:22.542][983][info][main] [source/server/server.cc:467] initializing epoch 0 (base id=0, hot restart version=11.104)3990lb # [ 18.928174] envoy[983]: [2026-09-20 13:33:22.544][983][info][main] [source/server/server.cc:469] statically linked extensions:3991lb # [ 18.929870] envoy[983]: [2026-09-20 13:33:22.544][983][info][main] [source/server/server.cc:471] envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata3992lb # [ 18.931926] envoy[983]: [2026-09-20 13:33:22.544][983][info][main] [source/server/server.cc:471] envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo, envoy.generic_proxy.codecs.http13993lb # [ 18.933832] envoy[983]: [2026-09-20 13:33:22.544][983][info][main] [source/server/server.cc:471] envoy.load_balancing_policies: envoy.load_balancing_policies.client_side_weighted_round_robin, envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.dynamic_modules, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.load_aware_locality, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.override_host, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset, envoy.load_balancing_policies.wrr_locality3994lb # [ 18.947621] envoy[983]: [2026-09-20 13:33:22.544][983][info][main] [source/server/server.cc:471] envoy.retry_priorities: envoy.retry_priorities.previous_priorities3995lb # [ 18.949491] envoy[983]: [2026-09-20 13:33:22.544][983][info][main] [source/server/server.cc:471] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor3996lb # [ 18.952087] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.upstream_options: envoy.extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions, envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options3997lb # [ 18.956099] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.bootstrap: envoy.bootstrap.dynamic_modules, envoy.bootstrap.internal_listener, envoy.bootstrap.reverse_tunnel.downstream_socket_interface, envoy.bootstrap.reverse_tunnel.upstream_socket_interface, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface, envoy.extensions.network.socket_interface.sockmap3998lb # [ 18.960685] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] quic.http_server_connection: quic.http_server_connection.default3999lb # [ 18.962407] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.route.early_data_policy: envoy.route.early_data_policy.default4000lb # [ 18.964075] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation4001lb # [ 18.966283] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port4002lb # [ 18.967913] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche4003lb # [ 18.969718] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts4004lb # [ 18.972423] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.quic.connection_id_generator: envoy.quic.connection_id_generator.quic_lb, envoy.quic.deterministic_connection_id_generator4005lb # [ 18.974914] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.built_in_formatters: envoy.built_in_formatters.cel, envoy.built_in_formatters.http.default, envoy.built_in_formatters.metadata, envoy.built_in_formatters.stream_info.default, envoy.built_in_formatters.xfcc_value4006lb # [ 18.978072] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain4007lb # [ 18.980244] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.quic.server_preferred_address: quic.server_preferred_address.datasource, quic.server_preferred_address.fixed4008lb # [ 18.982398] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.health_check.event_sinks: envoy.health_check.event_sink.file4009lb # [ 18.984063] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.content_parsers: envoy.content_parsers.json4010lb # [ 18.985591] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.dynamic_modules, envoy.matching.matchers.ip, envoy.matching.matchers.metadata_matcher, envoy.matching.matchers.runtime_fraction4011lb # [ 18.989487] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.cel, envoy.tracers.opentelemetry.samplers.dynatrace, envoy.tracers.opentelemetry.samplers.parent_based, envoy.tracers.opentelemetry.samplers.trace_id_ratio_based4012lb # [ 18.993660] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.tls.certificate_mappers: envoy.tls.certificate_mappers.sni, envoy.tls.certificate_mappers.static_name4013lb # [ 18.995803] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls4014lb # [ 18.999774] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.http.header_validators: envoy.http.header_validators.envoy_default4015lb # [ 19.001484] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment, envoy.tracers.opentelemetry.resource_detectors.static_config4016lb # [ 19.004568] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.dynamic_modules, envoy.tls.cert_validator.spiffe4017lb # [ 19.006802] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff4018lb # [ 19.009150] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.string_matcher: envoy.string_matcher.lua4019lb # [ 19.010941] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.upstreams: envoy.filters.connection_pools.tcp.generic4020lb # [ 19.013091] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest4021lb # [ 19.017831] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.http.ext_proc.response_processors: envoy.extensions.http.ext_proc.save_processing_response4022lb # [ 19.019796] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.dynamic_modules, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd4023lb # [ 19.024134] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr4024lb # [ 19.026644] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter4025lb # [ 19.028511] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_mappers: envoy.tls.certificate_mappers.static_name, envoy.tls.upstream_certificate_mappers.filter_state_override4026lb # [ 19.031033] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.filters.listener: envoy.filters.listener.dynamic_modules, envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.set_filter_state, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector4027lb # [ 19.036476] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.srds_factory: envoy.srds_factory.default4028lb # [ 19.037970] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory4029lb # [ 19.040737] envoy[983]: [2026-09-20 13:33:22.546][983][info][main] [source/server/server.cc:471] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.ext_authz, envoy.filters.udp.session.http_capsule4030lb # [ 19.043447] envoy[983]: [2026-09-20 13:33:22.556][983][info][main] [source/server/server.cc:471] envoy.quic.connection_debug_visitor: envoy.quic.connection_debug_visitor.basic, envoy.quic.connection_debug_visitor.quic_stats4031lb # [ 19.045872] envoy[983]: [2026-09-20 13:33:22.556][983][info][main] [source/server/server.cc:471] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.fault, envoy.filters.http.a2a, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.ai_protocol_manager, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.api_key_auth, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.bandwidth_share, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cache_v2, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_server, envoy.filters.http.file_system_buffer, envoy.filters.http.filter_chain, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_reverse_transcoder, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.mcp, envoy.filters.http.mcp_json_rest_bridge, envoy.filters.http.mcp_router, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.proto_api_scrubber, envoy.filters.http.proto_message_extraction, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.sse_to_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.thrift_to_metadata, envoy.filters.http.transform, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router4032lb # [ 19.071575] envoy[983]: [2026-09-20 13:33:22.556][983][info][main] [source/server/server.cc:471] envoy.generic_proxy.filters: envoy.filters.generic.router4033lb # [ 19.073357] envoy[983]: [2026-09-20 13:33:22.556][983][info][main] [source/server/server.cc:471] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso4034lb # [ 19.075371] envoy[983]: [2026-09-20 13:33:22.556][983][info][main] [source/server/server.cc:471] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor4035lb # [ 19.077983] envoy[983]: [2026-09-20 13:33:22.556][983][info][main] [source/server/server.cc:471] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.dynamic_modules, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.ext_proc, envoy.filters.network.generic_proxy, envoy.filters.network.geoip, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.reverse_tunnel, envoy.filters.network.reverse_tunnel_drain_aware_http_connection_manager, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_bandwidth_limit, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy4036lb # [ 19.089470] envoy[983]: [2026-09-20 13:33:22.556][983][info][main] [source/server/server.cc:471] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation4037lb # [ 19.091475] envoy[983]: [2026-09-20 13:33:22.556][983][info][main] [source/server/server.cc:471] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme4038lb # [ 19.094418] envoy[983]: [2026-09-20 13:33:22.556][983][info][main] [source/server/server.cc:471] envoy.matching.generic_proxy_request_input.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service4039lb # [ 19.098531] envoy[983]: [2026-09-20 13:33:22.556][983][info][main] [source/server/server.cc:471] envoy.http.forward_client_cert_details: forward_client_cert4040lb # [ 19.100386] envoy[983]: [2026-09-20 13:33:22.556][983][info][main] [source/server/server.cc:471] envoy.http.injected_credentials: envoy.http.injected_credentials.generic, envoy.http.injected_credentials.oauth24041lb # [ 19.102540] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable4042lb # [ 19.104456] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.resolvers: envoy.ip, envoy.resolvers.reverse_connection4043lb # [ 19.106293] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector, envoy.upstream.local_address_selector.filter_state_override4044lb # [ 19.109078] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple4045lb # [ 19.111280] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel, envoy.access_loggers.extension_filters.process_ratelimit4046lb # [ 19.113869] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.request_id: envoy.request_id.uuid4047lb # [ 19.115485] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router4048lb # [ 19.118326] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.ssl.server_context_factory: envoy.ssl.server_context_factory.default4049lb # [ 19.120366] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.regex_engines: envoy.regex_engines.google_re24050lb # [ 19.122147] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] network.connection.client: default, envoy_internal4051lb # [ 19.123916] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret4052lb # [ 19.125941] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo, envoy.network.dns_resolver.hickory4053lb # [ 19.128687] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.serializers: dubbo.hessian24054lb # [ 19.130188] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.http.cache_v2: envoy.extensions.http.cache_v2.file_system_http_cache, envoy.extensions.http.cache_v2.simple4055lb # [ 19.132516] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v84056lb # [ 19.134936] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.formatter: envoy.formatter.cel, envoy.formatter.dynamic_modules, envoy.formatter.file_content, envoy.formatter.generic_secret, envoy.formatter.metadata, envoy.formatter.req_without_query4057lb # [ 19.137981] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher4058lb # [ 19.140472] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.credential_injector, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec, envoy.filters.http.upstream_rbac, envoy.filters.http.wasm4059lb # [ 19.146760] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.thrift_proxy.transports: auto, framed, header, unframed4060lb # [ 19.148489] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.protocols: dubbo4061lb # [ 19.149987] envoy[983]: [2026-09-20 13:33:22.557][983][info][main] [source/server/server.cc:471] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy4062lb # [ 19.152465] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.network_namespace, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san4063lb # [ 19.158692] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case4064lb # [ 19.161072] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.quic.client_packet_writer: envoy.quic.packet_writer.default4065lb # [ 19.163079] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.matching.action: envoy.matching.action.transport_socket.name, envoy.matching.actions.format_string, filter-chain-name4066lb # [ 19.165603] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.rbac.principals: envoy.rbac.principals.mtls_authenticated4067lb # [ 19.167573] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls4068lb # [ 19.173081] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.access_loggers: envoy.access_loggers.dynamic_modules, envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stats, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log4069lb # [ 19.179313] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router4070lb # [ 19.181050] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action4071lb # [ 19.183079] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.http.ext_proc.processing_request_modifiers: envoy.extensions.http.ext_proc.mapped_attribute_builder4072lb # [ 19.185295] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.tls.certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret4073lb # [ 19.187290] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.health_checkers: envoy.health_checkers.dynamic_modules, envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift4074lb # [ 19.190389] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.dynamic_metadata, envoy.matching.inputs.dynamic_module_data_input, envoy.matching.inputs.filter_state, envoy.matching.inputs.local_reply, envoy.matching.inputs.network_namespace, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params4075lb # [ 19.200110] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua, envoy.router.cluster_specifier_plugin.matcher4076lb # [ 19.202554] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.filters.upstream_network: envoy.filters.upstream_network.reverse_tunnel_lifecycle4077lb # [ 19.204553] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.common.key_value: envoy.key_value.file_based4078lb # [ 19.206230] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator4079lb # [ 19.208711] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.route_config_update_requester: envoy.route_config_update_requester.default4080lb # [ 19.210944] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.dynamic_modules, envoy.filters.udp_listener.udp_proxy4081lb # [ 19.213289] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.clusters: envoy.cluster.dns, envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.composite, envoy.clusters.dynamic_forward_proxy, envoy.clusters.dynamic_modules, envoy.clusters.mcp_multicluster, envoy.clusters.redis, envoy.clusters.reverse_connection4082lb # [ 19.217955] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.rds_factory: envoy.rds_factory.default4083lb # [ 19.219929] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.matching.stat_tag_matching_data.input: stat_tag_value_input4084lb # [ 19.222166] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.matching.transport_socket.input: envoy.matching.inputs.endpoint_metadata, envoy.matching.inputs.locality_metadata, envoy.matching.inputs.transport_socket_filter_state4085lb # [ 19.224895] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.hashable_string, envoy.network.application_protocols, envoy.network.ip, envoy.network.network_namespace, envoy.network.transport_socket.http_11_proxy.info, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.ratelimit.hits_addend, envoy.router.debug_config, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port4086lb # [ 19.231978] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.resource_monitors: envoy.resource_monitors.cgroup_memory, envoy.resource_monitors.cpu_utilization, envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource4087lb # [ 19.234983] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.geoip_providers: envoy.geoip_providers.maxmind4088lb # [ 19.237246] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher4089lb # [ 19.239859] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.tracers: envoy.tracers.datadog, envoy.tracers.dynamic_modules, envoy.tracers.fluentd, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin4090lb # [ 19.243144] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher4091lb # [ 19.245290] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.envelope, envoy.http.stateful_session.header4092lb # [ 19.247742] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.connection_handler: envoy.connection_handler.default4093lb # [ 19.249866] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.matching.stat_matching_data.input: stat_full_name_match_input4094lb # [ 19.252080] envoy[983]: [2026-09-20 13:33:22.558][983][info][main] [source/server/server.cc:471] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter4095lb # [ 19.254334] envoy[983]: [2026-09-20 13:33:22.694][983][info][main] [source/server/server.cc:531] HTTP header map info:4096lb # [ 19.255622] envoy[983]: [2026-09-20 13:33:22.740][983][info][main] [source/server/server.cc:534] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-host,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id4097lb # [ 19.268665] envoy[983]: [2026-09-20 13:33:22.740][983][info][main] [source/server/server.cc:534] request trailer map: 120 bytes:4098lb # [ 19.270455] envoy[983]: [2026-09-20 13:33:22.740][983][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id4099lb # [ 19.278119] envoy[983]: [2026-09-20 13:33:22.740][983][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status4100lb # [ 19.315645] envoy[983]: [2026-09-20 13:33:22.932][983][info][main] [source/server/server.cc:968] runtime: {}4101lb # [ 19.323282] envoy[983]: [2026-09-20 13:33:22.939][983][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288.4102lb # [ 19.325175] envoy[983]: [2026-09-20 13:33:22.941][983][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:99014103lb # [ 19.327812] envoy[983]: [2026-09-20 13:33:22.944][983][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:99014104lb # [ 19.330161] envoy[983]: [2026-09-20 13:33:22.946][983][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration4105lb # [ 19.331869] envoy[983]: [2026-09-20 13:33:22.948][983][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s)4106lb # [ 19.333376] envoy[983]: [2026-09-20 13:33:22.948][983][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s)4107lb # [ 19.376151] envoy[983]: [2026-09-20 13:33:22.992][983][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s)4108worker2 # [ 19.363769] systemd[1]: Reached target Network is Online.4109worker1 # [ 19.375632] systemd[1]: Starting Nix worker-protocol over gRPC...4110worker2 # [ 19.371497] systemd[1]: Starting scheduler-token.service...4111lb # [ 19.391764] envoy[983]: [2026-09-20 13:33:23.008][983][info][config] [source/server/configuration_impl.cc:164] loading stats configuration4112lb # [ 19.393870] envoy[983]: [2026-09-20 13:33:23.010][983][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor.4113lb # [ 19.396686] envoy[983]: [2026-09-20 13:33:23.013][983][info][main] [source/server/server.cc:1077] starting main dispatch loop4114worker2 # [ 19.480577] systemd[1]: scheduler-token.service: Deactivated successfully.4115worker2 # [ 19.481728] systemd[1]: Finished scheduler-token.service.4116worker2 # [ 19.483161] systemd[1]: scheduler-token.service: Consumed 34ms CPU time over 109ms wall clock time, 3.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4117worker2 # [ 19.488598] systemd[1]: Starting Nix worker-protocol over gRPC...4118worker1 # [ 19.571051] nix-grpc-daemon[784]: ts=2026-09-20T13:33:23Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4119worker1 # [ 19.579644] nix-grpc-daemon[784]: ts=2026-09-20T13:33:23Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4120worker1 # [ 19.586969] nix-grpc-daemon[784]: ts=2026-09-20T13:33:23Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=14121worker1 # [ 19.622492] nix-grpc-daemon[784]: ts=2026-09-20T13:33:23Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4122worker1 # [ 19.626798] systemd[1]: Started Nix worker-protocol over gRPC.4123worker1 # [ 19.655261] nix-grpc-daemon[784]: ts=2026-09-20T13:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4124lb # [ 19.679719] envoy[983]: [2026-09-20 13:33:23.295][983][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization4125lb # [ 19.684353] envoy[983]: [2026-09-20 13:33:23.296][983][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized4126worker2 # [ 19.673731] nix-grpc-daemon[787]: ts=2026-09-20T13:33:22Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4127lb # [ 19.688455] envoy[983]: [2026-09-20 13:33:23.296][983][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager4128lb # [ 19.692271] envoy[983]: [2026-09-20 13:33:23.296][983][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers4129worker2 # [ 19.683092] nix-grpc-daemon[787]: ts=2026-09-20T13:33:22Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4130worker2 # [ 19.688662] nix-grpc-daemon[787]: ts=2026-09-20T13:33:22Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14131worker2 # [ 19.729232] nix-grpc-daemon[787]: ts=2026-09-20T13:33:23Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4132worker2 # [ 19.733460] systemd[1]: Started Nix worker-protocol over gRPC.4133worker2 # [ 19.783459] nix-grpc-daemon[787]: ts=2026-09-20T13:33:23Z level=info event=scheduler_connected addr=lb:500514134worker2 # [ 19.797670] nix-grpc-daemon[787]: ts=2026-09-20T13:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4135worker1 # [ 19.856300] nix-grpc-daemon[784]: ts=2026-09-20T13:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4136worker2 # [ 20.000438] nix-grpc-daemon[787]: ts=2026-09-20T13:33:23Z level=info event=scheduler_connected addr=lb:500514137worker2 # [ 20.004646] nix-grpc-daemon[787]: ts=2026-09-20T13:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4138worker2 # [ 20.205904] nix-grpc-daemon[787]: ts=2026-09-20T13:33:23Z level=info event=scheduler_connected addr=lb:500514139worker2 # [ 20.209702] nix-grpc-daemon[787]: ts=2026-09-20T13:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4140worker1 # [ 20.256929] nix-grpc-daemon[784]: ts=2026-09-20T13:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4141worker2 # [ 20.410422] nix-grpc-daemon[787]: ts=2026-09-20T13:33:23Z level=info event=scheduler_connected addr=lb:500514142worker2 # [ 20.412236] nix-grpc-daemon[787]: ts=2026-09-20T13:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4143worker2 # [ 20.612886] nix-grpc-daemon[787]: ts=2026-09-20T13:33:23Z level=info event=scheduler_connected addr=lb:500514144worker2 # [ 20.614581] nix-grpc-daemon[787]: ts=2026-09-20T13:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4145worker2 # [ 20.817092] nix-grpc-daemon[787]: ts=2026-09-20T13:33:24Z level=info event=scheduler_connected addr=lb:500514146worker2 # [ 20.820224] nix-grpc-daemon[787]: ts=2026-09-20T13:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4147worker2 # [ 21.019097] nix-grpc-daemon[787]: ts=2026-09-20T13:33:24Z level=info event=scheduler_connected addr=lb:500514148worker2 # [ 21.023109] nix-grpc-daemon[787]: ts=2026-09-20T13:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4149worker1 # [ 21.059374] nix-grpc-daemon[784]: ts=2026-09-20T13:33:24Z level=info event=scheduler_connected addr=lb:500514150worker1 # [ 21.061603] nix-grpc-daemon[784]: ts=2026-09-20T13:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4151worker2 # [ 21.224122] nix-grpc-daemon[787]: ts=2026-09-20T13:33:24Z level=info event=scheduler_connected addr=lb:500514152worker2 # [ 21.227912] nix-grpc-daemon[787]: ts=2026-09-20T13:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4153worker1 # [ 21.262834] nix-grpc-daemon[784]: ts=2026-09-20T13:33:25Z level=info event=scheduler_connected addr=lb:500514154worker1 # [ 21.266841] nix-grpc-daemon[784]: ts=2026-09-20T13:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4155worker2 # [ 21.428918] nix-grpc-daemon[787]: ts=2026-09-20T13:33:24Z level=info event=scheduler_connected addr=lb:500514156worker2 # [ 21.432821] nix-grpc-daemon[787]: ts=2026-09-20T13:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4157worker1 # [ 21.467787] nix-grpc-daemon[784]: ts=2026-09-20T13:33:25Z level=info event=scheduler_connected addr=lb:500514158worker1 # [ 21.471735] nix-grpc-daemon[784]: ts=2026-09-20T13:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4159worker2 # [ 21.634168] nix-grpc-daemon[787]: ts=2026-09-20T13:33:24Z level=info event=scheduler_connected addr=lb:500514160worker2 # [ 21.638083] nix-grpc-daemon[787]: ts=2026-09-20T13:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4161worker1 # [ 21.672445] nix-grpc-daemon[784]: ts=2026-09-20T13:33:25Z level=info event=scheduler_connected addr=lb:500514162worker1 # [ 21.677769] nix-grpc-daemon[784]: ts=2026-09-20T13:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4163worker2 # [ 21.840343] nix-grpc-daemon[787]: ts=2026-09-20T13:33:25Z level=info event=scheduler_connected addr=lb:500514164worker2 # [ 21.843390] nix-grpc-daemon[787]: ts=2026-09-20T13:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4165worker1 # [ 21.878593] nix-grpc-daemon[784]: ts=2026-09-20T13:33:25Z level=info event=scheduler_connected addr=lb:500514166worker1 # [ 21.882554] nix-grpc-daemon[784]: ts=2026-09-20T13:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4167worker2 # [ 22.041347] nix-grpc-daemon[787]: ts=2026-09-20T13:33:25Z level=info event=scheduler_connected addr=lb:500514168worker2 # [ 22.045209] nix-grpc-daemon[787]: ts=2026-09-20T13:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4169worker1 # [ 22.081616] nix-grpc-daemon[784]: ts=2026-09-20T13:33:25Z level=info event=scheduler_connected addr=lb:500514170worker1 # [ 22.085217] nix-grpc-daemon[784]: ts=2026-09-20T13:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4171worker2 # [ 22.246251] nix-grpc-daemon[787]: ts=2026-09-20T13:33:25Z level=info event=scheduler_connected addr=lb:500514172worker2 # [ 22.250091] nix-grpc-daemon[787]: ts=2026-09-20T13:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4173worker1 # [ 22.285777] nix-grpc-daemon[784]: ts=2026-09-20T13:33:26Z level=info event=scheduler_connected addr=lb:500514174worker1 # [ 22.287727] nix-grpc-daemon[784]: ts=2026-09-20T13:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4175worker2 # [ 22.450907] nix-grpc-daemon[787]: ts=2026-09-20T13:33:25Z level=info event=scheduler_connected addr=lb:500514176worker2 # [ 22.453162] nix-grpc-daemon[787]: ts=2026-09-20T13:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4177worker1 # [ 22.488699] nix-grpc-daemon[784]: ts=2026-09-20T13:33:26Z level=info event=scheduler_connected addr=lb:500514178worker1 # [ 22.492628] nix-grpc-daemon[784]: ts=2026-09-20T13:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4179worker2 # [ 22.654037] nix-grpc-daemon[787]: ts=2026-09-20T13:33:25Z level=info event=scheduler_connected addr=lb:500514180worker2 # [ 22.656143] nix-grpc-daemon[787]: ts=2026-09-20T13:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4181worker1 # [ 22.693298] nix-grpc-daemon[784]: ts=2026-09-20T13:33:26Z level=info event=scheduler_connected addr=lb:500514182worker1 # [ 22.697132] nix-grpc-daemon[784]: ts=2026-09-20T13:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4183worker2 # [ 22.857159] nix-grpc-daemon[787]: ts=2026-09-20T13:33:26Z level=info event=scheduler_connected addr=lb:500514184worker2 # [ 22.859157] nix-grpc-daemon[787]: ts=2026-09-20T13:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4185worker1 # [ 22.894913] nix-grpc-daemon[784]: ts=2026-09-20T13:33:26Z level=info event=scheduler_connected addr=lb:500514186worker1 # [ 22.898954] nix-grpc-daemon[784]: ts=2026-09-20T13:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4187worker2 # [ 23.060027] nix-grpc-daemon[787]: ts=2026-09-20T13:33:26Z level=info event=scheduler_connected addr=lb:500514188worker2 # [ 23.061849] nix-grpc-daemon[787]: ts=2026-09-20T13:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4189worker1 # [ 23.101247] nix-grpc-daemon[784]: ts=2026-09-20T13:33:26Z level=info event=scheduler_connected addr=lb:500514190worker1 # [ 23.104416] nix-grpc-daemon[784]: ts=2026-09-20T13:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4191lb # [ 23.206033] rustfs-bucket-start[1037]: mb s3://farm4192lb # [ 23.211655] systemd[1]: Finished rustfs-bucket.service.4193lb # [ 23.218336] systemd[1]: Starting niks3 server...4194worker2 # [ 23.263268] nix-grpc-daemon[787]: ts=2026-09-20T13:33:26Z level=info event=scheduler_connected addr=lb:500514195worker2 # [ 23.267383] nix-grpc-daemon[787]: ts=2026-09-20T13:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4196worker1 # [ 23.301966] nix-grpc-daemon[784]: ts=2026-09-20T13:33:27Z level=info event=scheduler_connected addr=lb:500514197worker1 # [ 23.303912] nix-grpc-daemon[784]: ts=2026-09-20T13:33:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4198lb # [ 23.446605] postgres[1053]: [1053] ERROR: relation "goose_db_version" does not exist at character 364199lb # [ 23.447940] postgres[1053]: [1053] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4200worker2 # [ 23.468718] nix-grpc-daemon[787]: ts=2026-09-20T13:33:26Z level=info event=scheduler_connected addr=lb:500514201lb # [ 23.488358] niks3-server[1047]: 2026/09/20 13:33:27 OK 20241026095416_initial_model.sql (21.74ms)4202worker2 # [ 23.472924] nix-grpc-daemon[787]: ts=2026-09-20T13:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4203lb # [ 23.491709] niks3-server[1047]: 2026/09/20 13:33:27 OK 20251210153512_drop_unused_gin_index.sql (3.49ms)4204lb # [ 23.496865] niks3-server[1047]: 2026/09/20 13:33:27 OK 20251218171726_add_pins.sql (5.06ms)4205lb # [ 23.501678] niks3-server[1047]: 2026/09/20 13:33:27 OK 20260628120000_add_object_size_and_stats.sql (4.89ms)4206lb # [ 23.506267] niks3-server[1047]: 2026/09/20 13:33:27 OK 20260905000000_add_claims.sql (3.09ms)4207lb # [ 23.510837] niks3-server[1047]: 2026/09/20 13:33:27 OK 20260920000000_drop_claims.sql (4.44ms)4208lb # [ 23.512610] niks3-server[1047]: 2026/09/20 13:33:27 goose: successfully migrated database to version: 202609200000004209worker1 # [ 23.505090] nix-grpc-daemon[784]: ts=2026-09-20T13:33:27Z level=info event=scheduler_connected addr=lb:500514210lb # [ 23.518065] niks3-server[1047]: 2026/09/20 13:33:27 OK 1_commit_pending_closure.sql (7.29ms)4211worker1 # [ 23.509131] nix-grpc-daemon[784]: ts=2026-09-20T13:33:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4212lb # [ 23.520617] niks3-server[1047]: 2026/09/20 13:33:27 OK 2_object_stats_trigger.sql (1.35ms)4213lb # [ 23.522053] niks3-server[1047]: 2026/09/20 13:33:27 goose: up to current file version: 24214lb # [ 23.527764] niks3-server[1047]: 2026/09/20 13:33:27 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key4215lb # [ 23.603797] niks3-server[1047]: 2026/09/20 13:33:27 INFO Read proxy enabled — serving cache objects from S34216lb # [ 23.605803] niks3-server[1047]: 2026/09/20 13:33:27 INFO Using socket-activated listener address=0.0.0.0:57514217lb # [ 23.612416] niks3-server[1047]: 2026/09/20 13:33:27 INFO systemd watchdog enabled interval=15s4218lb # [ 23.613692] systemd[1]: Started niks3 server.4219lb # [ 23.616913] systemd[1]: Reached target Multi-User System.4220lb # [ 23.617846] niks3-server[1047]: 2026/09/20 13:33:27 INFO Starting HTTP server address=0.0.0.0:57514221lb # [ 23.619851] systemd[1]: Startup finished in 876ms (kernel) + 5.130s (initrd) + 17.611s (userspace) = 23.618s.4222lb # [ 23.626132] niks3-server[1047]: 2026/09/20 13:33:27 INFO lead: acquired remote=192.168.1.4:375384223worker2 # [ 23.619445] nix-grpc-daemon[787]: ts=2026-09-20T13:33:26Z level=info event=scheduler_take_over4224lb: (finished: waiting for unit niks3.service, in 10.94 seconds)4225worker2 # [ 23.674197] nix-grpc-daemon[787]: ts=2026-09-20T13:33:26Z level=info event=scheduler_connected addr=lb:500514226lb: waiting for TCP port 5751 on localhost4227worker2 # [ 23.678153] nix-grpc-daemon[787]: ts=2026-09-20T13:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4228worker1 # [ 23.709814] nix-grpc-daemon[784]: ts=2026-09-20T13:33:27Z level=info event=scheduler_connected addr=lb:500514229worker1 # [ 23.713663] nix-grpc-daemon[784]: ts=2026-09-20T13:33:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4230lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!4231lb: (finished: waiting for TCP port 5751 on localhost, in 0.07 seconds)4232lb: waiting for unit envoy.service4233lb: (finished: waiting for unit envoy.service, in 0.05 seconds)4234lb: waiting for TCP port 50051 on localhost4235lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!4236lb # [ 23.847514] envoy[983]: conn peer=[::1]:52554 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=24237lb: (finished: waiting for TCP port 50051 on localhost, in 0.03 seconds)4238worker2 # [ 23.881218] nix-grpc-daemon[787]: ts=2026-09-20T13:33:27Z level=info event=scheduler_connected addr=lb:500514239worker1 # [ 23.914113] nix-grpc-daemon[784]: ts=2026-09-20T13:33:27Z level=info event=scheduler_connected addr=lb:500514240worker2 # [ 23.905987] nix-grpc-daemon[787]: ts=2026-09-20T13:33:27Z level=info event=oidc_jwks provider=mock keys=14241worker2 # [ 23.912383] nix-grpc-daemon[787]: ts=2026-09-20T13:33:27Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:504704242worker2 # [ 23.916202] nix-grpc-daemon[787]: ts=2026-09-20T13:33:27Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:504704243worker2 # [ 23.920696] nix-grpc-daemon[787]: ts=2026-09-20T13:33:27Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04244worker2 # [ 23.927173] nix-grpc-daemon[787]: ts=2026-09-20T13:33:27Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04245worker2 # [ 23.972226] systemd[1]: Starting scheduler-token.service...4246worker2 # [ 24.039190] systemd[1]: scheduler-token.service: Deactivated successfully.4247worker2 # [ 24.040820] systemd[1]: Finished scheduler-token.service.4248worker2 # [ 24.042201] systemd[1]: scheduler-token.service: Consumed 19ms CPU time over 69ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4249lb: waiting until exactly no endpoints of x86_64-linux are unhealthy4250??? Warning (UserWarning): retry(): The 'timeout_seconds' argument is deprecated. Use 'timeout' instead.4251 File "/nix/store/3gycnalwgpix76jjky7ljgh88gwgdb0l-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/machine/__init__.py", line 1284252lb: must succeed: curl -sf localhost:9901/clusters4253lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4254lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.08 seconds)4255subtest: one node schedules and both builders hold a WorkerSession on it4256lb: must succeed: curl -sf localhost:9901/clusters4257lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4258worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4259worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)4260lb: must succeed: curl -sf localhost:9901/clusters4261lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4262lb: must succeed: curl -sf localhost:9901/clusters4263lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4264worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4265worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.04 seconds)4266(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.18 seconds)4267subtest: balancer auth: client cert, bearer token, nothing, unknown CN4268worker2 # [ 24.581695] nix-grpc-daemon[787]: ts=2026-09-20T13:33:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504824269worker2 # [ 24.595069] systemd[1]: Started Nix Daemon.4270worker2 # [ 24.799273] nix-daemon[854]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4271worker2 # [ 24.848464] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 23 ms (attempt 1/5)4272worker2 # [ 24.873696] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 70 ms (attempt 2/5)4273worker2 # [ 24.947334] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 306 ms (attempt 3/5)4274worker2 # [ 25.259095] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 323 ms (attempt 4/5)4275worker2 # [ 25.593788] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4276worker1 # [ 25.650662] nix-grpc-daemon[784]: ts=2026-09-20T13:33:29Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:561064277worker1 # [ 25.664440] systemd[1]: Started Nix Daemon.4278worker1 # [ 25.877197] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4279worker1 # [ 25.907244] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 74 ms (attempt 1/5)4280worker1 # [ 25.983940] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 119 ms (attempt 2/5)4281worker1 # [ 26.105916] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 279 ms (attempt 3/5)4282worker1 # [ 26.390725] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 345 ms (attempt 4/5)4283worker1 # [ 26.745686] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4284worker1 # [ 26.757702] nix-grpc-daemon[784]: ts=2026-09-20T13:33:30Z level=debug event=path_info_miss path=jwg6hsvasqr1bfb4p1nn6ayia3202nfr-nixos-system-client-test error="error: path '/nix/store/jwg6hsvasqr1bfb4p1nn6ayia3202nfr-nixos-system-client-test' is required, but there is no substituter that can build it"4285worker1 # [ 26.767410] nix-grpc-daemon[784]: ts=2026-09-20T13:33:30Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:561064286worker1 # [ 26.772985] nix-grpc-daemon[784]: ts=2026-09-20T13:33:30Z level=debug event=path_info_miss path=jwg6hsvasqr1bfb4p1nn6ayia3202nfr-nixos-system-client-test error="error: path '/nix/store/jwg6hsvasqr1bfb4p1nn6ayia3202nfr-nixos-system-client-test' is required, but there is no substituter that can build it"4287client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4288client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.09 seconds)4289client: must succeed: install -D /nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4290client: (finished: must succeed: install -D /nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.03 seconds)4291worker1 # [ 27.122954] nix-grpc-daemon[784]: ts=2026-09-20T13:33:30Z level=info event=oidc_jwks provider=mock keys=14292worker1 # [ 27.128188] nix-grpc-daemon[784]: ts=2026-09-20T13:33:30Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:561064293worker1 # [ 27.146569] nix-grpc-daemon[784]: ts=2026-09-20T13:33:30Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:561064294worker1 # [ 27.156063] nix-grpc-daemon[784]: ts=2026-09-20T13:33:30Z level=debug event=path_info_miss path=jwg6hsvasqr1bfb4p1nn6ayia3202nfr-nixos-system-client-test error="error: path '/nix/store/jwg6hsvasqr1bfb4p1nn6ayia3202nfr-nixos-system-client-test' is required, but there is no substituter that can build it"4295worker2 # [ 27.154181] nix-grpc-daemon[787]: ts=2026-09-20T13:33:30Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:504824296worker2 # [ 27.172149] nix-grpc-daemon[787]: ts=2026-09-20T13:33:30Z level=debug event=path_info_miss path=jwg6hsvasqr1bfb4p1nn6ayia3202nfr-nixos-system-client-test error="error: path '/nix/store/jwg6hsvasqr1bfb4p1nn6ayia3202nfr-nixos-system-client-test' is required, but there is no substituter that can build it"4297client: must succeed: rm -r /var/lib/nix-grpc-store4298client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds)4299worker2 # [ 27.399546] nix-grpc-daemon[787]: ts=2026-09-20T13:33:30Z level=info event=denied method=QueryMissing cn=- role=none4300worker1 # [ 27.595147] nix-grpc-daemon[784]: ts=2026-09-20T13:33:31Z level=info event=denied method=QueryMissing cn=stranger role=none4301(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.31 seconds)4302subtest: DAG build is scheduled across workers and lands in the cache4303client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24304worker2 # [ 27.789847] nix-grpc-daemon[787]: ts=2026-09-20T13:33:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504824305client # don't know how to build these paths:4306client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4307worker2 # [ 27.803254] nix-grpc-daemon[787]: ts=2026-09-20T13:33:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504824308worker2 # [ 27.844433] nix-grpc-daemon[787]: ts=2026-09-20T13:33:31Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04309worker2 # [ 27.849384] nix-grpc-daemon[787]: ts=2026-09-20T13:33:31Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14310worker1 # [ 27.911157] nix-grpc-daemon[784]: ts=2026-09-20T13:33:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064311worker1 # [ 27.932216] nix-grpc-daemon[784]: ts=2026-09-20T13:33:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:561064312client # copying 1 paths to 'lb:50051'...4313worker1 # [ 27.944494] nix-grpc-daemon[784]: ts=2026-09-20T13:33:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:561064314client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4315worker1 # [ 27.969989] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4316worker1 # [ 27.998724] nix-grpc-daemon[784]: ts=2026-09-20T13:33:31Z level=info event=niks3_push_started pid=8504317worker1 # [ 28.269893] nix-grpc-daemon[850]: time=2026-09-20T13:33:32.078Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.key ca=""4318worker1 # [ 28.462257] nix-daemon[840]: accepted connection from pid 863, user nix-grpc-daemon (trusted)4319worker1 # [ 28.477333] nix-daemon[840]: reaped child process 866, status = succeeded4320worker1 # [ 28.629542] nix-daemon[840]: accepted connection from pid 868, user nix-grpc-daemon (trusted)4321worker1 # [ 28.642937] nix-daemon[840]: reaped child process 871, status = succeeded4322lb # [ 28.658857] niks3-server[1047]: 2026/09/20 13:33:32 INFO Received uploads request method=POST path=/api/pending_closures4323worker1 # [ 28.679257] nix-grpc-daemon[850]: time=2026-09-20T13:33:32.491Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4324worker1 # [ 28.681560] nix-grpc-daemon[850]: time=2026-09-20T13:33:32.493Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4325lb # [ 28.716323] niks3-server[1047]: 2026/09/20 13:33:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4326lb # [ 28.719991] niks3-server[1047]: 2026/09/20 13:33:32 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4327lb # [ 28.736796] niks3-server[1047]: 2026/09/20 13:33:32 INFO Signed narinfos id=1 count=14328worker1 # [ 28.731722] nix-grpc-daemon[850]: time=2026-09-20T13:33:32.543Z level=INFO msg="Uploading 1 narinfos"4329lb # [ 28.742924] niks3-server[1047]: 2026/09/20 13:33:32 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4330lb # [ 28.751041] niks3-server[1047]: 2026/09/20 13:33:32 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4331lb # [ 28.762434] niks3-server[1047]: 2026/09/20 13:33:32 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4332lb # [ 28.767070] niks3-server[1047]: 2026/09/20 13:33:32 INFO Completed upload id=14333worker1 # [ 28.760796] nix-grpc-daemon[850]: time=2026-09-20T13:33:32.572Z level=INFO msg="Upload complete. (476ms)"4334worker1 # [ 28.765367] nix-grpc-daemon[784]: ts=2026-09-20T13:33:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=0 paths=1 nar_bytes_in=5044335worker1 # [ 28.771738] nix-grpc-daemon[784]: ts=2026-09-20T13:33:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064336worker1 # [ 28.783160] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4337worker1 # [ 28.791862] nix-daemon[840]: reaped child process 848, status = succeeded4338client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4339worker1 # [ 28.799721] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4340client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4341lb # [ 29.382354] envoy[983]: conn peer=[2001:db8:1::1]:40736 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=406 ms=22274342lb # [ 29.385952] envoy[983]: conn peer=[2001:db8:1::1]:40748 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=804343lb # [ 29.389376] envoy[983]: conn peer=[2001:db8:1::1]:40754 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=134344lb # [ 29.392666] envoy[983]: conn peer=[2001:db8:1::1]:40768 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=134345client # farm-a> LOG-farm-a-t14346worker1 # [ 30.014061] nix-daemon[840]: reaped child process 875, status = succeeded4347worker1 # [ 30.153869] nix-daemon[840]: accepted connection from pid 890, user nix-grpc-daemon (trusted)4348worker1 # [ 30.166398] nix-daemon[840]: reaped child process 893, status = succeeded4349lb # [ 30.187631] niks3-server[1047]: 2026/09/20 13:33:33 INFO Received uploads request method=POST path=/api/pending_closures4350worker1 # [ 30.192776] nix-grpc-daemon[850]: time=2026-09-20T13:33:34.004Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4351worker1 # [ 30.194400] nix-grpc-daemon[850]: time=2026-09-20T13:33:34.006Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4352lb # [ 30.232384] niks3-server[1047]: 2026/09/20 13:33:33 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4353lb # [ 30.239961] niks3-server[1047]: 2026/09/20 13:33:33 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4354lb # [ 30.250199] niks3-server[1047]: 2026/09/20 13:33:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4355lb # [ 30.255519] niks3-server[1047]: 2026/09/20 13:33:33 INFO Signed narinfos id=2 count=14356worker1 # [ 30.252434] nix-grpc-daemon[850]: time=2026-09-20T13:33:34.063Z level=INFO msg="Uploading 1 narinfos"4357lb # [ 30.264623] niks3-server[1047]: 2026/09/20 13:33:33 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4358lb # [ 30.274232] niks3-server[1047]: 2026/09/20 13:33:33 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4359lb # [ 30.282397] niks3-server[1047]: 2026/09/20 13:33:33 INFO Completed upload id=24360worker1 # [ 30.276092] nix-grpc-daemon[850]: time=2026-09-20T13:33:34.087Z level=INFO msg="Upload complete. (267ms)"4361lb # [ 30.284458] niks3-server[1047]: 2026/09/20 13:33:33 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4362worker1 # [ 30.279652] nix-grpc-daemon[784]: ts=2026-09-20T13:33:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14363worker1 # [ 30.287704] nix-daemon[840]: reaped child process 873, status = succeeded4364worker2 # [ 30.289967] nix-grpc-daemon[787]: ts=2026-09-20T13:33:33Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04365worker2 # [ 30.293880] nix-grpc-daemon[787]: ts=2026-09-20T13:33:33Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14366worker1 # [ 30.343188] nix-grpc-daemon[784]: ts=2026-09-20T13:33:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064367worker1 # [ 30.349339] nix-grpc-daemon[784]: ts=2026-09-20T13:33:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:561064368worker1 # [ 30.355465] nix-grpc-daemon[850]: time=2026-09-20T13:33:34.167Z level=INFO msg="All 1 paths already cached"4369client # copying 1 paths to 'lb:50051'...4370worker1 # [ 30.359060] nix-grpc-daemon[784]: ts=2026-09-20T13:33:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:561064371client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4372worker1 # [ 30.368363] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4373worker1 # [ 30.524719] nix-daemon[840]: accepted connection from pid 897, user nix-grpc-daemon (trusted)4374worker1 # [ 30.538777] nix-daemon[840]: reaped child process 900, status = succeeded4375worker1 # [ 30.704590] nix-daemon[840]: accepted connection from pid 902, user nix-grpc-daemon (trusted)4376worker1 # [ 30.717704] nix-daemon[840]: reaped child process 906, status = succeeded4377lb # [ 30.737240] niks3-server[1047]: 2026/09/20 13:33:34 INFO Received uploads request method=POST path=/api/pending_closures4378worker1 # [ 30.740702] nix-grpc-daemon[850]: time=2026-09-20T13:33:34.552Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4379worker1 # [ 30.742258] nix-grpc-daemon[850]: time=2026-09-20T13:33:34.554Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4380lb # [ 30.769859] niks3-server[1047]: 2026/09/20 13:33:34 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4381lb # [ 30.779874] niks3-server[1047]: 2026/09/20 13:33:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4382lb # [ 30.786600] niks3-server[1047]: 2026/09/20 13:33:34 INFO Signed narinfos id=3 count=14383worker1 # [ 30.783275] nix-grpc-daemon[850]: time=2026-09-20T13:33:34.595Z level=INFO msg="Uploading 1 narinfos"4384lb # [ 30.790733] niks3-server[1047]: 2026/09/20 13:33:34 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4385lb # [ 30.805691] niks3-server[1047]: 2026/09/20 13:33:34 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4386lb # [ 30.811309] niks3-server[1047]: 2026/09/20 13:33:34 INFO Completed upload id=34387worker1 # [ 30.807286] nix-grpc-daemon[850]: time=2026-09-20T13:33:34.619Z level=INFO msg="Upload complete. (431ms)"4388lb # [ 30.816481] niks3-server[1047]: 2026/09/20 13:33:34 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4389worker1 # [ 30.810927] nix-grpc-daemon[784]: ts=2026-09-20T13:33:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=0 paths=1 nar_bytes_in=6244390worker1 # [ 30.816101] nix-grpc-daemon[784]: ts=2026-09-20T13:33:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064391worker1 # [ 30.825120] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4392worker1 # [ 30.833429] nix-daemon[840]: reaped child process 895, status = succeeded4393client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4394worker1 # [ 30.838321] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4395client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4396client # farm-b> LOG-farm-b-t14397worker1 # [ 32.054196] nix-daemon[840]: reaped child process 910, status = succeeded4398worker1 # [ 32.193864] nix-daemon[840]: accepted connection from pid 925, user nix-grpc-daemon (trusted)4399worker1 # [ 32.207079] nix-daemon[840]: reaped child process 928, status = succeeded4400lb # [ 32.228203] niks3-server[1047]: 2026/09/20 13:33:35 INFO Received uploads request method=POST path=/api/pending_closures4401worker1 # [ 32.233557] nix-grpc-daemon[850]: time=2026-09-20T13:33:36.045Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4402worker1 # [ 32.235093] nix-grpc-daemon[850]: time=2026-09-20T13:33:36.047Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4403lb # [ 32.266709] niks3-server[1047]: 2026/09/20 13:33:35 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4404lb # [ 32.282567] niks3-server[1047]: 2026/09/20 13:33:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4405lb # [ 32.288783] niks3-server[1047]: 2026/09/20 13:33:35 INFO Signed narinfos id=4 count=14406worker1 # [ 32.282731] nix-grpc-daemon[850]: time=2026-09-20T13:33:36.094Z level=INFO msg="Uploading 1 narinfos"4407lb # [ 32.291271] niks3-server[1047]: 2026/09/20 13:33:35 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4408lb # [ 32.294168] niks3-server[1047]: 2026/09/20 13:33:35 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4409lb # [ 32.299219] niks3-server[1047]: 2026/09/20 13:33:35 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4410lb # [ 32.303920] niks3-server[1047]: 2026/09/20 13:33:35 INFO Completed upload id=44411worker1 # [ 32.297592] nix-grpc-daemon[850]: time=2026-09-20T13:33:36.109Z level=INFO msg="Upload complete. (252ms)"4412lb # [ 32.305391] niks3-server[1047]: 2026/09/20 13:33:35 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4413worker1 # [ 32.302615] nix-grpc-daemon[784]: ts=2026-09-20T13:33:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14414worker2 # [ 32.300440] nix-grpc-daemon[787]: ts=2026-09-20T13:33:35Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04415worker2 # [ 32.304523] nix-grpc-daemon[787]: ts=2026-09-20T13:33:35Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14416worker1 # [ 32.315775] nix-daemon[840]: reaped child process 908, status = succeeded4417worker1 # [ 32.353117] nix-grpc-daemon[784]: ts=2026-09-20T13:33:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064418worker1 # [ 32.366124] nix-grpc-daemon[784]: ts=2026-09-20T13:33:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:561064419worker1 # [ 32.377585] nix-grpc-daemon[850]: time=2026-09-20T13:33:36.189Z level=INFO msg="All 2 paths already cached"4420client # copying 1 paths to 'lb:50051'...4421client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4422worker1 # [ 32.385482] nix-grpc-daemon[784]: ts=2026-09-20T13:33:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:561064423worker1 # [ 32.399356] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4424worker1 # [ 32.558068] nix-daemon[840]: accepted connection from pid 932, user nix-grpc-daemon (trusted)4425worker1 # [ 32.571387] nix-daemon[840]: reaped child process 935, status = succeeded4426worker1 # [ 32.717037] nix-daemon[840]: accepted connection from pid 937, user nix-grpc-daemon (trusted)4427worker1 # [ 32.729829] nix-daemon[840]: reaped child process 940, status = succeeded4428lb # [ 32.749691] niks3-server[1047]: 2026/09/20 13:33:36 INFO Received uploads request method=POST path=/api/pending_closures4429worker1 # [ 32.755397] nix-grpc-daemon[850]: time=2026-09-20T13:33:36.567Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4430worker1 # [ 32.756792] nix-grpc-daemon[850]: time=2026-09-20T13:33:36.567Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4431lb # [ 32.786268] niks3-server[1047]: 2026/09/20 13:33:36 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4432lb # [ 32.797968] niks3-server[1047]: 2026/09/20 13:33:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4433lb # [ 32.804882] niks3-server[1047]: 2026/09/20 13:33:36 INFO Signed narinfos id=5 count=14434worker1 # [ 32.803296] nix-grpc-daemon[850]: time=2026-09-20T13:33:36.614Z level=INFO msg="Uploading 1 narinfos"4435lb # [ 32.813545] niks3-server[1047]: 2026/09/20 13:33:36 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4436lb # [ 32.823270] niks3-server[1047]: 2026/09/20 13:33:36 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4437lb # [ 32.830298] niks3-server[1047]: 2026/09/20 13:33:36 INFO Completed upload id=54438worker1 # [ 32.823987] nix-grpc-daemon[850]: time=2026-09-20T13:33:36.635Z level=INFO msg="Upload complete. (412ms)"4439lb # [ 32.832084] niks3-server[1047]: 2026/09/20 13:33:36 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4440worker1 # [ 32.827084] nix-grpc-daemon[784]: ts=2026-09-20T13:33:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=0 paths=1 nar_bytes_in=7524441worker1 # [ 32.830665] nix-grpc-daemon[784]: ts=2026-09-20T13:33:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064442worker1 # [ 32.837332] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4443worker1 # [ 32.841434] nix-daemon[840]: reaped child process 930, status = succeeded4444worker1 # [ 32.845475] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4445client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4446client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4447client # farm-top> LOG-farm-top-t14448worker1 # [ 33.673520] nix-daemon[840]: reaped child process 944, status = succeeded4449worker1 # [ 33.825959] nix-daemon[840]: accepted connection from pid 959, user nix-grpc-daemon (trusted)4450worker1 # [ 33.852549] nix-daemon[840]: reaped child process 962, status = succeeded4451lb # [ 33.865463] niks3-server[1047]: 2026/09/20 13:33:37 INFO Received uploads request method=POST path=/api/pending_closures4452worker1 # [ 33.866272] nix-grpc-daemon[850]: time=2026-09-20T13:33:37.676Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4453worker1 # [ 33.867671] nix-grpc-daemon[850]: time=2026-09-20T13:33:37.676Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4454lb # [ 33.900794] niks3-server[1047]: 2026/09/20 13:33:37 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4455lb # [ 33.916549] niks3-server[1047]: 2026/09/20 13:33:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4456lb # [ 33.921171] niks3-server[1047]: 2026/09/20 13:33:37 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4457lb # [ 33.926600] niks3-server[1047]: 2026/09/20 13:33:37 INFO Signed narinfos id=6 count=14458worker1 # [ 33.922994] nix-grpc-daemon[850]: time=2026-09-20T13:33:37.734Z level=INFO msg="Uploading 1 narinfos"4459lb # [ 33.932977] niks3-server[1047]: 2026/09/20 13:33:37 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4460lb # [ 33.938242] niks3-server[1047]: 2026/09/20 13:33:37 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4461lb # [ 33.943458] niks3-server[1047]: 2026/09/20 13:33:37 INFO Completed upload id=64462worker1 # [ 33.936931] nix-grpc-daemon[850]: time=2026-09-20T13:33:37.748Z level=INFO msg="Upload complete. (284ms)"4463lb # [ 33.944902] niks3-server[1047]: 2026/09/20 13:33:37 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4464worker1 # [ 33.939737] nix-grpc-daemon[784]: ts=2026-09-20T13:33:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14465worker1 # [ 33.947115] nix-daemon[840]: reaped child process 942, status = succeeded4466client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 6.37 seconds)4467client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4468client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.21 seconds)4469client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4470client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.02 seconds)4471client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4472client # copying 3 paths...4473client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4474client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4475client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4476client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.28 seconds)4477(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 6.94 seconds)4478subtest: repeat is answered Cached by the scheduler without building4479worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4480worker1 # finding garbage collector roots...4481worker1 # deleting specified paths...4482worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4483worker1 # deleting unused links...4484worker1 # note: hard linking is currently saving 0.0 KiB4485worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.25 seconds)4486worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4487worker2 # finding garbage collector roots...4488worker2 # deleting specified paths...4489worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4490worker2 # deleting unused links...4491worker2 # note: hard linking is currently saving 0.0 KiB4492worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.22 seconds)4493client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24494client # don't know how to build these paths:4495client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4496worker2 # [ 35.214094] nix-grpc-daemon[787]: ts=2026-09-20T13:33:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504824497worker2 # [ 35.219855] nix-grpc-daemon[787]: ts=2026-09-20T13:33:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504824498worker2 # [ 35.295110] nix-grpc-daemon[787]: ts=2026-09-20T13:33:38Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14499client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.31 seconds)4500client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4501client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.18 seconds)4502lb: must succeed: curl -sf localhost:9901/clusters4503lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4504lb: must succeed: curl -sf localhost:9901/clusters4505lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4506worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4507worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4508(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.17 seconds)4509subtest: inputs already in the cache are not sent to the scheduler4510lb: must succeed: curl -sf localhost:9901/clusters4511lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4512lb: must succeed: curl -sf localhost:9901/clusters4513lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4514worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4515worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.04 seconds)4516client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24517worker2 # [ 36.038227] nix-grpc-daemon[787]: ts=2026-09-20T13:33:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504824518client # don't know how to build these paths:4519client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4520worker1 # [ 36.056485] nix-grpc-daemon[784]: ts=2026-09-20T13:33:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:561064521worker1 # [ 36.065103] nix-grpc-daemon[850]: time=2026-09-20T13:33:39.876Z level=INFO msg="All 2 paths already cached"4522worker2 # [ 36.069157] nix-grpc-daemon[787]: ts=2026-09-20T13:33:39Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04523worker2 # [ 36.073383] nix-grpc-daemon[787]: ts=2026-09-20T13:33:39Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14524worker1 # [ 36.122082] nix-grpc-daemon[784]: ts=2026-09-20T13:33:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064525worker1 # [ 36.134933] nix-grpc-daemon[784]: ts=2026-09-20T13:33:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:561064526worker1 # [ 36.146396] nix-grpc-daemon[850]: time=2026-09-20T13:33:39.958Z level=INFO msg="All 2 paths already cached"4527client # copying 1 paths to 'lb:50051'...4528worker1 # [ 36.154396] nix-grpc-daemon[784]: ts=2026-09-20T13:33:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:561064529client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4530worker1 # [ 36.165854] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4531worker1 # [ 36.334262] nix-daemon[840]: accepted connection from pid 980, user nix-grpc-daemon (trusted)4532worker1 # [ 36.348946] nix-daemon[840]: reaped child process 983, status = succeeded4533worker1 # [ 36.512077] nix-daemon[840]: accepted connection from pid 985, user nix-grpc-daemon (trusted)4534worker1 # [ 36.523476] nix-daemon[840]: reaped child process 988, status = succeeded4535lb # [ 36.547649] niks3-server[1047]: 2026/09/20 13:33:40 INFO Received uploads request method=POST path=/api/pending_closures4536worker1 # [ 36.550706] nix-grpc-daemon[850]: time=2026-09-20T13:33:40.362Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4537worker1 # [ 36.552357] nix-grpc-daemon[850]: time=2026-09-20T13:33:40.364Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4538lb # [ 36.581151] niks3-server[1047]: 2026/09/20 13:33:40 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4539lb # [ 36.595358] niks3-server[1047]: 2026/09/20 13:33:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4540lb # [ 36.602966] niks3-server[1047]: 2026/09/20 13:33:40 INFO Signed narinfos id=7 count=14541worker1 # [ 36.599335] nix-grpc-daemon[850]: time=2026-09-20T13:33:40.411Z level=INFO msg="Uploading 1 narinfos"4542lb # [ 36.607733] niks3-server[1047]: 2026/09/20 13:33:40 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4543lb # [ 36.618190] niks3-server[1047]: 2026/09/20 13:33:40 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4544lb # [ 36.625586] niks3-server[1047]: 2026/09/20 13:33:40 INFO Completed upload id=74545lb # [ 36.626784] niks3-server[1047]: 2026/09/20 13:33:40 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4546worker1 # [ 36.619331] nix-grpc-daemon[850]: time=2026-09-20T13:33:40.431Z level=INFO msg="Upload complete. (437ms)"4547worker1 # [ 36.624321] nix-grpc-daemon[784]: ts=2026-09-20T13:33:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=0 paths=1 nar_bytes_in=7524548worker1 # [ 36.630360] nix-grpc-daemon[784]: ts=2026-09-20T13:33:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064549worker1 # [ 36.641308] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4550worker1 # [ 36.651453] nix-daemon[840]: reaped child process 978, status = succeeded4551client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4552client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4553worker1 # [ 36.655150] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4554client # farm-top> LOG-farm-top-t1b4555worker1 # [ 37.481981] nix-daemon[840]: reaped child process 992, status = succeeded4556worker1 # [ 37.641899] nix-daemon[840]: accepted connection from pid 1008, user nix-grpc-daemon (trusted)4557worker1 # [ 37.654706] nix-daemon[840]: reaped child process 1011, status = succeeded4558lb # [ 37.670973] niks3-server[1047]: 2026/09/20 13:33:41 INFO Received uploads request method=POST path=/api/pending_closures4559worker1 # [ 37.673575] nix-grpc-daemon[850]: time=2026-09-20T13:33:41.485Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4560worker1 # [ 37.675143] nix-grpc-daemon[850]: time=2026-09-20T13:33:41.487Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4561lb # [ 37.698617] niks3-server[1047]: 2026/09/20 13:33:41 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4562lb # [ 37.701909] niks3-server[1047]: 2026/09/20 13:33:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4563lb # [ 37.704241] niks3-server[1047]: 2026/09/20 13:33:41 INFO Signed narinfos id=8 count=14564worker1 # [ 37.698030] nix-grpc-daemon[850]: time=2026-09-20T13:33:41.509Z level=INFO msg="Uploading 1 narinfos"4565lb # [ 37.708080] niks3-server[1047]: 2026/09/20 13:33:41 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4566lb # [ 37.712242] niks3-server[1047]: 2026/09/20 13:33:41 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4567lb # [ 37.716246] niks3-server[1047]: 2026/09/20 13:33:41 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4568lb # [ 37.719356] niks3-server[1047]: 2026/09/20 13:33:41 INFO Completed upload id=84569worker1 # [ 37.714051] nix-grpc-daemon[850]: time=2026-09-20T13:33:41.525Z level=INFO msg="Upload complete. (249ms)"4570lb # [ 37.720901] niks3-server[1047]: 2026/09/20 13:33:41 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4571worker1 # [ 37.716630] nix-grpc-daemon[784]: ts=2026-09-20T13:33:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14572worker1 # [ 37.723492] nix-daemon[840]: reaped child process 990, status = succeeded4573client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.88 seconds)4574client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4575client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.20 seconds)4576lb: must succeed: curl -sf localhost:9901/clusters4577lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4578lb: must succeed: curl -sf localhost:9901/clusters4579lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4580worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4581worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.04 seconds)4582(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.35 seconds)4583subtest: two clients wanting the same drv build it once4584client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4585client # [ 38.137257] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4586client # Running as unit: dedup1.service; invocation ID: ada44e8078824acda1b5a0663c5ff1684587client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.04 seconds)4588client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4589worker2 # [ 38.288433] nix-grpc-daemon[787]: ts=2026-09-20T13:33:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504824590client # [ 38.318847] nix[883]: don't know how to build these paths:4591client # [ 38.319695] nix[883]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4592worker2 # [ 38.296706] nix-grpc-daemon[787]: ts=2026-09-20T13:33:41Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04593worker2 # [ 38.299091] nix-grpc-daemon[787]: ts=2026-09-20T13:33:41Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=14594worker2 # [ 38.301383] nix-grpc-daemon[787]: ts=2026-09-20T13:33:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504824595worker2 # [ 38.310788] nix-grpc-daemon[787]: ts=2026-09-20T13:33:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504824596client # [ 38.344420] nix[883]: copying 1 paths to 'lb:50051'...4597worker2 # [ 38.317919] nix-grpc-daemon[787]: ts=2026-09-20T13:33:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504824598client # [ 38.345222] nix[883]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4599worker2 # [ 38.323906] nix-daemon[854]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4600worker2 # [ 38.334721] nix-grpc-daemon[787]: ts=2026-09-20T13:33:41Z level=info event=niks3_push_started pid=9054601worker2 # [ 38.576948] nix-grpc-daemon[905]: time=2026-09-20T13:33:41.861Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.key ca=""4602worker2 # [ 38.760992] nix-daemon[854]: accepted connection from pid 918, user nix-grpc-daemon (trusted)4603worker2 # [ 38.780501] nix-daemon[854]: reaped child process 921, status = succeeded4604worker2 # [ 38.946130] nix-daemon[854]: accepted connection from pid 923, user nix-grpc-daemon (trusted)4605worker2 # [ 38.958166] nix-daemon[854]: reaped child process 927, status = succeeded4606lb # [ 38.988261] niks3-server[1047]: 2026/09/20 13:33:42 INFO Received uploads request method=POST path=/api/pending_closures4607worker2 # [ 38.978434] nix-grpc-daemon[905]: time=2026-09-20T13:33:42.265Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4608worker2 # [ 38.981045] nix-grpc-daemon[905]: time=2026-09-20T13:33:42.268Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4609lb # [ 39.012866] niks3-server[1047]: 2026/09/20 13:33:42 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4610lb # [ 39.015558] niks3-server[1047]: 2026/09/20 13:33:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4611lb # [ 39.017387] niks3-server[1047]: 2026/09/20 13:33:42 INFO Signed narinfos id=9 count=14612worker2 # [ 39.004064] nix-grpc-daemon[905]: time=2026-09-20T13:33:42.291Z level=INFO msg="Uploading 1 narinfos"4613lb # [ 39.023800] niks3-server[1047]: 2026/09/20 13:33:42 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4614lb # [ 39.028169] niks3-server[1047]: 2026/09/20 13:33:42 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4615lb # [ 39.030809] niks3-server[1047]: 2026/09/20 13:33:42 INFO Completed upload id=94616worker2 # [ 39.016136] nix-grpc-daemon[905]: time=2026-09-20T13:33:42.303Z level=INFO msg="Upload complete. (421ms)"4617worker2 # [ 39.018212] nix-grpc-daemon[787]: ts=2026-09-20T13:33:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50482 duration_s=0 paths=1 nar_bytes_in=4324618lb # [ 39.033547] niks3-server[1047]: 2026/09/20 13:33:42 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4619worker2 # [ 39.021473] nix-grpc-daemon[787]: ts=2026-09-20T13:33:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504824620worker2 # [ 39.028237] nix-daemon[854]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4621worker2 # [ 39.030517] nix-daemon[854]: reaped child process 903, status = succeeded4622client # [ 39.061939] nix[883]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4623worker2 # [ 39.035897] nix-daemon[854]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4624client # [ 39.192754] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4625client # Running as unit: dedup2.service; invocation ID: d17cb68cae944e66802c36eecbebc5144626client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.06 seconds)4627lb # [ 39.383282] envoy[983]: conn peer=[2001:db8:1::1]:40776 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9468 tx=3842 ms=61904628lb # [ 39.384981] envoy[983]: conn peer=[2001:db8:1::1]:47760 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=954629lb # [ 39.386675] envoy[983]: conn peer=[2001:db8:1::1]:47772 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4169 tx=1728 ms=16844630worker1 # [ 39.455658] nix-grpc-daemon[784]: ts=2026-09-20T13:33:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:561064631client # [ 39.475368] nix[899]: don't know how to build these paths:4632client # [ 39.477054] nix[899]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4633worker2 # [ 39.458244] nix-grpc-daemon[787]: ts=2026-09-20T13:33:42Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04634worker2 # [ 39.461349] nix-grpc-daemon[787]: ts=2026-09-20T13:33:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504824635client # [ 39.488751] nix[899]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4636client: must succeed: sleep 24637client: (finished: must succeed: sleep 2, in 2.05 seconds)4638??? Warning (UserWarning): wait_until_succeeds(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead.4639 File "/nix/store/3gycnalwgpix76jjky7ljgh88gwgdb0l-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 394640client: waiting for success: ! systemctl is-active dedup1 dedup24641??? Warning (UserWarning): execute(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead.4642 File "/nix/store/3gycnalwgpix76jjky7ljgh88gwgdb0l-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 394643worker2 # [ 42.652340] nix-daemon[854]: reaped child process 931, status = succeeded4644worker2 # [ 42.809201] nix-daemon[854]: accepted connection from pid 963, user nix-grpc-daemon (trusted)4645worker2 # [ 42.821652] nix-daemon[854]: reaped child process 966, status = succeeded4646lb # [ 42.848288] niks3-server[1047]: 2026/09/20 13:33:46 INFO Received uploads request method=POST path=/api/pending_closures4647worker2 # [ 42.844505] nix-grpc-daemon[905]: time=2026-09-20T13:33:46.130Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4648worker2 # [ 42.846422] nix-grpc-daemon[905]: time=2026-09-20T13:33:46.130Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4649lb # [ 42.897064] niks3-server[1047]: 2026/09/20 13:33:46 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4650lb # [ 42.905333] niks3-server[1047]: 2026/09/20 13:33:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4651lb # [ 42.911434] niks3-server[1047]: 2026/09/20 13:33:46 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4652lb # [ 42.917841] niks3-server[1047]: 2026/09/20 13:33:46 INFO Signed narinfos id=10 count=14653worker2 # [ 42.903844] nix-grpc-daemon[905]: time=2026-09-20T13:33:46.190Z level=INFO msg="Uploading 1 narinfos"4654lb # [ 42.924804] niks3-server[1047]: 2026/09/20 13:33:46 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4655lb # [ 42.931059] niks3-server[1047]: 2026/09/20 13:33:46 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4656lb # [ 42.935598] niks3-server[1047]: 2026/09/20 13:33:46 INFO Completed upload id=104657worker2 # [ 42.921074] nix-grpc-daemon[905]: time=2026-09-20T13:33:46.208Z level=INFO msg="Upload complete. (275ms)"4658lb # [ 42.937592] niks3-server[1047]: 2026/09/20 13:33:46 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo4659worker2 # [ 42.927612] nix-grpc-daemon[787]: ts=2026-09-20T13:33:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50482 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14660worker2 # [ 42.934283] nix-grpc-daemon[787]: ts=2026-09-20T13:33:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50482 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14661worker2 # [ 42.943543] nix-daemon[854]: reaped child process 929, status = succeeded4662client # [ 42.980193] systemd[1]: dedup1.service: Deactivated successfully.4663client # [ 42.981944] systemd[1]: dedup1.service: Consumed 94ms CPU time over 4.844s wall clock time, 39.1M memory peak, 8.6K incoming IP traffic, 9.1K outgoing IP traffic.4664client # [ 43.026115] systemd[1]: dedup2.service: Deactivated successfully.4665client # [ 43.027532] systemd[1]: dedup2.service: Consumed 123ms CPU time over 3.835s wall clock time, 31.2M memory peak, 7.5K incoming IP traffic, 6.5K outgoing IP traffic.4666client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.10 seconds)4667client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4668client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)4669worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4670worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.03 seconds)4671worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4672worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.03 seconds)4673worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4674worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4675worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4676worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4677(finished: subtest: two clients wanting the same drv build it once, in 5.75 seconds)4678subtest: build hook: nix-daemon with builders = grpc://lb4679client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log4680client # [ 44.037821] systemd[1]: Started Nix Daemon.4681client # [ 44.192078] nix-daemon[943]: accepted connection from pid 940, user root (trusted)4682worker2 # [ 44.327846] nix-grpc-daemon[787]: ts=2026-09-20T13:33:47Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:504824683worker2 # [ 44.332048] nix-grpc-daemon[787]: ts=2026-09-20T13:33:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504824684worker2 # [ 44.340580] nix-grpc-daemon[787]: ts=2026-09-20T13:33:47Z level=debug event=want client=6 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=04685worker2 # [ 44.342450] nix-grpc-daemon[787]: ts=2026-09-20T13:33:47Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=14686worker1 # [ 44.393511] nix-grpc-daemon[784]: ts=2026-09-20T13:33:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064687worker1 # [ 44.403894] nix-grpc-daemon[784]: ts=2026-09-20T13:33:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:561064688worker1 # [ 44.413086] nix-grpc-daemon[784]: ts=2026-09-20T13:33:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:561064689worker1 # [ 44.421954] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4690worker1 # [ 44.597274] nix-daemon[840]: accepted connection from pid 1038, user nix-grpc-daemon (trusted)4691worker1 # [ 44.612280] nix-daemon[840]: reaped child process 1041, status = succeeded4692worker1 # [ 44.769331] nix-daemon[840]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)4693worker1 # [ 44.782549] nix-daemon[840]: reaped child process 1046, status = succeeded4694lb # [ 44.799583] niks3-server[1047]: 2026/09/20 13:33:48 INFO Received uploads request method=POST path=/api/pending_closures4695worker1 # [ 44.803307] nix-grpc-daemon[850]: time=2026-09-20T13:33:48.614Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4696worker1 # [ 44.805247] nix-grpc-daemon[850]: time=2026-09-20T13:33:48.616Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"4697lb # [ 44.828992] niks3-server[1047]: 2026/09/20 13:33:48 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst4698lb # [ 44.839652] niks3-server[1047]: 2026/09/20 13:33:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4699worker1 # [ 44.838624] nix-grpc-daemon[850]: time=2026-09-20T13:33:48.650Z level=INFO msg="Uploading 1 narinfos"4700lb # [ 44.844955] niks3-server[1047]: 2026/09/20 13:33:48 INFO Signed narinfos id=11 count=14701lb # [ 44.848602] niks3-server[1047]: 2026/09/20 13:33:48 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls4702lb # [ 44.853369] niks3-server[1047]: 2026/09/20 13:33:48 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4703lb # [ 44.857404] niks3-server[1047]: 2026/09/20 13:33:48 INFO Completed upload id=114704worker1 # [ 44.851161] nix-grpc-daemon[850]: time=2026-09-20T13:33:48.663Z level=INFO msg="Upload complete. (415ms)"4705lb # [ 44.858836] niks3-server[1047]: 2026/09/20 13:33:48 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo4706worker1 # [ 44.853165] nix-grpc-daemon[784]: ts=2026-09-20T13:33:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=0 paths=1 nar_bytes_in=5124707worker1 # [ 44.856911] nix-grpc-daemon[784]: ts=2026-09-20T13:33:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064708worker1 # [ 44.863517] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4709worker1 # [ 44.866543] nix-daemon[840]: reaped child process 1036, status = succeeded4710worker1 # [ 44.871406] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4711worker1 # [ 46.500606] nix-daemon[840]: reaped child process 1050, status = succeeded4712worker1 # [ 46.655095] nix-daemon[840]: accepted connection from pid 1065, user nix-grpc-daemon (trusted)4713worker1 # [ 46.667910] nix-daemon[840]: reaped child process 1068, status = succeeded4714lb # [ 46.688241] niks3-server[1047]: 2026/09/20 13:33:50 INFO Received uploads request method=POST path=/api/pending_closures4715worker1 # [ 46.688255] nix-grpc-daemon[850]: time=2026-09-20T13:33:50.499Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4716worker1 # [ 46.691871] nix-grpc-daemon[850]: time=2026-09-20T13:33:50.501Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"4717lb # [ 46.714569] niks3-server[1047]: 2026/09/20 13:33:50 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4718lb # [ 46.717738] niks3-server[1047]: 2026/09/20 13:33:50 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv4719lb # [ 46.720644] niks3-server[1047]: 2026/09/20 13:33:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4720lb # [ 46.722680] niks3-server[1047]: 2026/09/20 13:33:50 INFO Signed narinfos id=12 count=14721worker1 # [ 46.716665] nix-grpc-daemon[850]: time=2026-09-20T13:33:50.528Z level=INFO msg="Uploading 1 narinfos"4722lb # [ 46.726052] niks3-server[1047]: 2026/09/20 13:33:50 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls4723lb # [ 46.731371] niks3-server[1047]: 2026/09/20 13:33:50 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4724lb # [ 46.734688] niks3-server[1047]: 2026/09/20 13:33:50 INFO Completed upload id=124725worker1 # [ 46.729410] nix-grpc-daemon[850]: time=2026-09-20T13:33:50.541Z level=INFO msg="Upload complete. (249ms)"4726lb # [ 46.736385] niks3-server[1047]: 2026/09/20 13:33:50 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo4727worker1 # [ 46.733167] nix-grpc-daemon[784]: ts=2026-09-20T13:33:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=6 outputs=14728worker1 # [ 46.736224] nix-grpc-daemon[784]: ts=2026-09-20T13:33:50Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:561064729worker1 # [ 46.741586] nix-daemon[840]: reaped child process 1048, status = succeeded4730worker2 # [ 46.742496] nix-grpc-daemon[787]: ts=2026-09-20T13:33:50Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:504824731worker2 # [ 46.753189] nix-grpc-daemon[787]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...4732worker2 # [ 46.762265] nix-grpc-daemon[787]: ts=2026-09-20T13:33:50Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50482 duration_s=0 paths=1 nar_bytes_out=1284733worker2 # [ 47.023655] nix-grpc-daemon[787]: ts=2026-09-20T13:33:50Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:504824734worker2 # [ 47.029451] nix-grpc-daemon[787]: ts=2026-09-20T13:33:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504824735worker2 # [ 47.034795] nix-grpc-daemon[905]: time=2026-09-20T13:33:50.320Z level=INFO msg="All 1 paths already cached"4736worker2 # [ 47.043632] nix-grpc-daemon[787]: ts=2026-09-20T13:33:50Z level=debug event=want client=7 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=04737worker1 # [ 47.054310] nix-grpc-daemon[784]: ts=2026-09-20T13:33:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064738worker2 # [ 47.047646] nix-grpc-daemon[787]: ts=2026-09-20T13:33:50Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=14739worker1 # [ 47.063835] nix-grpc-daemon[784]: ts=2026-09-20T13:33:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:561064740worker1 # [ 47.073739] nix-grpc-daemon[850]: time=2026-09-20T13:33:50.885Z level=INFO msg="All 1 paths already cached"4741worker1 # [ 47.079349] nix-grpc-daemon[784]: ts=2026-09-20T13:33:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:561064742worker1 # [ 47.089162] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4743worker1 # [ 47.269568] nix-daemon[840]: accepted connection from pid 1072, user nix-grpc-daemon (trusted)4744worker1 # [ 47.283342] nix-daemon[840]: reaped child process 1075, status = succeeded4745worker1 # [ 47.448302] nix-daemon[840]: accepted connection from pid 1077, user nix-grpc-daemon (trusted)4746worker1 # [ 47.460820] nix-daemon[840]: reaped child process 1080, status = succeeded4747lb # [ 47.481630] niks3-server[1047]: 2026/09/20 13:33:51 INFO Received uploads request method=POST path=/api/pending_closures4748worker1 # [ 47.482570] nix-grpc-daemon[850]: time=2026-09-20T13:33:51.294Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4749worker1 # [ 47.484130] nix-grpc-daemon[850]: time=2026-09-20T13:33:51.296Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"4750lb # [ 47.501158] niks3-server[1047]: 2026/09/20 13:33:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4751worker1 # [ 47.496885] nix-grpc-daemon[850]: time=2026-09-20T13:33:51.308Z level=INFO msg="Uploading 1 narinfos"4752lb # [ 47.503844] niks3-server[1047]: 2026/09/20 13:33:51 INFO Signed narinfos id=13 count=14753lb # [ 47.507860] niks3-server[1047]: 2026/09/20 13:33:51 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst4754lb # [ 47.510590] niks3-server[1047]: 2026/09/20 13:33:51 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls4755lb # [ 47.514059] niks3-server[1047]: 2026/09/20 13:33:51 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4756lb # [ 47.517622] niks3-server[1047]: 2026/09/20 13:33:51 INFO Completed upload id=134757worker1 # [ 47.512500] nix-grpc-daemon[850]: time=2026-09-20T13:33:51.324Z level=INFO msg="Upload complete. (410ms)"4758lb # [ 47.519519] niks3-server[1047]: 2026/09/20 13:33:51 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo4759worker1 # [ 47.514394] nix-grpc-daemon[784]: ts=2026-09-20T13:33:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=0 paths=1 nar_bytes_in=6244760worker1 # [ 47.517515] nix-grpc-daemon[784]: ts=2026-09-20T13:33:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064761worker1 # [ 47.524543] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4762worker1 # [ 47.527573] nix-daemon[840]: reaped child process 1070, status = succeeded4763worker1 # [ 47.532447] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4764worker1 # [ 48.554436] nix-daemon[840]: reaped child process 1084, status = succeeded4765worker1 # [ 48.697097] nix-daemon[840]: accepted connection from pid 1099, user nix-grpc-daemon (trusted)4766worker1 # [ 48.708449] nix-daemon[840]: reaped child process 1102, status = succeeded4767lb # [ 48.727986] niks3-server[1047]: 2026/09/20 13:33:52 INFO Received uploads request method=POST path=/api/pending_closures4768worker1 # [ 48.730429] nix-grpc-daemon[850]: time=2026-09-20T13:33:52.542Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4769worker1 # [ 48.732820] nix-grpc-daemon[850]: time=2026-09-20T13:33:52.543Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"4770lb # [ 48.765365] niks3-server[1047]: 2026/09/20 13:33:52 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv4771lb # [ 48.775554] niks3-server[1047]: 2026/09/20 13:33:52 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst4772lb # [ 48.787165] niks3-server[1047]: 2026/09/20 13:33:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4773lb # [ 48.790804] niks3-server[1047]: 2026/09/20 13:33:52 INFO Signed narinfos id=14 count=14774worker1 # [ 48.786872] nix-grpc-daemon[850]: time=2026-09-20T13:33:52.598Z level=INFO msg="Uploading 1 narinfos"4775lb # [ 48.796280] niks3-server[1047]: 2026/09/20 13:33:52 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls4776lb # [ 48.805801] niks3-server[1047]: 2026/09/20 13:33:52 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4777lb # [ 48.809091] niks3-server[1047]: 2026/09/20 13:33:52 INFO Completed upload id=144778lb # [ 48.810758] niks3-server[1047]: 2026/09/20 13:33:52 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo4779worker1 # [ 48.804128] nix-grpc-daemon[850]: time=2026-09-20T13:33:52.615Z level=INFO msg="Upload complete. (264ms)"4780worker1 # [ 48.810714] nix-grpc-daemon[784]: ts=2026-09-20T13:33:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=7 outputs=14781worker2 # [ 48.805816] nix-grpc-daemon[787]: ts=2026-09-20T13:33:52Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:504824782worker2 # [ 48.811508] nix-grpc-daemon[787]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...4783worker1 # [ 48.823356] nix-daemon[840]: reaped child process 1082, status = succeeded4784worker1 # [ 48.840497] nix-grpc-daemon[784]: ts=2026-09-20T13:33:52Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:561064785worker1 # [ 48.845133] nix-grpc-daemon[784]: ts=2026-09-20T13:33:52Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=0 paths=1 nar_bytes_out=1764786worker2 # [ 49.102580] nix-grpc-daemon[787]: ts=2026-09-20T13:33:52Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:504824787worker2 # [ 49.108718] nix-grpc-daemon[787]: ts=2026-09-20T13:33:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504824788worker2 # [ 49.114180] nix-grpc-daemon[905]: time=2026-09-20T13:33:52.399Z level=INFO msg="All 2 paths already cached"4789worker2 # [ 49.122884] nix-grpc-daemon[787]: ts=2026-09-20T13:33:52Z level=debug event=want client=8 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=04790worker2 # [ 49.124758] nix-grpc-daemon[787]: ts=2026-09-20T13:33:52Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=14791worker1 # [ 49.133877] nix-grpc-daemon[784]: ts=2026-09-20T13:33:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064792worker1 # [ 49.146076] nix-grpc-daemon[784]: ts=2026-09-20T13:33:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:561064793worker1 # [ 49.166069] nix-grpc-daemon[850]: time=2026-09-20T13:33:52.977Z level=INFO msg="All 2 paths already cached"4794worker1 # [ 49.174153] nix-grpc-daemon[784]: ts=2026-09-20T13:33:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:561064795worker1 # [ 49.185155] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4796worker1 # [ 49.346165] nix-daemon[840]: accepted connection from pid 1106, user nix-grpc-daemon (trusted)4797worker1 # [ 49.360413] nix-daemon[840]: reaped child process 1109, status = succeeded4798lb # [ 49.383399] envoy[983]: conn peer=[2001:db8:1::1]:47774 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2972 tx=1371 ms=46564799lb # [ 49.387238] envoy[983]: conn peer=[2001:db8:1::1]:47790 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1440 tx=1085 ms=35444800lb # [ 49.390713] envoy[983]: conn peer=[2001:db8:1::1]:39016 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3401 tx=1660 ms=24864801lb # [ 49.394205] envoy[983]: conn peer=[2001:db8:1::1]:39030 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=334 ms=734802lb # [ 49.397627] envoy[983]: conn peer=[2001:db8:1::1]:39036 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3893 tx=1986 ms=18654803lb # [ 49.401857] envoy[983]: conn peer=[2001:db8:1::1]:39044 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=574804worker1 # [ 49.492182] nix-daemon[840]: accepted connection from pid 1111, user nix-grpc-daemon (trusted)4805worker1 # [ 49.505380] nix-daemon[840]: reaped child process 1114, status = succeeded4806lb # [ 49.521755] niks3-server[1047]: 2026/09/20 13:33:53 INFO Received uploads request method=POST path=/api/pending_closures4807worker1 # [ 49.522917] nix-grpc-daemon[850]: time=2026-09-20T13:33:53.334Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4808worker1 # [ 49.524783] nix-grpc-daemon[850]: time=2026-09-20T13:33:53.336Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"4809lb # [ 49.552749] niks3-server[1047]: 2026/09/20 13:33:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4810lb # [ 49.557098] niks3-server[1047]: 2026/09/20 13:33:53 INFO Signed narinfos id=15 count=14811worker1 # [ 49.553826] nix-grpc-daemon[850]: time=2026-09-20T13:33:53.365Z level=INFO msg="Uploading 1 narinfos"4812lb # [ 49.564067] niks3-server[1047]: 2026/09/20 13:33:53 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst4813lb # [ 49.572972] niks3-server[1047]: 2026/09/20 13:33:53 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls4814lb # [ 49.583467] niks3-server[1047]: 2026/09/20 13:33:53 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4815worker1 # [ 49.581749] nix-grpc-daemon[850]: time=2026-09-20T13:33:53.393Z level=INFO msg="Upload complete. (383ms)"4816lb # [ 49.588272] niks3-server[1047]: 2026/09/20 13:33:53 INFO Completed upload id=154817lb # [ 49.592459] niks3-server[1047]: 2026/09/20 13:33:53 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo4818worker1 # [ 49.585090] nix-grpc-daemon[784]: ts=2026-09-20T13:33:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=0 paths=1 nar_bytes_in=7524819worker1 # [ 49.590459] nix-grpc-daemon[784]: ts=2026-09-20T13:33:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064820worker1 # [ 49.598452] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4821worker1 # [ 49.604158] nix-daemon[840]: reaped child process 1104, status = succeeded4822worker1 # [ 49.609297] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4823worker1 # [ 51.024973] nix-daemon[840]: reaped child process 1118, status = succeeded4824worker1 # [ 51.154098] nix-daemon[840]: accepted connection from pid 1133, user nix-grpc-daemon (trusted)4825worker1 # [ 51.166402] nix-daemon[840]: reaped child process 1136, status = succeeded4826lb # [ 51.187634] niks3-server[1047]: 2026/09/20 13:33:54 INFO Received uploads request method=POST path=/api/pending_closures4827worker1 # [ 51.185590] nix-grpc-daemon[850]: time=2026-09-20T13:33:54.997Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4828worker1 # [ 51.188060] nix-grpc-daemon[850]: time=2026-09-20T13:33:54.999Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"4829lb # [ 51.210846] niks3-server[1047]: 2026/09/20 13:33:54 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv4830lb # [ 51.212983] niks3-server[1047]: 2026/09/20 13:33:54 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst4831lb # [ 51.217876] niks3-server[1047]: 2026/09/20 13:33:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4832lb # [ 51.219534] niks3-server[1047]: 2026/09/20 13:33:54 INFO Signed narinfos id=16 count=14833worker1 # [ 51.214479] nix-grpc-daemon[850]: time=2026-09-20T13:33:55.026Z level=INFO msg="Uploading 1 narinfos"4834lb # [ 51.223545] niks3-server[1047]: 2026/09/20 13:33:54 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls4835lb # [ 51.227664] niks3-server[1047]: 2026/09/20 13:33:54 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4836lb # [ 51.231937] niks3-server[1047]: 2026/09/20 13:33:54 INFO Completed upload id=164837worker1 # [ 51.225468] nix-grpc-daemon[850]: time=2026-09-20T13:33:55.037Z level=INFO msg="Upload complete. (209ms)"4838lb # [ 51.233493] niks3-server[1047]: 2026/09/20 13:33:54 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo4839worker1 # [ 51.227902] nix-grpc-daemon[784]: ts=2026-09-20T13:33:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=8 outputs=14840worker1 # [ 51.234419] nix-grpc-daemon[784]: ts=2026-09-20T13:33:55Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:561064841worker1 # [ 51.236853] nix-daemon[840]: reaped child process 1116, status = succeeded4842worker2 # [ 51.241718] nix-grpc-daemon[787]: ts=2026-09-20T13:33:54Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:504824843worker2 # [ 51.249544] nix-grpc-daemon[787]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...4844worker2 # [ 51.258666] nix-grpc-daemon[787]: ts=2026-09-20T13:33:54Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50482 duration_s=0 paths=1 nar_bytes_out=2324845client # [ 51.396923] nix-daemon[943]: reaped child process 945, status = succeeded4846client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.55 seconds)4847client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top4848client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.03 seconds)4849client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4850client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.03 seconds)4851client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4852client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.02 seconds)4853(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.63 seconds)4854subtest: an input only one worker has still reaches the builder4855client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only4856client # this derivation will be built:4857client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv4858client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...4859client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.37 seconds)4860client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure4861client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.17 seconds)4862worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4863worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.19 seconds)4864worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only4865worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)4866client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&24867client # [ 52.420419] nix-daemon[943]: accepted connection from pid 1026, user root (trusted)4868client # this derivation will be built:4869client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv4870worker2 # [ 52.566650] nix-grpc-daemon[787]: ts=2026-09-20T13:33:55Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:504824871client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux'...4872worker1 # [ 52.578657] nix-grpc-daemon[784]: ts=2026-09-20T13:33:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:561064873worker1 # [ 52.752660] nix-daemon[840]: accepted connection from pid 1143, user nix-grpc-daemon (trusted)4874worker1 # [ 52.764175] nix-daemon[840]: reaped child process 1146, status = succeeded4875lb # [ 52.780949] niks3-server[1047]: 2026/09/20 13:33:56 INFO Received uploads request method=POST path=/api/pending_closures4876worker1 # [ 52.779145] nix-grpc-daemon[850]: time=2026-09-20T13:33:56.590Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4877worker1 # [ 52.780592] nix-grpc-daemon[850]: time=2026-09-20T13:33:56.590Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"4878lb # [ 52.794326] niks3-server[1047]: 2026/09/20 13:33:56 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4879lb # [ 52.798059] niks3-server[1047]: 2026/09/20 13:33:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4880lb # [ 52.800590] niks3-server[1047]: 2026/09/20 13:33:56 INFO Signed narinfos id=17 count=14881worker1 # [ 52.794354] nix-grpc-daemon[850]: time=2026-09-20T13:33:56.606Z level=INFO msg="Uploading 1 narinfos"4882lb # [ 52.806681] niks3-server[1047]: 2026/09/20 13:33:56 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls4883lb # [ 52.810607] niks3-server[1047]: 2026/09/20 13:33:56 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4884lb # [ 52.813793] niks3-server[1047]: 2026/09/20 13:33:56 INFO Completed upload id=174885worker1 # [ 52.807363] nix-grpc-daemon[850]: time=2026-09-20T13:33:56.619Z level=INFO msg="Upload complete. (223ms)"4886lb # [ 52.816354] niks3-server[1047]: 2026/09/20 13:33:56 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo4887worker2 # [ 52.813113] nix-grpc-daemon[787]: ts=2026-09-20T13:33:56Z level=debug event=want client=9 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=04888worker2 # [ 52.816076] nix-grpc-daemon[787]: ts=2026-09-20T13:33:56Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14889worker2 # [ 52.857138] nix-grpc-daemon[787]: ts=2026-09-20T13:33:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504824890worker2 # [ 52.868084] nix-grpc-daemon[787]: ts=2026-09-20T13:33:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504824891client # copying 2 paths to 'lb:50051'...4892client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...4893worker2 # [ 52.879948] nix-grpc-daemon[787]: ts=2026-09-20T13:33:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504824894client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...4895worker2 # [ 52.894540] nix-daemon[854]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4896worker2 # [ 53.074303] nix-daemon[854]: accepted connection from pid 993, user nix-grpc-daemon (trusted)4897worker2 # [ 53.088351] nix-daemon[854]: reaped child process 996, status = succeeded4898worker2 # [ 53.248176] nix-daemon[854]: accepted connection from pid 998, user nix-grpc-daemon (trusted)4899worker2 # [ 53.260440] nix-daemon[854]: reaped child process 1001, status = succeeded4900lb # [ 53.285350] niks3-server[1047]: 2026/09/20 13:33:56 INFO Received uploads request method=POST path=/api/pending_closures4901lb # [ 53.293694] niks3-server[1047]: 2026/09/20 13:33:56 INFO Received uploads request method=POST path=/api/pending_closures4902worker2 # [ 53.286156] nix-grpc-daemon[905]: time=2026-09-20T13:33:56.572Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4903worker2 # [ 53.287789] nix-grpc-daemon[905]: time=2026-09-20T13:33:56.574Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"4904worker2 # [ 53.290187] nix-grpc-daemon[905]: time=2026-09-20T13:33:56.574Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"4905lb # [ 53.324149] niks3-server[1047]: 2026/09/20 13:33:56 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst4906lb # [ 53.340583] niks3-server[1047]: 2026/09/20 13:33:56 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst4907lb # [ 53.353243] niks3-server[1047]: 2026/09/20 13:33:56 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls4908lb # [ 53.359268] niks3-server[1047]: 2026/09/20 13:33:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4909lb # [ 53.362808] niks3-server[1047]: 2026/09/20 13:33:56 INFO Signed narinfos id=19 count=24910lb # [ 53.364859] niks3-server[1047]: 2026/09/20 13:33:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4911lb # [ 53.366753] niks3-server[1047]: 2026/09/20 13:33:56 INFO Signed narinfos id=18 count=14912worker2 # [ 53.353772] nix-grpc-daemon[905]: time=2026-09-20T13:33:56.640Z level=INFO msg="Uploading 3 narinfos"4913lb # [ 53.368207] niks3-server[1047]: 2026/09/20 13:33:56 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls4914lb # [ 53.381947] niks3-server[1047]: 2026/09/20 13:33:56 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4915lb # [ 53.385184] niks3-server[1047]: 2026/09/20 13:33:57 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo4916lb # [ 53.387942] niks3-server[1047]: 2026/09/20 13:33:57 INFO Completed upload id=184917lb # [ 53.390284] niks3-server[1047]: 2026/09/20 13:33:57 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4918lb # [ 53.392187] niks3-server[1047]: 2026/09/20 13:33:57 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo4919lb # [ 53.394667] niks3-server[1047]: 2026/09/20 13:33:57 INFO Completed upload id=194920worker2 # [ 53.379915] nix-grpc-daemon[905]: time=2026-09-20T13:33:56.666Z level=INFO msg="Upload complete. (470ms)"4921lb # [ 53.396130] niks3-server[1047]: 2026/09/20 13:33:57 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo4922worker2 # [ 53.381964] nix-grpc-daemon[787]: ts=2026-09-20T13:33:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50482 duration_s=0 paths=2 nar_bytes_in=9764923worker2 # [ 53.386387] nix-grpc-daemon[787]: ts=2026-09-20T13:33:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504824924worker2 # [ 53.391975] nix-daemon[854]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4925worker2 # [ 53.394218] nix-daemon[854]: reaped child process 991, status = succeeded4926worker2 # [ 53.400769] nix-grpc-daemon[787]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...4927worker2 # [ 53.407436] nix-daemon[854]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4928client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...4929client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv4930worker2 # [ 54.219770] nix-daemon[854]: reaped child process 1006, status = succeeded4931worker2 # [ 54.350792] nix-daemon[854]: accepted connection from pid 1021, user nix-grpc-daemon (trusted)4932worker2 # [ 54.362922] nix-daemon[854]: reaped child process 1024, status = succeeded4933lb # [ 54.392540] niks3-server[1047]: 2026/09/20 13:33:58 INFO Received uploads request method=POST path=/api/pending_closures4934worker2 # [ 54.386216] nix-grpc-daemon[905]: time=2026-09-20T13:33:57.672Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4935lb # [ 54.426375] niks3-server[1047]: 2026/09/20 13:33:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4936lb # [ 54.432289] niks3-server[1047]: 2026/09/20 13:33:58 INFO Signed narinfos id=20 count=14937worker2 # [ 54.420558] nix-grpc-daemon[905]: time=2026-09-20T13:33:57.707Z level=INFO msg="Uploading 1 narinfos"4938lb # [ 54.439602] niks3-server[1047]: 2026/09/20 13:33:58 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls4939lb # [ 54.449668] niks3-server[1047]: 2026/09/20 13:33:58 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv4940lb # [ 54.461933] niks3-server[1047]: 2026/09/20 13:33:58 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4941worker2 # [ 54.450605] nix-grpc-daemon[905]: time=2026-09-20T13:33:57.737Z level=INFO msg="Upload complete. (237ms)"4942lb # [ 54.465549] niks3-server[1047]: 2026/09/20 13:33:58 INFO Completed upload id=204943worker2 # [ 54.453577] nix-grpc-daemon[787]: ts=2026-09-20T13:33:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50482 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=9 outputs=14944client # copying 1 paths...4945worker2 # [ 54.458177] nix-grpc-daemon[905]: time=2026-09-20T13:33:57.743Z level=WARN msg="Failed to register uploaded object" key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo error="server returned 404: object is not pending upload\n"4946worker2 # [ 54.465407] nix-daemon[854]: reaped child process 1003, status = succeeded4947worker1 # [ 54.470857] nix-grpc-daemon[784]: ts=2026-09-20T13:33:58Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:561064948worker1 # [ 54.482335] nix-grpc-daemon[784]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...4949client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...4950worker1 # [ 54.519140] nix-grpc-daemon[784]: ts=2026-09-20T13:33:58Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:561064951worker1 # [ 54.523793] nix-grpc-daemon[784]: ts=2026-09-20T13:33:58Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=0 paths=1 nar_bytes_out=1204952client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.41 seconds)4953client # [ 54.656240] nix-daemon[943]: reaped child process 1029, status = succeeded4954client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&24955client # [ 54.806388] nix-daemon[943]: accepted connection from pid 1048, user root (trusted)4956client # this derivation will be built:4957client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv4958worker2 # [ 54.943784] nix-grpc-daemon[787]: ts=2026-09-20T13:33:58Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:504824959worker2 # [ 54.946584] nix-grpc-daemon[787]: ts=2026-09-20T13:33:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504824960client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux'...4961worker2 # [ 54.963906] nix-grpc-daemon[905]: time=2026-09-20T13:33:58.250Z level=INFO msg="All 1 paths already cached"4962worker2 # [ 54.974459] nix-grpc-daemon[787]: ts=2026-09-20T13:33:58Z level=debug event=want client=10 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=04963worker2 # [ 54.976505] nix-grpc-daemon[787]: ts=2026-09-20T13:33:58Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=14964worker1 # [ 55.027639] nix-grpc-daemon[784]: ts=2026-09-20T13:33:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064965worker1 # [ 55.037708] nix-grpc-daemon[784]: ts=2026-09-20T13:33:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:561064966client # copying 1 paths to 'lb:50051'...4967worker1 # [ 55.054113] nix-grpc-daemon[784]: ts=2026-09-20T13:33:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:561064968client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...4969worker1 # [ 55.062834] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4970worker1 # [ 55.071179] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 72 ms (attempt 1/5)4971worker1 # [ 55.145914] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 158 ms (attempt 2/5)4972worker1 # [ 55.306720] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 257 ms (attempt 3/5)4973worker1 # [ 55.565295] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 685 ms (attempt 4/5)4974worker1 # [ 56.251252] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4975worker1 # [ 56.257125] nix-grpc-daemon[784]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...4976worker1 # [ 56.471060] nix-daemon[840]: accepted connection from pid 1155, user nix-grpc-daemon (trusted)4977worker1 # [ 56.484117] nix-daemon[840]: reaped child process 1158, status = succeeded4978worker1 # [ 56.650721] nix-daemon[840]: accepted connection from pid 1160, user nix-grpc-daemon (trusted)4979worker1 # [ 56.663807] nix-daemon[840]: reaped child process 1163, status = succeeded4980lb # [ 56.683996] niks3-server[1047]: 2026/09/20 13:34:00 INFO Received uploads request method=POST path=/api/pending_closures4981worker1 # [ 56.686357] nix-grpc-daemon[850]: time=2026-09-20T13:34:00.498Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4982worker1 # [ 56.688196] nix-grpc-daemon[850]: time=2026-09-20T13:34:00.499Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"4983lb # [ 56.713843] niks3-server[1047]: 2026/09/20 13:34:00 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst4984lb # [ 56.724136] niks3-server[1047]: 2026/09/20 13:34:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4985worker1 # [ 56.722447] nix-grpc-daemon[850]: time=2026-09-20T13:34:00.534Z level=INFO msg="Uploading 1 narinfos"4986lb # [ 56.728764] niks3-server[1047]: 2026/09/20 13:34:00 INFO Signed narinfos id=21 count=14987lb # [ 56.737467] niks3-server[1047]: 2026/09/20 13:34:00 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls4988lb # [ 56.747631] niks3-server[1047]: 2026/09/20 13:34:00 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4989worker1 # [ 56.747164] nix-grpc-daemon[850]: time=2026-09-20T13:34:00.559Z level=INFO msg="Upload complete. (469ms)"4990lb # [ 56.753618] niks3-server[1047]: 2026/09/20 13:34:00 INFO Completed upload id=214991worker1 # [ 56.749428] nix-grpc-daemon[784]: ts=2026-09-20T13:34:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=1 paths=1 nar_bytes_in=5604992lb # [ 56.756802] niks3-server[1047]: 2026/09/20 13:34:00 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo4993worker1 # [ 56.754523] nix-grpc-daemon[784]: ts=2026-09-20T13:34:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:561064994worker1 # [ 56.760488] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4995worker1 # [ 56.762757] nix-daemon[840]: reaped child process 1149, status = succeeded4996worker1 # [ 56.768051] nix-daemon[840]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4997client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...4998client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv4999worker1 # [ 57.994969] nix-daemon[840]: reaped child process 1167, status = succeeded5000worker1 # [ 58.158885] nix-daemon[840]: accepted connection from pid 1182, user nix-grpc-daemon (trusted)5001worker1 # [ 58.170549] nix-daemon[840]: reaped child process 1185, status = succeeded5002lb # [ 58.193943] niks3-server[1047]: 2026/09/20 13:34:01 INFO Received uploads request method=POST path=/api/pending_closures5003worker1 # [ 58.438855] nix-grpc-daemon[850]: time=2026-09-20T13:34:02.247Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5004lb # [ 58.464255] niks3-server[1047]: 2026/09/20 13:34:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5005lb # [ 58.472936] niks3-server[1047]: 2026/09/20 13:34:02 INFO Signed narinfos id=22 count=15006worker1 # [ 58.466570] nix-grpc-daemon[850]: time=2026-09-20T13:34:02.278Z level=INFO msg="Uploading 1 narinfos"5007lb # [ 58.479602] niks3-server[1047]: 2026/09/20 13:34:02 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5008lb # [ 58.489464] niks3-server[1047]: 2026/09/20 13:34:02 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5009lb # [ 58.496306] niks3-server[1047]: 2026/09/20 13:34:02 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5010lb # [ 58.526440] niks3-server[1047]: 2026/09/20 13:34:02 INFO Completed upload id=225011worker1 # [ 58.520445] nix-grpc-daemon[850]: time=2026-09-20T13:34:02.332Z level=INFO msg="Upload complete. (543ms)"5012lb # [ 58.529896] niks3-server[1047]: 2026/09/20 13:34:02 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5013client # copying 1 paths...5014worker1 # [ 58.527319] nix-grpc-daemon[784]: ts=2026-09-20T13:34:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56106 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=10 outputs=15015worker1 # [ 58.534606] nix-grpc-daemon[784]: ts=2026-09-20T13:34:02Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:561065016client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5017worker1 # [ 58.545908] nix-daemon[840]: reaped child process 1165, status = succeeded5018worker2 # [ 58.544777] nix-grpc-daemon[787]: ts=2026-09-20T13:34:01Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:504825019worker2 # [ 58.556632] nix-grpc-daemon[787]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5020worker2 # [ 58.569670] nix-grpc-daemon[787]: ts=2026-09-20T13:34:01Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50482 duration_s=0 paths=1 nar_bytes_out=1205021client # [ 58.702160] nix-daemon[943]: reaped child process 1051, status = succeeded5022client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 4.06 seconds)5023client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25024client # [ 58.874368] nix-daemon[943]: accepted connection from pid 1070, user root (trusted)5025client # this derivation will be built:5026client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5027worker2 # [ 59.006662] nix-grpc-daemon[787]: ts=2026-09-20T13:34:02Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:504825028client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux'...5029worker1 # [ 59.018832] nix-grpc-daemon[784]: ts=2026-09-20T13:34:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:561065030worker1 # [ 59.044739] nix-grpc-daemon[850]: time=2026-09-20T13:34:02.856Z level=INFO msg="All 1 paths already cached"5031worker2 # [ 59.051202] nix-grpc-daemon[787]: ts=2026-09-20T13:34:02Z level=debug event=want client=11 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05032worker2 # [ 59.056879] nix-grpc-daemon[787]: ts=2026-09-20T13:34:02Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15033worker2 # [ 59.062406] nix-grpc-daemon[787]: ts=2026-09-20T13:34:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504825034worker2 # [ 59.074450] nix-grpc-daemon[787]: ts=2026-09-20T13:34:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504825035worker2 # [ 59.088194] nix-grpc-daemon[905]: time=2026-09-20T13:34:02.374Z level=INFO msg="All 1 paths already cached"5036client # copying 1 paths to 'lb:50051'...5037client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5038worker2 # [ 59.096174] nix-grpc-daemon[787]: ts=2026-09-20T13:34:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504825039worker2 # [ 59.104865] nix-daemon[854]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5040worker2 # [ 59.272590] nix-daemon[854]: accepted connection from pid 1033, user nix-grpc-daemon (trusted)5041worker2 # [ 59.285520] nix-daemon[854]: reaped child process 1036, status = succeeded5042lb # [ 59.382986] envoy[983]: conn peer=[2001:db8:1::1]:39052 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4298 tx=2090 ms=22115043lb # [ 59.386130] envoy[983]: conn peer=[2001:db8:1::1]:39066 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=411 ms=705044lb # [ 59.388883] envoy[983]: conn peer=[2001:db8:1::1]:39068 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3827 tx=1876 ms=20025045lb # [ 59.391668] envoy[983]: conn peer=[2001:db8:1::1]:48314 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=635046lb # [ 59.394390] envoy[983]: conn peer=[2001:db8:1::1]:48318 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3701 tx=1797 ms=36795047lb # [ 59.397209] envoy[983]: conn peer=[2001:db8:1::1]:48334 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=329 ms=795048worker2 # [ 59.443311] nix-daemon[854]: accepted connection from pid 1038, user nix-grpc-daemon (trusted)5049worker2 # [ 59.456351] nix-daemon[854]: reaped child process 1041, status = succeeded5050lb # [ 59.486083] niks3-server[1047]: 2026/09/20 13:34:03 INFO Received uploads request method=POST path=/api/pending_closures5051worker2 # [ 59.480777] nix-grpc-daemon[905]: time=2026-09-20T13:34:02.767Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5052worker2 # [ 59.482262] nix-grpc-daemon[905]: time=2026-09-20T13:34:02.769Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5053lb # [ 59.515091] niks3-server[1047]: 2026/09/20 13:34:03 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5054lb # [ 59.523318] niks3-server[1047]: 2026/09/20 13:34:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5055worker2 # [ 59.512237] nix-grpc-daemon[905]: time=2026-09-20T13:34:02.799Z level=INFO msg="Uploading 1 narinfos"5056lb # [ 59.527092] niks3-server[1047]: 2026/09/20 13:34:03 INFO Signed narinfos id=23 count=15057lb # [ 59.532963] niks3-server[1047]: 2026/09/20 13:34:03 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5058lb # [ 59.541695] niks3-server[1047]: 2026/09/20 13:34:03 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5059lb # [ 59.546913] niks3-server[1047]: 2026/09/20 13:34:03 INFO Completed upload id=235060worker2 # [ 59.532266] nix-grpc-daemon[905]: time=2026-09-20T13:34:02.819Z level=INFO msg="Upload complete. (412ms)"5061worker2 # [ 59.534544] nix-grpc-daemon[787]: ts=2026-09-20T13:34:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50482 duration_s=0 paths=1 nar_bytes_in=5605062worker2 # [ 59.537712] nix-grpc-daemon[787]: ts=2026-09-20T13:34:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504825063lb # [ 59.555675] niks3-server[1047]: 2026/09/20 13:34:03 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5064worker2 # [ 59.544556] nix-daemon[854]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5065worker2 # [ 59.547578] nix-daemon[854]: reaped child process 1031, status = succeeded5066worker2 # [ 59.552783] nix-daemon[854]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5067client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5068client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5069worker2 # [ 60.173661] nix-daemon[854]: reaped child process 1045, status = succeeded5070worker2 # [ 60.322040] nix-daemon[854]: accepted connection from pid 1060, user nix-grpc-daemon (trusted)5071worker2 # [ 60.334191] nix-daemon[854]: reaped child process 1063, status = succeeded5072lb # [ 60.364629] niks3-server[1047]: 2026/09/20 13:34:03 INFO Received uploads request method=POST path=/api/pending_closures5073worker2 # [ 60.406738] nix-grpc-daemon[905]: time=2026-09-20T13:34:03.693Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5074lb # [ 60.448631] niks3-server[1047]: 2026/09/20 13:34:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5075lb # [ 60.459492] niks3-server[1047]: 2026/09/20 13:34:04 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5076worker2 # [ 60.445356] nix-grpc-daemon[905]: time=2026-09-20T13:34:03.731Z level=INFO msg="Uploading 1 narinfos"5077lb # [ 60.462368] niks3-server[1047]: 2026/09/20 13:34:04 INFO Signed narinfos id=24 count=15078lb # [ 60.473826] niks3-server[1047]: 2026/09/20 13:34:04 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5079lb # [ 60.476075] niks3-server[1047]: 2026/09/20 13:34:04 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5080lb # [ 60.490433] niks3-server[1047]: 2026/09/20 13:34:04 INFO Completed upload id=245081worker2 # [ 60.475528] nix-grpc-daemon[905]: time=2026-09-20T13:34:03.762Z level=INFO msg="Upload complete. (315ms)"5082lb # [ 60.495549] niks3-server[1047]: 2026/09/20 13:34:04 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5083worker2 # [ 60.482632] nix-grpc-daemon[787]: ts=2026-09-20T13:34:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50482 duration_s=0 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=11 outputs=15084worker2 # [ 60.496839] nix-daemon[854]: reaped child process 1043, status = succeeded5085client # copying 1 paths...5086worker2 # [ 60.526340] nix-grpc-daemon[787]: ts=2026-09-20T13:34:03Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:504825087client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5088worker2 # [ 60.547523] nix-grpc-daemon[787]: ts=2026-09-20T13:34:03Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:504825089worker2 # [ 60.552431] nix-grpc-daemon[787]: ts=2026-09-20T13:34:03Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50482 duration_s=0 paths=1 nar_bytes_out=1205090client # [ 60.679833] nix-daemon[943]: reaped child process 1073, status = succeeded5091client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 1.98 seconds)5092(finished: subtest: an input only one worker has still reaches the builder, in 9.20 seconds)5093subtest: upload whose reference the worker lacks is completed from the cache5094client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5095client # this derivation will be built:5096client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5097client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5098client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.41 seconds)5099client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5100client # this derivation will be built:5101client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5102client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5103client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.37 seconds)5104client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5105client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.18 seconds)5106worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5107worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.22 seconds)5108worker2 # [ 61.892311] systemd[1]: Stopping Nix worker-protocol over gRPC...5109worker2 # [ 62.752882] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=signal_exit5110worker2 # [ 62.988847] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=scheduler_yield5111worker2 # [ 62.993311] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=scheduler_restarting addr=lb:500515112lb # [ 63.009340] niks3-server[1047]: 2026/09/20 13:34:06 INFO lead: released remote=192.168.1.4:375385113worker2 # [ 63.000466] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:504705114worker1 # [ 63.010820] nix-grpc-daemon[784]: ts=2026-09-20T13:34:06Z level=info event=scheduler_restarting addr=lb:500515115worker2 # [ 63.005149] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=worker_gone addr=192.168.1.4:500515116worker2 # [ 63.008278] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:504705117worker2 # [ 63.011922] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=worker_gone addr=192.168.1.3:500515118worker2 # [ 63.194520] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=scheduler_connected addr=lb:500515119worker2 # [ 63.198454] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5120worker1 # [ 63.211262] nix-grpc-daemon[784]: ts=2026-09-20T13:34:07Z level=info event=scheduler_connected addr=lb:500515121worker1 # [ 63.215267] nix-grpc-daemon[784]: ts=2026-09-20T13:34:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5122worker2 # [ 63.399047] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=scheduler_connected addr=lb:500515123worker2 # [ 63.400798] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5124worker1 # [ 63.415602] nix-grpc-daemon[784]: ts=2026-09-20T13:34:07Z level=info event=scheduler_connected addr=lb:500515125worker1 # [ 63.417357] nix-grpc-daemon[784]: ts=2026-09-20T13:34:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5126worker2 # [ 63.602150] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=scheduler_connected addr=lb:500515127worker2 # [ 63.605987] nix-grpc-daemon[787]: ts=2026-09-20T13:34:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5128worker1 # [ 63.617978] nix-grpc-daemon[784]: ts=2026-09-20T13:34:07Z level=info event=scheduler_connected addr=lb:500515129worker1 # [ 63.619758] nix-grpc-daemon[784]: ts=2026-09-20T13:34:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5130worker1 # [ 63.621481] nix-grpc-daemon[784]: ts=2026-09-20T13:34:07Z level=info event=scheduler_take_over5131lb # [ 63.626668] niks3-server[1047]: 2026/09/20 13:34:07 INFO lead: acquired remote=192.168.1.3:553365132worker2 # [ 63.806711] nix-grpc-daemon[787]: ts=2026-09-20T13:34:07Z level=info event=scheduler_connected addr=lb:500515133worker2 # [ 63.809972] nix-grpc-daemon[787]: ts=2026-09-20T13:34:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5134worker1 # [ 63.820889] nix-grpc-daemon[784]: ts=2026-09-20T13:34:07Z level=info event=scheduler_connected addr=lb:500515135worker1 # [ 63.824691] nix-grpc-daemon[784]: ts=2026-09-20T13:34:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5136worker2 # [ 63.992766] nix-grpc-daemon[787]: ts=2026-09-20T13:34:07Z level=info event=scheduler_restarting5137worker1 # [ 64.026749] nix-grpc-daemon[784]: ts=2026-09-20T13:34:07Z level=info event=scheduler_connected addr=lb:500515138worker2 # [ 64.022508] nix-grpc-daemon[787]: ts=2026-09-20T13:34:07Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5139worker1 # [ 64.030961] nix-grpc-daemon[784]: ts=2026-09-20T13:34:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5140worker2 # [ 64.034912] nix-daemon[854]: reaped child process 856, status = succeeded5141worker1 # [ 64.231908] nix-grpc-daemon[784]: ts=2026-09-20T13:34:08Z level=info event=scheduler_connected addr=lb:500515142worker1 # [ 64.235873] nix-grpc-daemon[784]: ts=2026-09-20T13:34:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5143worker1 # [ 64.436771] nix-grpc-daemon[784]: ts=2026-09-20T13:34:08Z level=info event=scheduler_connected addr=lb:500515144worker1 # [ 64.440846] nix-grpc-daemon[784]: ts=2026-09-20T13:34:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5145worker1 # [ 64.641721] nix-grpc-daemon[784]: ts=2026-09-20T13:34:08Z level=info event=scheduler_connected addr=lb:500515146worker1 # [ 64.646263] nix-grpc-daemon[784]: ts=2026-09-20T13:34:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5147worker1 # [ 64.846704] nix-grpc-daemon[784]: ts=2026-09-20T13:34:08Z level=info event=scheduler_connected addr=lb:500515148worker2 # [ 64.838350] nix-grpc-daemon[787]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5149worker2 # [ 64.842488] nix-grpc-daemon[787]: E0000 00:00:1789911248.123904 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5150worker1 # [ 64.854745] nix-grpc-daemon[784]: ts=2026-09-20T13:34:08Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:412785151worker1 # [ 64.857051] nix-grpc-daemon[784]: ts=2026-09-20T13:34:08Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05152worker2 # [ 64.847447] nix-grpc-daemon[787]: E0000 00:00:1789911248.129364 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5153worker2 # [ 65.877638] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5154worker2 # [ 65.881800] systemd[1]: Stopped Nix worker-protocol over gRPC.5155worker2 # [ 65.885908] systemd[1]: nix-grpc-daemon.service: Consumed 1.277s CPU time over 46.387s wall clock time, 50.2M memory peak, 49.1K incoming IP traffic, 106.2K outgoing IP traffic.5156worker2 # [ 65.898507] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5157worker2 # [ 65.903110] systemd[1]: Closed Nix worker-protocol over gRPC.5158worker2 # [ 65.905946] systemd[1]: nix-grpc-daemon.socket: Consumed 404us CPU time over 56.701s wall clock time, 256K memory peak, 69.7K incoming IP traffic, 57K outgoing IP traffic.5159lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5160lb: must succeed: curl -sf localhost:9901/clusters5161lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5162lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.07 seconds)5163client: must succeed: nix path-info --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25164worker1 # [ 66.189275] nix-grpc-daemon[784]: ts=2026-09-20T13:34:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:561065165worker1 # [ 66.195799] nix-grpc-daemon[784]: ts=2026-09-20T13:34:10Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:561065166client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5167client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.23 seconds)5168worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5169worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.02 seconds)5170worker2 # [ 66.291700] systemd[1]: Listening on Nix worker-protocol over gRPC.5171worker1 # [ 66.358814] systemd[1]: Stopping Nix worker-protocol over gRPC...5172worker1 # [ 66.646538] nix-grpc-daemon[784]: ts=2026-09-20T13:34:10Z level=info event=signal_exit5173worker2 # [ 66.867165] systemd[1]: Starting scheduler-token.service...5174worker1 # [ 66.891345] nix-grpc-daemon[784]: ts=2026-09-20T13:34:10Z level=info event=scheduler_yield5175lb # [ 66.900537] niks3-server[1047]: 2026/09/20 13:34:10 INFO lead: released remote=192.168.1.3:553365176worker1 # [ 66.896287] nix-grpc-daemon[784]: ts=2026-09-20T13:34:10Z level=info event=scheduler_restarting addr=lb:500515177worker1 # [ 66.898442] nix-grpc-daemon[784]: ts=2026-09-20T13:34:10Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:412785178worker1 # [ 66.899786] nix-grpc-daemon[784]: ts=2026-09-20T13:34:10Z level=info event=worker_gone addr=192.168.1.3:500515179worker2 # [ 66.954445] systemd[1]: scheduler-token.service: Deactivated successfully.5180worker2 # [ 66.957557] systemd[1]: Finished scheduler-token.service.5181worker2 # [ 66.959843] systemd[1]: scheduler-token.service: Consumed 36ms CPU time over 89ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5182worker2 # [ 66.967707] systemd[1]: Starting Nix worker-protocol over gRPC...5183worker1 # [ 67.097515] nix-grpc-daemon[784]: ts=2026-09-20T13:34:10Z level=info event=scheduler_connected addr=lb:500515184worker1 # [ 67.101777] nix-grpc-daemon[784]: ts=2026-09-20T13:34:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5185worker2 # [ 67.102611] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:10Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5186worker2 # [ 67.104197] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:10Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5187worker2 # [ 67.107490] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:10Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15188worker2 # [ 67.114678] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:10Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5189worker2 # [ 67.117911] systemd[1]: Started Nix worker-protocol over gRPC.5190worker2 # [ 67.124234] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:10Z level=info event=scheduler_take_over5191lb # [ 67.139214] niks3-server[1047]: 2026/09/20 13:34:10 INFO lead: acquired remote=192.168.1.4:465345192worker2 # [ 67.134696] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:10Z level=info event=scheduler_connected addr=lb:500515193worker2 # [ 67.137524] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5194worker1 # [ 67.302839] nix-grpc-daemon[784]: ts=2026-09-20T13:34:11Z level=info event=scheduler_connected addr=lb:500515195worker1 # [ 67.304744] nix-grpc-daemon[784]: ts=2026-09-20T13:34:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5196worker2 # [ 67.338740] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:10Z level=info event=scheduler_connected addr=lb:500515197worker2 # [ 67.342579] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5198worker1 # [ 67.505911] nix-grpc-daemon[784]: ts=2026-09-20T13:34:11Z level=info event=scheduler_connected addr=lb:500515199worker1 # [ 67.509960] nix-grpc-daemon[784]: ts=2026-09-20T13:34:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5200worker2 # [ 67.543634] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:10Z level=info event=scheduler_connected addr=lb:500515201worker2 # [ 67.547501] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5202worker1 # [ 67.710775] nix-grpc-daemon[784]: ts=2026-09-20T13:34:11Z level=info event=scheduler_connected addr=lb:500515203worker1 # [ 67.714755] nix-grpc-daemon[784]: ts=2026-09-20T13:34:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5204worker2 # [ 67.748650] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:11Z level=info event=scheduler_connected addr=lb:500515205worker2 # [ 67.752533] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5206worker1 # [ 67.891855] nix-grpc-daemon[784]: ts=2026-09-20T13:34:11Z level=info event=scheduler_restarting5207worker1 # [ 67.920813] nix-grpc-daemon[784]: ts=2026-09-20T13:34:11Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5208worker2 # [ 67.950965] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:11Z level=info event=scheduler_connected addr=lb:500515209worker2 # [ 67.954727] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5210worker1 # [ 67.996789] nix-daemon[840]: reaped child process 842, status = succeeded5211worker2 # [ 68.156213] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:11Z level=info event=scheduler_connected addr=lb:500515212worker2 # [ 68.169139] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:11Z level=info event=oidc_jwks provider=mock keys=15213worker2 # [ 68.172739] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:11Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:364705214worker2 # [ 68.176571] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:11Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05215worker1 # [ 68.752972] nix-grpc-daemon[784]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5216worker1 # [ 68.754513] nix-grpc-daemon[784]: E0000 00:00:1789911252.563971 827 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5217worker1 # [ 68.756847] nix-grpc-daemon[784]: E0000 00:00:1789911252.568787 827 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5218lb # [ 69.383861] envoy[983]: conn peer=[2001:db8:1::1]:48348 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3715 tx=1803 ms=15935219lb # [ 69.387509] envoy[983]: conn peer=[2001:db8:1::1]:48360 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=525220lb # [ 69.390956] envoy[983]: conn peer=[2001:db8:1::4]:60732 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=21606 tx=1321 ms=442535221lb # [ 69.394661] envoy[983]: conn peer=[2001:db8:1::1]:35116 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=195222lb # [ 69.398274] envoy[983]: conn peer=[2001:db8:1::3]:50538 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=7545 tx=1872 ms=471995223worker1 # [ 69.856636] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5224worker1 # [ 69.861847] systemd[1]: Stopped Nix worker-protocol over gRPC.5225worker1 # [ 69.865311] systemd[1]: nix-grpc-daemon.service: Consumed 2.648s CPU time over 50.481s wall clock time, 58.6M memory peak, 83K incoming IP traffic, 150.6K outgoing IP traffic.5226worker1 # [ 69.876598] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5227worker1 # [ 69.880190] systemd[1]: Closed Nix worker-protocol over gRPC.5228worker1 # [ 69.882411] systemd[1]: nix-grpc-daemon.socket: Consumed 345us CPU time over 59.213s wall clock time, 256K memory peak, 75.5K incoming IP traffic, 62.3K outgoing IP traffic.5229lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5230lb: must succeed: curl -sf localhost:9901/clusters5231lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5232lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.06 seconds)5233client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25234worker2 # [ 70.154379] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364865235worker2 # [ 70.166497] nix-daemon[854]: accepted connection from pid 1087, user nix-grpc-daemon (trusted)5236worker2 # [ 70.183256] nix-grpc-daemon[1087]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 83 ms (attempt 1/5)5237worker2 # [ 70.272612] nix-grpc-daemon[1087]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 73 ms (attempt 2/5)5238worker2 # [ 70.351629] nix-grpc-daemon[1087]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 58 ms (attempt 3/5)5239worker2 # [ 70.415677] nix-grpc-daemon[1087]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 788 ms (attempt 4/5)5240worker2 # [ 71.211186] nix-grpc-daemon[1087]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5241client # copying 2 paths...5242client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5243worker2 # [ 71.232875] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:364865244client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5245worker2 # [ 71.247214] nix-daemon[854]: accepted connection from pid 1087, user nix-grpc-daemon (trusted)5246worker2 # [ 71.268298] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:14Z level=info event=niks3_push_started pid=11435247worker2 # [ 71.445289] nix-grpc-daemon[1143]: time=2026-09-20T13:34:14.731Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.key ca=""5248worker2 # [ 71.599095] nix-daemon[854]: accepted connection from pid 1156, user nix-grpc-daemon (trusted)5249worker2 # [ 71.611956] nix-daemon[854]: reaped child process 1159, status = succeeded5250lb # [ 71.643396] niks3-server[1047]: 2026/09/20 13:34:15 INFO Received uploads request method=POST path=/api/pending_closures5251lb # [ 71.652137] niks3-server[1047]: 2026/09/20 13:34:15 INFO Received uploads request method=POST path=/api/pending_closures5252worker2 # [ 71.644489] nix-grpc-daemon[1143]: time=2026-09-20T13:34:14.931Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5253worker2 # [ 71.646138] nix-grpc-daemon[1143]: time=2026-09-20T13:34:14.933Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5254worker2 # [ 71.648185] nix-grpc-daemon[1143]: time=2026-09-20T13:34:14.935Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5255lb # [ 71.689786] niks3-server[1047]: 2026/09/20 13:34:15 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5256lb # [ 71.703684] niks3-server[1047]: 2026/09/20 13:34:15 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5257lb # [ 71.720172] niks3-server[1047]: 2026/09/20 13:34:15 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5258lb # [ 71.722044] niks3-server[1047]: 2026/09/20 13:34:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5259lb # [ 71.724462] niks3-server[1047]: 2026/09/20 13:34:15 INFO Signed narinfos id=25 count=15260lb # [ 71.727049] niks3-server[1047]: 2026/09/20 13:34:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5261lb # [ 71.729482] niks3-server[1047]: 2026/09/20 13:34:15 INFO Signed narinfos id=26 count=25262worker2 # [ 71.716157] nix-grpc-daemon[1143]: time=2026-09-20T13:34:15.001Z level=INFO msg="Uploading 3 narinfos"5263lb # [ 71.755875] niks3-server[1047]: 2026/09/20 13:34:15 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5264lb # [ 71.758221] niks3-server[1047]: 2026/09/20 13:34:15 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5265lb # [ 71.766773] niks3-server[1047]: 2026/09/20 13:34:15 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5266lb # [ 71.770945] niks3-server[1047]: 2026/09/20 13:34:15 INFO Completed upload id=265267lb # [ 71.772348] niks3-server[1047]: 2026/09/20 13:34:15 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5268lb # [ 71.776116] niks3-server[1047]: 2026/09/20 13:34:15 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5269lb # [ 71.779228] niks3-server[1047]: 2026/09/20 13:34:15 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5270lb # [ 71.782111] niks3-server[1047]: 2026/09/20 13:34:15 INFO Completed upload id=255271worker2 # [ 71.767715] nix-grpc-daemon[1143]: time=2026-09-20T13:34:15.054Z level=INFO msg="Upload complete. (319ms)"5272worker2 # [ 71.772766] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36486 duration_s=0 paths=2 nar_bytes_in=3125273worker2 # [ 71.786098] nix-daemon[854]: reaped child process 1141, status = succeeded5274client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.86 seconds)5275worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5276worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.03 seconds)5277worker1 # [ 71.904747] systemd[1]: Listening on Nix worker-protocol over gRPC.5278worker1 # [ 71.912660] systemd[1]: Starting Nix worker-protocol over gRPC...5279worker1 # [ 72.078908] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:15Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5280worker1 # [ 72.081149] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:15Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5281worker1 # [ 72.085048] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:15Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15282worker1 # [ 72.092056] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:15Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5283worker1 # [ 72.095811] systemd[1]: Started Nix worker-protocol over gRPC.5284lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5285lb: must succeed: curl -sf localhost:9901/clusters5286worker1 # [ 72.111474] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:15Z level=info event=scheduler_connected addr=lb:500515287worker2 # [ 72.105664] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:15Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:364705288worker2 # [ 72.109947] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:15Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05289lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5290lb: must succeed: curl -sf localhost:9901/clusters5291lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5292lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.14 seconds)5293(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 12.57 seconds)5294subtest: scheduler down: the other node takes the lock and keeps it5295lb: must succeed: curl -sf localhost:9901/clusters5296lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5297lb: must succeed: curl -sf localhost:9901/clusters5298lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5299lb: must succeed: curl -sf localhost:9901/clusters5300lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5301lb: must succeed: curl -sf localhost:9901/clusters5302lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5303worker2 # [ 73.464337] systemd[1]: Stopping Nix worker-protocol over gRPC...5304worker2 # [ 74.122488] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:17Z level=info event=signal_exit5305worker2 # [ 74.330614] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:17Z level=info event=scheduler_yield5306lb # [ 74.349102] niks3-server[1047]: 2026/09/20 13:34:17 INFO lead: released remote=192.168.1.4:465345307worker1 # [ 74.344108] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:18Z level=info event=scheduler_restarting addr=lb:500515308worker2 # [ 74.337885] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:17Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:364705309worker2 # [ 74.339819] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:17Z level=info event=worker_gone addr=192.168.1.3:500515310worker2 # [ 74.341810] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:17Z level=info event=scheduler_restarting addr=lb:500515311worker2 # [ 74.343064] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:17Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:364705312worker2 # [ 74.344407] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:17Z level=info event=worker_gone addr=192.168.1.4:500515313worker1 # [ 74.544192] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:18Z level=info event=scheduler_connected addr=lb:500515314worker1 # [ 74.546206] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5315worker2 # [ 74.540842] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:17Z level=info event=scheduler_connected addr=lb:500515316worker2 # [ 74.542668] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5317worker1 # [ 74.746896] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:18Z level=info event=scheduler_connected addr=lb:500515318worker1 # [ 74.748752] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5319worker2 # [ 74.744317] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:18Z level=info event=scheduler_connected addr=lb:500515320worker2 # [ 74.748310] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5321worker1 # [ 74.949800] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:18Z level=info event=scheduler_connected addr=lb:500515322worker1 # [ 74.953858] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5323worker2 # [ 74.949568] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:18Z level=info event=scheduler_connected addr=lb:500515324worker2 # [ 74.953457] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5325worker1 # [ 75.154697] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:18Z level=info event=scheduler_connected addr=lb:500515326worker1 # [ 75.158700] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5327worker2 # [ 75.155985] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:18Z level=info event=scheduler_connected addr=lb:500515328worker2 # [ 75.161894] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5329worker2 # [ 75.332807] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:18Z level=info event=scheduler_restarting5330worker2 # [ 75.351346] nix-daemon[854]: reaped child process 1136, status = succeeded5331worker1 # [ 75.360683] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:19Z level=info event=scheduler_connected addr=lb:500515332worker1 # [ 75.364392] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5333worker2 # [ 75.370423] nix-grpc-daemon[1087]: ts=2026-09-20T13:34:18Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5334worker1 # [ 75.565286] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:19Z level=info event=scheduler_connected addr=lb:500515335worker1 # [ 75.569391] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5336worker1 # [ 75.770412] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:19Z level=info event=scheduler_connected addr=lb:500515337worker1 # [ 75.774408] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5338worker1 # [ 75.975288] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:19Z level=info event=scheduler_connected addr=lb:500515339worker1 # [ 75.979335] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5340worker2 # [ 76.160887] nix-grpc-daemon[1087]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5341worker2 # [ 76.164540] nix-grpc-daemon[1087]: E0000 00:00:1789911259.446837 1130 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5342worker2 # [ 76.169469] nix-grpc-daemon[1087]: E0000 00:00:1789911259.447092 1130 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5343worker1 # [ 76.180348] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:19Z level=info event=scheduler_connected addr=lb:500515344worker1 # [ 76.184798] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5345worker1 # [ 76.385709] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:20Z level=info event=scheduler_connected addr=lb:500515346worker1 # [ 76.387555] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5347worker1 # [ 76.588420] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:20Z level=info event=scheduler_connected addr=lb:500515348worker1 # [ 76.590312] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5349worker1 # [ 76.791142] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:20Z level=info event=scheduler_connected addr=lb:500515350worker1 # [ 76.792956] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5351worker1 # [ 76.993952] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:20Z level=info event=scheduler_connected addr=lb:500515352worker1 # [ 76.998128] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5353worker1 # [ 77.103712] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:20Z level=info event=scheduler_take_over5354lb # [ 77.110390] niks3-server[1047]: 2026/09/20 13:34:20 INFO lead: acquired remote=192.168.1.3:469165355worker2 # [ 77.147566] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5356worker2 # [ 77.152555] systemd[1]: Stopped Nix worker-protocol over gRPC.5357worker2 # [ 77.154692] systemd[1]: nix-grpc-daemon.service: Consumed 353ms CPU time over 10.178s wall clock time, 30.8M memory peak, 20.1K incoming IP traffic, 31.7K outgoing IP traffic.5358worker2 # [ 77.166469] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5359worker2 # [ 77.169868] systemd[1]: Closed Nix worker-protocol over gRPC.5360worker2 # [ 77.172628] systemd[1]: nix-grpc-daemon.socket: Consumed 285us CPU time over 10.881s wall clock time, 256K memory peak, 20.7K incoming IP traffic, 15.3K outgoing IP traffic.5361worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5362worker1 # [ 77.199158] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:21Z level=info event=scheduler_connected addr=lb:500515363worker1 # [ 77.203620] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5364worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds)5365worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5366worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5367worker1 # [ 77.405763] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:21Z level=info event=scheduler_connected addr=lb:500515368worker1 # [ 77.408923] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5369worker1 # [ 77.607113] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:21Z level=info event=scheduler_connected addr=lb:500515370worker1 # [ 77.611146] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5371worker1 # [ 77.812181] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:21Z level=info event=scheduler_connected addr=lb:500515372worker1 # [ 77.816275] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5373worker1 # [ 78.017127] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:21Z level=info event=scheduler_connected addr=lb:500515374worker1 # [ 78.024168] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:21Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:540985375worker1 # [ 78.028339] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:21Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05376worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5377worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5378client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&25379worker1 # [ 78.598669] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:541105380worker1 # [ 78.609643] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5381client # don't know how to build these paths:5382client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5383worker1 # [ 78.625355] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:541105384worker1 # [ 78.636763] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 63 ms (attempt 1/5)5385worker1 # [ 78.705606] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 176 ms (attempt 2/5)5386worker1 # [ 78.887928] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 150 ms (attempt 3/5)5387worker1 # [ 79.044143] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 295 ms (attempt 4/5)5388worker1 # [ 79.345138] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5389worker1 # [ 79.370574] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:23Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=05390worker1 # [ 79.373451] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:23Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=15391lb # [ 79.383902] envoy[983]: conn peer=[2001:db8:1::1]:35118 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1009 tx=305 ms=16335392lb # [ 79.389660] envoy[983]: conn peer=[2001:db8:1::4]:38754 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=8603 tx=535 ms=82315393worker1 # [ 79.415680] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:541105394worker1 # [ 79.425505] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:541105395client # copying 1 paths to 'lb:50051'...5396client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...5397worker1 # [ 79.435215] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:541105398worker1 # [ 79.442090] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5399worker1 # [ 79.455693] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:23Z level=info event=niks3_push_started pid=12785400worker1 # [ 79.662182] nix-grpc-daemon[1278]: time=2026-09-20T13:34:23.473Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.key ca=""5401worker1 # [ 79.833661] nix-daemon[840]: accepted connection from pid 1291, user nix-grpc-daemon (trusted)5402worker1 # [ 79.847546] nix-daemon[840]: reaped child process 1294, status = succeeded5403worker1 # [ 80.028535] nix-daemon[840]: accepted connection from pid 1296, user nix-grpc-daemon (trusted)5404worker1 # [ 80.043122] nix-daemon[840]: reaped child process 1299, status = succeeded5405lb # [ 80.067456] niks3-server[1047]: 2026/09/20 13:34:23 INFO Received uploads request method=POST path=/api/pending_closures5406worker1 # [ 80.072919] nix-grpc-daemon[1278]: time=2026-09-20T13:34:23.884Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5407worker1 # [ 80.076399] nix-grpc-daemon[1278]: time=2026-09-20T13:34:23.886Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"5408lb # [ 80.108717] niks3-server[1047]: 2026/09/20 13:34:23 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst5409lb # [ 80.118248] niks3-server[1047]: 2026/09/20 13:34:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5410lb # [ 80.128128] niks3-server[1047]: 2026/09/20 13:34:23 INFO Signed narinfos id=27 count=15411worker1 # [ 80.121790] nix-grpc-daemon[1278]: time=2026-09-20T13:34:23.933Z level=INFO msg="Uploading 1 narinfos"5412lb # [ 80.137811] niks3-server[1047]: 2026/09/20 13:34:23 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls5413lb # [ 80.142937] niks3-server[1047]: 2026/09/20 13:34:23 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5414lb # [ 80.152219] niks3-server[1047]: 2026/09/20 13:34:23 INFO Completed upload id=275415worker1 # [ 80.145967] nix-grpc-daemon[1278]: time=2026-09-20T13:34:23.957Z level=INFO msg="Upload complete. (479ms)"5416lb # [ 80.154279] niks3-server[1047]: 2026/09/20 13:34:23 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo5417worker1 # [ 80.150079] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54110 duration_s=0 paths=1 nar_bytes_in=5205418worker1 # [ 80.153127] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:541105419worker1 # [ 80.158657] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5420worker1 # [ 80.165520] nix-daemon[840]: reaped child process 1276, status = succeeded5421worker1 # [ 80.171369] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5422client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...5423client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5424client # farm-a> LOG-farm-a-on-standby5425worker1 # [ 81.387172] nix-daemon[840]: reaped child process 1303, status = succeeded5426worker1 # [ 81.546313] nix-daemon[840]: accepted connection from pid 1318, user nix-grpc-daemon (trusted)5427worker1 # [ 81.559943] nix-daemon[840]: reaped child process 1321, status = succeeded5428lb # [ 81.583469] niks3-server[1047]: 2026/09/20 13:34:25 INFO Received uploads request method=POST path=/api/pending_closures5429worker1 # [ 81.944676] nix-grpc-daemon[1278]: time=2026-09-20T13:34:25.755Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5430worker1 # [ 81.956732] nix-grpc-daemon[1278]: time=2026-09-20T13:34:25.755Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"5431lb # [ 81.990875] niks3-server[1047]: 2026/09/20 13:34:25 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5432lb # [ 82.003362] niks3-server[1047]: 2026/09/20 13:34:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5433lb # [ 82.009188] niks3-server[1047]: 2026/09/20 13:34:25 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5434lb # [ 82.017746] niks3-server[1047]: 2026/09/20 13:34:25 INFO Signed narinfos id=28 count=15435worker1 # [ 82.012172] nix-grpc-daemon[1278]: time=2026-09-20T13:34:25.822Z level=INFO msg="Uploading 1 narinfos"5436lb # [ 82.028947] niks3-server[1047]: 2026/09/20 13:34:25 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5437lb # [ 82.035450] niks3-server[1047]: 2026/09/20 13:34:25 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls5438lb # [ 82.048972] niks3-server[1047]: 2026/09/20 13:34:25 INFO Completed upload id=285439worker1 # [ 82.042820] nix-grpc-daemon[1278]: time=2026-09-20T13:34:25.854Z level=INFO msg="Upload complete. (662ms)"5440lb # [ 82.063711] niks3-server[1047]: 2026/09/20 13:34:25 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo5441worker1 # [ 82.054598] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54110 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=15442worker1 # [ 82.072907] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:25Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=05443worker1 # [ 82.078265] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:25Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=15444worker1 # [ 82.086813] nix-daemon[840]: reaped child process 1301, status = succeeded5445worker1 # [ 82.117174] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:541105446worker1 # [ 82.130464] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:541105447worker1 # [ 82.146972] nix-grpc-daemon[1278]: time=2026-09-20T13:34:25.958Z level=INFO msg="All 1 paths already cached"5448client # copying 1 paths to 'lb:50051'...5449client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...5450worker1 # [ 82.152861] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:541105451worker1 # [ 82.159666] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5452worker1 # [ 82.331230] nix-daemon[840]: accepted connection from pid 1327, user nix-grpc-daemon (trusted)5453worker1 # [ 82.345338] nix-daemon[840]: reaped child process 1330, status = succeeded5454worker1 # [ 82.516294] nix-daemon[840]: accepted connection from pid 1332, user nix-grpc-daemon (trusted)5455worker1 # [ 82.529805] nix-daemon[840]: reaped child process 1335, status = succeeded5456lb # [ 82.553292] niks3-server[1047]: 2026/09/20 13:34:26 INFO Received uploads request method=POST path=/api/pending_closures5457worker1 # [ 82.559870] nix-grpc-daemon[1278]: time=2026-09-20T13:34:26.371Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5458worker1 # [ 82.561304] nix-grpc-daemon[1278]: time=2026-09-20T13:34:26.371Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"5459lb # [ 82.585322] niks3-server[1047]: 2026/09/20 13:34:26 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst5460lb # [ 82.601109] niks3-server[1047]: 2026/09/20 13:34:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5461worker1 # [ 82.601757] nix-grpc-daemon[1278]: time=2026-09-20T13:34:26.413Z level=INFO msg="Uploading 1 narinfos"5462lb # [ 82.607677] niks3-server[1047]: 2026/09/20 13:34:26 INFO Signed narinfos id=29 count=15463lb # [ 82.613168] niks3-server[1047]: 2026/09/20 13:34:26 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls5464lb # [ 82.622966] niks3-server[1047]: 2026/09/20 13:34:26 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5465lb # [ 82.629417] niks3-server[1047]: 2026/09/20 13:34:26 INFO Completed upload id=295466worker1 # [ 82.623033] nix-grpc-daemon[1278]: time=2026-09-20T13:34:26.434Z level=INFO msg="Upload complete. (458ms)"5467lb # [ 82.631789] niks3-server[1047]: 2026/09/20 13:34:26 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo5468worker1 # [ 82.626532] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54110 duration_s=0 paths=1 nar_bytes_in=6405469worker1 # [ 82.632312] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:541105470worker1 # [ 82.640513] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5471worker1 # [ 82.648364] nix-daemon[840]: reaped child process 1325, status = succeeded5472client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...5473worker1 # [ 82.654497] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5474client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5475client # farm-b> LOG-farm-b-on-standby5476worker1 # [ 83.266867] nix-daemon[840]: reaped child process 1339, status = succeeded5477worker1 # [ 83.421417] nix-daemon[840]: accepted connection from pid 1354, user nix-grpc-daemon (trusted)5478worker1 # [ 83.433169] nix-daemon[840]: reaped child process 1357, status = succeeded5479lb # [ 83.456370] niks3-server[1047]: 2026/09/20 13:34:27 INFO Received uploads request method=POST path=/api/pending_closures5480worker1 # [ 83.459103] nix-grpc-daemon[1278]: time=2026-09-20T13:34:27.270Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5481worker1 # [ 83.461353] nix-grpc-daemon[1278]: time=2026-09-20T13:34:27.272Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"5482lb # [ 83.488346] niks3-server[1047]: 2026/09/20 13:34:27 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5483lb # [ 83.504302] niks3-server[1047]: 2026/09/20 13:34:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5484lb # [ 83.509906] niks3-server[1047]: 2026/09/20 13:34:27 INFO Signed narinfos id=30 count=15485worker1 # [ 83.506514] nix-grpc-daemon[1278]: time=2026-09-20T13:34:27.318Z level=INFO msg="Uploading 1 narinfos"5486lb # [ 83.518538] niks3-server[1047]: 2026/09/20 13:34:27 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst5487lb # [ 83.524441] niks3-server[1047]: 2026/09/20 13:34:27 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls5488lb # [ 83.533325] niks3-server[1047]: 2026/09/20 13:34:27 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5489lb # [ 83.537325] niks3-server[1047]: 2026/09/20 13:34:27 INFO Completed upload id=305490worker1 # [ 83.531096] nix-grpc-daemon[1278]: time=2026-09-20T13:34:27.342Z level=INFO msg="Upload complete. (271ms)"5491lb # [ 83.538778] niks3-server[1047]: 2026/09/20 13:34:27 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo5492worker1 # [ 83.534433] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54110 duration_s=0 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=15493worker1 # [ 83.540647] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:27Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=05494worker1 # [ 83.542598] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:27Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=15495worker1 # [ 83.547620] nix-daemon[840]: reaped child process 1337, status = succeeded5496worker1 # [ 83.584650] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:541105497worker1 # [ 83.593228] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:541105498worker1 # [ 83.602893] nix-grpc-daemon[1278]: time=2026-09-20T13:34:27.414Z level=INFO msg="All 2 paths already cached"5499client # copying 1 paths to 'lb:50051'...5500client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...5501worker1 # [ 83.610045] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:541105502worker1 # [ 83.614175] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5503worker1 # [ 83.775576] nix-daemon[840]: accepted connection from pid 1363, user nix-grpc-daemon (trusted)5504worker1 # [ 83.789128] nix-daemon[840]: reaped child process 1366, status = succeeded5505worker1 # [ 83.937154] nix-daemon[840]: accepted connection from pid 1368, user nix-grpc-daemon (trusted)5506worker1 # [ 83.949875] nix-daemon[840]: reaped child process 1371, status = succeeded5507lb # [ 83.970450] niks3-server[1047]: 2026/09/20 13:34:27 INFO Received uploads request method=POST path=/api/pending_closures5508worker1 # [ 83.973667] nix-grpc-daemon[1278]: time=2026-09-20T13:34:27.785Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5509worker1 # [ 83.975063] nix-grpc-daemon[1278]: time=2026-09-20T13:34:27.785Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"5510lb # [ 83.996162] niks3-server[1047]: 2026/09/20 13:34:27 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst5511lb # [ 84.008492] niks3-server[1047]: 2026/09/20 13:34:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5512lb # [ 84.014958] niks3-server[1047]: 2026/09/20 13:34:27 INFO Signed narinfos id=31 count=15513worker1 # [ 84.011789] nix-grpc-daemon[1278]: time=2026-09-20T13:34:27.823Z level=INFO msg="Uploading 1 narinfos"5514lb # [ 84.020091] niks3-server[1047]: 2026/09/20 13:34:27 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls5515lb # [ 84.031310] niks3-server[1047]: 2026/09/20 13:34:27 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5516worker1 # [ 84.032401] nix-grpc-daemon[1278]: time=2026-09-20T13:34:27.844Z level=INFO msg="Upload complete. (411ms)"5517lb # [ 84.038903] niks3-server[1047]: 2026/09/20 13:34:27 INFO Completed upload id=315518worker1 # [ 84.034501] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54110 duration_s=0 paths=1 nar_bytes_in=7685519lb # [ 84.042283] niks3-server[1047]: 2026/09/20 13:34:27 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo5520worker1 # [ 84.039947] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:541105521worker1 # [ 84.048464] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5522worker1 # [ 84.051841] nix-daemon[840]: reaped child process 1361, status = succeeded5523worker1 # [ 84.056922] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5524client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...5525client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5526client # farm-top> LOG-farm-top-on-standby5527worker1 # [ 85.271875] nix-daemon[840]: reaped child process 1375, status = succeeded5528worker1 # [ 85.430470] nix-daemon[840]: accepted connection from pid 1390, user nix-grpc-daemon (trusted)5529worker1 # [ 85.443471] nix-daemon[840]: reaped child process 1393, status = succeeded5530lb # [ 85.468671] niks3-server[1047]: 2026/09/20 13:34:29 INFO Received uploads request method=POST path=/api/pending_closures5531worker1 # [ 85.470683] nix-grpc-daemon[1278]: time=2026-09-20T13:34:29.282Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5532worker1 # [ 85.472352] nix-grpc-daemon[1278]: time=2026-09-20T13:34:29.284Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"5533lb # [ 85.502081] niks3-server[1047]: 2026/09/20 13:34:29 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5534lb # [ 85.517478] niks3-server[1047]: 2026/09/20 13:34:29 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst5535lb # [ 85.525929] niks3-server[1047]: 2026/09/20 13:34:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5536lb # [ 85.531130] niks3-server[1047]: 2026/09/20 13:34:29 INFO Signed narinfos id=32 count=15537worker1 # [ 85.527349] nix-grpc-daemon[1278]: time=2026-09-20T13:34:29.339Z level=INFO msg="Uploading 1 narinfos"5538lb # [ 85.535403] niks3-server[1047]: 2026/09/20 13:34:29 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls5539lb # [ 85.544989] niks3-server[1047]: 2026/09/20 13:34:29 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5540lb # [ 85.552461] niks3-server[1047]: 2026/09/20 13:34:29 INFO Completed upload id=325541lb # [ 85.554155] niks3-server[1047]: 2026/09/20 13:34:29 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo5542worker1 # [ 85.546183] nix-grpc-daemon[1278]: time=2026-09-20T13:34:29.358Z level=INFO msg="Upload complete. (282ms)"5543worker1 # [ 85.554520] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54110 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=15544worker1 # [ 85.568531] nix-daemon[840]: reaped child process 1373, status = succeeded5545client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 7.20 seconds)5546client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath5547client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.20 seconds)5548worker2 # [ 85.845509] systemd[1]: Listening on Nix worker-protocol over gRPC.5549worker2 # [ 85.852192] systemd[1]: Starting scheduler-token.service...5550worker2 # [ 85.927985] systemd[1]: scheduler-token.service: Deactivated successfully.5551worker2 # [ 85.929124] systemd[1]: Finished scheduler-token.service.5552worker2 # [ 85.930627] systemd[1]: scheduler-token.service: Consumed 19ms CPU time over 76ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5553worker2 # [ 85.936107] systemd[1]: Starting Nix worker-protocol over gRPC...5554worker2 # [ 86.108366] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:29Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5555worker2 # [ 86.110349] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:29Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5556worker2 # [ 86.115779] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:29Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15557worker2 # [ 86.123056] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:29Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5558worker2 # [ 86.126134] systemd[1]: Started Nix worker-protocol over gRPC.5559lb: must succeed: curl -sf localhost:9901/clusters5560worker2 # [ 86.142915] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:29Z level=info event=scheduler_connected addr=lb:500515561worker1 # [ 86.161106] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:29Z level=info event=oidc_jwks provider=mock keys=15562worker1 # [ 86.163170] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:29Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:540985563worker1 # [ 86.164853] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:29Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05564lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5565worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5566worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5567lb: must succeed: curl -sf localhost:9901/clusters5568lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5569lb: must succeed: curl -sf localhost:9901/clusters5570lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5571worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5572worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5573(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 13.21 seconds)5574subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5575client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs5576client # [ 86.525639] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.5577client # Running as unit: rs.service; invocation ID: 38d656afb9744fbb8a1c5bc22bb8abd75578client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.07 seconds)5579client # [ 86.730740] nix[1173]: don't know how to build these paths:5580client # [ 86.732204] nix[1173]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5581worker1 # [ 86.711534] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:541105582worker1 # [ 86.723141] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:30Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=05583worker1 # [ 86.728289] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:30Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=15584worker1 # [ 86.767063] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:541105585worker1 # [ 86.780433] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:541105586client # [ 86.809349] nix[1173]: copying 1 paths to 'lb:50051'...5587worker1 # [ 86.791899] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:541105588client # [ 86.812142] nix[1173]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...5589worker1 # [ 86.803665] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5590worker1 # [ 86.992599] nix-daemon[840]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)5591worker1 # [ 87.006669] nix-daemon[840]: reaped child process 1409, status = succeeded5592worker1 # [ 87.187384] nix-daemon[840]: accepted connection from pid 1411, user nix-grpc-daemon (trusted)5593worker1 # [ 87.201517] nix-daemon[840]: reaped child process 1414, status = succeeded5594lb # [ 87.221633] niks3-server[1047]: 2026/09/20 13:34:30 INFO Received uploads request method=POST path=/api/pending_closures5595worker1 # [ 87.224283] nix-grpc-daemon[1278]: time=2026-09-20T13:34:31.035Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5596worker1 # [ 87.225684] nix-grpc-daemon[1278]: time=2026-09-20T13:34:31.035Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"5597lb # [ 87.249632] niks3-server[1047]: 2026/09/20 13:34:30 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst5598lb # [ 87.261287] niks3-server[1047]: 2026/09/20 13:34:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5599worker1 # [ 87.260264] nix-grpc-daemon[1278]: time=2026-09-20T13:34:31.071Z level=INFO msg="Uploading 1 narinfos"5600lb # [ 87.266439] niks3-server[1047]: 2026/09/20 13:34:30 INFO Signed narinfos id=33 count=15601lb # [ 87.277314] niks3-server[1047]: 2026/09/20 13:34:30 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls5602lb # [ 87.287286] niks3-server[1047]: 2026/09/20 13:34:30 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5603lb # [ 87.291492] niks3-server[1047]: 2026/09/20 13:34:30 INFO Completed upload id=335604worker1 # [ 87.285275] nix-grpc-daemon[1278]: time=2026-09-20T13:34:31.097Z level=INFO msg="Upload complete. (469ms)"5605lb # [ 87.293030] niks3-server[1047]: 2026/09/20 13:34:30 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo5606worker1 # [ 87.289611] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54110 duration_s=0 paths=1 nar_bytes_in=4165607worker1 # [ 87.295159] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:541105608worker1 # [ 87.303859] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5609worker1 # [ 87.311128] nix-daemon[840]: reaped child process 1404, status = succeeded5610worker1 # [ 87.316554] nix-daemon[840]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5611client # [ 87.334843] nix[1173]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...5612lb: must succeed: curl -sf localhost:9901/clusters5613lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5614lb: must succeed: curl -sf localhost:9901/clusters5615lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5616lb: must succeed: curl -sf localhost:9901/clusters5617lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5618lb: must succeed: curl -sf localhost:9901/clusters5619lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5620worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5621worker2 # [ 89.389529] systemd-journald[286]: Received client request to rotate journal, rotating.5622worker2 # [ 89.407559] systemd-journald[286]: Vacuuming done, freed 0B of archived journals from /var/log/journal/8e611fb0edcd4400a82975be9986475b.5623worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.07 seconds)5624worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5625worker1 # [ 89.334907] systemd[1]: Stopping Nix worker-protocol over gRPC...5626worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)5627??? Warning (UserWarning): sleep(): The 'secs' argument is deprecated. Use 'duration' instead.5628 File "/nix/store/3gycnalwgpix76jjky7ljgh88gwgdb0l-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/machine/__init__.py", line 5735629client: must succeed: sleep 5.05630lb # [ 89.384547] envoy[983]: conn peer=[2001:db8:1::1]:38226 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9577 tx=3843 ms=69695631worker1 # [ 90.097789] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:33Z level=info event=draining5632client: (finished: must succeed: sleep 5.0, in 5.04 seconds)5633worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5634worker1 # [ 94.751406] nix-daemon[840]: reaped child process 1418, status = succeeded5635worker1 # [ 94.912622] nix-daemon[840]: accepted connection from pid 1458, user nix-grpc-daemon (trusted)5636worker1 # [ 94.923843] nix-daemon[840]: reaped child process 1461, status = succeeded5637lb # [ 94.947423] niks3-server[1047]: 2026/09/20 13:34:38 INFO Received uploads request method=POST path=/api/pending_closures5638worker1 # [ 94.949482] nix-grpc-daemon[1278]: time=2026-09-20T13:34:38.761Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5639worker1 # [ 94.951337] nix-grpc-daemon[1278]: time=2026-09-20T13:34:38.763Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"5640lb # [ 94.989745] niks3-server[1047]: 2026/09/20 13:34:38 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5641lb # [ 95.003166] niks3-server[1047]: 2026/09/20 13:34:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5642worker1 # [ 95.001320] nix-grpc-daemon[1278]: time=2026-09-20T13:34:38.813Z level=INFO msg="Uploading 1 narinfos"5643lb # [ 95.008603] niks3-server[1047]: 2026/09/20 13:34:38 INFO Signed narinfos id=34 count=15644lb # [ 95.015529] niks3-server[1047]: 2026/09/20 13:34:38 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls5645lb # [ 95.020806] niks3-server[1047]: 2026/09/20 13:34:38 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5646lb # [ 95.028592] niks3-server[1047]: 2026/09/20 13:34:38 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5647lb # [ 95.032373] niks3-server[1047]: 2026/09/20 13:34:38 INFO Completed upload id=345648worker1 # [ 95.026108] nix-grpc-daemon[1278]: time=2026-09-20T13:34:38.837Z level=INFO msg="Upload complete. (295ms)"5649lb # [ 95.034727] niks3-server[1047]: 2026/09/20 13:34:38 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo5650worker1 # [ 95.033882] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54110 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=15651worker1 # [ 95.047087] nix-daemon[840]: reaped child process 1416, status = succeeded5652client # [ 95.086814] systemd[1]: rs.service: Deactivated successfully.5653client # [ 95.090401] systemd[1]: rs.service: Consumed 114ms CPU time over 8.564s wall clock time, 39.1M memory peak, 8.6K incoming IP traffic, 9K outgoing IP traffic.5654worker1 # [ 95.101352] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:38Z level=info event=signal_exit5655worker1 # [ 95.323586] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=scheduler_yield5656lb # [ 95.332318] niks3-server[1047]: 2026/09/20 13:34:38 INFO lead: released remote=192.168.1.3:469165657worker2 # [ 95.322830] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:38Z level=info event=scheduler_restarting addr=lb:500515658worker1 # [ 95.330840] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=scheduler_restarting addr=lb:500515659worker1 # [ 95.335140] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:540985660worker1 # [ 95.338897] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:540985661worker1 # [ 95.342237] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=worker_gone addr=192.168.1.3:500515662worker1 # [ 95.345436] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=worker_gone addr=192.168.1.4:500515663worker2 # [ 95.523409] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:38Z level=info event=scheduler_connected addr=lb:500515664worker1 # [ 95.531972] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=scheduler_connected addr=lb:500515665worker2 # [ 95.529406] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:38Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5666worker1 # [ 95.537749] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5667worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 1.09 seconds)5668client: waiting for success: ! systemctl is-active rs5669client: (finished: waiting for success: ! systemctl is-active rs, in 0.05 seconds)5670client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5671worker1 # [ 95.738553] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=scheduler_connected addr=lb:500515672worker2 # [ 95.730952] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:39Z level=info event=scheduler_connected addr=lb:500515673worker1 # [ 95.744652] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5674worker2 # [ 95.736617] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:39Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5675client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.04 seconds)5676client: must fail: journalctl -u rs | grep -q 'reconnecting'5677client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.02 seconds)5678lb: must succeed: curl -sf localhost:9901/clusters5679lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5680worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5681worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5682worker2 # [ 95.937890] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:39Z level=info event=scheduler_connected addr=lb:500515683worker1 # [ 95.949791] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=scheduler_connected addr=lb:500515684worker2 # [ 95.941943] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5685worker1 # [ 95.953734] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5686worker2 # [ 96.134191] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:39Z level=info event=scheduler_take_over5687lb # [ 96.149287] niks3-server[1047]: 2026/09/20 13:34:39 INFO lead: acquired remote=192.168.1.4:500545688worker2 # [ 96.142713] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:39Z level=info event=scheduler_connected addr=lb:500515689worker2 # [ 96.146487] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5690worker1 # [ 96.156306] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=scheduler_connected addr=lb:500515691worker1 # [ 96.159539] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5692worker1 # [ 96.327708] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:40Z level=info event=scheduler_restarting5693worker1 # [ 96.341891] nix-daemon[840]: reaped child process 1269, status = succeeded5694worker2 # [ 96.348472] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:39Z level=info event=scheduler_connected addr=lb:500515695worker1 # [ 96.361483] nix-grpc-daemon[1208]: ts=2026-09-20T13:34:40Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5696worker2 # [ 96.352283] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5697worker2 # [ 96.552986] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:39Z level=info event=scheduler_connected addr=lb:500515698worker2 # [ 96.556916] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5699worker2 # [ 96.758176] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:40Z level=info event=scheduler_connected addr=lb:500515700worker2 # [ 96.762353] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5701worker1 # [ 96.896966] nix-grpc-daemon[1208]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5702worker1 # [ 96.900490] nix-grpc-daemon[1208]: E0000 00:00:1789911280.708441 1251 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5703worker1 # [ 96.905124] nix-grpc-daemon[1208]: E0000 00:00:1789911280.708725 1251 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5704lb: must succeed: curl -sf localhost:9901/clusters5705worker2 # [ 96.963312] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:40Z level=info event=scheduler_connected addr=lb:500515706worker2 # [ 96.967264] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5707lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5708worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5709worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5710worker2 # [ 97.167667] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:40Z level=info event=scheduler_connected addr=lb:500515711worker2 # [ 97.169402] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5712worker2 # [ 97.371437] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:40Z level=info event=scheduler_connected addr=lb:500515713worker2 # [ 97.385141] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:40Z level=info event=oidc_jwks provider=mock keys=15714worker2 # [ 97.388914] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:40Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:460585715worker2 # [ 97.392901] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:40Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05716worker1 # [ 97.956072] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5717worker1 # [ 97.960301] systemd[1]: Stopped Nix worker-protocol over gRPC.5718worker1 # [ 97.962443] systemd[1]: nix-grpc-daemon.service: Consumed 1.349s CPU time over 26.043s wall clock time, 33.3M memory peak, 49.3K incoming IP traffic, 101.5K outgoing IP traffic.5719worker1 # [ 97.978324] systemd[1]: Starting Nix worker-protocol over gRPC...5720lb: must succeed: curl -sf localhost:9901/clusters5721lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5722worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5723worker1 # [ 98.152434] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:41Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5724worker1 # [ 98.154980] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:41Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5725worker1 # [ 98.159123] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:41Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15726worker1 # [ 98.166059] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:41Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5727worker1 # [ 98.169143] systemd[1]: Started Nix worker-protocol over gRPC.5728worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5729worker1 # [ 98.182246] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:41Z level=info event=scheduler_connected addr=lb:500515730worker2 # [ 98.175834] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:41Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:460585731worker2 # [ 98.179191] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:41Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05732lb: must succeed: curl -sf localhost:9901/clusters5733lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5734worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5735worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5736(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.85 seconds)5737subtest: scheduler restart mid-build: build finishes, no double build5738client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid5739client # [ 99.378662] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.5740client # Running as unit: mid.service; invocation ID: ed1aff65f6774e149daf95744b1e83275741client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)5742lb # [ 99.385741] envoy[983]: conn peer=[2001:db8:1::1]:41076 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2960 tx=1324 ms=83325743lb # [ 99.389769] envoy[983]: conn peer=[2001:db8:1::3]:41908 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=5767 tx=1349 ms=242535744worker2 # [ 99.563089] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:42Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:460745745worker2 # [ 99.574489] nix-daemon[854]: accepted connection from pid 1183, user nix-grpc-daemon (trusted)5746client # [ 99.612956] nix[1209]: don't know how to build these paths:5747client # [ 99.613786] nix[1209]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv5748worker2 # [ 99.594947] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:42Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=05749worker2 # [ 99.598296] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:42Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=15750worker2 # [ 99.642351] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460745751worker2 # [ 99.648417] nix-grpc-daemon[1183]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 72 ms (attempt 1/5)5752worker2 # [ 99.722880] nix-grpc-daemon[1183]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 193 ms (attempt 2/5)5753worker2 # [ 99.918499] nix-grpc-daemon[1183]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 167 ms (attempt 3/5)5754worker2 # [ 100.085906] nix-grpc-daemon[1183]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 206 ms (attempt 4/5)5755worker2 # [ 100.298309] nix-grpc-daemon[1183]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5756worker2 # [ 100.316302] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460745757client # [ 100.354156] nix[1209]: copying 1 paths to 'lb:50051'...5758client # [ 100.358157] nix[1209]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...5759worker2 # [ 100.331374] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460745760worker2 # [ 100.339603] nix-daemon[854]: accepted connection from pid 1183, user nix-grpc-daemon (trusted)5761worker2 # [ 100.359074] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:43Z level=info event=niks3_push_started pid=12985762worker2 # [ 100.547476] nix-grpc-daemon[1298]: time=2026-09-20T13:34:43.834Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.key ca=""5763worker2 # [ 100.719462] nix-daemon[854]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5764worker2 # [ 100.733371] nix-daemon[854]: reaped child process 1318, status = succeeded5765worker2 # [ 100.902475] nix-daemon[854]: accepted connection from pid 1320, user nix-grpc-daemon (trusted)5766worker2 # [ 100.916173] nix-daemon[854]: reaped child process 1323, status = succeeded5767lb # [ 100.947161] niks3-server[1047]: 2026/09/20 13:34:44 INFO Received uploads request method=POST path=/api/pending_closures5768worker2 # [ 100.937826] nix-grpc-daemon[1298]: time=2026-09-20T13:34:44.224Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5769worker2 # [ 100.941115] nix-grpc-daemon[1298]: time=2026-09-20T13:34:44.226Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"5770lb # [ 100.969679] niks3-server[1047]: 2026/09/20 13:34:44 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst5771lb # [ 100.972148] niks3-server[1047]: 2026/09/20 13:34:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5772lb # [ 100.974700] niks3-server[1047]: 2026/09/20 13:34:44 INFO Signed narinfos id=35 count=15773worker2 # [ 100.960440] nix-grpc-daemon[1298]: time=2026-09-20T13:34:44.247Z level=INFO msg="Uploading 1 narinfos"5774lb # [ 100.979906] niks3-server[1047]: 2026/09/20 13:34:44 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5775lb # [ 100.986073] niks3-server[1047]: 2026/09/20 13:34:44 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls5776lb # [ 100.989247] niks3-server[1047]: 2026/09/20 13:34:44 INFO Completed upload id=355777worker2 # [ 100.974644] nix-grpc-daemon[1298]: time=2026-09-20T13:34:44.261Z level=INFO msg="Upload complete. (423ms)"5778worker2 # [ 100.977215] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46074 duration_s=0 paths=1 nar_bytes_in=4245779worker2 # [ 100.980462] nix-grpc-daemon[1183]: ts=2026-09-20T13:34:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460745780worker2 # [ 100.982399] nix-grpc-daemon[1298]: time=2026-09-20T13:34:44.267Z level=WARN msg="Failed to register uploaded object" key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo error="server returned 404: object is not pending upload\n"5781worker2 # [ 100.988302] nix-daemon[854]: accepted connection from pid 1183, user nix-grpc-daemon (trusted)5782worker2 # [ 100.994793] nix-daemon[854]: reaped child process 1296, status = succeeded5783worker2 # [ 100.999188] nix-daemon[854]: accepted connection from pid 1183, user nix-grpc-daemon (trusted)5784client # [ 101.026387] nix[1209]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...5785lb: must succeed: curl -sf localhost:9901/clusters5786lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5787lb: must succeed: curl -sf localhost:9901/clusters5788lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5789worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5790worker2 # [ 103.158438] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1183 (nix-grpc-daemon) on client request.5791worker2 # [ 103.162355] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5792lb # [ 103.183952] niks3-server[1047]: 2026/09/20 13:34:46 INFO lead: released remote=192.168.1.4:500545793lb # [ 103.188311] niks3-server[1047]: 2026/09/20 13:34:46 INFO lead: acquired remote=192.168.1.3:380365794worker1 # [ 103.182226] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:46Z level=info event=scheduler_take_over5795client # [ 103.207915] nix[1209]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5796worker2 # [ 103.182852] systemd[1]: Starting scheduler-token.service...5797worker1 # [ 103.187925] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:46Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5798client # [ 103.209524] nix[1209]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5799worker2 # [ 103.187272] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5800worker2 # [ 103.192203] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5801worker2 # [ 103.194333] systemd[1]: nix-grpc-daemon.service: Consumed 451ms CPU time over 17.251s wall clock time, 31.1M memory peak, 17.6K incoming IP traffic, 32.4K outgoing IP traffic.5802worker2 # [ 103.211705] nix-daemon[854]: reaped child process 1325, status = succeeded5803worker2 # [ 103.213192] nix-daemon[854]: reaped child process 1289, status = succeeded5804worker2 # [ 103.221249] nix-daemon[1327]: unexpected Nix daemon error: error: interrupted by the user5805worker2 # [ 103.227398] nix-daemon[854]: reaped child process 1327, status = failed with exit code 15806worker2 # [ 103.261731] systemd[1]: scheduler-token.service: Deactivated successfully.5807worker2 # [ 103.263361] systemd[1]: Finished scheduler-token.service.5808worker2 # [ 103.264660] systemd[1]: scheduler-token.service: Consumed 18ms CPU time over 80ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5809worker2 # [ 103.267658] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.5810worker2 # [ 103.272087] systemd[1]: Starting scheduler-token.service...5811worker2 # [ 103.327098] systemd[1]: scheduler-token.service: Deactivated successfully.5812worker2 # [ 103.328729] systemd[1]: Finished scheduler-token.service.5813worker2 # [ 103.330077] systemd[1]: scheduler-token.service: Consumed 18ms CPU time over 56ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5814worker2 # [ 103.335670] systemd[1]: Starting Nix worker-protocol over gRPC...5815worker1 # [ 103.389113] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:47Z level=info event=scheduler_connected addr=lb:500515816worker2 # [ 103.473496] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:46Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5817worker2 # [ 103.475757] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:46Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5818worker2 # [ 103.479518] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:46Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15819worker2 # [ 103.486059] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:46Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5820worker2 # [ 103.490345] systemd[1]: Started Nix worker-protocol over gRPC.5821worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.38 seconds)5822client: must succeed: sleep 5.05823worker2 # [ 103.511055] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:46Z level=info event=scheduler_connected addr=lb:500515824worker2 # [ 103.512610] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5825worker1 # [ 103.517927] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:47Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5826client # [ 103.710535] nix[1209]: scheduler at lb:50051: no healthy upstream, reconnecting5827worker1 # [ 103.719117] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:47Z level=info event=scheduler_connected addr=lb:500515828worker2 # [ 103.714251] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:47Z level=info event=scheduler_connected addr=lb:500515829worker1 # [ 103.723180] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5830worker2 # [ 103.718162] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5831worker1 # [ 103.923953] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:47Z level=info event=scheduler_connected addr=lb:500515832worker1 # [ 103.926146] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5833worker2 # [ 103.919236] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:47Z level=info event=scheduler_connected addr=lb:500515834worker2 # [ 103.923222] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5835worker1 # [ 104.127166] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:47Z level=info event=scheduler_connected addr=lb:500515836worker1 # [ 104.131290] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5837worker2 # [ 104.124396] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:47Z level=info event=scheduler_connected addr=lb:500515838worker2 # [ 104.128298] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5839worker1 # [ 104.333552] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=scheduler_connected addr=lb:500515840worker2 # [ 104.329068] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:47Z level=info event=scheduler_connected addr=lb:500515841worker2 # [ 104.331061] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5842worker1 # [ 104.337497] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5843worker1 # [ 104.538604] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=scheduler_connected addr=lb:500515844worker2 # [ 104.533648] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:47Z level=info event=scheduler_connected addr=lb:500515845worker1 # [ 104.542889] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5846worker2 # [ 104.536827] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5847client # [ 104.713263] nix[1209]: scheduler at lb:50051: no healthy upstream, reconnecting5848worker2 # [ 104.734639] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:48Z level=info event=scheduler_connected addr=lb:500515849worker1 # [ 104.743835] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=scheduler_connected addr=lb:500515850worker2 # [ 104.738785] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5851worker1 # [ 104.747878] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5852worker2 # [ 104.939638] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:48Z level=info event=scheduler_connected addr=lb:500515853worker2 # [ 104.941451] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5854worker1 # [ 104.948987] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=scheduler_connected addr=lb:500515855worker1 # [ 104.953967] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5856worker2 # [ 105.142556] nix-grpc-daemon[1369]: ts=2026-09-20T13:34:48Z level=info event=scheduler_connected addr=lb:500515857worker1 # [ 105.154577] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=scheduler_connected addr=lb:500515858worker1 # [ 105.165144] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:578585859worker1 # [ 105.170233] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05860worker1 # [ 105.181186] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=oidc_jwks provider=mock keys=15861worker1 # [ 105.184150] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:48Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:578585862worker1 # [ 105.188382] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:49Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05863worker1 # [ 106.706347] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:50Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=05864worker1 # [ 106.710559] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:50Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=15865worker1 # [ 106.755100] nix-grpc-daemon[1471]: ts=2026-09-20T13:34:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578685866worker1 # [ 106.766524] nix-daemon[840]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5867worker1 # [ 106.783672] nix-grpc-daemon[1471]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 29 ms (attempt 1/5)5868worker1 # [ 106.818687] nix-grpc-daemon[1471]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 132 ms (attempt 2/5)5869worker1 # [ 106.956831] nix-grpc-daemon[1471]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 148 ms (attempt 3/5)5870worker1 # [ 107.110771] nix-grpc-daemon[1471]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 521 ms (attempt 4/5)5871worker1 # [ 107.638375] nix-grpc-daemon[1471]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5872worker1 # [ 107.651858] nix-grpc-daemon[1471]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' from 'http://lb:5751'...5873worker1 # [ 107.672305] nix-daemon[840]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5874client # [ 107.704449] nix[1209]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...5875worker1 # [ 107.686719] nix-daemon[840]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5876client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5877client: waiting for success: ! systemctl is-active mid5878lb # [ 109.385883] envoy[983]: conn peer=[2001:db8:1::4]:59298 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=9471 tx=675 ms=170445879worker1 # [ 199.196722] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:23Z level=info event=niks3_push_started pid=15745880worker1 # [ 199.229608] nix-daemon[840]: reaped child process 1550, status = succeeded5881worker1 # [ 199.365086] nix-grpc-daemon[1574]: time=2026-09-20T13:36:23.176Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.key ca=""5882worker1 # [ 199.488921] nix-daemon[840]: accepted connection from pid 1587, user nix-grpc-daemon (trusted)5883worker1 # [ 199.501580] nix-daemon[840]: reaped child process 1590, status = succeeded5884lb # [ 199.521944] niks3-server[1047]: 2026/09/20 13:36:23 INFO Received uploads request method=POST path=/api/pending_closures5885worker1 # [ 199.524319] nix-grpc-daemon[1574]: time=2026-09-20T13:36:23.336Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5886worker1 # [ 199.525816] nix-grpc-daemon[1574]: time=2026-09-20T13:36:23.337Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"5887lb # [ 199.577103] niks3-server[1047]: 2026/09/20 13:36:23 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv5888lb # [ 199.581453] niks3-server[1047]: 2026/09/20 13:36:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5889lb # [ 199.586104] niks3-server[1047]: 2026/09/20 13:36:23 INFO Signed narinfos id=36 count=15890worker1 # [ 199.582826] nix-grpc-daemon[1574]: time=2026-09-20T13:36:23.394Z level=INFO msg="Uploading 1 narinfos"5891lb # [ 199.596434] niks3-server[1047]: 2026/09/20 13:36:23 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5892lb # [ 199.599449] niks3-server[1047]: 2026/09/20 13:36:23 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls5893lb # [ 199.603709] niks3-server[1047]: 2026/09/20 13:36:23 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5894lb # [ 199.608131] niks3-server[1047]: 2026/09/20 13:36:23 INFO Completed upload id=365895worker1 # [ 199.601949] nix-grpc-daemon[1574]: time=2026-09-20T13:36:23.413Z level=INFO msg="Upload complete. (233ms)"5896lb # [ 199.609646] niks3-server[1047]: 2026/09/20 13:36:23 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo5897worker1 # [ 199.608280] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57868 duration_s=92 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=15898worker1 # [ 199.619503] nix-daemon[840]: reaped child process 1548, status = succeeded5899client # [ 199.669361] systemd[1]: mid.service: Deactivated successfully.5900client # [ 199.673838] systemd[1]: mid.service: Consumed 257ms CPU time over 1min 40.294s wall clock time, 37.3M memory peak, 4K written to disk, 10K incoming IP traffic, 11.3K outgoing IP traffic.5901client: (finished: waiting for success: ! systemctl is-active mid, in 91.13 seconds)5902client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5903client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.03 seconds)5904(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 100.66 seconds)5905subtest: interrupting the client stops the build on the worker5906client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr5907client # [ 200.012340] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.5908client # Running as unit: intr.service; invocation ID: 5ad424a3b53d45bea63e6d1d8befc2af5909client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.03 seconds)5910worker1 # [ 200.174153] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:578685911client # [ 200.199845] nix[1675]: don't know how to build these paths:5912client # [ 200.200837] nix[1675]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv5913worker1 # [ 200.188932] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:24Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=05914worker1 # [ 200.193419] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:24Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=15915worker2 # [ 200.230919] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346765916worker2 # [ 200.236422] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)5917worker2 # [ 200.244031] nix-grpc-daemon[1369]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 7 ms (attempt 1/5)5918worker2 # [ 200.253398] nix-grpc-daemon[1369]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 23 ms (attempt 2/5)5919worker2 # [ 200.278749] nix-grpc-daemon[1369]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 166 ms (attempt 3/5)5920worker2 # [ 200.447347] nix-grpc-daemon[1369]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 271 ms (attempt 4/5)5921worker2 # [ 200.721755] nix-grpc-daemon[1369]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5922worker2 # [ 200.737872] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346765923client # [ 200.775104] nix[1675]: copying 1 paths to 'lb:50051'...5924client # [ 200.775960] nix[1675]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...5925worker2 # [ 200.748790] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346765926worker2 # [ 200.757286] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)5927worker2 # [ 200.778289] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:24Z level=info event=niks3_push_started pid=14375928worker2 # [ 200.938403] nix-grpc-daemon[1437]: time=2026-09-20T13:36:24.225Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.key ca=""5929worker2 # [ 201.077316] nix-daemon[854]: accepted connection from pid 1450, user nix-grpc-daemon (trusted)5930worker2 # [ 201.089299] nix-daemon[854]: reaped child process 1453, status = succeeded5931worker2 # [ 201.259587] nix-daemon[854]: accepted connection from pid 1455, user nix-grpc-daemon (trusted)5932worker2 # [ 201.270033] nix-daemon[854]: reaped child process 1462, status = succeeded5933lb # [ 201.301702] niks3-server[1047]: 2026/09/20 13:36:24 INFO Received uploads request method=POST path=/api/pending_closures5934worker2 # [ 201.295631] nix-grpc-daemon[1437]: time=2026-09-20T13:36:24.582Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5935worker2 # [ 201.297195] nix-grpc-daemon[1437]: time=2026-09-20T13:36:24.584Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"5936lb # [ 201.335965] niks3-server[1047]: 2026/09/20 13:36:24 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst5937lb # [ 201.345491] niks3-server[1047]: 2026/09/20 13:36:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5938worker2 # [ 201.339126] nix-grpc-daemon[1437]: time=2026-09-20T13:36:24.625Z level=INFO msg="Uploading 1 narinfos"5939lb # [ 201.353980] niks3-server[1047]: 2026/09/20 13:36:24 INFO Signed narinfos id=37 count=15940lb # [ 201.364894] niks3-server[1047]: 2026/09/20 13:36:24 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls5941lb # [ 201.373350] niks3-server[1047]: 2026/09/20 13:36:24 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5942lb # [ 201.379835] niks3-server[1047]: 2026/09/20 13:36:24 INFO Completed upload id=375943worker2 # [ 201.365100] nix-grpc-daemon[1437]: time=2026-09-20T13:36:24.652Z level=INFO msg="Upload complete. (422ms)"5944lb # [ 201.382808] niks3-server[1047]: 2026/09/20 13:36:24 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo5945worker2 # [ 201.368994] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=0 paths=1 nar_bytes_in=4245946worker2 # [ 201.375339] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346765947worker2 # [ 201.384548] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)5948worker2 # [ 201.387910] nix-daemon[854]: reaped child process 1435, status = succeeded5949worker2 # [ 201.392908] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)5950client # [ 201.420261] nix[1675]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...5951client: must succeed: systemctl kill -s INT intr5952client # [ 202.381429] systemd[1]: intr.service: Sent signal SIGINT to main process 1675 (nix) on client request.5953worker2 # [ 202.360660] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:25Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv5954client # [ 202.388282] nix[1675]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5955client # [ 202.394766] nix[1675]: error: interrupted by the user5956client: (finished: must succeed: systemctl kill -s INT intr, in 0.06 seconds)5957client # [ 202.423551] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5958client # [ 202.425243] systemd[1]: intr.service: Failed with result 'exit-code'.5959client # [ 202.427277] systemd[1]: intr.service: Consumed 101ms CPU time over 2.412s wall clock time, 39.1M memory peak, 7.8K incoming IP traffic, 8.7K outgoing IP traffic.5960worker2 # [ 202.408936] nix-daemon[854]: reaped child process 1464, status = succeeded5961worker2 # [ 202.411405] nix-daemon[1466]: unexpected Nix daemon error: error: interrupted by the user5962worker2 # [ 202.415154] nix-daemon[854]: reaped child process 1466, status = failed with exit code 15963(finished: subtest: interrupting the client stops the build on the worker, in 2.50 seconds)5964subtest: a deploy mid-build drains: the build finishes, the new generation takes over5965client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw5966client # [ 202.520152] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.5967client # Running as unit: sw.service; invocation ID: e67ecf1c13694628a15ac7635468d48b5968client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.04 seconds)5969worker2 # [ 202.683456] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:346765970client # [ 202.721030] nix[1696]: don't know how to build these paths:5971client # [ 202.721908] nix[1696]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv5972worker1 # [ 202.711572] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:26Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=05973worker1 # [ 202.715676] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:26Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=15974worker1 # [ 202.754186] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578685975worker1 # [ 202.768369] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578685976client # [ 202.796156] nix[1696]: copying 1 paths to 'lb:50051'...5977worker1 # [ 202.778000] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578685978client # [ 202.798259] nix[1696]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...5979worker1 # [ 202.785922] nix-daemon[840]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5980worker1 # [ 202.944795] nix-daemon[840]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)5981worker1 # [ 202.957932] nix-daemon[840]: reaped child process 1623, status = succeeded5982worker1 # [ 203.103548] nix-daemon[840]: accepted connection from pid 1625, user nix-grpc-daemon (trusted)5983worker1 # [ 203.117530] nix-daemon[840]: reaped child process 1628, status = succeeded5984lb # [ 203.136711] niks3-server[1047]: 2026/09/20 13:36:26 INFO Received uploads request method=POST path=/api/pending_closures5985worker1 # [ 203.135665] nix-grpc-daemon[1574]: time=2026-09-20T13:36:26.947Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5986worker1 # [ 203.137754] nix-grpc-daemon[1574]: time=2026-09-20T13:36:26.949Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"5987lb # [ 203.153302] niks3-server[1047]: 2026/09/20 13:36:26 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst5988lb # [ 203.157488] niks3-server[1047]: 2026/09/20 13:36:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5989lb # [ 203.159480] niks3-server[1047]: 2026/09/20 13:36:26 INFO Signed narinfos id=38 count=15990worker1 # [ 203.153330] nix-grpc-daemon[1574]: time=2026-09-20T13:36:26.965Z level=INFO msg="Uploading 1 narinfos"5991lb # [ 203.163314] niks3-server[1047]: 2026/09/20 13:36:26 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls5992lb # [ 203.168149] niks3-server[1047]: 2026/09/20 13:36:26 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5993lb # [ 203.172971] niks3-server[1047]: 2026/09/20 13:36:26 INFO Completed upload id=385994worker1 # [ 203.166385] nix-grpc-daemon[1574]: time=2026-09-20T13:36:26.978Z level=INFO msg="Upload complete. (373ms)"5995lb # [ 203.174685] niks3-server[1047]: 2026/09/20 13:36:26 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo5996worker1 # [ 203.168364] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57868 duration_s=0 paths=1 nar_bytes_in=4165997worker1 # [ 203.172238] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578685998worker1 # [ 203.178798] nix-daemon[840]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5999worker1 # [ 203.181315] nix-daemon[840]: reaped child process 1618, status = succeeded6000worker1 # [ 203.186386] nix-daemon[840]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)6001client # [ 203.204552] nix[1696]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6002worker1: must succeed: systemctl reload nix-grpc-daemon.service6003worker1 # [ 205.008351] systemd[1]: Reloading Nix worker-protocol over gRPC...6004worker1 # [ 205.048610] systemd[1]: Reloaded Nix worker-protocol over gRPC.6005worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.08 seconds)6006worker1: must succeed: pgrep -f 'read -t [9]0 x'6007worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.04 seconds)6008client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26009worker1 # [ 205.209229] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:29Z level=info event=draining6010client # don't know how to build these paths:6011client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6012worker2 # [ 205.298801] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:346766013worker2 # [ 205.304558] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346766014worker1 # [ 205.327953] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:29Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06015worker1 # [ 205.331171] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:29Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16016worker2 # [ 205.362974] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766017worker2 # [ 205.378511] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346766018client # copying 1 paths to 'lb:50051'...6019worker2 # [ 205.389385] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346766020client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6021worker2 # [ 205.410802] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6022worker2 # [ 205.568661] nix-daemon[854]: accepted connection from pid 1508, user nix-grpc-daemon (trusted)6023worker2 # [ 205.580919] nix-daemon[854]: reaped child process 1511, status = succeeded6024worker2 # [ 205.720419] nix-daemon[854]: accepted connection from pid 1513, user nix-grpc-daemon (trusted)6025worker2 # [ 205.732220] nix-daemon[854]: reaped child process 1516, status = succeeded6026lb # [ 205.757828] niks3-server[1047]: 2026/09/20 13:36:29 INFO Received uploads request method=POST path=/api/pending_closures6027worker2 # [ 205.748049] nix-grpc-daemon[1437]: time=2026-09-20T13:36:29.034Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6028worker2 # [ 205.749385] nix-grpc-daemon[1437]: time=2026-09-20T13:36:29.034Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6029lb # [ 205.774291] niks3-server[1047]: 2026/09/20 13:36:29 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6030lb # [ 205.778597] niks3-server[1047]: 2026/09/20 13:36:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6031lb # [ 205.780511] niks3-server[1047]: 2026/09/20 13:36:29 INFO Signed narinfos id=39 count=16032worker2 # [ 205.767330] nix-grpc-daemon[1437]: time=2026-09-20T13:36:29.054Z level=INFO msg="Uploading 1 narinfos"6033lb # [ 205.786086] niks3-server[1047]: 2026/09/20 13:36:29 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6034lb # [ 205.790799] niks3-server[1047]: 2026/09/20 13:36:29 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6035lb # [ 205.793938] niks3-server[1047]: 2026/09/20 13:36:29 INFO Completed upload id=396036worker2 # [ 205.779211] nix-grpc-daemon[1437]: time=2026-09-20T13:36:29.066Z level=INFO msg="Upload complete. (357ms)"6037worker2 # [ 205.781058] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=0 paths=1 nar_bytes_in=5286038lb # [ 205.796693] niks3-server[1047]: 2026/09/20 13:36:29 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6039worker2 # [ 205.785229] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766040worker2 # [ 205.789917] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6041worker2 # [ 205.793946] nix-daemon[854]: reaped child process 1506, status = succeeded6042worker2 # [ 205.798777] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6043client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6044client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6045client # farm-a> LOG-farm-a-during-drain6046worker2 # [ 207.014586] nix-daemon[854]: reaped child process 1520, status = succeeded6047worker2 # [ 207.158554] nix-daemon[854]: accepted connection from pid 1535, user nix-grpc-daemon (trusted)6048worker2 # [ 207.170478] nix-daemon[854]: reaped child process 1538, status = succeeded6049lb # [ 207.202149] niks3-server[1047]: 2026/09/20 13:36:30 INFO Received uploads request method=POST path=/api/pending_closures6050worker2 # [ 207.197474] nix-grpc-daemon[1437]: time=2026-09-20T13:36:30.484Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6051worker2 # [ 207.201373] nix-grpc-daemon[1437]: time=2026-09-20T13:36:30.484Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6052lb # [ 207.234700] niks3-server[1047]: 2026/09/20 13:36:30 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6053lb # [ 207.255804] niks3-server[1047]: 2026/09/20 13:36:30 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6054lb # [ 207.259448] niks3-server[1047]: 2026/09/20 13:36:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6055lb # [ 207.264176] niks3-server[1047]: 2026/09/20 13:36:30 INFO Signed narinfos id=40 count=16056worker2 # [ 207.250124] nix-grpc-daemon[1437]: time=2026-09-20T13:36:30.536Z level=INFO msg="Uploading 1 narinfos"6057lb # [ 207.272503] niks3-server[1047]: 2026/09/20 13:36:30 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6058lb # [ 207.278971] niks3-server[1047]: 2026/09/20 13:36:30 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6059lb # [ 207.282183] niks3-server[1047]: 2026/09/20 13:36:30 INFO Completed upload id=406060worker2 # [ 207.268734] nix-grpc-daemon[1437]: time=2026-09-20T13:36:30.555Z level=INFO msg="Upload complete. (262ms)"6061lb # [ 207.284968] niks3-server[1047]: 2026/09/20 13:36:30 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6062worker2 # [ 207.274328] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16063worker1 # [ 207.285382] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:31Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06064worker1 # [ 207.287442] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:31Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16065worker2 # [ 207.285472] nix-daemon[854]: reaped child process 1518, status = succeeded6066worker2 # [ 207.323211] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766067worker2 # [ 207.334970] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346766068worker2 # [ 207.348518] nix-grpc-daemon[1437]: time=2026-09-20T13:36:30.635Z level=INFO msg="All 1 paths already cached"6069client # copying 1 paths to 'lb:50051'...6070worker2 # [ 207.356503] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346766071client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6072worker2 # [ 207.376163] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6073worker2 # [ 207.547318] nix-daemon[854]: accepted connection from pid 1542, user nix-grpc-daemon (trusted)6074worker2 # [ 207.560453] nix-daemon[854]: reaped child process 1545, status = succeeded6075worker2 # [ 207.721182] nix-daemon[854]: accepted connection from pid 1547, user nix-grpc-daemon (trusted)6076worker2 # [ 207.733605] nix-daemon[854]: reaped child process 1550, status = succeeded6077lb # [ 207.762672] niks3-server[1047]: 2026/09/20 13:36:31 INFO Received uploads request method=POST path=/api/pending_closures6078worker2 # [ 207.755434] nix-grpc-daemon[1437]: time=2026-09-20T13:36:31.042Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6079worker2 # [ 207.756986] nix-grpc-daemon[1437]: time=2026-09-20T13:36:31.042Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6080lb # [ 207.786194] niks3-server[1047]: 2026/09/20 13:36:31 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6081lb # [ 207.798592] niks3-server[1047]: 2026/09/20 13:36:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6082worker2 # [ 207.790551] nix-grpc-daemon[1437]: time=2026-09-20T13:36:31.077Z level=INFO msg="Uploading 1 narinfos"6083lb # [ 207.805278] niks3-server[1047]: 2026/09/20 13:36:31 INFO Signed narinfos id=41 count=16084lb # [ 207.809770] niks3-server[1047]: 2026/09/20 13:36:31 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6085lb # [ 207.823776] niks3-server[1047]: 2026/09/20 13:36:31 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6086worker2 # [ 207.816236] nix-grpc-daemon[1437]: time=2026-09-20T13:36:31.103Z level=INFO msg="Upload complete. (427ms)"6087lb # [ 207.830941] niks3-server[1047]: 2026/09/20 13:36:31 INFO Completed upload id=416088worker2 # [ 207.819070] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=0 paths=1 nar_bytes_in=6406089lb # [ 207.834588] niks3-server[1047]: 2026/09/20 13:36:31 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6090worker2 # [ 207.824422] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766091worker2 # [ 207.829182] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6092worker2 # [ 207.831406] nix-daemon[854]: reaped child process 1540, status = succeeded6093worker2 # [ 207.836903] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6094client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6095client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6096client # farm-b> LOG-farm-b-during-drain6097worker2 # [ 208.650634] nix-daemon[854]: reaped child process 1554, status = succeeded6098worker2 # [ 208.801898] nix-daemon[854]: accepted connection from pid 1569, user nix-grpc-daemon (trusted)6099worker2 # [ 208.812918] nix-daemon[854]: reaped child process 1572, status = succeeded6100lb # [ 208.845643] niks3-server[1047]: 2026/09/20 13:36:32 INFO Received uploads request method=POST path=/api/pending_closures6101worker2 # [ 208.842063] nix-grpc-daemon[1437]: time=2026-09-20T13:36:32.128Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6102worker2 # [ 208.843823] nix-grpc-daemon[1437]: time=2026-09-20T13:36:32.130Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6103lb # [ 208.886742] niks3-server[1047]: 2026/09/20 13:36:32 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6104lb # [ 208.905354] niks3-server[1047]: 2026/09/20 13:36:32 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6105lb # [ 208.910784] niks3-server[1047]: 2026/09/20 13:36:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6106lb # [ 208.915081] niks3-server[1047]: 2026/09/20 13:36:32 INFO Signed narinfos id=42 count=16107worker2 # [ 208.903168] nix-grpc-daemon[1437]: time=2026-09-20T13:36:32.189Z level=INFO msg="Uploading 1 narinfos"6108lb # [ 208.922808] niks3-server[1047]: 2026/09/20 13:36:32 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6109lb # [ 208.929358] niks3-server[1047]: 2026/09/20 13:36:32 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6110lb # [ 208.933617] niks3-server[1047]: 2026/09/20 13:36:32 INFO Completed upload id=426111worker2 # [ 208.918965] nix-grpc-daemon[1437]: time=2026-09-20T13:36:32.206Z level=INFO msg="Upload complete. (275ms)"6112lb # [ 208.935936] niks3-server[1047]: 2026/09/20 13:36:32 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6113worker2 # [ 208.922264] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16114worker2 # [ 208.928450] nix-daemon[854]: reaped child process 1552, status = succeeded6115worker1 # [ 208.937538] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:32Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06116worker1 # [ 208.940375] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:32Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16117worker2 # [ 208.976569] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766118worker2 # [ 208.986631] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346766119worker2 # [ 208.996511] nix-grpc-daemon[1437]: time=2026-09-20T13:36:32.283Z level=INFO msg="All 2 paths already cached"6120client # copying 1 paths to 'lb:50051'...6121worker2 # [ 209.002077] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346766122client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6123worker2 # [ 209.014780] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6124worker2 # [ 209.172438] nix-daemon[854]: accepted connection from pid 1576, user nix-grpc-daemon (trusted)6125worker2 # [ 209.184814] nix-daemon[854]: reaped child process 1579, status = succeeded6126worker2 # [ 209.343129] nix-daemon[854]: accepted connection from pid 1581, user nix-grpc-daemon (trusted)6127worker2 # [ 209.354875] nix-daemon[854]: reaped child process 1584, status = succeeded6128lb # [ 209.385166] niks3-server[1047]: 2026/09/20 13:36:33 INFO Received uploads request method=POST path=/api/pending_closures6129lb # [ 209.392093] envoy[983]: conn peer=[2001:db8:1::1]:56870 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4179 tx=1758 ms=1000596130lb # [ 209.398204] envoy[983]: conn peer=[2001:db8:1::1]:49784 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2937 tx=731 ms=22136131worker2 # [ 209.389880] nix-grpc-daemon[1437]: time=2026-09-20T13:36:32.676Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6132worker2 # [ 209.391971] nix-grpc-daemon[1437]: time=2026-09-20T13:36:32.678Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6133lb # [ 209.426160] niks3-server[1047]: 2026/09/20 13:36:33 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6134lb # [ 209.444134] niks3-server[1047]: 2026/09/20 13:36:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6135lb # [ 209.449949] niks3-server[1047]: 2026/09/20 13:36:33 INFO Signed narinfos id=43 count=16136lb # [ 209.453288] niks3-server[1047]: 2026/09/20 13:36:33 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6137worker2 # [ 209.438355] nix-grpc-daemon[1437]: time=2026-09-20T13:36:32.724Z level=INFO msg="Uploading 1 narinfos"6138lb # [ 209.464471] niks3-server[1047]: 2026/09/20 13:36:33 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6139lb # [ 209.467304] niks3-server[1047]: 2026/09/20 13:36:33 INFO Completed upload id=436140worker2 # [ 209.452742] nix-grpc-daemon[1437]: time=2026-09-20T13:36:32.739Z level=INFO msg="Upload complete. (432ms)"6141lb # [ 209.469861] niks3-server[1047]: 2026/09/20 13:36:33 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6142worker2 # [ 209.458132] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=0 paths=1 nar_bytes_in=7686143worker2 # [ 209.463658] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766144worker2 # [ 209.476105] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6145worker2 # [ 209.482200] nix-daemon[854]: reaped child process 1574, status = succeeded6146client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6147worker2 # [ 209.490157] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6148client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6149client # farm-top> LOG-farm-top-during-drain6150worker2 # [ 210.515179] nix-daemon[854]: reaped child process 1588, status = succeeded6151worker2 # [ 210.640563] nix-daemon[854]: accepted connection from pid 1603, user nix-grpc-daemon (trusted)6152worker2 # [ 210.653224] nix-daemon[854]: reaped child process 1606, status = succeeded6153lb # [ 210.678054] niks3-server[1047]: 2026/09/20 13:36:34 INFO Received uploads request method=POST path=/api/pending_closures6154worker2 # [ 210.669279] nix-grpc-daemon[1437]: time=2026-09-20T13:36:33.954Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6155worker2 # [ 210.670769] nix-grpc-daemon[1437]: time=2026-09-20T13:36:33.954Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6156lb # [ 210.695038] niks3-server[1047]: 2026/09/20 13:36:34 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6157lb # [ 210.699153] niks3-server[1047]: 2026/09/20 13:36:34 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6158lb # [ 210.702042] niks3-server[1047]: 2026/09/20 13:36:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6159lb # [ 210.703788] niks3-server[1047]: 2026/09/20 13:36:34 INFO Signed narinfos id=44 count=16160worker2 # [ 210.689259] nix-grpc-daemon[1437]: time=2026-09-20T13:36:33.976Z level=INFO msg="Uploading 1 narinfos"6161lb # [ 210.709613] niks3-server[1047]: 2026/09/20 13:36:34 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6162lb # [ 210.712661] niks3-server[1047]: 2026/09/20 13:36:34 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6163lb # [ 210.715213] niks3-server[1047]: 2026/09/20 13:36:34 INFO Completed upload id=446164worker2 # [ 210.701588] nix-grpc-daemon[1437]: time=2026-09-20T13:36:33.988Z level=INFO msg="Upload complete. (202ms)"6165lb # [ 210.717642] niks3-server[1047]: 2026/09/20 13:36:34 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6166worker2 # [ 210.704127] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16167worker2 # [ 210.711310] nix-daemon[854]: reaped child process 1586, status = succeeded6168client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 5.66 seconds)6169client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6170client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.19 seconds)6171worker1: must succeed: pkill -f 'read -t [9]0 x'6172worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6173client: waiting for success: ! systemctl is-active sw6174worker1 # [ 211.209659] nix-daemon[840]: reaped child process 1632, status = succeeded6175worker1 # [ 211.363094] nix-daemon[840]: accepted connection from pid 1678, user nix-grpc-daemon (trusted)6176worker1 # [ 211.376792] nix-daemon[840]: reaped child process 1681, status = succeeded6177lb # [ 211.399638] niks3-server[1047]: 2026/09/20 13:36:35 INFO Received uploads request method=POST path=/api/pending_closures6178worker1 # [ 211.403399] nix-grpc-daemon[1574]: time=2026-09-20T13:36:35.214Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6179worker1 # [ 211.404788] nix-grpc-daemon[1574]: time=2026-09-20T13:36:35.214Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6180lb # [ 211.433619] niks3-server[1047]: 2026/09/20 13:36:35 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6181lb # [ 211.441613] niks3-server[1047]: 2026/09/20 13:36:35 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6182lb # [ 211.453201] niks3-server[1047]: 2026/09/20 13:36:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6183lb # [ 211.457690] niks3-server[1047]: 2026/09/20 13:36:35 INFO Signed narinfos id=45 count=16184worker1 # [ 211.454212] nix-grpc-daemon[1574]: time=2026-09-20T13:36:35.265Z level=INFO msg="Uploading 1 narinfos"6185lb # [ 211.462791] niks3-server[1047]: 2026/09/20 13:36:35 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6186lb # [ 211.472134] niks3-server[1047]: 2026/09/20 13:36:35 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6187lb # [ 211.477230] niks3-server[1047]: 2026/09/20 13:36:35 INFO Completed upload id=456188worker1 # [ 211.471184] nix-grpc-daemon[1574]: time=2026-09-20T13:36:35.282Z level=INFO msg="Upload complete. (270ms)"6189lb # [ 211.480425] niks3-server[1047]: 2026/09/20 13:36:35 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6190worker1 # [ 211.477738] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57868 duration_s=8 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16191worker1 # [ 211.490179] nix-daemon[840]: reaped child process 1630, status = succeeded6192client # [ 211.540119] systemd[1]: sw.service: Deactivated successfully.6193client # [ 211.544224] systemd[1]: sw.service: Consumed 108ms CPU time over 9.022s wall clock time, 37.1M memory peak, 8.5K incoming IP traffic, 9K outgoing IP traffic.6194client: (finished: waiting for success: ! systemctl is-active sw, in 1.09 seconds)6195client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6196client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds)6197worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6198worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 0.05 seconds)6199lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6200lb: must succeed: curl -sf localhost:9901/clusters6201worker1 # [ 212.211926] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=signal_exit6202lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6203lb # [ 212.487454] niks3-server[1047]: 2026/09/20 13:36:36 INFO lead: released remote=192.168.1.3:380366204worker1 # [ 212.480818] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=scheduler_yield6205worker2 # [ 212.476778] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:35Z level=info event=scheduler_restarting addr=lb:500516206worker1 # [ 212.485213] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:578586207worker1 # [ 212.490191] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=worker_gone addr=192.168.1.4:500516208worker1 # [ 212.493305] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=scheduler_restarting addr=lb:500516209worker1 # [ 212.496294] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:578586210worker1 # [ 212.499522] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=worker_gone addr=192.168.1.3:500516211worker2 # [ 212.677261] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:35Z level=info event=scheduler_connected addr=lb:500516212worker1 # [ 212.690168] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=scheduler_connected addr=lb:500516213worker2 # [ 212.681247] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:35Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6214worker1 # [ 212.692236] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6215worker2 # [ 212.882360] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:36Z level=info event=scheduler_connected addr=lb:500516216worker1 # [ 212.893488] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=scheduler_connected addr=lb:500516217worker2 # [ 212.886397] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:36Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6218worker1 # [ 212.897967] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6219worker2 # [ 213.087546] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:36Z level=info event=scheduler_connected addr=lb:500516220worker1 # [ 213.098853] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=scheduler_connected addr=lb:500516221worker2 # [ 213.091597] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:36Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6222worker1 # [ 213.103741] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:36Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6223lb: must succeed: curl -sf localhost:9901/clusters6224lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6225worker2 # [ 213.292544] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:36Z level=info event=scheduler_connected addr=lb:500516226worker2 # [ 213.296374] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6227worker1 # [ 213.305259] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:37Z level=info event=scheduler_connected addr=lb:500516228worker1 # [ 213.308491] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6229worker1 # [ 213.486331] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:37Z level=info event=scheduler_restarting6230lb # [ 213.515531] niks3-server[1047]: 2026/09/20 13:36:37 INFO lead: acquired remote=192.168.1.4:457546231worker2 # [ 213.498769] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:36Z level=info event=scheduler_connected addr=lb:500516232worker2 # [ 213.502408] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6233worker2 # [ 213.506286] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:36Z level=info event=scheduler_take_over6234worker1 # [ 213.515550] nix-daemon[840]: reaped child process 1542, status = succeeded6235worker1 # [ 213.518583] nix-grpc-daemon[1471]: ts=2026-09-20T13:36:37Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6236worker2 # [ 213.699793] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:36Z level=info event=scheduler_connected addr=lb:500516237worker2 # [ 213.703677] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6238worker2 # [ 213.904909] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:37Z level=info event=scheduler_connected addr=lb:500516239worker2 # [ 213.908813] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6240worker2 # [ 214.109965] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:37Z level=info event=scheduler_connected addr=lb:500516241worker2 # [ 214.113789] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6242worker1 # [ 214.127419] nix-grpc-daemon[1471]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6243worker1 # [ 214.130791] nix-grpc-daemon[1471]: E0000 00:00:1789911397.938932 1514 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true6244worker1 # [ 214.135491] nix-grpc-daemon[1471]: E0000 00:00:1789911397.939285 1514 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6245lb: must succeed: curl -sf localhost:9901/clusters6246worker1 # [ 214.307574] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6247worker1 # [ 214.310659] systemd[1]: nix-grpc-daemon.service: Consumed 913ms CPU time over 1min 56.328s wall clock time, 30.9M memory peak, 29.1K incoming IP traffic, 38.7K outgoing IP traffic.6248worker2 # [ 214.314945] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:37Z level=info event=scheduler_connected addr=lb:500516249worker2 # [ 214.318803] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6250lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6251worker2 # [ 214.520531] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:37Z level=info event=scheduler_connected addr=lb:500516252worker2 # [ 214.524274] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6253worker2 # [ 214.725106] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:38Z level=info event=scheduler_connected addr=lb:500516254worker2 # [ 214.743139] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:38Z level=info event=oidc_jwks provider=mock keys=16255worker2 # [ 214.747067] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:38Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:381926256worker2 # [ 214.750913] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:38Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06257worker1 # [ 215.297756] systemd[1]: Starting Nix worker-protocol over gRPC...6258lb: must succeed: curl -sf localhost:9901/clusters6259lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6260worker1 # [ 215.476245] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:39Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6261worker1 # [ 215.478135] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:39Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6262worker1 # [ 215.482771] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:39Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16263worker1 # [ 215.489923] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:39Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6264worker1 # [ 215.497176] systemd[1]: Started Nix worker-protocol over gRPC.6265worker1 # [ 215.510225] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:39Z level=info event=scheduler_connected addr=lb:500516266worker2 # [ 215.504155] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:38Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:381926267worker2 # [ 215.508507] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:38Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06268lb: must succeed: curl -sf localhost:9901/clusters6269lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6270lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 4.29 seconds)6271lb: must succeed: curl -sf localhost:9901/clusters6272lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6273worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6274worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6275(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 14.11 seconds)6276subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6277client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16278worker1 # [ 216.804815] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:395826279worker1 # [ 216.815779] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6280worker1 # [ 216.830390] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395826281worker1 # [ 216.841978] nix-grpc-daemon[1693]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 28 ms (attempt 1/5)6282worker1 # [ 216.875773] nix-grpc-daemon[1693]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 82 ms (attempt 2/5)6283worker1 # [ 216.963704] nix-grpc-daemon[1693]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 163 ms (attempt 3/5)6284worker1 # [ 217.132474] nix-grpc-daemon[1693]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 113 ms (attempt 4/5)6285worker1 # [ 217.251939] nix-grpc-daemon[1693]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6286worker2 # [ 217.296785] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:40Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06287worker2 # [ 217.301351] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:40Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16288worker2 # [ 217.305664] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:40Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06289worker2 # [ 217.309438] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:40Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06290worker1 # [ 217.354249] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826291worker1 # [ 217.367624] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395826292worker1 # [ 217.379277] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395826293worker1 # [ 217.389269] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6294worker1 # [ 217.409763] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:41Z level=info event=niks3_push_started pid=17496295worker1 # [ 217.594642] nix-grpc-daemon[1749]: time=2026-09-20T13:36:41.406Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.key ca=""6296worker1 # [ 217.748154] nix-daemon[840]: accepted connection from pid 1762, user nix-grpc-daemon (trusted)6297worker1 # [ 217.760258] nix-daemon[840]: reaped child process 1765, status = succeeded6298worker1 # [ 217.920820] nix-daemon[840]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6299worker1 # [ 217.933700] nix-daemon[840]: reaped child process 1770, status = succeeded6300lb # [ 217.956847] niks3-server[1047]: 2026/09/20 13:36:41 INFO Received uploads request method=POST path=/api/pending_closures6301worker1 # [ 217.962338] nix-grpc-daemon[1749]: time=2026-09-20T13:36:41.774Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6302worker1 # [ 217.963926] nix-grpc-daemon[1749]: time=2026-09-20T13:36:41.775Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6303lb # [ 218.003069] niks3-server[1047]: 2026/09/20 13:36:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6304worker1 # [ 218.011386] nix-grpc-daemon[1749]: time=2026-09-20T13:36:41.823Z level=INFO msg="Uploading 1 narinfos"6305lb # [ 218.018273] niks3-server[1047]: 2026/09/20 13:36:41 INFO Signed narinfos id=46 count=16306lb # [ 218.044984] niks3-server[1047]: 2026/09/20 13:36:41 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6307lb # [ 218.047384] niks3-server[1047]: 2026/09/20 13:36:41 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6308lb # [ 218.052779] niks3-server[1047]: 2026/09/20 13:36:41 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6309lb # [ 218.055961] niks3-server[1047]: 2026/09/20 13:36:41 INFO Completed upload id=466310lb # [ 218.057401] niks3-server[1047]: 2026/09/20 13:36:41 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6311worker1 # [ 218.049706] nix-grpc-daemon[1749]: time=2026-09-20T13:36:41.861Z level=INFO msg="Upload complete. (450ms)"6312worker1 # [ 218.056213] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=0 paths=1 nar_bytes_in=5046313worker1 # [ 218.062628] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826314worker1 # [ 218.074111] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6315worker1 # [ 218.081476] nix-daemon[840]: reaped child process 1747, status = succeeded6316worker1 # [ 218.089626] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6317worker1 # [ 219.114559] nix-daemon[840]: reaped child process 1774, status = succeeded6318worker1 # [ 219.267088] nix-daemon[840]: accepted connection from pid 1789, user nix-grpc-daemon (trusted)6319worker1 # [ 219.279883] nix-daemon[840]: reaped child process 1792, status = succeeded6320lb # [ 219.303544] niks3-server[1047]: 2026/09/20 13:36:42 INFO Received uploads request method=POST path=/api/pending_closures6321worker1 # [ 219.311358] nix-grpc-daemon[1749]: time=2026-09-20T13:36:43.121Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6322worker1 # [ 219.312727] nix-grpc-daemon[1749]: time=2026-09-20T13:36:43.121Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6323lb # [ 219.350957] niks3-server[1047]: 2026/09/20 13:36:42 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6324lb # [ 219.355845] niks3-server[1047]: 2026/09/20 13:36:42 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6325lb # [ 219.367115] niks3-server[1047]: 2026/09/20 13:36:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6326lb # [ 219.371286] niks3-server[1047]: 2026/09/20 13:36:42 INFO Signed narinfos id=47 count=16327worker1 # [ 219.368243] nix-grpc-daemon[1749]: time=2026-09-20T13:36:43.179Z level=INFO msg="Uploading 1 narinfos"6328lb # [ 219.375621] niks3-server[1047]: 2026/09/20 13:36:42 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6329lb # [ 219.386634] niks3-server[1047]: 2026/09/20 13:36:43 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6330lb # [ 219.390425] niks3-server[1047]: 2026/09/20 13:36:43 INFO Completed upload id=476331lb # [ 219.392106] envoy[983]: conn peer=[2001:db8:1::1]:34424 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9598 tx=3847 ms=54226332worker1 # [ 219.384253] nix-grpc-daemon[1749]: time=2026-09-20T13:36:43.196Z level=INFO msg="Upload complete. (286ms)"6333lb # [ 219.393824] envoy[983]: conn peer=[2001:db8:1::1]:49800 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2949 tx=1322 ms=87996334lb # [ 219.395938] envoy[983]: conn peer=[2001:db8:1::3]:39722 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=3588 tx=984 ms=1153316335lb # [ 219.397736] niks3-server[1047]: 2026/09/20 13:36:43 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6336worker1 # [ 219.391166] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=16337worker2 # [ 219.384565] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:42Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16338worker1 # [ 219.401548] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826339worker2 # [ 219.394274] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:42Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06340worker1 # [ 219.410234] nix-daemon[840]: reaped child process 1772, status = succeeded6341worker1 # [ 219.415318] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395826342worker1 # [ 219.423327] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395826343worker1 # [ 219.431131] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6344worker1 # [ 219.589641] nix-daemon[840]: accepted connection from pid 1796, user nix-grpc-daemon (trusted)6345worker1 # [ 219.602580] nix-daemon[840]: reaped child process 1799, status = succeeded6346worker1 # [ 219.732971] nix-daemon[840]: accepted connection from pid 1801, user nix-grpc-daemon (trusted)6347worker1 # [ 219.745359] nix-daemon[840]: reaped child process 1804, status = succeeded6348lb # [ 219.766778] niks3-server[1047]: 2026/09/20 13:36:43 INFO Received uploads request method=POST path=/api/pending_closures6349worker1 # [ 219.769035] nix-grpc-daemon[1749]: time=2026-09-20T13:36:43.580Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6350worker1 # [ 219.770559] nix-grpc-daemon[1749]: time=2026-09-20T13:36:43.582Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6351lb # [ 219.795433] niks3-server[1047]: 2026/09/20 13:36:43 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6352lb # [ 219.808456] niks3-server[1047]: 2026/09/20 13:36:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6353worker1 # [ 219.806723] nix-grpc-daemon[1749]: time=2026-09-20T13:36:43.618Z level=INFO msg="Uploading 1 narinfos"6354lb # [ 219.813669] niks3-server[1047]: 2026/09/20 13:36:43 INFO Signed narinfos id=48 count=16355lb # [ 219.820862] niks3-server[1047]: 2026/09/20 13:36:43 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6356lb # [ 219.832794] niks3-server[1047]: 2026/09/20 13:36:43 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6357lb # [ 219.840248] niks3-server[1047]: 2026/09/20 13:36:43 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6358lb # [ 219.845511] niks3-server[1047]: 2026/09/20 13:36:43 INFO Completed upload id=486359worker1 # [ 219.839196] nix-grpc-daemon[1749]: time=2026-09-20T13:36:43.651Z level=INFO msg="Upload complete. (403ms)"6360worker1 # [ 219.842830] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=0 paths=1 nar_bytes_in=5046361worker1 # [ 219.849299] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826362worker1 # [ 219.857268] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6363worker1 # [ 219.866266] nix-daemon[840]: reaped child process 1794, status = succeeded6364worker1 # [ 219.871031] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6365worker1 # [ 220.297691] nix-daemon[840]: reaped child process 1808, status = succeeded6366worker1 # [ 220.435961] nix-daemon[840]: accepted connection from pid 1823, user nix-grpc-daemon (trusted)6367worker1 # [ 220.448792] nix-daemon[840]: reaped child process 1826, status = succeeded6368lb # [ 220.469331] niks3-server[1047]: 2026/09/20 13:36:44 INFO Received uploads request method=POST path=/api/pending_closures6369worker1 # [ 220.475528] nix-grpc-daemon[1749]: time=2026-09-20T13:36:44.287Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6370worker1 # [ 220.477310] nix-grpc-daemon[1749]: time=2026-09-20T13:36:44.289Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6371lb # [ 220.509120] niks3-server[1047]: 2026/09/20 13:36:44 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6372lb # [ 220.523207] niks3-server[1047]: 2026/09/20 13:36:44 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6373lb # [ 220.531894] niks3-server[1047]: 2026/09/20 13:36:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6374lb # [ 220.535197] niks3-server[1047]: 2026/09/20 13:36:44 INFO Signed narinfos id=49 count=16375worker1 # [ 220.531439] nix-grpc-daemon[1749]: time=2026-09-20T13:36:44.343Z level=INFO msg="Uploading 1 narinfos"6376lb # [ 220.544458] niks3-server[1047]: 2026/09/20 13:36:44 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6377lb # [ 220.548946] niks3-server[1047]: 2026/09/20 13:36:44 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6378lb # [ 220.552084] niks3-server[1047]: 2026/09/20 13:36:44 INFO Completed upload id=496379worker1 # [ 220.545592] nix-grpc-daemon[1749]: time=2026-09-20T13:36:44.357Z level=INFO msg="Upload complete. (268ms)"6380lb # [ 220.553891] niks3-server[1047]: 2026/09/20 13:36:44 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6381worker2 # [ 220.543167] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:43Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16382worker1 # [ 220.552721] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=0 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=16383worker1 # [ 220.558867] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826384worker2 # [ 220.552763] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:43Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06385worker1 # [ 220.569555] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395826386worker1 # [ 220.573100] nix-daemon[840]: reaped child process 1806, status = succeeded6387worker1 # [ 220.576641] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395826388worker1 # [ 220.580506] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6389worker1 # [ 220.720291] nix-daemon[840]: accepted connection from pid 1830, user nix-grpc-daemon (trusted)6390worker1 # [ 220.733232] nix-daemon[840]: reaped child process 1833, status = succeeded6391worker1 # [ 220.874805] nix-daemon[840]: accepted connection from pid 1835, user nix-grpc-daemon (trusted)6392worker1 # [ 220.887531] nix-daemon[840]: reaped child process 1838, status = succeeded6393lb # [ 220.904255] niks3-server[1047]: 2026/09/20 13:36:44 INFO Received uploads request method=POST path=/api/pending_closures6394worker1 # [ 220.906404] nix-grpc-daemon[1749]: time=2026-09-20T13:36:44.718Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6395worker1 # [ 220.907822] nix-grpc-daemon[1749]: time=2026-09-20T13:36:44.719Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6396lb # [ 220.934322] niks3-server[1047]: 2026/09/20 13:36:44 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6397lb # [ 220.946125] niks3-server[1047]: 2026/09/20 13:36:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6398worker1 # [ 220.945236] nix-grpc-daemon[1749]: time=2026-09-20T13:36:44.756Z level=INFO msg="Uploading 1 narinfos"6399lb # [ 220.951766] niks3-server[1047]: 2026/09/20 13:36:44 INFO Signed narinfos id=50 count=16400lb # [ 220.958285] niks3-server[1047]: 2026/09/20 13:36:44 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6401lb # [ 220.969832] niks3-server[1047]: 2026/09/20 13:36:44 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6402worker1 # [ 220.969226] nix-grpc-daemon[1749]: time=2026-09-20T13:36:44.781Z level=INFO msg="Upload complete. (383ms)"6403lb # [ 220.975730] niks3-server[1047]: 2026/09/20 13:36:44 INFO Completed upload id=506404worker1 # [ 220.971942] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=0 paths=1 nar_bytes_in=5046405lb # [ 220.980560] niks3-server[1047]: 2026/09/20 13:36:44 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6406worker1 # [ 220.978391] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826407worker1 # [ 220.987128] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6408worker1 # [ 220.991060] nix-daemon[840]: reaped child process 1828, status = succeeded6409worker1 # [ 220.995822] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6410worker1 # [ 222.223744] nix-daemon[840]: reaped child process 1842, status = succeeded6411worker1 # [ 222.373793] nix-daemon[840]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6412worker1 # [ 222.386908] nix-daemon[840]: reaped child process 1860, status = succeeded6413lb # [ 222.409774] niks3-server[1047]: 2026/09/20 13:36:46 INFO Received uploads request method=POST path=/api/pending_closures6414worker1 # [ 222.410570] nix-grpc-daemon[1749]: time=2026-09-20T13:36:46.222Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6415worker1 # [ 222.412707] nix-grpc-daemon[1749]: time=2026-09-20T13:36:46.224Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"6416lb # [ 222.451362] niks3-server[1047]: 2026/09/20 13:36:46 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6417lb # [ 222.455284] niks3-server[1047]: 2026/09/20 13:36:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6418lb # [ 222.460665] niks3-server[1047]: 2026/09/20 13:36:46 INFO Signed narinfos id=51 count=16419worker1 # [ 222.457328] nix-grpc-daemon[1749]: time=2026-09-20T13:36:46.269Z level=INFO msg="Uploading 1 narinfos"6420lb # [ 222.468788] niks3-server[1047]: 2026/09/20 13:36:46 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv6421lb # [ 222.480668] niks3-server[1047]: 2026/09/20 13:36:46 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6422lb # [ 222.486626] niks3-server[1047]: 2026/09/20 13:36:46 INFO Completed upload id=516423worker1 # [ 222.482483] nix-grpc-daemon[1749]: time=2026-09-20T13:36:46.294Z level=INFO msg="Upload complete. (279ms)"6424lb # [ 222.489969] niks3-server[1047]: 2026/09/20 13:36:46 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls6425lb # [ 222.494708] niks3-server[1047]: 2026/09/20 13:36:46 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo6426worker1 # [ 222.487551] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=16427worker2 # [ 222.480831] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:45Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=16428worker1 # [ 222.498939] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826429worker2 # [ 222.490761] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:45Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=06430worker1 # [ 222.506542] nix-daemon[840]: reaped child process 1840, status = succeeded6431worker1 # [ 222.512155] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395826432worker1 # [ 222.524197] nix-grpc-daemon[1749]: time=2026-09-20T13:36:46.336Z level=INFO msg="All 1 paths already cached"6433worker1 # [ 222.529081] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395826434worker1 # [ 222.532797] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6435worker1 # [ 222.683810] nix-daemon[840]: accepted connection from pid 1864, user nix-grpc-daemon (trusted)6436worker1 # [ 222.697108] nix-daemon[840]: reaped child process 1867, status = succeeded6437worker1 # [ 222.852835] nix-daemon[840]: accepted connection from pid 1869, user nix-grpc-daemon (trusted)6438worker1 # [ 222.865517] nix-daemon[840]: reaped child process 1872, status = succeeded6439lb # [ 222.887304] niks3-server[1047]: 2026/09/20 13:36:46 INFO Received uploads request method=POST path=/api/pending_closures6440worker1 # [ 222.889999] nix-grpc-daemon[1749]: time=2026-09-20T13:36:46.701Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6441worker1 # [ 222.891527] nix-grpc-daemon[1749]: time=2026-09-20T13:36:46.701Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"6442lb # [ 222.914076] niks3-server[1047]: 2026/09/20 13:36:46 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst6443lb # [ 222.927071] niks3-server[1047]: 2026/09/20 13:36:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6444lb # [ 222.933955] niks3-server[1047]: 2026/09/20 13:36:46 INFO Signed narinfos id=52 count=16445worker1 # [ 222.930874] nix-grpc-daemon[1749]: time=2026-09-20T13:36:46.742Z level=INFO msg="Uploading 1 narinfos"6446lb # [ 222.939125] niks3-server[1047]: 2026/09/20 13:36:46 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls6447lb # [ 222.951567] niks3-server[1047]: 2026/09/20 13:36:46 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6448lb # [ 222.958536] niks3-server[1047]: 2026/09/20 13:36:46 INFO Completed upload id=526449worker1 # [ 222.952235] nix-grpc-daemon[1749]: time=2026-09-20T13:36:46.764Z level=INFO msg="Upload complete. (413ms)"6450lb # [ 222.963159] niks3-server[1047]: 2026/09/20 13:36:46 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo6451worker1 # [ 222.956112] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=0 paths=1 nar_bytes_in=6246452worker1 # [ 222.962108] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826453worker1 # [ 222.970500] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6454worker1 # [ 222.977545] nix-daemon[840]: reaped child process 1862, status = succeeded6455worker1 # [ 222.982537] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6456worker1 # [ 224.007620] nix-daemon[840]: reaped child process 1876, status = succeeded6457worker1 # [ 224.148396] nix-daemon[840]: accepted connection from pid 1891, user nix-grpc-daemon (trusted)6458worker1 # [ 224.161376] nix-daemon[840]: reaped child process 1894, status = succeeded6459lb # [ 224.184250] niks3-server[1047]: 2026/09/20 13:36:47 INFO Received uploads request method=POST path=/api/pending_closures6460worker1 # [ 224.187546] nix-grpc-daemon[1749]: time=2026-09-20T13:36:47.999Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6461worker1 # [ 224.189360] nix-grpc-daemon[1749]: time=2026-09-20T13:36:48.001Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"6462lb # [ 224.221715] niks3-server[1047]: 2026/09/20 13:36:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6463lb # [ 224.226196] niks3-server[1047]: 2026/09/20 13:36:47 INFO Signed narinfos id=53 count=16464worker1 # [ 224.223484] nix-grpc-daemon[1749]: time=2026-09-20T13:36:48.035Z level=INFO msg="Uploading 1 narinfos"6465lb # [ 224.235268] niks3-server[1047]: 2026/09/20 13:36:47 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst6466lb # [ 224.239950] niks3-server[1047]: 2026/09/20 13:36:47 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv6467lb # [ 224.245299] niks3-server[1047]: 2026/09/20 13:36:47 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls6468lb # [ 224.248120] niks3-server[1047]: 2026/09/20 13:36:47 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6469lb # [ 224.251911] niks3-server[1047]: 2026/09/20 13:36:47 INFO Completed upload id=536470worker1 # [ 224.245510] nix-grpc-daemon[1749]: time=2026-09-20T13:36:48.057Z level=INFO msg="Upload complete. (255ms)"6471lb # [ 224.253589] niks3-server[1047]: 2026/09/20 13:36:47 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo6472worker2 # [ 224.242414] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:47Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=16473worker1 # [ 224.249311] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=16474worker1 # [ 224.255554] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826475worker2 # [ 224.250429] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:47Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=06476worker1 # [ 224.265159] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395826477worker1 # [ 224.267378] nix-daemon[840]: reaped child process 1874, status = succeeded6478worker1 # [ 224.271575] nix-grpc-daemon[1749]: time=2026-09-20T13:36:48.083Z level=INFO msg="All 1 paths already cached"6479worker1 # [ 224.274979] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395826480worker1 # [ 224.280034] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6481worker1 # [ 224.428398] nix-daemon[840]: accepted connection from pid 1898, user nix-grpc-daemon (trusted)6482worker1 # [ 224.441491] nix-daemon[840]: reaped child process 1901, status = succeeded6483worker1 # [ 224.597349] nix-daemon[840]: accepted connection from pid 1903, user nix-grpc-daemon (trusted)6484worker1 # [ 224.609974] nix-daemon[840]: reaped child process 1906, status = succeeded6485lb # [ 224.626581] niks3-server[1047]: 2026/09/20 13:36:48 INFO Received uploads request method=POST path=/api/pending_closures6486worker1 # [ 224.624904] nix-grpc-daemon[1749]: time=2026-09-20T13:36:48.436Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6487worker1 # [ 224.626384] nix-grpc-daemon[1749]: time=2026-09-20T13:36:48.438Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"6488lb # [ 224.644297] niks3-server[1047]: 2026/09/20 13:36:48 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst6489lb # [ 224.647206] niks3-server[1047]: 2026/09/20 13:36:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6490lb # [ 224.649544] niks3-server[1047]: 2026/09/20 13:36:48 INFO Signed narinfos id=54 count=16491worker1 # [ 224.644297] nix-grpc-daemon[1749]: time=2026-09-20T13:36:48.456Z level=INFO msg="Uploading 1 narinfos"6492lb # [ 224.652719] niks3-server[1047]: 2026/09/20 13:36:48 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls6493lb # [ 224.658133] niks3-server[1047]: 2026/09/20 13:36:48 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6494lb # [ 224.661963] niks3-server[1047]: 2026/09/20 13:36:48 INFO Completed upload id=546495worker1 # [ 224.655510] nix-grpc-daemon[1749]: time=2026-09-20T13:36:48.467Z level=INFO msg="Upload complete. (370ms)"6496lb # [ 224.663494] niks3-server[1047]: 2026/09/20 13:36:48 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo6497worker1 # [ 224.657501] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=0 paths=1 nar_bytes_in=6246498worker1 # [ 224.661434] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826499worker1 # [ 224.668867] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6500worker1 # [ 224.670135] nix-daemon[840]: reaped child process 1896, status = succeeded6501worker1 # [ 224.675483] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6502worker1 # [ 226.304503] nix-daemon[840]: reaped child process 1910, status = succeeded6503worker1 # [ 226.454626] nix-daemon[840]: accepted connection from pid 1925, user nix-grpc-daemon (trusted)6504worker1 # [ 226.466464] nix-daemon[840]: reaped child process 1928, status = succeeded6505lb # [ 226.482964] niks3-server[1047]: 2026/09/20 13:36:50 INFO Received uploads request method=POST path=/api/pending_closures6506worker1 # [ 226.482468] nix-grpc-daemon[1749]: time=2026-09-20T13:36:50.294Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6507worker1 # [ 226.484065] nix-grpc-daemon[1749]: time=2026-09-20T13:36:50.295Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"6508lb # [ 226.504437] niks3-server[1047]: 2026/09/20 13:36:50 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv6509lb # [ 226.507445] niks3-server[1047]: 2026/09/20 13:36:50 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst6510lb # [ 226.511051] niks3-server[1047]: 2026/09/20 13:36:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6511lb # [ 226.513497] niks3-server[1047]: 2026/09/20 13:36:50 INFO Signed narinfos id=55 count=16512worker1 # [ 226.506940] nix-grpc-daemon[1749]: time=2026-09-20T13:36:50.318Z level=INFO msg="Uploading 1 narinfos"6513lb # [ 226.516940] niks3-server[1047]: 2026/09/20 13:36:50 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls6514lb # [ 226.521152] niks3-server[1047]: 2026/09/20 13:36:50 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6515lb # [ 226.524630] niks3-server[1047]: 2026/09/20 13:36:50 INFO Completed upload id=556516lb # [ 226.526264] niks3-server[1047]: 2026/09/20 13:36:50 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo6517worker1 # [ 226.519177] nix-grpc-daemon[1749]: time=2026-09-20T13:36:50.331Z level=INFO msg="Upload complete. (234ms)"6518worker1 # [ 226.523282] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=16519worker2 # [ 226.517658] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:49Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=16520worker1 # [ 226.529907] nix-daemon[840]: reaped child process 1908, status = succeeded6521worker2 # [ 226.525279] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:49Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=06522worker1 # [ 226.567054] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826523worker1 # [ 226.573137] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395826524worker1 # [ 226.581958] nix-grpc-daemon[1749]: time=2026-09-20T13:36:50.393Z level=INFO msg="All 1 paths already cached"6525worker1 # [ 226.586868] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395826526worker1 # [ 226.595133] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6527worker1 # [ 226.729333] nix-daemon[840]: accepted connection from pid 1932, user nix-grpc-daemon (trusted)6528worker1 # [ 226.742447] nix-daemon[840]: reaped child process 1935, status = succeeded6529worker1 # [ 226.885275] nix-daemon[840]: accepted connection from pid 1937, user nix-grpc-daemon (trusted)6530worker1 # [ 226.897884] nix-daemon[840]: reaped child process 1940, status = succeeded6531lb # [ 226.920563] niks3-server[1047]: 2026/09/20 13:36:50 INFO Received uploads request method=POST path=/api/pending_closures6532worker1 # [ 226.922140] nix-grpc-daemon[1749]: time=2026-09-20T13:36:50.734Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6533worker1 # [ 226.923669] nix-grpc-daemon[1749]: time=2026-09-20T13:36:50.735Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"6534lb # [ 226.954178] niks3-server[1047]: 2026/09/20 13:36:50 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst6535lb # [ 226.961153] niks3-server[1047]: 2026/09/20 13:36:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6536lb # [ 226.966124] niks3-server[1047]: 2026/09/20 13:36:50 INFO Signed narinfos id=56 count=16537worker1 # [ 226.963193] nix-grpc-daemon[1749]: time=2026-09-20T13:36:50.775Z level=INFO msg="Uploading 1 narinfos"6538lb # [ 226.975177] niks3-server[1047]: 2026/09/20 13:36:50 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls6539lb # [ 226.988123] niks3-server[1047]: 2026/09/20 13:36:50 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6540worker1 # [ 226.988076] nix-grpc-daemon[1749]: time=2026-09-20T13:36:50.799Z level=INFO msg="Upload complete. (387ms)"6541lb # [ 226.994524] niks3-server[1047]: 2026/09/20 13:36:50 INFO Completed upload id=566542lb # [ 226.996680] niks3-server[1047]: 2026/09/20 13:36:50 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo6543worker1 # [ 226.990955] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=0 paths=1 nar_bytes_in=6246544worker1 # [ 226.996899] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826545worker1 # [ 227.005808] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6546worker1 # [ 227.013164] nix-daemon[840]: reaped child process 1930, status = succeeded6547worker1 # [ 227.019177] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6548worker1 # [ 227.845895] nix-daemon[840]: reaped child process 1944, status = succeeded6549worker1 # [ 227.968351] nix-daemon[840]: accepted connection from pid 1959, user nix-grpc-daemon (trusted)6550worker1 # [ 227.981458] nix-daemon[840]: reaped child process 1962, status = succeeded6551lb # [ 227.998988] niks3-server[1047]: 2026/09/20 13:36:51 INFO Received uploads request method=POST path=/api/pending_closures6552worker1 # [ 228.001944] nix-grpc-daemon[1749]: time=2026-09-20T13:36:51.812Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6553worker1 # [ 228.003311] nix-grpc-daemon[1749]: time=2026-09-20T13:36:51.812Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"6554lb # [ 228.027286] niks3-server[1047]: 2026/09/20 13:36:51 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv6555lb # [ 228.041101] niks3-server[1047]: 2026/09/20 13:36:51 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst6556lb # [ 228.051122] niks3-server[1047]: 2026/09/20 13:36:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6557lb # [ 228.056536] niks3-server[1047]: 2026/09/20 13:36:51 INFO Signed narinfos id=57 count=16558worker1 # [ 228.052743] nix-grpc-daemon[1749]: time=2026-09-20T13:36:51.864Z level=INFO msg="Uploading 1 narinfos"6559lb # [ 228.061457] niks3-server[1047]: 2026/09/20 13:36:51 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls6560lb # [ 228.070556] niks3-server[1047]: 2026/09/20 13:36:51 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6561lb # [ 228.074981] niks3-server[1047]: 2026/09/20 13:36:51 INFO Completed upload id=576562worker1 # [ 228.068531] nix-grpc-daemon[1749]: time=2026-09-20T13:36:51.880Z level=INFO msg="Upload complete. (243ms)"6563worker1 # [ 228.073097] nix-grpc-daemon[1749]: time=2026-09-20T13:36:51.883Z level=WARN msg="Failed to register uploaded object" key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo error="server returned 404: object is not pending upload\n"6564worker2 # [ 228.066449] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:51Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=16565worker1 # [ 228.080820] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=16566worker1 # [ 228.092417] nix-daemon[840]: reaped child process 1942, status = succeeded6567worker1 # [ 228.117794] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826568worker2 # [ 228.112596] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:51Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=06569worker1 # [ 228.130535] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395826570worker1 # [ 228.143192] nix-grpc-daemon[1749]: time=2026-09-20T13:36:51.955Z level=INFO msg="All 2 paths already cached"6571worker1 # [ 228.150140] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395826572worker1 # [ 228.160169] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6573worker1 # [ 228.325265] nix-daemon[840]: accepted connection from pid 1966, user nix-grpc-daemon (trusted)6574worker1 # [ 228.338407] nix-daemon[840]: reaped child process 1969, status = succeeded6575worker1 # [ 228.498799] nix-daemon[840]: accepted connection from pid 1971, user nix-grpc-daemon (trusted)6576worker1 # [ 228.512055] nix-daemon[840]: reaped child process 1974, status = succeeded6577lb # [ 228.532624] niks3-server[1047]: 2026/09/20 13:36:52 INFO Received uploads request method=POST path=/api/pending_closures6578worker1 # [ 228.534874] nix-grpc-daemon[1749]: time=2026-09-20T13:36:52.346Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6579worker1 # [ 228.536462] nix-grpc-daemon[1749]: time=2026-09-20T13:36:52.348Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"6580lb # [ 228.564068] niks3-server[1047]: 2026/09/20 13:36:52 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst6581lb # [ 228.573606] niks3-server[1047]: 2026/09/20 13:36:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6582worker1 # [ 228.572042] nix-grpc-daemon[1749]: time=2026-09-20T13:36:52.383Z level=INFO msg="Uploading 1 narinfos"6583lb # [ 228.578720] niks3-server[1047]: 2026/09/20 13:36:52 INFO Signed narinfos id=58 count=16584lb # [ 228.583670] niks3-server[1047]: 2026/09/20 13:36:52 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls6585lb # [ 228.596441] niks3-server[1047]: 2026/09/20 13:36:52 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6586lb # [ 228.601392] niks3-server[1047]: 2026/09/20 13:36:52 INFO Completed upload id=586587worker1 # [ 228.597128] nix-grpc-daemon[1749]: time=2026-09-20T13:36:52.409Z level=INFO msg="Upload complete. (427ms)"6588lb # [ 228.603937] niks3-server[1047]: 2026/09/20 13:36:52 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo6589worker1 # [ 228.599893] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=0 paths=1 nar_bytes_in=7526590worker1 # [ 228.605464] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826591worker1 # [ 228.614542] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6592worker1 # [ 228.619696] nix-daemon[840]: reaped child process 1964, status = succeeded6593worker1 # [ 228.625765] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6594worker1 # [ 230.046317] nix-daemon[840]: reaped child process 1978, status = succeeded6595worker1 # [ 230.175178] nix-daemon[840]: accepted connection from pid 1993, user nix-grpc-daemon (trusted)6596worker1 # [ 230.188321] nix-daemon[840]: reaped child process 1996, status = succeeded6597lb # [ 230.206867] niks3-server[1047]: 2026/09/20 13:36:53 INFO Received uploads request method=POST path=/api/pending_closures6598worker1 # [ 230.205657] nix-grpc-daemon[1749]: time=2026-09-20T13:36:54.017Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6599worker1 # [ 230.207747] nix-grpc-daemon[1749]: time=2026-09-20T13:36:54.019Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"6600lb # [ 230.229254] niks3-server[1047]: 2026/09/20 13:36:53 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv6601lb # [ 230.235240] niks3-server[1047]: 2026/09/20 13:36:53 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst6602lb # [ 230.238508] niks3-server[1047]: 2026/09/20 13:36:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6603lb # [ 230.240189] niks3-server[1047]: 2026/09/20 13:36:53 INFO Signed narinfos id=59 count=16604worker1 # [ 230.235081] nix-grpc-daemon[1749]: time=2026-09-20T13:36:54.046Z level=INFO msg="Uploading 1 narinfos"6605lb # [ 230.245579] niks3-server[1047]: 2026/09/20 13:36:53 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls6606lb # [ 230.249212] niks3-server[1047]: 2026/09/20 13:36:53 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6607lb # [ 230.251878] niks3-server[1047]: 2026/09/20 13:36:53 INFO Completed upload id=596608worker1 # [ 230.245417] nix-grpc-daemon[1749]: time=2026-09-20T13:36:54.057Z level=INFO msg="Upload complete. (210ms)"6609lb # [ 230.254737] niks3-server[1047]: 2026/09/20 13:36:53 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo6610worker1 # [ 230.249185] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=16611worker1 # [ 230.252354] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826612worker2 # [ 230.242276] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:53Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=16613worker1 # [ 230.258344] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395826614worker1 # [ 230.260947] nix-daemon[840]: reaped child process 1976, status = succeeded6615worker1 # [ 230.264387] nix-grpc-daemon[1749]: time=2026-09-20T13:36:54.076Z level=INFO msg="All 2 paths already cached"6616worker1 # [ 230.267878] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395826617worker1 # [ 230.271942] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6618worker1 # [ 230.406517] nix-daemon[840]: accepted connection from pid 2000, user nix-grpc-daemon (trusted)6619worker1 # [ 230.419747] nix-daemon[840]: reaped child process 2003, status = succeeded6620worker1 # [ 230.564101] nix-daemon[840]: accepted connection from pid 2005, user nix-grpc-daemon (trusted)6621worker1 # [ 230.577642] nix-daemon[840]: reaped child process 2008, status = succeeded6622lb # [ 230.594167] niks3-server[1047]: 2026/09/20 13:36:54 INFO Received uploads request method=POST path=/api/pending_closures6623worker1 # [ 230.597610] nix-grpc-daemon[1749]: time=2026-09-20T13:36:54.409Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6624worker1 # [ 230.599561] nix-grpc-daemon[1749]: time=2026-09-20T13:36:54.411Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"6625lb # [ 230.624632] niks3-server[1047]: 2026/09/20 13:36:54 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst6626lb # [ 230.637234] niks3-server[1047]: 2026/09/20 13:36:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6627worker1 # [ 230.635621] nix-grpc-daemon[1749]: time=2026-09-20T13:36:54.447Z level=INFO msg="Uploading 1 narinfos"6628lb # [ 230.643143] niks3-server[1047]: 2026/09/20 13:36:54 INFO Signed narinfos id=60 count=16629lb # [ 230.649129] niks3-server[1047]: 2026/09/20 13:36:54 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls6630lb # [ 230.660811] niks3-server[1047]: 2026/09/20 13:36:54 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6631worker1 # [ 230.659374] nix-grpc-daemon[1749]: time=2026-09-20T13:36:54.471Z level=INFO msg="Upload complete. (381ms)"6632lb # [ 230.665892] niks3-server[1047]: 2026/09/20 13:36:54 INFO Completed upload id=606633worker1 # [ 230.662298] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=0 paths=1 nar_bytes_in=7526634lb # [ 230.669658] niks3-server[1047]: 2026/09/20 13:36:54 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo6635worker1 # [ 230.667799] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826636worker1 # [ 230.676244] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6637worker1 # [ 230.682058] nix-daemon[840]: reaped child process 1998, status = succeeded6638worker1 # [ 230.687205] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6639worker1 # [ 231.510303] nix-daemon[840]: reaped child process 2012, status = succeeded6640worker1 # [ 231.645513] nix-daemon[840]: accepted connection from pid 2027, user nix-grpc-daemon (trusted)6641worker1 # [ 231.657620] nix-daemon[840]: reaped child process 2030, status = succeeded6642lb # [ 231.679112] niks3-server[1047]: 2026/09/20 13:36:55 INFO Received uploads request method=POST path=/api/pending_closures6643worker1 # [ 231.677121] nix-grpc-daemon[1749]: time=2026-09-20T13:36:55.488Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6644worker1 # [ 231.678710] nix-grpc-daemon[1749]: time=2026-09-20T13:36:55.490Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"6645lb # [ 231.699090] niks3-server[1047]: 2026/09/20 13:36:55 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv6646lb # [ 231.702834] niks3-server[1047]: 2026/09/20 13:36:55 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst6647lb # [ 231.706049] niks3-server[1047]: 2026/09/20 13:36:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6648lb # [ 231.707981] niks3-server[1047]: 2026/09/20 13:36:55 INFO Signed narinfos id=61 count=16649worker1 # [ 231.702844] nix-grpc-daemon[1749]: time=2026-09-20T13:36:55.514Z level=INFO msg="Uploading 1 narinfos"6650lb # [ 231.710489] niks3-server[1047]: 2026/09/20 13:36:55 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls6651lb # [ 231.715559] niks3-server[1047]: 2026/09/20 13:36:55 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6652lb # [ 231.718687] niks3-server[1047]: 2026/09/20 13:36:55 INFO Completed upload id=616653worker1 # [ 231.712157] nix-grpc-daemon[1749]: time=2026-09-20T13:36:55.524Z level=INFO msg="Upload complete. (218ms)"6654lb # [ 231.721357] niks3-server[1047]: 2026/09/20 13:36:55 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo6655worker1 # [ 231.715062] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=16656worker1 # [ 231.718677] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826657worker2 # [ 231.708963] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:54Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=16658worker1 # [ 231.725075] nix-daemon[840]: reaped child process 2010, status = succeeded6659worker1 # [ 231.726628] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395826660worker1 # [ 231.732054] nix-grpc-daemon[1749]: time=2026-09-20T13:36:55.543Z level=INFO msg="All 2 paths already cached"6661worker1 # [ 231.735606] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395826662worker1 # [ 231.740497] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6663worker1 # [ 231.870026] nix-daemon[840]: accepted connection from pid 2034, user nix-grpc-daemon (trusted)6664worker1 # [ 231.883369] nix-daemon[840]: reaped child process 2037, status = succeeded6665worker1 # [ 232.040810] nix-daemon[840]: accepted connection from pid 2039, user nix-grpc-daemon (trusted)6666worker1 # [ 232.052426] nix-daemon[840]: reaped child process 2042, status = succeeded6667lb # [ 232.074243] niks3-server[1047]: 2026/09/20 13:36:55 INFO Received uploads request method=POST path=/api/pending_closures6668worker1 # [ 232.072435] nix-grpc-daemon[1749]: time=2026-09-20T13:36:55.884Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6669worker1 # [ 232.073956] nix-grpc-daemon[1749]: time=2026-09-20T13:36:55.885Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"6670lb # [ 232.091578] niks3-server[1047]: 2026/09/20 13:36:55 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst6671lb # [ 232.094756] niks3-server[1047]: 2026/09/20 13:36:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6672lb # [ 232.096548] niks3-server[1047]: 2026/09/20 13:36:55 INFO Signed narinfos id=62 count=16673worker1 # [ 232.089884] nix-grpc-daemon[1749]: time=2026-09-20T13:36:55.901Z level=INFO msg="Uploading 1 narinfos"6674lb # [ 232.100123] niks3-server[1047]: 2026/09/20 13:36:55 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls6675lb # [ 232.104513] niks3-server[1047]: 2026/09/20 13:36:55 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6676lb # [ 232.107397] niks3-server[1047]: 2026/09/20 13:36:55 INFO Completed upload id=626677worker1 # [ 232.102242] nix-grpc-daemon[1749]: time=2026-09-20T13:36:55.914Z level=INFO msg="Upload complete. (357ms)"6678lb # [ 232.109494] niks3-server[1047]: 2026/09/20 13:36:55 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo6679worker1 # [ 232.104358] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=0 paths=1 nar_bytes_in=7526680worker1 # [ 232.107526] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395826681worker1 # [ 232.114268] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6682worker1 # [ 232.116953] nix-daemon[840]: reaped child process 2032, status = succeeded6683worker1 # [ 232.123248] nix-daemon[840]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6684worker1 # [ 232.943448] nix-daemon[840]: reaped child process 2046, status = succeeded6685worker1 # [ 233.089413] nix-daemon[840]: accepted connection from pid 2061, user nix-grpc-daemon (trusted)6686worker1 # [ 233.102375] nix-daemon[840]: reaped child process 2064, status = succeeded6687lb # [ 233.125386] niks3-server[1047]: 2026/09/20 13:36:56 INFO Received uploads request method=POST path=/api/pending_closures6688worker1 # [ 233.128382] nix-grpc-daemon[1749]: time=2026-09-20T13:36:56.940Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6689worker1 # [ 233.131066] nix-grpc-daemon[1749]: time=2026-09-20T13:36:56.941Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"6690lb # [ 233.166256] niks3-server[1047]: 2026/09/20 13:36:56 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv6691lb # [ 233.175127] niks3-server[1047]: 2026/09/20 13:36:56 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst6692lb # [ 233.186955] niks3-server[1047]: 2026/09/20 13:36:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6693worker1 # [ 233.185811] nix-grpc-daemon[1749]: time=2026-09-20T13:36:56.997Z level=INFO msg="Uploading 1 narinfos"6694lb # [ 233.192481] niks3-server[1047]: 2026/09/20 13:36:56 INFO Signed narinfos id=63 count=16695lb # [ 233.197756] niks3-server[1047]: 2026/09/20 13:36:56 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls6696lb # [ 233.206915] niks3-server[1047]: 2026/09/20 13:36:56 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6697lb # [ 233.211460] niks3-server[1047]: 2026/09/20 13:36:56 INFO Completed upload id=636698lb # [ 233.215496] niks3-server[1047]: 2026/09/20 13:36:56 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo6699worker1 # [ 233.207669] nix-grpc-daemon[1749]: time=2026-09-20T13:36:57.019Z level=INFO msg="Upload complete. (278ms)"6700worker1 # [ 233.213733] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39582 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=16701worker1 # [ 233.227289] nix-daemon[840]: reaped child process 2044, status = succeeded6702client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 16.68 seconds)6703worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6704worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6705client: must fail: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]' 2>&16706worker2 # [ 233.477523] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:56Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:346766707worker1 # [ 233.493749] nix-grpc-daemon[1693]: ts=2026-09-20T13:36:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395826708worker2 # [ 233.505389] nix-grpc-daemon[1369]: ts=2026-09-20T13:36:56Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=06709lb # [ 239.392939] envoy[983]: conn peer=[2001:db8:1::1]:40476 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27094 tx=10686 ms=164346710client: (finished: must fail: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]' 2>&1, in 30.23 seconds)6711(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 46.97 seconds)6712subtest: low disk drains a worker and builds go to the other6713worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6714worker1 # [ 264.518555] nix-grpc-daemon[1693]: ts=2026-09-20T13:37:28Z level=info event=unhealthy reason=min_free6715worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.03 seconds)6716worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6717worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.05 seconds)6718client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&26719worker2 # [ 264.788701] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:346766720client # don't know how to build these paths:6721client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv6722worker2 # [ 264.797661] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346766723worker2 # [ 264.817882] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:28Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=06724worker2 # [ 264.823152] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:28Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=16725worker2 # [ 264.861196] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766726worker2 # [ 264.875958] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346766727client # copying 1 paths to 'lb:50051'...6728client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...6729worker2 # [ 264.885395] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346766730worker2 # [ 264.897285] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6731worker2 # [ 265.069649] nix-daemon[854]: accepted connection from pid 1631, user nix-grpc-daemon (trusted)6732worker2 # [ 265.082792] nix-daemon[854]: reaped child process 1634, status = succeeded6733worker2 # [ 265.227425] nix-daemon[854]: accepted connection from pid 1636, user nix-grpc-daemon (trusted)6734worker2 # [ 265.239712] nix-daemon[854]: reaped child process 1639, status = succeeded6735lb # [ 265.267975] niks3-server[1047]: 2026/09/20 13:37:28 INFO Received uploads request method=POST path=/api/pending_closures6736worker2 # [ 265.260874] nix-grpc-daemon[1437]: time=2026-09-20T13:37:28.547Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6737worker2 # [ 265.262278] nix-grpc-daemon[1437]: time=2026-09-20T13:37:28.547Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"6738lb # [ 265.285669] niks3-server[1047]: 2026/09/20 13:37:28 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst6739lb # [ 265.291332] niks3-server[1047]: 2026/09/20 13:37:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6740lb # [ 265.294174] niks3-server[1047]: 2026/09/20 13:37:28 INFO Signed narinfos id=64 count=16741worker2 # [ 265.279403] nix-grpc-daemon[1437]: time=2026-09-20T13:37:28.566Z level=INFO msg="Uploading 1 narinfos"6742lb # [ 265.300196] niks3-server[1047]: 2026/09/20 13:37:28 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6743lb # [ 265.305964] niks3-server[1047]: 2026/09/20 13:37:28 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls6744lb # [ 265.308800] niks3-server[1047]: 2026/09/20 13:37:28 INFO Completed upload id=646745worker2 # [ 265.294174] nix-grpc-daemon[1437]: time=2026-09-20T13:37:28.581Z level=INFO msg="Upload complete. (385ms)"6746lb # [ 265.312250] niks3-server[1047]: 2026/09/20 13:37:28 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo6747worker2 # [ 265.297069] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=0 paths=1 nar_bytes_in=5126748worker2 # [ 265.300119] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766749worker2 # [ 265.304742] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6750worker2 # [ 265.309496] nix-daemon[854]: reaped child process 1629, status = succeeded6751client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...6752worker2 # [ 265.315362] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6753client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv6754client # farm-a> LOG-farm-a-drain6755worker2 # [ 265.927396] nix-daemon[854]: reaped child process 1643, status = succeeded6756worker2 # [ 266.054162] nix-daemon[854]: accepted connection from pid 1658, user nix-grpc-daemon (trusted)6757worker2 # [ 266.066364] nix-daemon[854]: reaped child process 1661, status = succeeded6758lb # [ 266.090662] niks3-server[1047]: 2026/09/20 13:37:29 INFO Received uploads request method=POST path=/api/pending_closures6759worker2 # [ 266.084850] nix-grpc-daemon[1437]: time=2026-09-20T13:37:29.371Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6760worker2 # [ 266.086329] nix-grpc-daemon[1437]: time=2026-09-20T13:37:29.373Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"6761lb # [ 266.130399] niks3-server[1047]: 2026/09/20 13:37:29 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6762lb # [ 266.138266] niks3-server[1047]: 2026/09/20 13:37:29 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv6763lb # [ 266.144832] niks3-server[1047]: 2026/09/20 13:37:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6764worker2 # [ 266.134209] nix-grpc-daemon[1437]: time=2026-09-20T13:37:29.421Z level=INFO msg="Uploading 1 narinfos"6765lb # [ 266.148893] niks3-server[1047]: 2026/09/20 13:37:29 INFO Signed narinfos id=65 count=16766lb # [ 266.153529] niks3-server[1047]: 2026/09/20 13:37:29 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls6767lb # [ 266.161534] niks3-server[1047]: 2026/09/20 13:37:29 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6768lb # [ 266.164552] niks3-server[1047]: 2026/09/20 13:37:29 INFO Completed upload id=656769worker2 # [ 266.150172] nix-grpc-daemon[1437]: time=2026-09-20T13:37:29.437Z level=INFO msg="Upload complete. (230ms)"6770lb # [ 266.166668] niks3-server[1047]: 2026/09/20 13:37:29 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo6771worker2 # [ 266.156530] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=0 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=16772worker2 # [ 266.165524] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:29Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=06773worker2 # [ 266.167658] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:29Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=16774worker2 # [ 266.172547] nix-daemon[854]: reaped child process 1641, status = succeeded6775worker2 # [ 266.210489] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766776worker2 # [ 266.221803] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346766777worker2 # [ 266.236550] nix-grpc-daemon[1437]: time=2026-09-20T13:37:29.523Z level=INFO msg="All 1 paths already cached"6778client # copying 1 paths to 'lb:50051'...6779client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...6780worker2 # [ 266.244787] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346766781worker2 # [ 266.252901] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6782worker2 # [ 266.394226] nix-daemon[854]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6783worker2 # [ 266.407946] nix-daemon[854]: reaped child process 1670, status = succeeded6784worker2 # [ 266.535029] nix-daemon[854]: accepted connection from pid 1672, user nix-grpc-daemon (trusted)6785worker2 # [ 266.546982] nix-daemon[854]: reaped child process 1675, status = succeeded6786lb # [ 266.577283] niks3-server[1047]: 2026/09/20 13:37:30 INFO Received uploads request method=POST path=/api/pending_closures6787worker2 # [ 266.572362] nix-grpc-daemon[1437]: time=2026-09-20T13:37:29.859Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6788worker2 # [ 266.574248] nix-grpc-daemon[1437]: time=2026-09-20T13:37:29.860Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"6789lb # [ 266.607721] niks3-server[1047]: 2026/09/20 13:37:30 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst6790lb # [ 266.618801] niks3-server[1047]: 2026/09/20 13:37:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6791lb # [ 266.622732] niks3-server[1047]: 2026/09/20 13:37:30 INFO Signed narinfos id=66 count=16792worker2 # [ 266.610523] nix-grpc-daemon[1437]: time=2026-09-20T13:37:29.897Z level=INFO msg="Uploading 1 narinfos"6793lb # [ 266.632951] niks3-server[1047]: 2026/09/20 13:37:30 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls6794lb # [ 266.641803] niks3-server[1047]: 2026/09/20 13:37:30 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6795lb # [ 266.644317] niks3-server[1047]: 2026/09/20 13:37:30 INFO Completed upload id=666796worker2 # [ 266.630881] nix-grpc-daemon[1437]: time=2026-09-20T13:37:29.917Z level=INFO msg="Upload complete. (365ms)"6797lb # [ 266.646502] niks3-server[1047]: 2026/09/20 13:37:30 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo6798worker2 # [ 266.634296] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=0 paths=1 nar_bytes_in=6246799worker2 # [ 266.640088] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766800worker2 # [ 266.648148] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6801worker2 # [ 266.653645] nix-daemon[854]: reaped child process 1665, status = succeeded6802worker2 # [ 266.657932] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6803client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...6804client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv6805worker2 # [ 267.670761] nix-daemon[854]: reaped child process 1679, status = succeeded6806client # farm-b> LOG-farm-b-drain6807worker2 # [ 267.790276] nix-daemon[854]: accepted connection from pid 1694, user nix-grpc-daemon (trusted)6808worker2 # [ 267.803105] nix-daemon[854]: reaped child process 1697, status = succeeded6809lb # [ 267.829469] niks3-server[1047]: 2026/09/20 13:37:31 INFO Received uploads request method=POST path=/api/pending_closures6810worker2 # [ 267.823917] nix-grpc-daemon[1437]: time=2026-09-20T13:37:31.110Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6811worker2 # [ 267.825617] nix-grpc-daemon[1437]: time=2026-09-20T13:37:31.112Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"6812lb # [ 267.870152] niks3-server[1047]: 2026/09/20 13:37:31 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst6813lb # [ 267.879340] niks3-server[1047]: 2026/09/20 13:37:31 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv6814lb # [ 267.889837] niks3-server[1047]: 2026/09/20 13:37:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6815worker2 # [ 267.881872] nix-grpc-daemon[1437]: time=2026-09-20T13:37:31.168Z level=INFO msg="Uploading 1 narinfos"6816lb # [ 267.896441] niks3-server[1047]: 2026/09/20 13:37:31 INFO Signed narinfos id=67 count=16817lb # [ 267.899795] niks3-server[1047]: 2026/09/20 13:37:31 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls6818lb # [ 267.909939] niks3-server[1047]: 2026/09/20 13:37:31 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6819lb # [ 267.915145] niks3-server[1047]: 2026/09/20 13:37:31 INFO Completed upload id=676820worker2 # [ 267.901809] nix-grpc-daemon[1437]: time=2026-09-20T13:37:31.188Z level=INFO msg="Upload complete. (237ms)"6821lb # [ 267.918019] niks3-server[1047]: 2026/09/20 13:37:31 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo6822worker2 # [ 267.909676] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=16823worker2 # [ 267.918628] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:31Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=06824worker2 # [ 267.922587] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:31Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=16825worker2 # [ 267.931716] nix-daemon[854]: reaped child process 1677, status = succeeded6826worker2 # [ 267.962856] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766827worker2 # [ 267.975517] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346766828worker2 # [ 267.988139] nix-grpc-daemon[1437]: time=2026-09-20T13:37:31.275Z level=INFO msg="All 2 paths already cached"6829client # copying 1 paths to 'lb:50051'...6830client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...6831worker2 # [ 267.996537] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346766832worker2 # [ 268.006168] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6833worker2 # [ 268.171701] nix-daemon[854]: accepted connection from pid 1703, user nix-grpc-daemon (trusted)6834worker2 # [ 268.184712] nix-daemon[854]: reaped child process 1706, status = succeeded6835worker2 # [ 268.326096] nix-daemon[854]: accepted connection from pid 1708, user nix-grpc-daemon (trusted)6836worker2 # [ 268.338339] nix-daemon[854]: reaped child process 1711, status = succeeded6837lb # [ 268.366963] niks3-server[1047]: 2026/09/20 13:37:31 INFO Received uploads request method=POST path=/api/pending_closures6838worker2 # [ 268.361298] nix-grpc-daemon[1437]: time=2026-09-20T13:37:31.648Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6839worker2 # [ 268.362671] nix-grpc-daemon[1437]: time=2026-09-20T13:37:31.648Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"6840lb # [ 268.393780] niks3-server[1047]: 2026/09/20 13:37:32 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst6841lb # [ 268.405131] niks3-server[1047]: 2026/09/20 13:37:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6842lb # [ 268.409556] niks3-server[1047]: 2026/09/20 13:37:32 INFO Signed narinfos id=68 count=16843worker2 # [ 268.397718] nix-grpc-daemon[1437]: time=2026-09-20T13:37:31.684Z level=INFO msg="Uploading 1 narinfos"6844lb # [ 268.415278] niks3-server[1047]: 2026/09/20 13:37:32 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls6845lb # [ 268.425867] niks3-server[1047]: 2026/09/20 13:37:32 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6846worker2 # [ 268.416112] nix-grpc-daemon[1437]: time=2026-09-20T13:37:31.703Z level=INFO msg="Upload complete. (398ms)"6847lb # [ 268.430840] niks3-server[1047]: 2026/09/20 13:37:32 INFO Completed upload id=686848worker2 # [ 268.418170] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=0 paths=1 nar_bytes_in=7526849lb # [ 268.436246] niks3-server[1047]: 2026/09/20 13:37:32 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo6850worker2 # [ 268.423080] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346766851worker2 # [ 268.433413] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6852worker2 # [ 268.435452] nix-daemon[854]: reaped child process 1701, status = succeeded6853worker2 # [ 268.440363] nix-daemon[854]: accepted connection from pid 1369, user nix-grpc-daemon (trusted)6854client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...6855client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv6856client # farm-top> LOG-farm-top-drain6857worker2 # [ 269.060993] nix-daemon[854]: reaped child process 1715, status = succeeded6858worker2 # [ 269.203972] nix-daemon[854]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6859worker2 # [ 269.216633] nix-daemon[854]: reaped child process 1733, status = succeeded6860lb # [ 269.246282] niks3-server[1047]: 2026/09/20 13:37:32 INFO Received uploads request method=POST path=/api/pending_closures6861worker2 # [ 269.236329] nix-grpc-daemon[1437]: time=2026-09-20T13:37:32.523Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6862worker2 # [ 269.237785] nix-grpc-daemon[1437]: time=2026-09-20T13:37:32.524Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"6863lb # [ 269.265851] niks3-server[1047]: 2026/09/20 13:37:32 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv6864lb # [ 269.270091] niks3-server[1047]: 2026/09/20 13:37:32 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst6865lb # [ 269.274058] niks3-server[1047]: 2026/09/20 13:37:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6866lb # [ 269.276632] niks3-server[1047]: 2026/09/20 13:37:32 INFO Signed narinfos id=69 count=16867worker2 # [ 269.262242] nix-grpc-daemon[1437]: time=2026-09-20T13:37:32.549Z level=INFO msg="Uploading 1 narinfos"6868lb # [ 269.280106] niks3-server[1047]: 2026/09/20 13:37:32 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls6869lb # [ 269.284149] niks3-server[1047]: 2026/09/20 13:37:32 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6870lb # [ 269.287648] niks3-server[1047]: 2026/09/20 13:37:32 INFO Completed upload id=696871worker2 # [ 269.272788] nix-grpc-daemon[1437]: time=2026-09-20T13:37:32.559Z level=INFO msg="Upload complete. (226ms)"6872lb # [ 269.289349] niks3-server[1047]: 2026/09/20 13:37:32 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo6873worker2 # [ 269.275193] nix-grpc-daemon[1369]: ts=2026-09-20T13:37:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34676 duration_s=0 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=16874worker2 # [ 269.282505] nix-daemon[854]: reaped child process 1713, status = succeeded6875client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 4.69 seconds)6876client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath6877lb # [ 269.393886] envoy[983]: conn peer=[2001:db8:1::1]:56162 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300476878lb # [ 269.395468] envoy[983]: conn peer=[2001:db8:1::1]:54556 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9486 tx=3794 ms=45036879client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.19 seconds)6880worker1: must succeed: rm /nix/.rw-store/fill6881worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.13 seconds)6882worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6883worker1 # [ 270.521748] nix-grpc-daemon[1693]: ts=2026-09-20T13:37:34Z level=info event=healthy reason=min_free6884worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.10 seconds)6885(finished: subtest: low disk drains a worker and builds go to the other, in 7.28 seconds)6886(finished: run the VM test script, in 272.07 seconds)6887test script finished in 272.25s6888cleanup6889kill QemuMachine (pid 49)6890client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)6891kill QemuMachine (pid 50)6892lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)6893kill QemuMachine (pid 51)6894worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)6895kill QemuMachine (pid 52)6896worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)6897(finished: cleanup, in 0.98 seconds)6898additionally exposed symbols:6899 client, lb, worker1, worker2,6900 vlan1,6901 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh

succeeded vm-test-run-nix-grpc-farm checks.x86_64-linux.farm · build #419 · raw

succeeded `vm-test-run-nix-grpc-farm` `checks.x86_64-linux.farm` · build #419 · raw