vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #477
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12lb # Disk image does not exist, creating the virtualisation disk image...13lb: QEMU running (pid 49)14worker1 # Disk image does not exist, creating the virtualisation disk image...15worker1: QEMU running (pid 51)16worker1 # Formatting '/build/vm-state-worker1/tmp.cgEa4SnSjp', fmt=raw size=107374182417lb # Formatting '/build/vm-state-lb/tmp.EblM4DtEcq', fmt=raw size=107374182418worker1 # mke2fs 1.47.4 (6-Mar-2025)19lb # mke2fs 1.47.4 (6-Mar-2025)20worker1 # Discarding device blocks: 0/262144 done21lb # Discarding device blocks: 0/262144 done22worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes23worker1 # Filesystem UUID: 5259bd96-8cd7-4831-9064-71d4b9d7efaf24lb # Creating filesystem with 262144 4k blocks and 65536 inodes25worker2 # Disk image does not exist, creating the virtualisation disk image...26worker2: QEMU running (pid 52)27client: QEMU running (pid 50)28client # Disk image does not exist, creating the virtualisation disk image...29lb # Filesystem UUID: f7ee3118-0757-4b77-98a4-0713a02c5a8e30worker2 # Formatting '/build/vm-state-worker2/tmp.pObkHtLtE5', fmt=raw size=107374182431worker1 # Superblock backups stored on blocks:32client # Formatting '/build/vm-state-client/tmp.2eoLU433ML', fmt=raw size=107374182433worker2 # mke2fs 1.47.4 (6-Mar-2025)34worker1 # 32768, 98304, 163840, 22937635client # mke2fs 1.47.4 (6-Mar-2025)36worker2 # Discarding device blocks: 0/262144 done37worker1 # 38client # Discarding device blocks: 0/262144 done39worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes40worker1 # Allocating group tables: 0/8 done41client # Creating filesystem with 262144 4k blocks and 65536 inodes42worker2 # Filesystem UUID: 7389df41-9423-448c-93b5-8496b1a23aaf43worker1 # Writing inode tables: 0/8 done44client # Filesystem UUID: 252b84b7-5432-4802-86ca-2266ee4a3fc445worker2 # Superblock backups stored on blocks:46worker1 # Creating journal (8192 blocks): done47client # Superblock backups stored on blocks:48worker2 # 32768, 98304, 163840, 22937649worker1 # Writing superblocks and filesystem accounting information: 0/8 done50client # 32768, 98304, 163840, 22937651(finished: start all VMs, in 0.19 seconds)52worker1 # 53client # 54client: waiting for unit network-addresses-eth1.service55worker1 # Virtualisation disk image created.56client # Allocating group tables: 0/8 done57client: waiting for the VM to finish booting58worker2 # 59lb # Superblock backups stored on blocks:60worker2 # Allocating group tables: 0/8 done61lb # 32768, 98304, 163840, 22937662worker2 # Writing inode tables: 0/8 done63lb # 64worker2 # Creating journal (8192 blocks): done65lb # Allocating group tables: 0/8 done66worker2 # Writing superblocks and filesystem accounting information: 0/8 done67lb # Writing inode tables: 0/8 done68worker2 # 69lb # Creating journal (8192 blocks): done70worker2 # Virtualisation disk image created.71lb # Writing superblocks and filesystem accounting information: 0/8 done72client # Writing inode tables: 0/8 done73lb # 74client # Creating journal (8192 blocks): done75lb # Virtualisation disk image created.76client # Writing superblocks and filesystem accounting information: 0/8 done77client # 78client # Virtualisation disk image created.79lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83worker1 # 84worker1 # 85worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0086lb # 87lb # 88lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0089worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...90worker1 # 91worker1 # 92worker1 # 93worker1 # 94worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB0095lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...96lb # 97lb # 98lb # 99lb # 100lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00101worker2 # 102worker2 # 103worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...104worker1 # 105worker1 # 106worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00107lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...108lb # 109lb # 110worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...111worker2 # 112worker2 # 113worker2 # 114worker2 # 115worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00116client # 117client # 118worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...119worker2 # 120worker2 # 121client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00122client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...123client # 124client # 125client # 126client # 127client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00128client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129client # 130client # 131worker1 # Booting from ROM...132lb # Booting from ROM...133worker2 # Booting from ROM...134client # Booting from ROM...135worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/pk0jp94yycs3bpxqsr991pq4ma6d15m4-nixos-system-worker1-test/init regInfo=/nix/store/x795jz123l8nr1yq1b29j4ikywqys74b-closure-info/registration console=ttyS0,115200n8 console=tty0137worker1 # [ 0.000000] BIOS-provided physical RAM map:138worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved141worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable142worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved143worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved144worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved145worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved146worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved147worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved148worker1 # [ 0.000000] NX (Execute Disable) protection: active149worker1 # [ 0.000000] APIC: Static calls initialized150worker1 # [ 0.000000] SMBIOS 2.8 present.151worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014152worker1 # [ 0.000000] DMI: Memory slots populated: 1/1153worker1 # [ 0.000000] Hypervisor detected: KVM154worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000155worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00156lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026157worker1 # [ 0.000000] kvm-clock: using sched offset of 464075159 cycles158worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns159worker1 # [ 0.000004] tsc: Detected 2400.012 MHz processor160worker1 # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000161lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/y7n4gfnnxcd9ip1gkd933mnr9bppn4da-nixos-system-lb-test/init regInfo=/nix/store/q685cp7hahmfghdqgwvns1qfgrgy9fga-closure-info/registration console=ttyS0,115200n8 console=tty0162worker1 # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs163lb # [ 0.000000] BIOS-provided physical RAM map:164worker1 # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT165lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable166worker1 # [ 0.002727] found SMP MP-table at [mem 0x000f5450-0x000f545f]167lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved168worker1 # [ 0.002738] Using GB pages for direct mapping169worker1 # [ 0.002874] RAMDISK: [mem 0x5e367000-0x5ffcffff]170lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved171worker1 # [ 0.002883] ACPI: Early table checksum verification disabled172lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable173worker1 # [ 0.002886] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )174lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved175worker1 # [ 0.002889] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)176lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved177worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026178worker1 # [ 0.002893] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)179lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved180lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved181client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026182worker1 # [ 0.002900] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)183worker1 # [ 0.002902] ACPI: FACS 0x000000005FFE0000 000040184lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved185worker1 # [ 0.002904] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)186lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved187lb # [ 0.000000] NX (Execute Disable) protection: active188worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ydfsr3h0qrrw30ippmh3hy2qppn6jwqd-nixos-system-worker2-test/init regInfo=/nix/store/q144d5gxymiw79c8bj2qbii20acjcvra-closure-info/registration console=ttyS0,115200n8 console=tty0189worker1 # [ 0.002905] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)190lb # [ 0.000000] APIC: Static calls initialized191lb # [ 0.000000] SMBIOS 2.8 present.192worker2 # [ 0.000000] BIOS-provided physical RAM map:193client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test/init regInfo=/nix/store/v58y4g30w74cah80479jy4r0f6qilcdy-closure-info/registration console=ttyS0,115200n8 console=tty0194worker1 # [ 0.002907] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)195worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable196client # [ 0.000000] BIOS-provided physical RAM map:197lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014198worker1 # [ 0.002908] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)199worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved200lb # [ 0.000000] DMI: Memory slots populated: 1/1201client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable202lb # [ 0.000000] Hypervisor detected: KVM203worker1 # [ 0.002910] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]204worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved205client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved206lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000207worker1 # [ 0.002911] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]208worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable209lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00210client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved211worker1 # [ 0.002911] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]212lb # [ 0.000001] kvm-clock: using sched offset of 489802803 cycles213worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved214client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable215worker1 # [ 0.002912] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]216worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved217client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved218lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns219worker1 # [ 0.002912] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]220worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved221client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved222lb # [ 0.000004] tsc: Detected 2400.012 MHz processor223worker1 # [ 0.002913] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]224worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved225lb # [ 0.000810] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000226client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved227worker1 # [ 0.002913] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]228worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved229lb # [ 0.000837] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs230client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved231worker1 # [ 0.003135] No NUMA configuration found232worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved233lb # [ 0.000839] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT234client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved235worker1 # [ 0.003136] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]236worker2 # [ 0.000000] NX (Execute Disable) protection: active237lb # [ 0.002732] found SMP MP-table at [mem 0x000f5450-0x000f545f]238client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved239worker1 # [ 0.003138] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]240worker2 # [ 0.000000] APIC: Static calls initialized241worker1 # [ 0.005766] Zone ranges:242client # [ 0.000000] NX (Execute Disable) protection: active243worker2 # [ 0.000000] SMBIOS 2.8 present.244lb # [ 0.002743] Using GB pages for direct mapping245lb # [ 0.002857] RAMDISK: [mem 0x5e367000-0x5ffcffff]246client # [ 0.000000] APIC: Static calls initialized247worker1 # [ 0.005767] DMA [mem 0x0000000000001000-0x0000000000ffffff]248client # [ 0.000000] SMBIOS 2.8 present.249lb # [ 0.002865] ACPI: Early table checksum verification disabled250worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014251worker1 # [ 0.005769] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]252worker1 # [ 0.005770] Normal empty253lb # [ 0.002868] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )254worker2 # [ 0.000000] DMI: Memory slots populated: 1/1255worker1 # [ 0.005770] Device empty256client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014257worker2 # [ 0.000000] Hypervisor detected: KVM258worker1 # [ 0.005771] Movable zone start for each node259client # [ 0.000000] DMI: Memory slots populated: 1/1260lb # [ 0.002872] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)261worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000262worker1 # [ 0.005772] Early memory node ranges263client # [ 0.000000] Hypervisor detected: KVM264worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00265client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000266lb # [ 0.002876] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)267worker1 # [ 0.005772] node 0: [mem 0x0000000000001000-0x000000000009efff]268worker2 # [ 0.000000] kvm-clock: using sched offset of 492094460 cycles269client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00270worker1 # [ 0.005773] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]271lb # [ 0.002884] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)272client # [ 0.000001] kvm-clock: using sched offset of 486036951 cycles273worker1 # [ 0.005774] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]274lb # [ 0.002886] ACPI: FACS 0x000000005FFE0000 000040275worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns276worker1 # [ 0.005803] On node 0, zone DMA: 1 pages in unavailable ranges277worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor278client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns279lb # [ 0.002887] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)280worker1 # [ 0.006069] On node 0, zone DMA: 97 pages in unavailable ranges281client # [ 0.000005] tsc: Detected 2400.012 MHz processor282worker2 # [ 0.000804] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000283lb # [ 0.002889] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)284client # [ 0.000809] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000285worker1 # [ 0.033028] On node 0, zone DMA32: 40 pages in unavailable ranges286worker2 # [ 0.000832] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs287worker1 # [ 0.033478] ACPI: PM-Timer IO Port: 0x608288lb # [ 0.002890] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)289client # [ 0.000835] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs290worker2 # [ 0.000835] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT291worker1 # [ 0.033489] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])292worker2 # [ 0.002747] found SMP MP-table at [mem 0x000f5450-0x000f545f]293client # [ 0.000838] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT294lb # [ 0.002892] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)295worker1 # [ 0.033515] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23296worker2 # [ 0.002758] Using GB pages for direct mapping297client # [ 0.002727] found SMP MP-table at [mem 0x000f5450-0x000f545f]298lb # [ 0.002893] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]299worker2 # [ 0.002913] RAMDISK: [mem 0x5e367000-0x5ffcffff]300worker1 # [ 0.033518] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)301client # [ 0.002739] Using GB pages for direct mapping302worker2 # [ 0.002922] ACPI: Early table checksum verification disabled303worker1 # [ 0.033519] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)304client # [ 0.002835] RAMDISK: [mem 0x5e366000-0x5ffcffff]305lb # [ 0.002894] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]306worker2 # [ 0.002924] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )307worker1 # [ 0.033520] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)308client # [ 0.002844] ACPI: Early table checksum verification disabled309lb # [ 0.002895] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]310client # [ 0.002847] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )311worker1 # [ 0.033521] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)312worker2 # [ 0.002929] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)313lb # [ 0.002895] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]314worker1 # [ 0.033522] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)315client # [ 0.002850] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)316lb # [ 0.002896] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]317worker2 # [ 0.002933] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)318worker1 # [ 0.033524] ACPI: Using ACPI (MADT) for SMP configuration information319lb # [ 0.002896] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]320client # [ 0.002854] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)321worker1 # [ 0.033525] ACPI: HPET id: 0x8086a201 base: 0xfed00000322worker2 # [ 0.002940] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)323lb # [ 0.002897] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]324worker1 # [ 0.033528] TSC deadline timer available325worker2 # [ 0.002943] ACPI: FACS 0x000000005FFE0000 000040326client # [ 0.002861] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)327lb # [ 0.003131] No NUMA configuration found328worker1 # [ 0.033533] CPU topo: Max. logical packages: 1329client # [ 0.002863] ACPI: FACS 0x000000005FFE0000 000040330worker2 # [ 0.002944] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)331worker1 # [ 0.033533] CPU topo: Max. logical dies: 1332lb # [ 0.003132] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]333worker1 # [ 0.033534] CPU topo: Max. dies per package: 1334client # [ 0.002864] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)335lb # [ 0.003134] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]336worker2 # [ 0.002946] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)337worker1 # [ 0.033537] CPU topo: Max. threads per core: 1338lb # [ 0.005544] Zone ranges:339worker1 # [ 0.033538] CPU topo: Num. cores per package: 1340client # [ 0.002866] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)341lb # [ 0.005545] DMA [mem 0x0000000000001000-0x0000000000ffffff]342worker2 # [ 0.002947] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)343worker1 # [ 0.033538] CPU topo: Num. threads per package: 1344lb # [ 0.005547] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]345client # [ 0.002868] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)346worker1 # [ 0.033538] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs347lb # [ 0.005548] Normal empty348worker2 # [ 0.002949] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)349lb # [ 0.005549] Device empty350worker1 # [ 0.033556] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()351client # [ 0.002869] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)352worker2 # [ 0.002950] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]353lb # [ 0.005549] Movable zone start for each node354lb # [ 0.005550] Early memory node ranges355client # [ 0.002870] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]356worker1 # [ 0.033585] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]357worker2 # [ 0.002951] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]358lb # [ 0.005550] node 0: [mem 0x0000000000001000-0x000000000009efff]359client # [ 0.002871] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]360worker1 # [ 0.033587] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]361worker2 # [ 0.002952] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]362lb # [ 0.005552] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]363client # [ 0.002872] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]364worker1 # [ 0.033588] [mem 0x60000000-0xafffffff] available for PCI devices365worker2 # [ 0.002952] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]366lb # [ 0.005553] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]367worker1 # [ 0.033590] Booting paravirtualized kernel on KVM368client # [ 0.002872] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]369worker2 # [ 0.002953] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]370lb # [ 0.005574] On node 0, zone DMA: 1 pages in unavailable ranges371client # [ 0.002873] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]372lb # [ 0.005857] On node 0, zone DMA: 97 pages in unavailable ranges373worker1 # [ 0.033592] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns374worker2 # [ 0.002953] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]375lb # [ 0.034233] On node 0, zone DMA32: 40 pages in unavailable ranges376client # [ 0.002873] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]377worker1 # [ 0.038022] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1378worker2 # [ 0.002954] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]379lb # [ 0.034704] ACPI: PM-Timer IO Port: 0x608380worker2 # [ 0.003187] No NUMA configuration found381client # [ 0.002874] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]382worker1 # [ 0.040206] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152383lb # [ 0.034716] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])384client # [ 0.003097] No NUMA configuration found385worker2 # [ 0.003188] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]386worker1 # [ 0.040250] kvm-guest: PV spinlocks disabled, single CPU387lb # [ 0.034747] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23388client # [ 0.003097] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]389worker2 # [ 0.003190] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]390worker2 # [ 0.005611] Zone ranges:391lb # [ 0.034750] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)392client # [ 0.003100] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]393client # [ 0.005391] Zone ranges:394worker2 # [ 0.005612] DMA [mem 0x0000000000001000-0x0000000000ffffff]395lb # [ 0.034751] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)396client # [ 0.005392] DMA [mem 0x0000000000001000-0x0000000000ffffff]397worker2 # [ 0.005614] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]398lb # [ 0.034752] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)399worker2 # [ 0.005615] Normal empty400client # [ 0.005393] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]401worker2 # [ 0.005616] Device empty402client # [ 0.005395] Normal empty403lb # [ 0.034753] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)404worker2 # [ 0.005616] Movable zone start for each node405client # [ 0.005395] Device empty406worker1 # [ 0.040251] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/pk0jp94yycs3bpxqsr991pq4ma6d15m4-nixos-system-worker1-test/init regInfo=/nix/store/x795jz123l8nr1yq1b29j4ikywqys74b-closure-info/registration console=ttyS0,115200n8 console=tty0407worker2 # [ 0.005617] Early memory node ranges408client # [ 0.005396] Movable zone start for each node409lb # [ 0.034754] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)410client # [ 0.005397] Early memory node ranges411worker2 # [ 0.005618] node 0: [mem 0x0000000000001000-0x000000000009efff]412lb # [ 0.034756] ACPI: Using ACPI (MADT) for SMP configuration information413client # [ 0.005397] node 0: [mem 0x0000000000001000-0x000000000009efff]414worker1 # [ 0.040343] Unknown kernel command line parameters "regInfo=/nix/store/x795jz123l8nr1yq1b29j4ikywqys74b-closure-info/registration", will be passed to user space.415lb # [ 0.034757] ACPI: HPET id: 0x8086a201 base: 0xfed00000416worker2 # [ 0.005619] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]417worker1 # [ 0.040355] random: crng init done418lb # [ 0.034761] TSC deadline timer available419client # [ 0.005398] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]420worker2 # [ 0.005620] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]421lb # [ 0.034765] CPU topo: Max. logical packages: 1422worker1 # [ 0.040356] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes423client # [ 0.005400] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]424worker2 # [ 0.005641] On node 0, zone DMA: 1 pages in unavailable ranges425lb # [ 0.034766] CPU topo: Max. logical dies: 1426client # [ 0.005417] On node 0, zone DMA: 1 pages in unavailable ranges427worker1 # [ 0.042543] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)428worker2 # [ 0.005925] On node 0, zone DMA: 97 pages in unavailable ranges429lb # [ 0.034766] CPU topo: Max. dies per package: 1430lb # [ 0.034770] CPU topo: Max. threads per core: 1431client # [ 0.005687] On node 0, zone DMA: 97 pages in unavailable ranges432worker2 # [ 0.034152] On node 0, zone DMA32: 40 pages in unavailable ranges433worker1 # [ 0.043623] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)434lb # [ 0.034770] CPU topo: Num. cores per package: 1435worker2 # [ 0.034614] ACPI: PM-Timer IO Port: 0x608436client # [ 0.032792] On node 0, zone DMA32: 40 pages in unavailable ranges437worker1 # [ 0.043653] Fallback order for Node 0: 0438lb # [ 0.034771] CPU topo: Num. threads per package: 1439client # [ 0.033244] ACPI: PM-Timer IO Port: 0x608440worker2 # [ 0.034626] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])441worker1 # [ 0.043656] Built 1 zonelists, mobility grouping on. Total pages: 393078442worker1 # [ 0.043657] Policy zone: DMA32443lb # [ 0.034771] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs444client # [ 0.033255] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])445worker2 # [ 0.034656] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23446lb # [ 0.034790] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()447worker1 # [ 0.046461] mem auto-init: stack:all(zero), heap alloc:on, heap free:off448client # [ 0.033281] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23449worker2 # [ 0.034658] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)450worker1 # [ 0.049539] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1451client # [ 0.033284] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)452lb # [ 0.034826] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]453worker2 # [ 0.034660] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)454worker1 # [ 0.052940] allocated 3145728 bytes of page_ext455client # [ 0.033285] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)456lb # [ 0.034827] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]457worker2 # [ 0.034661] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)458worker1 # [ 0.062569] ftrace: allocating 48731 entries in 192 pages459client # [ 0.033286] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)460lb # [ 0.034829] [mem 0x60000000-0xafffffff] available for PCI devices461worker1 # [ 0.062571] ftrace: allocated 192 pages with 2 groups462worker2 # [ 0.034662] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)463worker1 # [ 0.063415] Dynamic Preempt: lazy464lb # [ 0.034830] Booting paravirtualized kernel on KVM465client # [ 0.033288] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)466worker2 # [ 0.034663] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)467worker1 # [ 0.063590] rcu: Preemptible hierarchical RCU implementation.468client # [ 0.033288] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)469worker1 # [ 0.063591] rcu: RCU event tracing is enabled.470worker2 # [ 0.034665] ACPI: Using ACPI (MADT) for SMP configuration information471lb # [ 0.034834] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns472client # [ 0.033291] ACPI: Using ACPI (MADT) for SMP configuration information473worker2 # [ 0.034666] ACPI: HPET id: 0x8086a201 base: 0xfed00000474worker1 # [ 0.063591] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.475lb # [ 0.039289] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1476worker2 # [ 0.034670] TSC deadline timer available477client # [ 0.033292] ACPI: HPET id: 0x8086a201 base: 0xfed00000478worker1 # [ 0.063593] Trampoline variant of Tasks RCU enabled.479client # [ 0.033295] TSC deadline timer available480lb # [ 0.041580] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152481worker2 # [ 0.034674] CPU topo: Max. logical packages: 1482worker1 # [ 0.063594] Rude variant of Tasks RCU enabled.483client # [ 0.033300] CPU topo: Max. logical packages: 1484worker2 # [ 0.034675] CPU topo: Max. logical dies: 1485lb # [ 0.041625] kvm-guest: PV spinlocks disabled, single CPU486worker1 # [ 0.063594] Tracing variant of Tasks RCU enabled.487client # [ 0.033300] CPU topo: Max. logical dies: 1488worker2 # [ 0.034675] CPU topo: Max. dies per package: 1489client # [ 0.033301] CPU topo: Max. dies per package: 1490worker2 # [ 0.034679] CPU topo: Max. threads per core: 1491worker1 # [ 0.063595] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.492client # [ 0.033304] CPU topo: Max. threads per core: 1493worker2 # [ 0.034680] CPU topo: Num. cores per package: 1494worker1 # [ 0.063595] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1495client # [ 0.033305] CPU topo: Num. cores per package: 1496worker2 # [ 0.034680] CPU topo: Num. threads per package: 1497client # [ 0.033305] CPU topo: Num. threads per package: 1498worker1 # [ 0.063610] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.499worker2 # [ 0.034681] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs500client # [ 0.033305] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs501lb # [ 0.041626] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/y7n4gfnnxcd9ip1gkd933mnr9bppn4da-nixos-system-lb-test/init regInfo=/nix/store/q685cp7hahmfghdqgwvns1qfgrgy9fga-closure-info/registration console=ttyS0,115200n8 console=tty0502worker2 # [ 0.034700] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()503worker1 # [ 0.063612] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.504client # [ 0.033323] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()505worker2 # [ 0.034731] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]506worker1 # [ 0.063612] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.507client # [ 0.033355] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]508lb # [ 0.041729] Unknown kernel command line parameters "regInfo=/nix/store/q685cp7hahmfghdqgwvns1qfgrgy9fga-closure-info/registration", will be passed to user space.509worker2 # [ 0.034732] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]510worker1 # [ 0.067970] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16511lb # [ 0.041742] random: crng init done512client # [ 0.033356] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]513worker2 # [ 0.034733] [mem 0x60000000-0xafffffff] available for PCI devices514worker1 # [ 0.068249] rcu: srcu_init: Setting srcu_struct sizes based on contention.515lb # [ 0.041743] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes516client # [ 0.033357] [mem 0x60000000-0xafffffff] available for PCI devices517worker2 # [ 0.034735] Booting paravirtualized kernel on KVM518client # [ 0.033359] Booting paravirtualized kernel on KVM519lb # [ 0.043994] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)520worker1 # [ 0.068256] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns521worker2 # [ 0.034738] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns522client # [ 0.033362] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns523lb # [ 0.045397] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)524worker1 # [ 0.068358] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)525lb # [ 0.045429] Fallback order for Node 0: 0526worker2 # [ 0.039266] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1527worker1 # [ 0.071876] Console: colour VGA+ 80x25528client # [ 0.037803] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1529lb # [ 0.045432] Built 1 zonelists, mobility grouping on. Total pages: 393078530worker2 # [ 0.041833] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152531worker1 # [ 0.071879] printk: legacy console [tty0] enabled532lb # [ 0.045433] Policy zone: DMA32533client # [ 0.040250] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152534worker2 # [ 0.041889] kvm-guest: PV spinlocks disabled, single CPU535worker1 # [ 0.101279] printk: legacy console [ttyS0] enabled536client # [ 0.040291] kvm-guest: PV spinlocks disabled, single CPU537lb # [ 0.048072] mem auto-init: stack:all(zero), heap alloc:on, heap free:off538worker1 # [ 0.198780] ACPI: Core revision 20250807539lb # [ 0.051333] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1540lb # [ 0.054848] allocated 3145728 bytes of page_ext541lb # [ 0.064889] ftrace: allocating 48731 entries in 192 pages542worker1 # [ 0.199620] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns543lb # [ 0.064891] ftrace: allocated 192 pages with 2 groups544worker1 # [ 0.201098] APIC: Switch to symmetric I/O mode setup545lb # [ 0.065763] Dynamic Preempt: lazy546worker2 # [ 0.041891] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ydfsr3h0qrrw30ippmh3hy2qppn6jwqd-nixos-system-worker2-test/init regInfo=/nix/store/q144d5gxymiw79c8bj2qbii20acjcvra-closure-info/registration console=ttyS0,115200n8 console=tty0547worker1 # [ 0.202023] x2apic enabled548lb # [ 0.065946] rcu: Preemptible hierarchical RCU implementation.549client # [ 0.040293] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test/init regInfo=/nix/store/v58y4g30w74cah80479jy4r0f6qilcdy-closure-info/registration console=ttyS0,115200n8 console=tty0550lb # [ 0.065946] rcu: RCU event tracing is enabled.551worker1 # [ 0.202753] APIC: Switched APIC routing to: physical x2apic552lb # [ 0.065947] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.553worker2 # [ 0.041986] Unknown kernel command line parameters "regInfo=/nix/store/q144d5gxymiw79c8bj2qbii20acjcvra-closure-info/registration", will be passed to user space.554client # [ 0.040382] Unknown kernel command line parameters "regInfo=/nix/store/v58y4g30w74cah80479jy4r0f6qilcdy-closure-info/registration", will be passed to user space.555worker2 # [ 0.041999] random: crng init done556lb # [ 0.065948] Trampoline variant of Tasks RCU enabled.557client # [ 0.040395] random: crng init done558lb # [ 0.065949] Rude variant of Tasks RCU enabled.559worker2 # [ 0.042000] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes560client # [ 0.040395] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes561lb # [ 0.065949] Tracing variant of Tasks RCU enabled.562worker1 # [ 0.204664] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1563worker2 # [ 0.044226] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)564client # [ 0.042521] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)565lb # [ 0.065950] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.566worker1 # [ 0.205606] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns567worker2 # [ 0.045342] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)568lb # [ 0.065951] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1569client # [ 0.043590] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)570worker2 # [ 0.045374] Fallback order for Node 0: 0571client # [ 0.043620] Fallback order for Node 0: 0572lb # [ 0.065966] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.573worker2 # [ 0.045377] Built 1 zonelists, mobility grouping on. Total pages: 393078574worker1 # [ 0.207202] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)575client # [ 0.043622] Built 1 zonelists, mobility grouping on. Total pages: 393078576worker2 # [ 0.045378] Policy zone: DMA32577client # [ 0.043623] Policy zone: DMA32578lb # [ 0.065968] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.579worker2 # [ 0.047973] mem auto-init: stack:all(zero), heap alloc:on, heap free:off580worker1 # [ 0.208287] x86/cpu: User Mode Instruction Prevention (UMIP) activated581client # [ 0.046114] mem auto-init: stack:all(zero), heap alloc:on, heap free:off582worker2 # [ 0.051255] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1583lb # [ 0.065969] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.584client # [ 0.049200] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1585worker1 # [ 0.209363] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127586worker2 # [ 0.055069] allocated 3145728 bytes of page_ext587client # [ 0.052854] allocated 3145728 bytes of page_ext588lb # [ 0.070741] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16589worker2 # [ 0.065190] ftrace: allocating 48731 entries in 192 pages590worker1 # [ 0.210200] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0591client # [ 0.062368] ftrace: allocating 48731 entries in 192 pages592lb # [ 0.071021] rcu: srcu_init: Setting srcu_struct sizes based on contention.593worker2 # [ 0.065192] ftrace: allocated 192 pages with 2 groups594client # [ 0.062370] ftrace: allocated 192 pages with 2 groups595worker2 # [ 0.066067] Dynamic Preempt: lazy596client # [ 0.063210] Dynamic Preempt: lazy597lb # [ 0.071028] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns598worker2 # [ 0.066239] rcu: Preemptible hierarchical RCU implementation.599worker1 # [ 0.211203] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto600client # [ 0.063371] rcu: Preemptible hierarchical RCU implementation.601worker2 # [ 0.066240] rcu: RCU event tracing is enabled.602client # [ 0.063371] rcu: RCU event tracing is enabled.603lb # [ 0.071136] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)604worker2 # [ 0.066240] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.605worker1 # [ 0.212200] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl606lb # [ 0.074676] Console: colour VGA+ 80x25607client # [ 0.063372] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.608worker2 # [ 0.066242] Trampoline variant of Tasks RCU enabled.609client # [ 0.063373] Trampoline variant of Tasks RCU enabled.610lb # [ 0.074679] printk: legacy console [tty0] enabled611worker1 # [ 0.213200] Transient Scheduler Attacks: Vulnerable: No microcode612worker2 # [ 0.066242] Rude variant of Tasks RCU enabled.613client # [ 0.063374] Rude variant of Tasks RCU enabled.614lb # [ 0.104098] printk: legacy console [ttyS0] enabled615worker2 # [ 0.066243] Tracing variant of Tasks RCU enabled.616client # [ 0.063374] Tracing variant of Tasks RCU enabled.617worker1 # [ 0.214199] Spectre V2 : Mitigation: Enhanced / Automatic IBRS618lb # [ 0.199718] ACPI: Core revision 20250807619worker2 # [ 0.066243] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.620client # [ 0.063375] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.621worker1 # [ 0.215199] Speculative Return Stack Overflow: Mitigation: Safe RET622worker2 # [ 0.066244] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1623client # [ 0.063376] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1624lb # [ 0.200564] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns625worker2 # [ 0.066266] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.626client # [ 0.063401] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.627lb # [ 0.202083] APIC: Switch to symmetric I/O mode setup628lb # [ 0.203035] x2apic enabled629worker2 # [ 0.066268] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.630client # [ 0.063403] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.631lb # [ 0.203822] APIC: Switched APIC routing to: physical x2apic632worker1 # [ 0.216199] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization633worker2 # [ 0.066269] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.634client # [ 0.063403] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.635worker2 # [ 0.070693] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16636worker1 # [ 0.218207] Spectre V2 : Enabling IBPB for BPF637client # [ 0.067730] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16638worker2 # [ 0.070970] rcu: srcu_init: Setting srcu_struct sizes based on contention.639client # [ 0.068006] rcu: srcu_init: Setting srcu_struct sizes based on contention.640lb # [ 0.205618] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1641worker1 # [ 0.218939] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier642worker2 # [ 0.070978] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns643client # [ 0.068012] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns644worker1 # [ 0.220200] active return thunk: srso_alias_return_thunk645lb # [ 0.206573] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns646client # [ 0.068113] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)647worker2 # [ 0.071084] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)648worker1 # [ 0.221030] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'649worker2 # [ 0.074672] Console: colour VGA+ 80x25650client # [ 0.071654] Console: colour VGA+ 80x25651lb # [ 0.208198] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)652client # [ 0.071657] printk: legacy console [tty0] enabled653worker2 # [ 0.074675] printk: legacy console [tty0] enabled654worker1 # [ 0.222199] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'655client # [ 0.101056] printk: legacy console [ttyS0] enabled656worker2 # [ 0.104258] printk: legacy console [ttyS0] enabled657lb # [ 0.209283] x86/cpu: User Mode Instruction Prevention (UMIP) activated658worker1 # [ 0.223199] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'659client # [ 0.196413] ACPI: Core revision 20250807660worker2 # [ 0.200671] ACPI: Core revision 20250807661lb # [ 0.210340] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127662worker1 # [ 0.224149] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'663lb # [ 0.211195] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0664client # [ 0.197318] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns665worker2 # [ 0.201614] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns666worker1 # [ 0.224968] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'667client # [ 0.198809] APIC: Switch to symmetric I/O mode setup668worker1 # [ 0.225907] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'669worker2 # [ 0.203190] APIC: Switch to symmetric I/O mode setup670client # [ 0.199768] x2apic enabled671lb # [ 0.212198] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto672worker2 # [ 0.204132] x2apic enabled673worker1 # [ 0.226943] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'674client # [ 0.200504] APIC: Switched APIC routing to: physical x2apic675worker2 # [ 0.204920] APIC: Switched APIC routing to: physical x2apic676lb # [ 0.214194] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl677worker1 # [ 0.228200] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256678lb # [ 0.215195] Transient Scheduler Attacks: Vulnerable: No microcode679worker1 # [ 0.229200] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64680client # [ 0.202308] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1681worker2 # [ 0.206755] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1682lb # [ 0.216194] Spectre V2 : Mitigation: Enhanced / Automatic IBRS683worker1 # [ 0.230199] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512684lb # [ 0.217195] Speculative Return Stack Overflow: Mitigation: Safe RET685worker1 # [ 0.231121] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024686client # [ 0.203275] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns687worker2 # [ 0.207740] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns688worker1 # [ 0.231893] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8689client # [ 0.204927] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)690worker2 # [ 0.209411] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)691worker1 # [ 0.232937] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.692client # [ 0.206010] x86/cpu: User Mode Instruction Prevention (UMIP) activated693worker2 # [ 0.210506] x86/cpu: User Mode Instruction Prevention (UMIP) activated694client # [ 0.207053] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127695lb # [ 0.218194] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization696worker2 # [ 0.211532] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127697lb # [ 0.220202] Spectre V2 : Enabling IBPB for BPF698client # [ 0.208618] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0699worker2 # [ 0.213102] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0700lb # [ 0.221195] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier701client # [ 0.208928] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto702worker2 # [ 0.214184] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto703lb # [ 0.222195] active return thunk: srso_alias_return_thunk704client # [ 0.210925] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl705worker2 # [ 0.215408] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl706lb # [ 0.223215] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'707client # [ 0.211925] Transient Scheduler Attacks: Vulnerable: No microcode708worker2 # [ 0.216408] Transient Scheduler Attacks: Vulnerable: No microcode709lb # [ 0.224195] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'710client # [ 0.213630] Spectre V2 : Mitigation: Enhanced / Automatic IBRS711worker2 # [ 0.218133] Spectre V2 : Mitigation: Enhanced / Automatic IBRS712lb # [ 0.225936] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'713client # [ 0.214611] Speculative Return Stack Overflow: Mitigation: Safe RET714worker2 # [ 0.219102] Speculative Return Stack Overflow: Mitigation: Safe RET715lb # [ 0.226922] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'716lb # [ 0.227964] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'717lb # [ 0.229013] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'718lb # [ 0.230053] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'719client # [ 0.214924] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization720worker2 # [ 0.220154] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization721client # [ 0.216931] Spectre V2 : Enabling IBPB for BPF722lb # [ 0.231195] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256723worker2 # [ 0.221416] Spectre V2 : Enabling IBPB for BPF724lb # [ 0.232194] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64725client # [ 0.217925] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier726worker2 # [ 0.222408] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier727lb # [ 0.233194] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512728client # [ 0.218925] active return thunk: srso_alias_return_thunk729worker2 # [ 0.223408] active return thunk: srso_alias_return_thunk730lb # [ 0.234194] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024731client # [ 0.220572] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'732worker2 # [ 0.225067] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'733lb # [ 0.235194] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8734client # [ 0.221841] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'735worker2 # [ 0.226407] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'736lb # [ 0.236194] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.737client # [ 0.222924] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'738worker2 # [ 0.227407] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'739client # [ 0.223883] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'740worker2 # [ 0.228367] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'741client # [ 0.224924] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'742worker2 # [ 0.229407] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'743client # [ 0.225924] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'744worker2 # [ 0.230406] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'745client # [ 0.226924] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'746worker2 # [ 0.231407] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'747client # [ 0.227925] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256748worker2 # [ 0.232408] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256749client # [ 0.228924] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64750worker2 # [ 0.233407] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64751client # [ 0.229924] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512752worker2 # [ 0.234408] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512753client # [ 0.230924] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024754worker2 # [ 0.235407] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024755client # [ 0.231924] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8756worker2 # [ 0.236407] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8757client # [ 0.232924] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.758worker2 # [ 0.237408] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759worker1 # [ 0.267092] Freeing SMP alternatives memory: 44K760worker1 # [ 0.267802] pid_max: default: 32768 minimum: 301761worker1 # [ 0.268288] LSM: initializing lsm=capability,landlock,yama,bpf,ima762worker1 # [ 0.269298] landlock: Up and running.763worker1 # [ 0.270200] Yama: becoming mindful.764worker1 # [ 0.271392] LSM support for eBPF active765worker1 # [ 0.272171] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)766worker1 # [ 0.273238] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)767worker1 # [ 0.276138] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)768worker1 # [ 0.276730] Performance Events: Fam17h+ core perfctr, AMD PMU driver.769worker1 # [ 0.277204] ... version: 2770worker1 # [ 0.277938] ... bit width: 48771worker1 # [ 0.278202] ... generic counters: 6772worker1 # [ 0.278946] ... generic bitmap: 000000000000003f773worker1 # [ 0.279255] ... fixed-purpose counters: 0774worker1 # [ 0.280007] ... fixed-purpose bitmap: 0000000000000000775worker1 # [ 0.280995] ... value mask: 0000ffffffffffff776worker1 # [ 0.281926] ... max period: 00007fffffffffff777worker1 # [ 0.282202] ... global_ctrl mask: 000000000000003f778lb # [ 0.271696] Freeing SMP alternatives memory: 44K779worker1 # [ 0.283240] signal: max sigframe size: 3376780lb # [ 0.272198] pid_max: default: 32768 minimum: 301781worker1 # [ 0.284048] rcu: Hierarchical SRCU implementation.782lb # [ 0.273303] LSM: initializing lsm=capability,landlock,yama,bpf,ima783worker1 # [ 0.284805] rcu: Max phase no-delay instances is 400.784lb # [ 0.274316] landlock: Up and running.785lb # [ 0.274953] Yama: becoming mindful.786lb # [ 0.275859] LSM support for eBPF active787lb # [ 0.276828] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)788lb # [ 0.278235] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)789worker1 # [ 0.289763] smp: Bringing up secondary CPUs ...790worker1 # [ 0.290216] smp: Brought up 1 node, 1 CPU791worker1 # [ 0.290941] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)792lb # [ 0.280649] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)793client # [ 0.268062] Freeing SMP alternatives memory: 44K794worker2 # [ 0.271880] Freeing SMP alternatives memory: 44K795client # [ 0.268838] pid_max: default: 32768 minimum: 301796lb # [ 0.281766] Performance Events: Fam17h+ core perfctr, AMD PMU driver.797worker1 # [ 0.291363] Memory: 1454536K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)798worker2 # [ 0.272411] pid_max: default: 32768 minimum: 301799lb # [ 0.282203] ... version: 2800client # [ 0.269541] LSM: initializing lsm=capability,landlock,yama,bpf,ima801worker1 # [ 0.292468] devtmpfs: initialized802lb # [ 0.282898] ... bit width: 48803worker2 # [ 0.273516] LSM: initializing lsm=capability,landlock,yama,bpf,ima804client # [ 0.270029] landlock: Up and running.805worker1 # [ 0.293266] x86/mm: Memory block size: 128MB806lb # [ 0.283197] ... generic counters: 6807client # [ 0.271344] Yama: becoming mindful.808worker2 # [ 0.274546] landlock: Up and running.809lb # [ 0.283885] ... generic bitmap: 000000000000003f810worker2 # [ 0.275169] Yama: becoming mindful.811client # [ 0.272138] LSM support for eBPF active812lb # [ 0.284197] ... fixed-purpose counters: 0813worker2 # [ 0.276038] LSM support for eBPF active814worker1 # [ 0.294986] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)815client # [ 0.272977] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)816lb # [ 0.284903] ... fixed-purpose bitmap: 0000000000000000817worker2 # [ 0.276547] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)818lb # [ 0.285197] ... value mask: 0000ffffffffffff819worker1 # [ 0.296090] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).820client # [ 0.273975] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)821lb # [ 0.286025] ... max period: 00007fffffffffff822worker1 # [ 0.297265] pinctrl core: initialized pinctrl subsystem823worker2 # [ 0.277452] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)824lb # [ 0.286819] ... global_ctrl mask: 000000000000003f825worker1 # [ 0.298351] PM: RTC time: 13:44:54, date: 2026-09-21826lb # [ 0.287294] signal: max sigframe size: 3376827client # [ 0.276424] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)828lb # [ 0.288094] rcu: Hierarchical SRCU implementation.829worker2 # [ 0.280611] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)830lb # [ 0.288810] rcu: Max phase no-delay instances is 400.831client # [ 0.277479] Performance Events: Fam17h+ core perfctr, AMD PMU driver.832client # [ 0.277929] ... version: 2833worker2 # [ 0.281988] Performance Events: Fam17h+ core perfctr, AMD PMU driver.834worker1 # [ 0.301887] NET: Registered PF_NETLINK/PF_ROUTE protocol family835client # [ 0.278655] ... bit width: 48836worker2 # [ 0.282417] ... version: 2837client # [ 0.278927] ... generic counters: 6838worker2 # [ 0.283148] ... bit width: 48839worker1 # [ 0.302708] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations840worker2 # [ 0.283410] ... generic counters: 6841client # [ 0.279633] ... generic bitmap: 000000000000003f842client # [ 0.279976] ... fixed-purpose counters: 0843worker2 # [ 0.284137] ... generic bitmap: 000000000000003f844worker1 # [ 0.303225] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations845worker2 # [ 0.284411] ... fixed-purpose counters: 0846client # [ 0.280700] ... fixed-purpose bitmap: 0000000000000000847lb # [ 0.293797] smp: Bringing up secondary CPUs ...848worker2 # [ 0.285150] ... fixed-purpose bitmap: 0000000000000000849client # [ 0.280927] ... value mask: 0000ffffffffffff850worker1 # [ 0.304491] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations851lb # [ 0.294210] smp: Brought up 1 node, 1 CPU852worker2 # [ 0.285410] ... value mask: 0000ffffffffffff853worker1 # [ 0.305211] audit: initializing netlink subsys (disabled)854client # [ 0.281791] ... max period: 00007fffffffffff855lb # [ 0.294865] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)856worker2 # [ 0.286255] ... max period: 00007fffffffffff857client # [ 0.282587] ... global_ctrl mask: 000000000000003f858worker1 # [ 0.306204] audit: type=2000 audit(1789998295.183:1): state=initialized audit_enabled=0 res=1859client # [ 0.283037] signal: max sigframe size: 3376860worker2 # [ 0.287062] ... global_ctrl mask: 000000000000003f861worker2 # [ 0.287527] signal: max sigframe size: 3376862worker1 # [ 0.307291] thermal_sys: Registered thermal governor 'fair_share'863lb # [ 0.295365] Memory: 1454552K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)864client # [ 0.283789] rcu: Hierarchical SRCU implementation.865client # [ 0.284467] rcu: Max phase no-delay instances is 400.866worker2 # [ 0.288299] rcu: Hierarchical SRCU implementation.867worker1 # [ 0.307293] thermal_sys: Registered thermal governor 'bang_bang'868lb # [ 0.296477] devtmpfs: initialized869worker2 # [ 0.288998] rcu: Max phase no-delay instances is 400.870worker1 # [ 0.308203] thermal_sys: Registered thermal governor 'step_wise'871lb # [ 0.297297] x86/mm: Memory block size: 128MB872worker1 # [ 0.309125] thermal_sys: Registered thermal governor 'user_space'873worker1 # [ 0.309928] thermal_sys: Registered thermal governor 'power_allocator'874lb # [ 0.299051] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)875worker1 # [ 0.310215] cpuidle: using governor menu876lb # [ 0.300146] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).877client # [ 0.289423] smp: Bringing up secondary CPUs ...878worker1 # [ 0.312431] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5879lb # [ 0.301297] pinctrl core: initialized pinctrl subsystem880client # [ 0.289940] smp: Brought up 1 node, 1 CPU881worker2 # [ 0.294067] smp: Bringing up secondary CPUs ...882worker1 # [ 0.313428] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]883lb # [ 0.302468] PM: RTC time: 13:44:54, date: 2026-09-21884client # [ 0.290627] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)885worker2 # [ 0.294425] smp: Brought up 1 node, 1 CPU886worker1 # [ 0.314205] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry887worker2 # [ 0.295107] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)888worker1 # [ 0.315215] PCI: Using configuration type 1 for base access889client # [ 0.291112] Memory: 1454560K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)890client # [ 0.292180] devtmpfs: initialized891worker1 # [ 0.316303] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.892worker2 # [ 0.295557] Memory: 1454556K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)893lb # [ 0.306039] NET: Registered PF_NETLINK/PF_ROUTE protocol family894client # [ 0.292981] x86/mm: Memory block size: 128MB895worker2 # [ 0.296727] devtmpfs: initialized896worker2 # [ 0.297559] x86/mm: Memory block size: 128MB897lb # [ 0.307376] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations898client # [ 0.294758] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)899lb # [ 0.308222] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations900worker2 # [ 0.299374] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)901client # [ 0.295868] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).902lb # [ 0.309496] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations903client # [ 0.297020] pinctrl core: initialized pinctrl subsystem904worker2 # [ 0.300327] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).905lb # [ 0.310208] audit: initializing netlink subsys (disabled)906client # [ 0.298143] PM: RTC time: 13:44:54, date: 2026-09-21907worker2 # [ 0.301504] pinctrl core: initialized pinctrl subsystem908lb # [ 0.311386] thermal_sys: Registered thermal governor 'fair_share'909worker1 # [ 0.323220] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages910worker2 # [ 0.302612] PM: RTC time: 13:44:54, date: 2026-09-21911lb # [ 0.311388] thermal_sys: Registered thermal governor 'bang_bang'912worker1 # [ 0.324203] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page913lb # [ 0.312198] thermal_sys: Registered thermal governor 'step_wise'914worker1 # [ 0.325188] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages915lb # [ 0.313151] thermal_sys: Registered thermal governor 'user_space'916client # [ 0.301982] NET: Registered PF_NETLINK/PF_ROUTE protocol family917lb # [ 0.313965] audit: type=2000 audit(1789998295.206:1): state=initialized audit_enabled=0 res=1918worker2 # [ 0.306637] NET: Registered PF_NETLINK/PF_ROUTE protocol family919client # [ 0.303408] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations920lb # [ 0.315200] thermal_sys: Registered thermal governor 'power_allocator'921lb # [ 0.315216] cpuidle: using governor menu922client # [ 0.303948] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations923worker2 # [ 0.307947] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations924worker1 # [ 0.330204] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page925worker2 # [ 0.308440] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations926lb # [ 0.317422] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5927client # [ 0.305211] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations928client # [ 0.305937] audit: initializing netlink subsys (disabled)929worker2 # [ 0.309714] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations930lb # [ 0.318440] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]931worker2 # [ 0.310420] audit: initializing netlink subsys (disabled)932client # [ 0.306929] audit: type=2000 audit(1789998295.216:1): state=initialized audit_enabled=0 res=1933lb # [ 0.319201] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry934worker2 # [ 0.311516] thermal_sys: Registered thermal governor 'fair_share'935lb # [ 0.320210] PCI: Using configuration type 1 for base access936client # [ 0.308073] thermal_sys: Registered thermal governor 'fair_share'937worker2 # [ 0.311519] thermal_sys: Registered thermal governor 'bang_bang'938client # [ 0.308075] thermal_sys: Registered thermal governor 'bang_bang'939lb # [ 0.321297] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.940worker2 # [ 0.312412] thermal_sys: Registered thermal governor 'step_wise'941client # [ 0.308928] thermal_sys: Registered thermal governor 'step_wise'942worker2 # [ 0.313370] thermal_sys: Registered thermal governor 'user_space'943client # [ 0.309901] thermal_sys: Registered thermal governor 'user_space'944client # [ 0.310657] thermal_sys: Registered thermal governor 'power_allocator'945worker1 # [ 0.338636] ACPI: Added _OSI(Module Device)946worker2 # [ 0.314139] audit: type=2000 audit(1789998295.215:1): state=initialized audit_enabled=0 res=1947client # [ 0.310949] cpuidle: using governor menu948worker1 # [ 0.339204] ACPI: Added _OSI(Processor Device)949worker2 # [ 0.315413] thermal_sys: Registered thermal governor 'power_allocator'950worker1 # [ 0.339940] ACPI: Added _OSI(Processor Aggregator Device)951worker2 # [ 0.315427] cpuidle: using governor menu952client # [ 0.313145] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5953lb # [ 0.326322] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages954worker2 # [ 0.317647] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5955lb # [ 0.327198] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page956client # [ 0.314169] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]957client # [ 0.314930] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry958worker2 # [ 0.318663] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]959client # [ 0.315938] PCI: Using configuration type 1 for base access960worker2 # [ 0.319413] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry961worker2 # [ 0.320423] PCI: Using configuration type 1 for base access962client # [ 0.317073] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.963worker2 # [ 0.321547] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.964lb # [ 0.332198] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages965lb # [ 0.333197] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page966worker1 # [ 0.347712] ACPI: 1 ACPI AML tables successfully acquired and loaded967worker2 # [ 0.326702] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages968worker1 # [ 0.351615] ACPI: Interpreter enabled969client # [ 0.323945] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages970worker1 # [ 0.352219] ACPI: PM: (supports S0 S3 S4 S5)971worker2 # [ 0.327412] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page972client # [ 0.324928] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page973worker1 # [ 0.352939] ACPI: Using IOAPIC for interrupt routing974worker1 # [ 0.355272] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug975lb # [ 0.343569] ACPI: Added _OSI(Module Device)976worker2 # [ 0.332411] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages977worker1 # [ 0.356203] PCI: Using E820 reservations for host bridge windows978lb # [ 0.344199] ACPI: Added _OSI(Processor Device)979client # [ 0.329928] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages980worker2 # [ 0.333410] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page981lb # [ 0.344907] ACPI: Added _OSI(Processor Aggregator Device)982client # [ 0.330928] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page983worker1 # [ 0.359252] ACPI: Enabled 2 GPEs in block 00 to 3F984lb # [ 0.353472] ACPI: 1 ACPI AML tables successfully acquired and loaded985worker2 # [ 0.343843] ACPI: Added _OSI(Module Device)986worker1 # [ 0.367314] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])987worker2 # [ 0.344412] ACPI: Added _OSI(Processor Device)988client # [ 0.342168] ACPI: Added _OSI(Module Device)989worker1 # [ 0.368208] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]990worker2 # [ 0.345162] ACPI: Added _OSI(Processor Aggregator Device)991client # [ 0.342897] ACPI: Added _OSI(Processor Device)992worker1 # [ 0.369279] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]993client # [ 0.343486] ACPI: Added _OSI(Processor Aggregator Device)994lb # [ 0.357641] ACPI: Interpreter enabled995worker1 # [ 0.370323] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]996lb # [ 0.358213] ACPI: PM: (supports S0 S3 S4 S5)997worker1 # [ 0.371565] PCI host bridge to bus 0000:00998lb # [ 0.358950] ACPI: Using IOAPIC for interrupt routing999worker1 # [ 0.372208] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1000worker1 # [ 0.373202] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1001worker1 # [ 0.374202] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1002lb # [ 0.361249] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1003client # [ 0.349575] ACPI: 1 ACPI AML tables successfully acquired and loaded1004worker1 # [ 0.375202] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1005lb # [ 0.362198] PCI: Using E820 reservations for host bridge windows1006worker1 # [ 0.376202] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1007worker2 # [ 0.354037] ACPI: 1 ACPI AML tables successfully acquired and loaded1008worker1 # [ 0.377202] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1009lb # [ 0.365285] ACPI: Enabled 2 GPEs in block 00 to 3F1010worker1 # [ 0.378202] pci_bus 0000:00: root bus resource [bus 00-ff]1011client # [ 0.353357] ACPI: Interpreter enabled1012client # [ 0.353942] ACPI: PM: (supports S0 S3 S4 S5)1013worker1 # [ 0.379181] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1014worker2 # [ 0.357815] ACPI: Interpreter enabled1015client # [ 0.354700] ACPI: Using IOAPIC for interrupt routing1016worker2 # [ 0.358429] ACPI: PM: (supports S0 S3 S4 S5)1017worker1 # [ 0.380648] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1018worker2 # [ 0.359139] ACPI: Using IOAPIC for interrupt routing1019worker2 # [ 0.359472] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1020client # [ 0.357004] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1021worker2 # [ 0.360411] PCI: Using E820 reservations for host bridge windows1022client # [ 0.357928] PCI: Using E820 reservations for host bridge windows1023lb # [ 0.372803] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1024lb # [ 0.373204] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1025worker2 # [ 0.363520] ACPI: Enabled 2 GPEs in block 00 to 3F1026client # [ 0.361034] ACPI: Enabled 2 GPEs in block 00 to 3F1027lb # [ 0.374284] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1028lb # [ 0.375320] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1029worker1 # [ 0.383267] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1030lb # [ 0.376540] PCI host bridge to bus 0000:001031worker1 # [ 0.384214] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1032lb # [ 0.377202] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1033worker1 # [ 0.385180] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1034lb # [ 0.378197] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1035worker1 # [ 0.386218] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1036lb # [ 0.379198] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1037lb # [ 0.380197] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1038client # [ 0.368380] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1039worker1 # [ 0.387962] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1040worker2 # [ 0.372140] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1041lb # [ 0.381198] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1042client # [ 0.368933] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1043lb # [ 0.382197] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1044worker2 # [ 0.372417] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1045client # [ 0.370012] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1046lb # [ 0.383197] pci_bus 0000:00: root bus resource [bus 00-ff]1047worker1 # [ 0.390222] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1048worker2 # [ 0.373975] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1049client # [ 0.371050] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1050worker1 # [ 0.391065] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1051lb # [ 0.384180] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1052client # [ 0.372276] PCI host bridge to bus 0000:001053worker2 # [ 0.374537] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1054worker1 # [ 0.391906] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1055client # [ 0.372932] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1056worker2 # [ 0.375761] PCI host bridge to bus 0000:001057lb # [ 0.385627] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1058worker1 # [ 0.392208] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1059client # [ 0.373928] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1060worker2 # [ 0.376415] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1061worker2 # [ 0.377411] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1062client # [ 0.374927] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1063worker1 # [ 0.393926] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1064worker2 # [ 0.378411] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1065client # [ 0.375928] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1066worker2 # [ 0.379410] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1067client # [ 0.376927] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1068worker1 # [ 0.396210] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1069worker1 # [ 0.397055] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1070worker2 # [ 0.380410] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1071client # [ 0.377927] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1072worker1 # [ 0.397928] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1073client # [ 0.378927] pci_bus 0000:00: root bus resource [bus 00-ff]1074worker2 # [ 0.381410] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1075worker2 # [ 0.382410] pci_bus 0000:00: root bus resource [bus 00-ff]1076client # [ 0.379887] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1077worker1 # [ 0.398764] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1078lb # [ 0.388387] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1079worker2 # [ 0.383393] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1080client # [ 0.381400] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1081lb # [ 0.389210] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1082worker2 # [ 0.384884] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1083lb # [ 0.390176] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1084worker1 # [ 0.401364] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1085lb # [ 0.391572] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1086worker1 # [ 0.402209] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1087worker1 # [ 0.403202] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1088lb # [ 0.392934] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1089worker1 # [ 0.404758] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1090lb # [ 0.396232] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1091client # [ 0.384003] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1092worker1 # [ 0.406209] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1093lb # [ 0.397124] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1094worker1 # [ 0.407043] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1095client # [ 0.385047] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1096worker2 # [ 0.387478] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1097lb # [ 0.397910] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1098worker1 # [ 0.407927] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1099client # [ 0.385950] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1100worker2 # [ 0.388422] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1101lb # [ 0.398203] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1102worker2 # [ 0.389385] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1103client # [ 0.387289] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1104worker1 # [ 0.408753] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1105lb # [ 0.399771] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1106worker2 # [ 0.390796] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1107client # [ 0.388650] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1108worker1 # [ 0.410210] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1109worker1 # [ 0.411066] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1110worker2 # [ 0.392128] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1111lb # [ 0.402240] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1112worker1 # [ 0.411971] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1113lb # [ 0.403099] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1114client # [ 0.391961] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1115lb # [ 0.403972] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1116client # [ 0.392817] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1117worker1 # [ 0.413643] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1118worker2 # [ 0.395419] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1119client # [ 0.393643] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1120worker2 # [ 0.396284] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1121lb # [ 0.405730] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1122client # [ 0.393933] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1123worker2 # [ 0.397155] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1124worker1 # [ 0.416121] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1125worker2 # [ 0.397416] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1126client # [ 0.395522] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1127worker1 # [ 0.416855] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1128worker1 # [ 0.417225] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1129lb # [ 0.408232] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1130worker2 # [ 0.398966] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1131lb # [ 0.409081] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1132client # [ 0.397877] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1133worker1 # [ 0.418803] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1134lb # [ 0.409956] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1135client # [ 0.398590] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1136worker2 # [ 0.401411] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1137worker1 # [ 0.420210] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1138client # [ 0.398949] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1139lb # [ 0.410751] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1140worker2 # [ 0.402260] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1141worker1 # [ 0.421024] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1142worker2 # [ 0.403155] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1143client # [ 0.400474] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1144worker1 # [ 0.421950] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1145worker1 # [ 0.422208] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1146lb # [ 0.412231] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1147worker2 # [ 0.403967] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1148lb # [ 0.413073] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1149worker1 # [ 0.423738] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1150client # [ 0.402959] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1151lb # [ 0.413965] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1152client # [ 0.403852] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1153worker1 # [ 0.425215] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1154lb # [ 0.414743] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1155worker2 # [ 0.407310] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1156client # [ 0.404726] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1157worker1 # [ 0.426185] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1158worker2 # [ 0.408042] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1159client # [ 0.406289] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1160worker2 # [ 0.408432] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1161worker1 # [ 0.427536] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1162lb # [ 0.416994] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1163lb # [ 0.417861] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1164worker2 # [ 0.409970] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1165worker1 # [ 0.429211] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1166lb # [ 0.418219] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1167client # [ 0.408901] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1168worker1 # [ 0.430035] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1169client # [ 0.409574] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1170worker1 # [ 0.430947] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1171lb # [ 0.419853] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1172worker2 # [ 0.412429] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1173client # [ 0.409949] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1174worker2 # [ 0.413277] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1175worker1 # [ 0.431757] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1176client # [ 0.411497] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1177worker2 # [ 0.414166] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1178lb # [ 0.422229] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1179worker1 # [ 0.433210] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1180lb # [ 0.423122] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1181worker1 # [ 0.434088] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1182worker2 # [ 0.414957] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1183client # [ 0.413818] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1184lb # [ 0.423938] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1185worker1 # [ 0.434925] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1186client # [ 0.414538] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1187client # [ 0.414949] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1188worker2 # [ 0.417173] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1189lb # [ 0.424769] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1190worker1 # [ 0.435756] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1191worker2 # [ 0.417417] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1192worker1 # [ 0.436751] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1193client # [ 0.416581] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1194worker2 # [ 0.418397] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1195lb # [ 0.426229] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1196worker1 # [ 0.437405] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1197lb # [ 0.427112] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1198worker2 # [ 0.419855] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1199worker1 # [ 0.439223] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1200lb # [ 0.427943] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1201client # [ 0.418963] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1202lb # [ 0.428204] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1203client # [ 0.419845] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1204client # [ 0.420684] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1205worker2 # [ 0.423418] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1206lb # [ 0.429754] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1207worker2 # [ 0.424278] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1208client # [ 0.421495] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1209worker2 # [ 0.425158] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1210lb # [ 0.431238] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1211lb # [ 0.432219] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1212worker2 # [ 0.426911] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1213client # [ 0.422960] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1214client # [ 0.423859] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1215lb # [ 0.433764] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1216worker2 # [ 0.428444] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1217client # [ 0.424744] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1218worker2 # [ 0.429417] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1219client # [ 0.425933] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1220lb # [ 0.435584] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1221worker1 # [ 0.440300] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1222worker2 # [ 0.430358] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1223client # [ 0.427470] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1224lb # [ 0.436867] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1225worker2 # [ 0.431416] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1226lb # [ 0.437219] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1227worker1 # [ 0.443718] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1228client # [ 0.428940] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1229worker2 # [ 0.432948] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1230lb # [ 0.438747] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1231client # [ 0.429901] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1232worker1 # [ 0.444455] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1233worker2 # [ 0.434430] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1234client # [ 0.431291] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1235worker1 # [ 0.446084] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1236worker2 # [ 0.435381] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1237lb # [ 0.440895] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1238lb # [ 0.441204] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1239worker1 # [ 0.447156] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1240worker2 # [ 0.436775] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1241lb # [ 0.442189] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1242worker1 # [ 0.448464] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1243client # [ 0.432935] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1244client # [ 0.434577] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1245worker1 # [ 0.449446] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1246lb # [ 0.444242] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1247client # [ 0.434949] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1248worker2 # [ 0.439336] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1249lb # [ 0.446012] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1250worker2 # [ 0.440055] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1251client # [ 0.436499] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1252worker1 # [ 0.451764] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1253worker2 # [ 0.440432] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1254worker1 # [ 0.452209] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1255lb # [ 0.447040] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1256client # [ 0.438618] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1257worker1 # [ 0.453458] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1258worker2 # [ 0.441988] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1259lb # [ 0.448935] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1260client # [ 0.438934] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1261worker1 # [ 0.454969] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1262client # [ 0.439943] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1263worker2 # [ 0.444115] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1264worker2 # [ 0.444417] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1265client # [ 0.441517] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1266worker2 # [ 0.445334] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1267worker1 # [ 0.458108] ACPI: PCI: Interrupt link LNKA configured for IRQ 101268client # [ 0.443720] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1269worker1 # [ 0.459085] ACPI: PCI: Interrupt link LNKB configured for IRQ 101270worker2 # [ 0.446786] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1271client # [ 0.444817] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1272worker1 # [ 0.460303] ACPI: PCI: Interrupt link LNKC configured for IRQ 111273worker1 # [ 0.461298] ACPI: PCI: Interrupt link LNKD configured for IRQ 111274worker2 # [ 0.449248] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1275client # [ 0.446616] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1276worker1 # [ 0.462297] ACPI: PCI: Interrupt link LNKE configured for IRQ 101277worker2 # [ 0.450255] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1278worker1 # [ 0.463298] ACPI: PCI: Interrupt link LNKF configured for IRQ 101279worker2 # [ 0.452130] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1280worker1 # [ 0.464303] ACPI: PCI: Interrupt link LNKG configured for IRQ 111281lb # [ 0.449387] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1282worker1 # [ 0.465301] ACPI: PCI: Interrupt link LNKH configured for IRQ 111283worker1 # [ 0.466237] ACPI: PCI: Interrupt link GSIA configured for IRQ 161284lb # [ 0.451977] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1285worker1 # [ 0.467217] ACPI: PCI: Interrupt link GSIB configured for IRQ 171286worker1 # [ 0.468215] ACPI: PCI: Interrupt link GSIC configured for IRQ 181287lb # [ 0.453288] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1288worker1 # [ 0.469184] ACPI: PCI: Interrupt link GSID configured for IRQ 191289lb # [ 0.454883] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1290worker1 # [ 0.470003] ACPI: PCI: Interrupt link GSIE configured for IRQ 201291worker1 # [ 0.471016] ACPI: PCI: Interrupt link GSIF configured for IRQ 211292lb # [ 0.455420] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1293worker1 # [ 0.471940] ACPI: PCI: Interrupt link GSIG configured for IRQ 221294lb # [ 0.456499] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1295worker1 # [ 0.472216] ACPI: PCI: Interrupt link GSIH configured for IRQ 231296client # [ 0.447124] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1297lb # [ 0.457467] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1298worker1 # [ 0.474166] iommu: Default domain type: Translated1299client # [ 0.449669] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1300worker1 # [ 0.474815] iommu: DMA domain TLB invalidation policy: lazy mode1301worker1 # [ 0.475478] ACPI: bus type USB registered1302worker2 # [ 0.452604] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1303lb # [ 0.459228] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1304client # [ 0.450235] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1305worker1 # [ 0.476216] usbcore: registered new interface driver usbfs1306lb # [ 0.460204] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1307worker2 # [ 0.455185] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1308worker1 # [ 0.477125] usbcore: registered new interface driver hub1309client # [ 0.451642] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1310worker1 # [ 0.477848] usbcore: registered new device driver usb1311lb # [ 0.461489] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1312client # [ 0.452160] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1313worker2 # [ 0.455650] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1314worker1 # [ 0.479052] NetLabel: Initializing1315lb # [ 0.462968] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1316worker1 # [ 0.479630] NetLabel: domain hash size = 1281317client # [ 0.453213] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1318worker2 # [ 0.457132] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1319worker1 # [ 0.480202] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1320client # [ 0.454172] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1321worker2 # [ 0.457685] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1322worker1 # [ 0.481208] NetLabel: unlabeled traffic allowed by default1323worker1 # [ 0.482093] PCI: Using ACPI for IRQ routing1324worker2 # [ 0.458694] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1325lb # [ 0.466234] ACPI: PCI: Interrupt link LNKA configured for IRQ 101326client # [ 0.455965] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1327lb # [ 0.467298] ACPI: PCI: Interrupt link LNKB configured for IRQ 101328worker2 # [ 0.459666] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1329client # [ 0.456793] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1330lb # [ 0.468293] ACPI: PCI: Interrupt link LNKC configured for IRQ 111331client # [ 0.458250] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1332lb # [ 0.469299] ACPI: PCI: Interrupt link LNKD configured for IRQ 111333worker2 # [ 0.461464] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1334lb # [ 0.470312] ACPI: PCI: Interrupt link LNKE configured for IRQ 101335client # [ 0.459732] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1336worker2 # [ 0.462367] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1337lb # [ 0.471295] ACPI: PCI: Interrupt link LNKF configured for IRQ 101338lb # [ 0.472300] ACPI: PCI: Interrupt link LNKG configured for IRQ 111339worker2 # [ 0.463757] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1340lb # [ 0.473296] ACPI: PCI: Interrupt link LNKH configured for IRQ 111341worker2 # [ 0.465190] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1342lb # [ 0.474233] ACPI: PCI: Interrupt link GSIA configured for IRQ 161343client # [ 0.463492] ACPI: PCI: Interrupt link LNKA configured for IRQ 101344lb # [ 0.475189] ACPI: PCI: Interrupt link GSIB configured for IRQ 171345client # [ 0.464038] ACPI: PCI: Interrupt link LNKB configured for IRQ 101346lb # [ 0.476170] ACPI: PCI: Interrupt link GSIC configured for IRQ 181347client # [ 0.465047] ACPI: PCI: Interrupt link LNKC configured for IRQ 111348lb # [ 0.476959] ACPI: PCI: Interrupt link GSID configured for IRQ 191349client # [ 0.466021] ACPI: PCI: Interrupt link LNKD configured for IRQ 111350worker2 # [ 0.468415] ACPI: PCI: Interrupt link LNKA configured for IRQ 101351lb # [ 0.477213] ACPI: PCI: Interrupt link GSIE configured for IRQ 201352client # [ 0.467027] ACPI: PCI: Interrupt link LNKE configured for IRQ 101353worker2 # [ 0.469470] ACPI: PCI: Interrupt link LNKB configured for IRQ 101354lb # [ 0.478210] ACPI: PCI: Interrupt link GSIF configured for IRQ 211355client # [ 0.468024] ACPI: PCI: Interrupt link LNKF configured for IRQ 101356worker2 # [ 0.470504] ACPI: PCI: Interrupt link LNKC configured for IRQ 111357lb # [ 0.479188] ACPI: PCI: Interrupt link GSIG configured for IRQ 221358client # [ 0.469030] ACPI: PCI: Interrupt link LNKG configured for IRQ 111359lb # [ 0.479940] ACPI: PCI: Interrupt link GSIH configured for IRQ 231360worker2 # [ 0.471506] ACPI: PCI: Interrupt link LNKD configured for IRQ 111361client # [ 0.470023] ACPI: PCI: Interrupt link LNKH configured for IRQ 111362worker2 # [ 0.472519] ACPI: PCI: Interrupt link LNKE configured for IRQ 101363client # [ 0.470964] ACPI: PCI: Interrupt link GSIA configured for IRQ 161364lb # [ 0.481318] iommu: Default domain type: Translated1365worker2 # [ 0.473507] ACPI: PCI: Interrupt link LNKF configured for IRQ 101366client # [ 0.471943] ACPI: PCI: Interrupt link GSIB configured for IRQ 171367lb # [ 0.482129] iommu: DMA domain TLB invalidation policy: lazy mode1368worker2 # [ 0.474512] ACPI: PCI: Interrupt link LNKG configured for IRQ 111369client # [ 0.472894] ACPI: PCI: Interrupt link GSIC configured for IRQ 181370lb # [ 0.483220] ACPI: bus type USB registered1371worker2 # [ 0.475508] ACPI: PCI: Interrupt link LNKH configured for IRQ 111372client # [ 0.473664] ACPI: PCI: Interrupt link GSID configured for IRQ 191373lb # [ 0.483956] usbcore: registered new interface driver usbfs1374worker2 # [ 0.476445] ACPI: PCI: Interrupt link GSIA configured for IRQ 161375client # [ 0.473939] ACPI: PCI: Interrupt link GSIE configured for IRQ 201376lb # [ 0.484216] usbcore: registered new interface driver hub1377worker2 # [ 0.477392] ACPI: PCI: Interrupt link GSIB configured for IRQ 171378lb # [ 0.485075] usbcore: registered new device driver usb1379client # [ 0.474884] ACPI: PCI: Interrupt link GSIF configured for IRQ 211380worker2 # [ 0.478138] ACPI: PCI: Interrupt link GSIC configured for IRQ 181381client # [ 0.475791] ACPI: PCI: Interrupt link GSIG configured for IRQ 221382worker2 # [ 0.478425] ACPI: PCI: Interrupt link GSID configured for IRQ 191383lb # [ 0.486773] NetLabel: Initializing1384client # [ 0.476686] ACPI: PCI: Interrupt link GSIH configured for IRQ 231385lb # [ 0.487198] NetLabel: domain hash size = 1281386worker2 # [ 0.479370] ACPI: PCI: Interrupt link GSIE configured for IRQ 201387lb # [ 0.487997] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1388worker2 # [ 0.480135] ACPI: PCI: Interrupt link GSIF configured for IRQ 211389client # [ 0.477977] iommu: Default domain type: Translated1390lb # [ 0.488989] NetLabel: unlabeled traffic allowed by default1391worker2 # [ 0.480429] ACPI: PCI: Interrupt link GSIG configured for IRQ 221392client # [ 0.478779] iommu: DMA domain TLB invalidation policy: lazy mode1393lb # [ 0.489900] PCI: Using ACPI for IRQ routing1394worker2 # [ 0.481427] ACPI: PCI: Interrupt link GSIH configured for IRQ 231395client # [ 0.479905] ACPI: bus type USB registered1396client # [ 0.480480] usbcore: registered new interface driver usbfs1397worker2 # [ 0.483444] iommu: Default domain type: Translated1398client # [ 0.480949] usbcore: registered new interface driver hub1399client # [ 0.481788] usbcore: registered new device driver usb1400worker2 # [ 0.484234] iommu: DMA domain TLB invalidation policy: lazy mode1401worker2 # [ 0.485470] ACPI: bus type USB registered1402client # [ 0.483497] NetLabel: Initializing1403worker2 # [ 0.486186] usbcore: registered new interface driver usbfs1404client # [ 0.483928] NetLabel: domain hash size = 1281405worker2 # [ 0.486429] usbcore: registered new interface driver hub1406client # [ 0.484752] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1407worker2 # [ 0.487296] usbcore: registered new device driver usb1408client # [ 0.485699] NetLabel: unlabeled traffic allowed by default1409client # [ 0.485940] PCI: Using ACPI for IRQ routing1410worker2 # [ 0.488975] NetLabel: Initializing1411worker2 # [ 0.489411] NetLabel: domain hash size = 1281412worker2 # [ 0.490174] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413worker2 # [ 0.490458] NetLabel: unlabeled traffic allowed by default1414worker2 # [ 0.491386] PCI: Using ACPI for IRQ routing1415worker1 # [ 0.525874] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416worker1 # [ 0.526198] pci 0000:00:01.0: vgaarb: bridge control possible1417worker1 # [ 0.526198] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418worker1 # [ 0.526208] vgaarb: loaded1419worker1 # [ 0.526953] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01420worker1 # [ 0.527202] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1421worker1 # [ 0.532290] clocksource: Switched to clocksource kvm-clock1422worker1 # [ 0.535780] VFS: Disk quotas dquot_6.6.01423worker1 # [ 0.536509] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1424worker1 # [ 0.537897] pnp: PnP ACPI init1425worker1 # [ 0.538728] ACPI: IRQ 4 override to edge(!), high(!)1426worker1 # [ 0.539730] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1427worker1 # [ 0.541305] pnp: PnP ACPI: found 6 devices1428worker1 # [ 0.548865] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1429worker1 # [ 0.550366] clocksource: Switched to clocksource acpi_pm1430lb # [ 0.533494] pci 0000:00:01.0: vgaarb: setting as boot VGA device1431worker1 # [ 0.551336] NET: Registered PF_INET protocol family1432lb # [ 0.534193] pci 0000:00:01.0: vgaarb: bridge control possible1433worker1 # [ 0.552491] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1434lb # [ 0.534193] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1435lb # [ 0.534202] vgaarb: loaded1436lb # [ 0.534960] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01437lb # [ 0.535199] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1438client # [ 0.529475] pci 0000:00:01.0: vgaarb: setting as boot VGA device1439client # [ 0.529923] pci 0000:00:01.0: vgaarb: bridge control possible1440client # [ 0.529923] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1441client # [ 0.529932] vgaarb: loaded1442lb # [ 0.541293] clocksource: Switched to clocksource kvm-clock1443client # [ 0.530665] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01444worker2 # [ 0.535208] pci 0000:00:01.0: vgaarb: setting as boot VGA device1445client # [ 0.530927] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1446worker2 # [ 0.535406] pci 0000:00:01.0: vgaarb: bridge control possible1447lb # [ 0.545087] VFS: Disk quotas dquot_6.6.01448worker2 # [ 0.535406] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1449worker2 # [ 0.535416] vgaarb: loaded1450lb # [ 0.545801] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1451worker2 # [ 0.536146] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01452lb # [ 0.547185] pnp: PnP ACPI init1453worker2 # [ 0.536412] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1454lb # [ 0.547985] ACPI: IRQ 4 override to edge(!), high(!)1455client # [ 0.535041] clocksource: Switched to clocksource kvm-clock1456worker1 # [ 0.568848] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1457lb # [ 0.548949] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1458lb # [ 0.550499] pnp: PnP ACPI: found 6 devices1459worker1 # [ 0.570414] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1460client # [ 0.538462] VFS: Disk quotas dquot_6.6.01461worker2 # [ 0.539498] clocksource: Switched to clocksource kvm-clock1462worker1 # [ 0.571723] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1463client # [ 0.539168] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1464worker1 # [ 0.573048] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1465worker2 # [ 0.541200] VFS: Disk quotas dquot_6.6.01466client # [ 0.540506] pnp: PnP ACPI init1467worker1 # [ 0.574307] TCP: Hash tables configured (established 16384 bind 16384)1468client # [ 0.541322] ACPI: IRQ 4 override to edge(!), high(!)1469worker1 # [ 0.575424] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1470client # [ 0.542321] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1471worker2 # [ 0.543763] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1472worker1 # [ 0.576697] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1473client # [ 0.543843] pnp: PnP ACPI: found 6 devices1474worker2 # [ 0.545062] pnp: PnP ACPI init1475worker1 # [ 0.577790] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1476worker2 # [ 0.545861] ACPI: IRQ 4 override to edge(!), high(!)1477worker1 # [ 0.579073] NET: Registered PF_UNIX/PF_LOCAL protocol family1478lb # [ 0.558210] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1479worker2 # [ 0.546832] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1480worker1 # [ 0.579989] NET: Registered PF_XDP protocol family1481lb # [ 0.559723] clocksource: Switched to clocksource acpi_pm1482worker2 # [ 0.548304] pnp: PnP ACPI: found 6 devices1483worker1 # [ 0.580799] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1484lb # [ 0.560688] NET: Registered PF_INET protocol family1485worker1 # [ 0.581774] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1486lb # [ 0.561879] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1487worker1 # [ 0.582801] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1488worker1 # [ 0.583896] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1489worker1 # [ 0.585051] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1490client # [ 0.551417] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1491worker1 # [ 0.586161] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1492client # [ 0.552901] clocksource: Switched to clocksource acpi_pm1493client # [ 0.553880] NET: Registered PF_INET protocol family1494worker1 # [ 0.587894] ACPI: \_SB_.GSIA: Enabled at IRQ 161495client # [ 0.555087] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1496worker2 # [ 0.556265] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1497worker2 # [ 0.557671] clocksource: Switched to clocksource acpi_pm1498worker1 # [ 0.590400] ACPI: \_SB_.GSIB: Enabled at IRQ 171499worker2 # [ 0.558624] NET: Registered PF_INET protocol family1500worker1 # [ 0.592256] ACPI: \_SB_.GSIC: Enabled at IRQ 181501worker2 # [ 0.559788] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1502worker1 # [ 0.594194] ACPI: \_SB_.GSID: Enabled at IRQ 191503worker1 # [ 0.595835] PCI: CLS 0 bytes, default 641504worker1 # [ 0.596753] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1505worker1 # [ 0.598548] Trying to unpack rootfs image as initramfs...1506lb # [ 0.578679] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1507lb # [ 0.580296] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1508lb # [ 0.581668] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1509lb # [ 0.583065] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1510lb # [ 0.584400] TCP: Hash tables configured (established 16384 bind 16384)1511client # [ 0.571749] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1512lb # [ 0.585556] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1513client # [ 0.573290] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1514lb # [ 0.586870] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1515client # [ 0.574638] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1516lb # [ 0.587934] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1517worker2 # [ 0.576550] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1518lb # [ 0.589127] NET: Registered PF_UNIX/PF_LOCAL protocol family1519client # [ 0.575951] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1520lb # [ 0.590128] NET: Registered PF_XDP protocol family1521client # [ 0.577144] TCP: Hash tables configured (established 16384 bind 16384)1522worker2 # [ 0.578049] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1523lb # [ 0.590927] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1524client # [ 0.578216] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1525worker2 # [ 0.579299] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1526lb # [ 0.591932] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1527client # [ 0.579429] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1528worker2 # [ 0.580599] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1529lb # [ 0.592911] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1530client # [ 0.580478] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1531worker2 # [ 0.581791] TCP: Hash tables configured (established 16384 bind 16384)1532lb # [ 0.593968] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1533client # [ 0.581651] NET: Registered PF_UNIX/PF_LOCAL protocol family1534lb # [ 0.595013] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1535worker2 # [ 0.582842] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1536client # [ 0.582597] NET: Registered PF_XDP protocol family1537worker2 # [ 0.584060] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1538lb # [ 0.596106] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1539client # [ 0.583392] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1540worker2 # [ 0.585101] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1541client # [ 0.584375] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1542lb # [ 0.597817] ACPI: \_SB_.GSIA: Enabled at IRQ 161543worker2 # [ 0.586323] NET: Registered PF_UNIX/PF_LOCAL protocol family1544client # [ 0.585351] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1545worker2 # [ 0.587265] NET: Registered PF_XDP protocol family1546client # [ 0.586435] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1547worker2 # [ 0.588098] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1548lb # [ 0.600842] ACPI: \_SB_.GSIB: Enabled at IRQ 171549client # [ 0.587549] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1550worker2 # [ 0.589081] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1551client # [ 0.588614] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1552worker2 # [ 0.590063] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1553lb # [ 0.602782] ACPI: \_SB_.GSIC: Enabled at IRQ 181554client # [ 0.590294] ACPI: \_SB_.GSIA: Enabled at IRQ 161555worker2 # [ 0.591135] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1556worker2 # [ 0.592196] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1557lb # [ 0.604774] ACPI: \_SB_.GSID: Enabled at IRQ 191558client # [ 0.592597] ACPI: \_SB_.GSIB: Enabled at IRQ 171559worker2 # [ 0.593261] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1560lb # [ 0.606564] PCI: CLS 0 bytes, default 641561worker2 # [ 0.594972] ACPI: \_SB_.GSIA: Enabled at IRQ 161562client # [ 0.594569] ACPI: \_SB_.GSIC: Enabled at IRQ 181563lb # [ 0.607479] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1564lb # [ 0.609288] Trying to unpack rootfs image as initramfs...1565client # [ 0.596509] ACPI: \_SB_.GSID: Enabled at IRQ 191566worker2 # [ 0.597628] ACPI: \_SB_.GSIB: Enabled at IRQ 171567client # [ 0.598181] PCI: CLS 0 bytes, default 641568worker2 # [ 0.599640] ACPI: \_SB_.GSIC: Enabled at IRQ 181569client # [ 0.599068] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1570worker2 # [ 0.601604] ACPI: \_SB_.GSID: Enabled at IRQ 191571client # [ 0.600842] Trying to unpack rootfs image as initramfs...1572worker2 # [ 0.603283] PCI: CLS 0 bytes, default 641573worker2 # [ 0.604160] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1574worker2 # [ 0.605848] Trying to unpack rootfs image as initramfs...1575worker1 # [ 0.641210] Initialise system trusted keyrings1576worker1 # [ 0.644117] workingset: timestamp_bits=40 max_order=19 bucket_order=01577worker1 # [ 0.668040] Key type asymmetric registered1578worker1 # [ 0.668774] Asymmetric key parser 'x509' registered1579worker1 # [ 0.669676] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1580worker1 # [ 0.673129] io scheduler mq-deadline registered1581worker1 # [ 0.673904] io scheduler kyber registered1582lb # [ 0.654400] Initialise system trusted keyrings1583lb # [ 0.657233] workingset: timestamp_bits=40 max_order=19 bucket_order=01584worker1 # [ 0.678168] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1585client # [ 0.645016] Initialise system trusted keyrings1586worker1 # [ 0.679419] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1587client # [ 0.647976] workingset: timestamp_bits=40 max_order=19 bucket_order=01588worker2 # [ 0.650902] Initialise system trusted keyrings1589worker1 # [ 0.684754] Linux agpgart interface v0.1031590worker1 # [ 0.685563] ACPI: bus type drm_connector registered1591worker2 # [ 0.654501] workingset: timestamp_bits=40 max_order=19 bucket_order=01592worker1 # [ 0.688399] usbcore: registered new interface driver usbserial_generic1593worker1 # [ 0.689517] usbserial: USB Serial support registered for generic1594worker1 # [ 0.692984] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1595worker1 # [ 0.692984] Please enable it if your BIOS has the CPPC option.1596worker1 # [ 0.695257] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1597lb # [ 0.677107] Key type asymmetric registered1598worker1 # [ 0.699120] drop_monitor: Initializing network drop monitor service1599worker1 # [ 0.700333] NET: Registered PF_INET6 protocol family1600lb # [ 0.681205] Asymmetric key parser 'x509' registered1601client # [ 0.669203] Key type asymmetric registered1602lb # [ 0.682088] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1603worker1 # [ 0.703455] Segment Routing with IPv61604client # [ 0.669934] Asymmetric key parser 'x509' registered1605client # [ 0.670745] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1606lb # [ 0.684294] io scheduler mq-deadline registered1607lb # [ 0.685040] io scheduler kyber registered1608worker1 # [ 0.705994] In-situ OAM (IOAM) with IPv61609worker1 # [ 0.707077] IPI shorthand broadcast: enabled1610worker2 # [ 0.675810] Key type asymmetric registered1611lb # [ 0.687803] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1612worker2 # [ 0.676541] Asymmetric key parser 'x509' registered1613client # [ 0.676096] io scheduler mq-deadline registered1614lb # [ 0.689144] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1615client # [ 0.676897] io scheduler kyber registered1616worker2 # [ 0.677390] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1617client # [ 0.679467] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1618client # [ 0.680792] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1619worker2 # [ 0.682638] io scheduler mq-deadline registered1620lb # [ 0.694847] Linux agpgart interface v0.1031621worker2 # [ 0.683393] io scheduler kyber registered1622worker1 # [ 0.715373] sched_clock: Marking stable (580014493, 134942815)->(787651241, -72693933)1623lb # [ 0.695610] ACPI: bus type drm_connector registered1624worker2 # [ 0.686046] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1625lb # [ 0.698686] usbcore: registered new interface driver usbserial_generic1626client # [ 0.686327] Linux agpgart interface v0.1031627worker1 # [ 0.720223] registered taskstats version 11628worker2 # [ 0.687313] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1629lb # [ 0.699802] usbserial: USB Serial support registered for generic1630client # [ 0.687105] ACPI: bus type drm_connector registered1631worker1 # [ 0.721152] Loading compiled-in X.509 certificates1632client # [ 0.690377] usbcore: registered new interface driver usbserial_generic1633lb # [ 0.703202] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1634lb # [ 0.703202] Please enable it if your BIOS has the CPPC option.1635client # [ 0.691453] usbserial: USB Serial support registered for generic1636worker2 # [ 0.692848] Linux agpgart interface v0.1031637worker2 # [ 0.693601] ACPI: bus type drm_connector registered1638lb # [ 0.705422] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1639client # [ 0.693945] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1640client # [ 0.693945] Please enable it if your BIOS has the CPPC option.1641worker2 # [ 0.696858] usbcore: registered new interface driver usbserial_generic1642lb # [ 0.709314] drop_monitor: Initializing network drop monitor service1643client # [ 0.696236] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1644worker2 # [ 0.697877] usbserial: USB Serial support registered for generic1645lb # [ 0.710497] NET: Registered PF_INET6 protocol family1646client # [ 0.699076] drop_monitor: Initializing network drop monitor service1647worker2 # [ 0.700477] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1648client # [ 0.700273] NET: Registered PF_INET6 protocol family1649worker2 # [ 0.700477] Please enable it if your BIOS has the CPPC option.1650lb # [ 0.714337] Segment Routing with IPv61651lb # [ 0.715009] In-situ OAM (IOAM) with IPv61652worker2 # [ 0.702603] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1653lb # [ 0.716097] IPI shorthand broadcast: enabled1654client # [ 0.705400] Segment Routing with IPv61655worker2 # [ 0.706631] drop_monitor: Initializing network drop monitor service1656worker1 # [ 0.739973] Demotion targets for Node 0: null1657worker2 # [ 0.707770] NET: Registered PF_INET6 protocol family1658worker1 # [ 0.740854] Key type .fscrypt registered1659client # [ 0.707955] In-situ OAM (IOAM) with IPv61660worker1 # [ 0.741579] Key type fscrypt-provisioning registered1661client # [ 0.709019] IPI shorthand broadcast: enabled1662worker2 # [ 0.710894] Segment Routing with IPv61663worker1 # [ 0.745123] ima: No TPM chip found, activating TPM-bypass!1664lb # [ 0.724545] sched_clock: Marking stable (591016688, 133165805)->(801217343, -77034850)1665worker2 # [ 0.713479] In-situ OAM (IOAM) with IPv61666worker1 # [ 0.746028] ima: Allocated hash algorithm: sha11667worker1 # [ 0.746784] ima: No architecture policies found1668worker2 # [ 0.714519] IPI shorthand broadcast: enabled1669lb # [ 0.728284] registered taskstats version 11670worker1 # [ 0.749144] PM: Magic number: 10:659:7361671lb # [ 0.729287] Loading compiled-in X.509 certificates1672worker1 # [ 0.750730] RAS: Correctable Errors collector initialized.1673client # [ 0.717356] sched_clock: Marking stable (584014026, 132911256)->(792960105, -76034823)1674client # [ 0.722109] registered taskstats version 11675worker2 # [ 0.722804] sched_clock: Marking stable (588014549, 134428838)->(801701491, -79258104)1676client # [ 0.723062] Loading compiled-in X.509 certificates1677worker2 # [ 0.727538] registered taskstats version 11678worker2 # [ 0.728510] Loading compiled-in X.509 certificates1679worker1 # [ 0.761639] clk: Disabling unused clocks1680worker1 # [ 0.763973] PM: genpd: Disabling unused power domains1681lb # [ 0.749193] Demotion targets for Node 0: null1682lb # [ 0.750087] Key type .fscrypt registered1683lb # [ 0.753199] Key type fscrypt-provisioning registered1684lb # [ 0.754157] ima: No TPM chip found, activating TPM-bypass!1685client # [ 0.741941] Demotion targets for Node 0: null1686lb # [ 0.755126] ima: Allocated hash algorithm: sha11687client # [ 0.742800] Key type .fscrypt registered1688client # [ 0.743511] Key type fscrypt-provisioning registered1689lb # [ 0.757218] ima: No architecture policies found1690client # [ 0.744493] ima: No TPM chip found, activating TPM-bypass!1691lb # [ 0.759201] PM: Magic number: 10:659:7361692worker2 # [ 0.747460] Demotion targets for Node 0: null1693worker2 # [ 0.748343] Key type .fscrypt registered1694lb # [ 0.760864] RAS: Correctable Errors collector initialized.1695worker2 # [ 0.749041] Key type fscrypt-provisioning registered1696client # [ 0.748944] ima: Allocated hash algorithm: sha11697worker2 # [ 0.749937] ima: No TPM chip found, activating TPM-bypass!1698client # [ 0.749716] ima: No architecture policies found1699client # [ 0.750694] PM: Magic number: 10:659:7361700worker2 # [ 0.752457] ima: Allocated hash algorithm: sha11701worker2 # [ 0.753243] ima: No architecture policies found1702client # [ 0.754346] RAS: Correctable Errors collector initialized.1703worker2 # [ 0.757473] PM: Magic number: 10:659:7361704lb # [ 0.769870] clk: Disabling unused clocks1705worker2 # [ 0.759125] RAS: Correctable Errors collector initialized.1706lb # [ 0.774200] PM: genpd: Disabling unused power domains1707client # [ 0.763052] clk: Disabling unused clocks1708client # [ 0.763793] PM: genpd: Disabling unused power domains1709worker2 # [ 0.767385] clk: Disabling unused clocks1710worker2 # [ 0.772464] PM: genpd: Disabling unused power domains1711worker1 # [ 0.928465] Freeing initrd memory: 29092K1712worker1 # [ 0.931628] Freeing unused decrypted memory: 2028K1713worker1 # [ 0.934294] Freeing unused kernel image (initmem) memory: 3644K1714worker1 # [ 0.935386] Write protecting the kernel read-only data: 32768k1715worker1 # [ 0.937275] Freeing unused kernel image (text/rodata gap) memory: 1208K1716worker1 # [ 0.938784] Freeing unused kernel image (rodata/data gap) memory: 752K1717client # [ 0.930252] Freeing initrd memory: 29096K1718client # [ 0.933481] Freeing unused decrypted memory: 2028K1719lb # [ 0.947748] Freeing initrd memory: 29092K1720client # [ 0.936153] Freeing unused kernel image (initmem) memory: 3644K1721client # [ 0.937265] Write protecting the kernel read-only data: 32768k1722lb # [ 0.951053] Freeing unused decrypted memory: 2028K1723client # [ 0.939157] Freeing unused kernel image (text/rodata gap) memory: 1208K1724client # [ 0.940695] Freeing unused kernel image (rodata/data gap) memory: 752K1725lb # [ 0.953797] Freeing unused kernel image (initmem) memory: 3644K1726lb # [ 0.954913] Write protecting the kernel read-only data: 32768k1727lb # [ 0.956837] Freeing unused kernel image (text/rodata gap) memory: 1208K1728lb # [ 0.958455] Freeing unused kernel image (rodata/data gap) memory: 752K1729worker2 # [ 0.948103] Freeing initrd memory: 29092K1730worker2 # [ 0.951348] Freeing unused decrypted memory: 2028K1731worker2 # [ 0.954101] Freeing unused kernel image (initmem) memory: 3644K1732worker2 # [ 0.955233] Write protecting the kernel read-only data: 32768k1733worker2 # [ 0.957181] Freeing unused kernel image (text/rodata gap) memory: 1208K1734worker1 # [ 0.989977] x86/mm: Checked W+X mappings: passed, no W+X pages found.1735worker1 # [ 0.991046] Run /init as init process1736worker2 # [ 0.958754] Freeing unused kernel image (rodata/data gap) memory: 752K1737worker1 # [ 1.001542] systemd[1]: Inserted module 'autofs4'1738worker1 # [ 1.017740] fuse: init (API version 7.45)1739worker1 # [ 1.023472] ACPI: \_SB_.GSIG: Enabled at IRQ 221740client # [ 0.991521] x86/mm: Checked W+X mappings: passed, no W+X pages found.1741worker1 # [ 1.025847] ACPI: \_SB_.GSIH: Enabled at IRQ 231742client # [ 0.992638] Run /init as init process1743worker1 # [ 1.028659] ACPI: \_SB_.GSIE: Enabled at IRQ 201744lb # [ 1.009700] x86/mm: Checked W+X mappings: passed, no W+X pages found.1745worker1 # [ 1.030870] ACPI: \_SB_.GSIF: Enabled at IRQ 211746lb # [ 1.010817] Run /init as init process1747client # [ 1.002788] systemd[1]: Inserted module 'autofs4'1748lb # [ 1.021353] systemd[1]: Inserted module 'autofs4'1749worker2 # [ 1.009987] x86/mm: Checked W+X mappings: passed, no W+X pages found.1750worker2 # [ 1.011118] Run /init as init process1751client # [ 1.019074] fuse: init (API version 7.45)1752worker2 # [ 1.021699] systemd[1]: Inserted module 'autofs4'1753worker1 # [ 1.056769] systemd[1]: Successfully made /usr/ read-only.1754lb # [ 1.037649] fuse: init (API version 7.45)1755client # [ 1.024854] ACPI: \_SB_.GSIG: Enabled at IRQ 221756client # [ 1.027238] ACPI: \_SB_.GSIH: Enabled at IRQ 231757client # [ 1.030335] ACPI: \_SB_.GSIE: Enabled at IRQ 201758lb # [ 1.043470] ACPI: \_SB_.GSIG: Enabled at IRQ 221759client # [ 1.032840] ACPI: \_SB_.GSIF: Enabled at IRQ 211760lb # [ 1.046294] ACPI: \_SB_.GSIH: Enabled at IRQ 231761lb # [ 1.049661] ACPI: \_SB_.GSIE: Enabled at IRQ 201762worker2 # [ 1.038153] fuse: init (API version 7.45)1763lb # [ 1.052241] ACPI: \_SB_.GSIF: Enabled at IRQ 211764worker2 # [ 1.044017] ACPI: \_SB_.GSIG: Enabled at IRQ 221765worker2 # [ 1.046762] ACPI: \_SB_.GSIH: Enabled at IRQ 231766worker2 # [ 1.050045] ACPI: \_SB_.GSIE: Enabled at IRQ 201767worker2 # [ 1.052708] ACPI: \_SB_.GSIF: Enabled at IRQ 211768client # [ 1.058605] systemd[1]: Successfully made /usr/ read-only.1769lb # [ 1.082309] systemd[1]: Successfully made /usr/ read-only.1770worker2 # [ 1.081633] systemd[1]: Successfully made /usr/ read-only.1771worker1 # [ 1.393231] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1772worker1 # [ 1.405112] systemd[1]: Detected virtualization kvm.1773worker1 # [ 1.407189] systemd[1]: Detected architecture x86-64.1774worker1 # [ 1.409268] systemd[1]: Running in initrd.1775worker1 # [ 1.411669] systemd[1]: Initializing machine ID from random generator.1776worker1 # [ 1.414529] systemd[1]: Hostname set to <worker1>.1777client # [ 1.395261] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1778client # [ 1.407055] systemd[1]: Detected virtualization kvm.1779client # [ 1.409137] systemd[1]: Detected architecture x86-64.1780client # [ 1.411220] systemd[1]: Running in initrd.1781client # [ 1.413661] systemd[1]: Initializing machine ID from random generator.1782lb # [ 1.418983] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1783client # [ 1.416494] systemd[1]: Hostname set to <client>.1784lb # [ 1.430920] systemd[1]: Detected virtualization kvm.1785lb # [ 1.433047] systemd[1]: Detected architecture x86-64.1786lb # [ 1.435223] systemd[1]: Running in initrd.1787lb # [ 1.437908] systemd[1]: Initializing machine ID from random generator.1788worker2 # [ 1.418375] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1789lb # [ 1.440778] systemd[1]: Hostname set to <lb>.1790worker2 # [ 1.430291] systemd[1]: Detected virtualization kvm.1791worker2 # [ 1.432404] systemd[1]: Detected architecture x86-64.1792worker2 # [ 1.434529] systemd[1]: Running in initrd.1793worker2 # [ 1.437118] systemd[1]: Initializing machine ID from random generator.1794worker2 # [ 1.440180] systemd[1]: Hostname set to <worker2>.1795worker1 # [ 1.629158] systemd[1]: bpf-restrict-fs: LSM BPF program attached1796client # [ 1.641972] systemd[1]: bpf-restrict-fs: LSM BPF program attached1797lb # [ 1.661406] systemd[1]: bpf-restrict-fs: LSM BPF program attached1798worker1 # [ 1.683445] systemd[1]: Queued start job for default target Initrd Default Target.1799worker1 # [ 1.687383] systemd[1]: Created slice Slice /system/modprobe.1800worker1 # [ 1.688570] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1801worker1 # [ 1.690008] systemd[1]: Expecting device /dev/disk/by-label/nixos...1802worker1 # [ 1.691152] systemd[1]: Reached target Path Units.1803worker1 # [ 1.692060] systemd[1]: Reached target Slice Units.1804worker1 # [ 1.692991] systemd[1]: Reached target Swaps.1805worker1 # [ 1.693856] systemd[1]: Reached target Timer Units.1806worker1 # [ 1.694887] systemd[1]: Listening on D-Bus System Message Bus Socket.1807worker2 # [ 1.661679] systemd[1]: bpf-restrict-fs: LSM BPF program attached1808worker1 # [ 1.696143] systemd[1]: Listening on Journal Socket (/dev/log).1809worker1 # [ 1.697297] systemd[1]: Listening on Journal Sockets.1810worker1 # [ 1.698285] systemd[1]: Listening on udev Control Socket.1811worker1 # [ 1.699244] systemd[1]: Listening on udev Kernel Socket.1812worker1 # [ 1.700193] systemd[1]: Reached target Socket Units.1813worker1 # [ 1.701989] systemd[1]: Starting Create List of Static Device Nodes...1814worker1 # [ 1.705763] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1815worker1 # [ 1.712175] systemd[1]: Starting Load Kernel Module configfs...1816worker1 # [ 1.727025] systemd[1]: Starting Journal Service...1817client # [ 1.694192] systemd[1]: Queued start job for default target Initrd Default Target.1818client # [ 1.698218] systemd[1]: Created slice Slice /system/modprobe.1819client # [ 1.699388] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1820client # [ 1.700869] systemd[1]: Expecting device /dev/disk/by-label/nixos...1821client # [ 1.702017] systemd[1]: Reached target Path Units.1822client # [ 1.702891] systemd[1]: Reached target Slice Units.1823lb # [ 1.716122] systemd[1]: Queued start job for default target Initrd Default Target.1824client # [ 1.703809] systemd[1]: Reached target Swaps.1825worker1 # [ 1.737652] systemd[1]: Starting Load Kernel Modules...1826client # [ 1.704580] systemd[1]: Reached target Timer Units.1827client # [ 1.705545] systemd[1]: Listening on D-Bus System Message Bus Socket.1828client # [ 1.706792] systemd[1]: Listening on Journal Socket (/dev/log).1829lb # [ 1.720572] systemd[1]: Created slice Slice /system/modprobe.1830client # [ 1.708002] systemd[1]: Listening on Journal Sockets.1831client # [ 1.709025] systemd[1]: Listening on udev Control Socket.1832lb # [ 1.721787] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1833client # [ 1.710077] systemd[1]: Listening on udev Kernel Socket.1834lb # [ 1.723222] systemd[1]: Expecting device /dev/disk/by-label/nixos...1835client # [ 1.711098] systemd[1]: Reached target Socket Units.1836lb # [ 1.724413] systemd[1]: Reached target Path Units.1837lb # [ 1.725320] systemd[1]: Reached target Slice Units.1838lb # [ 1.726243] systemd[1]: Reached target Swaps.1839client # [ 1.712867] systemd[1]: Starting Create List of Static Device Nodes...1840lb # [ 1.727017] systemd[1]: Reached target Timer Units.1841lb # [ 1.728059] systemd[1]: Listening on D-Bus System Message Bus Socket.1842worker2 # [ 1.716348] systemd[1]: Queued start job for default target Initrd Default Target.1843lb # [ 1.729334] systemd[1]: Listening on Journal Socket (/dev/log).1844client # [ 1.716753] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1845lb # [ 1.730509] systemd[1]: Listening on Journal Sockets.1846lb # [ 1.731534] systemd[1]: Listening on udev Control Socket.1847lb # [ 1.732589] systemd[1]: Listening on udev Kernel Socket.1848worker2 # [ 1.720890] systemd[1]: Created slice Slice /system/modprobe.1849lb # [ 1.733559] systemd[1]: Reached target Socket Units.1850worker2 # [ 1.722036] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1851worker1 # [ 1.744005] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1852lb # [ 1.735451] systemd[1]: Starting Create List of Static Device Nodes...1853worker2 # [ 1.723559] systemd[1]: Expecting device /dev/disk/by-label/nixos...1854worker2 # [ 1.724627] systemd[1]: Reached target Path Units.1855worker2 # [ 1.725519] systemd[1]: Reached target Slice Units.1856client # [ 1.725154] systemd[1]: Starting Load Kernel Module configfs...1857worker2 # [ 1.726397] systemd[1]: Reached target Swaps.1858worker2 # [ 1.727191] systemd[1]: Reached target Timer Units.1859lb # [ 1.739252] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1860worker2 # [ 1.728194] systemd[1]: Listening on D-Bus System Message Bus Socket.1861worker1 # [ 1.761913] netfs: FS-Cache loaded1862worker2 # [ 1.729464] systemd[1]: Listening on Journal Socket (/dev/log).1863worker2 # [ 1.730626] systemd[1]: Listening on Journal Sockets.1864worker2 # [ 1.731638] systemd[1]: Listening on udev Control Socket.1865worker2 # [ 1.732677] systemd[1]: Listening on udev Kernel Socket.1866worker2 # [ 1.733649] systemd[1]: Reached target Socket Units.1867worker1 # [ 1.767151] 9pnet: Installing 9P2000 support1868lb # [ 1.747260] systemd[1]: Starting Load Kernel Module configfs...1869worker2 # [ 1.735512] systemd[1]: Starting Create List of Static Device Nodes...1870worker2 # [ 1.739305] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1871client # [ 1.739278] systemd[1]: Starting Journal Service...1872worker2 # [ 1.746523] systemd[1]: Starting Load Kernel Module configfs...1873lb # [ 1.759832] systemd[1]: Starting Journal Service...1874client # [ 1.749109] systemd[1]: Starting Load Kernel Modules...1875worker1 # [ 1.789065] systemd[1]: Starting Coldplug All udev Devices...1876worker2 # [ 1.758106] systemd[1]: Starting Journal Service...1877lb # [ 1.771250] systemd[1]: Starting Load Kernel Modules...1878worker1 # [ 1.794156] systemd-journald[67]: Collecting audit messages is disabled.1879client # [ 1.755004] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1880worker1 # [ 1.802195] systemd[1]: Finished Create List of Static Device Nodes.1881worker2 # [ 1.771513] systemd[1]: Starting Load Kernel Modules...1882client # [ 1.772616] netfs: FS-Cache loaded1883worker1 # [ 1.807590] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1884lb # [ 1.777271] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1885client # [ 1.777299] 9pnet: Installing 9P2000 support1886worker1 # [ 1.816421] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1887lb # [ 1.798305] netfs: FS-Cache loaded1888client # [ 1.787997] systemd[1]: Starting Coldplug All udev Devices...1889worker1 # [ 1.822433] systemd[1]: modprobe@configfs.service: Deactivated successfully.1890worker2 # [ 1.778573] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1891lb # [ 1.803037] 9pnet: Installing 9P2000 support1892worker1 # [ 1.828318] systemd[1]: Finished Load Kernel Module configfs.1893worker2 # [ 1.798833] netfs: FS-Cache loaded1894lb # [ 1.811372] systemd[1]: Starting Coldplug All udev Devices...1895client # [ 1.800406] systemd[1]: Finished Create List of Static Device Nodes.1896worker1 # [ 1.834450] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1897worker2 # [ 1.806611] 9pnet: Installing 9P2000 support1898client # [ 1.805480] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1899worker1 # [ 1.839286] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1900worker1 # [ 1.844920] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1901client # [ 1.813349] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1902worker2 # [ 1.814735] systemd[1]: Starting Coldplug All udev Devices...1903lb # [ 1.827653] systemd[1]: Finished Create List of Static Device Nodes.1904worker1 # [ 1.847994] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1905client # [ 1.818331] systemd[1]: modprobe@configfs.service: Deactivated successfully.1906lb # [ 1.835246] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1907client # [ 1.822437] systemd-journald[67]: Collecting audit messages is disabled.1908client # [ 1.825324] systemd[1]: Finished Load Kernel Module configfs.1909lb # [ 1.840570] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1910worker2 # [ 1.830619] systemd[1]: Finished Create List of Static Device Nodes.1911client # [ 1.830737] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1912lb # [ 1.845656] systemd[1]: modprobe@configfs.service: Deactivated successfully.1913worker2 # [ 1.835724] systemd-journald[67]: Collecting audit messages is disabled.1914lb # [ 1.847748] systemd-journald[67]: Collecting audit messages is disabled.1915client # [ 1.835404] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1916worker2 # [ 1.838140] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1917lb # [ 1.853282] systemd[1]: Finished Load Kernel Module configfs.1918client # [ 1.843025] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1919worker2 # [ 1.845882] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1920worker1 # [ 1.878518] systemd[1]: Finished Load Kernel Modules.1921client # [ 1.844480] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1922lb # [ 1.859093] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1923lb # [ 1.862708] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1924worker2 # [ 1.851878] systemd[1]: modprobe@configfs.service: Deactivated successfully.1925worker1 # [ 1.886220] systemd[1]: Starting Apply Kernel Variables...1926worker2 # [ 1.859819] systemd[1]: Finished Load Kernel Module configfs.1927lb # [ 1.872265] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1928lb # [ 1.873790] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1929worker2 # [ 1.863883] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1930worker1 # [ 1.897100] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1931worker2 # [ 1.868115] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1932worker1 # [ 1.906548] systemd[1]: Starting Create Static Device Nodes in /dev...1933client # [ 1.873468] systemd[1]: Finished Load Kernel Modules.1934worker2 # [ 1.876544] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1935worker2 # [ 1.878043] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1936worker1 # [ 1.776319] systemd-modules-load[68]: Inserted module 'dm_mod'1937worker1 # [ 1.779275] systemd-modules-load[68]: Inserted module 'virtio_balloon'1938client # [ 1.881987] systemd[1]: Starting Apply Kernel Variables...1939worker1 # [ 1.782309] systemd-modules-load[68]: Inserted module 'virtio_gpu'1940worker1 # [ 1.921323] systemd[1]: Started Journal Service.1941client # [ 1.891077] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1942lb # [ 1.904745] systemd[1]: Finished Load Kernel Modules.1943worker1 # [ 1.795167] systemd[1]: Finished Apply Kernel Variables.1944lb # [ 1.910368] systemd[1]: Starting Apply Kernel Variables...1945client # [ 1.901025] systemd[1]: Starting Create Static Device Nodes in /dev...1946worker2 # [ 1.905976] systemd[1]: Finished Load Kernel Modules.1947lb # [ 1.922183] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1948worker1 # [ 1.809106] systemd[1]: Finished Create Static Device Nodes in /dev.1949worker1 # [ 1.810130] systemd[1]: Reached target Preparation for Local File Systems.1950worker1 # [ 1.811093] systemd[1]: Reached target Local File Systems.1951worker2 # [ 1.913672] systemd[1]: Starting Apply Kernel Variables...1952worker1 # [ 1.814079] systemd[1]: Starting Create System Files and Directories...1953client # [ 1.917026] systemd[1]: Finished Apply Kernel Variables.1954worker1 # [ 1.819076] systemd[1]: Starting Rule-based Manager for Device Events and Files...1955lb # [ 1.934381] systemd[1]: Starting Create Static Device Nodes in /dev...1956worker2 # [ 1.926571] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1957lb # [ 1.945831] systemd[1]: Finished Apply Kernel Variables.1958client # [ 1.934467] systemd[1]: Finished Create Static Device Nodes in /dev.1959worker2 # [ 1.935661] systemd[1]: Starting Create Static Device Nodes in /dev...1960client # [ 1.803398] systemd-modules-load[68]: Inserted module 'dm_mod'1961client # [ 1.805256] systemd-modules-load[68]: Inserted module 'virtio_balloon'1962client # [ 1.939164] systemd[1]: Started Journal Service.1963client # [ 1.808314] systemd-modules-load[68]: Inserted module 'virtio_gpu'1964worker1 # [ 1.843187] systemd[1]: Finished Create System Files and Directories.1965client # [ 1.818100] systemd[1]: Reached target Preparation for Local File Systems.1966lb # [ 1.830192] systemd-modules-load[68]: Inserted module 'dm_mod'[ 1.964495] systemd[1]: Started Journal Service.1967client # [ 1.819138] systemd[1]: Reached target Local File Systems.1968lb # 1969lb # [ 1.833380] systemd-modules-load[68]: Inserted module 'virtio_balloon'1970worker2 # [ 1.954626] systemd[1]: Started Journal Service.1971client # [ 1.821244] systemd[1]: Starting Create System Files and Directories...1972lb # [ 1.835286] systemd-modules-load[68]: Inserted module 'virtio_gpu'1973worker2 # [ 1.823121] systemd-modules-load[68]: Inserted module 'dm_mod'1974worker2 # [ 1.824357] systemd-modules-load[68]: Inserted module 'virtio_balloon'1975lb # [ 1.839095] systemd[1]: Finished Create Static Device Nodes in /dev.1976client # [ 1.828331] systemd[1]: Starting Rule-based Manager for Device Events and Files...1977worker2 # [ 1.829341] systemd-modules-load[68]: Inserted module 'virtio_gpu'1978worker2 # [ 1.830332] systemd[1]: Finished Apply Kernel Variables.1979lb # [ 1.848128] systemd[1]: Reached target Preparation for Local File Systems.1980lb # [ 1.849146] systemd[1]: Reached target Local File Systems.1981worker1 # [ 1.868641] systemd-udevd[83]: Using default interface naming scheme 'v261'.1982lb # [ 1.852064] systemd[1]: Starting Create System Files and Directories...1983worker2 # [ 1.841042] systemd[1]: Finished Create Static Device Nodes in /dev.1984worker2 # [ 1.844075] systemd[1]: Reached target Preparation for Local File Systems.1985worker2 # [ 1.844984] systemd[1]: Reached target Local File Systems.1986lb # [ 1.859070] systemd[1]: Starting Rule-based Manager for Device Events and Files...1987worker2 # [ 1.847258] systemd[1]: Starting Create System Files and Directories...1988client # [ 1.853439] systemd[1]: Finished Create System Files and Directories.1989worker2 # [ 1.854142] systemd[1]: Starting Rule-based Manager for Device Events and Files...1990worker1 # [ 1.893089] systemd[1]: Started Rule-based Manager for Device Events and Files.1991lb # [ 1.882309] systemd[1]: Finished Create System Files and Directories.1992client # [ 1.874834] systemd-udevd[83]: Using default interface naming scheme 'v261'.1993worker2 # [ 1.880650] systemd[1]: Finished Create System Files and Directories.1994lb # [ 1.905251] systemd-udevd[83]: Using default interface naming scheme 'v261'.1995client # [ 1.899280] systemd[1]: Started Rule-based Manager for Device Events and Files.1996worker2 # [ 1.904353] systemd-udevd[83]: Using default interface naming scheme 'v261'.1997worker1 # [ 1.943106] systemd[1]: Finished Coldplug All udev Devices.1998worker1 # [ 1.945804] systemd[1]: Reached target System Initialization.1999worker1 # [ 1.946625] systemd[1]: Reached target Basic System.2000lb # [ 1.927806] systemd[1]: Started Rule-based Manager for Device Events and Files.
loading 2,363 hidden lines…
4364lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)4365lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.10 seconds)4366subtest: one node schedules and both builders hold a WorkerSession on it4367lb: must succeed: curl -sf localhost:9901/clusters4368worker2 # [ 27.729303] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_connected addr=lb:500514369worker2 # [ 27.733113] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4370lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4371worker1 # [ 27.813585] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_connected addr=lb:500514372worker1 # [ 27.817725] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4373worker2 # [ 27.934899] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_connected addr=lb:500514374worker2 # [ 27.938656] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4375worker1 # [ 28.015501] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_connected addr=lb:500514376worker1 # [ 28.019629] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4377worker2 # [ 28.139906] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_connected addr=lb:500514378worker2 # [ 28.143830] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4379worker1 # [ 28.220560] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_connected addr=lb:500514380worker1 # [ 28.224759] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4381worker2 # [ 28.345120] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_connected addr=lb:500514382worker2 # [ 28.348955] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4383worker1 # [ 28.427208] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_connected addr=lb:500514384worker1 # [ 28.430563] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4385worker2 # [ 28.550119] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_connected addr=lb:500514386worker2 # [ 28.553965] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4387worker1 # [ 28.628260] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_connected addr=lb:500514388worker1 # [ 28.632437] nix-grpc-daemon[787]: ts=2026-09-21T13:45:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4389worker2 # [ 28.754856] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_connected addr=lb:500514390worker2 # [ 28.758991] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4391lb: must succeed: curl -sf localhost:9901/clusters4392worker1 # [ 28.833219] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_connected addr=lb:500514393worker1 # [ 28.837358] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4394lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4395worker2 # [ 28.960670] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_connected addr=lb:500514396worker2 # [ 28.964498] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4397worker1 # [ 29.038142] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_connected addr=lb:500514398worker1 # [ 29.042350] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4399worker2 # [ 29.165707] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_connected addr=lb:500514400worker2 # [ 29.169625] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4401worker1 # [ 29.242977] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_connected addr=lb:500514402worker1 # [ 29.247454] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4403worker2 # [ 29.370653] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_connected addr=lb:500514404worker2 # [ 29.374532] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4405worker1 # [ 29.449797] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_connected addr=lb:500514406worker1 # [ 29.453266] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4407worker2 # [ 29.575499] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_connected addr=lb:500514408worker2 # [ 29.578464] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4409worker1 # [ 29.650887] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_connected addr=lb:500514410worker1 # [ 29.655197] nix-grpc-daemon[787]: ts=2026-09-21T13:45:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4411worker2 # [ 29.779648] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_connected addr=lb:500514412worker2 # [ 29.783799] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4413worker1 # [ 29.855970] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_connected addr=lb:500514414worker1 # [ 29.859878] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4415lb: must succeed: curl -sf localhost:9901/clusters4416lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4417worker2 # [ 29.986329] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_connected addr=lb:500514418worker2 # [ 29.989495] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4419worker1 # [ 30.060711] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_connected addr=lb:500514420worker1 # [ 30.064546] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4421worker2 # [ 30.187682] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_connected addr=lb:500514422worker2 # [ 30.191527] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4423worker1 # [ 30.265529] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_connected addr=lb:500514424worker1 # [ 30.269437] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4425worker2 # [ 30.392869] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_connected addr=lb:500514426worker2 # [ 30.396810] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4427worker1 # [ 30.471794] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_connected addr=lb:500514428worker1 # [ 30.474835] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4429worker2 # [ 30.598057] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_connected addr=lb:500514430worker2 # [ 30.601930] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4431worker1 # [ 30.672755] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_connected addr=lb:500514432worker1 # [ 30.676741] nix-grpc-daemon[787]: ts=2026-09-21T13:45:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4433worker2 # [ 30.802979] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_connected addr=lb:500514434worker2 # [ 30.806863] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4435worker1 # [ 30.877356] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_connected addr=lb:500514436worker1 # [ 30.881153] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4437lb: must succeed: curl -sf localhost:9901/clusters4438lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4439worker2 # [ 31.008697] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_connected addr=lb:500514440worker2 # [ 31.012535] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4441worker1 # [ 31.082129] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_connected addr=lb:500514442worker1 # [ 31.085945] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4443worker2 # [ 31.213896] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_connected addr=lb:500514444worker2 # [ 31.217750] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4445worker1 # [ 31.286658] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_connected addr=lb:500514446worker1 # [ 31.290499] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4447worker2 # [ 31.418910] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_connected addr=lb:500514448worker2 # [ 31.422835] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4449worker1 # [ 31.492798] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_connected addr=lb:500514450worker1 # [ 31.495838] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4451worker2 # [ 31.624327] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_connected addr=lb:500514452worker2 # [ 31.628473] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4453worker1 # [ 31.693830] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_connected addr=lb:500514454worker1 # [ 31.697673] nix-grpc-daemon[787]: ts=2026-09-21T13:45:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4455worker2 # [ 31.829223] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_connected addr=lb:500514456worker2 # [ 31.832980] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4457worker1 # [ 31.898670] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_connected addr=lb:500514458worker1 # [ 31.902612] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4459lb: must succeed: curl -sf localhost:9901/clusters4460worker2 # [ 32.035606] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_connected addr=lb:500514461worker2 # [ 32.038828] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4462lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4463worker1 # [ 32.103313] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_connected addr=lb:500514464worker1 # [ 32.107100] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4465worker2 # [ 32.239854] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_connected addr=lb:500514466worker2 # [ 32.243750] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4467worker1 # [ 32.307815] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_connected addr=lb:500514468worker1 # [ 32.312339] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4469worker2 # [ 32.444985] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_connected addr=lb:500514470worker2 # [ 32.448915] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4471worker1 # [ 32.514130] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_connected addr=lb:500514472worker1 # [ 32.517243] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4473worker2 # [ 32.650081] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_connected addr=lb:500514474worker2 # [ 32.653936] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4475worker1 # [ 32.715137] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_connected addr=lb:500514476worker1 # [ 32.718952] nix-grpc-daemon[787]: ts=2026-09-21T13:45:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4477worker2 # [ 32.854902] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_connected addr=lb:500514478worker2 # [ 32.859384] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4479worker1 # [ 32.919811] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_connected addr=lb:500514480worker1 # [ 32.923656] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4481lb: must succeed: curl -sf localhost:9901/clusters4482worker2 # [ 33.061148] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_connected addr=lb:500514483worker2 # [ 33.064813] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4484worker1 # [ 33.124614] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_connected addr=lb:500514485worker1 # [ 33.128542] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4486lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4487worker2 # [ 33.266239] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_connected addr=lb:500514488worker2 # [ 33.270157] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4489worker1 # [ 33.329333] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_connected addr=lb:500514490worker1 # [ 33.333134] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4491worker2 # [ 33.471082] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_connected addr=lb:500514492worker2 # [ 33.474810] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4493worker1 # [ 33.535782] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_connected addr=lb:500514494worker1 # [ 33.538847] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4495worker2 # [ 33.675953] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_connected addr=lb:500514496worker2 # [ 33.679945] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4497worker1 # [ 33.736758] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_connected addr=lb:500514498worker1 # [ 33.741172] nix-grpc-daemon[787]: ts=2026-09-21T13:45:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4499worker2 # [ 33.880773] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_connected addr=lb:500514500worker2 # [ 33.884719] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4501worker1 # [ 33.941580] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_connected addr=lb:500514502worker1 # [ 33.945429] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4503worker2 # [ 34.086787] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_connected addr=lb:500514504worker2 # [ 34.090495] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4505worker1 # [ 34.146378] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_connected addr=lb:500514506worker1 # [ 34.150262] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4507lb: must succeed: curl -sf localhost:9901/clusters4508lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4509worker2 # [ 34.288792] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_connected addr=lb:500514510worker2 # [ 34.292743] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4511worker1 # [ 34.351117] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_connected addr=lb:500514512worker1 # [ 34.354887] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4513worker2 # [ 34.494073] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_connected addr=lb:500514514worker2 # [ 34.497904] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4515worker1 # [ 34.556909] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_connected addr=lb:500514516worker1 # [ 34.559995] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4517worker2 # [ 34.699093] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_connected addr=lb:500514518worker2 # [ 34.703703] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4519worker1 # [ 34.757970] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_connected addr=lb:500514520worker1 # [ 34.761822] nix-grpc-daemon[787]: ts=2026-09-21T13:45:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4521worker2 # [ 34.904793] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_connected addr=lb:500514522worker2 # [ 34.908547] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4523worker1 # [ 34.962712] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_connected addr=lb:500514524worker1 # [ 34.966769] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4525worker2 # [ 35.110511] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_connected addr=lb:500514526worker2 # [ 35.114220] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4527worker1 # [ 35.167807] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_connected addr=lb:500514528worker1 # [ 35.171720] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4529lb: must succeed: curl -sf localhost:9901/clusters4530lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4531worker2 # [ 35.315425] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_connected addr=lb:500514532worker2 # [ 35.319230] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4533worker1 # [ 35.372618] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_connected addr=lb:500514534worker1 # [ 35.376515] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4535worker2 # [ 35.520468] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_connected addr=lb:500514536worker2 # [ 35.524381] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4537worker1 # [ 35.577417] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_connected addr=lb:500514538worker1 # [ 35.579129] nix-grpc-daemon[787]: ts=2026-09-21T13:45:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4539worker2 # [ 35.725575] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_connected addr=lb:500514540worker2 # [ 35.729507] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4541worker1 # [ 35.780091] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_connected addr=lb:500514542worker1 # [ 35.783870] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4543worker2 # [ 35.930933] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_connected addr=lb:500514544worker2 # [ 35.934741] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4545worker1 # [ 35.984758] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_connected addr=lb:500514546worker1 # [ 35.988667] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4547worker2 # [ 36.135975] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_connected addr=lb:500514548worker2 # [ 36.139645] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4549worker1 # [ 36.189568] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_connected addr=lb:500514550worker1 # [ 36.193436] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4551lb: must succeed: curl -sf localhost:9901/clusters4552worker2 # [ 36.337887] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_connected addr=lb:500514553lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4554worker2 # [ 36.341833] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4555worker1 # [ 36.394551] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_connected addr=lb:500514556worker1 # [ 36.398581] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4557worker2 # [ 36.542980] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_connected addr=lb:500514558worker2 # [ 36.546883] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4559worker1 # [ 36.600378] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_connected addr=lb:500514560worker1 # [ 36.604188] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4561worker1 # [ 36.708984] nix-grpc-daemon[787]: ts=2026-09-21T13:45:30Z level=info event=scheduler_take_over4562lb # [ 36.690452] niks3-server[1024]: 2026/09/21 13:45:31 INFO lead: acquired remote=192.168.1.3:567084563worker2 # [ 36.747942] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_connected addr=lb:500514564worker2 # [ 36.751807] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4565worker1 # [ 36.804862] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_connected addr=lb:500514566worker1 # [ 36.808800] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4567worker2 # [ 36.952954] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_connected addr=lb:500514568worker1 # [ 37.009439] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=scheduler_connected addr=lb:500514569worker1 # [ 37.015604] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:504164570worker1 # [ 37.021104] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04571worker1 # [ 37.028430] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=oidc_jwks provider=mock keys=14572worker1 # [ 37.032560] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:504164573worker1 # [ 37.036718] nix-grpc-daemon[787]: ts=2026-09-21T13:45:31Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04574lb: must succeed: curl -sf localhost:9901/clusters4575lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4576worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4577worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds)4578lb: must succeed: curl -sf localhost:9901/clusters4579lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4580lb: must succeed: curl -sf localhost:9901/clusters4581lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4582lb: must succeed: curl -sf localhost:9901/clusters4583lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4584lb: must succeed: curl -sf localhost:9901/clusters4585lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4586worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4587worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.06 seconds)4588worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4589worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4590lb: must succeed: curl -sf localhost:9901/clusters4591lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4592lb: must succeed: curl -sf localhost:9901/clusters4593lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4594worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4595worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4596worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4597worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4598worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4599worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4600worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4601worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4602(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 10.47 seconds)4603subtest: balancer auth: client cert, bearer token, nothing, unknown CN4604worker2 # [ 38.579353] nix-grpc-daemon[787]: ts=2026-09-21T13:45:33Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:568524605worker2 # [ 38.592139] systemd[1]: Started Nix Daemon.4606worker2 # [ 38.822271] nix-daemon[856]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4607worker2 # [ 38.848766] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 35 ms (attempt 1/5)4608worker2 # [ 38.886201] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 157 ms (attempt 2/5)4609worker2 # [ 39.045863] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 122 ms (attempt 3/5)4610worker2 # [ 39.173902] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 580 ms (attempt 4/5)4611worker2 # [ 39.759264] nix-grpc-daemon[787]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4612worker2 # [ 39.763271] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4613worker1 # [ 39.841343] nix-grpc-daemon[787]: ts=2026-09-21T13:45:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:504304614worker1 # [ 39.855243] systemd[1]: Started Nix Daemon.4615worker1 # [ 40.114122] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4616worker1 # [ 40.139290] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 58 ms (attempt 1/5)4617worker1 # [ 40.199808] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 55 ms (attempt 2/5)4618worker1 # [ 40.255357] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 395 ms (attempt 3/5)4619worker1 # [ 40.650755] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 203 ms (attempt 4/5)4620worker1 # [ 40.863911] nix-grpc-daemon[787]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4621worker1 # [ 40.868667] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4622worker1 # [ 40.887709] nix-grpc-daemon[787]: ts=2026-09-21T13:45:35Z level=debug event=path_info_miss path=6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test error="error: path '/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test' is required, but there is no substituter that can build it"4623worker2 # [ 40.872288] nix-grpc-daemon[787]: ts=2026-09-21T13:45:35Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:568524624worker2 # [ 40.892183] nix-grpc-daemon[787]: ts=2026-09-21T13:45:35Z level=debug event=path_info_miss path=6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test error="error: path '/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test' is required, but there is no substituter that can build it"4625client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4626client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds)4627client: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4628client: (finished: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4629worker2 # [ 41.281137] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=info event=oidc_jwks provider=mock keys=14630worker2 # [ 41.285845] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:568524631worker2 # [ 41.298937] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:568524632worker2 # [ 41.308451] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=path_info_miss path=6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test error="error: path '/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test' is required, but there is no substituter that can build it"4633worker2 # [ 41.317484] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:568524634worker2 # [ 41.326591] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=path_info_miss path=6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test error="error: path '/nix/store/6ya3np4gk5g3zb085kx6dfrg466qaw4r-nixos-system-client-test' is required, but there is no substituter that can build it"4635client: must succeed: rm -r /var/lib/nix-grpc-store4636client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds)4637worker1 # [ 41.612579] nix-grpc-daemon[787]: ts=2026-09-21T13:45:35Z level=info event=denied method=QueryMissing cn=- role=none4638worker2 # [ 41.800423] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=info event=denied method=QueryMissing cn=stranger role=none4639(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.63 seconds)4640subtest: DAG build is scheduled across workers and lands in the cache4641client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24642client # don't know how to build these paths:4643client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4644worker2 # [ 42.038806] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:568524645worker1 # [ 42.080944] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504304646worker1 # [ 42.116674] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04647worker1 # [ 42.121298] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14648worker1 # [ 42.175267] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504304649worker1 # [ 42.198435] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504304650client # copying 1 paths to 'lb:50051'...4651worker1 # [ 42.211714] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504304652client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4653worker1 # [ 42.240217] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4654worker1 # [ 42.268574] nix-grpc-daemon[787]: ts=2026-09-21T13:45:36Z level=info event=niks3_push_started pid=8864655worker1 # [ 42.603323] nix-grpc-daemon[886]: time=2026-09-21T13:45:36.827Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""4656worker1 # [ 42.841360] nix-daemon[871]: accepted connection from pid 897, user nix-grpc-daemon (trusted)4657worker1 # [ 42.859765] nix-daemon[871]: reaped child process 901, status = succeeded4658worker1 # [ 43.058825] nix-daemon[871]: accepted connection from pid 904, user nix-grpc-daemon (trusted)4659worker1 # [ 43.070623] nix-daemon[871]: reaped child process 907, status = succeeded4660lb # [ 43.073899] niks3-server[1024]: 2026/09/21 13:45:37 INFO Received uploads request method=POST path=/api/pending_closures4661worker1 # [ 43.119614] nix-grpc-daemon[886]: time=2026-09-21T13:45:37.347Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4662worker1 # [ 43.122293] nix-grpc-daemon[886]: time=2026-09-21T13:45:37.350Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4663lb # [ 43.147889] niks3-server[1024]: 2026/09/21 13:45:38 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4664lb # [ 43.151063] niks3-server[1024]: 2026/09/21 13:45:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4665lb # [ 43.158735] niks3-server[1024]: 2026/09/21 13:45:38 INFO Signed narinfos id=1 count=14666worker1 # [ 43.177910] nix-grpc-daemon[886]: time=2026-09-21T13:45:37.405Z level=INFO msg="Uploading 1 narinfos"4667lb # [ 43.173800] niks3-server[1024]: 2026/09/21 13:45:38 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4668lb # [ 43.179282] niks3-server[1024]: 2026/09/21 13:45:38 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4669lb # [ 43.185984] niks3-server[1024]: 2026/09/21 13:45:38 INFO Completed upload id=14670worker1 # [ 43.205145] nix-grpc-daemon[886]: time=2026-09-21T13:45:37.432Z level=INFO msg="Upload complete. (577ms)"4671worker1 # [ 43.211634] nix-grpc-daemon[787]: ts=2026-09-21T13:45:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 paths=1 nar_bytes_in=5044672worker1 # [ 43.218396] nix-grpc-daemon[886]: time=2026-09-21T13:45:37.437Z level=WARN msg="Failed to register uploaded object" key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo error="server returned 404: object is not pending upload\n"4673worker1 # [ 43.224315] nix-grpc-daemon[787]: ts=2026-09-21T13:45:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504304674worker1 # [ 43.232942] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4675worker1 # [ 43.243359] nix-daemon[871]: reaped child process 884, status = succeeded4676worker1 # [ 43.246618] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4677client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4678client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4679client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4680client # farm-a> LOG-farm-a-t14681worker1 # [ 44.459775] nix-daemon[871]: reaped child process 911, status = succeeded4682worker1 # [ 44.642986] nix-daemon[871]: accepted connection from pid 927, user nix-grpc-daemon (trusted)4683worker1 # [ 44.654462] nix-daemon[871]: reaped child process 930, status = succeeded4684lb # [ 44.655944] niks3-server[1024]: 2026/09/21 13:45:39 INFO Received uploads request method=POST path=/api/pending_closures4685worker1 # [ 44.688520] nix-grpc-daemon[886]: time=2026-09-21T13:45:38.916Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4686worker1 # [ 44.690145] nix-grpc-daemon[886]: time=2026-09-21T13:45:38.917Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4687lb # [ 44.714418] niks3-server[1024]: 2026/09/21 13:45:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4688worker1 # [ 44.739393] nix-grpc-daemon[886]: time=2026-09-21T13:45:38.966Z level=INFO msg="Uploading 1 narinfos"4689lb # [ 44.720813] niks3-server[1024]: 2026/09/21 13:45:39 INFO Signed narinfos id=2 count=14690lb # [ 44.727737] niks3-server[1024]: 2026/09/21 13:45:39 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4691lb # [ 44.732265] niks3-server[1024]: 2026/09/21 13:45:39 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4692lb # [ 44.738997] niks3-server[1024]: 2026/09/21 13:45:39 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4693lb # [ 44.744149] niks3-server[1024]: 2026/09/21 13:45:39 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4694lb # [ 44.746873] niks3-server[1024]: 2026/09/21 13:45:39 INFO Completed upload id=24695worker1 # [ 44.765910] nix-grpc-daemon[886]: time=2026-09-21T13:45:38.993Z level=INFO msg="Upload complete. (310ms)"4696lb # [ 44.749370] niks3-server[1024]: 2026/09/21 13:45:39 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4697worker1 # [ 44.772893] nix-grpc-daemon[787]: ts=2026-09-21T13:45:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14698worker1 # [ 44.787685] nix-daemon[871]: reaped child process 909, status = succeeded4699worker1 # [ 44.792174] nix-grpc-daemon[787]: ts=2026-09-21T13:45:39Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04700worker1 # [ 44.793903] nix-grpc-daemon[787]: ts=2026-09-21T13:45:39Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14701worker1 # [ 44.841258] nix-grpc-daemon[787]: ts=2026-09-21T13:45:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504304702worker1 # [ 44.854474] nix-grpc-daemon[787]: ts=2026-09-21T13:45:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504304703worker1 # [ 44.868204] nix-grpc-daemon[886]: time=2026-09-21T13:45:39.096Z level=INFO msg="All 1 paths already cached"4704client # copying 1 paths to 'lb:50051'...4705worker1 # [ 44.876549] nix-grpc-daemon[787]: ts=2026-09-21T13:45:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504304706client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4707worker1 # [ 44.895364] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4708worker1 # [ 45.112529] nix-daemon[871]: accepted connection from pid 936, user nix-grpc-daemon (trusted)4709worker1 # [ 45.125827] nix-daemon[871]: reaped child process 939, status = succeeded4710worker1 # [ 45.318301] nix-daemon[871]: accepted connection from pid 941, user nix-grpc-daemon (trusted)4711worker1 # [ 45.329707] nix-daemon[871]: reaped child process 944, status = succeeded4712lb # [ 45.330290] niks3-server[1024]: 2026/09/21 13:45:40 INFO Received uploads request method=POST path=/api/pending_closures4713worker1 # [ 45.358999] nix-grpc-daemon[886]: time=2026-09-21T13:45:39.586Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4714worker1 # [ 45.360541] nix-grpc-daemon[886]: time=2026-09-21T13:45:39.588Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4715lb # [ 45.363336] niks3-server[1024]: 2026/09/21 13:45:40 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4716lb # [ 45.374200] niks3-server[1024]: 2026/09/21 13:45:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4717worker1 # [ 45.397716] nix-grpc-daemon[886]: time=2026-09-21T13:45:39.625Z level=INFO msg="Uploading 1 narinfos"4718lb # [ 45.379401] niks3-server[1024]: 2026/09/21 13:45:40 INFO Signed narinfos id=3 count=14719lb # [ 45.385932] niks3-server[1024]: 2026/09/21 13:45:40 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4720lb # [ 45.395579] niks3-server[1024]: 2026/09/21 13:45:40 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4721lb # [ 45.403098] niks3-server[1024]: 2026/09/21 13:45:40 INFO Completed upload id=34722worker1 # [ 45.421960] nix-grpc-daemon[886]: time=2026-09-21T13:45:39.649Z level=INFO msg="Upload complete. (511ms)"4723lb # [ 45.405398] niks3-server[1024]: 2026/09/21 13:45:40 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4724worker1 # [ 45.427114] nix-grpc-daemon[787]: ts=2026-09-21T13:45:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 paths=1 nar_bytes_in=6244725worker1 # [ 45.432589] nix-grpc-daemon[787]: ts=2026-09-21T13:45:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504304726worker1 # [ 45.441580] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4727worker1 # [ 45.449316] nix-daemon[871]: reaped child process 934, status = succeeded4728worker1 # [ 45.454762] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4729client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4730client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4731client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4732client # farm-b> LOG-farm-b-t14733worker1 # [ 46.265970] nix-daemon[871]: reaped child process 948, status = succeeded4734worker1 # [ 46.436183] nix-daemon[871]: accepted connection from pid 964, user nix-grpc-daemon (trusted)4735worker1 # [ 46.447855] nix-daemon[871]: reaped child process 967, status = succeeded4736lb # [ 46.448660] niks3-server[1024]: 2026/09/21 13:45:41 INFO Received uploads request method=POST path=/api/pending_closures4737worker1 # [ 46.479280] nix-grpc-daemon[886]: time=2026-09-21T13:45:40.706Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4738worker1 # [ 46.480785] nix-grpc-daemon[886]: time=2026-09-21T13:45:40.708Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4739lb # [ 46.493553] niks3-server[1024]: 2026/09/21 13:45:41 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4740lb # [ 46.501110] niks3-server[1024]: 2026/09/21 13:45:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4741lb # [ 46.506070] niks3-server[1024]: 2026/09/21 13:45:41 INFO Signed narinfos id=4 count=14742worker1 # [ 46.528045] nix-grpc-daemon[886]: time=2026-09-21T13:45:40.755Z level=INFO msg="Uploading 1 narinfos"4743lb # [ 46.519707] niks3-server[1024]: 2026/09/21 13:45:41 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4744lb # [ 46.601438] niks3-server[1024]: 2026/09/21 13:45:41 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4745lb # [ 46.608750] niks3-server[1024]: 2026/09/21 13:45:41 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4746lb # [ 46.615114] niks3-server[1024]: 2026/09/21 13:45:41 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4747lb # [ 46.621725] niks3-server[1024]: 2026/09/21 13:45:41 INFO Completed upload id=44748worker1 # [ 46.640792] nix-grpc-daemon[886]: time=2026-09-21T13:45:40.868Z level=INFO msg="Upload complete. (380ms)"4749worker1 # [ 46.650282] nix-grpc-daemon[787]: ts=2026-09-21T13:45:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14750worker1 # [ 46.661918] nix-grpc-daemon[787]: ts=2026-09-21T13:45:40Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04751worker1 # [ 46.666263] nix-grpc-daemon[787]: ts=2026-09-21T13:45:40Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14752worker1 # [ 46.675236] nix-daemon[871]: reaped child process 946, status = succeeded4753worker1 # [ 46.706342] nix-grpc-daemon[787]: ts=2026-09-21T13:45:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504304754worker1 # [ 46.720236] nix-grpc-daemon[787]: ts=2026-09-21T13:45:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504304755worker1 # [ 46.733786] nix-grpc-daemon[886]: time=2026-09-21T13:45:40.961Z level=INFO msg="All 2 paths already cached"4756client # copying 1 paths to 'lb:50051'...4757worker1 # [ 46.742059] nix-grpc-daemon[787]: ts=2026-09-21T13:45:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504304758client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4759worker1 # [ 46.755350] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4760worker1 # [ 46.951042] nix-daemon[871]: accepted connection from pid 973, user nix-grpc-daemon (trusted)4761worker1 # [ 46.962837] nix-daemon[871]: reaped child process 976, status = succeeded4762worker1 # [ 47.151301] nix-daemon[871]: accepted connection from pid 978, user nix-grpc-daemon (trusted)4763worker1 # [ 47.162890] nix-daemon[871]: reaped child process 981, status = succeeded4764lb # [ 47.163594] niks3-server[1024]: 2026/09/21 13:45:42 INFO Received uploads request method=POST path=/api/pending_closures4765worker1 # [ 47.194208] nix-grpc-daemon[886]: time=2026-09-21T13:45:41.421Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4766worker1 # [ 47.195568] nix-grpc-daemon[886]: time=2026-09-21T13:45:41.421Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4767lb # [ 47.196310] niks3-server[1024]: 2026/09/21 13:45:42 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4768lb # [ 47.206648] niks3-server[1024]: 2026/09/21 13:45:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4769worker1 # [ 47.231922] nix-grpc-daemon[886]: time=2026-09-21T13:45:41.459Z level=INFO msg="Uploading 1 narinfos"4770lb # [ 47.213082] niks3-server[1024]: 2026/09/21 13:45:42 INFO Signed narinfos id=5 count=14771lb # [ 47.217438] niks3-server[1024]: 2026/09/21 13:45:42 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4772lb # [ 47.228656] niks3-server[1024]: 2026/09/21 13:45:42 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4773lb # [ 47.235663] niks3-server[1024]: 2026/09/21 13:45:42 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4774lb # [ 47.239474] niks3-server[1024]: 2026/09/21 13:45:42 INFO Completed upload id=54775worker1 # [ 47.258447] nix-grpc-daemon[886]: time=2026-09-21T13:45:41.486Z level=INFO msg="Upload complete. (490ms)"4776worker1 # [ 47.262261] nix-grpc-daemon[787]: ts=2026-09-21T13:45:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 paths=1 nar_bytes_in=7524777worker1 # [ 47.267875] nix-grpc-daemon[787]: ts=2026-09-21T13:45:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504304778worker1 # [ 47.276546] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4779worker1 # [ 47.284808] nix-daemon[871]: reaped child process 971, status = succeeded4780worker1 # [ 47.290103] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4781client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4782client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4783lb # [ 47.403273] envoy[914]: conn peer=[2001:db8:1::1]:40282 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=407 ms=23664784lb # [ 47.406968] envoy[914]: conn peer=[2001:db8:1::1]:40290 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=714785lb # [ 47.410328] envoy[914]: conn peer=[2001:db8:1::1]:40306 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=144786lb # [ 47.413602] envoy[914]: conn peer=[2001:db8:1::1]:40308 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=134787worker1 # [ 47.762768] nix-grpc-daemon[787]: E0000 00:00:1789998341.989719 826 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4788worker1 # [ 47.793185] nix-grpc-daemon[787]: E0000 00:00:1789998342.020856 826 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4789worker1 # [ 47.801320] nix-grpc-daemon[787]: E0000 00:00:1789998342.027257 826 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4790worker1 # [ 47.824027] nix-grpc-daemon[787]: E0000 00:00:1789998342.051879 826 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4791client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4792client # farm-top> LOG-farm-top-t14793worker1 # [ 48.515688] nix-daemon[871]: reaped child process 985, status = succeeded4794worker1 # [ 48.708802] nix-daemon[871]: accepted connection from pid 1001, user nix-grpc-daemon (trusted)4795worker1 # [ 48.719735] nix-daemon[871]: reaped child process 1004, status = succeeded4796lb # [ 48.723130] niks3-server[1024]: 2026/09/21 13:45:43 INFO Received uploads request method=POST path=/api/pending_closures4797worker1 # [ 48.749878] nix-grpc-daemon[886]: time=2026-09-21T13:45:42.977Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4798worker1 # [ 48.751336] nix-grpc-daemon[886]: time=2026-09-21T13:45:42.979Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4799lb # [ 48.764069] niks3-server[1024]: 2026/09/21 13:45:43 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4800lb # [ 48.770342] niks3-server[1024]: 2026/09/21 13:45:43 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4801lb # [ 48.778793] niks3-server[1024]: 2026/09/21 13:45:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4802lb # [ 48.783477] niks3-server[1024]: 2026/09/21 13:45:43 INFO Signed narinfos id=6 count=14803worker1 # [ 48.804909] nix-grpc-daemon[886]: time=2026-09-21T13:45:43.032Z level=INFO msg="Uploading 1 narinfos"4804lb # [ 48.794645] niks3-server[1024]: 2026/09/21 13:45:43 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4805lb # [ 48.797385] niks3-server[1024]: 2026/09/21 13:45:43 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4806lb # [ 48.806651] niks3-server[1024]: 2026/09/21 13:45:43 INFO Completed upload id=64807worker1 # [ 48.825341] nix-grpc-daemon[886]: time=2026-09-21T13:45:43.053Z level=INFO msg="Upload complete. (327ms)"4808lb # [ 48.812745] niks3-server[1024]: 2026/09/21 13:45:43 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4809worker1 # [ 48.832744] nix-grpc-daemon[787]: ts=2026-09-21T13:45:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14810worker1 # [ 48.848558] nix-daemon[871]: reaped child process 983, status = succeeded4811client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 7.02 seconds)4812client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4813client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.19 seconds)4814client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4815client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.02 seconds)4816client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4817client # copying 3 paths...4818client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4819client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4820client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4821client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.26 seconds)4822(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.57 seconds)4823subtest: repeat is answered Cached by the scheduler without building4824worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4825worker1 # finding garbage collector roots...4826worker1 # deleting specified paths...4827worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4828worker1 # deleting unused links...4829worker1 # note: hard linking is currently saving 0.0 KiB4830worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.28 seconds)4831worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4832worker2 # finding garbage collector roots...4833worker2 # deleting specified paths...4834worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4835worker2 # deleting unused links...4836worker2 # note: hard linking is currently saving 0.0 KiB4837worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.29 seconds)4838client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24839worker1 # [ 50.208322] nix-grpc-daemon[787]: ts=2026-09-21T13:45:44Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504304840client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4841client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4842worker2 # [ 50.198440] nix-grpc-daemon[787]: ts=2026-09-21T13:45:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:568524843worker1 # [ 50.258249] nix-grpc-daemon[787]: ts=2026-09-21T13:45:44Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14844client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.30 seconds)4845client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4846client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.17 seconds)4847lb: must succeed: curl -sf localhost:9901/clusters4848lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4849lb: must succeed: curl -sf localhost:9901/clusters4850lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4851worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4852worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4853(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.32 seconds)4854subtest: inputs already in the cache are not sent to the scheduler4855lb: must succeed: curl -sf localhost:9901/clusters4856lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4857lb: must succeed: curl -sf localhost:9901/clusters4858lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4859worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4860worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4861client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24862worker2 # [ 51.097394] nix-grpc-daemon[787]: ts=2026-09-21T13:45:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:568524863client # don't know how to build these paths:4864client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4865worker1 # [ 51.138756] nix-grpc-daemon[787]: ts=2026-09-21T13:45:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504304866worker1 # [ 51.148293] nix-grpc-daemon[886]: time=2026-09-21T13:45:45.375Z level=INFO msg="All 2 paths already cached"4867worker1 # [ 51.160237] nix-grpc-daemon[787]: ts=2026-09-21T13:45:45Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04868worker1 # [ 51.165689] nix-grpc-daemon[787]: ts=2026-09-21T13:45:45Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14869worker1 # [ 51.205273] nix-grpc-daemon[787]: ts=2026-09-21T13:45:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504304870worker1 # [ 51.218552] nix-grpc-daemon[787]: ts=2026-09-21T13:45:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504304871worker1 # [ 51.230308] nix-grpc-daemon[886]: time=2026-09-21T13:45:45.458Z level=INFO msg="All 2 paths already cached"4872client # copying 1 paths to 'lb:50051'...4873client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4874worker1 # [ 51.238433] nix-grpc-daemon[787]: ts=2026-09-21T13:45:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504304875worker1 # [ 51.249756] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4876worker1 # [ 51.443759] nix-daemon[871]: accepted connection from pid 1033, user nix-grpc-daemon (trusted)4877worker1 # [ 51.456951] nix-daemon[871]: reaped child process 1036, status = succeeded4878worker1 # [ 51.621300] nix-daemon[871]: accepted connection from pid 1038, user nix-grpc-daemon (trusted)4879worker1 # [ 51.632846] nix-daemon[871]: reaped child process 1041, status = succeeded4880lb # [ 51.634416] niks3-server[1024]: 2026/09/21 13:45:46 INFO Received uploads request method=POST path=/api/pending_closures4881worker1 # [ 51.662132] nix-grpc-daemon[886]: time=2026-09-21T13:45:45.889Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4882worker1 # [ 51.663534] nix-grpc-daemon[886]: time=2026-09-21T13:45:45.889Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4883lb # [ 51.660650] niks3-server[1024]: 2026/09/21 13:45:46 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4884lb # [ 51.672940] niks3-server[1024]: 2026/09/21 13:45:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4885worker1 # [ 51.696636] nix-grpc-daemon[886]: time=2026-09-21T13:45:45.924Z level=INFO msg="Uploading 1 narinfos"4886lb # [ 51.677592] niks3-server[1024]: 2026/09/21 13:45:46 INFO Signed narinfos id=7 count=14887lb # [ 51.684925] niks3-server[1024]: 2026/09/21 13:45:46 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4888lb # [ 51.696258] niks3-server[1024]: 2026/09/21 13:45:46 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4889worker1 # [ 51.721918] nix-grpc-daemon[886]: time=2026-09-21T13:45:45.949Z level=INFO msg="Upload complete. (460ms)"4890lb # [ 51.703143] niks3-server[1024]: 2026/09/21 13:45:46 INFO Completed upload id=74891lb # [ 51.705654] niks3-server[1024]: 2026/09/21 13:45:46 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4892worker1 # [ 51.724208] nix-grpc-daemon[787]: ts=2026-09-21T13:45:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 paths=1 nar_bytes_in=7524893worker1 # [ 51.729820] nix-grpc-daemon[787]: ts=2026-09-21T13:45:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504304894worker1 # [ 51.738774] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4895worker1 # [ 51.743730] nix-daemon[871]: reaped child process 1031, status = succeeded4896worker1 # [ 51.749133] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4897client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4898client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4899client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4900client # farm-top> LOG-farm-top-t1b4901worker1 # [ 52.972552] nix-daemon[871]: reaped child process 1045, status = succeeded4902worker1 # [ 53.170361] nix-daemon[871]: accepted connection from pid 1061, user nix-grpc-daemon (trusted)4903worker1 # [ 53.181230] nix-daemon[871]: reaped child process 1064, status = succeeded4904lb # [ 53.182625] niks3-server[1024]: 2026/09/21 13:45:48 INFO Received uploads request method=POST path=/api/pending_closures4905worker1 # [ 53.211712] nix-grpc-daemon[886]: time=2026-09-21T13:45:47.439Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4906worker1 # [ 53.213931] nix-grpc-daemon[886]: time=2026-09-21T13:45:47.441Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4907lb # [ 53.218072] niks3-server[1024]: 2026/09/21 13:45:48 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4908lb # [ 53.233624] niks3-server[1024]: 2026/09/21 13:45:48 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4909lb # [ 53.244941] niks3-server[1024]: 2026/09/21 13:45:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4910lb # [ 53.248584] niks3-server[1024]: 2026/09/21 13:45:48 INFO Signed narinfos id=8 count=14911worker1 # [ 53.270064] nix-grpc-daemon[886]: time=2026-09-21T13:45:47.497Z level=INFO msg="Uploading 1 narinfos"4912lb # [ 53.254366] niks3-server[1024]: 2026/09/21 13:45:48 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4913lb # [ 53.263350] niks3-server[1024]: 2026/09/21 13:45:48 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4914lb # [ 53.266232] niks3-server[1024]: 2026/09/21 13:45:48 INFO Completed upload id=84915worker1 # [ 53.285372] nix-grpc-daemon[886]: time=2026-09-21T13:45:47.513Z level=INFO msg="Upload complete. (327ms)"4916lb # [ 53.268331] niks3-server[1024]: 2026/09/21 13:45:48 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4917worker1 # [ 53.293383] nix-grpc-daemon[787]: ts=2026-09-21T13:45:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14918worker1 # [ 53.304807] nix-daemon[871]: reaped child process 1043, status = succeeded4919client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.41 seconds)4920client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4921client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.19 seconds)4922lb: must succeed: curl -sf localhost:9901/clusters4923lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4924lb: must succeed: curl -sf localhost:9901/clusters4925lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4926worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4927worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4928(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.96 seconds)4929subtest: two clients wanting the same drv build it once4930client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4931client # [ 53.729151] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4932client # Running as unit: dedup1.service; invocation ID: 82393226cc8045d6884281f2db8bf34a4933client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.07 seconds)4934client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4935worker1 # [ 53.973459] nix-grpc-daemon[787]: ts=2026-09-21T13:45:48Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504304936client # [ 53.948156] nix[892]: don't know how to build these paths:4937client # [ 53.949073] nix[892]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4938worker1 # [ 53.987235] nix-grpc-daemon[787]: ts=2026-09-21T13:45:48Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04939worker1 # [ 53.991729] nix-grpc-daemon[787]: ts=2026-09-21T13:45:48Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=14940worker2 # [ 53.964452] nix-grpc-daemon[787]: ts=2026-09-21T13:45:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:568524941worker2 # [ 53.976976] nix-grpc-daemon[787]: ts=2026-09-21T13:45:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:568524942client # [ 53.987123] nix[892]: copying 1 paths to 'lb:50051'...4943client # [ 53.987870] nix[892]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4944worker2 # [ 53.987492] nix-grpc-daemon[787]: ts=2026-09-21T13:45:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:568524945worker2 # [ 53.999384] nix-daemon[856]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4946worker2 # [ 54.021895] nix-grpc-daemon[787]: ts=2026-09-21T13:45:48Z level=info event=niks3_push_started pid=8834947worker2 # [ 54.331130] nix-grpc-daemon[883]: time=2026-09-21T13:45:49.207Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""4948worker2 # [ 54.551701] nix-daemon[856]: accepted connection from pid 896, user nix-grpc-daemon (trusted)4949worker2 # [ 54.565961] nix-daemon[856]: reaped child process 899, status = succeeded4950worker2 # [ 54.746639] nix-daemon[856]: accepted connection from pid 901, user nix-grpc-daemon (trusted)4951worker2 # [ 54.759244] nix-daemon[856]: reaped child process 904, status = succeeded4952lb # [ 54.791657] niks3-server[1024]: 2026/09/21 13:45:49 INFO Received uploads request method=POST path=/api/pending_closures4953worker2 # [ 54.788704] nix-grpc-daemon[883]: time=2026-09-21T13:45:49.669Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4954worker2 # [ 54.791252] nix-grpc-daemon[883]: time=2026-09-21T13:45:49.671Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4955client # [ 54.808106] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4956client # Running as unit: dedup2.service; invocation ID: 1e500b2ab1564c8889078053d2ac3c144957lb # [ 54.833941] niks3-server[1024]: 2026/09/21 13:45:49 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4958client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.08 seconds)4959lb # [ 54.845871] niks3-server[1024]: 2026/09/21 13:45:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4960lb # [ 54.851686] niks3-server[1024]: 2026/09/21 13:45:49 INFO Signed narinfos id=9 count=14961worker2 # [ 54.842791] nix-grpc-daemon[883]: time=2026-09-21T13:45:49.723Z level=INFO msg="Uploading 1 narinfos"4962lb # [ 54.860108] niks3-server[1024]: 2026/09/21 13:45:49 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4963lb # [ 54.873446] niks3-server[1024]: 2026/09/21 13:45:49 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4964lb # [ 54.878781] niks3-server[1024]: 2026/09/21 13:45:49 INFO Completed upload id=94965worker2 # [ 54.866928] nix-grpc-daemon[883]: time=2026-09-21T13:45:49.747Z level=INFO msg="Upload complete. (512ms)"4966lb # [ 54.880519] niks3-server[1024]: 2026/09/21 13:45:49 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4967worker2 # [ 54.869729] nix-grpc-daemon[787]: ts=2026-09-21T13:45:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56852 duration_s=0 paths=1 nar_bytes_in=4324968worker2 # [ 54.875296] nix-grpc-daemon[787]: ts=2026-09-21T13:45:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:568524969worker2 # [ 54.883954] nix-daemon[856]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4970worker2 # [ 54.889616] nix-daemon[856]: reaped child process 881, status = succeeded4971client # [ 54.894489] nix[892]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4972worker2 # [ 54.895560] nix-daemon[856]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4973worker1 # [ 55.103426] nix-grpc-daemon[787]: ts=2026-09-21T13:45:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504304974client # [ 55.078320] nix[908]: don't know how to build these paths:4975client # [ 55.079197] nix[908]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4976worker1 # [ 55.119611] nix-grpc-daemon[787]: ts=2026-09-21T13:45:49Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04977worker2 # [ 55.092847] nix-grpc-daemon[787]: ts=2026-09-21T13:45:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:568524978client # [ 55.098735] nix[908]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4979client: must succeed: sleep 24980lb # [ 57.403606] envoy[914]: conn peer=[2001:db8:1::1]:40322 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9524 tx=4073 ms=68054981lb # [ 57.407420] envoy[914]: conn peer=[2001:db8:1::1]:44044 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=634982lb # [ 57.410797] envoy[914]: conn peer=[2001:db8:1::1]:44052 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4169 tx=1807 ms=21714983client: (finished: must succeed: sleep 2, in 2.06 seconds)4984client: waiting for success: ! systemctl is-active dedup1 dedup24985worker2 # [ 58.310670] nix-daemon[856]: reaped child process 908, status = succeeded4986worker2 # [ 58.481158] nix-daemon[856]: accepted connection from pid 938, user nix-grpc-daemon (trusted)4987worker2 # [ 58.494077] nix-daemon[856]: reaped child process 941, status = succeeded4988lb # [ 58.526847] niks3-server[1024]: 2026/09/21 13:45:53 INFO Received uploads request method=POST path=/api/pending_closures4989worker2 # [ 58.522377] nix-grpc-daemon[883]: time=2026-09-21T13:45:53.402Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4990worker2 # [ 58.524051] nix-grpc-daemon[883]: time=2026-09-21T13:45:53.404Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4991lb # [ 58.571187] niks3-server[1024]: 2026/09/21 13:45:53 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4992lb # [ 58.579422] niks3-server[1024]: 2026/09/21 13:45:53 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4993lb # [ 58.591539] niks3-server[1024]: 2026/09/21 13:45:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4994lb # [ 58.597196] niks3-server[1024]: 2026/09/21 13:45:53 INFO Signed narinfos id=10 count=14995worker2 # [ 58.584793] nix-grpc-daemon[883]: time=2026-09-21T13:45:53.464Z level=INFO msg="Uploading 1 narinfos"4996lb # [ 58.600644] niks3-server[1024]: 2026/09/21 13:45:53 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4997lb # [ 58.610438] niks3-server[1024]: 2026/09/21 13:45:53 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4998lb # [ 58.614089] niks3-server[1024]: 2026/09/21 13:45:53 INFO Completed upload id=104999worker2 # [ 58.600896] nix-grpc-daemon[883]: time=2026-09-21T13:45:53.481Z level=INFO msg="Upload complete. (293ms)"5000lb # [ 58.615772] niks3-server[1024]: 2026/09/21 13:45:53 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo5001worker2 # [ 58.604326] nix-grpc-daemon[787]: ts=2026-09-21T13:45:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56852 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=15002worker2 # [ 58.608149] nix-grpc-daemon[787]: ts=2026-09-21T13:45:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56852 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=15003worker2 # [ 58.613318] nix-daemon[856]: reaped child process 906, status = succeeded5004client # [ 58.655710] systemd[1]: dedup2.service: Deactivated successfully.5005client # [ 58.659239] systemd[1]: dedup2.service: Consumed 142ms CPU time over 3.850s wall clock time, 35.2M memory peak, 5.8K incoming IP traffic, 5.6K outgoing IP traffic.5006client # [ 58.665496] systemd[1]: dedup1.service: Deactivated successfully.5007client # [ 58.667462] systemd[1]: dedup1.service: Consumed 119ms CPU time over 4.936s wall clock time, 41.4M memory peak, 7.1K incoming IP traffic, 8.6K outgoing IP traffic.5008client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.11 seconds)5009client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }5010client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)5011worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5012worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5013worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5014worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)5015worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5016worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5017worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5018worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5019(finished: subtest: two clients wanting the same drv build it once, in 6.01 seconds)5020subtest: build hook: nix-daemon with builders = grpc://lb5021client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log5022client # [ 59.869384] systemd[1]: Started Nix Daemon.5023client # [ 60.008472] nix-daemon[950]: accepted connection from pid 947, user root (trusted)5024worker2 # [ 60.187798] nix-grpc-daemon[787]: ts=2026-09-21T13:45:55Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:568525025worker2 # [ 60.193220] nix-grpc-daemon[787]: ts=2026-09-21T13:45:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:568525026worker1 # [ 60.240178] nix-grpc-daemon[787]: ts=2026-09-21T13:45:54Z level=debug event=want client=6 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05027worker1 # [ 60.244143] nix-grpc-daemon[787]: ts=2026-09-21T13:45:54Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=15028worker1 # [ 60.282965] nix-grpc-daemon[787]: ts=2026-09-21T13:45:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504305029worker1 # [ 60.296095] nix-grpc-daemon[787]: ts=2026-09-21T13:45:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504305030worker1 # [ 60.307624] nix-grpc-daemon[787]: ts=2026-09-21T13:45:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504305031worker1 # [ 60.318564] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5032worker1 # [ 60.513569] nix-daemon[871]: accepted connection from pid 1100, user nix-grpc-daemon (trusted)5033worker1 # [ 60.527133] nix-daemon[871]: reaped child process 1103, status = succeeded5034worker1 # [ 60.702058] nix-daemon[871]: accepted connection from pid 1105, user nix-grpc-daemon (trusted)5035worker1 # [ 60.713845] nix-daemon[871]: reaped child process 1108, status = succeeded5036lb # [ 60.713234] niks3-server[1024]: 2026/09/21 13:45:55 INFO Received uploads request method=POST path=/api/pending_closures5037worker1 # [ 60.739349] nix-grpc-daemon[886]: time=2026-09-21T13:45:54.967Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5038worker1 # [ 60.740850] nix-grpc-daemon[886]: time=2026-09-21T13:45:54.968Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5039lb # [ 60.741750] niks3-server[1024]: 2026/09/21 13:45:55 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5040lb # [ 60.751934] niks3-server[1024]: 2026/09/21 13:45:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign5041worker1 # [ 60.778936] nix-grpc-daemon[886]: time=2026-09-21T13:45:55.006Z level=INFO msg="Uploading 1 narinfos"5042lb # [ 60.761213] niks3-server[1024]: 2026/09/21 13:45:55 INFO Signed narinfos id=11 count=15043lb # [ 60.771294] niks3-server[1024]: 2026/09/21 13:45:55 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5044lb # [ 60.780775] niks3-server[1024]: 2026/09/21 13:45:55 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete5045lb # [ 60.787410] niks3-server[1024]: 2026/09/21 13:45:55 INFO Completed upload id=115046worker1 # [ 60.806561] nix-grpc-daemon[886]: time=2026-09-21T13:45:55.034Z level=INFO msg="Upload complete. (476ms)"5047lb # [ 60.790670] niks3-server[1024]: 2026/09/21 13:45:55 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo5048worker1 # [ 60.810306] nix-grpc-daemon[787]: ts=2026-09-21T13:45:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 paths=1 nar_bytes_in=5125049worker1 # [ 60.815398] nix-grpc-daemon[787]: ts=2026-09-21T13:45:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504305050worker1 # [ 60.825083] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5051worker1 # [ 60.834026] nix-daemon[871]: reaped child process 1098, status = succeeded5052worker1 # [ 60.838181] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5053worker1 # [ 61.863739] nix-daemon[871]: reaped child process 1112, status = succeeded5054worker1 # [ 62.038586] nix-daemon[871]: accepted connection from pid 1128, user nix-grpc-daemon (trusted)5055worker1 # [ 62.050661] nix-daemon[871]: reaped child process 1131, status = succeeded5056lb # [ 62.049350] niks3-server[1024]: 2026/09/21 13:45:56 INFO Received uploads request method=POST path=/api/pending_closures5057worker1 # [ 62.289656] nix-grpc-daemon[886]: time=2026-09-21T13:45:56.516Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5058worker1 # [ 62.301201] nix-grpc-daemon[886]: time=2026-09-21T13:45:56.516Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5059lb # [ 62.316425] niks3-server[1024]: 2026/09/21 13:45:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign5060lb # [ 62.322339] niks3-server[1024]: 2026/09/21 13:45:57 INFO Signed narinfos id=12 count=15061worker1 # [ 62.341198] nix-grpc-daemon[886]: time=2026-09-21T13:45:56.568Z level=INFO msg="Uploading 1 narinfos"5062lb # [ 62.337670] niks3-server[1024]: 2026/09/21 13:45:57 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5063lb # [ 62.344198] niks3-server[1024]: 2026/09/21 13:45:57 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete5064lb # [ 62.350120] niks3-server[1024]: 2026/09/21 13:45:57 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5065lb # [ 62.360573] niks3-server[1024]: 2026/09/21 13:45:57 INFO Completed upload id=125066worker1 # [ 62.379564] nix-grpc-daemon[886]: time=2026-09-21T13:45:56.607Z level=INFO msg="Upload complete. (534ms)"5067lb # [ 62.366509] niks3-server[1024]: 2026/09/21 13:45:57 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5068lb # [ 62.371516] niks3-server[1024]: 2026/09/21 13:45:57 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5069worker1 # [ 62.385967] nix-grpc-daemon[787]: ts=2026-09-21T13:45:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=6 outputs=15070worker1 # [ 62.397238] nix-grpc-daemon[787]: ts=2026-09-21T13:45:56Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:504305071worker1 # [ 62.405929] nix-daemon[871]: reaped child process 1110, status = succeeded5072worker1 # [ 62.415883] nix-grpc-daemon[787]: ts=2026-09-21T13:45:56Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:504305073worker1 # [ 62.421709] nix-grpc-daemon[787]: ts=2026-09-21T13:45:56Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 paths=1 nar_bytes_out=1285074worker1 # [ 62.680140] nix-grpc-daemon[787]: ts=2026-09-21T13:45:56Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:504305075worker1 # [ 62.687138] nix-grpc-daemon[787]: ts=2026-09-21T13:45:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504305076worker1 # [ 62.696820] nix-grpc-daemon[886]: time=2026-09-21T13:45:56.924Z level=INFO msg="All 1 paths already cached"5077worker1 # [ 62.712254] nix-grpc-daemon[787]: ts=2026-09-21T13:45:56Z level=debug event=want client=7 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05078worker1 # [ 62.717592] nix-grpc-daemon[787]: ts=2026-09-21T13:45:56Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=15079worker1 # [ 62.723075] nix-grpc-daemon[787]: ts=2026-09-21T13:45:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504305080worker1 # [ 62.735114] nix-grpc-daemon[787]: ts=2026-09-21T13:45:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504305081worker1 # [ 62.747402] nix-grpc-daemon[886]: time=2026-09-21T13:45:56.975Z level=INFO msg="All 1 paths already cached"5082worker1 # [ 62.754584] nix-grpc-daemon[787]: ts=2026-09-21T13:45:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504305083worker1 # [ 62.762733] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5084worker1 # [ 62.931396] nix-daemon[871]: accepted connection from pid 1137, user nix-grpc-daemon (trusted)5085worker1 # [ 62.944261] nix-daemon[871]: reaped child process 1140, status = succeeded5086worker1 # [ 63.093246] nix-daemon[871]: accepted connection from pid 1142, user nix-grpc-daemon (trusted)5087worker1 # [ 63.104786] nix-daemon[871]: reaped child process 1145, status = succeeded5088lb # [ 63.101292] niks3-server[1024]: 2026/09/21 13:45:57 INFO Received uploads request method=POST path=/api/pending_closures5089worker1 # [ 63.128640] nix-grpc-daemon[886]: time=2026-09-21T13:45:57.356Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5090worker1 # [ 63.130451] nix-grpc-daemon[886]: time=2026-09-21T13:45:57.358Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5091lb # [ 63.130139] niks3-server[1024]: 2026/09/21 13:45:58 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5092lb # [ 63.140918] niks3-server[1024]: 2026/09/21 13:45:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign5093worker1 # [ 63.164465] nix-grpc-daemon[886]: time=2026-09-21T13:45:57.392Z level=INFO msg="Uploading 1 narinfos"5094lb # [ 63.145594] niks3-server[1024]: 2026/09/21 13:45:58 INFO Signed narinfos id=13 count=15095lb # [ 63.152657] niks3-server[1024]: 2026/09/21 13:45:58 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5096lb # [ 63.162922] niks3-server[1024]: 2026/09/21 13:45:58 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete5097worker1 # [ 63.189075] nix-grpc-daemon[886]: time=2026-09-21T13:45:57.416Z level=INFO msg="Upload complete. (413ms)"5098lb # [ 63.169949] niks3-server[1024]: 2026/09/21 13:45:58 INFO Completed upload id=135099worker1 # [ 63.192081] nix-grpc-daemon[787]: ts=2026-09-21T13:45:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 paths=1 nar_bytes_in=6245100lb # [ 63.174109] niks3-server[1024]: 2026/09/21 13:45:58 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5101worker1 # [ 63.197428] nix-grpc-daemon[787]: ts=2026-09-21T13:45:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504305102worker1 # [ 63.201277] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5103worker1 # [ 63.203466] nix-daemon[871]: reaped child process 1135, status = succeeded5104worker1 # [ 63.208761] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5105worker1 # [ 64.421865] nix-daemon[871]: reaped child process 1149, status = succeeded5106worker1 # [ 64.580927] nix-daemon[871]: accepted connection from pid 1165, user nix-grpc-daemon (trusted)5107worker1 # [ 64.593297] nix-daemon[871]: reaped child process 1168, status = succeeded5108lb # [ 64.591924] niks3-server[1024]: 2026/09/21 13:45:59 INFO Received uploads request method=POST path=/api/pending_closures5109worker1 # [ 64.619770] nix-grpc-daemon[886]: time=2026-09-21T13:45:58.847Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5110worker1 # [ 64.621271] nix-grpc-daemon[886]: time=2026-09-21T13:45:58.849Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5111lb # [ 64.637097] niks3-server[1024]: 2026/09/21 13:45:59 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5112lb # [ 64.642321] niks3-server[1024]: 2026/09/21 13:45:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign5113lb # [ 64.645159] niks3-server[1024]: 2026/09/21 13:45:59 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5114lb # [ 64.650408] niks3-server[1024]: 2026/09/21 13:45:59 INFO Signed narinfos id=14 count=15115worker1 # [ 64.671941] nix-grpc-daemon[886]: time=2026-09-21T13:45:58.899Z level=INFO msg="Uploading 1 narinfos"5116lb # [ 64.653135] niks3-server[1024]: 2026/09/21 13:45:59 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5117lb # [ 64.663970] niks3-server[1024]: 2026/09/21 13:45:59 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete5118lb # [ 64.901286] niks3-server[1024]: 2026/09/21 13:45:59 INFO Completed upload id=145119worker1 # [ 64.923733] nix-grpc-daemon[886]: time=2026-09-21T13:45:59.151Z level=INFO msg="Upload complete. (508ms)"5120lb # [ 64.910359] niks3-server[1024]: 2026/09/21 13:45:59 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5121worker1 # [ 64.930608] nix-grpc-daemon[787]: ts=2026-09-21T13:45:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=7 outputs=15122worker1 # [ 64.944140] nix-daemon[871]: reaped child process 1147, status = succeeded5123worker1 # [ 64.975991] nix-grpc-daemon[787]: ts=2026-09-21T13:45:59Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:504305124worker1 # [ 64.993140] nix-grpc-daemon[787]: ts=2026-09-21T13:45:59Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:504305125worker1 # [ 64.996919] nix-grpc-daemon[787]: ts=2026-09-21T13:45:59Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 paths=1 nar_bytes_out=1765126worker2 # [ 65.244402] nix-grpc-daemon[787]: ts=2026-09-21T13:46:00Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:568525127worker1 # [ 65.278983] nix-grpc-daemon[787]: ts=2026-09-21T13:45:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504305128worker1 # [ 65.287866] nix-grpc-daemon[886]: time=2026-09-21T13:45:59.515Z level=INFO msg="All 2 paths already cached"5129worker1 # [ 65.301254] nix-grpc-daemon[787]: ts=2026-09-21T13:45:59Z level=debug event=want client=8 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05130worker1 # [ 65.305099] nix-grpc-daemon[787]: ts=2026-09-21T13:45:59Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=15131worker1 # [ 65.311820] nix-grpc-daemon[787]: ts=2026-09-21T13:45:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504305132worker1 # [ 65.322659] nix-grpc-daemon[787]: ts=2026-09-21T13:45:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504305133worker1 # [ 65.333589] nix-grpc-daemon[886]: time=2026-09-21T13:45:59.561Z level=INFO msg="All 2 paths already cached"5134worker1 # [ 65.339814] nix-grpc-daemon[787]: ts=2026-09-21T13:45:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504305135worker1 # [ 65.348266] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5136worker1 # [ 65.527203] nix-daemon[871]: accepted connection from pid 1174, user nix-grpc-daemon (trusted)5137worker1 # [ 65.539442] nix-daemon[871]: reaped child process 1177, status = succeeded5138worker1 # [ 65.708262] nix-daemon[871]: accepted connection from pid 1179, user nix-grpc-daemon (trusted)5139worker1 # [ 65.719695] nix-daemon[871]: reaped child process 1182, status = succeeded5140lb # [ 65.719769] niks3-server[1024]: 2026/09/21 13:46:00 INFO Received uploads request method=POST path=/api/pending_closures5141worker1 # [ 65.746989] nix-grpc-daemon[886]: time=2026-09-21T13:45:59.974Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5142worker1 # [ 65.748483] nix-grpc-daemon[886]: time=2026-09-21T13:45:59.976Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5143lb # [ 65.748576] niks3-server[1024]: 2026/09/21 13:46:00 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5144lb # [ 65.762130] niks3-server[1024]: 2026/09/21 13:46:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5145lb # [ 65.766938] niks3-server[1024]: 2026/09/21 13:46:00 INFO Signed narinfos id=15 count=15146worker1 # [ 65.788749] nix-grpc-daemon[886]: time=2026-09-21T13:46:00.016Z level=INFO msg="Uploading 1 narinfos"5147lb # [ 65.771131] niks3-server[1024]: 2026/09/21 13:46:00 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5148lb # [ 65.784212] niks3-server[1024]: 2026/09/21 13:46:00 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5149lb # [ 65.788946] niks3-server[1024]: 2026/09/21 13:46:00 INFO Completed upload id=155150worker1 # [ 65.808068] nix-grpc-daemon[886]: time=2026-09-21T13:46:00.035Z level=INFO msg="Upload complete. (446ms)"5151lb # [ 65.791758] niks3-server[1024]: 2026/09/21 13:46:00 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5152worker1 # [ 65.813132] nix-grpc-daemon[787]: ts=2026-09-21T13:46:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 paths=1 nar_bytes_in=7525153worker1 # [ 65.817712] nix-grpc-daemon[787]: ts=2026-09-21T13:46:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504305154worker1 # [ 65.826237] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5155worker1 # [ 65.834135] nix-daemon[871]: reaped child process 1172, status = succeeded5156worker1 # [ 65.837940] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5157worker1 # [ 66.862609] nix-daemon[871]: reaped child process 1186, status = succeeded5158worker1 # [ 67.039656] nix-daemon[871]: accepted connection from pid 1202, user nix-grpc-daemon (trusted)5159worker1 # [ 67.050280] nix-daemon[871]: reaped child process 1205, status = succeeded5160lb # [ 67.050397] niks3-server[1024]: 2026/09/21 13:46:01 INFO Received uploads request method=POST path=/api/pending_closures5161worker1 # [ 67.078968] nix-grpc-daemon[886]: time=2026-09-21T13:46:01.306Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5162worker1 # [ 67.080428] nix-grpc-daemon[886]: time=2026-09-21T13:46:01.308Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5163lb # [ 67.098455] niks3-server[1024]: 2026/09/21 13:46:01 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5164lb # [ 67.107189] niks3-server[1024]: 2026/09/21 13:46:02 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5165lb # [ 67.118609] niks3-server[1024]: 2026/09/21 13:46:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5166worker1 # [ 67.141931] nix-grpc-daemon[886]: time=2026-09-21T13:46:01.369Z level=INFO msg="Uploading 1 narinfos"5167lb # [ 67.124107] niks3-server[1024]: 2026/09/21 13:46:02 INFO Signed narinfos id=16 count=15168lb # [ 67.128490] niks3-server[1024]: 2026/09/21 13:46:02 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5169lb # [ 67.136063] niks3-server[1024]: 2026/09/21 13:46:02 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5170lb # [ 67.141579] niks3-server[1024]: 2026/09/21 13:46:02 INFO Completed upload id=165171lb # [ 67.142874] niks3-server[1024]: 2026/09/21 13:46:02 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5172worker1 # [ 67.160672] nix-grpc-daemon[886]: time=2026-09-21T13:46:01.388Z level=INFO msg="Upload complete. (315ms)"5173worker1 # [ 67.166738] nix-grpc-daemon[787]: ts=2026-09-21T13:46:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=8 outputs=15174worker1 # [ 67.177118] nix-grpc-daemon[787]: ts=2026-09-21T13:46:01Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:504305175worker1 # [ 67.185764] nix-daemon[871]: reaped child process 1184, status = succeeded5176worker2 # [ 67.162136] nix-grpc-daemon[787]: ts=2026-09-21T13:46:02Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:568525177worker2 # [ 67.177955] nix-grpc-daemon[787]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5178worker2 # [ 67.193121] nix-grpc-daemon[787]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...5179worker2 # [ 67.203370] nix-grpc-daemon[787]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...5180worker2 # [ 67.216329] nix-grpc-daemon[787]: ts=2026-09-21T13:46:02Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56852 duration_s=0 paths=1 nar_bytes_out=2325181client # [ 67.328289] nix-daemon[950]: reaped child process 952, status = succeeded5182client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.66 seconds)5183client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5184client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.03 seconds)5185client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5186lb # [ 67.403495] envoy[914]: conn peer=[2001:db8:1::1]:45412 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2998 tx=1451 ms=46805187lb # [ 67.408270] envoy[914]: conn peer=[2001:db8:1::1]:45428 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1440 tx=1082 ms=35645188lb # [ 67.412185] envoy[914]: conn peer=[2001:db8:1::1]:45456 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=575189lb # [ 67.415601] envoy[914]: conn peer=[2001:db8:1::1]:45442 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3438 tx=1751 ms=22575190client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.04 seconds)5191client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5192lb # [ 67.419326] envoy[914]: conn peer=[2001:db8:1::1]:45464 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3893 tx=1930 ms=23655193lb # [ 67.423219] envoy[914]: conn peer=[2001:db8:1::1]:34092 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=555194lb # [ 67.427703] envoy[914]: conn peer=[2001:db8:1::1]:34120 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=411 ms=1105195lb # [ 67.431199] envoy[914]: conn peer=[2001:db8:1::1]:34104 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4340 tx=2185 ms=20255196client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)5197(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.75 seconds)5198subtest: an input only one worker has still reaches the builder5199client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only5200client # this derivation will be built:5201client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5202client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5203client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.61 seconds)5204client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5205client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.19 seconds)5206worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5207worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.23 seconds)5208worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5209worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.03 seconds)5210client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&25211client # [ 68.687723] nix-daemon[950]: accepted connection from pid 1033, user root (trusted)5212client # this derivation will be built:5213client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5214client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5215worker1 # [ 68.957647] nix-grpc-daemon[787]: ts=2026-09-21T13:46:03Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:504305216worker1 # [ 68.964116] nix-grpc-daemon[787]: ts=2026-09-21T13:46:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504305217worker1 # [ 69.146741] nix-daemon[871]: accepted connection from pid 1212, user nix-grpc-daemon (trusted)5218worker1 # [ 69.157618] nix-daemon[871]: reaped child process 1215, status = succeeded5219lb # [ 69.156736] niks3-server[1024]: 2026/09/21 13:46:04 INFO Received uploads request method=POST path=/api/pending_closures5220worker1 # [ 69.185346] nix-grpc-daemon[886]: time=2026-09-21T13:46:03.412Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5221worker1 # [ 69.186781] nix-grpc-daemon[886]: time=2026-09-21T13:46:03.412Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5222lb # [ 69.186102] niks3-server[1024]: 2026/09/21 13:46:04 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5223lb # [ 69.195520] niks3-server[1024]: 2026/09/21 13:46:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5224lb # [ 69.202378] niks3-server[1024]: 2026/09/21 13:46:04 INFO Signed narinfos id=17 count=15225worker1 # [ 69.224375] nix-grpc-daemon[886]: time=2026-09-21T13:46:03.452Z level=INFO msg="Uploading 1 narinfos"5226lb # [ 69.205872] niks3-server[1024]: 2026/09/21 13:46:04 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5227lb # [ 69.221106] niks3-server[1024]: 2026/09/21 13:46:04 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5228lb # [ 69.225905] niks3-server[1024]: 2026/09/21 13:46:04 INFO Completed upload id=175229worker1 # [ 69.247303] nix-grpc-daemon[886]: time=2026-09-21T13:46:03.475Z level=INFO msg="Upload complete. (284ms)"5230lb # [ 69.229288] niks3-server[1024]: 2026/09/21 13:46:04 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5231worker1 # [ 69.265906] nix-grpc-daemon[787]: ts=2026-09-21T13:46:03Z level=debug event=want client=9 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05232worker1 # [ 69.270724] nix-grpc-daemon[787]: ts=2026-09-21T13:46:03Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=15233worker2 # [ 69.280226] nix-grpc-daemon[787]: ts=2026-09-21T13:46:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:568525234worker2 # [ 69.294225] nix-grpc-daemon[787]: ts=2026-09-21T13:46:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:568525235client # copying 2 paths to 'lb:50051'...5236worker2 # [ 69.310330] nix-grpc-daemon[787]: ts=2026-09-21T13:46:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:568525237client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5238client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5239worker2 # [ 69.327863] nix-daemon[856]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5240worker2 # [ 69.538212] nix-daemon[856]: accepted connection from pid 960, user nix-grpc-daemon (trusted)5241worker2 # [ 69.551341] nix-daemon[856]: reaped child process 963, status = succeeded5242worker2 # [ 69.724657] nix-daemon[856]: accepted connection from pid 965, user nix-grpc-daemon (trusted)5243worker2 # [ 69.736972] nix-daemon[856]: reaped child process 968, status = succeeded5244lb # [ 69.767587] niks3-server[1024]: 2026/09/21 13:46:04 INFO Received uploads request method=POST path=/api/pending_closures5245lb # [ 69.777250] niks3-server[1024]: 2026/09/21 13:46:04 INFO Received uploads request method=POST path=/api/pending_closures5246worker2 # [ 69.772850] nix-grpc-daemon[883]: time=2026-09-21T13:46:04.652Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5247worker2 # [ 69.774232] nix-grpc-daemon[883]: time=2026-09-21T13:46:04.652Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5248worker2 # [ 69.775969] nix-grpc-daemon[883]: time=2026-09-21T13:46:04.652Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5249lb # [ 69.814142] niks3-server[1024]: 2026/09/21 13:46:04 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5250lb # [ 69.823885] niks3-server[1024]: 2026/09/21 13:46:04 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5251lb # [ 69.833624] niks3-server[1024]: 2026/09/21 13:46:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5252lb # [ 69.838113] niks3-server[1024]: 2026/09/21 13:46:04 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5253lb # [ 69.841062] niks3-server[1024]: 2026/09/21 13:46:04 INFO Signed narinfos id=18 count=15254lb # [ 69.842391] niks3-server[1024]: 2026/09/21 13:46:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5255lb # [ 69.844818] niks3-server[1024]: 2026/09/21 13:46:04 INFO Signed narinfos id=19 count=25256lb # [ 69.847302] niks3-server[1024]: 2026/09/21 13:46:04 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5257worker2 # [ 69.833708] nix-grpc-daemon[883]: time=2026-09-21T13:46:04.713Z level=INFO msg="Uploading 3 narinfos"5258lb # [ 69.859703] niks3-server[1024]: 2026/09/21 13:46:04 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5259lb # [ 69.863962] niks3-server[1024]: 2026/09/21 13:46:04 INFO Completed upload id=185260lb # [ 69.865726] niks3-server[1024]: 2026/09/21 13:46:04 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5261lb # [ 69.867186] niks3-server[1024]: 2026/09/21 13:46:04 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5262lb # [ 69.869886] niks3-server[1024]: 2026/09/21 13:46:04 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5263lb # [ 69.872510] niks3-server[1024]: 2026/09/21 13:46:04 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5264lb # [ 69.874870] niks3-server[1024]: 2026/09/21 13:46:04 INFO Completed upload id=195265worker2 # [ 69.863105] nix-grpc-daemon[883]: time=2026-09-21T13:46:04.743Z level=INFO msg="Upload complete. (518ms)"5266worker2 # [ 69.868181] nix-grpc-daemon[787]: ts=2026-09-21T13:46:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56852 duration_s=0 paths=2 nar_bytes_in=9765267worker2 # [ 69.873747] nix-grpc-daemon[787]: ts=2026-09-21T13:46:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:568525268worker2 # [ 69.882203] nix-daemon[856]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5269worker2 # [ 69.892152] nix-daemon[856]: reaped child process 958, status = succeeded5270worker2 # [ 69.897151] nix-grpc-daemon[787]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5271worker2 # [ 69.904225] nix-daemon[856]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5272client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5273client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5274client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5275worker2 # [ 71.517784] nix-daemon[856]: reaped child process 973, status = succeeded5276worker2 # [ 71.679049] nix-daemon[856]: accepted connection from pid 989, user nix-grpc-daemon (trusted)5277worker2 # [ 71.691610] nix-daemon[856]: reaped child process 992, status = succeeded5278lb # [ 71.723897] niks3-server[1024]: 2026/09/21 13:46:06 INFO Received uploads request method=POST path=/api/pending_closures5279worker2 # [ 71.719085] nix-grpc-daemon[883]: time=2026-09-21T13:46:06.599Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5280lb # [ 71.755117] niks3-server[1024]: 2026/09/21 13:46:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5281lb # [ 71.764453] niks3-server[1024]: 2026/09/21 13:46:06 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5282worker2 # [ 71.757525] nix-grpc-daemon[883]: time=2026-09-21T13:46:06.637Z level=INFO msg="Uploading 1 narinfos"5283lb # [ 71.770647] niks3-server[1024]: 2026/09/21 13:46:06 INFO Signed narinfos id=20 count=15284lb # [ 71.775066] niks3-server[1024]: 2026/09/21 13:46:06 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5285lb # [ 71.788298] niks3-server[1024]: 2026/09/21 13:46:06 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5286lb # [ 71.795192] niks3-server[1024]: 2026/09/21 13:46:06 INFO Completed upload id=205287worker2 # [ 71.782122] nix-grpc-daemon[883]: time=2026-09-21T13:46:06.662Z level=INFO msg="Upload complete. (267ms)"5288lb # [ 71.797704] niks3-server[1024]: 2026/09/21 13:46:06 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5289worker2 # [ 71.786630] nix-grpc-daemon[787]: ts=2026-09-21T13:46:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56852 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=9 outputs=15290client # copying 1 paths...5291worker2 # [ 71.797725] nix-grpc-daemon[787]: ts=2026-09-21T13:46:06Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:568525292worker2 # [ 71.802750] nix-daemon[856]: reaped child process 970, status = succeeded5293client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5294worker1 # [ 71.852843] nix-grpc-daemon[787]: ts=2026-09-21T13:46:06Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:504305295worker1 # [ 71.867206] nix-grpc-daemon[787]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5296worker1 # [ 71.885605] nix-grpc-daemon[787]: ts=2026-09-21T13:46:06Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 paths=1 nar_bytes_out=1205297client # [ 71.961951] nix-daemon[950]: reaped child process 1036, status = succeeded5298client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.49 seconds)5299client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25300client # [ 72.140484] nix-daemon[950]: accepted connection from pid 1055, user root (trusted)5301client # this derivation will be built:5302client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5303worker2 # [ 72.325294] nix-grpc-daemon[787]: ts=2026-09-21T13:46:07Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:568525304client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5305worker1 # [ 72.365119] nix-grpc-daemon[787]: ts=2026-09-21T13:46:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504305306worker1 # [ 72.375736] nix-grpc-daemon[886]: time=2026-09-21T13:46:06.603Z level=INFO msg="All 1 paths already cached"5307worker1 # [ 72.390263] nix-grpc-daemon[787]: ts=2026-09-21T13:46:06Z level=debug event=want client=10 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05308worker1 # [ 72.395704] nix-grpc-daemon[787]: ts=2026-09-21T13:46:06Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15309worker1 # [ 72.401400] nix-grpc-daemon[787]: ts=2026-09-21T13:46:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504305310worker1 # [ 72.413498] nix-grpc-daemon[787]: ts=2026-09-21T13:46:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504305311client # copying 1 paths to 'lb:50051'...5312client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5313worker1 # [ 72.432125] nix-grpc-daemon[787]: ts=2026-09-21T13:46:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504305314worker1 # [ 72.440879] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5315worker1 # [ 72.458748] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 20 ms (attempt 1/5)5316worker1 # [ 72.483398] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 19 ms (attempt 2/5)5317worker1 # [ 72.508077] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 220 ms (attempt 3/5)5318worker1 # [ 72.733762] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 265 ms (attempt 4/5)5319worker1 # [ 73.005800] nix-grpc-daemon[787]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5320worker1 # [ 73.009928] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5321worker1 # [ 73.015436] nix-grpc-daemon[787]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5322worker1 # [ 73.225791] nix-daemon[871]: accepted connection from pid 1230, user nix-grpc-daemon (trusted)5323worker1 # [ 73.238497] nix-daemon[871]: reaped child process 1233, status = succeeded5324worker1 # [ 73.407830] nix-daemon[871]: accepted connection from pid 1235, user nix-grpc-daemon (trusted)5325worker1 # [ 73.420107] nix-daemon[871]: reaped child process 1238, status = succeeded5326lb # [ 73.420499] niks3-server[1024]: 2026/09/21 13:46:08 INFO Received uploads request method=POST path=/api/pending_closures5327worker1 # [ 73.447001] nix-grpc-daemon[886]: time=2026-09-21T13:46:07.674Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5328worker1 # [ 73.448868] nix-grpc-daemon[886]: time=2026-09-21T13:46:07.676Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5329lb # [ 73.447297] niks3-server[1024]: 2026/09/21 13:46:08 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5330lb # [ 73.466424] niks3-server[1024]: 2026/09/21 13:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5331worker1 # [ 73.493818] nix-grpc-daemon[886]: time=2026-09-21T13:46:07.721Z level=INFO msg="Uploading 1 narinfos"5332lb # [ 73.474923] niks3-server[1024]: 2026/09/21 13:46:08 INFO Signed narinfos id=21 count=15333lb # [ 73.480775] niks3-server[1024]: 2026/09/21 13:46:08 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5334lb # [ 73.490944] niks3-server[1024]: 2026/09/21 13:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5335lb # [ 73.493019] niks3-server[1024]: 2026/09/21 13:46:08 INFO Completed upload id=215336worker1 # [ 73.513236] nix-grpc-daemon[886]: time=2026-09-21T13:46:07.741Z level=INFO msg="Upload complete. (478ms)"5337lb # [ 73.496123] niks3-server[1024]: 2026/09/21 13:46:08 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5338worker1 # [ 73.517734] nix-grpc-daemon[787]: ts=2026-09-21T13:46:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=1 paths=1 nar_bytes_in=5605339worker1 # [ 73.523762] nix-grpc-daemon[787]: ts=2026-09-21T13:46:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504305340worker1 # [ 73.533297] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5341worker1 # [ 73.547201] nix-daemon[871]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5342client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5343worker1 # [ 73.549387] nix-daemon[871]: reaped child process 1222, status = succeeded5344client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5345client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5346worker1 # [ 74.168522] nix-daemon[871]: reaped child process 1242, status = succeeded5347worker1 # [ 74.346072] nix-daemon[871]: accepted connection from pid 1258, user nix-grpc-daemon (trusted)5348worker1 # [ 74.357747] nix-daemon[871]: reaped child process 1261, status = succeeded5349lb # [ 74.357495] niks3-server[1024]: 2026/09/21 13:46:09 INFO Received uploads request method=POST path=/api/pending_closures5350worker1 # [ 74.384849] nix-grpc-daemon[886]: time=2026-09-21T13:46:08.612Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5351lb # [ 74.389181] niks3-server[1024]: 2026/09/21 13:46:09 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5352lb # [ 74.398278] niks3-server[1024]: 2026/09/21 13:46:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5353worker1 # [ 74.423344] nix-grpc-daemon[886]: time=2026-09-21T13:46:08.651Z level=INFO msg="Uploading 1 narinfos"5354lb # [ 74.404398] niks3-server[1024]: 2026/09/21 13:46:09 INFO Signed narinfos id=22 count=15355lb # [ 74.409587] niks3-server[1024]: 2026/09/21 13:46:09 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5356lb # [ 74.422877] niks3-server[1024]: 2026/09/21 13:46:09 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5357lb # [ 74.429451] niks3-server[1024]: 2026/09/21 13:46:09 INFO Completed upload id=225358worker1 # [ 74.448650] nix-grpc-daemon[886]: time=2026-09-21T13:46:08.676Z level=INFO msg="Upload complete. (292ms)"5359lb # [ 74.431852] niks3-server[1024]: 2026/09/21 13:46:09 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5360worker1 # [ 74.454388] nix-grpc-daemon[787]: ts=2026-09-21T13:46:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=10 outputs=15361client # copying 1 paths...5362worker1 # [ 74.463809] nix-grpc-daemon[787]: ts=2026-09-21T13:46:08Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:504305363client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5364worker1 # [ 74.469889] nix-daemon[871]: reaped child process 1240, status = succeeded5365worker1 # [ 74.480110] nix-grpc-daemon[787]: ts=2026-09-21T13:46:08Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:504305366worker1 # [ 74.483191] nix-grpc-daemon[787]: ts=2026-09-21T13:46:08Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50430 duration_s=0 paths=1 nar_bytes_out=1205367client # [ 74.555318] nix-daemon[950]: reaped child process 1058, status = succeeded5368client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.59 seconds)5369client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25370client # [ 74.735162] nix-daemon[950]: accepted connection from pid 1077, user root (trusted)5371client # this derivation will be built:5372client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5373client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5374worker2 # [ 74.934659] nix-grpc-daemon[787]: ts=2026-09-21T13:46:09Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:568525375worker1 # [ 74.968623] nix-grpc-daemon[787]: ts=2026-09-21T13:46:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504305376worker1 # [ 74.978281] nix-grpc-daemon[886]: time=2026-09-21T13:46:09.206Z level=INFO msg="All 1 paths already cached"5377worker1 # [ 74.991885] nix-grpc-daemon[787]: ts=2026-09-21T13:46:09Z level=debug event=want client=11 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05378worker1 # [ 74.997165] nix-grpc-daemon[787]: ts=2026-09-21T13:46:09Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15379worker2 # [ 74.967114] nix-grpc-daemon[787]: ts=2026-09-21T13:46:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:568525380worker2 # [ 74.979899] nix-grpc-daemon[787]: ts=2026-09-21T13:46:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:568525381worker2 # [ 74.993318] nix-grpc-daemon[883]: time=2026-09-21T13:46:09.873Z level=INFO msg="All 1 paths already cached"5382client # copying 1 paths to 'lb:50051'...5383client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5384worker2 # [ 75.000308] nix-grpc-daemon[787]: ts=2026-09-21T13:46:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:568525385worker2 # [ 75.011910] nix-daemon[856]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5386worker2 # [ 75.199194] nix-daemon[856]: accepted connection from pid 996, user nix-grpc-daemon (trusted)5387worker2 # [ 75.211847] nix-daemon[856]: reaped child process 999, status = succeeded5388worker2 # [ 75.386516] nix-daemon[856]: accepted connection from pid 1001, user nix-grpc-daemon (trusted)5389worker2 # [ 75.399057] nix-daemon[856]: reaped child process 1004, status = succeeded5390lb # [ 75.431954] niks3-server[1024]: 2026/09/21 13:46:10 INFO Received uploads request method=POST path=/api/pending_closures5391worker2 # [ 75.428029] nix-grpc-daemon[883]: time=2026-09-21T13:46:10.308Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5392worker2 # [ 75.429461] nix-grpc-daemon[883]: time=2026-09-21T13:46:10.308Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5393lb # [ 75.463351] niks3-server[1024]: 2026/09/21 13:46:10 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5394lb # [ 75.470903] niks3-server[1024]: 2026/09/21 13:46:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5395lb # [ 75.476864] niks3-server[1024]: 2026/09/21 13:46:10 INFO Signed narinfos id=23 count=15396worker2 # [ 75.467127] nix-grpc-daemon[883]: time=2026-09-21T13:46:10.347Z level=INFO msg="Uploading 1 narinfos"5397lb # [ 75.482289] niks3-server[1024]: 2026/09/21 13:46:10 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5398lb # [ 75.492345] niks3-server[1024]: 2026/09/21 13:46:10 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5399worker2 # [ 75.485365] nix-grpc-daemon[883]: time=2026-09-21T13:46:10.365Z level=INFO msg="Upload complete. (461ms)"5400lb # [ 75.498568] niks3-server[1024]: 2026/09/21 13:46:10 INFO Completed upload id=235401worker2 # [ 75.487378] nix-grpc-daemon[787]: ts=2026-09-21T13:46:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56852 duration_s=0 paths=1 nar_bytes_in=5605402lb # [ 75.503266] niks3-server[1024]: 2026/09/21 13:46:10 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5403worker2 # [ 75.492523] nix-grpc-daemon[787]: ts=2026-09-21T13:46:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:568525404worker2 # [ 75.498277] nix-daemon[856]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5405worker2 # [ 75.500600] nix-daemon[856]: reaped child process 994, status = succeeded5406worker2 # [ 75.505990] nix-daemon[856]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5407client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5408client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5409client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5410worker2 # [ 76.519772] nix-daemon[856]: reaped child process 1008, status = succeeded5411worker2 # [ 76.687547] nix-daemon[856]: accepted connection from pid 1024, user nix-grpc-daemon (trusted)5412worker2 # [ 76.699756] nix-daemon[856]: reaped child process 1027, status = succeeded5413lb # [ 76.733093] niks3-server[1024]: 2026/09/21 13:46:11 INFO Received uploads request method=POST path=/api/pending_closures5414worker2 # [ 76.727944] nix-grpc-daemon[883]: time=2026-09-21T13:46:11.608Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5415lb # [ 76.764155] niks3-server[1024]: 2026/09/21 13:46:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5416lb # [ 76.772344] niks3-server[1024]: 2026/09/21 13:46:11 INFO Signed narinfos id=24 count=15417worker2 # [ 76.762314] nix-grpc-daemon[883]: time=2026-09-21T13:46:11.642Z level=INFO msg="Uploading 1 narinfos"5418lb # [ 76.776895] niks3-server[1024]: 2026/09/21 13:46:11 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5419lb # [ 76.783260] niks3-server[1024]: 2026/09/21 13:46:11 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5420lb # [ 76.796707] niks3-server[1024]: 2026/09/21 13:46:11 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5421lb # [ 76.801552] niks3-server[1024]: 2026/09/21 13:46:11 INFO Completed upload id=245422worker2 # [ 76.790845] nix-grpc-daemon[883]: time=2026-09-21T13:46:11.671Z level=INFO msg="Upload complete. (278ms)"5423lb # [ 76.804671] niks3-server[1024]: 2026/09/21 13:46:11 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5424worker2 # [ 76.797280] nix-grpc-daemon[787]: ts=2026-09-21T13:46:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56852 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=11 outputs=15425worker2 # [ 76.809259] nix-daemon[856]: reaped child process 1006, status = succeeded5426client # copying 1 paths...5427worker1 # [ 76.874534] nix-grpc-daemon[787]: ts=2026-09-21T13:46:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:504305428worker1 # [ 76.886290] nix-grpc-daemon[787]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5429client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5430worker2 # [ 76.888172] nix-grpc-daemon[787]: ts=2026-09-21T13:46:11Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:568525431worker2 # [ 76.893109] nix-grpc-daemon[787]: ts=2026-09-21T13:46:11Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56852 duration_s=0 paths=1 nar_bytes_out=1205432client # [ 76.954614] nix-daemon[950]: reaped child process 1080, status = succeeded5433client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.40 seconds)5434(finished: subtest: an input only one worker has still reaches the builder, in 9.53 seconds)5435subtest: upload whose reference the worker lacks is completed from the cache5436client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5437client # this derivation will be built:5438client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5439client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5440client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.40 seconds)5441client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5442lb # [ 77.403869] envoy[914]: conn peer=[2001:db8:1::1]:34124 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=329 ms=875443lb # [ 77.407591] envoy[914]: conn peer=[2001:db8:1::1]:34122 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3880 tx=1861 ms=29815444lb # [ 77.411145] envoy[914]: conn peer=[2001:db8:1::1]:34132 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3684 tx=1857 ms=21745445lb # [ 77.414619] envoy[914]: conn peer=[2001:db8:1::1]:41520 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=525446lb # [ 77.418900] envoy[914]: conn peer=[2001:db8:1::1]:41530 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3755 tx=2015 ms=19715447lb # [ 77.422368] envoy[914]: conn peer=[2001:db8:1::1]:41538 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=421 tx=328 ms=185448client # this derivation will be built:5449client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5450client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5451client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.40 seconds)5452client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5453client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.19 seconds)5454worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5455worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.22 seconds)5456worker2 # [ 78.239880] systemd[1]: Stopping Nix worker-protocol over gRPC...5457worker2 # [ 78.819109] nix-grpc-daemon[787]: ts=2026-09-21T13:46:13Z level=info event=signal_exit5458worker2 # [ 80.071451] nix-grpc-daemon[787]: ts=2026-09-21T13:46:14Z level=info event=scheduler_restarting5459worker1 # [ 80.112791] nix-grpc-daemon[787]: ts=2026-09-21T13:46:14Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:504165460worker1 # [ 80.116715] nix-grpc-daemon[787]: ts=2026-09-21T13:46:14Z level=info event=worker_gone addr=192.168.1.4:500515461worker2 # [ 80.091382] nix-grpc-daemon[787]: ts=2026-09-21T13:46:14Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5462worker2 # [ 80.111260] nix-daemon[856]: reaped child process 858, status = succeeded5463worker2 # [ 80.960341] nix-grpc-daemon[787]: E0000 00:00:1789998375.840390 832 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true5464worker2 # [ 80.965261] nix-grpc-daemon[787]: E0000 00:00:1789998375.840658 832 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5465worker2 # [ 81.735816] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5466worker2 # [ 81.739614] systemd[1]: Stopped Nix worker-protocol over gRPC.5467worker2 # [ 81.743470] systemd[1]: nix-grpc-daemon.service: Consumed 1.449s CPU time over 1min 4.223s wall clock time, 49.9M memory peak, 55.5K incoming IP traffic, 165.3K outgoing IP traffic.5468worker2 # [ 81.754704] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5469worker2 # [ 81.757777] systemd[1]: Closed Nix worker-protocol over gRPC.5470worker2 # [ 81.760379] systemd[1]: nix-grpc-daemon.socket: Consumed 291us CPU time over 1min 12.158s wall clock time, 256K memory peak, 60.8K incoming IP traffic, 50.6K outgoing IP traffic.5471lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5472lb: must succeed: curl -sf localhost:9901/clusters5473lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5474lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.08 seconds)5475client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25476worker1 # [ 82.116870] nix-grpc-daemon[787]: ts=2026-09-21T13:46:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504305477worker1 # [ 82.123586] nix-grpc-daemon[787]: ts=2026-09-21T13:46:16Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:504305478client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5479client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.27 seconds)5480worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5481worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.03 seconds)5482worker2 # [ 82.204143] systemd[1]: Listening on Nix worker-protocol over gRPC.5483worker1 # [ 82.299760] systemd[1]: Stopping Nix worker-protocol over gRPC...5484worker2 # [ 82.719767] systemd[1]: Starting scheduler-token.service...5485worker1 # [ 82.759883] nix-grpc-daemon[787]: ts=2026-09-21T13:46:16Z level=info event=signal_exit5486worker2 # [ 82.827412] systemd[1]: scheduler-token.service: Deactivated successfully.5487worker2 # [ 82.828753] systemd[1]: Finished scheduler-token.service.5488worker2 # [ 82.831154] systemd[1]: scheduler-token.service: Consumed 36ms CPU time over 107ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5489worker2 # [ 82.835644] systemd[1]: Starting Nix worker-protocol over gRPC...5490lb # [ 83.009509] niks3-server[1024]: 2026/09/21 13:46:17 INFO lead: released remote=192.168.1.3:567085491worker2 # [ 83.023551] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:17Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5492worker2 # [ 83.025377] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:17Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5493worker2 # [ 83.029068] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:17Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15494worker2 # [ 83.037062] nix-grpc-daemon[1051]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5495worker2 # [ 83.039927] nix-grpc-daemon[1051]: E0000 00:00:1789998377.917225 1051 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5496worker2 # [ 83.043830] nix-grpc-daemon[1051]: E0000 00:00:1789998377.920182 1051 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5497worker2 # [ 83.047985] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:17Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5498worker2 # [ 83.051739] systemd[1]: Started Nix worker-protocol over gRPC.5499lb # [ 83.072336] niks3-server[1024]: 2026/09/21 13:46:17 INFO lead: acquired remote=192.168.1.4:565865500worker2 # [ 83.060073] nix-grpc-daemon[1051]: E0000 00:00:1789998377.939847 1092 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5501worker2 # [ 83.063423] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:17Z level=info event=scheduler_take_over5502worker2 # [ 83.073412] nix-grpc-daemon[1051]: E0000 00:00:1789998377.953797 1092 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5503worker2 # [ 83.078055] nix-grpc-daemon[1051]: E0000 00:00:1789998377.956031 1092 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5504worker2 # [ 83.087392] nix-grpc-daemon[1051]: E0000 00:00:1789998377.967772 1092 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5505worker2 # [ 83.096445] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:17Z level=info event=scheduler_connected addr=lb:500515506worker2 # [ 83.098076] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5507worker2 # [ 83.299312] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:18Z level=info event=scheduler_connected addr=lb:500515508worker2 # [ 83.303231] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5509worker2 # [ 83.504263] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:18Z level=info event=scheduler_connected addr=lb:500515510worker2 # [ 83.508232] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5511worker2 # [ 83.709391] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:18Z level=info event=scheduler_connected addr=lb:500515512worker2 # [ 83.713271] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5513worker2 # [ 83.914354] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:18Z level=info event=scheduler_connected addr=lb:500515514worker2 # [ 83.918179] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5515worker1 # [ 84.029760] nix-grpc-daemon[787]: ts=2026-09-21T13:46:18Z level=info event=scheduler_restarting5516worker1 # [ 84.035815] nix-grpc-daemon[787]: ts=2026-09-21T13:46:18Z level=info event=scheduler_restarting addr=lb:500515517worker1 # [ 84.039397] nix-grpc-daemon[787]: ts=2026-09-21T13:46:18Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:504165518worker1 # [ 84.043710] nix-grpc-daemon[787]: ts=2026-09-21T13:46:18Z level=info event=worker_gone addr=192.168.1.3:500515519worker1 # [ 84.065923] nix-daemon[871]: reaped child process 873, status = succeeded5520worker2 # [ 84.119836] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:18Z level=info event=scheduler_connected addr=lb:500515521worker2 # [ 84.133141] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:19Z level=info event=oidc_jwks provider=mock keys=15522worker2 # [ 84.136701] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:19Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:578005523worker2 # [ 84.141570] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:19Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05524worker1 # [ 84.245442] nix-grpc-daemon[787]: ts=2026-09-21T13:46:18Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5525worker1 # [ 84.929195] nix-grpc-daemon[787]: E0000 00:00:1789998379.156939 832 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5526worker1 # [ 84.934314] nix-grpc-daemon[787]: E0000 00:00:1789998379.161835 832 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5527worker1 # [ 85.703301] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5528worker1 # [ 85.706530] systemd[1]: Stopped Nix worker-protocol over gRPC.5529worker1 # [ 85.709339] systemd[1]: nix-grpc-daemon.service: Consumed 2.902s CPU time over 1min 8.282s wall clock time, 64.8M memory peak, 104K incoming IP traffic, 231.3K outgoing IP traffic.5530worker1 # [ 85.720282] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5531worker1 # [ 85.723586] systemd[1]: Closed Nix worker-protocol over gRPC.5532worker1 # [ 85.726472] systemd[1]: nix-grpc-daemon.socket: Consumed 289us CPU time over 1min 16.495s wall clock time, 256K memory peak, 99.2K incoming IP traffic, 82.8K outgoing IP traffic.5533lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5534lb: must succeed: curl -sf localhost:9901/clusters5535lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5536lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.08 seconds)5537client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25538worker2 # [ 86.024393] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578065539worker2 # [ 86.035549] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5540worker2 # [ 86.053629] nix-grpc-daemon[1051]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 32 ms (attempt 1/5)5541worker2 # [ 86.091344] nix-grpc-daemon[1051]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 30 ms (attempt 2/5)5542worker2 # [ 86.127107] nix-grpc-daemon[1051]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 82 ms (attempt 3/5)5543worker2 # [ 86.215138] nix-grpc-daemon[1051]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 4 ms (attempt 4/5)5544worker2 # [ 86.225177] nix-grpc-daemon[1051]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5545worker2 # [ 86.228276] nix-grpc-daemon[1051]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5546client # copying 2 paths...5547client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5548worker2 # [ 86.251151] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578065549client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5550worker2 # [ 86.263183] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5551worker2 # [ 86.285822] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:21Z level=info event=niks3_push_started pid=11115552worker2 # [ 86.497803] nix-grpc-daemon[1111]: time=2026-09-21T13:46:21.377Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5553worker2 # [ 86.681062] nix-daemon[856]: accepted connection from pid 1124, user nix-grpc-daemon (trusted)5554worker2 # [ 86.692249] nix-daemon[856]: reaped child process 1127, status = succeeded5555lb # [ 86.726623] niks3-server[1024]: 2026/09/21 13:46:21 INFO Received uploads request method=POST path=/api/pending_closures5556lb # [ 86.735759] niks3-server[1024]: 2026/09/21 13:46:21 INFO Received uploads request method=POST path=/api/pending_closures5557worker2 # [ 86.730067] nix-grpc-daemon[1111]: time=2026-09-21T13:46:21.610Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5558worker2 # [ 86.731642] nix-grpc-daemon[1111]: time=2026-09-21T13:46:21.612Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5559worker2 # [ 86.733464] nix-grpc-daemon[1111]: time=2026-09-21T13:46:21.613Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5560lb # [ 86.790188] niks3-server[1024]: 2026/09/21 13:46:21 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5561lb # [ 86.793423] niks3-server[1024]: 2026/09/21 13:46:21 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5562lb # [ 86.796358] niks3-server[1024]: 2026/09/21 13:46:21 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5563lb # [ 86.799582] niks3-server[1024]: 2026/09/21 13:46:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5564lb # [ 86.801382] niks3-server[1024]: 2026/09/21 13:46:21 INFO Signed narinfos id=25 count=15565lb # [ 86.803371] niks3-server[1024]: 2026/09/21 13:46:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5566lb # [ 86.805814] niks3-server[1024]: 2026/09/21 13:46:21 INFO Signed narinfos id=26 count=25567worker2 # [ 86.792600] nix-grpc-daemon[1111]: time=2026-09-21T13:46:21.672Z level=INFO msg="Uploading 3 narinfos"5568lb # [ 86.807706] niks3-server[1024]: 2026/09/21 13:46:21 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5569lb # [ 86.820077] niks3-server[1024]: 2026/09/21 13:46:21 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5570lb # [ 86.822795] niks3-server[1024]: 2026/09/21 13:46:21 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5571lb # [ 86.826056] niks3-server[1024]: 2026/09/21 13:46:21 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5572lb # [ 86.829693] niks3-server[1024]: 2026/09/21 13:46:21 INFO Completed upload id=265573lb # [ 86.830961] niks3-server[1024]: 2026/09/21 13:46:21 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5574lb # [ 86.833257] niks3-server[1024]: 2026/09/21 13:46:21 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5575lb # [ 86.835556] niks3-server[1024]: 2026/09/21 13:46:21 INFO Completed upload id=255576worker2 # [ 86.823705] nix-grpc-daemon[1111]: time=2026-09-21T13:46:21.704Z level=INFO msg="Upload complete. (322ms)"5577worker2 # [ 86.828227] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57806 duration_s=0 paths=2 nar_bytes_in=3125578worker2 # [ 86.840271] nix-daemon[856]: reaped child process 1109, status = succeeded5579client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.08 seconds)5580worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5581worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.03 seconds)5582worker1 # [ 86.987561] systemd[1]: Listening on Nix worker-protocol over gRPC.5583worker1 # [ 86.994997] systemd[1]: Starting Nix worker-protocol over gRPC...5584worker1 # [ 87.163364] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:21Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5585worker1 # [ 87.166133] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:21Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5586worker1 # [ 87.169853] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:21Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15587worker1 # [ 87.177064] nix-grpc-daemon[1287]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5588worker1 # [ 87.178432] nix-grpc-daemon[1287]: E0000 00:00:1789998381.404951 1287 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5589worker1 # [ 87.184891] nix-grpc-daemon[1287]: E0000 00:00:1789998381.410186 1287 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5590worker1 # [ 87.188224] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:21Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5591worker1 # [ 87.190684] systemd[1]: Started Nix worker-protocol over gRPC.5592lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5593lb: must succeed: curl -sf localhost:9901/clusters5594worker1 # [ 87.197178] nix-grpc-daemon[1287]: E0000 00:00:1789998381.424264 1328 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5595worker1 # [ 87.208539] nix-grpc-daemon[1287]: E0000 00:00:1789998381.436454 1328 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5596worker1 # [ 87.213053] nix-grpc-daemon[1287]: E0000 00:00:1789998381.438657 1328 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5597worker1 # [ 87.222458] nix-grpc-daemon[1287]: E0000 00:00:1789998381.450374 1328 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5598worker1 # [ 87.233730] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:21Z level=info event=scheduler_connected addr=lb:500515599worker2 # [ 87.205667] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:22Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:578005600worker2 # [ 87.209486] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:22Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05601lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5602lb # [ 87.404777] envoy[914]: conn peer=[2001:db8:1::4]:34866 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=78488 tx=3517 ms=622055603lb # [ 87.410313] envoy[914]: conn peer=[2001:db8:1::1]:41548 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=215604lb # [ 87.413697] envoy[914]: conn peer=[2001:db8:1::3]:56766 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=19429 tx=3761 ms=653265605lb # [ 87.417646] envoy[914]: conn peer=[2001:db8:1::1]:58434 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1006 tx=304 ms=8185606lb: must succeed: curl -sf localhost:9901/clusters5607lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5608lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.16 seconds)5609(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.35 seconds)5610subtest: a niks3 restart does not cost the leader its role5611lb: must succeed: curl -sf localhost:9901/clusters5612lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5613lb: must succeed: curl -sf localhost:9901/clusters5614lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5615lb # [ 88.480704] niks3-server[1024]: 2026/09/21 13:46:23 INFO Shutdown signal received, draining in-flight requests timeout=10s5616lb # [ 88.483370] systemd[1]: Stopping niks3 server...5617lb # [ 88.486041] niks3-server[1024]: 2026/09/21 13:46:23 INFO lead: released remote=192.168.1.4:565865618worker1 # [ 88.503479] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:22Z level=info event=scheduler_lock_lost status=2005619worker2 # [ 88.472699] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:23Z level=info event=scheduler_lock_lost status=2005620lb # [ 88.497307] systemd[1]: niks3.service: Deactivated successfully.5621lb # [ 88.499175] systemd[1]: Stopped niks3 server.5622lb # [ 88.499796] systemd[1]: niks3.service: Consumed 632ms CPU time over 1min 2.304s wall clock time, 20.5M memory peak, 73K incoming IP traffic, 103.3K outgoing IP traffic.5623lb # [ 88.508454] systemd[1]: Starting niks3 server...5624lb # [ 88.581165] niks3-server[1158]: 2026/09/21 13:46:23 goose: no migrations to run. current version: 202609200000005625lb # [ 88.587975] niks3-server[1158]: 2026/09/21 13:46:23 OK 1_commit_pending_closure.sql (4.94ms)5626lb # [ 88.591759] niks3-server[1158]: 2026/09/21 13:46:23 OK 2_object_stats_trigger.sql (2.13ms)5627lb # [ 88.592870] niks3-server[1158]: 2026/09/21 13:46:23 goose: up to current file version: 25628lb # [ 88.594928] niks3-server[1158]: 2026/09/21 13:46:23 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key5629lb # [ 88.607785] niks3-server[1158]: 2026/09/21 13:46:23 INFO Read proxy enabled — serving cache objects from S35630lb # [ 88.609265] niks3-server[1158]: 2026/09/21 13:46:23 INFO Using socket-activated listener address=0.0.0.0:57515631lb # [ 88.610712] niks3-server[1158]: 2026/09/21 13:46:23 INFO systemd watchdog enabled interval=15s5632lb # [ 88.612202] niks3-server[1158]: 2026/09/21 13:46:23 INFO Starting HTTP server address=0.0.0.0:57515633lb # [ 88.613993] systemd[1]: Started niks3 server.5634lb: waiting for unit niks3.service5635lb: (finished: waiting for unit niks3.service, in 0.04 seconds)5636lb: must succeed: curl -sf localhost:9901/clusters5637lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5638worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5639worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5640lb: must succeed: curl -sf localhost:9901/clusters5641lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5642lb: must succeed: curl -sf localhost:9901/clusters5643lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5644worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5645worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.06 seconds)5646client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&25647worker2 # [ 89.173768] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:578065648client # don't know how to build these paths:5649client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5650worker2 # [ 89.182645] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578065651worker2 # [ 89.210200] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:24Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=05652worker2 # [ 89.216429] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:24Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.4:50051 clients=15653worker2 # [ 89.256482] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578065654worker2 # [ 89.269809] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578065655client # copying 1 paths to 'lb:50051'...5656client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...5657worker2 # [ 89.282969] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578065658worker2 # [ 89.294098] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5659lb # [ 89.492472] niks3-server[1158]: 2026/09/21 13:46:24 INFO lead: acquired remote=192.168.1.4:566625660worker2 # [ 89.494983] nix-daemon[856]: accepted connection from pid 1144, user nix-grpc-daemon (trusted)5661worker2 # [ 89.507151] nix-daemon[856]: reaped child process 1149, status = succeeded5662worker2 # [ 89.687577] nix-daemon[856]: accepted connection from pid 1151, user nix-grpc-daemon (trusted)5663worker2 # [ 89.699531] nix-daemon[856]: reaped child process 1154, status = succeeded5664lb # [ 89.732800] niks3-server[1158]: 2026/09/21 13:46:24 INFO Received uploads request method=POST path=/api/pending_closures5665worker2 # [ 89.743808] nix-grpc-daemon[1111]: time=2026-09-21T13:46:24.623Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5666worker2 # [ 89.745367] nix-grpc-daemon[1111]: time=2026-09-21T13:46:24.623Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"5667lb # [ 89.784386] niks3-server[1158]: 2026/09/21 13:46:24 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst5668lb # [ 89.791170] niks3-server[1158]: 2026/09/21 13:46:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5669lb # [ 89.802194] niks3-server[1158]: 2026/09/21 13:46:24 INFO Signed narinfos id=27 count=15670worker2 # [ 89.790557] nix-grpc-daemon[1111]: time=2026-09-21T13:46:24.670Z level=INFO msg="Uploading 1 narinfos"5671lb # [ 89.811183] niks3-server[1158]: 2026/09/21 13:46:24 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5672lb # [ 89.818912] niks3-server[1158]: 2026/09/21 13:46:24 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls5673lb # [ 89.824417] niks3-server[1158]: 2026/09/21 13:46:24 INFO Completed upload id=275674worker2 # [ 89.811126] nix-grpc-daemon[1111]: time=2026-09-21T13:46:24.691Z level=INFO msg="Upload complete. (504ms)"5675lb # [ 89.829862] niks3-server[1158]: 2026/09/21 13:46:24 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo5676worker2 # [ 89.817114] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57806 duration_s=0 paths=1 nar_bytes_in=5365677worker2 # [ 89.822973] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578065678worker2 # [ 89.835116] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5679worker2 # [ 89.841593] nix-daemon[856]: reaped child process 1142, status = succeeded5680worker2 # [ 89.849481] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5681client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...5682client # farm-a> worker2: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5683client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5684client # farm-a> LOG-farm-a-after-niks3-restart5685worker2 # [ 90.873596] nix-daemon[856]: reaped child process 1158, status = succeeded5686worker2 # [ 91.071390] nix-daemon[856]: accepted connection from pid 1174, user nix-grpc-daemon (trusted)5687worker2 # [ 91.084631] nix-daemon[856]: reaped child process 1177, status = succeeded5688lb # [ 91.118546] niks3-server[1158]: 2026/09/21 13:46:26 INFO Received uploads request method=POST path=/api/pending_closures5689worker2 # [ 91.117472] nix-grpc-daemon[1111]: time=2026-09-21T13:46:25.997Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5690worker2 # [ 91.119093] nix-grpc-daemon[1111]: time=2026-09-21T13:46:25.999Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"5691lb # [ 91.158311] niks3-server[1158]: 2026/09/21 13:46:26 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5692lb # [ 91.174500] niks3-server[1158]: 2026/09/21 13:46:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5693lb # [ 91.178876] niks3-server[1158]: 2026/09/21 13:46:26 INFO Signed narinfos id=28 count=15694worker2 # [ 91.171856] nix-grpc-daemon[1111]: time=2026-09-21T13:46:26.051Z level=INFO msg="Uploading 1 narinfos"5695lb # [ 91.187157] niks3-server[1158]: 2026/09/21 13:46:26 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5696lb # [ 91.193313] niks3-server[1158]: 2026/09/21 13:46:26 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls5697lb # [ 91.198055] niks3-server[1158]: 2026/09/21 13:46:26 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5698lb # [ 91.203546] niks3-server[1158]: 2026/09/21 13:46:26 INFO Completed upload id=285699worker2 # [ 91.190553] nix-grpc-daemon[1111]: time=2026-09-21T13:46:26.070Z level=INFO msg="Upload complete. (332ms)"5700lb # [ 91.205902] niks3-server[1158]: 2026/09/21 13:46:26 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo5701worker2 # [ 91.201610] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57806 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=15702worker2 # [ 91.218954] nix-daemon[856]: reaped child process 1156, status = succeeded5703worker2 # [ 91.221963] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:26Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=05704worker2 # [ 91.223952] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:26Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.4:50051 clients=15705worker2 # [ 91.268258] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578065706worker2 # [ 91.281060] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578065707worker2 # [ 91.295452] nix-grpc-daemon[1111]: time=2026-09-21T13:46:26.175Z level=INFO msg="All 1 paths already cached"5708client # copying 1 paths to 'lb:50051'...5709client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...5710worker2 # [ 91.303544] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578065711worker2 # [ 91.311822] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5712worker2 # [ 91.512444] nix-daemon[856]: accepted connection from pid 1183, user nix-grpc-daemon (trusted)5713worker2 # [ 91.525071] nix-daemon[856]: reaped child process 1186, status = succeeded5714worker2 # [ 91.720343] nix-daemon[856]: accepted connection from pid 1188, user nix-grpc-daemon (trusted)5715worker2 # [ 91.732515] nix-daemon[856]: reaped child process 1191, status = succeeded5716lb # [ 91.765476] niks3-server[1158]: 2026/09/21 13:46:26 INFO Received uploads request method=POST path=/api/pending_closures5717worker2 # [ 91.761307] nix-grpc-daemon[1111]: time=2026-09-21T13:46:26.641Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5718worker2 # [ 91.763479] nix-grpc-daemon[1111]: time=2026-09-21T13:46:26.643Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"5719lb # [ 91.794195] niks3-server[1158]: 2026/09/21 13:46:26 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst5720lb # [ 91.805274] niks3-server[1158]: 2026/09/21 13:46:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5721lb # [ 91.810669] niks3-server[1158]: 2026/09/21 13:46:26 INFO Signed narinfos id=29 count=15722worker2 # [ 91.800561] nix-grpc-daemon[1111]: time=2026-09-21T13:46:26.680Z level=INFO msg="Uploading 1 narinfos"5723lb # [ 91.818488] niks3-server[1158]: 2026/09/21 13:46:26 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls5724lb # [ 91.828386] niks3-server[1158]: 2026/09/21 13:46:26 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5725lb # [ 91.835553] niks3-server[1158]: 2026/09/21 13:46:26 INFO Completed upload id=295726worker2 # [ 91.822584] nix-grpc-daemon[1111]: time=2026-09-21T13:46:26.702Z level=INFO msg="Upload complete. (497ms)"5727lb # [ 91.840133] niks3-server[1158]: 2026/09/21 13:46:26 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo5728worker2 # [ 91.826283] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57806 duration_s=0 paths=1 nar_bytes_in=6565729worker2 # [ 91.831358] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578065730worker2 # [ 91.840492] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5731worker2 # [ 91.847219] nix-daemon[856]: reaped child process 1181, status = succeeded5732worker2 # [ 91.852408] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5733client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...5734client # farm-b> worker2: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5735client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5736client # farm-b> LOG-farm-b-after-niks3-restart5737worker2 # [ 92.877584] nix-daemon[856]: reaped child process 1195, status = succeeded5738worker2 # [ 93.068586] nix-daemon[856]: accepted connection from pid 1211, user nix-grpc-daemon (trusted)5739worker2 # [ 93.080646] nix-daemon[856]: reaped child process 1214, status = succeeded5740lb # [ 93.117978] niks3-server[1158]: 2026/09/21 13:46:28 INFO Received uploads request method=POST path=/api/pending_closures5741worker2 # [ 93.112540] nix-grpc-daemon[1111]: time=2026-09-21T13:46:27.992Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5742worker2 # [ 93.114084] nix-grpc-daemon[1111]: time=2026-09-21T13:46:27.994Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"5743lb # [ 93.161665] niks3-server[1158]: 2026/09/21 13:46:28 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst5744lb # [ 93.169586] niks3-server[1158]: 2026/09/21 13:46:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5745worker2 # [ 93.161896] nix-grpc-daemon[1111]: time=2026-09-21T13:46:28.042Z level=INFO msg="Uploading 1 narinfos"5746lb # [ 93.174685] niks3-server[1158]: 2026/09/21 13:46:28 INFO Signed narinfos id=30 count=15747lb # [ 93.181663] niks3-server[1158]: 2026/09/21 13:46:28 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5748lb # [ 93.189072] niks3-server[1158]: 2026/09/21 13:46:28 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls5749lb # [ 93.192559] niks3-server[1158]: 2026/09/21 13:46:28 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5750lb # [ 93.195623] niks3-server[1158]: 2026/09/21 13:46:28 INFO Completed upload id=305751lb # [ 93.197901] niks3-server[1158]: 2026/09/21 13:46:28 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo5752worker2 # [ 93.183950] nix-grpc-daemon[1111]: time=2026-09-21T13:46:28.064Z level=INFO msg="Upload complete. (324ms)"5753worker2 # [ 93.190559] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57806 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=15754worker2 # [ 93.200917] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:28Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=05755worker2 # [ 93.204899] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:28Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.4:50051 clients=15756worker2 # [ 93.213885] nix-daemon[856]: reaped child process 1193, status = succeeded5757worker2 # [ 93.246470] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578065758worker2 # [ 93.259661] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578065759worker2 # [ 93.274740] nix-grpc-daemon[1111]: time=2026-09-21T13:46:28.155Z level=INFO msg="All 2 paths already cached"5760client # copying 1 paths to 'lb:50051'...5761client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...5762worker2 # [ 93.283572] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578065763worker2 # [ 93.292327] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5764worker2 # [ 93.483700] nix-daemon[856]: accepted connection from pid 1220, user nix-grpc-daemon (trusted)5765worker2 # [ 93.496087] nix-daemon[856]: reaped child process 1223, status = succeeded5766worker2 # [ 93.662406] nix-daemon[856]: accepted connection from pid 1225, user nix-grpc-daemon (trusted)5767worker2 # [ 93.674395] nix-daemon[856]: reaped child process 1228, status = succeeded5768lb # [ 93.705259] niks3-server[1158]: 2026/09/21 13:46:28 INFO Received uploads request method=POST path=/api/pending_closures5769worker2 # [ 93.701659] nix-grpc-daemon[1111]: time=2026-09-21T13:46:28.582Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5770worker2 # [ 93.703206] nix-grpc-daemon[1111]: time=2026-09-21T13:46:28.583Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"5771lb # [ 93.739276] niks3-server[1158]: 2026/09/21 13:46:28 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst5772lb # [ 93.751375] niks3-server[1158]: 2026/09/21 13:46:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5773lb # [ 93.756108] niks3-server[1158]: 2026/09/21 13:46:28 INFO Signed narinfos id=31 count=15774worker2 # [ 93.745351] nix-grpc-daemon[1111]: time=2026-09-21T13:46:28.625Z level=INFO msg="Uploading 1 narinfos"5775lb # [ 93.764094] niks3-server[1158]: 2026/09/21 13:46:28 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls5776lb # [ 93.771647] niks3-server[1158]: 2026/09/21 13:46:28 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5777lb # [ 93.778493] niks3-server[1158]: 2026/09/21 13:46:28 INFO Completed upload id=315778worker2 # [ 93.765557] nix-grpc-daemon[1111]: time=2026-09-21T13:46:28.645Z level=INFO msg="Upload complete. (460ms)"5779lb # [ 93.780951] niks3-server[1158]: 2026/09/21 13:46:28 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo5780worker2 # [ 93.769362] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57806 duration_s=0 paths=1 nar_bytes_in=7845781worker2 # [ 93.775298] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578065782worker2 # [ 93.783492] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5783worker2 # [ 93.790653] nix-daemon[856]: reaped child process 1218, status = succeeded5784worker2 # [ 93.795924] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5785client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...5786client # farm-top> worker2: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5787client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5788client # farm-top> LOG-farm-top-after-niks3-restart5789worker2 # [ 95.009543] nix-daemon[856]: reaped child process 1232, status = succeeded5790worker2 # [ 95.184463] nix-daemon[856]: accepted connection from pid 1248, user nix-grpc-daemon (trusted)5791worker2 # [ 95.196739] nix-daemon[856]: reaped child process 1251, status = succeeded5792lb # [ 95.230561] niks3-server[1158]: 2026/09/21 13:46:30 INFO Received uploads request method=POST path=/api/pending_closures5793worker2 # [ 95.228178] nix-grpc-daemon[1111]: time=2026-09-21T13:46:30.108Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5794worker2 # [ 95.229770] nix-grpc-daemon[1111]: time=2026-09-21T13:46:30.110Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"5795lb # [ 95.276288] niks3-server[1158]: 2026/09/21 13:46:30 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5796lb # [ 95.281217] niks3-server[1158]: 2026/09/21 13:46:30 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst5797lb # [ 95.293075] niks3-server[1158]: 2026/09/21 13:46:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5798lb # [ 95.298116] niks3-server[1158]: 2026/09/21 13:46:30 INFO Signed narinfos id=32 count=15799worker2 # [ 95.287961] nix-grpc-daemon[1111]: time=2026-09-21T13:46:30.167Z level=INFO msg="Uploading 1 narinfos"5800lb # [ 95.300751] niks3-server[1158]: 2026/09/21 13:46:30 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls5801lb # [ 95.310242] niks3-server[1158]: 2026/09/21 13:46:30 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5802lb # [ 95.313854] niks3-server[1158]: 2026/09/21 13:46:30 INFO Completed upload id=325803worker2 # [ 95.300945] nix-grpc-daemon[1111]: time=2026-09-21T13:46:30.181Z level=INFO msg="Upload complete. (297ms)"5804lb # [ 95.316233] niks3-server[1158]: 2026/09/21 13:46:30 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo5805worker2 # [ 95.309977] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57806 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=15806worker2 # [ 95.322211] nix-daemon[856]: reaped child process 1230, status = succeeded5807client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&2, in 6.41 seconds)5808client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath5809client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath, in 0.21 seconds)5810(finished: subtest: a niks3 restart does not cost the leader its role, in 7.24 seconds)5811subtest: scheduler down: the other node takes the lock and keeps it5812lb: must succeed: curl -sf localhost:9901/clusters5813lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5814lb: must succeed: curl -sf localhost:9901/clusters5815lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5816lb: must succeed: curl -sf localhost:9901/clusters5817lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5818lb: must succeed: curl -sf localhost:9901/clusters5819lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5820worker2 # [ 95.823706] systemd[1]: Stopping Nix worker-protocol over gRPC...5821worker2 # [ 96.048180] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:30Z level=info event=signal_exit5822lb # [ 96.698910] niks3-server[1158]: 2026/09/21 13:46:31 INFO lead: released remote=192.168.1.4:566625823lb # [ 97.404148] envoy[914]: conn peer=[2001:db8:1::1]:58448 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9752 tx=4147 ms=61485824worker1 # [ 97.719309] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:31Z level=info event=scheduler_restarting addr=lb:500515825worker2 # [ 97.690930] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:32Z level=info event=scheduler_restarting5826worker2 # [ 97.694890] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:32Z level=info event=scheduler_restarting addr=lb:500515827worker2 # [ 97.698155] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:32Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:578005828worker2 # [ 97.701583] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:32Z level=info event=worker_gone addr=192.168.1.3:500515829worker2 # [ 97.704916] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:32Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:578005830worker2 # [ 97.708826] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:32Z level=info event=worker_gone addr=192.168.1.4:500515831worker2 # [ 97.723703] nix-daemon[856]: reaped child process 1103, status = succeeded5832worker1 # [ 97.918902] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:32Z level=info event=scheduler_connected addr=lb:500515833worker1 # [ 97.923119] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5834worker2 # [ 97.898621] nix-grpc-daemon[1051]: ts=2026-09-21T13:46:32Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5835worker1 # [ 98.123985] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:32Z level=info event=scheduler_connected addr=lb:500515836worker1 # [ 98.128178] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5837worker2 # [ 98.095525] nix-grpc-daemon[1051]: E0000 00:00:1789998392.975699 1096 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5838worker2 # [ 98.100648] nix-grpc-daemon[1051]: E0000 00:00:1789998392.976206 1096 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5839worker1 # [ 98.328936] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:32Z level=info event=scheduler_connected addr=lb:500515840worker1 # [ 98.332874] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5841worker1 # [ 98.533837] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:32Z level=info event=scheduler_connected addr=lb:500515842worker1 # [ 98.537737] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5843worker1 # [ 98.739659] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:32Z level=info event=scheduler_connected addr=lb:500515844worker1 # [ 98.743112] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5845worker2 # [ 98.809961] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5846worker2 # [ 98.813199] systemd[1]: Stopped Nix worker-protocol over gRPC.5847worker2 # [ 98.816836] systemd[1]: nix-grpc-daemon.service: Consumed 1.124s CPU time over 15.973s wall clock time, 35.3M memory peak, 46.1K incoming IP traffic, 83.1K outgoing IP traffic.5848worker2 # [ 98.827838] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5849worker2 # [ 98.831535] systemd[1]: Closed Nix worker-protocol over gRPC.5850worker2 # [ 98.834715] systemd[1]: nix-grpc-daemon.socket: Consumed 306us CPU time over 16.631s wall clock time, 256K memory peak, 38.4K incoming IP traffic, 28.7K outgoing IP traffic.5851worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5852worker1 # [ 98.944206] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:33Z level=info event=scheduler_connected addr=lb:500515853worker1 # [ 98.947375] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5854worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds)5855worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5856worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5857worker1 # [ 99.148314] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:33Z level=info event=scheduler_connected addr=lb:500515858worker1 # [ 99.152218] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5859worker1 # [ 99.353432] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:33Z level=info event=scheduler_connected addr=lb:500515860worker1 # [ 99.357397] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5861worker1 # [ 99.513290] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:33Z level=info event=scheduler_take_over5862lb # [ 99.494725] niks3-server[1158]: 2026/09/21 13:46:34 INFO lead: acquired remote=192.168.1.3:510225863worker1 # [ 99.558235] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:33Z level=info event=scheduler_connected addr=lb:500515864worker1 # [ 99.562146] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5865worker1 # [ 99.763636] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:33Z level=info event=scheduler_connected addr=lb:500515866worker1 # [ 99.770181] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:33Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:593105867worker1 # [ 99.775141] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:34Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05868worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5869worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5870client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&25871worker1 # [ 100.299575] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:593185872worker1 # [ 100.311371] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)5873client # don't know how to build these paths:5874client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5875worker1 # [ 100.325808] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593185876worker1 # [ 100.337337] nix-grpc-daemon[1287]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 70 ms (attempt 1/5)5877worker1 # [ 100.412915] nix-grpc-daemon[1287]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 2/5)5878worker1 # [ 100.470869] nix-grpc-daemon[1287]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 119 ms (attempt 3/5)5879worker1 # [ 100.595901] nix-grpc-daemon[1287]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 593 ms (attempt 4/5)5880worker1 # [ 101.195732] nix-grpc-daemon[1287]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5881worker1 # [ 101.199981] nix-grpc-daemon[1287]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5882worker1 # [ 101.230238] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:35Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=05883worker1 # [ 101.235646] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:35Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=15884worker1 # [ 101.275747] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593185885worker1 # [ 101.288552] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593185886client # copying 1 paths to 'lb:50051'...5887client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...5888worker1 # [ 101.301369] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:593185889worker1 # [ 101.312796] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)5890worker1 # [ 101.332923] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:35Z level=info event=niks3_push_started pid=13635891worker1 # [ 101.530580] nix-grpc-daemon[1363]: time=2026-09-21T13:46:35.757Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""5892worker1 # [ 101.697375] nix-daemon[871]: accepted connection from pid 1376, user nix-grpc-daemon (trusted)5893worker1 # [ 101.707606] nix-daemon[871]: reaped child process 1379, status = succeeded5894worker1 # [ 101.882055] nix-daemon[871]: accepted connection from pid 1381, user nix-grpc-daemon (trusted)5895worker1 # [ 101.893607] nix-daemon[871]: reaped child process 1384, status = succeeded5896lb # [ 101.893427] niks3-server[1158]: 2026/09/21 13:46:36 INFO Received uploads request method=POST path=/api/pending_closures5897worker1 # [ 101.923050] nix-grpc-daemon[1363]: time=2026-09-21T13:46:36.150Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5898worker1 # [ 101.926048] nix-grpc-daemon[1363]: time=2026-09-21T13:46:36.152Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"5899lb # [ 101.930599] niks3-server[1158]: 2026/09/21 13:46:36 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst5900lb # [ 101.943588] niks3-server[1158]: 2026/09/21 13:46:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5901worker1 # [ 101.973314] nix-grpc-daemon[1363]: time=2026-09-21T13:46:36.200Z level=INFO msg="Uploading 1 narinfos"5902lb # [ 101.954272] niks3-server[1158]: 2026/09/21 13:46:36 INFO Signed narinfos id=33 count=15903lb # [ 101.964106] niks3-server[1158]: 2026/09/21 13:46:36 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls5904lb # [ 101.974273] niks3-server[1158]: 2026/09/21 13:46:36 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5905lb # [ 101.979478] niks3-server[1158]: 2026/09/21 13:46:36 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo5906lb # [ 101.982044] niks3-server[1158]: 2026/09/21 13:46:36 INFO Completed upload id=335907worker1 # [ 102.001313] nix-grpc-daemon[1363]: time=2026-09-21T13:46:36.229Z level=INFO msg="Upload complete. (467ms)"5908worker1 # [ 102.006437] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59318 duration_s=0 paths=1 nar_bytes_in=5205909worker1 # [ 102.013365] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593185910worker1 # [ 102.022122] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)5911worker1 # [ 102.033138] nix-daemon[871]: reaped child process 1361, status = succeeded5912worker1 # [ 102.036808] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)5913client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...5914client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5915client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5916client # farm-a> LOG-farm-a-on-standby5917worker1 # [ 102.847572] nix-daemon[871]: reaped child process 1388, status = succeeded5918worker1 # [ 103.008639] nix-daemon[871]: accepted connection from pid 1404, user nix-grpc-daemon (trusted)5919worker1 # [ 103.019287] nix-daemon[871]: reaped child process 1407, status = succeeded5920lb # [ 103.018888] niks3-server[1158]: 2026/09/21 13:46:37 INFO Received uploads request method=POST path=/api/pending_closures5921worker1 # [ 103.049065] nix-grpc-daemon[1363]: time=2026-09-21T13:46:37.276Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5922worker1 # [ 103.050795] nix-grpc-daemon[1363]: time=2026-09-21T13:46:37.278Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"5923lb # [ 103.058457] niks3-server[1158]: 2026/09/21 13:46:37 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5924lb # [ 103.070704] niks3-server[1158]: 2026/09/21 13:46:37 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5925lb # [ 103.082231] niks3-server[1158]: 2026/09/21 13:46:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5926worker1 # [ 103.105563] nix-grpc-daemon[1363]: time=2026-09-21T13:46:37.333Z level=INFO msg="Uploading 1 narinfos"5927lb # [ 103.087098] niks3-server[1158]: 2026/09/21 13:46:37 INFO Signed narinfos id=34 count=15928lb # [ 103.093043] niks3-server[1158]: 2026/09/21 13:46:37 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls5929lb # [ 103.099830] niks3-server[1158]: 2026/09/21 13:46:37 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5930lb # [ 103.103990] niks3-server[1158]: 2026/09/21 13:46:38 INFO Completed upload id=345931lb # [ 103.105978] niks3-server[1158]: 2026/09/21 13:46:38 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo5932worker1 # [ 103.124263] nix-grpc-daemon[1363]: time=2026-09-21T13:46:37.352Z level=INFO msg="Upload complete. (282ms)"5933worker1 # [ 103.130758] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59318 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=15934worker1 # [ 103.142783] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:37Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=05935worker1 # [ 103.146003] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:37Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=15936worker1 # [ 103.153785] nix-daemon[871]: reaped child process 1386, status = succeeded5937worker1 # [ 103.186907] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593185938worker1 # [ 103.200992] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593185939worker1 # [ 103.214330] nix-grpc-daemon[1363]: time=2026-09-21T13:46:37.442Z level=INFO msg="All 1 paths already cached"5940client # copying 1 paths to 'lb:50051'...5941client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...5942worker1 # [ 103.222625] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:593185943worker1 # [ 103.233178] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)5944worker1 # [ 103.410113] nix-daemon[871]: accepted connection from pid 1413, user nix-grpc-daemon (trusted)5945worker1 # [ 103.421984] nix-daemon[871]: reaped child process 1416, status = succeeded5946worker1 # [ 103.598576] nix-daemon[871]: accepted connection from pid 1418, user nix-grpc-daemon (trusted)5947worker1 # [ 103.610281] nix-daemon[871]: reaped child process 1421, status = succeeded5948lb # [ 103.608905] niks3-server[1158]: 2026/09/21 13:46:38 INFO Received uploads request method=POST path=/api/pending_closures5949worker1 # [ 103.638272] nix-grpc-daemon[1363]: time=2026-09-21T13:46:37.866Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5950worker1 # [ 103.640270] nix-grpc-daemon[1363]: time=2026-09-21T13:46:37.867Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"5951lb # [ 103.638070] niks3-server[1158]: 2026/09/21 13:46:38 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst5952lb # [ 103.649119] niks3-server[1158]: 2026/09/21 13:46:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5953lb # [ 103.653909] niks3-server[1158]: 2026/09/21 13:46:38 INFO Signed narinfos id=35 count=15954worker1 # [ 103.675903] nix-grpc-daemon[1363]: time=2026-09-21T13:46:37.903Z level=INFO msg="Uploading 1 narinfos"5955lb # [ 103.662647] niks3-server[1158]: 2026/09/21 13:46:38 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls5956lb # [ 103.672221] niks3-server[1158]: 2026/09/21 13:46:38 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5957worker1 # [ 103.697626] nix-grpc-daemon[1363]: time=2026-09-21T13:46:37.925Z level=INFO msg="Upload complete. (454ms)"5958lb # [ 103.678756] niks3-server[1158]: 2026/09/21 13:46:38 INFO Completed upload id=355959worker1 # [ 103.700230] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59318 duration_s=0 paths=1 nar_bytes_in=6405960lb # [ 103.683118] niks3-server[1158]: 2026/09/21 13:46:38 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo5961worker1 # [ 103.706282] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593185962worker1 # [ 103.717107] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)5963worker1 # [ 103.721122] nix-daemon[871]: reaped child process 1411, status = succeeded5964worker1 # [ 103.724855] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)5965client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...5966client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5967client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5968client # farm-b> LOG-farm-b-on-standby5969worker1 # [ 104.951469] nix-daemon[871]: reaped child process 1425, status = succeeded5970worker1 # [ 105.125023] nix-daemon[871]: accepted connection from pid 1441, user nix-grpc-daemon (trusted)5971worker1 # [ 105.136608] nix-daemon[871]: reaped child process 1444, status = succeeded5972lb # [ 105.137302] niks3-server[1158]: 2026/09/21 13:46:40 INFO Received uploads request method=POST path=/api/pending_closures5973worker1 # [ 105.167879] nix-grpc-daemon[1363]: time=2026-09-21T13:46:39.395Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5974worker1 # [ 105.169394] nix-grpc-daemon[1363]: time=2026-09-21T13:46:39.397Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"5975lb # [ 105.176433] niks3-server[1158]: 2026/09/21 13:46:40 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst5976lb # [ 105.186209] niks3-server[1158]: 2026/09/21 13:46:40 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5977lb # [ 105.195646] niks3-server[1158]: 2026/09/21 13:46:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5978lb # [ 105.201119] niks3-server[1158]: 2026/09/21 13:46:40 INFO Signed narinfos id=36 count=15979worker1 # [ 105.220578] nix-grpc-daemon[1363]: time=2026-09-21T13:46:39.447Z level=INFO msg="Uploading 1 narinfos"5980lb # [ 105.206387] niks3-server[1158]: 2026/09/21 13:46:40 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls5981lb # [ 105.214853] niks3-server[1158]: 2026/09/21 13:46:40 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5982lb # [ 105.222212] niks3-server[1158]: 2026/09/21 13:46:40 INFO Completed upload id=365983lb # [ 105.223501] niks3-server[1158]: 2026/09/21 13:46:40 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo5984worker1 # [ 105.241373] nix-grpc-daemon[1363]: time=2026-09-21T13:46:39.469Z level=INFO msg="Upload complete. (308ms)"5985worker1 # [ 105.251219] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59318 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=15986worker1 # [ 105.259631] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:39Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=05987worker1 # [ 105.263745] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:39Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=15988worker1 # [ 105.272445] nix-daemon[871]: reaped child process 1423, status = succeeded5989worker1 # [ 105.303930] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593185990worker1 # [ 105.317113] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593185991worker1 # [ 105.330265] nix-grpc-daemon[1363]: time=2026-09-21T13:46:39.558Z level=INFO msg="All 2 paths already cached"5992client # copying 1 paths to 'lb:50051'...5993client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...5994worker1 # [ 105.338751] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:593185995worker1 # [ 105.349181] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)5996worker1 # [ 105.532988] nix-daemon[871]: accepted connection from pid 1450, user nix-grpc-daemon (trusted)5997worker1 # [ 105.544716] nix-daemon[871]: reaped child process 1453, status = succeeded5998worker1 # [ 105.721661] nix-daemon[871]: accepted connection from pid 1455, user nix-grpc-daemon (trusted)5999worker1 # [ 105.733247] nix-daemon[871]: reaped child process 1458, status = succeeded6000lb # [ 105.733811] niks3-server[1158]: 2026/09/21 13:46:40 INFO Received uploads request method=POST path=/api/pending_closures6001worker1 # [ 105.761422] nix-grpc-daemon[1363]: time=2026-09-21T13:46:39.989Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6002worker1 # [ 105.763264] nix-grpc-daemon[1363]: time=2026-09-21T13:46:39.990Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6003lb # [ 105.763638] niks3-server[1158]: 2026/09/21 13:46:40 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6004lb # [ 105.773818] niks3-server[1158]: 2026/09/21 13:46:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6005lb # [ 105.779124] niks3-server[1158]: 2026/09/21 13:46:40 INFO Signed narinfos id=37 count=16006worker1 # [ 105.801063] nix-grpc-daemon[1363]: time=2026-09-21T13:46:40.028Z level=INFO msg="Uploading 1 narinfos"6007lb # [ 105.786159] niks3-server[1158]: 2026/09/21 13:46:40 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6008lb # [ 105.798688] niks3-server[1158]: 2026/09/21 13:46:40 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6009lb # [ 105.804412] niks3-server[1158]: 2026/09/21 13:46:40 INFO Completed upload id=376010worker1 # [ 105.823137] nix-grpc-daemon[1363]: time=2026-09-21T13:46:40.050Z level=INFO msg="Upload complete. (462ms)"6011lb # [ 105.808114] niks3-server[1158]: 2026/09/21 13:46:40 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6012worker1 # [ 105.827050] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59318 duration_s=0 paths=1 nar_bytes_in=7686013worker1 # [ 105.832278] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593186014worker1 # [ 105.842136] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)6015worker1 # [ 105.846114] nix-daemon[871]: reaped child process 1448, status = succeeded6016worker1 # [ 105.851403] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)6017client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6018client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6019client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6020client # farm-top> LOG-farm-top-on-standby6021worker1 # [ 107.078424] nix-daemon[871]: reaped child process 1462, status = succeeded6022worker1 # [ 107.255883] nix-daemon[871]: accepted connection from pid 1478, user nix-grpc-daemon (trusted)6023worker1 # [ 107.266564] nix-daemon[871]: reaped child process 1481, status = succeeded6024lb # [ 107.268978] niks3-server[1158]: 2026/09/21 13:46:42 INFO Received uploads request method=POST path=/api/pending_closures6025worker1 # [ 107.294981] nix-grpc-daemon[1363]: time=2026-09-21T13:46:41.522Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6026worker1 # [ 107.296565] nix-grpc-daemon[1363]: time=2026-09-21T13:46:41.524Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6027lb # [ 107.300664] niks3-server[1158]: 2026/09/21 13:46:42 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6028lb # [ 107.315180] niks3-server[1158]: 2026/09/21 13:46:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6029worker1 # [ 107.338816] nix-grpc-daemon[1363]: time=2026-09-21T13:46:41.566Z level=INFO msg="Uploading 1 narinfos"6030lb # [ 107.321185] niks3-server[1158]: 2026/09/21 13:46:42 INFO Signed narinfos id=38 count=16031lb # [ 107.331797] niks3-server[1158]: 2026/09/21 13:46:42 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6032lb # [ 107.336741] niks3-server[1158]: 2026/09/21 13:46:42 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6033lb # [ 107.344321] niks3-server[1158]: 2026/09/21 13:46:42 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6034lb # [ 107.348709] niks3-server[1158]: 2026/09/21 13:46:42 INFO Completed upload id=386035worker1 # [ 107.367674] nix-grpc-daemon[1363]: time=2026-09-21T13:46:41.595Z level=INFO msg="Upload complete. (307ms)"6036lb # [ 107.350679] niks3-server[1158]: 2026/09/21 13:46:42 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6037worker1 # [ 107.373559] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59318 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16038worker1 # [ 107.384748] nix-daemon[871]: reaped child process 1460, status = succeeded6039client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 7.35 seconds)6040client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath6041lb # [ 107.405122] envoy[914]: conn peer=[2001:db8:1::4]:46268 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=5880 tx=648 ms=147976042lb # [ 107.408964] envoy[914]: conn peer=[2001:db8:1::1]:50034 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9620 tx=4131 ms=70926043client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.21 seconds)6044worker2 # [ 107.671619] systemd[1]: Listening on Nix worker-protocol over gRPC.6045worker2 # [ 107.678111] systemd[1]: Starting scheduler-token.service...6046worker2 # [ 107.761865] systemd[1]: scheduler-token.service: Deactivated successfully.6047worker2 # [ 107.763050] systemd[1]: Finished scheduler-token.service.6048worker2 # [ 107.764548] systemd[1]: scheduler-token.service: Consumed 18ms CPU time over 84ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6049worker2 # [ 107.770250] systemd[1]: Starting Nix worker-protocol over gRPC...6050worker2 # [ 107.953531] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:42Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6051worker2 # [ 107.955412] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:42Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6052worker2 # [ 107.959413] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:42Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16053worker2 # [ 107.967063] nix-grpc-daemon[1272]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6054worker2 # [ 107.969935] nix-grpc-daemon[1272]: E0000 00:00:1789998402.847198 1272 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6055worker2 # [ 107.973865] nix-grpc-daemon[1272]: E0000 00:00:1789998402.850188 1272 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6056worker2 # [ 107.979136] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:42Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6057worker2 # [ 107.982816] systemd[1]: Started Nix worker-protocol over gRPC.6058lb: must succeed: curl -sf localhost:9901/clusters6059worker2 # [ 107.991082] nix-grpc-daemon[1272]: E0000 00:00:1789998402.870623 1313 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6060worker2 # [ 108.004190] nix-grpc-daemon[1272]: E0000 00:00:1789998402.884554 1313 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6061worker2 # [ 108.009054] nix-grpc-daemon[1272]: E0000 00:00:1789998402.886906 1313 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6062worker2 # [ 108.018316] nix-grpc-daemon[1272]: E0000 00:00:1789998402.898693 1313 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6063worker2 # [ 108.027576] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:42Z level=info event=scheduler_connected addr=lb:500516064worker1 # [ 108.069456] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:42Z level=info event=oidc_jwks provider=mock keys=16065worker1 # [ 108.072632] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:42Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:593106066worker1 # [ 108.076469] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:42Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06067lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6068worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6069worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6070lb: must succeed: curl -sf localhost:9901/clusters6071lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6072lb: must succeed: curl -sf localhost:9901/clusters6073lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6074worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6075worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6076(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.77 seconds)6077subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6078client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs6079client # [ 108.382994] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.6080client # Running as unit: rs.service; invocation ID: 8867b08e78f141b0823d97d7900983686081client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.07 seconds)6082worker2 # [ 108.597552] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:526806083worker2 # [ 108.609516] nix-daemon[856]: accepted connection from pid 1272, user nix-grpc-daemon (trusted)6084client # [ 108.622356] nix[1203]: don't know how to build these paths:6085client # [ 108.623288] nix[1203]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6086worker1 # [ 108.663577] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:42Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06087worker1 # [ 108.667504] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:42Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=16088worker1 # [ 108.706441] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593186089worker1 # [ 108.719469] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593186090client # [ 108.698257] nix[1203]: copying 1 paths to 'lb:50051'...6091worker1 # [ 108.731148] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:593186092client # [ 108.701150] nix[1203]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6093worker1 # [ 108.743902] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)6094worker1 # [ 108.934887] nix-daemon[871]: accepted connection from pid 1494, user nix-grpc-daemon (trusted)6095worker1 # [ 108.946742] nix-daemon[871]: reaped child process 1497, status = succeeded6096worker1 # [ 109.118944] nix-daemon[871]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6097worker1 # [ 109.130422] nix-daemon[871]: reaped child process 1502, status = succeeded6098lb # [ 109.129742] niks3-server[1158]: 2026/09/21 13:46:44 INFO Received uploads request method=POST path=/api/pending_closures6099worker1 # [ 109.159149] nix-grpc-daemon[1363]: time=2026-09-21T13:46:43.386Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6100worker1 # [ 109.161043] nix-grpc-daemon[1363]: time=2026-09-21T13:46:43.388Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6101lb # [ 109.161575] niks3-server[1158]: 2026/09/21 13:46:44 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6102lb # [ 109.175257] niks3-server[1158]: 2026/09/21 13:46:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6103worker1 # [ 109.201906] nix-grpc-daemon[1363]: time=2026-09-21T13:46:43.429Z level=INFO msg="Uploading 1 narinfos"6104lb # [ 109.183067] niks3-server[1158]: 2026/09/21 13:46:44 INFO Signed narinfos id=39 count=16105lb # [ 109.191812] niks3-server[1158]: 2026/09/21 13:46:44 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6106lb # [ 109.202501] niks3-server[1158]: 2026/09/21 13:46:44 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6107lb # [ 109.205803] niks3-server[1158]: 2026/09/21 13:46:44 INFO Completed upload id=396108lb # [ 109.207770] niks3-server[1158]: 2026/09/21 13:46:44 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6109worker1 # [ 109.225916] nix-grpc-daemon[1363]: time=2026-09-21T13:46:43.453Z level=INFO msg="Upload complete. (469ms)"6110worker1 # [ 109.229791] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59318 duration_s=0 paths=1 nar_bytes_in=4166111worker1 # [ 109.236347] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593186112worker1 # [ 109.243991] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)6113worker1 # [ 109.252931] nix-daemon[871]: reaped child process 1492, status = succeeded6114worker1 # [ 109.258359] nix-daemon[871]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)6115client # [ 109.226500] nix[1203]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6116lb: must succeed: curl -sf localhost:9901/clusters6117lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6118lb: must succeed: curl -sf localhost:9901/clusters6119lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6120lb: must succeed: curl -sf localhost:9901/clusters6121lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6122lb: must succeed: curl -sf localhost:9901/clusters6123lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6124worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6125worker2 # [ 111.337645] systemd-journald[288]: Received client request to rotate journal, rotating.6126worker2 # [ 111.353810] systemd-journald[288]: Vacuuming done, freed 0B of archived journals from /var/log/journal/aa3665dbd8a6455c83a7fe91eb8c8e8c.6127worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.07 seconds)6128worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service6129worker1 # [ 111.300513] systemd[1]: Stopping Nix worker-protocol over gRPC...6130worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)6131client: must succeed: sleep 5.06132worker1 # [ 112.194426] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:46Z level=info event=draining6133client: (finished: must succeed: sleep 5.0, in 5.05 seconds)6134worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6135worker1 # [ 116.684430] nix-daemon[871]: reaped child process 1506, status = succeeded6136worker1 # [ 116.852874] nix-daemon[871]: accepted connection from pid 1547, user nix-grpc-daemon (trusted)6137worker1 # [ 116.864541] nix-daemon[871]: reaped child process 1550, status = succeeded6138lb # [ 116.864405] niks3-server[1158]: 2026/09/21 13:46:51 INFO Received uploads request method=POST path=/api/pending_closures6139worker1 # [ 116.891751] nix-grpc-daemon[1363]: time=2026-09-21T13:46:51.119Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6140worker1 # [ 116.893272] nix-grpc-daemon[1363]: time=2026-09-21T13:46:51.121Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6141lb # [ 116.912982] niks3-server[1158]: 2026/09/21 13:46:51 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6142lb # [ 116.924172] niks3-server[1158]: 2026/09/21 13:46:51 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6143lb # [ 116.930095] niks3-server[1158]: 2026/09/21 13:46:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6144lb # [ 116.934279] niks3-server[1158]: 2026/09/21 13:46:51 INFO Signed narinfos id=40 count=16145worker1 # [ 116.953257] nix-grpc-daemon[1363]: time=2026-09-21T13:46:51.180Z level=INFO msg="Uploading 1 narinfos"6146lb # [ 116.939226] niks3-server[1158]: 2026/09/21 13:46:51 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6147lb # [ 116.948068] niks3-server[1158]: 2026/09/21 13:46:51 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6148lb # [ 116.951299] niks3-server[1158]: 2026/09/21 13:46:51 INFO Completed upload id=406149lb # [ 116.953808] niks3-server[1158]: 2026/09/21 13:46:51 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6150worker1 # [ 116.970405] nix-grpc-daemon[1363]: time=2026-09-21T13:46:51.198Z level=INFO msg="Upload complete. (299ms)"6151worker1 # [ 116.979181] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59318 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16152worker1 # [ 116.991311] nix-daemon[871]: reaped child process 1504, status = succeeded6153client # [ 116.980325] systemd[1]: rs.service: Deactivated successfully.6154client # [ 116.984179] systemd[1]: rs.service: Consumed 128ms CPU time over 8.599s wall clock time, 43.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6155worker1 # [ 117.195717] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:51Z level=info event=signal_exit6156lb # [ 117.406709] envoy[914]: conn peer=[2001:db8:1::1]:44920 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2960 tx=1404 ms=83636157lb # [ 117.716301] niks3-server[1158]: 2026/09/21 13:46:52 INFO lead: released remote=192.168.1.3:510226158lb # [ 118.002860] niks3-server[1158]: 2026/09/21 13:46:52 INFO lead: acquired remote=192.168.1.4:510166159worker2 # [ 117.989976] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:52Z level=info event=scheduler_take_over6160worker2 # [ 118.705589] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:53Z level=info event=scheduler_restarting addr=lb:500516161worker1 # [ 118.741698] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:52Z level=info event=scheduler_restarting6162worker1 # [ 118.744896] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:52Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:593106163worker1 # [ 118.748721] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:52Z level=info event=worker_gone addr=192.168.1.4:500516164worker1 # [ 118.751874] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:52Z level=info event=scheduler_restarting addr=lb:500516165worker1 # [ 118.755124] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:52Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:593106166worker1 # [ 118.758825] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:52Z level=info event=worker_gone addr=192.168.1.3:500516167worker1 # [ 118.764884] nix-grpc-daemon[1287]: E0000 00:00:1789998412.992568 1332 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true6168worker1 # [ 118.769634] nix-grpc-daemon[1287]: E0000 00:00:1789998412.992780 1332 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6169worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.17 seconds)6170client: waiting for success: ! systemctl is-active rs6171worker1 # [ 118.786609] nix-daemon[871]: reaped child process 1353, status = succeeded6172client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)6173client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6174client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)6175client: must fail: journalctl -u rs | grep -q 'reconnecting'6176client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.02 seconds)6177lb: must succeed: curl -sf localhost:9901/clusters6178worker2 # [ 118.906972] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:53Z level=info event=scheduler_connected addr=lb:500516179worker2 # [ 118.911149] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6180worker1 # [ 118.952296] nix-grpc-daemon[1287]: ts=2026-09-21T13:46:53Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6181lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6182worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6183worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6184worker2 # [ 119.113096] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:53Z level=info event=scheduler_connected addr=lb:500516185worker2 # [ 119.126139] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:54Z level=info event=oidc_jwks provider=mock keys=16186worker2 # [ 119.130399] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:54Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:372526187worker2 # [ 119.134314] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:54Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06188lb: must succeed: curl -sf localhost:9901/clusters6189lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6190worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6191worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6192worker1 # [ 120.210910] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6193worker1 # [ 120.214585] systemd[1]: Stopped Nix worker-protocol over gRPC.6194worker1 # [ 120.217335] systemd[1]: nix-grpc-daemon.service: Consumed 1.322s CPU time over 33.216s wall clock time, 35.3M memory peak, 47.4K incoming IP traffic, 91.9K outgoing IP traffic.6195worker1 # [ 120.233148] systemd[1]: Starting Nix worker-protocol over gRPC...6196worker1 # [ 120.404760] nix-grpc-daemon[1557]: ts=2026-09-21T13:46:54Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6197worker1 # [ 120.406477] nix-grpc-daemon[1557]: ts=2026-09-21T13:46:54Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6198worker1 # [ 120.410391] nix-grpc-daemon[1557]: ts=2026-09-21T13:46:54Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16199worker1 # [ 120.418066] nix-grpc-daemon[1557]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6200worker1 # [ 120.419463] nix-grpc-daemon[1557]: E0000 00:00:1789998414.645111 1557 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6201worker1 # [ 120.424075] nix-grpc-daemon[1557]: E0000 00:00:1789998414.650290 1557 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6202worker1 # [ 120.426680] nix-grpc-daemon[1557]: ts=2026-09-21T13:46:54Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6203worker1 # [ 120.428811] systemd[1]: Started Nix worker-protocol over gRPC.6204worker1 # [ 120.436068] nix-grpc-daemon[1557]: E0000 00:00:1789998414.662875 1599 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6205worker1 # [ 120.446808] nix-grpc-daemon[1557]: E0000 00:00:1789998414.674724 1599 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6206worker1 # [ 120.451054] nix-grpc-daemon[1557]: E0000 00:00:1789998414.676967 1599 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6207worker1 # [ 120.460855] nix-grpc-daemon[1557]: E0000 00:00:1789998414.688729 1599 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6208worker1 # [ 120.470087] nix-grpc-daemon[1557]: ts=2026-09-21T13:46:54Z level=info event=scheduler_connected addr=lb:500516209worker2 # [ 120.440880] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:55Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:372526210worker2 # [ 120.445228] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:55Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06211lb: must succeed: curl -sf localhost:9901/clusters6212lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6213worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6214worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6215(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.98 seconds)6216subtest: scheduler restart mid-build: build finishes, no double build6217client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid6218client # [ 121.358316] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.6219client # Running as unit: mid.service; invocation ID: ffa50c9614264db583d1d8f6df9238ff6220client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)6221worker1 # [ 121.601557] nix-grpc-daemon[1557]: ts=2026-09-21T13:46:55Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:481646222worker1 # [ 121.608631] nix-daemon[871]: accepted connection from pid 1557, user nix-grpc-daemon (trusted)6223client # [ 121.583599] nix[1239]: don't know how to build these paths:6224client # [ 121.584688] nix[1239]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6225worker2 # [ 121.592861] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:56Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06226worker2 # [ 121.598483] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:56Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=16227worker2 # [ 121.642603] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:526806228worker2 # [ 121.654447] nix-grpc-daemon[1272]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 38 ms (attempt 1/5)6229worker2 # [ 121.698366] nix-grpc-daemon[1272]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 2/5)6230worker2 # [ 121.756230] nix-grpc-daemon[1272]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 58 ms (attempt 3/5)6231worker2 # [ 121.820224] nix-grpc-daemon[1272]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 515 ms (attempt 4/5)6232worker2 # [ 122.341262] nix-grpc-daemon[1272]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6233worker2 # [ 122.342530] nix-grpc-daemon[1272]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6234worker2 # [ 122.353836] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:526806235client # [ 122.358892] nix[1239]: copying 1 paths to 'lb:50051'...6236worker2 # [ 122.361848] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:526806237client # [ 122.362130] nix[1239]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6238worker2 # [ 122.368818] nix-daemon[856]: accepted connection from pid 1272, user nix-grpc-daemon (trusted)6239worker2 # [ 122.380194] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:57Z level=info event=niks3_push_started pid=13976240worker2 # [ 122.561458] nix-grpc-daemon[1397]: time=2026-09-21T13:46:57.441Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6241worker2 # [ 122.725315] nix-daemon[856]: accepted connection from pid 1413, user nix-grpc-daemon (trusted)6242worker2 # [ 122.739291] nix-daemon[856]: reaped child process 1417, status = succeeded6243worker2 # [ 122.904661] nix-daemon[856]: accepted connection from pid 1419, user nix-grpc-daemon (trusted)6244worker2 # [ 122.918272] nix-daemon[856]: reaped child process 1422, status = succeeded6245lb # [ 122.947868] niks3-server[1158]: 2026/09/21 13:46:57 INFO Received uploads request method=POST path=/api/pending_closures6246worker2 # [ 122.946973] nix-grpc-daemon[1397]: time=2026-09-21T13:46:57.827Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6247worker2 # [ 122.948814] nix-grpc-daemon[1397]: time=2026-09-21T13:46:57.828Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6248lb # [ 122.989729] niks3-server[1158]: 2026/09/21 13:46:57 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6249lb # [ 122.996099] niks3-server[1158]: 2026/09/21 13:46:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6250lb # [ 123.004576] niks3-server[1158]: 2026/09/21 13:46:57 INFO Signed narinfos id=41 count=16251worker2 # [ 122.995822] nix-grpc-daemon[1397]: time=2026-09-21T13:46:57.875Z level=INFO msg="Uploading 1 narinfos"6252lb # [ 123.018354] niks3-server[1158]: 2026/09/21 13:46:57 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6253lb # [ 123.022194] niks3-server[1158]: 2026/09/21 13:46:57 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6254lb # [ 123.026656] niks3-server[1158]: 2026/09/21 13:46:57 INFO Completed upload id=416255worker2 # [ 123.013779] nix-grpc-daemon[1397]: time=2026-09-21T13:46:57.894Z level=INFO msg="Upload complete. (448ms)"6256worker2 # [ 123.017976] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52680 duration_s=0 paths=1 nar_bytes_in=4246257worker2 # [ 123.023423] nix-grpc-daemon[1272]: ts=2026-09-21T13:46:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:526806258lb # [ 123.041156] niks3-server[1158]: 2026/09/21 13:46:57 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6259worker2 # [ 123.033029] nix-daemon[856]: accepted connection from pid 1272, user nix-grpc-daemon (trusted)6260worker2 # [ 123.037614] nix-daemon[856]: reaped child process 1395, status = succeeded6261worker2 # [ 123.042634] nix-daemon[856]: accepted connection from pid 1272, user nix-grpc-daemon (trusted)6262client # [ 123.043376] nix[1239]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6263lb: must succeed: curl -sf localhost:9901/clusters6264lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6265lb: must succeed: curl -sf localhost:9901/clusters6266lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6267worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6268worker2 # [ 124.999454] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1272 (nix-grpc-daemon) on client request.6269worker2 # [ 125.002420] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6270lb # [ 125.023502] niks3-server[1158]: 2026/09/21 13:46:59 INFO lead: released remote=192.168.1.4:510166271client # [ 125.015420] nix[1239]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6272worker1 # [ 125.046448] nix-grpc-daemon[1557]: ts=2026-09-21T13:46:59Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6273client # [ 125.017366] nix[1239]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6274worker2 # [ 125.022797] systemd[1]: Starting scheduler-token.service...6275worker2 # [ 125.034086] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6276worker2 # [ 125.035927] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6277worker2 # [ 125.037599] systemd[1]: nix-grpc-daemon.service: Consumed 460ms CPU time over 17.260s wall clock time, 33.7M memory peak, 16K incoming IP traffic, 30.9K outgoing IP traffic.6278worker2 # [ 125.054284] nix-daemon[856]: reaped child process 1424, status = succeeded6279worker2 # [ 125.055807] nix-daemon[856]: reaped child process 1331, status = succeeded6280worker2 # [ 125.066288] nix-daemon[1426]: unexpected Nix daemon error: error: interrupted by the user6281worker2 # [ 125.073531] nix-daemon[856]: reaped child process 1426, status = failed with exit code 16282worker2 # [ 125.112942] systemd[1]: scheduler-token.service: Deactivated successfully.6283worker2 # [ 125.114274] systemd[1]: Finished scheduler-token.service.6284worker2 # [ 125.116557] systemd[1]: scheduler-token.service: Consumed 20ms CPU time over 89ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6285worker2 # [ 125.118778] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6286worker2 # [ 125.123378] systemd[1]: Starting scheduler-token.service...6287worker2 # [ 125.171100] systemd[1]: scheduler-token.service: Deactivated successfully.6288worker2 # [ 125.172676] systemd[1]: Finished scheduler-token.service.6289worker2 # [ 125.174399] systemd[1]: scheduler-token.service: Consumed 19ms CPU time over 49ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6290worker2 # [ 125.179670] systemd[1]: Starting Nix worker-protocol over gRPC...6291worker1 # [ 125.246830] nix-grpc-daemon[1557]: ts=2026-09-21T13:46:59Z level=info event=scheduler_connected addr=lb:500516292worker2 # [ 125.306737] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6293worker2 # [ 125.308846] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6294worker2 # [ 125.312498] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16295worker2 # [ 125.318067] nix-grpc-daemon[1469]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6296worker2 # [ 125.319373] nix-grpc-daemon[1469]: E0000 00:00:1789998420.198027 1469 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6297worker2 # [ 125.324067] nix-grpc-daemon[1469]: E0000 00:00:1789998420.203143 1469 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6298worker2 # [ 125.326158] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6299worker2 # [ 125.332584] systemd[1]: Started Nix worker-protocol over gRPC.6300lb # [ 125.351622] niks3-server[1158]: 2026/09/21 13:47:00 INFO lead: acquired remote=192.168.1.4:341306301worker2 # [ 125.340088] nix-grpc-daemon[1469]: E0000 00:00:1789998420.218566 1510 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6302worker2 # [ 125.344543] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=scheduler_take_over6303worker2 # [ 125.347073] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:483326304worker2 # [ 125.349179] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06305worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.40 seconds)6306client: must succeed: sleep 5.06307worker2 # [ 125.361041] nix-grpc-daemon[1469]: E0000 00:00:1789998420.240359 1510 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6308worker2 # [ 125.364125] nix-grpc-daemon[1469]: E0000 00:00:1789998420.240550 1510 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6309worker2 # [ 125.374119] nix-grpc-daemon[1469]: E0000 00:00:1789998420.254474 1510 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6310worker2 # [ 125.381488] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=scheduler_connected addr=lb:500516311worker2 # [ 125.383115] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6312client # [ 125.519368] nix[1239]: scheduler at lb:50051: no healthy upstream, reconnecting6313worker2 # [ 125.583808] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=scheduler_connected addr=lb:500516314worker2 # [ 125.585572] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6315worker2 # [ 125.786348] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=scheduler_connected addr=lb:500516316worker2 # [ 125.788187] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6317worker2 # [ 125.988904] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=scheduler_connected addr=lb:500516318worker2 # [ 125.990711] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6319worker2 # [ 126.192101] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:01Z level=info event=scheduler_connected addr=lb:500516320worker2 # [ 126.195827] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6321worker2 # [ 126.397495] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:01Z level=info event=scheduler_connected addr=lb:500516322worker2 # [ 126.410153] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:01Z level=info event=oidc_jwks provider=mock keys=16323worker2 # [ 126.413102] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:01Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:483446324worker2 # [ 126.417464] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:01Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06325worker2 # [ 126.528991] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:01Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06326worker2 # [ 126.533612] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:01Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16327worker1 # [ 126.568850] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:481646328worker1 # [ 126.581952] nix-grpc-daemon[1557]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 47 ms (attempt 1/5)6329worker1 # [ 126.635231] nix-grpc-daemon[1557]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 107 ms (attempt 2/5)6330worker1 # [ 126.748116] nix-grpc-daemon[1557]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 281 ms (attempt 3/5)6331worker1 # [ 127.029455] nix-grpc-daemon[1557]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 505 ms (attempt 4/5)6332lb # [ 127.407410] envoy[914]: conn peer=[2001:db8:1::3]:51374 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=3387 tx=915 ms=317166333lb # [ 127.408962] envoy[914]: conn peer=[2001:db8:1::4]:41946 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=2871 tx=376 ms=169916334worker1 # [ 127.534896] nix-grpc-daemon[1557]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6335worker1 # [ 127.536186] nix-grpc-daemon[1557]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6336worker1 # [ 127.545147] nix-grpc-daemon[1557]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' from 'http://lb:5751'...6337worker1 # [ 127.560850] nix-daemon[871]: accepted connection from pid 1557, user nix-grpc-daemon (trusted)6338worker1 # [ 127.566784] nix-daemon[871]: accepted connection from pid 1557, user nix-grpc-daemon (trusted)6339client # [ 127.535847] nix[1239]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6340client: (finished: must succeed: sleep 5.0, in 5.02 seconds)6341worker1 # [ 130.577702] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:04Z level=info event=niks3_push_started pid=16636342worker1 # [ 130.590233] nix-daemon[871]: reaped child process 1638, status = succeeded6343client: waiting for success: ! systemctl is-active mid6344worker1 # [ 130.732303] nix-grpc-daemon[1663]: time=2026-09-21T13:47:04.958Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6345worker1 # [ 130.867893] nix-daemon[871]: accepted connection from pid 1676, user nix-grpc-daemon (trusted)6346worker1 # [ 130.881505] nix-daemon[871]: reaped child process 1679, status = succeeded6347lb # [ 130.873505] niks3-server[1158]: 2026/09/21 13:47:05 INFO Received uploads request method=POST path=/api/pending_closures6348worker1 # [ 130.899075] nix-grpc-daemon[1663]: time=2026-09-21T13:47:05.125Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6349worker1 # [ 130.900428] nix-grpc-daemon[1663]: time=2026-09-21T13:47:05.125Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6350lb # [ 130.902260] niks3-server[1158]: 2026/09/21 13:47:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6351lb # [ 130.904900] niks3-server[1158]: 2026/09/21 13:47:05 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6352lb # [ 130.908815] niks3-server[1158]: 2026/09/21 13:47:05 INFO Signed narinfos id=42 count=16353worker1 # [ 130.927739] nix-grpc-daemon[1663]: time=2026-09-21T13:47:05.154Z level=INFO msg="Uploading 1 narinfos"6354lb # [ 130.920181] niks3-server[1158]: 2026/09/21 13:47:05 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6355lb # [ 130.926330] niks3-server[1158]: 2026/09/21 13:47:05 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6356lb # [ 130.929786] niks3-server[1158]: 2026/09/21 13:47:05 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6357lb # [ 130.932084] niks3-server[1158]: 2026/09/21 13:47:05 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6358lb # [ 130.935483] niks3-server[1158]: 2026/09/21 13:47:05 INFO Completed upload id=426359worker1 # [ 130.954439] nix-grpc-daemon[1663]: time=2026-09-21T13:47:05.182Z level=INFO msg="Upload complete. (219ms)"6360worker1 # [ 130.958091] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48164 duration_s=4 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16361worker1 # [ 130.966727] nix-daemon[871]: reaped child process 1636, status = succeeded6362client # [ 130.950480] systemd[1]: mid.service: Deactivated successfully.6363client # [ 130.952091] systemd[1]: mid.service: Consumed 128ms CPU time over 9.593s wall clock time, 41.4M memory peak, 8.8K incoming IP traffic, 10.9K outgoing IP traffic.6364client: (finished: waiting for success: ! systemctl is-active mid, in 1.06 seconds)6365client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6366client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.02 seconds)6367(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.34 seconds)6368subtest: interrupting the client stops the build on the worker6369client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6370client # [ 131.671827] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6371client # Running as unit: intr.service; invocation ID: f4f25e82616e4c96b2c13581c7619f186372client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.03 seconds)6373client # [ 131.836979] nix[1275]: don't know how to build these paths:6374worker1 # [ 131.867476] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:481646375client # [ 131.837933] nix[1275]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6376worker2 # [ 131.844257] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:06Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06377worker2 # [ 131.846200] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:06Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16378worker2 # [ 131.888669] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483566379worker2 # [ 131.894794] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6380worker2 # [ 131.903900] nix-grpc-daemon[1469]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 48 ms (attempt 1/5)6381worker2 # [ 131.954444] nix-grpc-daemon[1469]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 173 ms (attempt 2/5)6382worker2 # [ 132.130696] nix-grpc-daemon[1469]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 203 ms (attempt 3/5)6383worker2 # [ 132.339717] nix-grpc-daemon[1469]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 406 ms (attempt 4/5)6384worker2 # [ 132.748834] nix-grpc-daemon[1469]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6385worker2 # [ 132.750803] nix-grpc-daemon[1469]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6386worker2 # [ 132.760642] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:483566387client # [ 132.767098] nix[1275]: copying 1 paths to 'lb:50051'...6388client # [ 132.768213] nix[1275]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6389worker2 # [ 132.767505] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:483566390worker2 # [ 132.773942] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6391worker2 # [ 132.786928] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:07Z level=info event=niks3_push_started pid=15476392worker2 # [ 132.951235] nix-grpc-daemon[1547]: time=2026-09-21T13:47:07.831Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6393worker2 # [ 133.098285] nix-daemon[856]: accepted connection from pid 1562, user nix-grpc-daemon (trusted)6394worker2 # [ 133.116366] nix-daemon[856]: reaped child process 1565, status = succeeded6395worker2 # [ 133.246905] nix-daemon[856]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6396worker2 # [ 133.262964] nix-daemon[856]: reaped child process 1570, status = succeeded6397lb # [ 133.288826] niks3-server[1158]: 2026/09/21 13:47:08 INFO Received uploads request method=POST path=/api/pending_closures6398worker2 # [ 133.282736] nix-grpc-daemon[1547]: time=2026-09-21T13:47:08.162Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6399worker2 # [ 133.284116] nix-grpc-daemon[1547]: time=2026-09-21T13:47:08.162Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6400lb # [ 133.312194] niks3-server[1158]: 2026/09/21 13:47:08 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6401lb # [ 133.315821] niks3-server[1158]: 2026/09/21 13:47:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6402lb # [ 133.318319] niks3-server[1158]: 2026/09/21 13:47:08 INFO Signed narinfos id=43 count=16403worker2 # [ 133.307204] nix-grpc-daemon[1547]: time=2026-09-21T13:47:08.187Z level=INFO msg="Uploading 1 narinfos"6404lb # [ 133.323694] niks3-server[1158]: 2026/09/21 13:47:08 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6405lb # [ 133.330258] niks3-server[1158]: 2026/09/21 13:47:08 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6406lb # [ 133.337793] niks3-server[1158]: 2026/09/21 13:47:08 INFO Completed upload id=436407worker2 # [ 133.324697] nix-grpc-daemon[1547]: time=2026-09-21T13:47:08.205Z level=INFO msg="Upload complete. (369ms)"6408lb # [ 133.339947] niks3-server[1158]: 2026/09/21 13:47:08 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6409worker2 # [ 133.327617] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=0 paths=1 nar_bytes_in=4246410worker2 # [ 133.331389] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483566411worker2 # [ 133.339308] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6412worker2 # [ 133.342764] nix-daemon[856]: reaped child process 1543, status = succeeded6413client # [ 133.349087] nix[1275]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6414worker2 # [ 133.348999] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6415client: must succeed: systemctl kill -s INT intr6416client # [ 135.056491] systemd[1]: intr.service: Sent signal SIGINT to main process 1275 (nix) on client request.6417client # [ 135.061834] nix[1275]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6418worker2 # [ 135.062575] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:09Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6419client: (finished: must succeed: systemctl kill -s INT intr, in 0.04 seconds)6420client # [ 135.067549] nix[1275]: error: interrupted by the user6421client # [ 135.087629] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6422client # [ 135.089341] systemd[1]: intr.service: Failed with result 'exit-code'.6423client # [ 135.091108] systemd[1]: intr.service: Consumed 114ms CPU time over 3.417s wall clock time, 39.4M memory peak, 6.4K incoming IP traffic, 8.2K outgoing IP traffic.6424worker2 # [ 135.178765] nix-daemon[856]: reaped child process 1572, status = succeeded6425worker2 # [ 135.182282] nix-daemon[1574]: unexpected Nix daemon error: error: interrupted by the user6426worker2 # [ 135.188889] nix-daemon[856]: reaped child process 1574, status = failed with exit code 16427(finished: subtest: interrupting the client stops the build on the worker, in 3.54 seconds)6428subtest: a deploy mid-build drains: the build finishes, the new generation takes over6429client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6430client # [ 135.220616] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6431client # Running as unit: sw.service; invocation ID: d5b73cac0a6e4ef0b76b7f08d8261ec86432client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.04 seconds)6433worker2 # [ 135.416193] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:483566434client # [ 135.424104] nix[1296]: don't know how to build these paths:6435client # [ 135.425175] nix[1296]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6436worker2 # [ 135.433158] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:10Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06437worker2 # [ 135.437666] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:10Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16438worker1 # [ 135.471993] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:481646439worker1 # [ 135.477083] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:481646440client # [ 135.450786] nix[1296]: copying 1 paths to 'lb:50051'...6441worker1 # [ 135.482454] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:481646442client # [ 135.451567] nix[1296]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6443worker1 # [ 135.487888] nix-daemon[871]: accepted connection from pid 1557, user nix-grpc-daemon (trusted)6444worker1 # [ 135.634211] nix-daemon[871]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6445worker1 # [ 135.646277] nix-daemon[871]: reaped child process 1710, status = succeeded6446worker1 # [ 135.775778] nix-daemon[871]: accepted connection from pid 1712, user nix-grpc-daemon (trusted)6447worker1 # [ 135.789234] nix-daemon[871]: reaped child process 1715, status = succeeded6448lb # [ 135.786539] niks3-server[1158]: 2026/09/21 13:47:10 INFO Received uploads request method=POST path=/api/pending_closures6449worker1 # [ 135.812404] nix-grpc-daemon[1663]: time=2026-09-21T13:47:10.039Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6450worker1 # [ 135.813834] nix-grpc-daemon[1663]: time=2026-09-21T13:47:10.039Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6451lb # [ 135.802028] niks3-server[1158]: 2026/09/21 13:47:10 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6452lb # [ 135.806307] niks3-server[1158]: 2026/09/21 13:47:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6453worker1 # [ 135.828323] nix-grpc-daemon[1663]: time=2026-09-21T13:47:10.055Z level=INFO msg="Uploading 1 narinfos"6454lb # [ 135.809477] niks3-server[1158]: 2026/09/21 13:47:10 INFO Signed narinfos id=44 count=16455lb # [ 135.817593] niks3-server[1158]: 2026/09/21 13:47:10 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6456lb # [ 135.822258] niks3-server[1158]: 2026/09/21 13:47:10 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6457lb # [ 135.826490] niks3-server[1158]: 2026/09/21 13:47:10 INFO Completed upload id=446458worker1 # [ 135.845168] nix-grpc-daemon[1663]: time=2026-09-21T13:47:10.073Z level=INFO msg="Upload complete. (352ms)"6459lb # [ 135.828109] niks3-server[1158]: 2026/09/21 13:47:10 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6460worker1 # [ 135.847228] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48164 duration_s=0 paths=1 nar_bytes_in=4166461worker1 # [ 135.853045] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:481646462worker1 # [ 135.858206] nix-daemon[871]: accepted connection from pid 1557, user nix-grpc-daemon (trusted)6463worker1 # [ 135.860584] nix-daemon[871]: reaped child process 1705, status = succeeded6464worker1 # [ 135.866148] nix-daemon[871]: accepted connection from pid 1557, user nix-grpc-daemon (trusted)6465client # [ 135.833966] nix[1296]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6466lb # [ 137.409232] envoy[914]: conn peer=[2001:db8:1::1]:33922 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4119 tx=1924 ms=93706467lb # [ 137.413159] envoy[914]: conn peer=[2001:db8:1::1]:36760 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2967 tx=828 ms=32436468worker1: must succeed: systemctl reload nix-grpc-daemon.service6469worker1 # [ 137.744688] systemd[1]: Reloading Nix worker-protocol over gRPC...6470worker1 # [ 137.779246] systemd[1]: Reloaded Nix worker-protocol over gRPC.6471worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.07 seconds)6472worker1: must succeed: pgrep -f 'read -t [9]0 x'6473worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.04 seconds)6474client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26475client # don't know how to build these paths:6476client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6477worker2 # [ 137.992099] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:483566478worker2 # [ 137.997283] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:483566479worker2 # [ 138.020763] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:12Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06480worker2 # [ 138.025912] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:12Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16481worker2 # [ 138.065757] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483566482worker2 # [ 138.082810] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:483566483client # copying 1 paths to 'lb:50051'...6484worker2 # [ 138.095132] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:483566485client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6486worker2 # [ 138.117914] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6487worker2 # [ 138.298202] nix-daemon[856]: accepted connection from pid 1625, user nix-grpc-daemon (trusted)6488worker2 # [ 138.312748] nix-daemon[856]: reaped child process 1628, status = succeeded6489worker1 # [ 138.430383] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:12Z level=info event=draining6490worker2 # [ 138.481525] nix-daemon[856]: accepted connection from pid 1630, user nix-grpc-daemon (trusted)6491worker2 # [ 138.495848] nix-daemon[856]: reaped child process 1633, status = succeeded6492lb # [ 138.524181] niks3-server[1158]: 2026/09/21 13:47:13 INFO Received uploads request method=POST path=/api/pending_closures6493worker2 # [ 138.520248] nix-grpc-daemon[1547]: time=2026-09-21T13:47:13.400Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6494worker2 # [ 138.522404] nix-grpc-daemon[1547]: time=2026-09-21T13:47:13.402Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6495lb # [ 138.547902] niks3-server[1158]: 2026/09/21 13:47:13 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6496lb # [ 138.550657] niks3-server[1158]: 2026/09/21 13:47:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6497lb # [ 138.552452] niks3-server[1158]: 2026/09/21 13:47:13 INFO Signed narinfos id=45 count=16498worker2 # [ 138.540836] nix-grpc-daemon[1547]: time=2026-09-21T13:47:13.421Z level=INFO msg="Uploading 1 narinfos"6499lb # [ 138.558077] niks3-server[1158]: 2026/09/21 13:47:13 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6500lb # [ 138.562897] niks3-server[1158]: 2026/09/21 13:47:13 INFO Completed upload id=456501worker2 # [ 138.549663] nix-grpc-daemon[1547]: time=2026-09-21T13:47:13.430Z level=INFO msg="Upload complete. (420ms)"6502lb # [ 138.564840] niks3-server[1158]: 2026/09/21 13:47:13 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6503worker2 # [ 138.552046] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=0 paths=1 nar_bytes_in=5286504lb # [ 138.567292] niks3-server[1158]: 2026/09/21 13:47:13 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6505worker2 # [ 138.556599] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483566506worker2 # [ 138.563309] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6507worker2 # [ 138.565786] nix-daemon[856]: reaped child process 1623, status = succeeded6508worker2 # [ 138.571124] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6509client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6510client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6511client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6512client # farm-a> LOG-farm-a-during-drain6513worker2 # [ 139.786277] nix-daemon[856]: reaped child process 1637, status = succeeded6514worker2 # [ 139.948224] nix-daemon[856]: accepted connection from pid 1653, user nix-grpc-daemon (trusted)6515worker2 # [ 139.964507] nix-daemon[856]: reaped child process 1656, status = succeeded6516lb # [ 139.993694] niks3-server[1158]: 2026/09/21 13:47:14 INFO Received uploads request method=POST path=/api/pending_closures6517worker2 # [ 139.992361] nix-grpc-daemon[1547]: time=2026-09-21T13:47:14.871Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6518worker2 # [ 139.993830] nix-grpc-daemon[1547]: time=2026-09-21T13:47:14.871Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6519lb # [ 140.036201] niks3-server[1158]: 2026/09/21 13:47:14 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6520lb # [ 140.042495] niks3-server[1158]: 2026/09/21 13:47:14 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6521lb # [ 140.053235] niks3-server[1158]: 2026/09/21 13:47:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6522lb # [ 140.058900] niks3-server[1158]: 2026/09/21 13:47:14 INFO Signed narinfos id=46 count=16523worker2 # [ 140.046373] nix-grpc-daemon[1547]: time=2026-09-21T13:47:14.926Z level=INFO msg="Uploading 1 narinfos"6524lb # [ 140.062722] niks3-server[1158]: 2026/09/21 13:47:14 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6525lb # [ 140.071179] niks3-server[1158]: 2026/09/21 13:47:14 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6526lb # [ 140.074333] niks3-server[1158]: 2026/09/21 13:47:14 INFO Completed upload id=466527worker2 # [ 140.061384] nix-grpc-daemon[1547]: time=2026-09-21T13:47:14.941Z level=INFO msg="Upload complete. (283ms)"6528lb # [ 140.077180] niks3-server[1158]: 2026/09/21 13:47:14 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6529worker2 # [ 140.068385] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16530worker2 # [ 140.077934] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:14Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06531worker2 # [ 140.081307] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:14Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16532worker2 # [ 140.086173] nix-daemon[856]: reaped child process 1635, status = succeeded6533worker2 # [ 140.122730] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483566534worker2 # [ 140.136894] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:483566535client # copying 1 paths to 'lb:50051'...6536worker2 # [ 140.151649] nix-grpc-daemon[1547]: time=2026-09-21T13:47:15.031Z level=INFO msg="All 1 paths already cached"6537client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6538worker2 # [ 140.157445] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:483566539worker2 # [ 140.171190] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6540worker2 # [ 140.351899] nix-daemon[856]: accepted connection from pid 1662, user nix-grpc-daemon (trusted)6541worker2 # [ 140.366085] nix-daemon[856]: reaped child process 1665, status = succeeded6542worker2 # [ 140.534458] nix-daemon[856]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6543worker2 # [ 140.548257] nix-daemon[856]: reaped child process 1670, status = succeeded6544lb # [ 140.577269] niks3-server[1158]: 2026/09/21 13:47:15 INFO Received uploads request method=POST path=/api/pending_closures6545worker2 # [ 140.575477] nix-grpc-daemon[1547]: time=2026-09-21T13:47:15.455Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6546worker2 # [ 140.576937] nix-grpc-daemon[1547]: time=2026-09-21T13:47:15.455Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6547lb # [ 140.607922] niks3-server[1158]: 2026/09/21 13:47:15 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6548lb # [ 140.619239] niks3-server[1158]: 2026/09/21 13:47:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6549worker2 # [ 140.611332] nix-grpc-daemon[1547]: time=2026-09-21T13:47:15.491Z level=INFO msg="Uploading 1 narinfos"6550lb # [ 140.624676] niks3-server[1158]: 2026/09/21 13:47:15 INFO Signed narinfos id=47 count=16551lb # [ 140.631474] niks3-server[1158]: 2026/09/21 13:47:15 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6552lb # [ 140.639342] niks3-server[1158]: 2026/09/21 13:47:15 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6553lb # [ 140.646942] niks3-server[1158]: 2026/09/21 13:47:15 INFO Completed upload id=476554worker2 # [ 140.636308] nix-grpc-daemon[1547]: time=2026-09-21T13:47:15.516Z level=INFO msg="Upload complete. (452ms)"6555lb # [ 140.650059] niks3-server[1158]: 2026/09/21 13:47:15 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6556worker2 # [ 140.640451] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=0 paths=1 nar_bytes_in=6406557worker2 # [ 140.645294] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483566558worker2 # [ 140.657286] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6559worker2 # [ 140.660580] nix-daemon[856]: reaped child process 1660, status = succeeded6560worker2 # [ 140.666177] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6561client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6562client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6563client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6564client # farm-b> LOG-farm-b-during-drain6565worker2 # [ 141.683324] nix-daemon[856]: reaped child process 1674, status = succeeded6566worker2 # [ 141.843696] nix-daemon[856]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6567worker2 # [ 141.858537] nix-daemon[856]: reaped child process 1693, status = succeeded6568lb # [ 141.886643] niks3-server[1158]: 2026/09/21 13:47:16 INFO Received uploads request method=POST path=/api/pending_closures6569worker2 # [ 141.882897] nix-grpc-daemon[1547]: time=2026-09-21T13:47:16.763Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6570worker2 # [ 141.885363] nix-grpc-daemon[1547]: time=2026-09-21T13:47:16.764Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6571lb # [ 141.919449] niks3-server[1158]: 2026/09/21 13:47:16 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6572lb # [ 141.931478] niks3-server[1158]: 2026/09/21 13:47:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6573lb # [ 141.935209] niks3-server[1158]: 2026/09/21 13:47:16 INFO Signed narinfos id=48 count=16574worker2 # [ 141.924599] nix-grpc-daemon[1547]: time=2026-09-21T13:47:16.804Z level=INFO msg="Uploading 1 narinfos"6575lb # [ 141.942562] niks3-server[1158]: 2026/09/21 13:47:16 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6576lb # [ 141.950186] niks3-server[1158]: 2026/09/21 13:47:16 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6577lb # [ 141.954535] niks3-server[1158]: 2026/09/21 13:47:16 INFO Completed upload id=486578worker2 # [ 141.941290] nix-grpc-daemon[1547]: time=2026-09-21T13:47:16.821Z level=INFO msg="Upload complete. (268ms)"6579lb # [ 141.956218] niks3-server[1158]: 2026/09/21 13:47:16 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6580lb # [ 141.959259] niks3-server[1158]: 2026/09/21 13:47:16 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6581worker2 # [ 141.947316] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16582worker2 # [ 141.956596] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:16Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06583worker2 # [ 141.958287] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:16Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16584worker2 # [ 141.963849] nix-daemon[856]: reaped child process 1672, status = succeeded6585worker2 # [ 142.001526] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483566586worker2 # [ 142.018509] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:483566587worker2 # [ 142.032521] nix-grpc-daemon[1547]: time=2026-09-21T13:47:16.912Z level=INFO msg="All 2 paths already cached"6588client # copying 1 paths to 'lb:50051'...6589worker2 # [ 142.040654] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:483566590client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6591worker2 # [ 142.060180] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6592worker2 # [ 142.231718] nix-daemon[856]: accepted connection from pid 1699, user nix-grpc-daemon (trusted)6593worker2 # [ 142.245560] nix-daemon[856]: reaped child process 1702, status = succeeded6594worker2 # [ 142.393116] nix-daemon[856]: accepted connection from pid 1704, user nix-grpc-daemon (trusted)6595worker2 # [ 142.405594] nix-daemon[856]: reaped child process 1707, status = succeeded6596lb # [ 142.429799] niks3-server[1158]: 2026/09/21 13:47:17 INFO Received uploads request method=POST path=/api/pending_closures6597worker2 # [ 142.428797] nix-grpc-daemon[1547]: time=2026-09-21T13:47:17.309Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6598worker2 # [ 142.430342] nix-grpc-daemon[1547]: time=2026-09-21T13:47:17.310Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6599lb # [ 142.467796] niks3-server[1158]: 2026/09/21 13:47:17 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6600lb # [ 142.471203] niks3-server[1158]: 2026/09/21 13:47:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6601lb # [ 142.476112] niks3-server[1158]: 2026/09/21 13:47:17 INFO Signed narinfos id=49 count=16602worker2 # [ 142.466049] nix-grpc-daemon[1547]: time=2026-09-21T13:47:17.346Z level=INFO msg="Uploading 1 narinfos"6603lb # [ 142.488965] niks3-server[1158]: 2026/09/21 13:47:17 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6604lb # [ 142.493916] niks3-server[1158]: 2026/09/21 13:47:17 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6605lb # [ 142.500041] niks3-server[1158]: 2026/09/21 13:47:17 INFO Completed upload id=496606worker2 # [ 142.487232] nix-grpc-daemon[1547]: time=2026-09-21T13:47:17.367Z level=INFO msg="Upload complete. (413ms)"6607lb # [ 142.502021] niks3-server[1158]: 2026/09/21 13:47:17 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6608worker2 # [ 142.490774] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=0 paths=1 nar_bytes_in=7686609worker2 # [ 142.495097] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483566610worker2 # [ 142.501842] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6611worker2 # [ 142.509107] nix-daemon[856]: reaped child process 1697, status = succeeded6612client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6613worker2 # [ 142.514727] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)6614client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6615client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6616client # farm-top> LOG-farm-top-during-drain6617worker2 # [ 143.329347] nix-daemon[856]: reaped child process 1711, status = succeeded6618worker2 # [ 143.453383] nix-daemon[856]: accepted connection from pid 1727, user nix-grpc-daemon (trusted)6619worker2 # [ 143.465145] nix-daemon[856]: reaped child process 1730, status = succeeded6620lb # [ 143.491688] niks3-server[1158]: 2026/09/21 13:47:18 INFO Received uploads request method=POST path=/api/pending_closures6621worker2 # [ 143.483540] nix-grpc-daemon[1547]: time=2026-09-21T13:47:18.363Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6622worker2 # [ 143.485102] nix-grpc-daemon[1547]: time=2026-09-21T13:47:18.365Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6623lb # [ 143.512431] niks3-server[1158]: 2026/09/21 13:47:18 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6624lb # [ 143.515994] niks3-server[1158]: 2026/09/21 13:47:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6625lb # [ 143.517800] niks3-server[1158]: 2026/09/21 13:47:18 INFO Signed narinfos id=50 count=16626worker2 # [ 143.504915] nix-grpc-daemon[1547]: time=2026-09-21T13:47:18.385Z level=INFO msg="Uploading 1 narinfos"6627lb # [ 143.521262] niks3-server[1158]: 2026/09/21 13:47:18 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6628lb # [ 143.524659] niks3-server[1158]: 2026/09/21 13:47:18 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6629lb # [ 143.529434] niks3-server[1158]: 2026/09/21 13:47:18 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6630lb # [ 143.532146] niks3-server[1158]: 2026/09/21 13:47:18 INFO Completed upload id=506631worker2 # [ 143.520250] nix-grpc-daemon[1547]: time=2026-09-21T13:47:18.400Z level=INFO msg="Upload complete. (199ms)"6632lb # [ 143.534148] niks3-server[1158]: 2026/09/21 13:47:18 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6633worker2 # [ 143.522754] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16634worker2 # [ 143.530084] nix-daemon[856]: reaped child process 1709, status = succeeded6635client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 5.77 seconds)6636client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6637client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.21 seconds)6638worker1: must succeed: pkill -f 'read -t [9]0 x'6639worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6640client: waiting for success: ! systemctl is-active sw6641worker1 # [ 143.886241] nix-daemon[871]: reaped child process 1719, status = succeeded6642worker1 # [ 144.033086] nix-daemon[871]: accepted connection from pid 1760, user nix-grpc-daemon (trusted)6643worker1 # [ 144.043645] nix-daemon[871]: reaped child process 1763, status = succeeded6644lb # [ 144.042176] niks3-server[1158]: 2026/09/21 13:47:18 INFO Received uploads request method=POST path=/api/pending_closures6645worker1 # [ 144.069663] nix-grpc-daemon[1663]: time=2026-09-21T13:47:18.297Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6646worker1 # [ 144.071822] nix-grpc-daemon[1663]: time=2026-09-21T13:47:18.299Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6647lb # [ 144.079950] niks3-server[1158]: 2026/09/21 13:47:18 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6648lb # [ 144.086348] niks3-server[1158]: 2026/09/21 13:47:18 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6649lb # [ 144.095981] niks3-server[1158]: 2026/09/21 13:47:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6650lb # [ 144.103299] niks3-server[1158]: 2026/09/21 13:47:19 INFO Signed narinfos id=51 count=16651worker1 # [ 144.125202] nix-grpc-daemon[1663]: time=2026-09-21T13:47:18.352Z level=INFO msg="Uploading 1 narinfos"6652lb # [ 144.108280] niks3-server[1158]: 2026/09/21 13:47:19 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6653lb # [ 144.119340] niks3-server[1158]: 2026/09/21 13:47:19 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6654lb # [ 144.124987] niks3-server[1158]: 2026/09/21 13:47:19 INFO Completed upload id=516655worker1 # [ 144.144214] nix-grpc-daemon[1663]: time=2026-09-21T13:47:18.371Z level=INFO msg="Upload complete. (265ms)"6656lb # [ 144.126862] niks3-server[1158]: 2026/09/21 13:47:19 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6657worker1 # [ 144.151513] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48164 duration_s=8 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16658worker1 # [ 144.163512] nix-daemon[871]: reaped child process 1717, status = succeeded6659client # [ 144.156873] systemd[1]: sw.service: Deactivated successfully.6660client # [ 144.161147] systemd[1]: sw.service: Consumed 116ms CPU time over 8.938s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6661worker1 # [ 144.433726] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:18Z level=info event=signal_exit6662client: (finished: waiting for success: ! systemctl is-active sw, in 1.09 seconds)6663client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6664client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds)6665worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6666worker2 # [ 145.630783] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:20Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:483326667worker2 # [ 145.634503] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:20Z level=info event=worker_gone addr=192.168.1.3:500516668worker1 # [ 145.667459] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:19Z level=info event=scheduler_restarting6669worker1 # [ 145.675811] nix-grpc-daemon[1557]: ts=2026-09-21T13:47:19Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6670worker1 # [ 145.690244] nix-daemon[871]: reaped child process 1613, status = succeeded6671worker1 # [ 146.239282] nix-grpc-daemon[1557]: E0000 00:00:1789998440.466839 1602 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true6672worker1 # [ 146.244195] nix-grpc-daemon[1557]: E0000 00:00:1789998440.467136 1602 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6673worker1 # [ 146.461540] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6674worker1 # [ 146.464770] systemd[1]: nix-grpc-daemon.service: Consumed 611ms CPU time over 26.228s wall clock time, 32.6M memory peak, 20.9K incoming IP traffic, 27.6K outgoing IP traffic.6675worker1 # [ 146.481595] systemd[1]: Starting Nix worker-protocol over gRPC...6676worker1 # [ 146.628715] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:20Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6677worker1 # [ 146.630964] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:20Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6678worker1 # [ 146.634568] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:20Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16679worker1 # [ 146.641065] nix-grpc-daemon[1777]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6680worker1 # [ 146.642399] nix-grpc-daemon[1777]: E0000 00:00:1789998440.868501 1777 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6681worker1 # [ 146.648094] nix-grpc-daemon[1777]: E0000 00:00:1789998440.873698 1777 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6682worker1 # [ 146.651574] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:20Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6683worker1 # [ 146.653578] systemd[1]: Started Nix worker-protocol over gRPC.6684worker1 # [ 146.659303] nix-grpc-daemon[1777]: E0000 00:00:1789998440.887263 1818 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6685worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 1.68 seconds)6686lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6687lb: must succeed: curl -sf localhost:9901/clusters6688worker1 # [ 146.671983] nix-grpc-daemon[1777]: E0000 00:00:1789998440.899894 1818 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6689worker1 # [ 146.675135] nix-grpc-daemon[1777]: E0000 00:00:1789998440.900082 1818 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6690worker1 # [ 146.686053] nix-grpc-daemon[1777]: E0000 00:00:1789998440.913977 1818 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6691worker1 # [ 146.698243] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:20Z level=info event=scheduler_connected addr=lb:500516692worker2 # [ 146.669888] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:21Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:483446693worker2 # [ 146.673679] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:21Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06694lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6695lb # [ 147.410107] envoy[914]: conn peer=[2001:db8:1::1]:50922 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9598 tx=4117 ms=55476696lb # [ 147.413862] envoy[914]: conn peer=[2001:db8:1::1]:50920 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2962 tx=1427 ms=87096697lb # [ 147.417370] envoy[914]: conn peer=[2001:db8:1::3]:36566 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1114 tx=448 ms=251966698lb: must succeed: curl -sf localhost:9901/clusters6699lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6700lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.16 seconds)6701lb: must succeed: curl -sf localhost:9901/clusters6702lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6703worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6704worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6705(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 12.71 seconds)6706subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6707client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16708worker1 # [ 148.147538] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:421066709worker1 # [ 148.158610] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6710worker1 # [ 148.172341] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421066711worker1 # [ 148.183441] nix-grpc-daemon[1777]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 86 ms (attempt 1/5)6712worker1 # [ 148.275113] nix-grpc-daemon[1777]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 84 ms (attempt 2/5)6713worker1 # [ 148.365133] nix-grpc-daemon[1777]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 337 ms (attempt 3/5)6714worker1 # [ 148.707966] nix-grpc-daemon[1777]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 466 ms (attempt 4/5)6715worker1 # [ 149.180811] nix-grpc-daemon[1777]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6716worker1 # [ 149.185098] nix-grpc-daemon[1777]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6717worker2 # [ 149.200794] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:24Z level=debug event=want client=5 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06718worker2 # [ 149.205144] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:24Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16719worker2 # [ 149.209397] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:24Z level=debug event=want client=5 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06720worker2 # [ 149.213223] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:24Z level=debug event=want client=5 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06721worker1 # [ 149.277431] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066722worker1 # [ 149.290723] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421066723worker1 # [ 149.303122] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421066724worker1 # [ 149.313675] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6725worker1 # [ 149.334245] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:23Z level=info event=niks3_push_started pid=18376726worker1 # [ 149.528289] nix-grpc-daemon[1837]: time=2026-09-21T13:47:23.755Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6727worker1 # [ 149.690569] nix-daemon[871]: accepted connection from pid 1850, user nix-grpc-daemon (trusted)6728worker1 # [ 149.701080] nix-daemon[871]: reaped child process 1853, status = succeeded6729worker1 # [ 149.871145] nix-daemon[871]: accepted connection from pid 1855, user nix-grpc-daemon (trusted)6730worker1 # [ 149.882737] nix-daemon[871]: reaped child process 1858, status = succeeded6731lb # [ 149.881934] niks3-server[1158]: 2026/09/21 13:47:24 INFO Received uploads request method=POST path=/api/pending_closures6732worker1 # [ 149.910389] nix-grpc-daemon[1837]: time=2026-09-21T13:47:24.138Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6733worker1 # [ 149.911957] nix-grpc-daemon[1837]: time=2026-09-21T13:47:24.139Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6734lb # [ 149.915576] niks3-server[1158]: 2026/09/21 13:47:24 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6735lb # [ 149.925963] niks3-server[1158]: 2026/09/21 13:47:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6736lb # [ 149.932810] niks3-server[1158]: 2026/09/21 13:47:24 INFO Signed narinfos id=52 count=16737worker1 # [ 149.954847] nix-grpc-daemon[1837]: time=2026-09-21T13:47:24.182Z level=INFO msg="Uploading 1 narinfos"6738lb # [ 149.943982] niks3-server[1158]: 2026/09/21 13:47:24 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6739lb # [ 149.950323] niks3-server[1158]: 2026/09/21 13:47:24 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6740worker1 # [ 149.976335] nix-grpc-daemon[1837]: time=2026-09-21T13:47:24.204Z level=INFO msg="Upload complete. (444ms)"6741lb # [ 149.957514] niks3-server[1158]: 2026/09/21 13:47:24 INFO Completed upload id=526742worker1 # [ 149.978350] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=0 paths=1 nar_bytes_in=5046743worker1 # [ 149.982413] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066744lb # [ 149.963449] niks3-server[1158]: 2026/09/21 13:47:24 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6745worker1 # [ 149.988539] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6746worker1 # [ 149.990760] nix-daemon[871]: reaped child process 1835, status = succeeded6747worker1 # [ 149.995780] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6748worker1 # [ 151.009458] nix-daemon[871]: reaped child process 1862, status = succeeded6749worker1 # [ 151.157847] nix-daemon[871]: accepted connection from pid 1878, user nix-grpc-daemon (trusted)6750worker1 # [ 151.168637] nix-daemon[871]: reaped child process 1881, status = succeeded6751lb # [ 151.166784] niks3-server[1158]: 2026/09/21 13:47:26 INFO Received uploads request method=POST path=/api/pending_closures6752worker1 # [ 151.191660] nix-grpc-daemon[1837]: time=2026-09-21T13:47:25.419Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6753worker1 # [ 151.193846] nix-grpc-daemon[1837]: time=2026-09-21T13:47:25.421Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6754lb # [ 151.203091] niks3-server[1158]: 2026/09/21 13:47:26 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6755lb # [ 151.214132] niks3-server[1158]: 2026/09/21 13:47:26 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6756lb # [ 151.222257] niks3-server[1158]: 2026/09/21 13:47:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6757worker1 # [ 151.247128] nix-grpc-daemon[1837]: time=2026-09-21T13:47:25.474Z level=INFO msg="Uploading 1 narinfos"6758lb # [ 151.228352] niks3-server[1158]: 2026/09/21 13:47:26 INFO Signed narinfos id=53 count=16759lb # [ 151.234295] niks3-server[1158]: 2026/09/21 13:47:26 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6760lb # [ 151.244608] niks3-server[1158]: 2026/09/21 13:47:26 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6761lb # [ 151.249976] niks3-server[1158]: 2026/09/21 13:47:26 INFO Completed upload id=536762worker1 # [ 151.268917] nix-grpc-daemon[1837]: time=2026-09-21T13:47:25.496Z level=INFO msg="Upload complete. (266ms)"6763lb # [ 151.251540] niks3-server[1158]: 2026/09/21 13:47:26 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6764worker1 # [ 151.274600] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=7 outputs=16765worker2 # [ 151.243721] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:26Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16766worker1 # [ 151.283199] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066767worker2 # [ 151.253154] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:26Z level=debug event=want client=5 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06768worker1 # [ 151.294502] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421066769worker1 # [ 151.298990] nix-daemon[871]: reaped child process 1860, status = succeeded6770worker1 # [ 151.304291] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421066771worker1 # [ 151.310321] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6772worker1 # [ 151.483127] nix-daemon[871]: accepted connection from pid 1885, user nix-grpc-daemon (trusted)6773worker1 # [ 151.494975] nix-daemon[871]: reaped child process 1888, status = succeeded6774worker1 # [ 151.659223] nix-daemon[871]: accepted connection from pid 1890, user nix-grpc-daemon (trusted)6775worker1 # [ 151.670810] nix-daemon[871]: reaped child process 1893, status = succeeded6776lb # [ 151.668647] niks3-server[1158]: 2026/09/21 13:47:26 INFO Received uploads request method=POST path=/api/pending_closures6777worker1 # [ 151.691495] nix-grpc-daemon[1837]: time=2026-09-21T13:47:25.919Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6778worker1 # [ 151.693323] nix-grpc-daemon[1837]: time=2026-09-21T13:47:25.920Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6779lb # [ 151.685421] niks3-server[1158]: 2026/09/21 13:47:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6780lb # [ 151.688246] niks3-server[1158]: 2026/09/21 13:47:26 INFO Signed narinfos id=54 count=16781worker1 # [ 151.709247] nix-grpc-daemon[1837]: time=2026-09-21T13:47:25.937Z level=INFO msg="Uploading 1 narinfos"6782lb # [ 151.690777] niks3-server[1158]: 2026/09/21 13:47:26 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6783lb # [ 151.694757] niks3-server[1158]: 2026/09/21 13:47:26 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6784lb # [ 151.698826] niks3-server[1158]: 2026/09/21 13:47:26 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6785lb # [ 151.701495] niks3-server[1158]: 2026/09/21 13:47:26 INFO Completed upload id=546786worker1 # [ 151.721543] nix-grpc-daemon[1837]: time=2026-09-21T13:47:25.949Z level=INFO msg="Upload complete. (405ms)"6787lb # [ 151.704046] niks3-server[1158]: 2026/09/21 13:47:26 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6788worker1 # [ 151.723558] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=0 paths=1 nar_bytes_in=5046789worker1 # [ 151.727378] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066790worker1 # [ 151.732926] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6791worker1 # [ 151.735105] nix-daemon[871]: reaped child process 1883, status = succeeded6792worker1 # [ 151.740332] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6793worker1 # [ 153.167391] nix-daemon[871]: reaped child process 1897, status = succeeded6794worker1 # [ 153.309815] nix-daemon[871]: accepted connection from pid 1913, user nix-grpc-daemon (trusted)6795worker1 # [ 153.320818] nix-daemon[871]: reaped child process 1916, status = succeeded6796lb # [ 153.318801] niks3-server[1158]: 2026/09/21 13:47:28 INFO Received uploads request method=POST path=/api/pending_closures6797worker1 # [ 153.346860] nix-grpc-daemon[1837]: time=2026-09-21T13:47:27.574Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6798worker1 # [ 153.348364] nix-grpc-daemon[1837]: time=2026-09-21T13:47:27.576Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6799lb # [ 153.356556] niks3-server[1158]: 2026/09/21 13:47:28 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6800lb # [ 153.364171] niks3-server[1158]: 2026/09/21 13:47:28 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6801lb # [ 153.371750] niks3-server[1158]: 2026/09/21 13:47:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6802lb # [ 153.375620] niks3-server[1158]: 2026/09/21 13:47:28 INFO Signed narinfos id=55 count=16803worker1 # [ 153.396856] nix-grpc-daemon[1837]: time=2026-09-21T13:47:27.624Z level=INFO msg="Uploading 1 narinfos"6804lb # [ 153.380294] niks3-server[1158]: 2026/09/21 13:47:28 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6805lb # [ 153.387227] niks3-server[1158]: 2026/09/21 13:47:28 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6806lb # [ 153.389528] niks3-server[1158]: 2026/09/21 13:47:28 INFO Completed upload id=556807lb # [ 153.391605] niks3-server[1158]: 2026/09/21 13:47:28 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6808worker1 # [ 153.409691] nix-grpc-daemon[1837]: time=2026-09-21T13:47:27.637Z level=INFO msg="Upload complete. (260ms)"6809worker1 # [ 153.414500] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=8 outputs=16810worker2 # [ 153.383296] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:28Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16811worker1 # [ 153.426470] nix-daemon[871]: reaped child process 1895, status = succeeded6812worker2 # [ 153.393964] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:28Z level=debug event=want client=5 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06813worker1 # [ 153.465742] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066814worker1 # [ 153.480404] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421066815worker1 # [ 153.491289] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421066816worker1 # [ 153.499679] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6817worker1 # [ 153.663874] nix-daemon[871]: accepted connection from pid 1920, user nix-grpc-daemon (trusted)6818worker1 # [ 153.675647] nix-daemon[871]: reaped child process 1923, status = succeeded6819worker1 # [ 153.835774] nix-daemon[871]: accepted connection from pid 1925, user nix-grpc-daemon (trusted)6820worker1 # [ 153.847701] nix-daemon[871]: reaped child process 1928, status = succeeded6821lb # [ 153.842863] niks3-server[1158]: 2026/09/21 13:47:28 INFO Received uploads request method=POST path=/api/pending_closures6822worker1 # [ 153.872437] nix-grpc-daemon[1837]: time=2026-09-21T13:47:28.100Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6823worker1 # [ 153.874041] nix-grpc-daemon[1837]: time=2026-09-21T13:47:28.101Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6824lb # [ 153.875153] niks3-server[1158]: 2026/09/21 13:47:28 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6825lb # [ 153.884645] niks3-server[1158]: 2026/09/21 13:47:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6826worker1 # [ 153.910240] nix-grpc-daemon[1837]: time=2026-09-21T13:47:28.138Z level=INFO msg="Uploading 1 narinfos"6827lb # [ 153.891401] niks3-server[1158]: 2026/09/21 13:47:28 INFO Signed narinfos id=56 count=16828lb # [ 153.895346] niks3-server[1158]: 2026/09/21 13:47:28 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6829lb # [ 153.902971] niks3-server[1158]: 2026/09/21 13:47:28 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6830lb # [ 153.907060] niks3-server[1158]: 2026/09/21 13:47:28 INFO Completed upload id=566831worker1 # [ 153.926129] nix-grpc-daemon[1837]: time=2026-09-21T13:47:28.153Z level=INFO msg="Upload complete. (413ms)"6832lb # [ 153.909167] niks3-server[1158]: 2026/09/21 13:47:28 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6833worker1 # [ 153.931130] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=0 paths=1 nar_bytes_in=5046834worker1 # [ 153.935539] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066835worker1 # [ 153.945637] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6836worker1 # [ 153.948209] nix-daemon[871]: reaped child process 1918, status = succeeded6837worker1 # [ 153.953168] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6838worker1 # [ 155.368294] nix-daemon[871]: reaped child process 1932, status = succeeded6839worker1 # [ 155.510299] nix-daemon[871]: accepted connection from pid 1948, user nix-grpc-daemon (trusted)6840worker1 # [ 155.522187] nix-daemon[871]: reaped child process 1951, status = succeeded6841lb # [ 155.514381] niks3-server[1158]: 2026/09/21 13:47:30 INFO Received uploads request method=POST path=/api/pending_closures6842worker1 # [ 155.538039] nix-grpc-daemon[1837]: time=2026-09-21T13:47:29.765Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6843worker1 # [ 155.539503] nix-grpc-daemon[1837]: time=2026-09-21T13:47:29.767Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"6844lb # [ 155.533865] niks3-server[1158]: 2026/09/21 13:47:30 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6845lb # [ 155.538449] niks3-server[1158]: 2026/09/21 13:47:30 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv6846lb # [ 155.540500] niks3-server[1158]: 2026/09/21 13:47:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6847lb # [ 155.542513] niks3-server[1158]: 2026/09/21 13:47:30 INFO Signed narinfos id=57 count=16848worker1 # [ 155.562695] nix-grpc-daemon[1837]: time=2026-09-21T13:47:29.790Z level=INFO msg="Uploading 1 narinfos"6849lb # [ 155.546059] niks3-server[1158]: 2026/09/21 13:47:30 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls6850lb # [ 155.551163] niks3-server[1158]: 2026/09/21 13:47:30 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6851lb # [ 155.554201] niks3-server[1158]: 2026/09/21 13:47:30 INFO Completed upload id=576852worker1 # [ 155.574206] nix-grpc-daemon[1837]: time=2026-09-21T13:47:29.802Z level=INFO msg="Upload complete. (214ms)"6853lb # [ 155.555947] niks3-server[1158]: 2026/09/21 13:47:30 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo6854worker1 # [ 155.576599] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=9 outputs=16855worker2 # [ 155.546102] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:30Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=16856worker1 # [ 155.583446] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066857worker1 # [ 155.586596] nix-daemon[871]: reaped child process 1930, status = succeeded6858worker2 # [ 155.554398] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:30Z level=debug event=want client=5 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=06859worker1 # [ 155.590608] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421066860worker1 # [ 155.596648] nix-grpc-daemon[1837]: time=2026-09-21T13:47:29.824Z level=INFO msg="All 1 paths already cached"6861worker1 # [ 155.600255] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421066862worker1 # [ 155.605260] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6863worker1 # [ 155.740071] nix-daemon[871]: accepted connection from pid 1955, user nix-grpc-daemon (trusted)6864worker1 # [ 155.752769] nix-daemon[871]: reaped child process 1958, status = succeeded6865worker1 # [ 155.886655] nix-daemon[871]: accepted connection from pid 1960, user nix-grpc-daemon (trusted)6866worker1 # [ 155.899276] nix-daemon[871]: reaped child process 1963, status = succeeded6867lb # [ 155.892643] niks3-server[1158]: 2026/09/21 13:47:30 INFO Received uploads request method=POST path=/api/pending_closures6868worker1 # [ 155.919696] nix-grpc-daemon[1837]: time=2026-09-21T13:47:30.147Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6869worker1 # [ 155.921260] nix-grpc-daemon[1837]: time=2026-09-21T13:47:30.149Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"6870lb # [ 155.920353] niks3-server[1158]: 2026/09/21 13:47:30 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst6871lb # [ 155.931405] niks3-server[1158]: 2026/09/21 13:47:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6872worker1 # [ 155.956690] nix-grpc-daemon[1837]: time=2026-09-21T13:47:30.184Z level=INFO msg="Uploading 1 narinfos"6873lb # [ 155.937860] niks3-server[1158]: 2026/09/21 13:47:30 INFO Signed narinfos id=58 count=16874lb # [ 155.944815] niks3-server[1158]: 2026/09/21 13:47:30 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls6875lb # [ 155.957824] niks3-server[1158]: 2026/09/21 13:47:30 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6876lb # [ 155.962755] niks3-server[1158]: 2026/09/21 13:47:30 INFO Completed upload id=586877worker1 # [ 155.981767] nix-grpc-daemon[1837]: time=2026-09-21T13:47:30.209Z level=INFO msg="Upload complete. (371ms)"6878lb # [ 155.965198] niks3-server[1158]: 2026/09/21 13:47:30 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo6879worker1 # [ 155.986133] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=0 paths=1 nar_bytes_in=6246880worker1 # [ 155.990902] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066881worker1 # [ 156.001063] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6882worker1 # [ 156.007103] nix-daemon[871]: reaped child process 1953, status = succeeded6883worker1 # [ 156.010195] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6884worker1 # [ 157.625909] nix-daemon[871]: reaped child process 1967, status = succeeded6885worker1 # [ 157.778387] nix-daemon[871]: accepted connection from pid 1983, user nix-grpc-daemon (trusted)6886worker1 # [ 157.790409] nix-daemon[871]: reaped child process 1986, status = succeeded6887lb # [ 157.788990] niks3-server[1158]: 2026/09/21 13:47:32 INFO Received uploads request method=POST path=/api/pending_closures6888worker1 # [ 157.812620] nix-grpc-daemon[1837]: time=2026-09-21T13:47:32.040Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6889worker1 # [ 157.814174] nix-grpc-daemon[1837]: time=2026-09-21T13:47:32.042Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"6890lb # [ 157.812531] niks3-server[1158]: 2026/09/21 13:47:32 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv6891lb # [ 157.815294] niks3-server[1158]: 2026/09/21 13:47:32 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst6892lb # [ 157.819061] niks3-server[1158]: 2026/09/21 13:47:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6893lb # [ 157.820690] niks3-server[1158]: 2026/09/21 13:47:32 INFO Signed narinfos id=59 count=16894worker1 # [ 157.841256] nix-grpc-daemon[1837]: time=2026-09-21T13:47:32.069Z level=INFO msg="Uploading 1 narinfos"6895lb # [ 157.824304] niks3-server[1158]: 2026/09/21 13:47:32 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls6896lb # [ 157.829285] niks3-server[1158]: 2026/09/21 13:47:32 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6897lb # [ 157.833197] niks3-server[1158]: 2026/09/21 13:47:32 INFO Completed upload id=596898worker1 # [ 157.851916] nix-grpc-daemon[1837]: time=2026-09-21T13:47:32.079Z level=INFO msg="Upload complete. (234ms)"6899lb # [ 157.834828] niks3-server[1158]: 2026/09/21 13:47:32 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo6900worker1 # [ 157.854370] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=10 outputs=16901worker2 # [ 157.825248] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:32Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=16902worker1 # [ 157.861560] nix-daemon[871]: reaped child process 1965, status = succeeded6903worker1 # [ 157.862962] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066904worker1 # [ 157.869071] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421066905worker2 # [ 157.834636] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:32Z level=debug event=want client=5 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=06906worker1 # [ 157.874874] nix-grpc-daemon[1837]: time=2026-09-21T13:47:32.102Z level=INFO msg="All 1 paths already cached"6907worker1 # [ 157.877955] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421066908worker1 # [ 157.882596] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6909worker1 # [ 158.036058] nix-daemon[871]: accepted connection from pid 1990, user nix-grpc-daemon (trusted)6910worker1 # [ 158.048486] nix-daemon[871]: reaped child process 1993, status = succeeded6911worker1 # [ 158.193581] nix-daemon[871]: accepted connection from pid 1995, user nix-grpc-daemon (trusted)6912worker1 # [ 158.205398] nix-daemon[871]: reaped child process 1998, status = succeeded6913lb # [ 158.197074] niks3-server[1158]: 2026/09/21 13:47:33 INFO Received uploads request method=POST path=/api/pending_closures6914worker1 # [ 158.220945] nix-grpc-daemon[1837]: time=2026-09-21T13:47:32.448Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6915worker1 # [ 158.222323] nix-grpc-daemon[1837]: time=2026-09-21T13:47:32.448Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"6916lb # [ 158.210440] niks3-server[1158]: 2026/09/21 13:47:33 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst6917lb # [ 158.213855] niks3-server[1158]: 2026/09/21 13:47:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6918lb # [ 158.215581] niks3-server[1158]: 2026/09/21 13:47:33 INFO Signed narinfos id=60 count=16919worker1 # [ 158.235691] nix-grpc-daemon[1837]: time=2026-09-21T13:47:32.463Z level=INFO msg="Uploading 1 narinfos"6920lb # [ 158.222318] niks3-server[1158]: 2026/09/21 13:47:33 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6921lb # [ 158.225111] niks3-server[1158]: 2026/09/21 13:47:33 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls6922lb # [ 158.227081] niks3-server[1158]: 2026/09/21 13:47:33 INFO Completed upload id=606923worker1 # [ 158.245935] nix-grpc-daemon[1837]: time=2026-09-21T13:47:32.473Z level=INFO msg="Upload complete. (358ms)"6924worker1 # [ 158.247909] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=0 paths=1 nar_bytes_in=6246925lb # [ 158.230256] niks3-server[1158]: 2026/09/21 13:47:33 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo6926worker1 # [ 158.251484] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066927worker1 # [ 158.257459] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6928worker1 # [ 158.259702] nix-daemon[871]: reaped child process 1988, status = succeeded6929worker1 # [ 158.264887] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6930worker1 # [ 159.281405] nix-daemon[871]: reaped child process 2002, status = succeeded6931worker1 # [ 159.422871] nix-daemon[871]: accepted connection from pid 2018, user nix-grpc-daemon (trusted)6932worker1 # [ 159.433583] nix-daemon[871]: reaped child process 2021, status = succeeded6933lb # [ 159.427902] niks3-server[1158]: 2026/09/21 13:47:34 INFO Received uploads request method=POST path=/api/pending_closures6934worker1 # [ 159.457031] nix-grpc-daemon[1837]: time=2026-09-21T13:47:33.684Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6935worker1 # [ 159.458497] nix-grpc-daemon[1837]: time=2026-09-21T13:47:33.686Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"6936lb # [ 159.465078] niks3-server[1158]: 2026/09/21 13:47:34 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv6937lb # [ 159.478846] niks3-server[1158]: 2026/09/21 13:47:34 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst6938lb # [ 159.490933] niks3-server[1158]: 2026/09/21 13:47:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6939worker1 # [ 159.515583] nix-grpc-daemon[1837]: time=2026-09-21T13:47:33.743Z level=INFO msg="Uploading 1 narinfos"6940lb # [ 159.496811] niks3-server[1158]: 2026/09/21 13:47:34 INFO Signed narinfos id=61 count=16941lb # [ 159.501719] niks3-server[1158]: 2026/09/21 13:47:34 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls6942lb # [ 159.510120] niks3-server[1158]: 2026/09/21 13:47:34 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6943lb # [ 159.513359] niks3-server[1158]: 2026/09/21 13:47:34 INFO Completed upload id=616944lb # [ 159.515223] niks3-server[1158]: 2026/09/21 13:47:34 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo6945worker1 # [ 159.533834] nix-grpc-daemon[1837]: time=2026-09-21T13:47:33.761Z level=INFO msg="Upload complete. (262ms)"6946worker1 # [ 159.540927] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=11 outputs=16947worker2 # [ 159.508949] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:34Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=16948worker1 # [ 159.552739] nix-daemon[871]: reaped child process 2000, status = succeeded6949worker2 # [ 159.519663] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:34Z level=debug event=want client=5 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=06950worker1 # [ 159.584718] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066951worker1 # [ 159.597260] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421066952worker1 # [ 159.609873] nix-grpc-daemon[1837]: time=2026-09-21T13:47:33.837Z level=INFO msg="All 1 paths already cached"6953worker1 # [ 159.616989] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421066954worker1 # [ 159.625329] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6955worker1 # [ 159.806581] nix-daemon[871]: accepted connection from pid 2025, user nix-grpc-daemon (trusted)6956worker1 # [ 159.818501] nix-daemon[871]: reaped child process 2028, status = succeeded6957worker1 # [ 159.978496] nix-daemon[871]: accepted connection from pid 2030, user nix-grpc-daemon (trusted)6958worker1 # [ 159.990462] nix-daemon[871]: reaped child process 2033, status = succeeded6959lb # [ 159.986742] niks3-server[1158]: 2026/09/21 13:47:34 INFO Received uploads request method=POST path=/api/pending_closures6960worker1 # [ 160.014822] nix-grpc-daemon[1837]: time=2026-09-21T13:47:34.242Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6961worker1 # [ 160.016246] nix-grpc-daemon[1837]: time=2026-09-21T13:47:34.242Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"6962lb # [ 160.014584] niks3-server[1158]: 2026/09/21 13:47:34 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst6963lb # [ 160.025089] niks3-server[1158]: 2026/09/21 13:47:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6964worker1 # [ 160.048922] nix-grpc-daemon[1837]: time=2026-09-21T13:47:34.276Z level=INFO msg="Uploading 1 narinfos"6965lb # [ 160.031386] niks3-server[1158]: 2026/09/21 13:47:34 INFO Signed narinfos id=62 count=16966lb # [ 160.036183] niks3-server[1158]: 2026/09/21 13:47:34 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls6967lb # [ 160.046456] niks3-server[1158]: 2026/09/21 13:47:34 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6968lb # [ 160.052374] niks3-server[1158]: 2026/09/21 13:47:34 INFO Completed upload id=626969worker1 # [ 160.071170] nix-grpc-daemon[1837]: time=2026-09-21T13:47:34.299Z level=INFO msg="Upload complete. (432ms)"6970lb # [ 160.056940] niks3-server[1158]: 2026/09/21 13:47:34 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo6971worker1 # [ 160.075063] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=0 paths=1 nar_bytes_in=6246972worker1 # [ 160.080465] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066973worker1 # [ 160.088569] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6974worker1 # [ 160.094694] nix-daemon[871]: reaped child process 2023, status = succeeded6975worker1 # [ 160.099140] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6976worker1 # [ 160.912388] nix-daemon[871]: reaped child process 2037, status = succeeded6977worker1 # [ 161.033218] nix-daemon[871]: accepted connection from pid 2053, user nix-grpc-daemon (trusted)6978worker1 # [ 161.045091] nix-daemon[871]: reaped child process 2056, status = succeeded6979lb # [ 161.038581] niks3-server[1158]: 2026/09/21 13:47:35 INFO Received uploads request method=POST path=/api/pending_closures6980worker1 # [ 161.065430] nix-grpc-daemon[1837]: time=2026-09-21T13:47:35.293Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6981worker1 # [ 161.066913] nix-grpc-daemon[1837]: time=2026-09-21T13:47:35.294Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"6982lb # [ 161.070557] niks3-server[1158]: 2026/09/21 13:47:35 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv6983lb # [ 161.086437] niks3-server[1158]: 2026/09/21 13:47:35 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst6984lb # [ 161.095988] niks3-server[1158]: 2026/09/21 13:47:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6985lb # [ 161.100828] niks3-server[1158]: 2026/09/21 13:47:35 INFO Signed narinfos id=63 count=16986worker1 # [ 161.122107] nix-grpc-daemon[1837]: time=2026-09-21T13:47:35.349Z level=INFO msg="Uploading 1 narinfos"6987lb # [ 161.104766] niks3-server[1158]: 2026/09/21 13:47:36 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls6988lb # [ 161.116355] niks3-server[1158]: 2026/09/21 13:47:36 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6989lb # [ 161.119229] niks3-server[1158]: 2026/09/21 13:47:36 INFO Completed upload id=636990worker1 # [ 161.139459] nix-grpc-daemon[1837]: time=2026-09-21T13:47:35.367Z level=INFO msg="Upload complete. (234ms)"6991lb # [ 161.121110] niks3-server[1158]: 2026/09/21 13:47:36 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo6992worker1 # [ 161.143081] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=12 outputs=16993worker1 # [ 161.148868] nix-daemon[871]: reaped child process 2035, status = succeeded6994worker2 # [ 161.114659] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:35Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=16995worker2 # [ 161.121832] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:36Z level=debug event=want client=5 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=06996worker1 # [ 161.188226] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421066997worker1 # [ 161.199882] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421066998worker1 # [ 161.211567] nix-grpc-daemon[1837]: time=2026-09-21T13:47:35.439Z level=INFO msg="All 2 paths already cached"6999worker1 # [ 161.218146] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421067000worker1 # [ 161.223367] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)7001worker1 # [ 161.363512] nix-daemon[871]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)7002worker1 # [ 161.375561] nix-daemon[871]: reaped child process 2063, status = succeeded7003worker1 # [ 161.525082] nix-daemon[871]: accepted connection from pid 2065, user nix-grpc-daemon (trusted)7004worker1 # [ 161.536870] nix-daemon[871]: reaped child process 2068, status = succeeded7005lb # [ 161.533885] niks3-server[1158]: 2026/09/21 13:47:36 INFO Received uploads request method=POST path=/api/pending_closures7006worker1 # [ 161.556751] nix-grpc-daemon[1837]: time=2026-09-21T13:47:35.784Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7007worker1 # [ 161.558623] nix-grpc-daemon[1837]: time=2026-09-21T13:47:35.786Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7008lb # [ 161.548638] niks3-server[1158]: 2026/09/21 13:47:36 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7009lb # [ 161.550806] niks3-server[1158]: 2026/09/21 13:47:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7010lb # [ 161.552799] niks3-server[1158]: 2026/09/21 13:47:36 INFO Signed narinfos id=64 count=17011worker1 # [ 161.571688] nix-grpc-daemon[1837]: time=2026-09-21T13:47:35.799Z level=INFO msg="Uploading 1 narinfos"7012lb # [ 161.556970] niks3-server[1158]: 2026/09/21 13:47:36 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7013lb # [ 161.560146] niks3-server[1158]: 2026/09/21 13:47:36 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7014lb # [ 161.563024] niks3-server[1158]: 2026/09/21 13:47:36 INFO Completed upload id=647015worker1 # [ 161.581842] nix-grpc-daemon[1837]: time=2026-09-21T13:47:35.809Z level=INFO msg="Upload complete. (353ms)"7016lb # [ 161.565491] niks3-server[1158]: 2026/09/21 13:47:36 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7017worker1 # [ 161.583694] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=0 paths=1 nar_bytes_in=7527018worker1 # [ 161.586528] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421067019worker1 # [ 161.593130] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)7020worker1 # [ 161.595289] nix-daemon[871]: reaped child process 2058, status = succeeded7021worker1 # [ 161.600563] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)7022worker1 # [ 162.413427] nix-daemon[871]: reaped child process 2072, status = succeeded7023worker1 # [ 162.579373] nix-daemon[871]: accepted connection from pid 2088, user nix-grpc-daemon (trusted)7024worker1 # [ 162.590758] nix-daemon[871]: reaped child process 2091, status = succeeded7025lb # [ 162.590364] niks3-server[1158]: 2026/09/21 13:47:37 INFO Received uploads request method=POST path=/api/pending_closures7026worker1 # [ 162.617991] nix-grpc-daemon[1837]: time=2026-09-21T13:47:36.845Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7027worker1 # [ 162.619575] nix-grpc-daemon[1837]: time=2026-09-21T13:47:36.847Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7028lb # [ 162.635179] niks3-server[1158]: 2026/09/21 13:47:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7029worker1 # [ 162.665627] nix-grpc-daemon[1837]: time=2026-09-21T13:47:36.893Z level=INFO msg="Uploading 1 narinfos"7030lb # [ 162.647264] niks3-server[1158]: 2026/09/21 13:47:37 INFO Signed narinfos id=65 count=17031lb # [ 162.652398] niks3-server[1158]: 2026/09/21 13:47:37 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7032lb # [ 162.660987] niks3-server[1158]: 2026/09/21 13:47:37 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7033lb # [ 162.664367] niks3-server[1158]: 2026/09/21 13:47:37 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7034lb # [ 162.673428] niks3-server[1158]: 2026/09/21 13:47:37 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7035lb # [ 162.678452] niks3-server[1158]: 2026/09/21 13:47:37 INFO Completed upload id=657036worker1 # [ 162.699266] nix-grpc-daemon[1837]: time=2026-09-21T13:47:36.927Z level=INFO msg="Upload complete. (292ms)"7037lb # [ 162.681774] niks3-server[1158]: 2026/09/21 13:47:37 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7038worker2 # [ 162.673976] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:37Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17039worker1 # [ 162.705609] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=13 outputs=17040worker1 # [ 162.719280] nix-daemon[871]: reaped child process 2070, status = succeeded7041worker1 # [ 162.750111] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421067042worker1 # [ 162.761937] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421067043worker1 # [ 162.775896] nix-grpc-daemon[1837]: time=2026-09-21T13:47:37.003Z level=INFO msg="All 2 paths already cached"7044worker1 # [ 162.782839] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421067045worker1 # [ 162.793173] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)7046worker1 # [ 162.965567] nix-daemon[871]: accepted connection from pid 2095, user nix-grpc-daemon (trusted)7047worker1 # [ 162.977713] nix-daemon[871]: reaped child process 2098, status = succeeded7048worker1 # [ 163.144145] nix-daemon[871]: accepted connection from pid 2100, user nix-grpc-daemon (trusted)7049worker1 # [ 163.155594] nix-daemon[871]: reaped child process 2103, status = succeeded7050lb # [ 163.153262] niks3-server[1158]: 2026/09/21 13:47:38 INFO Received uploads request method=POST path=/api/pending_closures7051worker1 # [ 163.181189] nix-grpc-daemon[1837]: time=2026-09-21T13:47:37.409Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7052worker1 # [ 163.183062] nix-grpc-daemon[1837]: time=2026-09-21T13:47:37.410Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7053lb # [ 163.181966] niks3-server[1158]: 2026/09/21 13:47:38 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7054lb # [ 163.193422] niks3-server[1158]: 2026/09/21 13:47:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7055lb # [ 163.199756] niks3-server[1158]: 2026/09/21 13:47:38 INFO Signed narinfos id=66 count=17056worker1 # [ 163.221840] nix-grpc-daemon[1837]: time=2026-09-21T13:47:37.449Z level=INFO msg="Uploading 1 narinfos"7057lb # [ 163.204706] niks3-server[1158]: 2026/09/21 13:47:38 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7058lb # [ 163.216856] niks3-server[1158]: 2026/09/21 13:47:38 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7059lb # [ 163.222884] niks3-server[1158]: 2026/09/21 13:47:38 INFO Completed upload id=667060worker1 # [ 163.243842] nix-grpc-daemon[1837]: time=2026-09-21T13:47:37.471Z level=INFO msg="Upload complete. (439ms)"7061lb # [ 163.225539] niks3-server[1158]: 2026/09/21 13:47:38 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7062worker1 # [ 163.246900] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=0 paths=1 nar_bytes_in=7527063worker1 # [ 163.251923] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421067064worker1 # [ 163.261078] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)7065worker1 # [ 163.268751] nix-daemon[871]: reaped child process 2093, status = succeeded7066worker1 # [ 163.274641] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)7067worker1 # [ 164.093210] nix-daemon[871]: reaped child process 2107, status = succeeded7068worker1 # [ 164.222267] nix-daemon[871]: accepted connection from pid 2123, user nix-grpc-daemon (trusted)7069worker1 # [ 164.232725] nix-daemon[871]: reaped child process 2126, status = succeeded7070lb # [ 164.225707] niks3-server[1158]: 2026/09/21 13:47:39 INFO Received uploads request method=POST path=/api/pending_closures7071worker1 # [ 164.252899] nix-grpc-daemon[1837]: time=2026-09-21T13:47:38.480Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7072worker1 # [ 164.254413] nix-grpc-daemon[1837]: time=2026-09-21T13:47:38.482Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7073lb # [ 164.268516] niks3-server[1158]: 2026/09/21 13:47:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7074lb # [ 164.272645] niks3-server[1158]: 2026/09/21 13:47:39 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7075worker1 # [ 164.297118] nix-grpc-daemon[1837]: time=2026-09-21T13:47:38.524Z level=INFO msg="Uploading 1 narinfos"7076lb # [ 164.279192] niks3-server[1158]: 2026/09/21 13:47:39 INFO Signed narinfos id=67 count=17077lb # [ 164.284701] niks3-server[1158]: 2026/09/21 13:47:39 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7078lb # [ 164.289268] niks3-server[1158]: 2026/09/21 13:47:39 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7079lb # [ 164.300101] niks3-server[1158]: 2026/09/21 13:47:39 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7080lb # [ 164.304906] niks3-server[1158]: 2026/09/21 13:47:39 INFO Completed upload id=677081lb # [ 164.307762] niks3-server[1158]: 2026/09/21 13:47:39 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7082worker1 # [ 164.326166] nix-grpc-daemon[1837]: time=2026-09-21T13:47:38.554Z level=INFO msg="Upload complete. (245ms)"7083worker2 # [ 164.300350] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:39Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17084worker1 # [ 164.330567] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=14 outputs=17085worker1 # [ 164.343592] nix-daemon[871]: reaped child process 2105, status = succeeded7086worker1 # [ 164.375981] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421067087worker1 # [ 164.387825] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421067088worker1 # [ 164.400360] nix-grpc-daemon[1837]: time=2026-09-21T13:47:38.628Z level=INFO msg="All 2 paths already cached"7089worker1 # [ 164.407547] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421067090worker1 # [ 164.417181] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)7091worker1 # [ 164.601324] nix-daemon[871]: accepted connection from pid 2130, user nix-grpc-daemon (trusted)7092worker1 # [ 164.613545] nix-daemon[871]: reaped child process 2133, status = succeeded7093worker1 # [ 164.778274] nix-daemon[871]: accepted connection from pid 2135, user nix-grpc-daemon (trusted)7094worker1 # [ 164.790047] nix-daemon[871]: reaped child process 2138, status = succeeded7095lb # [ 164.787182] niks3-server[1158]: 2026/09/21 13:47:39 INFO Received uploads request method=POST path=/api/pending_closures7096worker1 # [ 164.813954] nix-grpc-daemon[1837]: time=2026-09-21T13:47:39.041Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7097worker1 # [ 164.815324] nix-grpc-daemon[1837]: time=2026-09-21T13:47:39.041Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7098lb # [ 164.811120] niks3-server[1158]: 2026/09/21 13:47:39 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7099lb # [ 164.822974] niks3-server[1158]: 2026/09/21 13:47:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7100lb # [ 164.829590] niks3-server[1158]: 2026/09/21 13:47:39 INFO Signed narinfos id=68 count=17101worker1 # [ 164.851634] nix-grpc-daemon[1837]: time=2026-09-21T13:47:39.079Z level=INFO msg="Uploading 1 narinfos"7102lb # [ 164.834678] niks3-server[1158]: 2026/09/21 13:47:39 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7103lb # [ 164.846417] niks3-server[1158]: 2026/09/21 13:47:39 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7104worker1 # [ 164.870752] nix-grpc-daemon[1837]: time=2026-09-21T13:47:39.098Z level=INFO msg="Upload complete. (441ms)"7105lb # [ 164.852223] niks3-server[1158]: 2026/09/21 13:47:39 INFO Completed upload id=687106worker1 # [ 164.872676] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=0 paths=1 nar_bytes_in=7527107lb # [ 164.855622] niks3-server[1158]: 2026/09/21 13:47:39 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7108worker1 # [ 164.879655] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421067109worker1 # [ 164.883950] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)7110worker1 # [ 164.886257] nix-daemon[871]: reaped child process 2128, status = succeeded7111worker1 # [ 164.891810] nix-daemon[871]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)7112worker1 # [ 166.116316] nix-daemon[871]: reaped child process 2142, status = succeeded7113worker1 # [ 166.274716] nix-daemon[871]: accepted connection from pid 2158, user nix-grpc-daemon (trusted)7114worker1 # [ 166.286827] nix-daemon[871]: reaped child process 2161, status = succeeded7115lb # [ 166.279934] niks3-server[1158]: 2026/09/21 13:47:41 INFO Received uploads request method=POST path=/api/pending_closures7116worker1 # [ 166.306970] nix-grpc-daemon[1837]: time=2026-09-21T13:47:40.534Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7117worker1 # [ 166.308493] nix-grpc-daemon[1837]: time=2026-09-21T13:47:40.536Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7118lb # [ 166.315191] niks3-server[1158]: 2026/09/21 13:47:41 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7119lb # [ 166.324871] niks3-server[1158]: 2026/09/21 13:47:41 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7120lb # [ 166.338134] niks3-server[1158]: 2026/09/21 13:47:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7121lb # [ 166.342846] niks3-server[1158]: 2026/09/21 13:47:41 INFO Signed narinfos id=69 count=17122worker1 # [ 166.364308] nix-grpc-daemon[1837]: time=2026-09-21T13:47:40.592Z level=INFO msg="Uploading 1 narinfos"7123lb # [ 166.347582] niks3-server[1158]: 2026/09/21 13:47:41 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7124lb # [ 166.356873] niks3-server[1158]: 2026/09/21 13:47:41 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7125lb # [ 166.361417] niks3-server[1158]: 2026/09/21 13:47:41 INFO Completed upload id=697126lb # [ 166.364443] niks3-server[1158]: 2026/09/21 13:47:41 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7127worker1 # [ 166.382185] nix-grpc-daemon[1837]: time=2026-09-21T13:47:40.609Z level=INFO msg="Upload complete. (280ms)"7128worker1 # [ 166.387617] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42106 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=15 outputs=17129worker1 # [ 166.400613] nix-daemon[871]: reaped child process 2140, status = succeeded7130client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 18.51 seconds)7131worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7132worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7133client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'7134client # [ 166.530369] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7135client # Running as unit: gpu.service; invocation ID: 89044b85eb3c4485af4869b3b1c43d897136client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.06 seconds)7137lb: must succeed: curl -sf localhost:9901/clusters7138lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7139lb: must succeed: curl -sf localhost:9901/clusters7140lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7141worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7142worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7143worker1 # [ 166.771767] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:421067144client # [ 166.746596] nix[1376]: don't know how to build these paths:7145client # [ 166.747491] nix[1376]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7146worker1 # [ 166.779166] nix-grpc-daemon[1777]: ts=2026-09-21T13:47:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421067147client # [ 166.768946] nix[1376]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker7148worker2 # [ 166.768428] nix-grpc-daemon[1469]: ts=2026-09-21T13:47:41Z level=debug event=want client=6 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07149lb # [ 167.411571] envoy[914]: conn peer=[2001:db8:1::1]:43330 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27098 tx=11402 ms=182617150lb: must succeed: curl -sf localhost:9901/clusters7151lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7152lb: must succeed: curl -sf localhost:9901/clusters7153lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7154worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7155worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7156client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7157client # [ 196.771466] nix[1376]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}7158client # [ 196.777889] nix[1376]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7159client # [ 196.813427] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7160client # [ 196.815833] systemd[1]: gpu.service: Failed with result 'exit-code'.7161client # [ 196.819262] systemd[1]: gpu.service: Consumed 127ms CPU time over 30.283s wall clock time, 35.4M memory peak, 5.1K incoming IP traffic, 5.1K outgoing IP traffic.7162lb # [ 197.411800] envoy[914]: conn peer=[2001:db8:1::1]:40984 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300517163client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.61 seconds)7164client: must succeed: journalctl -u gpu -o cat7165client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)7166lb: must succeed: curl -sf localhost:9901/clusters7167lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7168lb: must succeed: curl -sf localhost:9901/clusters7169lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7170worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7171worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7172(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 49.86 seconds)7173subtest: low disk drains a worker and builds go to the other7174worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7175worker1 # [ 198.667119] nix-grpc-daemon[1777]: ts=2026-09-21T13:48:12Z level=info event=unhealthy reason=min_free7176worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.95 seconds)7177worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7178worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.05 seconds)7179client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&27180worker1 # [ 199.011201] nix-grpc-daemon[1777]: ts=2026-09-21T13:48:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:421067181client # don't know how to build these paths:7182client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7183worker1 # [ 199.019303] nix-grpc-daemon[1777]: ts=2026-09-21T13:48:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421067184worker2 # [ 199.008566] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:13Z level=debug event=want client=7 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07185worker2 # [ 199.012799] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:13Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17186worker2 # [ 199.050390] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483567187worker2 # [ 199.061623] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:483567188client # copying 1 paths to 'lb:50051'...7189client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7190worker2 # [ 199.072729] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:483567191worker2 # [ 199.084103] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)7192worker2 # [ 199.290181] nix-daemon[856]: accepted connection from pid 1764, user nix-grpc-daemon (trusted)7193worker2 # [ 199.304555] nix-daemon[856]: reaped child process 1767, status = succeeded7194worker2 # [ 199.479512] nix-daemon[856]: accepted connection from pid 1769, user nix-grpc-daemon (trusted)7195worker2 # [ 199.494937] nix-daemon[856]: reaped child process 1772, status = succeeded7196lb # [ 199.524923] niks3-server[1158]: 2026/09/21 13:48:14 INFO Received uploads request method=POST path=/api/pending_closures7197worker2 # [ 199.521083] nix-grpc-daemon[1547]: time=2026-09-21T13:48:14.401Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7198worker2 # [ 199.523058] nix-grpc-daemon[1547]: time=2026-09-21T13:48:14.403Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7199lb # [ 199.558504] niks3-server[1158]: 2026/09/21 13:48:14 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7200lb # [ 199.570317] niks3-server[1158]: 2026/09/21 13:48:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7201lb # [ 199.575490] niks3-server[1158]: 2026/09/21 13:48:14 INFO Signed narinfos id=70 count=17202worker2 # [ 199.565868] nix-grpc-daemon[1547]: time=2026-09-21T13:48:14.445Z level=INFO msg="Uploading 1 narinfos"7203lb # [ 199.585874] niks3-server[1158]: 2026/09/21 13:48:14 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7204lb # [ 199.594061] niks3-server[1158]: 2026/09/21 13:48:14 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7205worker2 # [ 199.586326] nix-grpc-daemon[1547]: time=2026-09-21T13:48:14.466Z level=INFO msg="Upload complete. (488ms)"7206lb # [ 199.599504] niks3-server[1158]: 2026/09/21 13:48:14 INFO Completed upload id=707207worker2 # [ 199.588563] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=0 paths=1 nar_bytes_in=5127208lb # [ 199.604235] niks3-server[1158]: 2026/09/21 13:48:14 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7209worker2 # [ 199.593560] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483567210worker2 # [ 199.600437] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)7211worker2 # [ 199.602984] nix-daemon[856]: reaped child process 1762, status = succeeded7212worker2 # [ 199.608643] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)7213client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7214client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7215client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7216client # farm-a> LOG-farm-a-drain7217worker2 # [ 200.837925] nix-daemon[856]: reaped child process 1776, status = succeeded7218worker2 # [ 201.024868] nix-daemon[856]: accepted connection from pid 1792, user nix-grpc-daemon (trusted)7219worker2 # [ 201.037523] nix-daemon[856]: reaped child process 1795, status = succeeded7220lb # [ 201.068189] niks3-server[1158]: 2026/09/21 13:48:15 INFO Received uploads request method=POST path=/api/pending_closures7221worker2 # [ 201.064580] nix-grpc-daemon[1547]: time=2026-09-21T13:48:15.944Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7222worker2 # [ 201.066143] nix-grpc-daemon[1547]: time=2026-09-21T13:48:15.946Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7223lb # [ 201.101688] niks3-server[1158]: 2026/09/21 13:48:16 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7224worker2 # [ 201.091510] nix-grpc-daemon[1547]: time=2026-09-21T13:48:15.971Z level=INFO msg="Uploading 1 narinfos"7225lb # [ 201.103979] niks3-server[1158]: 2026/09/21 13:48:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7226lb # [ 201.106060] niks3-server[1158]: 2026/09/21 13:48:16 INFO Signed narinfos id=71 count=17227lb # [ 201.109556] niks3-server[1158]: 2026/09/21 13:48:16 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7228lb # [ 201.113815] niks3-server[1158]: 2026/09/21 13:48:16 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7229lb # [ 201.115994] niks3-server[1158]: 2026/09/21 13:48:16 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7230lb # [ 201.119975] niks3-server[1158]: 2026/09/21 13:48:16 INFO Completed upload id=717231worker2 # [ 201.106858] nix-grpc-daemon[1547]: time=2026-09-21T13:48:15.987Z level=INFO msg="Upload complete. (290ms)"7232lb # [ 201.121635] niks3-server[1158]: 2026/09/21 13:48:16 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7233worker2 # [ 201.109536] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=16 outputs=17234worker2 # [ 201.116895] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:15Z level=debug event=want client=7 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07235worker2 # [ 201.118940] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:15Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17236worker2 # [ 201.123960] nix-daemon[856]: reaped child process 1774, status = succeeded7237worker2 # [ 201.161303] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483567238worker2 # [ 201.172593] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:483567239worker2 # [ 201.183705] nix-grpc-daemon[1547]: time=2026-09-21T13:48:16.064Z level=INFO msg="All 1 paths already cached"7240client # copying 1 paths to 'lb:50051'...7241client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7242worker2 # [ 201.191143] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:483567243worker2 # [ 201.198275] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)7244worker2 # [ 201.397150] nix-daemon[856]: accepted connection from pid 1801, user nix-grpc-daemon (trusted)7245worker2 # [ 201.412877] nix-daemon[856]: reaped child process 1804, status = succeeded7246worker2 # [ 201.594600] nix-daemon[856]: accepted connection from pid 1806, user nix-grpc-daemon (trusted)7247worker2 # [ 201.608429] nix-daemon[856]: reaped child process 1809, status = succeeded7248lb # [ 201.638766] niks3-server[1158]: 2026/09/21 13:48:16 INFO Received uploads request method=POST path=/api/pending_closures7249worker2 # [ 201.629654] nix-grpc-daemon[1547]: time=2026-09-21T13:48:16.510Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7250worker2 # [ 201.631732] nix-grpc-daemon[1547]: time=2026-09-21T13:48:16.511Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7251lb # [ 201.653823] niks3-server[1158]: 2026/09/21 13:48:16 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7252lb # [ 201.656745] niks3-server[1158]: 2026/09/21 13:48:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7253lb # [ 201.659255] niks3-server[1158]: 2026/09/21 13:48:16 INFO Signed narinfos id=72 count=17254worker2 # [ 201.646418] nix-grpc-daemon[1547]: time=2026-09-21T13:48:16.526Z level=INFO msg="Uploading 1 narinfos"7255lb # [ 201.661501] niks3-server[1158]: 2026/09/21 13:48:16 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7256lb # [ 201.666331] niks3-server[1158]: 2026/09/21 13:48:16 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7257lb # [ 201.669732] niks3-server[1158]: 2026/09/21 13:48:16 INFO Completed upload id=727258worker2 # [ 201.657816] nix-grpc-daemon[1547]: time=2026-09-21T13:48:16.538Z level=INFO msg="Upload complete. (448ms)"7259lb # [ 201.671453] niks3-server[1158]: 2026/09/21 13:48:16 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7260worker2 # [ 201.660069] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=0 paths=1 nar_bytes_in=6247261worker2 # [ 201.662850] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483567262worker2 # [ 201.671085] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)7263worker2 # [ 201.673496] nix-daemon[856]: reaped child process 1799, status = succeeded7264client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7265worker2 # [ 201.679084] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)7266client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7267client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7268worker2 # [ 202.493332] nix-daemon[856]: reaped child process 1813, status = succeeded7269client # farm-b> LOG-farm-b-drain7270worker2 # [ 202.666747] nix-daemon[856]: accepted connection from pid 1830, user nix-grpc-daemon (trusted)7271worker2 # [ 202.681302] nix-daemon[856]: reaped child process 1833, status = succeeded7272lb # [ 202.712829] niks3-server[1158]: 2026/09/21 13:48:17 INFO Received uploads request method=POST path=/api/pending_closures7273worker2 # [ 202.708348] nix-grpc-daemon[1547]: time=2026-09-21T13:48:17.588Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7274worker2 # [ 202.711443] nix-grpc-daemon[1547]: time=2026-09-21T13:48:17.590Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7275lb # [ 202.757907] niks3-server[1158]: 2026/09/21 13:48:17 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7276lb # [ 202.763879] niks3-server[1158]: 2026/09/21 13:48:17 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7277lb # [ 202.774201] niks3-server[1158]: 2026/09/21 13:48:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7278lb # [ 202.781444] niks3-server[1158]: 2026/09/21 13:48:17 INFO Signed narinfos id=73 count=17279worker2 # [ 202.768996] nix-grpc-daemon[1547]: time=2026-09-21T13:48:17.648Z level=INFO msg="Uploading 1 narinfos"7280lb # [ 202.785085] niks3-server[1158]: 2026/09/21 13:48:17 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7281lb # [ 202.794713] niks3-server[1158]: 2026/09/21 13:48:17 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7282lb # [ 202.799500] niks3-server[1158]: 2026/09/21 13:48:17 INFO Completed upload id=737283worker2 # [ 202.786600] nix-grpc-daemon[1547]: time=2026-09-21T13:48:17.666Z level=INFO msg="Upload complete. (301ms)"7284lb # [ 202.802078] niks3-server[1158]: 2026/09/21 13:48:17 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7285worker2 # [ 202.796139] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=17 outputs=17286worker2 # [ 202.804680] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:17Z level=debug event=want client=7 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07287worker2 # [ 202.808747] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:17Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17288worker2 # [ 202.812289] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483567289worker2 # [ 202.821785] nix-daemon[856]: reaped child process 1811, status = succeeded7290worker2 # [ 202.824130] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:483567291worker2 # [ 202.831780] nix-grpc-daemon[1547]: time=2026-09-21T13:48:17.712Z level=INFO msg="All 2 paths already cached"7292client # copying 1 paths to 'lb:50051'...7293client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7294worker2 # [ 202.836917] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:483567295worker2 # [ 202.841795] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)7296worker2 # [ 203.043828] nix-daemon[856]: accepted connection from pid 1839, user nix-grpc-daemon (trusted)7297worker2 # [ 203.058792] nix-daemon[856]: reaped child process 1842, status = succeeded7298worker2 # [ 203.217668] nix-daemon[856]: accepted connection from pid 1844, user nix-grpc-daemon (trusted)7299worker2 # [ 203.231463] nix-daemon[856]: reaped child process 1847, status = succeeded7300lb # [ 203.257682] niks3-server[1158]: 2026/09/21 13:48:18 INFO Received uploads request method=POST path=/api/pending_closures7301worker2 # [ 203.251731] nix-grpc-daemon[1547]: time=2026-09-21T13:48:18.132Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7302worker2 # [ 203.253355] nix-grpc-daemon[1547]: time=2026-09-21T13:48:18.133Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7303lb # [ 203.283934] niks3-server[1158]: 2026/09/21 13:48:18 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7304lb # [ 203.296097] niks3-server[1158]: 2026/09/21 13:48:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7305worker2 # [ 203.287471] nix-grpc-daemon[1547]: time=2026-09-21T13:48:18.167Z level=INFO msg="Uploading 1 narinfos"7306lb # [ 203.301115] niks3-server[1158]: 2026/09/21 13:48:18 INFO Signed narinfos id=74 count=17307lb # [ 203.307777] niks3-server[1158]: 2026/09/21 13:48:18 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7308lb # [ 203.318595] niks3-server[1158]: 2026/09/21 13:48:18 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7309worker2 # [ 203.311301] nix-grpc-daemon[1547]: time=2026-09-21T13:48:18.191Z level=INFO msg="Upload complete. (462ms)"7310lb # [ 203.324403] niks3-server[1158]: 2026/09/21 13:48:18 INFO Completed upload id=747311worker2 # [ 203.313570] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=0 paths=1 nar_bytes_in=7527312lb # [ 203.327523] niks3-server[1158]: 2026/09/21 13:48:18 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7313worker2 # [ 203.318581] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:483567314worker2 # [ 203.325490] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)7315worker2 # [ 203.327899] nix-daemon[856]: reaped child process 1837, status = succeeded7316worker2 # [ 203.333666] nix-daemon[856]: accepted connection from pid 1469, user nix-grpc-daemon (trusted)7317client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7318client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7319client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7320client # farm-top> LOG-farm-top-drain7321worker2 # [ 204.158746] nix-daemon[856]: reaped child process 1851, status = succeeded7322worker2 # [ 204.344105] nix-daemon[856]: accepted connection from pid 1867, user nix-grpc-daemon (trusted)7323worker2 # [ 204.356731] nix-daemon[856]: reaped child process 1870, status = succeeded7324lb # [ 204.388868] niks3-server[1158]: 2026/09/21 13:48:19 INFO Received uploads request method=POST path=/api/pending_closures7325worker2 # [ 204.384072] nix-grpc-daemon[1547]: time=2026-09-21T13:48:19.264Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7326worker2 # [ 204.385589] nix-grpc-daemon[1547]: time=2026-09-21T13:48:19.265Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7327lb # [ 204.422834] niks3-server[1158]: 2026/09/21 13:48:19 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7328lb # [ 204.431634] niks3-server[1158]: 2026/09/21 13:48:19 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7329lb # [ 204.443472] niks3-server[1158]: 2026/09/21 13:48:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7330worker2 # [ 204.435774] nix-grpc-daemon[1547]: time=2026-09-21T13:48:19.315Z level=INFO msg="Uploading 1 narinfos"7331lb # [ 204.448615] niks3-server[1158]: 2026/09/21 13:48:19 INFO Signed narinfos id=75 count=17332lb # [ 204.454398] niks3-server[1158]: 2026/09/21 13:48:19 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7333lb # [ 204.463316] niks3-server[1158]: 2026/09/21 13:48:19 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7334lb # [ 204.469388] niks3-server[1158]: 2026/09/21 13:48:19 INFO Completed upload id=757335worker2 # [ 204.456541] nix-grpc-daemon[1547]: time=2026-09-21T13:48:19.336Z level=INFO msg="Upload complete. (316ms)"7336lb # [ 204.471823] niks3-server[1158]: 2026/09/21 13:48:19 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7337worker2 # [ 204.464321] nix-grpc-daemon[1469]: ts=2026-09-21T13:48:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48356 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=18 outputs=17338worker2 # [ 204.477591] nix-daemon[856]: reaped child process 1849, status = succeeded7339client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 5.73 seconds)7340client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7341client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.18 seconds)7342worker1: must succeed: rm /nix/.rw-store/fill7343worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.13 seconds)7344worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7345worker1 # [ 205.670750] nix-grpc-daemon[1777]: ts=2026-09-21T13:48:19Z level=info event=healthy reason=min_free7346worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.11 seconds)7347(finished: subtest: low disk drains a worker and builds go to the other, in 8.24 seconds)7348(finished: run the VM test script, in 206.87 seconds)7349test script finished in 207.12s7350cleanup7351kill QemuMachine (pid 50)7352client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7353kill QemuMachine (pid 49)7354lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7355kill QemuMachine (pid 51)7356worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7357kill QemuMachine (pid 52)7358worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7359(finished: cleanup, in 0.99 seconds)7360additionally exposed symbols:7361 client, lb, worker1, worker2,7362 vlan1,7363 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh