vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #470
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12lb # Disk image does not exist, creating the virtualisation disk image...13lb: QEMU running (pid 51)14worker2 # Disk image does not exist, creating the virtualisation disk image...15worker2: QEMU running (pid 52)16worker2 # Formatting '/build/vm-state-worker2/tmp.gFgJdeTFcf', fmt=raw size=107374182417lb # Formatting '/build/vm-state-lb/tmp.a2yHzCg6Li', fmt=raw size=107374182418worker2 # mke2fs 1.47.4 (6-Mar-2025)19worker1 # Disk image does not exist, creating the virtualisation disk image...20lb # mke2fs 1.47.4 (6-Mar-2025)21worker1: QEMU running (pid 50)22worker2 # Discarding device blocks: 0/262144 done23worker1 # Formatting '/build/vm-state-worker1/tmp.JQ5XuSMymk', fmt=raw size=107374182424client: QEMU running (pid 49)25worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes26worker1 # mke2fs 1.47.4 (6-Mar-2025)27lb # Discarding device blocks: 0/262144 done28client # Disk image does not exist, creating the virtualisation disk image...29worker2 # Filesystem UUID: 093e4d20-0920-49d5-b536-615a67d9553830worker1 # Discarding device blocks: 0/262144 done31client # Formatting '/build/vm-state-client/tmp.ZFcPyExDdc', fmt=raw size=107374182432worker2 # Superblock backups stored on blocks:33worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes34client # mke2fs 1.47.4 (6-Mar-2025)35worker2 # 32768, 98304, 163840, 22937636worker1 # Filesystem UUID: 1728a11a-dfe7-4837-af6f-483421b73e9437client # Discarding device blocks: 0/262144 done38worker2 # 39worker1 # Superblock backups stored on blocks:40client # Creating filesystem with 262144 4k blocks and 65536 inodes41worker2 # Allocating group tables: 0/8 done42worker1 # 32768, 98304, 163840, 22937643client # Filesystem UUID: 633674e2-1c98-4bfb-a29a-14109e91e82044worker2 # Writing inode tables: 0/8 done45worker1 # 46client # Superblock backups stored on blocks:47worker2 # Creating journal (8192 blocks): done48worker1 # Allocating group tables: 0/8 done49client # 32768, 98304, 163840, 22937650worker2 # Writing superblocks and filesystem accounting information: 0/8 done51worker1 # Writing inode tables: 0/8 done52client # 53worker2 # 54worker1 # Creating journal (8192 blocks): done55client # Allocating group tables: 0/8 done56worker2 # Virtualisation disk image created.57worker1 # Writing superblocks and filesystem accounting information: 0/8 done58client # Writing inode tables: 0/8 done59lb # Creating filesystem with 262144 4k blocks and 65536 inodes60(finished: start all VMs, in 0.19 seconds)61worker1 # 62client # Creating journal (8192 blocks): done63lb # Filesystem UUID: 56c0f477-95d7-4b53-bc5b-01d73112d80e64client: waiting for unit network-addresses-eth1.service65worker1 # Virtualisation disk image created.66client # Writing superblocks and filesystem accounting information: 0/8 done67lb # Superblock backups stored on blocks:68client # 69lb # 32768, 98304, 163840, 22937670client # Virtualisation disk image created.71lb # 72client: waiting for the VM to finish booting73lb # Allocating group tables: 0/8 done74lb # Writing inode tables: 0/8 done75lb # Creating journal (8192 blocks): done76lb # Writing superblocks and filesystem accounting information: 0/8 done77lb # 78lb # Virtualisation disk image created.79client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83client # 84client # 85client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0086client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...87client # 88client # 89client # 90client # 91client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB0092lb # 93lb # 94lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0095client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...96client # 97client # 98lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...99lb # 100lb # 101lb # 102lb # 103worker1 # 104worker1 # 105lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00106worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00107worker2 # 108worker2 # 109worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00110worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...111worker1 # 112worker1 # 113worker1 # 114worker1 # 115lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...116lb # 117lb # 118worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00119worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...120worker2 # 121worker2 # 122worker2 # 123worker2 # 124worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00125worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...126worker1 # 127worker1 # 128worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129worker2 # 130worker2 # 131client # Booting from ROM...132lb # Booting from ROM...133worker1 # Booting from ROM...134worker2 # Booting from ROM...135client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026137client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/iwfnwmr510zyiiapzic7q3y408r3ypyx-nixos-system-client-test/init regInfo=/nix/store/9balbfgs3w3ldwc5x160r8h4fh1s60j0-closure-info/registration console=ttyS0,115200n8 console=tty0138lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/dlr5gkwdzlpfiks0wm9k3cgbj9l1v40b-nixos-system-lb-test/init regInfo=/nix/store/493p756s0f7jvdvcriannaqb5380jad0-closure-info/registration console=ttyS0,115200n8 console=tty0139client # [ 0.000000] BIOS-provided physical RAM map:140lb # [ 0.000000] BIOS-provided physical RAM map:141client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable142lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable143client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved144lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved145client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved146lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved147client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable148lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable149client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved150lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved151client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved152lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved153client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved154lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved155client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved156lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved157worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026158client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved159lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved160client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved161lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved162client # [ 0.000000] NX (Execute Disable) protection: active163lb # [ 0.000000] NX (Execute Disable) protection: active164client # [ 0.000000] APIC: Static calls initialized165lb # [ 0.000000] APIC: Static calls initialized166client # [ 0.000000] SMBIOS 2.8 present.167lb # [ 0.000000] SMBIOS 2.8 present.168worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/4p0rsf301ngzk6ydvvpd55qzgfl3sk2k-nixos-system-worker2-test/init regInfo=/nix/store/wsi5ixxy1rl93hi0l57rxfvizfzxgnaz-closure-info/registration console=ttyS0,115200n8 console=tty0169worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026170client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014171worker2 # [ 0.000000] BIOS-provided physical RAM map:172lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014173client # [ 0.000000] DMI: Memory slots populated: 1/1174lb # [ 0.000000] DMI: Memory slots populated: 1/1175client # [ 0.000000] Hypervisor detected: KVM176worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable177lb # [ 0.000000] Hypervisor detected: KVM178client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000179lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000180worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved181client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00182lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00183worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved184client # [ 0.000001] kvm-clock: using sched offset of 492254949 cycles185lb # [ 0.000000] kvm-clock: using sched offset of 474189268 cycles186worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable187worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/sv46bqjv56hpwfdlfbiqqk75c9wn49ss-nixos-system-worker1-test/init regInfo=/nix/store/9svqxhrgyf4k3m7ih0lqx23pv9hl752c-closure-info/registration console=ttyS0,115200n8 console=tty0188worker1 # [ 0.000000] BIOS-provided physical RAM map:189worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved190client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns191lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns192client # [ 0.000004] tsc: Detected 2400.012 MHz processor193worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable194lb # [ 0.000005] tsc: Detected 2400.012 MHz processor195worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved196client # [ 0.000811] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000197lb # [ 0.000811] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000198worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved199worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved200client # [ 0.000837] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs201worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved202lb # [ 0.000837] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs203worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved204client # [ 0.000840] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT205worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable206lb # [ 0.000840] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT207worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved208client # [ 0.002746] found SMP MP-table at [mem 0x000f5450-0x000f545f]209lb # [ 0.002729] found SMP MP-table at [mem 0x000f5450-0x000f545f]210worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved211worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved212client # [ 0.002757] Using GB pages for direct mapping213lb # [ 0.002740] Using GB pages for direct mapping214worker2 # [ 0.000000] NX (Execute Disable) protection: active215worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved216client # [ 0.002850] RAMDISK: [mem 0x5e366000-0x5ffcffff]217lb # [ 0.002910] RAMDISK: [mem 0x5e367000-0x5ffcffff]218worker2 # [ 0.000000] APIC: Static calls initialized219client # [ 0.002858] ACPI: Early table checksum verification disabled220worker2 # [ 0.000000] SMBIOS 2.8 present.221lb # [ 0.002919] ACPI: Early table checksum verification disabled222worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved223client # [ 0.002861] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )224lb # [ 0.002921] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )225worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved226worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014227client # [ 0.002865] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)228worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved229lb # [ 0.002925] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)230worker2 # [ 0.000000] DMI: Memory slots populated: 1/1231worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved232worker2 # [ 0.000000] Hypervisor detected: KVM233worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000234client # [ 0.002869] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)235lb # [ 0.002929] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)236worker1 # [ 0.000000] NX (Execute Disable) protection: active237worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00238worker1 # [ 0.000000] APIC: Static calls initialized239client # [ 0.002876] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)240lb # [ 0.002936] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)241worker1 # [ 0.000000] SMBIOS 2.8 present.242worker2 # [ 0.000001] kvm-clock: using sched offset of 514593398 cycles243client # [ 0.002878] ACPI: FACS 0x000000005FFE0000 000040244lb # [ 0.002938] ACPI: FACS 0x000000005FFE0000 000040245worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014246client # [ 0.002880] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)247worker2 # [ 0.000003] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns248lb # [ 0.002939] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)249worker1 # [ 0.000000] DMI: Memory slots populated: 1/1250worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor251worker1 # [ 0.000000] Hypervisor detected: KVM252client # [ 0.002881] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)253lb # [ 0.002941] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)254worker2 # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000255worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000256client # [ 0.002883] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)257worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00258lb # [ 0.002943] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)259worker2 # [ 0.000839] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs260worker1 # [ 0.000001] kvm-clock: using sched offset of 487999109 cycles261worker2 # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT262client # [ 0.002884] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)263lb # [ 0.002944] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)264worker2 # [ 0.002742] found SMP MP-table at [mem 0x000f5450-0x000f545f]265client # [ 0.002886] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]266worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns267lb # [ 0.002945] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]268worker2 # [ 0.002753] Using GB pages for direct mapping269worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor270client # [ 0.002887] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]271worker2 # [ 0.002861] RAMDISK: [mem 0x5e367000-0x5ffcffff]272lb # [ 0.002946] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]273worker1 # [ 0.000804] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000274worker2 # [ 0.002870] ACPI: Early table checksum verification disabled275client # [ 0.002887] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]276lb # [ 0.002947] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]277worker2 # [ 0.002873] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )278client # [ 0.002888] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]279worker1 # [ 0.000832] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs280lb # [ 0.002948] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]281worker1 # [ 0.000834] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT282worker2 # [ 0.002877] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)283client # [ 0.002888] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]284lb # [ 0.002948] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]285worker1 # [ 0.002725] found SMP MP-table at [mem 0x000f5450-0x000f545f]286client # [ 0.002889] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]287worker2 # [ 0.002881] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)288lb # [ 0.002948] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]289worker1 # [ 0.002737] Using GB pages for direct mapping290client # [ 0.002889] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]291worker1 # [ 0.002824] RAMDISK: [mem 0x5e367000-0x5ffcffff]292worker2 # [ 0.002888] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)293lb # [ 0.002949] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]294client # [ 0.003116] No NUMA configuration found295worker1 # [ 0.002834] ACPI: Early table checksum verification disabled296lb # [ 0.003176] No NUMA configuration found297worker2 # [ 0.002890] ACPI: FACS 0x000000005FFE0000 000040298client # [ 0.003117] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]299worker1 # [ 0.002837] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )300lb # [ 0.003176] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]301worker2 # [ 0.002891] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)302client # [ 0.003119] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]303lb # [ 0.003179] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]304client # [ 0.005524] Zone ranges:305worker1 # [ 0.002841] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)306lb # [ 0.005507] Zone ranges:307worker2 # [ 0.002893] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)308client # [ 0.005526] DMA [mem 0x0000000000001000-0x0000000000ffffff]309lb # [ 0.005508] DMA [mem 0x0000000000001000-0x0000000000ffffff]310worker1 # [ 0.002845] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)311client # [ 0.005527] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]312worker2 # [ 0.002894] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)313lb # [ 0.005509] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]314client # [ 0.005528] Normal empty315worker1 # [ 0.002853] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)316lb # [ 0.005511] Normal empty317client # [ 0.005529] Device empty318lb # [ 0.005511] Device empty319worker2 # [ 0.002896] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)320worker1 # [ 0.002855] ACPI: FACS 0x000000005FFE0000 000040321client # [ 0.005530] Movable zone start for each node322lb # [ 0.005512] Movable zone start for each node323client # [ 0.005531] Early memory node ranges324worker2 # [ 0.002897] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]325lb # [ 0.005513] Early memory node ranges326worker1 # [ 0.002856] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)327client # [ 0.005531] node 0: [mem 0x0000000000001000-0x000000000009efff]328lb # [ 0.005513] node 0: [mem 0x0000000000001000-0x000000000009efff]329worker2 # [ 0.002898] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]330worker1 # [ 0.002858] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)331client # [ 0.005532] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]332lb # [ 0.005514] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]333worker2 # [ 0.002899] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]334client # [ 0.005533] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]335worker1 # [ 0.002859] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)336lb # [ 0.005515] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]337worker2 # [ 0.002899] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]338client # [ 0.005554] On node 0, zone DMA: 1 pages in unavailable ranges339lb # [ 0.005534] On node 0, zone DMA: 1 pages in unavailable ranges340worker1 # [ 0.002861] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)341worker2 # [ 0.002900] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]342client # [ 0.005837] On node 0, zone DMA: 97 pages in unavailable ranges343lb # [ 0.005809] On node 0, zone DMA: 97 pages in unavailable ranges344worker2 # [ 0.002900] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]345worker1 # [ 0.002862] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]346client # [ 0.034089] On node 0, zone DMA32: 40 pages in unavailable ranges347lb # [ 0.032957] On node 0, zone DMA32: 40 pages in unavailable ranges348client # [ 0.034550] ACPI: PM-Timer IO Port: 0x608349lb # [ 0.033413] ACPI: PM-Timer IO Port: 0x608350worker1 # [ 0.002863] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]351worker2 # [ 0.002901] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]352client # [ 0.034561] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])353worker2 # [ 0.003123] No NUMA configuration found354lb # [ 0.033423] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])355worker1 # [ 0.002864] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]356client # [ 0.034591] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23357worker2 # [ 0.003124] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]358lb # [ 0.033449] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23359worker1 # [ 0.002864] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]360worker2 # [ 0.003126] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]361client # [ 0.034594] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)362lb # [ 0.033451] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)363worker2 # [ 0.005765] Zone ranges:364worker1 # [ 0.002865] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]365client # [ 0.034596] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)366lb # [ 0.033453] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)367worker2 # [ 0.005766] DMA [mem 0x0000000000001000-0x0000000000ffffff]368worker1 # [ 0.002865] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]369client # [ 0.034597] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)370worker2 # [ 0.005768] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]371lb # [ 0.033454] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)372worker2 # [ 0.005769] Normal empty373worker1 # [ 0.002865] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]374worker2 # [ 0.005770] Device empty375client # [ 0.034598] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)376lb # [ 0.033455] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)377worker1 # [ 0.003097] No NUMA configuration found378worker2 # [ 0.005770] Movable zone start for each node379client # [ 0.034599] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)380worker2 # [ 0.005771] Early memory node ranges381worker1 # [ 0.003098] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]382lb # [ 0.033456] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)383client # [ 0.034601] ACPI: Using ACPI (MADT) for SMP configuration information384worker2 # [ 0.005771] node 0: [mem 0x0000000000001000-0x000000000009efff]385worker1 # [ 0.003101] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]386lb # [ 0.033458] ACPI: Using ACPI (MADT) for SMP configuration information387worker1 # [ 0.005499] Zone ranges:388client # [ 0.034602] ACPI: HPET id: 0x8086a201 base: 0xfed00000389worker2 # [ 0.005772] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]390lb # [ 0.033459] ACPI: HPET id: 0x8086a201 base: 0xfed00000391client # [ 0.034605] TSC deadline timer available392worker1 # [ 0.005500] DMA [mem 0x0000000000001000-0x0000000000ffffff]393lb # [ 0.033462] TSC deadline timer available394client # [ 0.034609] CPU topo: Max. logical packages: 1395worker2 # [ 0.005773] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]396lb # [ 0.033466] CPU topo: Max. logical packages: 1397worker1 # [ 0.005502] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]398client # [ 0.034610] CPU topo: Max. logical dies: 1399worker1 # [ 0.005503] Normal empty400worker2 # [ 0.005793] On node 0, zone DMA: 1 pages in unavailable ranges401lb # [ 0.033467] CPU topo: Max. logical dies: 1402worker1 # [ 0.005504] Device empty403client # [ 0.034611] CPU topo: Max. dies per package: 1404worker2 # [ 0.006058] On node 0, zone DMA: 97 pages in unavailable ranges405lb # [ 0.033467] CPU topo: Max. dies per package: 1406worker1 # [ 0.005504] Movable zone start for each node407client # [ 0.034614] CPU topo: Max. threads per core: 1408worker1 # [ 0.005505] Early memory node ranges409lb # [ 0.033470] CPU topo: Max. threads per core: 1410worker2 # [ 0.033275] On node 0, zone DMA32: 40 pages in unavailable ranges411client # [ 0.034614] CPU topo: Num. cores per package: 1412lb # [ 0.033471] CPU topo: Num. cores per package: 1413worker2 # [ 0.033739] ACPI: PM-Timer IO Port: 0x608414worker1 # [ 0.005505] node 0: [mem 0x0000000000001000-0x000000000009efff]415client # [ 0.034615] CPU topo: Num. threads per package: 1416lb # [ 0.033471] CPU topo: Num. threads per package: 1417worker2 # [ 0.033750] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])418worker1 # [ 0.005506] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]419client # [ 0.034615] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs420lb # [ 0.033472] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs421worker2 # [ 0.033775] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23422worker1 # [ 0.005507] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]423client # [ 0.034643] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()424lb # [ 0.033488] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()425worker2 # [ 0.033777] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)426worker1 # [ 0.005527] On node 0, zone DMA: 1 pages in unavailable ranges427client # [ 0.034678] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]428lb # [ 0.033519] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]429worker2 # [ 0.033779] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)430worker1 # [ 0.005804] On node 0, zone DMA: 97 pages in unavailable ranges431client # [ 0.034680] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]432worker1 # [ 0.034017] On node 0, zone DMA32: 40 pages in unavailable ranges433worker2 # [ 0.033780] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)434lb # [ 0.033520] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]435worker1 # [ 0.034475] ACPI: PM-Timer IO Port: 0x608436client # [ 0.034681] [mem 0x60000000-0xafffffff] available for PCI devices437worker2 # [ 0.033781] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)438lb # [ 0.033522] [mem 0x60000000-0xafffffff] available for PCI devices439client # [ 0.034682] Booting paravirtualized kernel on KVM440worker1 # [ 0.034487] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])441lb # [ 0.033523] Booting paravirtualized kernel on KVM442worker2 # [ 0.033782] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)443worker1 # [ 0.034516] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23444client # [ 0.034686] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns445worker2 # [ 0.033784] ACPI: Using ACPI (MADT) for SMP configuration information446lb # [ 0.033526] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns447worker1 # [ 0.034519] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)448worker2 # [ 0.033785] ACPI: HPET id: 0x8086a201 base: 0xfed00000449client # [ 0.039153] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1450worker2 # [ 0.033789] TSC deadline timer available451worker1 # [ 0.034521] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)452lb # [ 0.037978] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1453worker2 # [ 0.033794] CPU topo: Max. logical packages: 1454client # [ 0.041459] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152455worker1 # [ 0.034522] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)456lb # [ 0.040169] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152457worker2 # [ 0.033794] CPU topo: Max. logical dies: 1458client # [ 0.041505] kvm-guest: PV spinlocks disabled, single CPU459lb # [ 0.040213] kvm-guest: PV spinlocks disabled, single CPU460worker2 # [ 0.033795] CPU topo: Max. dies per package: 1461worker1 # [ 0.034523] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)462worker2 # [ 0.033798] CPU topo: Max. threads per core: 1463worker1 # [ 0.034524] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)464worker2 # [ 0.033799] CPU topo: Num. cores per package: 1465worker1 # [ 0.034526] ACPI: Using ACPI (MADT) for SMP configuration information466worker2 # [ 0.033799] CPU topo: Num. threads per package: 1467worker1 # [ 0.034527] ACPI: HPET id: 0x8086a201 base: 0xfed00000468worker2 # [ 0.033799] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs469worker1 # [ 0.034531] TSC deadline timer available470worker2 # [ 0.033818] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()471worker1 # [ 0.034536] CPU topo: Max. logical packages: 1472client # [ 0.041506] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/iwfnwmr510zyiiapzic7q3y408r3ypyx-nixos-system-client-test/init regInfo=/nix/store/9balbfgs3w3ldwc5x160r8h4fh1s60j0-closure-info/registration console=ttyS0,115200n8 console=tty0473lb # [ 0.040214] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/dlr5gkwdzlpfiks0wm9k3cgbj9l1v40b-nixos-system-lb-test/init regInfo=/nix/store/493p756s0f7jvdvcriannaqb5380jad0-closure-info/registration console=ttyS0,115200n8 console=tty0474worker1 # [ 0.034536] CPU topo: Max. logical dies: 1475worker2 # [ 0.033849] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]476worker1 # [ 0.034537] CPU topo: Max. dies per package: 1477worker2 # [ 0.033851] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]478client # [ 0.041603] Unknown kernel command line parameters "regInfo=/nix/store/9balbfgs3w3ldwc5x160r8h4fh1s60j0-closure-info/registration", will be passed to user space.479worker1 # [ 0.034540] CPU topo: Max. threads per core: 1480lb # [ 0.040306] Unknown kernel command line parameters "regInfo=/nix/store/493p756s0f7jvdvcriannaqb5380jad0-closure-info/registration", will be passed to user space.481client # [ 0.041615] random: crng init done482worker1 # [ 0.034541] CPU topo: Num. cores per package: 1483worker2 # [ 0.033852] [mem 0x60000000-0xafffffff] available for PCI devices484lb # [ 0.040318] random: crng init done485worker1 # [ 0.034541] CPU topo: Num. threads per package: 1486worker2 # [ 0.033854] Booting paravirtualized kernel on KVM487client # [ 0.041616] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes488lb # [ 0.040319] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes489worker1 # [ 0.034541] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs490client # [ 0.043879] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)491worker2 # [ 0.033857] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns492lb # [ 0.042487] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)493worker1 # [ 0.034561] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()494client # [ 0.045298] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)495worker2 # [ 0.038298] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1496worker1 # [ 0.034592] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]497lb # [ 0.043554] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)498client # [ 0.045330] Fallback order for Node 0: 0499lb # [ 0.043586] Fallback order for Node 0: 0500worker2 # [ 0.040492] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152501client # [ 0.045333] Built 1 zonelists, mobility grouping on. Total pages: 393078502worker1 # [ 0.034594] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]503client # [ 0.045334] Policy zone: DMA32504lb # [ 0.043589] Built 1 zonelists, mobility grouping on. Total pages: 393078505worker2 # [ 0.040532] kvm-guest: PV spinlocks disabled, single CPU506worker1 # [ 0.034595] [mem 0x60000000-0xafffffff] available for PCI devices507lb # [ 0.043590] Policy zone: DMA32508client # [ 0.047971] mem auto-init: stack:all(zero), heap alloc:on, heap free:off509worker1 # [ 0.034597] Booting paravirtualized kernel on KVM510lb # [ 0.046386] mem auto-init: stack:all(zero), heap alloc:on, heap free:off511client # [ 0.051309] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1512lb # [ 0.049466] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1513worker1 # [ 0.034600] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns514client # [ 0.054806] allocated 3145728 bytes of page_ext515lb # [ 0.052821] allocated 3145728 bytes of page_ext516client # [ 0.064922] ftrace: allocating 48731 entries in 192 pages517worker1 # [ 0.039053] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1518lb # [ 0.062706] ftrace: allocating 48731 entries in 192 pages519client # [ 0.064924] ftrace: allocated 192 pages with 2 groups520worker2 # [ 0.040533] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/4p0rsf301ngzk6ydvvpd55qzgfl3sk2k-nixos-system-worker2-test/init regInfo=/nix/store/wsi5ixxy1rl93hi0l57rxfvizfzxgnaz-closure-info/registration console=ttyS0,115200n8 console=tty0521client # [ 0.065821] Dynamic Preempt: lazy522lb # [ 0.062708] ftrace: allocated 192 pages with 2 groups523worker1 # [ 0.041327] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152524lb # [ 0.063572] Dynamic Preempt: lazy525client # [ 0.065993] rcu: Preemptible hierarchical RCU implementation.526worker1 # [ 0.041375] kvm-guest: PV spinlocks disabled, single CPU527lb # [ 0.063723] rcu: Preemptible hierarchical RCU implementation.528client # [ 0.065994] rcu: RCU event tracing is enabled.529lb # [ 0.063724] rcu: RCU event tracing is enabled.530worker2 # [ 0.040625] Unknown kernel command line parameters "regInfo=/nix/store/wsi5ixxy1rl93hi0l57rxfvizfzxgnaz-closure-info/registration", will be passed to user space.531client # [ 0.065995] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.532worker2 # [ 0.040639] random: crng init done533lb # [ 0.063724] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.534client # [ 0.065996] Trampoline variant of Tasks RCU enabled.535worker2 # [ 0.040639] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes536lb # [ 0.063726] Trampoline variant of Tasks RCU enabled.537client # [ 0.065997] Rude variant of Tasks RCU enabled.538lb # [ 0.063726] Rude variant of Tasks RCU enabled.539client # [ 0.065997] Tracing variant of Tasks RCU enabled.540worker2 # [ 0.042792] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)541lb # [ 0.063726] Tracing variant of Tasks RCU enabled.542worker1 # [ 0.041377] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/sv46bqjv56hpwfdlfbiqqk75c9wn49ss-nixos-system-worker1-test/init regInfo=/nix/store/9svqxhrgyf4k3m7ih0lqx23pv9hl752c-closure-info/registration console=ttyS0,115200n8 console=tty0543client # [ 0.065998] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.544worker2 # [ 0.043869] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)545worker2 # [ 0.043901] Fallback order for Node 0: 0546lb # [ 0.063727] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.547client # [ 0.065999] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1548worker2 # [ 0.043904] Built 1 zonelists, mobility grouping on. Total pages: 393078549lb # [ 0.063728] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1550worker1 # [ 0.041472] Unknown kernel command line parameters "regInfo=/nix/store/9svqxhrgyf4k3m7ih0lqx23pv9hl752c-closure-info/registration", will be passed to user space.551worker2 # [ 0.043905] Policy zone: DMA32552client # [ 0.066020] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.553worker1 # [ 0.041485] random: crng init done554lb # [ 0.063750] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.555worker2 # [ 0.046702] mem auto-init: stack:all(zero), heap alloc:on, heap free:off556worker1 # [ 0.041486] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes557client # [ 0.066022] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.558worker2 # [ 0.049884] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1559lb # [ 0.063752] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.560worker1 # [ 0.043731] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)561worker2 # [ 0.053205] allocated 3145728 bytes of page_ext562client # [ 0.066023] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.563worker2 # [ 0.063032] ftrace: allocating 48731 entries in 192 pages564lb # [ 0.063753] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.565worker1 # [ 0.045138] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)566client # [ 0.070752] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16567worker2 # [ 0.063034] ftrace: allocated 192 pages with 2 groups568worker1 # [ 0.045170] Fallback order for Node 0: 0569lb # [ 0.068163] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16570worker2 # [ 0.063864] Dynamic Preempt: lazy571client # [ 0.071039] rcu: srcu_init: Setting srcu_struct sizes based on contention.572worker1 # [ 0.045173] Built 1 zonelists, mobility grouping on. Total pages: 393078573lb # [ 0.068452] rcu: srcu_init: Setting srcu_struct sizes based on contention.574worker2 # [ 0.064012] rcu: Preemptible hierarchical RCU implementation.575worker1 # [ 0.045174] Policy zone: DMA32576worker2 # [ 0.064012] rcu: RCU event tracing is enabled.577client # [ 0.071047] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns578worker1 # [ 0.047790] mem auto-init: stack:all(zero), heap alloc:on, heap free:off579lb # [ 0.068458] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns580worker2 # [ 0.064013] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.581client # [ 0.071159] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)582worker1 # [ 0.051148] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1583worker2 # [ 0.064014] Trampoline variant of Tasks RCU enabled.584client # [ 0.074716] Console: colour VGA+ 80x25585worker1 # [ 0.054628] allocated 3145728 bytes of page_ext586lb # [ 0.068558] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)587worker2 # [ 0.064015] Rude variant of Tasks RCU enabled.588lb # [ 0.072095] Console: colour VGA+ 80x25589client # [ 0.074719] printk: legacy console [tty0] enabled590worker1 # [ 0.064667] ftrace: allocating 48731 entries in 192 pages591worker2 # [ 0.064015] Tracing variant of Tasks RCU enabled.592client # [ 0.104141] printk: legacy console [ttyS0] enabled593lb # [ 0.072098] printk: legacy console [tty0] enabled594worker1 # [ 0.064669] ftrace: allocated 192 pages with 2 groups595worker2 # [ 0.064016] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.596lb # [ 0.101513] printk: legacy console [ttyS0] enabled597worker1 # [ 0.065570] Dynamic Preempt: lazy598client # [ 0.200333] ACPI: Core revision 20250807599worker1 # [ 0.065759] rcu: Preemptible hierarchical RCU implementation.600worker2 # [ 0.064016] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1601lb # [ 0.198034] ACPI: Core revision 20250807602worker1 # [ 0.065759] rcu: RCU event tracing is enabled.603worker2 # [ 0.064065] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.604client # [ 0.201265] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns605worker1 # [ 0.065760] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.606worker1 # [ 0.065761] Trampoline variant of Tasks RCU enabled.607lb # [ 0.199258] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns608client # [ 0.202756] APIC: Switch to symmetric I/O mode setup609worker2 # [ 0.064067] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.610worker1 # [ 0.065762] Rude variant of Tasks RCU enabled.611client # [ 0.203723] x2apic enabled612worker1 # [ 0.065762] Tracing variant of Tasks RCU enabled.613lb # [ 0.200768] APIC: Switch to symmetric I/O mode setup614worker2 # [ 0.064068] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.615lb # [ 0.201749] x2apic enabled616worker2 # [ 0.068399] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16617worker1 # [ 0.065763] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.618client # [ 0.204462] APIC: Switched APIC routing to: physical x2apic619worker2 # [ 0.068686] rcu: srcu_init: Setting srcu_struct sizes based on contention.620worker1 # [ 0.065764] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1621lb # [ 0.202521] APIC: Switched APIC routing to: physical x2apic622worker1 # [ 0.065785] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.623client # [ 0.206267] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1624worker2 # [ 0.068691] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns625lb # [ 0.204332] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1626worker1 # [ 0.065787] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.627worker2 # [ 0.068795] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)628worker2 # [ 0.072273] Console: colour VGA+ 80x25629client # [ 0.207261] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns630worker1 # [ 0.065788] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.631worker2 # [ 0.072276] printk: legacy console [tty0] enabled632lb # [ 0.205306] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns633worker2 # [ 0.101585] printk: legacy console [ttyS0] enabled634worker1 # [ 0.070478] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16635client # [ 0.208937] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)636worker2 # [ 0.195762] ACPI: Core revision 20250807637worker1 # [ 0.070773] rcu: srcu_init: Setting srcu_struct sizes based on contention.638lb # [ 0.206973] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)639client # [ 0.210024] x86/cpu: User Mode Instruction Prevention (UMIP) activated640worker1 # [ 0.070780] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns641worker2 # [ 0.196625] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns642lb # [ 0.208056] x86/cpu: User Mode Instruction Prevention (UMIP) activated643client # [ 0.211063] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127644worker1 # [ 0.070894] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)645worker2 # [ 0.198110] APIC: Switch to symmetric I/O mode setup646lb # [ 0.209155] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127647client # [ 0.211934] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0648worker1 # [ 0.074419] Console: colour VGA+ 80x25649worker2 # [ 0.199040] x2apic enabled650worker1 # [ 0.074422] printk: legacy console [tty0] enabled651lb # [ 0.210642] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0652worker1 # [ 0.103763] printk: legacy console [ttyS0] enabled653worker2 # [ 0.199780] APIC: Switched APIC routing to: physical x2apic654client # [ 0.212938] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto655worker1 # [ 0.198982] ACPI: Core revision 20250807656lb # [ 0.210974] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto657client # [ 0.214934] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl658worker2 # [ 0.201596] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1659worker1 # [ 0.199893] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns660client # [ 0.215934] Transient Scheduler Attacks: Vulnerable: No microcode661lb # [ 0.212970] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl662worker1 # [ 0.201373] APIC: Switch to symmetric I/O mode setup663worker2 # [ 0.202550] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns664client # [ 0.216934] Spectre V2 : Mitigation: Enhanced / Automatic IBRS665lb # [ 0.213970] Transient Scheduler Attacks: Vulnerable: No microcode666worker1 # [ 0.202342] x2apic enabled667lb # [ 0.215668] Spectre V2 : Mitigation: Enhanced / Automatic IBRS668client # [ 0.217934] Speculative Return Stack Overflow: Mitigation: Safe RET669worker2 # [ 0.204192] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)670worker1 # [ 0.203123] APIC: Switched APIC routing to: physical x2apic671lb # [ 0.216652] Speculative Return Stack Overflow: Mitigation: Safe RET672worker2 # [ 0.205276] x86/cpu: User Mode Instruction Prevention (UMIP) activated673worker1 # [ 0.204881] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1674worker2 # [ 0.206309] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127675worker2 # [ 0.207190] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0676client # [ 0.218934] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization677worker1 # [ 0.205890] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns678client # [ 0.220942] Spectre V2 : Enabling IBPB for BPF679lb # [ 0.216970] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization680worker2 # [ 0.208193] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto681worker1 # [ 0.207558] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)682lb # [ 0.218977] Spectre V2 : Enabling IBPB for BPF683client # [ 0.221935] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier684worker1 # [ 0.208644] x86/cpu: User Mode Instruction Prevention (UMIP) activated685worker2 # [ 0.209189] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl686client # [ 0.222934] active return thunk: srso_alias_return_thunk687lb # [ 0.219971] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier688worker2 # [ 0.210190] Transient Scheduler Attacks: Vulnerable: No microcode689worker1 # [ 0.209697] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127690lb # [ 0.220971] active return thunk: srso_alias_return_thunk691client # [ 0.223956] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'692worker2 # [ 0.211189] Spectre V2 : Mitigation: Enhanced / Automatic IBRS693worker1 # [ 0.210556] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0694client # [ 0.224934] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'695lb # [ 0.222605] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'696worker2 # [ 0.212189] Speculative Return Stack Overflow: Mitigation: Safe RET697client # [ 0.225934] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'698worker1 # [ 0.211559] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto699lb # [ 0.223849] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'700client # [ 0.226934] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'701lb # [ 0.224705] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'702worker1 # [ 0.213556] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl703lb # [ 0.225709] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'704client # [ 0.227934] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'705worker1 # [ 0.214556] Transient Scheduler Attacks: Vulnerable: No microcode706lb # [ 0.226728] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'707client # [ 0.228934] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'708worker2 # [ 0.213189] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization709worker1 # [ 0.215555] Spectre V2 : Mitigation: Enhanced / Automatic IBRS710worker2 # [ 0.215196] Spectre V2 : Enabling IBPB for BPF711lb # [ 0.227693] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'712worker1 # [ 0.216555] Speculative Return Stack Overflow: Mitigation: Safe RET713client # [ 0.229934] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'714worker2 # [ 0.215895] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier715client # [ 0.231834] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256716lb # [ 0.228740] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'717worker2 # [ 0.217190] active return thunk: srso_alias_return_thunk718client # [ 0.232620] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64719lb # [ 0.229970] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256720client # [ 0.233647] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512721worker2 # [ 0.218210] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'722lb # [ 0.230970] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64723client # [ 0.234640] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024724worker2 # [ 0.219189] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'725lb # [ 0.231970] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512726worker1 # [ 0.217555] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization727worker1 # [ 0.219563] Spectre V2 : Enabling IBPB for BPF728client # [ 0.234934] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8729lb # [ 0.232872] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024730worker2 # [ 0.220189] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'731lb # [ 0.233673] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8732worker2 # [ 0.221190] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'733worker1 # [ 0.220556] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier734client # [ 0.235934] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.735worker2 # [ 0.222189] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'736worker1 # [ 0.221556] active return thunk: srso_alias_return_thunk737lb # [ 0.234672] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.738worker2 # [ 0.223189] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'739worker1 # [ 0.222577] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'740worker2 # [ 0.224189] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'741worker1 # [ 0.223555] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'742worker2 # [ 0.225190] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256743worker1 # [ 0.224555] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'744worker2 # [ 0.226189] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64745worker1 # [ 0.225555] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'746worker2 # [ 0.227189] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512747worker1 # [ 0.226555] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'748worker2 # [ 0.228189] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024749worker1 # [ 0.228344] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'750worker2 # [ 0.229189] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8751worker1 # [ 0.229373] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'752worker2 # [ 0.230189] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.753worker1 # [ 0.230555] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256754worker1 # [ 0.231532] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64755worker1 # [ 0.232555] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512756worker1 # [ 0.233554] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024757worker1 # [ 0.234555] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8758worker1 # [ 0.235555] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759lb # [ 0.269074] Freeing SMP alternatives memory: 44K760client # [ 0.272153] Freeing SMP alternatives memory: 44K761lb # [ 0.269861] pid_max: default: 32768 minimum: 301762client # [ 0.272938] pid_max: default: 32768 minimum: 301763lb # [ 0.270616] LSM: initializing lsm=capability,landlock,yama,bpf,ima764client # [ 0.273783] LSM: initializing lsm=capability,landlock,yama,bpf,ima765lb # [ 0.271071] landlock: Up and running.766client # [ 0.274941] landlock: Up and running.767lb # [ 0.272415] Yama: becoming mindful.768client # [ 0.275563] Yama: becoming mindful.769lb # [ 0.273182] LSM support for eBPF active770client # [ 0.276153] LSM support for eBPF active771lb # [ 0.274071] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)772client # [ 0.277071] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)773lb # [ 0.275013] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)774client # [ 0.277978] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)775worker2 # [ 0.264725] Freeing SMP alternatives memory: 44K776worker2 # [ 0.265192] pid_max: default: 32768 minimum: 301777lb # [ 0.277866] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)778client # [ 0.280808] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)779worker2 # [ 0.266277] LSM: initializing lsm=capability,landlock,yama,bpf,ima780worker2 # [ 0.267283] landlock: Up and running.781lb # [ 0.278556] Performance Events: Fam17h+ core perfctr, AMD PMU driver.782worker2 # [ 0.267883] Yama: becoming mindful.783client # [ 0.281494] Performance Events: Fam17h+ core perfctr, AMD PMU driver.784client # [ 0.281947] ... version: 2785lb # [ 0.278975] ... version: 2786worker2 # [ 0.268403] LSM support for eBPF active787client # [ 0.282620] ... bit width: 48788lb # [ 0.279659] ... bit width: 48789client # [ 0.282937] ... generic counters: 6790worker2 # [ 0.269323] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)791lb # [ 0.279972] ... generic counters: 6792client # [ 0.283649] ... generic bitmap: 000000000000003f793lb # [ 0.280689] ... generic bitmap: 000000000000003f794worker2 # [ 0.270227] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)795client # [ 0.283994] ... fixed-purpose counters: 0796lb # [ 0.280983] ... fixed-purpose counters: 0797worker1 # [ 0.271152] Freeing SMP alternatives memory: 44K798client # [ 0.284663] ... fixed-purpose bitmap: 0000000000000000799lb # [ 0.281653] ... fixed-purpose bitmap: 0000000000000000800worker1 # [ 0.271559] pid_max: default: 32768 minimum: 301801client # [ 0.284937] ... value mask: 0000ffffffffffff802lb # [ 0.281972] ... value mask: 0000ffffffffffff803worker1 # [ 0.272656] LSM: initializing lsm=capability,landlock,yama,bpf,ima804worker2 # [ 0.272900] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)805client # [ 0.285800] ... max period: 00007fffffffffff806lb # [ 0.282851] ... max period: 00007fffffffffff807worker1 # [ 0.273669] landlock: Up and running.808worker1 # [ 0.274281] Yama: becoming mindful.809client # [ 0.286609] ... global_ctrl mask: 000000000000003f810lb # [ 0.283654] ... global_ctrl mask: 000000000000003f811worker1 # [ 0.275190] LSM support for eBPF active812client # [ 0.287062] signal: max sigframe size: 3376813lb # [ 0.284097] signal: max sigframe size: 3376814worker2 # [ 0.274445] Performance Events: Fam17h+ core perfctr, AMD PMU driver.815client # [ 0.287822] rcu: Hierarchical SRCU implementation.816worker1 # [ 0.276126] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)817lb # [ 0.284899] rcu: Hierarchical SRCU implementation.818worker2 # [ 0.275195] ... version: 2819client # [ 0.288473] rcu: Max phase no-delay instances is 400.820lb # [ 0.285557] rcu: Max phase no-delay instances is 400.821worker2 # [ 0.275872] ... bit width: 48822worker1 # [ 0.276596] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)823worker2 # [ 0.276243] ... generic counters: 6824worker2 # [ 0.276989] ... generic bitmap: 000000000000003f825worker2 # [ 0.277936] ... fixed-purpose counters: 0826worker1 # [ 0.279744] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)827worker2 # [ 0.278191] ... fixed-purpose bitmap: 0000000000000000828worker2 # [ 0.279122] ... value mask: 0000ffffffffffff829worker1 # [ 0.281118] Performance Events: Fam17h+ core perfctr, AMD PMU driver.830client # [ 0.293562] smp: Bringing up secondary CPUs ...831lb # [ 0.290574] smp: Bringing up secondary CPUs ...832worker2 # [ 0.279858] ... max period: 00007fffffffffff833worker1 # [ 0.281564] ... version: 2834client # [ 0.293950] smp: Brought up 1 node, 1 CPU835lb # [ 0.290987] smp: Brought up 1 node, 1 CPU836worker1 # [ 0.282256] ... bit width: 48837worker2 # [ 0.280191] ... global_ctrl mask: 000000000000003f838client # [ 0.294586] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)839worker1 # [ 0.282558] ... generic counters: 6840lb # [ 0.291627] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)841worker2 # [ 0.281171] signal: max sigframe size: 3376842worker1 # [ 0.283245] ... generic bitmap: 000000000000003f843worker2 # [ 0.281797] rcu: Hierarchical SRCU implementation.844worker1 # [ 0.283557] ... fixed-purpose counters: 0845worker2 # [ 0.282195] rcu: Max phase no-delay instances is 400.846client # [ 0.295169] Memory: 1454552K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)847lb # [ 0.292149] Memory: 1454552K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)848worker1 # [ 0.284217] ... fixed-purpose bitmap: 0000000000000000849client # [ 0.296236] devtmpfs: initialized850lb # [ 0.293223] devtmpfs: initialized851worker1 # [ 0.284557] ... value mask: 0000ffffffffffff852client # [ 0.297032] x86/mm: Memory block size: 128MB853lb # [ 0.294028] x86/mm: Memory block size: 128MB854worker1 # [ 0.285428] ... max period: 00007fffffffffff855worker1 # [ 0.286255] ... global_ctrl mask: 000000000000003f856worker1 # [ 0.286666] signal: max sigframe size: 3376857client # [ 0.298753] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)858lb # [ 0.295807] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)859worker1 # [ 0.287434] rcu: Hierarchical SRCU implementation.860worker2 # [ 0.287708] smp: Bringing up secondary CPUs ...861client # [ 0.299830] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).862worker1 # [ 0.288188] rcu: Max phase no-delay instances is 400.863lb # [ 0.296886] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).864worker2 # [ 0.288206] smp: Brought up 1 node, 1 CPU865client # [ 0.301020] pinctrl core: initialized pinctrl subsystem866lb # [ 0.298055] pinctrl core: initialized pinctrl subsystem867worker2 # [ 0.288890] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)868client # [ 0.302181] PM: RTC time: 12:07:48, date: 2026-09-21869lb # [ 0.299180] PM: RTC time: 12:07:48, date: 2026-09-21870worker2 # [ 0.289339] Memory: 1454588K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109340K reserved, 0K cma-reserved)871worker2 # [ 0.290447] devtmpfs: initialized872worker1 # [ 0.293152] smp: Bringing up secondary CPUs ...873worker2 # [ 0.291327] x86/mm: Memory block size: 128MB874client # [ 0.305875] NET: Registered PF_NETLINK/PF_ROUTE protocol family875lb # [ 0.302798] NET: Registered PF_NETLINK/PF_ROUTE protocol family876worker1 # [ 0.293572] smp: Brought up 1 node, 1 CPU877worker1 # [ 0.294242] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)878client # [ 0.307157] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations879lb # [ 0.304185] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations880worker2 # [ 0.293116] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)881lb # [ 0.304993] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations882client # [ 0.307962] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations883worker1 # [ 0.294672] Memory: 1454560K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)884worker2 # [ 0.294047] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).885worker1 # [ 0.295840] devtmpfs: initialized886worker2 # [ 0.295286] pinctrl core: initialized pinctrl subsystem887client # [ 0.309233] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations888lb # [ 0.306251] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations889lb # [ 0.306983] audit: initializing netlink subsys (disabled)890worker1 # [ 0.296669] x86/mm: Memory block size: 128MB891client # [ 0.309948] audit: initializing netlink subsys (disabled)892worker2 # [ 0.296452] PM: RTC time: 12:07:48, date: 2026-09-21893lb # [ 0.308051] thermal_sys: Registered thermal governor 'fair_share'894client # [ 0.310938] audit: type=2000 audit(1789992468.857:1): state=initialized audit_enabled=0 res=1895lb # [ 0.308053] thermal_sys: Registered thermal governor 'bang_bang'896worker1 # [ 0.298472] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)897client # [ 0.312111] thermal_sys: Registered thermal governor 'fair_share'898lb # [ 0.308973] thermal_sys: Registered thermal governor 'step_wise'899client # [ 0.312113] thermal_sys: Registered thermal governor 'bang_bang'900worker1 # [ 0.299501] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).901lb # [ 0.309947] thermal_sys: Registered thermal governor 'user_space'902worker2 # [ 0.300042] NET: Registered PF_NETLINK/PF_ROUTE protocol family903worker1 # [ 0.300656] pinctrl core: initialized pinctrl subsystem904client # [ 0.312938] thermal_sys: Registered thermal governor 'step_wise'905lb # [ 0.310692] audit: type=2000 audit(1789992468.857:1): state=initialized audit_enabled=0 res=1906client # [ 0.313896] thermal_sys: Registered thermal governor 'user_space'907worker2 # [ 0.301424] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations908worker1 # [ 0.301779] PM: RTC time: 12:07:48, date: 2026-09-21909lb # [ 0.311975] thermal_sys: Registered thermal governor 'power_allocator'910client # [ 0.314698] thermal_sys: Registered thermal governor 'power_allocator'911worker2 # [ 0.302214] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations912lb # [ 0.312011] cpuidle: using governor menu913client # [ 0.314950] cpuidle: using governor menu914worker2 # [ 0.303472] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations915lb # [ 0.314202] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5916client # [ 0.317180] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5917worker2 # [ 0.304202] audit: initializing netlink subsys (disabled)918worker1 # [ 0.305424] NET: Registered PF_NETLINK/PF_ROUTE protocol family919lb # [ 0.315202] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]920client # [ 0.318175] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]921worker2 # [ 0.305194] audit: type=2000 audit(1789992468.868:1): state=initialized audit_enabled=0 res=1922worker1 # [ 0.306768] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations923lb # [ 0.315975] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry924client # [ 0.318940] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry925worker2 # [ 0.306373] thermal_sys: Registered thermal governor 'fair_share'926lb # [ 0.316984] PCI: Using configuration type 1 for base access927worker1 # [ 0.307581] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations928client # [ 0.319949] PCI: Using configuration type 1 for base access929worker2 # [ 0.306375] thermal_sys: Registered thermal governor 'bang_bang'930worker2 # [ 0.307192] thermal_sys: Registered thermal governor 'step_wise'931worker1 # [ 0.308851] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations932lb # [ 0.318084] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.933client # [ 0.321033] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.934worker2 # [ 0.308143] thermal_sys: Registered thermal governor 'user_space'935worker1 # [ 0.309570] audit: initializing netlink subsys (disabled)936worker2 # [ 0.308868] thermal_sys: Registered thermal governor 'power_allocator'937worker1 # [ 0.310731] thermal_sys: Registered thermal governor 'fair_share'938worker2 # [ 0.309222] cpuidle: using governor menu939worker1 # [ 0.310733] thermal_sys: Registered thermal governor 'bang_bang'940worker2 # [ 0.311418] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5941worker1 # [ 0.311559] thermal_sys: Registered thermal governor 'step_wise'942worker1 # [ 0.312550] thermal_sys: Registered thermal governor 'user_space'943worker2 # [ 0.312620] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]944worker1 # [ 0.313363] audit: type=2000 audit(1789992468.874:1): state=initialized audit_enabled=0 res=1945worker2 # [ 0.313195] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry946client # [ 0.327971] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages947lb # [ 0.325230] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages948worker1 # [ 0.315560] thermal_sys: Registered thermal governor 'power_allocator'949worker2 # [ 0.314204] PCI: Using configuration type 1 for base access950client # [ 0.328938] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page951worker1 # [ 0.315574] cpuidle: using governor menu952lb # [ 0.325974] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page953worker2 # [ 0.315237] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.954worker1 # [ 0.317600] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5955worker1 # [ 0.318796] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]956worker1 # [ 0.319561] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry957client # [ 0.333938] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages958lb # [ 0.330974] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages959worker1 # [ 0.320570] PCI: Using configuration type 1 for base access960client # [ 0.334937] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page961lb # [ 0.331973] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page962worker2 # [ 0.320416] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages963worker1 # [ 0.321661] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.964worker2 # [ 0.321194] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page965worker1 # [ 0.326852] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages966worker2 # [ 0.326193] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages967worker1 # [ 0.327559] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page968worker2 # [ 0.327191] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page969lb # [ 0.340564] ACPI: Added _OSI(Module Device)970lb # [ 0.340974] ACPI: Added _OSI(Processor Device)971lb # [ 0.341746] ACPI: Added _OSI(Processor Aggregator Device)972client # [ 0.346197] ACPI: Added _OSI(Module Device)973client # [ 0.346938] ACPI: Added _OSI(Processor Device)974worker1 # [ 0.332560] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages975client # [ 0.347670] ACPI: Added _OSI(Processor Aggregator Device)976worker1 # [ 0.333557] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page977lb # [ 0.348515] ACPI: 1 ACPI AML tables successfully acquired and loaded978worker2 # [ 0.337641] ACPI: Added _OSI(Module Device)979worker2 # [ 0.338193] ACPI: Added _OSI(Processor Device)980client # [ 0.353673] ACPI: 1 ACPI AML tables successfully acquired and loaded981worker2 # [ 0.338959] ACPI: Added _OSI(Processor Aggregator Device)982lb # [ 0.351951] ACPI: Interpreter enabled983lb # [ 0.352461] ACPI: PM: (supports S0 S3 S4 S5)984lb # [ 0.352973] ACPI: Using IOAPIC for interrupt routing985worker1 # [ 0.343921] ACPI: Added _OSI(Module Device)986client # [ 0.357350] ACPI: Interpreter enabled987worker1 # [ 0.344559] ACPI: Added _OSI(Processor Device)988client # [ 0.357953] ACPI: PM: (supports S0 S3 S4 S5)989lb # [ 0.353872] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug990client # [ 0.358677] ACPI: Using IOAPIC for interrupt routing991worker1 # [ 0.345267] ACPI: Added _OSI(Processor Aggregator Device)992client # [ 0.360998] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug993worker2 # [ 0.347778] ACPI: 1 ACPI AML tables successfully acquired and loaded994lb # [ 0.358973] PCI: Using E820 reservations for host bridge windows995client # [ 0.361938] PCI: Using E820 reservations for host bridge windows996lb # [ 0.360109] ACPI: Enabled 2 GPEs in block 00 to 3F997client # [ 0.365074] ACPI: Enabled 2 GPEs in block 00 to 3F998worker2 # [ 0.351610] ACPI: Interpreter enabled999worker2 # [ 0.352209] ACPI: PM: (supports S0 S3 S4 S5)1000worker1 # [ 0.354107] ACPI: 1 ACPI AML tables successfully acquired and loaded1001worker2 # [ 0.352945] ACPI: Using IOAPIC for interrupt routing1002worker2 # [ 0.353253] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1003worker2 # [ 0.354193] PCI: Using E820 reservations for host bridge windows1004worker1 # [ 0.357967] ACPI: Interpreter enabled1005lb # [ 0.368786] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1006worker1 # [ 0.358573] ACPI: PM: (supports S0 S3 S4 S5)1007worker1 # [ 0.359331] ACPI: Using IOAPIC for interrupt routing1008client # [ 0.372608] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1009worker2 # [ 0.357326] ACPI: Enabled 2 GPEs in block 00 to 3F1010lb # [ 0.370309] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1011client # [ 0.372944] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1012worker1 # [ 0.359609] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1013lb # [ 0.371050] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1014worker1 # [ 0.360558] PCI: Using E820 reservations for host bridge windows1015client # [ 0.374024] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1016lb # [ 0.372095] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1017client # [ 0.375059] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1018lb # [ 0.373338] PCI host bridge to bus 0000:001019client # [ 0.376271] PCI host bridge to bus 0000:001020worker1 # [ 0.363671] ACPI: Enabled 2 GPEs in block 00 to 3F1021lb # [ 0.373977] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1022client # [ 0.376941] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1023lb # [ 0.374973] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1024client # [ 0.377937] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1025lb # [ 0.375972] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1026client # [ 0.378937] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1027worker2 # [ 0.365928] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1028lb # [ 0.376973] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1029client # [ 0.379936] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1030worker2 # [ 0.366198] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1031lb # [ 0.377973] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1032client # [ 0.380937] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1033lb # [ 0.378972] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1034worker2 # [ 0.367817] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1035client # [ 0.381937] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1036lb # [ 0.379973] pci_bus 0000:00: root bus resource [bus 00-ff]1037client # [ 0.382937] pci_bus 0000:00: root bus resource [bus 00-ff]1038worker2 # [ 0.368315] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1039lb # [ 0.380959] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1040worker2 # [ 0.369533] PCI host bridge to bus 0000:001041worker1 # [ 0.372329] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1042client # [ 0.383887] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1043worker2 # [ 0.370196] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1044worker1 # [ 0.372565] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1045lb # [ 0.382449] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1046client # [ 0.385404] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1047worker2 # [ 0.371192] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1048worker1 # [ 0.374169] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1049worker2 # [ 0.372192] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1050worker1 # [ 0.374689] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1051worker2 # [ 0.373192] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1052worker1 # [ 0.375953] PCI host bridge to bus 0000:001053worker2 # [ 0.374192] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1054worker1 # [ 0.376563] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1055worker2 # [ 0.375192] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1056worker1 # [ 0.377558] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1057worker2 # [ 0.376193] pci_bus 0000:00: root bus resource [bus 00-ff]1058worker1 # [ 0.378557] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1059worker2 # [ 0.377186] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1060worker1 # [ 0.379558] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1061lb # [ 0.385072] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1062client # [ 0.388025] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1063worker2 # [ 0.378627] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1064worker1 # [ 0.380558] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1065lb # [ 0.385985] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1066client # [ 0.389078] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1067worker1 # [ 0.381558] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1068lb # [ 0.386938] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1069client # [ 0.389959] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1070worker1 # [ 0.382558] pci_bus 0000:00: root bus resource [bus 00-ff]1071lb # [ 0.388337] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1072worker1 # [ 0.383571] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1073client # [ 0.391313] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1074worker1 # [ 0.385008] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1075lb # [ 0.389677] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1076client # [ 0.392707] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1077lb # [ 0.393009] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1078worker2 # [ 0.381295] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1079client # [ 0.395971] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1080lb # [ 0.393839] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1081worker2 # [ 0.382204] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1082client # [ 0.396820] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1083lb # [ 0.394670] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1084worker2 # [ 0.383184] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1085client # [ 0.397715] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1086lb # [ 0.394978] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1087worker2 # [ 0.384426] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1088client # [ 0.398815] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1089lb # [ 0.396574] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1090worker1 # [ 0.387753] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1091worker2 # [ 0.386097] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1092client # [ 0.400301] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1093worker1 # [ 0.388614] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1094lb # [ 0.398956] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1095worker1 # [ 0.389580] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1096lb # [ 0.399629] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1097worker2 # [ 0.389199] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1098client # [ 0.402970] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1099worker1 # [ 0.390879] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1100lb # [ 0.399995] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1101client # [ 0.403802] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1102worker2 # [ 0.390031] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1103worker2 # [ 0.390909] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1104client # [ 0.404654] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1105worker1 # [ 0.392277] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1106lb # [ 0.401508] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1107worker2 # [ 0.391198] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1108client # [ 0.405490] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1109worker2 # [ 0.392735] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1110worker1 # [ 0.394590] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1111worker1 # [ 0.395481] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1112lb # [ 0.404869] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1113lb # [ 0.405629] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1114worker1 # [ 0.396336] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1115worker2 # [ 0.394200] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1116client # [ 0.408971] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1117lb # [ 0.405994] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1118worker2 # [ 0.395139] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1119worker1 # [ 0.397444] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1120client # [ 0.409784] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1121worker2 # [ 0.395899] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1122client # [ 0.410704] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1123lb # [ 0.407557] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1124worker1 # [ 0.399097] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1125worker2 # [ 0.396782] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1126client # [ 0.412330] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1127worker1 # [ 0.400587] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1128lb # [ 0.409973] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1129lb # [ 0.410824] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1130worker1 # [ 0.401442] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1131client # [ 0.414967] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1132worker1 # [ 0.402386] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1133worker2 # [ 0.399199] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1134client # [ 0.415839] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1135lb # [ 0.411669] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1136worker2 # [ 0.400049] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1137worker1 # [ 0.403990] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1138worker2 # [ 0.400937] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1139client # [ 0.416696] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1140lb # [ 0.412511] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1141worker2 # [ 0.401759] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1142client # [ 0.417547] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1143worker1 # [ 0.406585] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1144lb # [ 0.414771] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1145worker1 # [ 0.407462] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1146lb # [ 0.415626] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1147worker2 # [ 0.403236] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1148worker1 # [ 0.408265] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1149lb # [ 0.415994] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1150client # [ 0.419864] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1151worker2 # [ 0.404088] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1152client # [ 0.420583] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1153worker1 # [ 0.409120] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1154lb # [ 0.417544] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1155client # [ 0.420959] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1156worker2 # [ 0.404928] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1157client # [ 0.422529] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1158worker2 # [ 0.405736] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1159worker1 # [ 0.410595] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1160worker1 # [ 0.411452] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1161worker1 # [ 0.412362] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1162lb # [ 0.419981] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1163worker2 # [ 0.407235] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1164lb # [ 0.421618] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1165worker2 # [ 0.408076] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1166client # [ 0.425964] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1167lb # [ 0.422031] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1168worker2 # [ 0.408977] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1169worker1 # [ 0.414385] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1170client # [ 0.426844] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1171client # [ 0.427686] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1172lb # [ 0.423556] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1173worker2 # [ 0.410607] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1174worker1 # [ 0.416586] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1175worker1 # [ 0.417456] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1176client # [ 0.428624] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1177lb # [ 0.425752] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1178worker1 # [ 0.418309] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1179lb # [ 0.425980] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1180worker2 # [ 0.413201] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1181client # [ 0.430965] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1182worker1 # [ 0.419996] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1183lb # [ 0.426977] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1184worker2 # [ 0.414832] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1185client # [ 0.431817] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1186worker2 # [ 0.415214] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1187lb # [ 0.427978] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1188client # [ 0.432702] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1189worker2 # [ 0.416776] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1190worker1 # [ 0.422242] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1191lb # [ 0.429514] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1192client # [ 0.432943] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1193worker1 # [ 0.422564] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1194client # [ 0.434422] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1195worker1 # [ 0.423557] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1196worker2 # [ 0.418199] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1197lb # [ 0.430985] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1198worker2 # [ 0.419008] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1199lb # [ 0.431923] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1200worker1 # [ 0.425182] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1201worker2 # [ 0.419981] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1202client # [ 0.435950] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1203worker2 # [ 0.420985] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1204lb # [ 0.433389] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1205client # [ 0.436923] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1206worker1 # [ 0.426588] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1207worker1 # [ 0.427478] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1208client # [ 0.438329] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1209worker2 # [ 0.422864] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1210lb # [ 0.435845] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1211worker1 # [ 0.428265] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1212worker1 # [ 0.428564] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1213worker2 # [ 0.424204] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1214lb # [ 0.436618] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1215worker2 # [ 0.425181] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1216lb # [ 0.436995] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1217client # [ 0.440408] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1218worker1 # [ 0.430661] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1219client # [ 0.441603] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1220worker2 # [ 0.426590] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1221lb # [ 0.438512] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1222client # [ 0.441959] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1223worker1 # [ 0.432367] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1224worker1 # [ 0.433302] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1225client # [ 0.443517] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1226lb # [ 0.439981] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1227worker2 # [ 0.429014] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1228lb # [ 0.440823] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1229worker1 # [ 0.434099] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1230worker2 # [ 0.429832] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1231lb # [ 0.441704] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1232client # [ 0.445666] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1233worker2 # [ 0.430215] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1234worker1 # [ 0.435566] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1235client # [ 0.445969] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1236lb # [ 0.442549] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1237worker1 # [ 0.436412] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1238client # [ 0.446880] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1239worker2 # [ 0.431769] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1240worker1 # [ 0.437334] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1241lb # [ 0.444462] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1242worker2 # [ 0.433227] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1243client # [ 0.448307] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1244worker1 # [ 0.439107] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1245worker2 # [ 0.434123] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1246lb # [ 0.445203] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1247client # [ 0.450794] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1248worker2 # [ 0.434922] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1249lb # [ 0.446723] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1250worker1 # [ 0.440589] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1251client # [ 0.451812] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1252worker1 # [ 0.441448] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1253worker2 # [ 0.435751] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1254worker1 # [ 0.442261] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1255client # [ 0.453677] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1256worker2 # [ 0.437255] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1257worker1 # [ 0.443143] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1258worker2 # [ 0.438313] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1259worker1 # [ 0.444156] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1260worker2 # [ 0.439894] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1261worker1 # [ 0.444756] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1262worker1 # [ 0.446094] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1263lb # [ 0.447168] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1264lb # [ 0.449769] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1265client # [ 0.454157] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1266lb # [ 0.450851] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1267client # [ 0.456734] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1268lb # [ 0.452676] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1269client # [ 0.458082] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1270worker2 # [ 0.440383] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1271lb # [ 0.453228] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1272client # [ 0.459637] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1273worker2 # [ 0.442847] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1274lb # [ 0.454298] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1275worker1 # [ 0.446754] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1276client # [ 0.460199] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1277worker2 # [ 0.443428] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1278lb # [ 0.455218] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1279worker1 # [ 0.449094] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1280client # [ 0.461220] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1281worker2 # [ 0.444753] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1282lb # [ 0.457029] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1283worker1 # [ 0.449782] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1284client # [ 0.462228] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1285lb # [ 0.457867] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1286worker2 # [ 0.446262] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1287worker1 # [ 0.451073] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1288worker2 # [ 0.447502] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1289lb # [ 0.458988] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1290worker1 # [ 0.452134] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1291client # [ 0.463988] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1292client # [ 0.464943] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1293worker2 # [ 0.448437] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1294lb # [ 0.460741] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1295worker1 # [ 0.452843] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1296client # [ 0.466217] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1297worker2 # [ 0.450250] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1298worker1 # [ 0.453836] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1299worker2 # [ 0.451068] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1300client # [ 0.467741] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1301worker1 # [ 0.455580] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1302lb # [ 0.464538] ACPI: PCI: Interrupt link LNKA configured for IRQ 101303worker2 # [ 0.452275] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1304worker1 # [ 0.456447] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1305lb # [ 0.465078] ACPI: PCI: Interrupt link LNKB configured for IRQ 101306lb # [ 0.466071] ACPI: PCI: Interrupt link LNKC configured for IRQ 111307worker2 # [ 0.453855] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1308worker1 # [ 0.457592] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1309lb # [ 0.467070] ACPI: PCI: Interrupt link LNKD configured for IRQ 111310client # [ 0.471598] ACPI: PCI: Interrupt link LNKA configured for IRQ 101311lb # [ 0.468070] ACPI: PCI: Interrupt link LNKE configured for IRQ 101312worker1 # [ 0.459163] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1313client # [ 0.472069] ACPI: PCI: Interrupt link LNKB configured for IRQ 101314lb # [ 0.469072] ACPI: PCI: Interrupt link LNKF configured for IRQ 101315client # [ 0.473046] ACPI: PCI: Interrupt link LNKC configured for IRQ 111316worker2 # [ 0.457197] ACPI: PCI: Interrupt link LNKA configured for IRQ 101317lb # [ 0.470074] ACPI: PCI: Interrupt link LNKG configured for IRQ 111318client # [ 0.474037] ACPI: PCI: Interrupt link LNKD configured for IRQ 111319lb # [ 0.471069] ACPI: PCI: Interrupt link LNKH configured for IRQ 111320worker2 # [ 0.458265] ACPI: PCI: Interrupt link LNKB configured for IRQ 101321client # [ 0.475036] ACPI: PCI: Interrupt link LNKE configured for IRQ 101322worker1 # [ 0.462570] ACPI: PCI: Interrupt link LNKA configured for IRQ 101323lb # [ 0.472010] ACPI: PCI: Interrupt link GSIA configured for IRQ 161324worker2 # [ 0.459292] ACPI: PCI: Interrupt link LNKC configured for IRQ 111325client # [ 0.476038] ACPI: PCI: Interrupt link LNKF configured for IRQ 101326worker1 # [ 0.463614] ACPI: PCI: Interrupt link LNKB configured for IRQ 101327lb # [ 0.472945] ACPI: PCI: Interrupt link GSIB configured for IRQ 171328worker2 # [ 0.460292] ACPI: PCI: Interrupt link LNKD configured for IRQ 111329client # [ 0.477038] ACPI: PCI: Interrupt link LNKG configured for IRQ 111330lb # [ 0.473697] ACPI: PCI: Interrupt link GSIC configured for IRQ 181331worker1 # [ 0.464636] ACPI: PCI: Interrupt link LNKC configured for IRQ 111332worker2 # [ 0.461291] ACPI: PCI: Interrupt link LNKE configured for IRQ 101333client # [ 0.478033] ACPI: PCI: Interrupt link LNKH configured for IRQ 111334lb # [ 0.473985] ACPI: PCI: Interrupt link GSID configured for IRQ 191335worker1 # [ 0.465657] ACPI: PCI: Interrupt link LNKD configured for IRQ 111336worker2 # [ 0.462290] ACPI: PCI: Interrupt link LNKF configured for IRQ 101337client # [ 0.478976] ACPI: PCI: Interrupt link GSIA configured for IRQ 161338lb # [ 0.474885] ACPI: PCI: Interrupt link GSIE configured for IRQ 201339worker1 # [ 0.466654] ACPI: PCI: Interrupt link LNKE configured for IRQ 101340worker2 # [ 0.463291] ACPI: PCI: Interrupt link LNKG configured for IRQ 111341client # [ 0.479917] ACPI: PCI: Interrupt link GSIB configured for IRQ 171342lb # [ 0.475661] ACPI: PCI: Interrupt link GSIF configured for IRQ 211343worker1 # [ 0.467652] ACPI: PCI: Interrupt link LNKF configured for IRQ 101344worker2 # [ 0.464287] ACPI: PCI: Interrupt link LNKH configured for IRQ 111345lb # [ 0.475985] ACPI: PCI: Interrupt link GSIG configured for IRQ 221346client # [ 0.480672] ACPI: PCI: Interrupt link GSIC configured for IRQ 181347worker1 # [ 0.468656] ACPI: PCI: Interrupt link LNKG configured for IRQ 111348worker2 # [ 0.465222] ACPI: PCI: Interrupt link GSIA configured for IRQ 161349client # [ 0.480952] ACPI: PCI: Interrupt link GSID configured for IRQ 191350lb # [ 0.476889] ACPI: PCI: Interrupt link GSIH configured for IRQ 231351worker1 # [ 0.469679] ACPI: PCI: Interrupt link LNKH configured for IRQ 111352worker2 # [ 0.466138] ACPI: PCI: Interrupt link GSIB configured for IRQ 171353client # [ 0.481879] ACPI: PCI: Interrupt link GSIE configured for IRQ 201354worker2 # [ 0.466892] ACPI: PCI: Interrupt link GSIC configured for IRQ 181355worker1 # [ 0.470627] ACPI: PCI: Interrupt link GSIA configured for IRQ 161356lb # [ 0.478737] iommu: Default domain type: Translated1357client # [ 0.482803] ACPI: PCI: Interrupt link GSIF configured for IRQ 211358worker2 # [ 0.467204] ACPI: PCI: Interrupt link GSID configured for IRQ 191359worker1 # [ 0.471573] ACPI: PCI: Interrupt link GSIB configured for IRQ 171360lb # [ 0.478980] iommu: DMA domain TLB invalidation policy: lazy mode1361client # [ 0.483657] ACPI: PCI: Interrupt link GSIG configured for IRQ 221362worker1 # [ 0.472491] ACPI: PCI: Interrupt link GSIC configured for IRQ 181363worker2 # [ 0.468151] ACPI: PCI: Interrupt link GSIE configured for IRQ 201364lb # [ 0.480170] ACPI: bus type USB registered1365client # [ 0.483948] ACPI: PCI: Interrupt link GSIH configured for IRQ 231366worker1 # [ 0.473248] ACPI: PCI: Interrupt link GSID configured for IRQ 191367worker2 # [ 0.468923] ACPI: PCI: Interrupt link GSIF configured for IRQ 211368lb # [ 0.480913] usbcore: registered new interface driver usbfs1369worker1 # [ 0.473570] ACPI: PCI: Interrupt link GSIE configured for IRQ 201370worker2 # [ 0.469205] ACPI: PCI: Interrupt link GSIG configured for IRQ 221371lb # [ 0.481648] usbcore: registered new interface driver hub1372client # [ 0.485986] iommu: Default domain type: Translated1373worker1 # [ 0.474523] ACPI: PCI: Interrupt link GSIF configured for IRQ 211374lb # [ 0.481983] usbcore: registered new device driver usb1375worker2 # [ 0.470163] ACPI: PCI: Interrupt link GSIH configured for IRQ 231376client # [ 0.486769] iommu: DMA domain TLB invalidation policy: lazy mode1377worker1 # [ 0.475284] ACPI: PCI: Interrupt link GSIG configured for IRQ 221378client # [ 0.487936] ACPI: bus type USB registered1379lb # [ 0.483701] NetLabel: Initializing1380worker1 # [ 0.475572] ACPI: PCI: Interrupt link GSIH configured for IRQ 231381worker2 # [ 0.472113] iommu: Default domain type: Translated1382lb # [ 0.483973] NetLabel: domain hash size = 1281383client # [ 0.488750] usbcore: registered new interface driver usbfs1384worker2 # [ 0.472774] iommu: DMA domain TLB invalidation policy: lazy mode1385lb # [ 0.484696] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1386worker1 # [ 0.477518] iommu: Default domain type: Translated1387worker2 # [ 0.473448] ACPI: bus type USB registered1388lb # [ 0.485036] NetLabel: unlabeled traffic allowed by default1389client # [ 0.490047] usbcore: registered new interface driver hub1390worker1 # [ 0.478146] iommu: DMA domain TLB invalidation policy: lazy mode1391worker2 # [ 0.474182] usbcore: registered new interface driver usbfs1392lb # [ 0.485979] PCI: Using ACPI for IRQ routing1393client # [ 0.490958] usbcore: registered new device driver usb1394worker1 # [ 0.478822] ACPI: bus type USB registered1395worker2 # [ 0.474858] usbcore: registered new interface driver hub1396worker2 # [ 0.475203] usbcore: registered new device driver usb1397worker1 # [ 0.479590] usbcore: registered new interface driver usbfs1398client # [ 0.493096] NetLabel: Initializing1399worker1 # [ 0.480534] usbcore: registered new interface driver hub1400worker2 # [ 0.476925] NetLabel: Initializing1401client # [ 0.493747] NetLabel: domain hash size = 1281402worker2 # [ 0.477198] NetLabel: domain hash size = 1281403client # [ 0.494494] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1404worker1 # [ 0.481250] usbcore: registered new device driver usb1405worker2 # [ 0.477944] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1406client # [ 0.494976] NetLabel: unlabeled traffic allowed by default1407client # [ 0.495895] PCI: Using ACPI for IRQ routing1408worker2 # [ 0.478226] NetLabel: unlabeled traffic allowed by default1409worker1 # [ 0.482459] NetLabel: Initializing1410worker2 # [ 0.479211] PCI: Using ACPI for IRQ routing1411worker1 # [ 0.482985] NetLabel: domain hash size = 1281412worker1 # [ 0.483558] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413worker1 # [ 0.484569] NetLabel: unlabeled traffic allowed by default1414worker1 # [ 0.485528] PCI: Using ACPI for IRQ routing1415lb # [ 0.529707] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416lb # [ 0.529969] pci 0000:00:01.0: vgaarb: bridge control possible1417lb # [ 0.529969] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418lb # [ 0.529979] vgaarb: loaded1419lb # [ 0.530729] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01420lb # [ 0.530975] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1421client # [ 0.539863] pci 0000:00:01.0: vgaarb: setting as boot VGA device1422client # [ 0.539933] pci 0000:00:01.0: vgaarb: bridge control possible1423worker2 # [ 0.523237] pci 0000:00:01.0: vgaarb: setting as boot VGA device1424client # [ 0.539933] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1425worker1 # [ 0.529051] pci 0000:00:01.0: vgaarb: setting as boot VGA device1426worker2 # [ 0.524188] pci 0000:00:01.0: vgaarb: bridge control possible1427client # [ 0.539942] vgaarb: loaded1428worker1 # [ 0.529554] pci 0000:00:01.0: vgaarb: bridge control possible1429worker2 # [ 0.524188] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1430client # [ 0.540695] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01431worker2 # [ 0.524196] vgaarb: loaded1432worker1 # [ 0.529554] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1433client # [ 0.540937] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1434worker1 # [ 0.529562] vgaarb: loaded1435worker2 # [ 0.524953] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01436lb # [ 0.536068] clocksource: Switched to clocksource kvm-clock1437worker1 # [ 0.530306] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01438worker2 # [ 0.525192] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1439worker1 # [ 0.530560] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1440lb # [ 0.539477] VFS: Disk quotas dquot_6.6.01441client # [ 0.544026] clocksource: Switched to clocksource kvm-clock1442lb # [ 0.540203] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1443client # [ 0.545798] VFS: Disk quotas dquot_6.6.01444lb # [ 0.541563] pnp: PnP ACPI init1445worker1 # [ 0.534642] clocksource: Switched to clocksource kvm-clock1446worker2 # [ 0.529282] clocksource: Switched to clocksource kvm-clock1447lb # [ 0.542345] ACPI: IRQ 4 override to edge(!), high(!)1448lb # [ 0.543308] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1449client # [ 0.548379] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1450lb # [ 0.544879] pnp: PnP ACPI: found 6 devices1451client # [ 0.549734] pnp: PnP ACPI init1452worker1 # [ 0.538248] VFS: Disk quotas dquot_6.6.01453worker2 # [ 0.532982] VFS: Disk quotas dquot_6.6.01454client # [ 0.550580] ACPI: IRQ 4 override to edge(!), high(!)1455worker1 # [ 0.538974] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1456worker2 # [ 0.533670] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1457worker1 # [ 0.540290] pnp: PnP ACPI init1458client # [ 0.551536] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1459worker2 # [ 0.534980] pnp: PnP ACPI init1460worker1 # [ 0.541103] ACPI: IRQ 4 override to edge(!), high(!)1461client # [ 0.553084] pnp: PnP ACPI: found 6 devices1462worker2 # [ 0.535781] ACPI: IRQ 4 override to edge(!), high(!)1463worker1 # [ 0.542042] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1464worker2 # [ 0.536744] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1465worker1 # [ 0.543599] pnp: PnP ACPI: found 6 devices1466worker2 # [ 0.538308] pnp: PnP ACPI: found 6 devices1467lb # [ 0.552495] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1468lb # [ 0.553920] clocksource: Switched to clocksource acpi_pm1469lb # [ 0.554871] NET: Registered PF_INET protocol family1470lb # [ 0.556002] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1471client # [ 0.560733] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1472client # [ 0.562197] clocksource: Switched to clocksource acpi_pm1473client # [ 0.563207] NET: Registered PF_INET protocol family1474worker1 # [ 0.551299] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1475worker2 # [ 0.545907] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1476worker1 # [ 0.552747] clocksource: Switched to clocksource acpi_pm1477worker2 # [ 0.547313] clocksource: Switched to clocksource acpi_pm1478client # [ 0.564386] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1479worker1 # [ 0.553708] NET: Registered PF_INET protocol family1480worker2 # [ 0.548265] NET: Registered PF_INET protocol family1481worker1 # [ 0.554931] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1482worker2 # [ 0.549536] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1483lb # [ 0.572777] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1484lb # [ 0.574275] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1485lb # [ 0.575566] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1486client # [ 0.581052] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1487lb # [ 0.576875] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1488lb # [ 0.578093] TCP: Hash tables configured (established 16384 bind 16384)1489client # [ 0.582589] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1490worker2 # [ 0.566022] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1491lb # [ 0.579186] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1492client # [ 0.583912] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1493worker1 # [ 0.572036] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1494lb # [ 0.580391] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1495worker2 # [ 0.567555] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1496client # [ 0.585205] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1497worker1 # [ 0.573521] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1498lb # [ 0.581427] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1499worker2 # [ 0.568789] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1500client # [ 0.586392] TCP: Hash tables configured (established 16384 bind 16384)1501worker1 # [ 0.574803] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1502lb # [ 0.582599] NET: Registered PF_UNIX/PF_LOCAL protocol family1503worker2 # [ 0.570105] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1504client # [ 0.587535] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1505lb # [ 0.583495] NET: Registered PF_XDP protocol family1506worker1 # [ 0.576096] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1507worker2 # [ 0.571298] TCP: Hash tables configured (established 16384 bind 16384)1508client # [ 0.588813] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1509lb # [ 0.584318] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1510worker1 # [ 0.577319] TCP: Hash tables configured (established 16384 bind 16384)1511worker2 # [ 0.572377] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1512lb # [ 0.585290] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1513client # [ 0.589954] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1514worker1 # [ 0.578402] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1515worker2 # [ 0.573620] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1516client # [ 0.591134] NET: Registered PF_UNIX/PF_LOCAL protocol family1517lb # [ 0.586254] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1518worker1 # [ 0.579628] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1519client # [ 0.592086] NET: Registered PF_XDP protocol family1520worker2 # [ 0.574645] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1521lb # [ 0.587301] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1522worker1 # [ 0.580656] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1523client # [ 0.592879] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1524worker2 # [ 0.575756] NET: Registered PF_UNIX/PF_LOCAL protocol family1525lb # [ 0.588344] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1526worker1 # [ 0.581833] NET: Registered PF_UNIX/PF_LOCAL protocol family1527worker2 # [ 0.576666] NET: Registered PF_XDP protocol family1528client # [ 0.593847] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1529lb # [ 0.589391] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1530worker1 # [ 0.582749] NET: Registered PF_XDP protocol family1531worker2 # [ 0.577436] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1532client # [ 0.594808] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1533worker1 # [ 0.583559] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1534worker2 # [ 0.578444] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1535lb # [ 0.591317] ACPI: \_SB_.GSIA: Enabled at IRQ 161536client # [ 0.595905] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1537worker1 # [ 0.584628] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1538worker2 # [ 0.579420] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1539client # [ 0.596950] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1540worker1 # [ 0.585614] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1541worker2 # [ 0.580490] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1542client # [ 0.598017] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1543lb # [ 0.593878] ACPI: \_SB_.GSIB: Enabled at IRQ 171544worker1 # [ 0.586698] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1545worker2 # [ 0.581591] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1546client # [ 0.599724] ACPI: \_SB_.GSIA: Enabled at IRQ 161547worker1 # [ 0.587769] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1548worker2 # [ 0.582682] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1549lb # [ 0.595842] ACPI: \_SB_.GSIC: Enabled at IRQ 181550worker1 # [ 0.588899] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1551worker2 # [ 0.584315] ACPI: \_SB_.GSIA: Enabled at IRQ 161552client # [ 0.602075] ACPI: \_SB_.GSIB: Enabled at IRQ 171553lb # [ 0.597757] ACPI: \_SB_.GSID: Enabled at IRQ 191554worker1 # [ 0.590664] ACPI: \_SB_.GSIA: Enabled at IRQ 161555lb # [ 0.599375] PCI: CLS 0 bytes, default 641556worker2 # [ 0.586937] ACPI: \_SB_.GSIB: Enabled at IRQ 171557client # [ 0.604250] ACPI: \_SB_.GSIC: Enabled at IRQ 181558worker1 # [ 0.592986] ACPI: \_SB_.GSIB: Enabled at IRQ 171559lb # [ 0.600267] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1560worker2 # [ 0.588894] ACPI: \_SB_.GSIC: Enabled at IRQ 181561client # [ 0.606442] ACPI: \_SB_.GSID: Enabled at IRQ 191562lb # [ 0.601993] Trying to unpack rootfs image as initramfs...1563worker1 # [ 0.595194] ACPI: \_SB_.GSIC: Enabled at IRQ 181564client # [ 0.608169] PCI: CLS 0 bytes, default 641565worker2 # [ 0.590944] ACPI: \_SB_.GSID: Enabled at IRQ 191566worker1 # [ 0.597224] ACPI: \_SB_.GSID: Enabled at IRQ 191567worker2 # [ 0.592691] PCI: CLS 0 bytes, default 641568client # [ 0.609138] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1569worker1 # [ 0.598960] PCI: CLS 0 bytes, default 641570worker2 # [ 0.593632] Trying to unpack rootfs image as initramfs...1571client # [ 0.610934] Trying to unpack rootfs image as initramfs...1572worker1 # [ 0.599909] Trying to unpack rootfs image as initramfs...1573worker2 # [ 0.597753] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1574worker1 # [ 0.606247] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1575lb # [ 0.644765] Initialise system trusted keyrings1576lb # [ 0.647991] workingset: timestamp_bits=40 max_order=19 bucket_order=01577client # [ 0.655870] Initialise system trusted keyrings1578worker2 # [ 0.639449] Initialise system trusted keyrings1579worker1 # [ 0.645192] Initialise system trusted keyrings1580client # [ 0.656840] workingset: timestamp_bits=40 max_order=19 bucket_order=01581worker2 # [ 0.642565] workingset: timestamp_bits=40 max_order=19 bucket_order=01582worker1 # [ 0.648846] workingset: timestamp_bits=40 max_order=19 bucket_order=01583lb # [ 0.671639] Key type asymmetric registered1584lb # [ 0.672332] Asymmetric key parser 'x509' registered1585lb # [ 0.673200] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1586worker1 # [ 0.668011] Key type asymmetric registered1587client # [ 0.680134] Key type asymmetric registered1588worker1 # [ 0.668741] Asymmetric key parser 'x509' registered1589client # [ 0.680925] Asymmetric key parser 'x509' registered1590worker2 # [ 0.663951] Key type asymmetric registered1591client # [ 0.681779] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1592lb # [ 0.678686] io scheduler mq-deadline registered1593worker2 # [ 0.666547] Asymmetric key parser 'x509' registered1594worker1 # [ 0.671851] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1595lb # [ 0.679425] io scheduler kyber registered1596worker2 # [ 0.667393] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1597worker1 # [ 0.674863] io scheduler mq-deadline registered1598client # [ 0.687076] io scheduler mq-deadline registered1599worker1 # [ 0.675638] io scheduler kyber registered1600client # [ 0.687855] io scheduler kyber registered1601worker2 # [ 0.670679] io scheduler mq-deadline registered1602lb # [ 0.683091] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1603worker2 # [ 0.671396] io scheduler kyber registered1604lb # [ 0.684334] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1605worker1 # [ 0.678371] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1606client # [ 0.691498] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1607worker1 # [ 0.679692] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1608lb # [ 0.688336] Linux agpgart interface v0.1031609worker2 # [ 0.675548] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1610client # [ 0.692773] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1611lb # [ 0.689104] ACPI: bus type drm_connector registered1612worker2 # [ 0.676872] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1613worker1 # [ 0.684758] Linux agpgart interface v0.1031614client # [ 0.696693] Linux agpgart interface v0.1031615lb # [ 0.692341] usbcore: registered new interface driver usbserial_generic1616worker1 # [ 0.685540] ACPI: bus type drm_connector registered1617client # [ 0.697457] ACPI: bus type drm_connector registered1618lb # [ 0.693398] usbserial: USB Serial support registered for generic1619worker2 # [ 0.682608] Linux agpgart interface v0.1031620worker2 # [ 0.683350] ACPI: bus type drm_connector registered1621lb # [ 0.696542] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1622worker1 # [ 0.690246] usbcore: registered new interface driver usbserial_generic1623client # [ 0.702064] usbcore: registered new interface driver usbserial_generic1624lb # [ 0.696542] Please enable it if your BIOS has the CPPC option.1625worker1 # [ 0.691282] usbserial: USB Serial support registered for generic1626client # [ 0.703117] usbserial: USB Serial support registered for generic1627lb # [ 0.698661] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1628worker2 # [ 0.686648] usbcore: registered new interface driver usbserial_generic1629worker2 # [ 0.687698] usbserial: USB Serial support registered for generic1630client # [ 0.705922] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1631worker1 # [ 0.694810] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1632client # [ 0.705922] Please enable it if your BIOS has the CPPC option.1633lb # [ 0.702689] drop_monitor: Initializing network drop monitor service1634worker1 # [ 0.694810] Please enable it if your BIOS has the CPPC option.1635worker2 # [ 0.690528] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1636lb # [ 0.703806] NET: Registered PF_INET6 protocol family1637client # [ 0.708179] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1638worker2 # [ 0.690528] Please enable it if your BIOS has the CPPC option.1639worker1 # [ 0.696955] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1640worker1 # [ 0.698309] drop_monitor: Initializing network drop monitor service1641worker2 # [ 0.692653] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1642worker1 # [ 0.699471] NET: Registered PF_INET6 protocol family1643lb # [ 0.706985] Segment Routing with IPv61644client # [ 0.712063] drop_monitor: Initializing network drop monitor service1645client # [ 0.713280] NET: Registered PF_INET6 protocol family1646lb # [ 0.709560] In-situ OAM (IOAM) with IPv61647worker1 # [ 0.703295] Segment Routing with IPv61648worker2 # [ 0.697666] drop_monitor: Initializing network drop monitor service1649lb # [ 0.710598] IPI shorthand broadcast: enabled1650worker2 # [ 0.698816] NET: Registered PF_INET6 protocol family1651client # [ 0.716383] Segment Routing with IPv61652worker1 # [ 0.705817] In-situ OAM (IOAM) with IPv61653worker1 # [ 0.706898] IPI shorthand broadcast: enabled1654client # [ 0.718938] In-situ OAM (IOAM) with IPv61655worker2 # [ 0.702002] Segment Routing with IPv61656client # [ 0.720018] IPI shorthand broadcast: enabled1657worker2 # [ 0.704542] In-situ OAM (IOAM) with IPv61658worker2 # [ 0.705656] IPI shorthand broadcast: enabled1659lb # [ 0.718704] sched_clock: Marking stable (584013125, 134511017)->(796540927, -78016785)1660worker1 # [ 0.715263] sched_clock: Marking stable (582014576, 132774506)->(787057665, -72268583)1661lb # [ 0.723779] registered taskstats version 11662client # [ 0.728328] sched_clock: Marking stable (594014810, 133886591)->(804112179, -76210778)1663lb # [ 0.724690] Loading compiled-in X.509 certificates1664worker2 # [ 0.713816] sched_clock: Marking stable (582014101, 131497113)->(792798246, -79287032)1665worker1 # [ 0.720971] registered taskstats version 11666worker1 # [ 0.721945] Loading compiled-in X.509 certificates1667client # [ 0.734008] registered taskstats version 11668client # [ 0.734986] Loading compiled-in X.509 certificates1669worker2 # [ 0.718767] registered taskstats version 11670worker2 # [ 0.719763] Loading compiled-in X.509 certificates1671lb # [ 0.741116] Demotion targets for Node 0: null1672lb # [ 0.744579] Key type .fscrypt registered1673lb # [ 0.745258] Key type fscrypt-provisioning registered1674lb # [ 0.746194] ima: No TPM chip found, activating TPM-bypass!1675client # [ 0.751534] Demotion targets for Node 0: null1676worker1 # [ 0.740805] Demotion targets for Node 0: null1677lb # [ 0.748543] ima: Allocated hash algorithm: sha11678worker1 # [ 0.741743] Key type .fscrypt registered1679client # [ 0.753954] Key type .fscrypt registered1680lb # [ 0.749310] ima: No architecture policies found1681client # [ 0.754629] Key type fscrypt-provisioning registered1682worker1 # [ 0.743805] Key type fscrypt-provisioning registered1683client # [ 0.755599] ima: No TPM chip found, activating TPM-bypass!1684worker2 # [ 0.738533] Demotion targets for Node 0: null1685lb # [ 0.751537] PM: Magic number: 10:932:1271686worker2 # [ 0.739425] Key type .fscrypt registered1687worker1 # [ 0.744732] ima: No TPM chip found, activating TPM-bypass!1688lb # [ 0.752259] acpi device:02: hash matches1689worker2 # [ 0.740117] Key type fscrypt-provisioning registered1690worker1 # [ 0.745655] ima: Allocated hash algorithm: sha11691client # [ 0.759917] ima: Allocated hash algorithm: sha11692worker2 # [ 0.742629] ima: No TPM chip found, activating TPM-bypass!1693worker1 # [ 0.748833] ima: No architecture policies found1694lb # [ 0.755886] RAS: Correctable Errors collector initialized.1695worker2 # [ 0.743520] ima: Allocated hash algorithm: sha11696client # [ 0.760719] ima: No architecture policies found1697worker1 # [ 0.749808] PM: Magic number: 10:932:1271698client # [ 0.761642] PM: Magic number: 10:932:1271699worker2 # [ 0.744287] ima: No architecture policies found1700client # [ 0.762379] acpi device:02: hash matches1701worker1 # [ 0.750552] acpi device:02: hash matches1702worker2 # [ 0.748696] PM: Magic number: 10:932:1271703client # [ 0.765782] RAS: Correctable Errors collector initialized.1704worker1 # [ 0.754191] RAS: Correctable Errors collector initialized.1705worker2 # [ 0.749447] acpi device:02: hash matches1706lb # [ 0.764448] clk: Disabling unused clocks1707worker2 # [ 0.753049] RAS: Correctable Errors collector initialized.1708lb # [ 0.766558] PM: genpd: Disabling unused power domains1709client # [ 0.774439] clk: Disabling unused clocks1710worker1 # [ 0.762857] clk: Disabling unused clocks1711worker1 # [ 0.763559] PM: genpd: Disabling unused power domains1712client # [ 0.776915] PM: genpd: Disabling unused power domains1713worker2 # [ 0.761809] clk: Disabling unused clocks1714worker2 # [ 0.762481] PM: genpd: Disabling unused power domains1715lb # [ 0.930703] Freeing initrd memory: 29092K1716lb # [ 0.933879] Freeing unused decrypted memory: 2028K1717lb # [ 0.936461] Freeing unused kernel image (initmem) memory: 3644K1718worker2 # [ 0.924139] Freeing initrd memory: 29092K1719client # [ 0.941658] Freeing initrd memory: 29096K1720lb # [ 0.937538] Write protecting the kernel read-only data: 32768k1721worker1 # [ 0.931343] Freeing initrd memory: 29092K1722lb # [ 0.939353] Freeing unused kernel image (text/rodata gap) memory: 1208K1723worker2 # [ 0.927291] Freeing unused decrypted memory: 2028K1724client # [ 0.944830] Freeing unused decrypted memory: 2028K1725lb # [ 0.940848] Freeing unused kernel image (rodata/data gap) memory: 752K1726worker1 # [ 0.934613] Freeing unused decrypted memory: 2028K1727worker2 # [ 0.929898] Freeing unused kernel image (initmem) memory: 3644K1728client # [ 0.947570] Freeing unused kernel image (initmem) memory: 3644K1729worker2 # [ 0.931035] Write protecting the kernel read-only data: 32768k1730client # [ 0.948643] Write protecting the kernel read-only data: 32768k1731worker1 # [ 0.937358] Freeing unused kernel image (initmem) memory: 3644K1732worker1 # [ 0.938478] Write protecting the kernel read-only data: 32768k1733worker2 # [ 0.932919] Freeing unused kernel image (text/rodata gap) memory: 1208K1734client # [ 0.950559] Freeing unused kernel image (text/rodata gap) memory: 1208K1735worker2 # [ 0.934413] Freeing unused kernel image (rodata/data gap) memory: 752K1736client # [ 0.952068] Freeing unused kernel image (rodata/data gap) memory: 752K1737worker1 # [ 0.940360] Freeing unused kernel image (text/rodata gap) memory: 1208K1738worker1 # [ 0.941888] Freeing unused kernel image (rodata/data gap) memory: 752K1739lb # [ 0.992074] x86/mm: Checked W+X mappings: passed, no W+X pages found.1740lb # [ 0.993129] Run /init as init process1741worker2 # [ 0.985604] x86/mm: Checked W+X mappings: passed, no W+X pages found.1742client # [ 1.003172] x86/mm: Checked W+X mappings: passed, no W+X pages found.1743worker2 # [ 0.986655] Run /init as init process1744client # [ 1.004237] Run /init as init process1745worker1 # [ 0.993150] x86/mm: Checked W+X mappings: passed, no W+X pages found.1746worker1 # [ 0.994245] Run /init as init process1747lb # [ 1.003194] systemd[1]: Inserted module 'autofs4'1748worker2 # [ 0.997034] systemd[1]: Inserted module 'autofs4'1749client # [ 1.014519] systemd[1]: Inserted module 'autofs4'1750worker1 # [ 1.004495] systemd[1]: Inserted module 'autofs4'1751lb # [ 1.019428] fuse: init (API version 7.45)1752lb # [ 1.025191] ACPI: \_SB_.GSIG: Enabled at IRQ 221753worker2 # [ 1.013309] fuse: init (API version 7.45)1754client # [ 1.030852] fuse: init (API version 7.45)1755worker1 # [ 1.020895] fuse: init (API version 7.45)1756lb # [ 1.028015] ACPI: \_SB_.GSIH: Enabled at IRQ 231757lb # [ 1.031254] ACPI: \_SB_.GSIE: Enabled at IRQ 201758worker2 # [ 1.019147] ACPI: \_SB_.GSIG: Enabled at IRQ 221759client # [ 1.036639] ACPI: \_SB_.GSIG: Enabled at IRQ 221760lb # [ 1.033657] ACPI: \_SB_.GSIF: Enabled at IRQ 211761worker1 # [ 1.026685] ACPI: \_SB_.GSIG: Enabled at IRQ 221762client # [ 1.039007] ACPI: \_SB_.GSIH: Enabled at IRQ 231763worker2 # [ 1.021843] ACPI: \_SB_.GSIH: Enabled at IRQ 231764worker1 # [ 1.029096] ACPI: \_SB_.GSIH: Enabled at IRQ 231765client # [ 1.042141] ACPI: \_SB_.GSIE: Enabled at IRQ 201766worker2 # [ 1.025461] ACPI: \_SB_.GSIE: Enabled at IRQ 201767worker1 # [ 1.032166] ACPI: \_SB_.GSIE: Enabled at IRQ 201768client # [ 1.044373] ACPI: \_SB_.GSIF: Enabled at IRQ 211769worker2 # [ 1.028002] ACPI: \_SB_.GSIF: Enabled at IRQ 211770worker1 # [ 1.034417] ACPI: \_SB_.GSIF: Enabled at IRQ 211771lb # [ 1.062694] systemd[1]: Successfully made /usr/ read-only.1772client # [ 1.072012] systemd[1]: Successfully made /usr/ read-only.1773worker1 # [ 1.060641] systemd[1]: Successfully made /usr/ read-only.1774worker2 # [ 1.056682] systemd[1]: Successfully made /usr/ read-only.1775worker1 # [ 1.396138] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1776worker1 # [ 1.400805] systemd[1]: Detected virtualization kvm.1777client # [ 1.408264] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1778worker1 # [ 1.401606] systemd[1]: Detected architecture x86-64.1779worker2 # [ 1.392364] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1780client # [ 1.413692] systemd[1]: Detected virtualization kvm.1781worker1 # [ 1.402439] systemd[1]: Running in initrd.1782worker2 # [ 1.397030] systemd[1]: Detected virtualization kvm.1783client # [ 1.414502] systemd[1]: Detected architecture x86-64.1784lb # [ 1.399299] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1785worker2 # [ 1.397853] systemd[1]: Detected architecture x86-64.1786client # [ 1.415404] systemd[1]: Running in initrd.1787worker1 # [ 1.403515] systemd[1]: Initializing machine ID from random generator.1788worker2 # [ 1.398709] systemd[1]: Running in initrd.1789worker1 # [ 1.404658] systemd[1]: Hostname set to <worker1>.1790client # [ 1.416569] systemd[1]: Initializing machine ID from random generator.1791lb # [ 1.411136] systemd[1]: Detected virtualization kvm.1792worker2 # [ 1.399865] systemd[1]: Initializing machine ID from random generator.1793client # [ 1.417738] systemd[1]: Hostname set to <client>.1794worker2 # [ 1.401071] systemd[1]: Hostname set to <worker2>.1795lb # [ 1.413281] systemd[1]: Detected architecture x86-64.1796lb # [ 1.415463] systemd[1]: Running in initrd.1797lb # [ 1.418076] systemd[1]: Initializing machine ID from random generator.1798lb # [ 1.420946] systemd[1]: Hostname set to <lb>.1799worker2 # [ 1.595117] systemd[1]: bpf-restrict-fs: LSM BPF program attached1800client # [ 1.622145] systemd[1]: bpf-restrict-fs: LSM BPF program attached1801worker1 # [ 1.611345] systemd[1]: bpf-restrict-fs: LSM BPF program attached1802lb # [ 1.630919] systemd[1]: bpf-restrict-fs: LSM BPF program attached1803worker2 # [ 1.635624] systemd[1]: Queued start job for default target Initrd Default Target.1804worker2 # [ 1.638955] systemd[1]: Created slice Slice /system/modprobe.1805worker2 # [ 1.640106] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1806worker2 # [ 1.641455] systemd[1]: Expecting device /dev/disk/by-label/nixos...1807worker2 # [ 1.642541] systemd[1]: Reached target Path Units.1808worker2 # [ 1.643365] systemd[1]: Reached target Slice Units.1809worker2 # [ 1.644255] systemd[1]: Reached target Swaps.1810worker2 # [ 1.645065] systemd[1]: Reached target Timer Units.1811worker2 # [ 1.646024] systemd[1]: Listening on D-Bus System Message Bus Socket.1812worker2 # [ 1.647205] systemd[1]: Listening on Journal Socket (/dev/log).1813worker2 # [ 1.648332] systemd[1]: Listening on Journal Sockets.1814worker2 # [ 1.649305] systemd[1]: Listening on udev Control Socket.1815worker2 # [ 1.650294] systemd[1]: Listening on udev Kernel Socket.1816worker2 # [ 1.651218] systemd[1]: Reached target Socket Units.1817worker2 # [ 1.653068] systemd[1]: Starting Create List of Static Device Nodes...1818worker2 # [ 1.657579] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1819lb # [ 1.671698] systemd[1]: Queued start job for default target Initrd Default Target.1820worker1 # [ 1.665223] systemd[1]: Queued start job for default target Initrd Default Target.1821client # [ 1.677313] systemd[1]: Queued start job for default target Initrd Default Target.1822lb # [ 1.675944] systemd[1]: Created slice Slice /system/modprobe.1823worker1 # [ 1.669103] systemd[1]: Created slice Slice /system/modprobe.1824client # [ 1.681303] systemd[1]: Created slice Slice /system/modprobe.1825lb # [ 1.677136] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1826worker1 # [ 1.670242] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1827client # [ 1.682494] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1828worker2 # [ 1.665587] systemd[1]: Starting Load Kernel Module configfs...1829lb # [ 1.678553] systemd[1]: Expecting device /dev/disk/by-label/nixos...1830worker1 # [ 1.671618] systemd[1]: Expecting device /dev/disk/by-label/nixos...1831client # [ 1.683928] systemd[1]: Expecting device /dev/disk/by-label/nixos...1832lb # [ 1.679597] systemd[1]: Reached target Path Units.1833worker1 # [ 1.672708] systemd[1]: Reached target Path Units.1834client # [ 1.685008] systemd[1]: Reached target Path Units.1835lb # [ 1.680392] systemd[1]: Reached target Slice Units.1836worker1 # [ 1.673558] systemd[1]: Reached target Slice Units.1837client # [ 1.685859] systemd[1]: Reached target Slice Units.1838lb # [ 1.681258] systemd[1]: Reached target Swaps.1839worker1 # [ 1.674488] systemd[1]: Reached target Swaps.1840client # [ 1.686753] systemd[1]: Reached target Swaps.1841lb # [ 1.682170] systemd[1]: Reached target Timer Units.1842worker1 # [ 1.675291] systemd[1]: Reached target Timer Units.1843client # [ 1.687572] systemd[1]: Reached target Timer Units.1844lb # [ 1.683145] systemd[1]: Listening on D-Bus System Message Bus Socket.1845worker1 # [ 1.676248] systemd[1]: Listening on D-Bus System Message Bus Socket.1846client # [ 1.688550] systemd[1]: Listening on D-Bus System Message Bus Socket.1847lb # [ 1.684400] systemd[1]: Listening on Journal Socket (/dev/log).1848worker1 # [ 1.677462] systemd[1]: Listening on Journal Socket (/dev/log).1849client # [ 1.689756] systemd[1]: Listening on Journal Socket (/dev/log).1850lb # [ 1.685641] systemd[1]: Listening on Journal Sockets.1851worker1 # [ 1.678677] systemd[1]: Listening on Journal Sockets.1852client # [ 1.690930] systemd[1]: Listening on Journal Sockets.1853lb # [ 1.686647] systemd[1]: Listening on udev Control Socket.1854worker1 # [ 1.679689] systemd[1]: Listening on udev Control Socket.1855client # [ 1.691943] systemd[1]: Listening on udev Control Socket.1856lb # [ 1.687654] systemd[1]: Listening on udev Kernel Socket.1857worker1 # [ 1.680736] systemd[1]: Listening on udev Kernel Socket.1858client # [ 1.692985] systemd[1]: Listening on udev Kernel Socket.1859lb # [ 1.688599] systemd[1]: Reached target Socket Units.1860worker1 # [ 1.681712] systemd[1]: Reached target Socket Units.1861client # [ 1.694013] systemd[1]: Reached target Socket Units.1862lb # [ 1.690454] systemd[1]: Starting Create List of Static Device Nodes...1863worker1 # [ 1.683485] systemd[1]: Starting Create List of Static Device Nodes...1864client # [ 1.695906] systemd[1]: Starting Create List of Static Device Nodes...1865worker2 # [ 1.679886] systemd[1]: Starting Journal Service...1866lb # [ 1.694392] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1867worker1 # [ 1.687858] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1868client # [ 1.699770] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1869client # [ 1.705065] systemd[1]: Starting Load Kernel Module configfs...1870lb # [ 1.700613] systemd[1]: Starting Load Kernel Module configfs...1871worker1 # [ 1.695858] systemd[1]: Starting Load Kernel Module configfs...1872lb # [ 1.713760] systemd[1]: Starting Journal Service...1873client # [ 1.719559] systemd[1]: Starting Journal Service...1874worker1 # [ 1.708525] systemd[1]: Starting Journal Service...1875worker2 # [ 1.693462] systemd[1]: Starting Load Kernel Modules...1876worker2 # [ 1.713019] netfs: FS-Cache loaded1877worker2 # [ 1.717810] 9pnet: Installing 9P2000 support1878lb # [ 1.725743] systemd[1]: Starting Load Kernel Modules...1879client # [ 1.731971] systemd[1]: Starting Load Kernel Modules...1880worker2 # [ 1.727188] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1881worker1 # [ 1.722884] systemd[1]: Starting Load Kernel Modules...1882worker1 # [ 1.742455] netfs: FS-Cache loaded1883lb # [ 1.749775] netfs: FS-Cache loaded1884client # [ 1.754462] netfs: FS-Cache loaded1885worker2 # [ 1.738741] systemd[1]: Starting Coldplug All udev Devices...1886worker1 # [ 1.747256] 9pnet: Installing 9P2000 support1887client # [ 1.761481] 9pnet: Installing 9P2000 support1888lb # [ 1.757740] 9pnet: Installing 9P2000 support1889client # [ 1.763222] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1890lb # [ 1.760651] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1891worker1 # [ 1.753966] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1892worker2 # [ 1.752305] systemd[1]: Finished Create List of Static Device Nodes.1893client # [ 1.774132] systemd[1]: Starting Coldplug All udev Devices...1894worker1 # [ 1.762871] systemd[1]: Starting Coldplug All udev Devices...1895worker2 # [ 1.758143] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1896lb # [ 1.772610] systemd[1]: Starting Coldplug All udev Devices...1897worker2 # [ 1.761262] systemd-journald[67]: Collecting audit messages is disabled.1898worker2 # [ 1.766067] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1899worker1 # [ 1.776895] systemd[1]: Finished Create List of Static Device Nodes.1900client # [ 1.789026] systemd[1]: Finished Create List of Static Device Nodes.1901worker2 # [ 1.772980] systemd[1]: modprobe@configfs.service: Deactivated successfully.1902worker2 # [ 1.777108] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1903worker1 # [ 1.783385] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1904lb # [ 1.790849] systemd[1]: Finished Create List of Static Device Nodes.1905client # [ 1.795597] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1906worker2 # [ 1.779482] systemd[1]: Finished Load Kernel Module configfs.1907lb # [ 1.796246] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1908worker1 # [ 1.789843] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1909client # [ 1.802713] systemd-journald[67]: Collecting audit messages is disabled.1910worker2 # [ 1.784942] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1911client # [ 1.804283] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1912worker1 # [ 1.794528] systemd-journald[67]: Collecting audit messages is disabled.1913lb # [ 1.802409] systemd-journald[67]: Collecting audit messages is disabled.1914worker2 # [ 1.789647] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1915worker1 # [ 1.797220] systemd[1]: modprobe@configfs.service: Deactivated successfully.1916client # [ 1.809304] systemd[1]: modprobe@configfs.service: Deactivated successfully.1917lb # [ 1.804958] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1918worker2 # [ 1.795612] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1919worker1 # [ 1.803254] systemd[1]: Finished Load Kernel Module configfs.1920client # [ 1.816300] systemd[1]: Finished Load Kernel Module configfs.1921lb # [ 1.812009] systemd[1]: modprobe@configfs.service: Deactivated successfully.1922worker1 # [ 1.806125] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1923client # [ 1.820378] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1924lb # [ 1.817929] systemd[1]: Finished Load Kernel Module configfs.1925worker1 # [ 1.810262] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1926client # [ 1.824085] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1927lb # [ 1.821755] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1928worker1 # [ 1.816822] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1929lb # [ 1.824434] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1930client # [ 1.829923] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1931worker1 # [ 1.821874] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1932worker2 # [ 1.817189] systemd[1]: Finished Load Kernel Modules.1933client # [ 1.835178] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1934lb # [ 1.830729] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1935lb # [ 1.837857] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1936worker2 # [ 1.825714] systemd[1]: Starting Apply Kernel Variables...1937worker1 # [ 1.847331] systemd[1]: Finished Load Kernel Modules.1938client # [ 1.861466] systemd[1]: Finished Load Kernel Modules.1939worker2 # [ 1.843745] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1940worker1 # [ 1.855983] systemd[1]: Starting Apply Kernel Variables...1941client # [ 1.868110] systemd[1]: Starting Apply Kernel Variables...1942lb # [ 1.866625] systemd[1]: Finished Load Kernel Modules.1943worker2 # [ 1.856005] systemd[1]: Starting Create Static Device Nodes in /dev...1944worker1 # [ 1.867904] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1945lb # [ 1.875723] systemd[1]: Starting Apply Kernel Variables...1946client # [ 1.883041] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1947worker1 # [ 1.877022] systemd[1]: Starting Create Static Device Nodes in /dev...1948worker2 # [ 1.874627] systemd[1]: Finished Apply Kernel Variables.1949client # [ 1.893120] systemd[1]: Starting Create Static Device Nodes in /dev...1950lb # [ 1.890868] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1951worker2 # [ 1.886095] systemd[1]: Started Journal Service.1952client # [ 1.906532] systemd[1]: Finished Apply Kernel Variables.1953worker2 # [ 1.755746] systemd-modules-load[68]: Inserted module 'dm_mod'1954lb # [ 1.902716] systemd[1]: Starting Create Static Device Nodes in /dev...1955worker2 # [ 1.758988] systemd-modules-load[68]: Inserted module 'virtio_balloon'1956worker2 # [ 1.760274] systemd-modules-load[68]: Inserted module 'virtio_gpu'1957worker1 # [ 1.897345] systemd[1]: Finished Apply Kernel Variables.1958worker2 # [ 1.767206] systemd[1]: Finished Create Static Device Nodes in /dev.1959lb # [ 1.915611] systemd[1]: Finished Apply Kernel Variables.1960worker2 # [ 1.774123] systemd[1]: Reached target Preparation for Local File Systems.1961worker1 # [ 1.911316] systemd[1]: Finished Create Static Device Nodes in /dev.1962worker2 # [ 1.775198] systemd[1]: Reached target Local File Systems.1963client # [ 1.924359] systemd[1]: Started Journal Service.1964worker1 # [ 1.780126] systemd-modules-load[68]: Inserted module 'dm_mod'1965worker2 # [ 1.778136] systemd[1]: Starting Create System Files and Directories...1966worker1 # [ 1.782383] systemd-modules-load[68]: Inserted module 'virtio_balloon'1967worker1 # [ 1.916176] systemd[1]: Started Journal Service.1968client # [ 1.791515] systemd-modules-load[68]: Inserted module 'dm_mod'1969worker1 # [ 1.784314] systemd-modules-load[68]: Inserted module 'virtio_gpu'1970client # [ 1.795313] systemd-modules-load[68]: Inserted module 'virtio_balloon'1971client # [ 1.796876] systemd-modules-load[68]: Inserted module 'virtio_gpu'1972worker2 # [ 1.782658] systemd[1]: Starting Rule-based Manager for Device Events and Files...1973lb # [ 1.928633] systemd[1]: Started Journal Service.1974lb # [ 1.796321] systemd-modules-load[68]: Inserted module 'dm_mod'1975client # [ 1.802474] systemd[1]: Finished Create Static Device Nodes in /dev.1976lb # [ 1.797722] systemd-modules-load[68]: Inserted module 'virtio_balloon'1977lb # [ 1.800294] systemd-modules-load[68]: Inserted module 'virtio_gpu'1978worker1 # [ 1.795082] systemd[1]: Reached target Preparation for Local File Systems.1979worker1 # [ 1.797067] systemd[1]: Reached target Local File Systems.1980client # [ 1.809070] systemd[1]: Reached target Preparation for Local File Systems.1981worker1 # [ 1.800044] systemd[1]: Starting Create System Files and Directories...1982client # [ 1.811131] systemd[1]: Reached target Local File Systems.1983client # [ 1.814078] systemd[1]: Starting Create System Files and Directories...1984lb # [ 1.810901] systemd[1]: Finished Create Static Device Nodes in /dev.1985worker1 # [ 1.807071] systemd[1]: Starting Rule-based Manager for Device Events and Files...1986client # [ 1.819072] systemd[1]: Starting Rule-based Manager for Device Events and Files...1987lb # [ 1.814085] systemd[1]: Reached target Preparation for Local File Systems.1988lb # [ 1.815088] systemd[1]: Reached target Local File Systems.1989lb # [ 1.818069] systemd[1]: Starting Create System Files and Directories...1990worker2 # [ 1.811060] systemd[1]: Finished Create System Files and Directories.1991lb # [ 1.822602] systemd[1]: Starting Rule-based Manager for Device Events and Files...1992worker1 # [ 1.831336] systemd[1]: Finished Create System Files and Directories.1993client # [ 1.846352] systemd[1]: Finished Create System Files and Directories.1994worker2 # [ 1.838282] systemd-udevd[83]: Using default interface naming scheme 'v261'.1995lb # [ 1.852209] systemd[1]: Finished Create System Files and Directories.1996worker1 # [ 1.854246] systemd-udevd[83]: Using default interface naming scheme 'v261'.1997client # [ 1.869966] systemd-udevd[83]: Using default interface naming scheme 'v261'.1998worker2 # [ 1.863073] systemd[1]: Started Rule-based Manager for Device Events and Files.1999lb # [ 1.873352] systemd-udevd[84]: Using default interface naming scheme 'v261'.2000worker1 # [ 1.878062] systemd[1]: Started Rule-based Manager for Device Events and Files.
loading 2,144 hidden lines…
4145worker2 # [ 17.923186] nix-grpc-daemon[787]: E0000 00:00:1789992486.922564 826 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4146worker2 # [ 17.936169] nix-grpc-daemon[787]: E0000 00:00:1789992486.935533 826 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4147worker1 # [ 17.940874] nix-grpc-daemon[763]: ts=2026-09-21T12:08:06Z level=info event=scheduler_connected addr=lb:500514148worker1 # [ 17.945481] nix-grpc-daemon[763]: ts=2026-09-21T12:08:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4149worker2 # [ 17.958851] nix-grpc-daemon[787]: ts=2026-09-21T12:08:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4150worker1 # [ 18.145756] nix-grpc-daemon[763]: ts=2026-09-21T12:08:06Z level=info event=scheduler_connected addr=lb:500514151worker1 # [ 18.147732] nix-grpc-daemon[763]: ts=2026-09-21T12:08:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4152worker2 # [ 18.162508] nix-grpc-daemon[787]: ts=2026-09-21T12:08:07Z level=info event=scheduler_connected addr=lb:500514153worker2 # [ 18.166157] nix-grpc-daemon[787]: ts=2026-09-21T12:08:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4154worker1 # [ 18.348878] nix-grpc-daemon[763]: ts=2026-09-21T12:08:06Z level=info event=scheduler_connected addr=lb:500514155worker1 # [ 18.353412] nix-grpc-daemon[763]: ts=2026-09-21T12:08:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4156worker2 # [ 18.364318] nix-grpc-daemon[787]: ts=2026-09-21T12:08:07Z level=info event=scheduler_connected addr=lb:500514157worker2 # [ 18.368214] nix-grpc-daemon[787]: ts=2026-09-21T12:08:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4158worker1 # [ 18.554419] nix-grpc-daemon[763]: ts=2026-09-21T12:08:06Z level=info event=scheduler_connected addr=lb:500514159worker1 # [ 18.558788] nix-grpc-daemon[763]: ts=2026-09-21T12:08:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4160worker2 # [ 18.569487] nix-grpc-daemon[787]: ts=2026-09-21T12:08:07Z level=info event=scheduler_connected addr=lb:500514161worker2 # [ 18.573285] nix-grpc-daemon[787]: ts=2026-09-21T12:08:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4162worker1 # [ 18.760802] nix-grpc-daemon[763]: ts=2026-09-21T12:08:06Z level=info event=scheduler_connected addr=lb:500514163worker1 # [ 18.764134] nix-grpc-daemon[763]: ts=2026-09-21T12:08:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4164worker2 # [ 18.774511] nix-grpc-daemon[787]: ts=2026-09-21T12:08:07Z level=info event=scheduler_connected addr=lb:500514165worker2 # [ 18.778318] nix-grpc-daemon[787]: ts=2026-09-21T12:08:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4166worker1 # [ 18.961491] nix-grpc-daemon[763]: ts=2026-09-21T12:08:07Z level=info event=scheduler_connected addr=lb:500514167worker1 # [ 18.965584] nix-grpc-daemon[763]: ts=2026-09-21T12:08:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4168worker2 # [ 18.981344] nix-grpc-daemon[787]: ts=2026-09-21T12:08:07Z level=info event=scheduler_connected addr=lb:500514169worker2 # [ 18.984497] nix-grpc-daemon[787]: ts=2026-09-21T12:08:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4170worker1 # [ 19.166505] nix-grpc-daemon[763]: ts=2026-09-21T12:08:07Z level=info event=scheduler_connected addr=lb:500514171worker1 # [ 19.170355] nix-grpc-daemon[763]: ts=2026-09-21T12:08:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4172worker2 # [ 19.182326] nix-grpc-daemon[787]: ts=2026-09-21T12:08:08Z level=info event=scheduler_connected addr=lb:500514173worker2 # [ 19.186200] nix-grpc-daemon[787]: ts=2026-09-21T12:08:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4174worker1 # [ 19.371297] nix-grpc-daemon[763]: ts=2026-09-21T12:08:07Z level=info event=scheduler_connected addr=lb:500514175worker1 # [ 19.375170] nix-grpc-daemon[763]: ts=2026-09-21T12:08:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4176worker2 # [ 19.387434] nix-grpc-daemon[787]: ts=2026-09-21T12:08:08Z level=info event=scheduler_connected addr=lb:500514177worker2 # [ 19.391378] nix-grpc-daemon[787]: ts=2026-09-21T12:08:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4178worker1 # [ 19.576188] nix-grpc-daemon[763]: ts=2026-09-21T12:08:07Z level=info event=scheduler_connected addr=lb:500514179worker1 # [ 19.580124] nix-grpc-daemon[763]: ts=2026-09-21T12:08:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4180worker2 # [ 19.592608] nix-grpc-daemon[787]: ts=2026-09-21T12:08:08Z level=info event=scheduler_connected addr=lb:500514181worker2 # [ 19.596478] nix-grpc-daemon[787]: ts=2026-09-21T12:08:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4182worker1 # [ 19.781605] nix-grpc-daemon[763]: ts=2026-09-21T12:08:08Z level=info event=scheduler_connected addr=lb:500514183worker1 # [ 19.785335] nix-grpc-daemon[763]: ts=2026-09-21T12:08:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4184worker2 # [ 19.797857] nix-grpc-daemon[787]: ts=2026-09-21T12:08:08Z level=info event=scheduler_connected addr=lb:500514185worker2 # [ 19.801716] nix-grpc-daemon[787]: ts=2026-09-21T12:08:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4186worker1 # [ 19.986336] nix-grpc-daemon[763]: ts=2026-09-21T12:08:08Z level=info event=scheduler_connected addr=lb:500514187worker1 # [ 19.990380] nix-grpc-daemon[763]: ts=2026-09-21T12:08:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4188worker2 # [ 20.004103] nix-grpc-daemon[787]: ts=2026-09-21T12:08:09Z level=info event=scheduler_connected addr=lb:500514189worker2 # [ 20.007281] nix-grpc-daemon[787]: ts=2026-09-21T12:08:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4190worker1 # [ 20.191174] nix-grpc-daemon[763]: ts=2026-09-21T12:08:08Z level=info event=scheduler_connected addr=lb:500514191worker1 # [ 20.195160] nix-grpc-daemon[763]: ts=2026-09-21T12:08:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4192worker2 # [ 20.205479] nix-grpc-daemon[787]: ts=2026-09-21T12:08:09Z level=info event=scheduler_connected addr=lb:500514193worker2 # [ 20.209319] nix-grpc-daemon[787]: ts=2026-09-21T12:08:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4194worker1 # [ 20.396229] nix-grpc-daemon[763]: ts=2026-09-21T12:08:08Z level=info event=scheduler_connected addr=lb:500514195worker1 # [ 20.400200] nix-grpc-daemon[763]: ts=2026-09-21T12:08:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4196worker2 # [ 20.410475] nix-grpc-daemon[787]: ts=2026-09-21T12:08:09Z level=info event=scheduler_connected addr=lb:500514197worker2 # [ 20.414460] nix-grpc-daemon[787]: ts=2026-09-21T12:08:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4198worker1 # [ 20.600802] nix-grpc-daemon[763]: ts=2026-09-21T12:08:08Z level=info event=scheduler_connected addr=lb:500514199worker1 # [ 20.604278] nix-grpc-daemon[763]: ts=2026-09-21T12:08:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4200worker2 # [ 20.615886] nix-grpc-daemon[787]: ts=2026-09-21T12:08:09Z level=info event=scheduler_connected addr=lb:500514201worker2 # [ 20.619749] nix-grpc-daemon[787]: ts=2026-09-21T12:08:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4202worker1 # [ 20.805758] nix-grpc-daemon[763]: ts=2026-09-21T12:08:09Z level=info event=scheduler_connected addr=lb:500514203worker1 # [ 20.809501] nix-grpc-daemon[763]: ts=2026-09-21T12:08:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4204worker2 # [ 20.820220] nix-grpc-daemon[787]: ts=2026-09-21T12:08:09Z level=info event=scheduler_connected addr=lb:500514205worker2 # [ 20.823285] nix-grpc-daemon[787]: ts=2026-09-21T12:08:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4206worker1 # [ 21.010554] nix-grpc-daemon[763]: ts=2026-09-21T12:08:09Z level=info event=scheduler_connected addr=lb:500514207worker1 # [ 21.014585] nix-grpc-daemon[763]: ts=2026-09-21T12:08:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4208worker2 # [ 21.025882] nix-grpc-daemon[787]: ts=2026-09-21T12:08:10Z level=info event=scheduler_connected addr=lb:500514209worker2 # [ 21.029117] nix-grpc-daemon[787]: ts=2026-09-21T12:08:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4210worker1 # [ 21.215369] nix-grpc-daemon[763]: ts=2026-09-21T12:08:09Z level=info event=scheduler_connected addr=lb:500514211worker1 # [ 21.219278] nix-grpc-daemon[763]: ts=2026-09-21T12:08:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4212worker2 # [ 21.226794] nix-grpc-daemon[787]: ts=2026-09-21T12:08:10Z level=info event=scheduler_connected addr=lb:500514213worker2 # [ 21.230598] nix-grpc-daemon[787]: ts=2026-09-21T12:08:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4214worker1 # [ 21.420446] nix-grpc-daemon[763]: ts=2026-09-21T12:08:09Z level=info event=scheduler_connected addr=lb:500514215worker1 # [ 21.425090] nix-grpc-daemon[763]: ts=2026-09-21T12:08:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4216worker2 # [ 21.431886] nix-grpc-daemon[787]: ts=2026-09-21T12:08:10Z level=info event=scheduler_connected addr=lb:500514217worker2 # [ 21.436109] nix-grpc-daemon[787]: ts=2026-09-21T12:08:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4218worker1 # [ 21.625896] nix-grpc-daemon[763]: ts=2026-09-21T12:08:09Z level=info event=scheduler_connected addr=lb:500514219worker1 # [ 21.629824] nix-grpc-daemon[763]: ts=2026-09-21T12:08:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4220worker2 # [ 21.637358] nix-grpc-daemon[787]: ts=2026-09-21T12:08:10Z level=info event=scheduler_connected addr=lb:500514221worker2 # [ 21.641204] nix-grpc-daemon[787]: ts=2026-09-21T12:08:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4222worker1 # [ 21.831637] nix-grpc-daemon[763]: ts=2026-09-21T12:08:10Z level=info event=scheduler_connected addr=lb:500514223worker1 # [ 21.835337] nix-grpc-daemon[763]: ts=2026-09-21T12:08:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4224worker2 # [ 21.842464] nix-grpc-daemon[787]: ts=2026-09-21T12:08:10Z level=info event=scheduler_connected addr=lb:500514225worker2 # [ 21.846244] nix-grpc-daemon[787]: ts=2026-09-21T12:08:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4226worker1 # [ 22.033742] nix-grpc-daemon[763]: ts=2026-09-21T12:08:10Z level=info event=scheduler_connected addr=lb:500514227worker1 # [ 22.037706] nix-grpc-daemon[763]: ts=2026-09-21T12:08:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4228worker2 # [ 22.048359] nix-grpc-daemon[787]: ts=2026-09-21T12:08:11Z level=info event=scheduler_connected addr=lb:500514229worker2 # [ 22.051902] nix-grpc-daemon[787]: ts=2026-09-21T12:08:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4230worker1 # [ 22.239141] nix-grpc-daemon[763]: ts=2026-09-21T12:08:10Z level=info event=scheduler_connected addr=lb:500514231worker1 # [ 22.243128] nix-grpc-daemon[763]: ts=2026-09-21T12:08:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4232worker2 # [ 22.253379] nix-grpc-daemon[787]: ts=2026-09-21T12:08:11Z level=info event=scheduler_connected addr=lb:500514233worker2 # [ 22.257170] nix-grpc-daemon[787]: ts=2026-09-21T12:08:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4234worker1 # [ 22.444122] nix-grpc-daemon[763]: ts=2026-09-21T12:08:10Z level=info event=scheduler_connected addr=lb:500514235worker1 # [ 22.448090] nix-grpc-daemon[763]: ts=2026-09-21T12:08:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4236worker2 # [ 22.458304] nix-grpc-daemon[787]: ts=2026-09-21T12:08:11Z level=info event=scheduler_connected addr=lb:500514237worker2 # [ 22.461980] nix-grpc-daemon[787]: ts=2026-09-21T12:08:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4238worker1 # [ 22.648804] nix-grpc-daemon[763]: ts=2026-09-21T12:08:10Z level=info event=scheduler_connected addr=lb:500514239worker1 # [ 22.652580] nix-grpc-daemon[763]: ts=2026-09-21T12:08:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4240worker2 # [ 22.663425] nix-grpc-daemon[787]: ts=2026-09-21T12:08:11Z level=info event=scheduler_connected addr=lb:500514241worker2 # [ 22.667187] nix-grpc-daemon[787]: ts=2026-09-21T12:08:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4242worker1 # [ 22.854432] nix-grpc-daemon[763]: ts=2026-09-21T12:08:11Z level=info event=scheduler_connected addr=lb:500514243worker1 # [ 22.858212] nix-grpc-daemon[763]: ts=2026-09-21T12:08:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4244worker2 # [ 22.868070] nix-grpc-daemon[787]: ts=2026-09-21T12:08:11Z level=info event=scheduler_connected addr=lb:500514245worker2 # [ 22.871789] nix-grpc-daemon[787]: ts=2026-09-21T12:08:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4246worker1 # [ 23.059175] nix-grpc-daemon[763]: ts=2026-09-21T12:08:11Z level=info event=scheduler_connected addr=lb:500514247worker1 # [ 23.063144] nix-grpc-daemon[763]: ts=2026-09-21T12:08:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4248worker2 # [ 23.074473] nix-grpc-daemon[787]: ts=2026-09-21T12:08:12Z level=info event=scheduler_connected addr=lb:500514249worker2 # [ 23.077546] nix-grpc-daemon[787]: ts=2026-09-21T12:08:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4250worker1 # [ 23.263975] nix-grpc-daemon[763]: ts=2026-09-21T12:08:11Z level=info event=scheduler_connected addr=lb:500514251worker1 # [ 23.267935] nix-grpc-daemon[763]: ts=2026-09-21T12:08:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4252worker2 # [ 23.275368] nix-grpc-daemon[787]: ts=2026-09-21T12:08:12Z level=info event=scheduler_connected addr=lb:500514253worker2 # [ 23.279159] nix-grpc-daemon[787]: ts=2026-09-21T12:08:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4254worker1 # [ 23.468846] nix-grpc-daemon[763]: ts=2026-09-21T12:08:11Z level=info event=scheduler_connected addr=lb:500514255worker1 # [ 23.472647] nix-grpc-daemon[763]: ts=2026-09-21T12:08:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4256worker2 # [ 23.480390] nix-grpc-daemon[787]: ts=2026-09-21T12:08:12Z level=info event=scheduler_connected addr=lb:500514257worker2 # [ 23.484160] nix-grpc-daemon[787]: ts=2026-09-21T12:08:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4258worker1 # [ 23.673629] nix-grpc-daemon[763]: ts=2026-09-21T12:08:11Z level=info event=scheduler_connected addr=lb:500514259worker1 # [ 23.677464] nix-grpc-daemon[763]: ts=2026-09-21T12:08:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4260worker2 # [ 23.685275] nix-grpc-daemon[787]: ts=2026-09-21T12:08:12Z level=info event=scheduler_connected addr=lb:500514261worker2 # [ 23.689111] nix-grpc-daemon[787]: ts=2026-09-21T12:08:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4262worker1 # [ 23.879219] nix-grpc-daemon[763]: ts=2026-09-21T12:08:12Z level=info event=scheduler_connected addr=lb:500514263worker1 # [ 23.882846] nix-grpc-daemon[763]: ts=2026-09-21T12:08:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4264worker2 # [ 23.890225] nix-grpc-daemon[787]: ts=2026-09-21T12:08:12Z level=info event=scheduler_connected addr=lb:500514265worker2 # [ 23.894075] nix-grpc-daemon[787]: ts=2026-09-21T12:08:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4266worker1 # [ 24.083721] nix-grpc-daemon[763]: ts=2026-09-21T12:08:12Z level=info event=scheduler_connected addr=lb:500514267worker1 # [ 24.087629] nix-grpc-daemon[763]: ts=2026-09-21T12:08:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4268worker2 # [ 24.096522] nix-grpc-daemon[787]: ts=2026-09-21T12:08:13Z level=info event=scheduler_connected addr=lb:500514269worker2 # [ 24.099584] nix-grpc-daemon[787]: ts=2026-09-21T12:08:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4270worker1 # [ 24.288557] nix-grpc-daemon[763]: ts=2026-09-21T12:08:12Z level=info event=scheduler_connected addr=lb:500514271worker1 # [ 24.292461] nix-grpc-daemon[763]: ts=2026-09-21T12:08:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4272worker2 # [ 24.297855] nix-grpc-daemon[787]: ts=2026-09-21T12:08:13Z level=info event=scheduler_connected addr=lb:500514273worker2 # [ 24.301665] nix-grpc-daemon[787]: ts=2026-09-21T12:08:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4274worker1 # [ 24.493362] nix-grpc-daemon[763]: ts=2026-09-21T12:08:12Z level=info event=scheduler_connected addr=lb:500514275worker1 # [ 24.497245] nix-grpc-daemon[763]: ts=2026-09-21T12:08:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4276worker2 # [ 24.502796] nix-grpc-daemon[787]: ts=2026-09-21T12:08:13Z level=info event=scheduler_connected addr=lb:500514277worker2 # [ 24.506534] nix-grpc-daemon[787]: ts=2026-09-21T12:08:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4278worker1 # [ 24.697935] nix-grpc-daemon[763]: ts=2026-09-21T12:08:12Z level=info event=scheduler_connected addr=lb:500514279worker1 # [ 24.701902] nix-grpc-daemon[763]: ts=2026-09-21T12:08:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4280worker2 # [ 24.707799] nix-grpc-daemon[787]: ts=2026-09-21T12:08:13Z level=info event=scheduler_connected addr=lb:500514281worker2 # [ 24.711600] nix-grpc-daemon[787]: ts=2026-09-21T12:08:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4282worker1 # [ 24.903613] nix-grpc-daemon[763]: ts=2026-09-21T12:08:13Z level=info event=scheduler_connected addr=lb:500514283worker1 # [ 24.907415] nix-grpc-daemon[763]: ts=2026-09-21T12:08:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4284worker2 # [ 24.913252] nix-grpc-daemon[787]: ts=2026-09-21T12:08:13Z level=info event=scheduler_connected addr=lb:500514285worker2 # [ 24.918147] nix-grpc-daemon[787]: ts=2026-09-21T12:08:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4286worker1 # [ 25.105455] nix-grpc-daemon[763]: ts=2026-09-21T12:08:13Z level=info event=scheduler_connected addr=lb:500514287worker1 # [ 25.109335] nix-grpc-daemon[763]: ts=2026-09-21T12:08:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4288worker2 # [ 25.116673] nix-grpc-daemon[787]: ts=2026-09-21T12:08:14Z level=info event=scheduler_connected addr=lb:500514289worker2 # [ 25.119784] nix-grpc-daemon[787]: ts=2026-09-21T12:08:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4290worker1 # [ 25.310314] nix-grpc-daemon[763]: ts=2026-09-21T12:08:13Z level=info event=scheduler_connected addr=lb:500514291worker1 # [ 25.314185] nix-grpc-daemon[763]: ts=2026-09-21T12:08:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4292worker2 # [ 25.317793] nix-grpc-daemon[787]: ts=2026-09-21T12:08:14Z level=info event=scheduler_connected addr=lb:500514293worker2 # [ 25.321781] nix-grpc-daemon[787]: ts=2026-09-21T12:08:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4294worker1 # [ 25.514804] nix-grpc-daemon[763]: ts=2026-09-21T12:08:13Z level=info event=scheduler_connected addr=lb:500514295worker1 # [ 25.518731] nix-grpc-daemon[763]: ts=2026-09-21T12:08:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4296worker2 # [ 25.523162] nix-grpc-daemon[787]: ts=2026-09-21T12:08:14Z level=info event=scheduler_connected addr=lb:500514297worker2 # [ 25.527122] nix-grpc-daemon[787]: ts=2026-09-21T12:08:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4298worker1 # [ 25.719309] nix-grpc-daemon[763]: ts=2026-09-21T12:08:13Z level=info event=scheduler_connected addr=lb:500514299worker1 # [ 25.723102] nix-grpc-daemon[763]: ts=2026-09-21T12:08:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4300worker2 # [ 25.728462] nix-grpc-daemon[787]: ts=2026-09-21T12:08:14Z level=info event=scheduler_connected addr=lb:500514301worker2 # [ 25.732272] nix-grpc-daemon[787]: ts=2026-09-21T12:08:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4302worker1 # [ 25.925254] nix-grpc-daemon[763]: ts=2026-09-21T12:08:14Z level=info event=scheduler_connected addr=lb:500514303worker1 # [ 25.928369] nix-grpc-daemon[763]: ts=2026-09-21T12:08:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4304worker2 # [ 25.933060] nix-grpc-daemon[787]: ts=2026-09-21T12:08:14Z level=info event=scheduler_connected addr=lb:500514305worker2 # [ 25.936863] nix-grpc-daemon[787]: ts=2026-09-21T12:08:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4306worker1 # [ 26.126451] nix-grpc-daemon[763]: ts=2026-09-21T12:08:14Z level=info event=scheduler_connected addr=lb:500514307worker1 # [ 26.130321] nix-grpc-daemon[763]: ts=2026-09-21T12:08:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4308worker2 # [ 26.139322] nix-grpc-daemon[787]: ts=2026-09-21T12:08:15Z level=info event=scheduler_connected addr=lb:500514309worker2 # [ 26.142592] nix-grpc-daemon[787]: ts=2026-09-21T12:08:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4310worker1 # [ 26.331136] nix-grpc-daemon[763]: ts=2026-09-21T12:08:14Z level=info event=scheduler_connected addr=lb:500514311worker1 # [ 26.335097] nix-grpc-daemon[763]: ts=2026-09-21T12:08:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4312worker2 # [ 26.340428] nix-grpc-daemon[787]: ts=2026-09-21T12:08:15Z level=info event=scheduler_connected addr=lb:500514313worker2 # [ 26.344257] nix-grpc-daemon[787]: ts=2026-09-21T12:08:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4314lb # [ 26.508534] rustfs-bucket-start[1011]: mb s3://farm4315lb # [ 26.515905] systemd[1]: Finished rustfs-bucket.service.4316lb # [ 26.521317] systemd[1]: Starting niks3 server...4317worker1 # [ 26.535918] nix-grpc-daemon[763]: ts=2026-09-21T12:08:14Z level=info event=scheduler_connected addr=lb:500514318worker1 # [ 26.539761] nix-grpc-daemon[763]: ts=2026-09-21T12:08:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4319worker2 # [ 26.545472] nix-grpc-daemon[787]: ts=2026-09-21T12:08:15Z level=info event=scheduler_connected addr=lb:500514320worker2 # [ 26.549466] nix-grpc-daemon[787]: ts=2026-09-21T12:08:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4321worker1 # [ 26.740436] nix-grpc-daemon[763]: ts=2026-09-21T12:08:14Z level=info event=scheduler_connected addr=lb:500514322worker1 # [ 26.744235] nix-grpc-daemon[763]: ts=2026-09-21T12:08:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4323worker2 # [ 26.748358] nix-grpc-daemon[787]: ts=2026-09-21T12:08:15Z level=info event=scheduler_connected addr=lb:500514324worker2 # [ 26.751567] nix-grpc-daemon[787]: ts=2026-09-21T12:08:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4325lb # [ 26.823095] postgres[1035]: [1035] ERROR: relation "goose_db_version" does not exist at character 364326lb # [ 26.824311] postgres[1035]: [1035] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4327lb # [ 26.870293] niks3-server[1029]: 2026/09/21 12:08:15 OK 20241026095416_initial_model.sql (24.08ms)4328lb # [ 26.874212] niks3-server[1029]: 2026/09/21 12:08:16 OK 20251210153512_drop_unused_gin_index.sql (3.97ms)4329lb # [ 26.879740] niks3-server[1029]: 2026/09/21 12:08:16 OK 20251218171726_add_pins.sql (4.16ms)4330lb # [ 26.885472] niks3-server[1029]: 2026/09/21 12:08:16 OK 20260628120000_add_object_size_and_stats.sql (5.77ms)4331lb # [ 26.890170] niks3-server[1029]: 2026/09/21 12:08:16 OK 20260905000000_add_claims.sql (3.27ms)4332lb # [ 26.894968] niks3-server[1029]: 2026/09/21 12:08:16 OK 20260920000000_drop_claims.sql (4.72ms)4333lb # [ 26.896119] niks3-server[1029]: 2026/09/21 12:08:16 goose: successfully migrated database to version: 202609200000004334lb # [ 26.902753] niks3-server[1029]: 2026/09/21 12:08:16 OK 1_commit_pending_closure.sql (7.82ms)4335lb # [ 26.905609] niks3-server[1029]: 2026/09/21 12:08:16 OK 2_object_stats_trigger.sql (1.59ms)4336lb # [ 26.906835] niks3-server[1029]: 2026/09/21 12:08:16 goose: up to current file version: 24337lb # [ 26.914731] niks3-server[1029]: 2026/09/21 12:08:16 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key4338worker1 # [ 26.946755] nix-grpc-daemon[763]: ts=2026-09-21T12:08:15Z level=info event=scheduler_connected addr=lb:500514339worker1 # [ 26.950211] nix-grpc-daemon[763]: ts=2026-09-21T12:08:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4340worker2 # [ 26.949772] nix-grpc-daemon[787]: ts=2026-09-21T12:08:15Z level=info event=scheduler_connected addr=lb:500514341worker2 # [ 26.953600] nix-grpc-daemon[787]: ts=2026-09-21T12:08:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4342lb # [ 27.021731] niks3-server[1029]: 2026/09/21 12:08:16 INFO Read proxy enabled — serving cache objects from S34343lb # [ 27.023715] niks3-server[1029]: 2026/09/21 12:08:16 INFO Using socket-activated listener address=0.0.0.0:57514344lb # [ 27.029742] systemd[1]: Started niks3 server.4345lb # [ 27.030475] systemd[1]: Reached target Multi-User System.4346lb # [ 27.031905] niks3-server[1029]: 2026/09/21 12:08:16 INFO systemd watchdog enabled interval=15s4347lb # [ 27.033407] systemd[1]: Startup finished in 862ms (kernel) + 4.029s (initrd) + 22.141s (userspace) = 27.032s.4348lb # [ 27.036212] niks3-server[1029]: 2026/09/21 12:08:16 INFO Starting HTTP server address=0.0.0.0:57514349lb # [ 27.042054] niks3-server[1029]: 2026/09/21 12:08:16 INFO lead: acquired remote=192.168.1.4:426744350worker2 # [ 27.039526] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=scheduler_take_over4351worker1 # [ 27.147195] nix-grpc-daemon[763]: ts=2026-09-21T12:08:15Z level=info event=scheduler_connected addr=lb:500514352worker1 # [ 27.151305] nix-grpc-daemon[763]: ts=2026-09-21T12:08:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4353worker2 # [ 27.156749] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=scheduler_connected addr=lb:500514354worker2 # [ 27.161338] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4355worker1 # [ 27.352480] nix-grpc-daemon[763]: ts=2026-09-21T12:08:15Z level=info event=scheduler_connected addr=lb:500514356worker1 # [ 27.356687] nix-grpc-daemon[763]: ts=2026-09-21T12:08:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4357worker2 # [ 27.357522] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=scheduler_connected addr=lb:500514358worker2 # [ 27.361327] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4359worker1 # [ 27.557594] nix-grpc-daemon[763]: ts=2026-09-21T12:08:15Z level=info event=scheduler_connected addr=lb:500514360worker1 # [ 27.561795] nix-grpc-daemon[763]: ts=2026-09-21T12:08:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4361worker2 # [ 27.562385] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=scheduler_connected addr=lb:500514362worker2 # [ 27.566236] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4363lb: (finished: waiting for unit niks3.service, in 16.56 seconds)4364lb: waiting for TCP port 5751 on localhost4365lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!4366lb: (finished: waiting for TCP port 5751 on localhost, in 0.10 seconds)4367lb: waiting for unit envoy.service4368worker1 # [ 27.762491] nix-grpc-daemon[763]: ts=2026-09-21T12:08:15Z level=info event=scheduler_connected addr=lb:500514369worker1 # [ 27.766616] nix-grpc-daemon[763]: ts=2026-09-21T12:08:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4370worker2 # [ 27.767365] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=scheduler_connected addr=lb:500514371worker2 # [ 27.771172] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4372lb: (finished: waiting for unit envoy.service, in 0.08 seconds)4373lb: waiting for TCP port 50051 on localhost4374lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!4375lb # [ 27.835532] envoy[909]: conn peer=[::1]:45818 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=34376lb: (finished: waiting for TCP port 50051 on localhost, in 0.05 seconds)4377worker1 # [ 27.968707] nix-grpc-daemon[763]: ts=2026-09-21T12:08:16Z level=info event=scheduler_connected addr=lb:500514378worker2 # [ 27.969513] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=scheduler_connected addr=lb:500514379worker2 # [ 27.981456] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:567684380worker2 # [ 27.985962] systemd[1]: Starting scheduler-token.service...4381worker2 # [ 27.988104] nix-grpc-daemon[787]: ts=2026-09-21T12:08:16Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04382worker2 # [ 28.001984] nix-grpc-daemon[787]: ts=2026-09-21T12:08:17Z level=info event=oidc_jwks provider=mock keys=14383worker2 # [ 28.004248] nix-grpc-daemon[787]: ts=2026-09-21T12:08:17Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:567684384worker2 # [ 28.005874] nix-grpc-daemon[787]: ts=2026-09-21T12:08:17Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04385worker2 # [ 28.069362] systemd[1]: scheduler-token.service: Deactivated successfully.4386worker2 # [ 28.070970] systemd[1]: Finished scheduler-token.service.4387worker2 # [ 28.072133] systemd[1]: scheduler-token.service: Consumed 19ms CPU time over 86ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4388lb: waiting until exactly no endpoints of x86_64-linux are unhealthy4389lb: must succeed: curl -sf localhost:9901/clusters4390lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.12 seconds)4391lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.12 seconds)4392subtest: one node schedules and both builders hold a WorkerSession on it4393lb: must succeed: curl -sf localhost:9901/clusters4394lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4395worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4396worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4397lb: must succeed: curl -sf localhost:9901/clusters4398lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4399lb: must succeed: curl -sf localhost:9901/clusters4400lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4401lb: must succeed: curl -sf localhost:9901/clusters4402lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4403lb: must succeed: curl -sf localhost:9901/clusters4404lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4405worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4406worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.05 seconds)4407worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4408worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4409lb: must succeed: curl -sf localhost:9901/clusters4410lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4411lb: must succeed: curl -sf localhost:9901/clusters4412lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4413worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4414worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4415worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4416worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4417worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4418worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4419worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4420worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4421(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.84 seconds)4422subtest: balancer auth: client cert, bearer token, nothing, unknown CN4423worker1 # [ 29.420265] nix-grpc-daemon[763]: ts=2026-09-21T12:08:17Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:371864424worker1 # [ 29.434132] systemd[1]: Started Nix Daemon.4425worker1 # [ 29.656522] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4426worker1 # [ 29.686750] nix-grpc-daemon[763]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 29 ms (attempt 1/5)4427worker1 # [ 29.718734] nix-grpc-daemon[763]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 31 ms (attempt 2/5)4428worker1 # [ 29.752622] nix-grpc-daemon[763]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 81 ms (attempt 3/5)4429worker1 # [ 29.834242] nix-grpc-daemon[763]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 388 ms (attempt 4/5)4430worker1 # [ 30.225073] nix-grpc-daemon[763]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4431worker1 # [ 30.228845] nix-grpc-daemon[763]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4432worker2 # [ 30.266588] nix-grpc-daemon[787]: ts=2026-09-21T12:08:19Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:567804433worker2 # [ 30.280267] systemd[1]: Started Nix Daemon.4434worker2 # [ 30.505111] nix-daemon[870]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4435worker2 # [ 30.529460] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 86 ms (attempt 1/5)4436worker2 # [ 30.618159] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 191 ms (attempt 2/5)4437worker2 # [ 30.814172] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 260 ms (attempt 3/5)4438worker2 # [ 31.080118] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 624 ms (attempt 4/5)4439worker2 # [ 31.714878] nix-grpc-daemon[787]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4440worker2 # [ 31.719806] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4441worker2 # [ 31.740394] nix-grpc-daemon[787]: ts=2026-09-21T12:08:20Z level=debug event=path_info_miss path=iwfnwmr510zyiiapzic7q3y408r3ypyx-nixos-system-client-test error="error: path '/nix/store/iwfnwmr510zyiiapzic7q3y408r3ypyx-nixos-system-client-test' is required, but there is no substituter that can build it"4442worker1 # [ 31.753374] nix-grpc-daemon[763]: ts=2026-09-21T12:08:19Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:371864443worker1 # [ 31.771213] nix-grpc-daemon[763]: ts=2026-09-21T12:08:19Z level=debug event=path_info_miss path=iwfnwmr510zyiiapzic7q3y408r3ypyx-nixos-system-client-test error="error: path '/nix/store/iwfnwmr510zyiiapzic7q3y408r3ypyx-nixos-system-client-test' is required, but there is no substituter that can build it"4444client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4445client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.10 seconds)4446client: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4447client: (finished: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.05 seconds)4448worker2 # [ 32.247906] nix-grpc-daemon[787]: ts=2026-09-21T12:08:21Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:567804449worker2 # [ 32.267681] nix-grpc-daemon[787]: ts=2026-09-21T12:08:21Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:567804450worker2 # [ 32.276705] nix-grpc-daemon[787]: ts=2026-09-21T12:08:21Z level=debug event=path_info_miss path=iwfnwmr510zyiiapzic7q3y408r3ypyx-nixos-system-client-test error="error: path '/nix/store/iwfnwmr510zyiiapzic7q3y408r3ypyx-nixos-system-client-test' is required, but there is no substituter that can build it"4451worker1 # [ 32.297984] nix-grpc-daemon[763]: ts=2026-09-21T12:08:20Z level=info event=oidc_jwks provider=mock keys=14452worker1 # [ 32.302388] nix-grpc-daemon[763]: ts=2026-09-21T12:08:20Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:371864453worker1 # [ 32.312906] nix-grpc-daemon[763]: ts=2026-09-21T12:08:20Z level=debug event=path_info_miss path=iwfnwmr510zyiiapzic7q3y408r3ypyx-nixos-system-client-test error="error: path '/nix/store/iwfnwmr510zyiiapzic7q3y408r3ypyx-nixos-system-client-test' is required, but there is no substituter that can build it"4454client: must succeed: rm -r /var/lib/nix-grpc-store4455client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds)4456worker2 # [ 32.605082] nix-grpc-daemon[787]: ts=2026-09-21T12:08:21Z level=info event=denied method=QueryMissing cn=- role=none4457worker2 # [ 32.887883] nix-grpc-daemon[787]: ts=2026-09-21T12:08:21Z level=info event=denied method=QueryMissing cn=stranger role=none4458(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.89 seconds)4459subtest: DAG build is scheduled across workers and lands in the cache4460client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24461worker2 # [ 33.170501] nix-grpc-daemon[787]: ts=2026-09-21T12:08:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:567804462client # don't know how to build these paths:4463client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4464worker2 # [ 33.184914] nix-grpc-daemon[787]: ts=2026-09-21T12:08:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:567804465worker2 # [ 33.222389] nix-grpc-daemon[787]: ts=2026-09-21T12:08:22Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04466worker2 # [ 33.226819] nix-grpc-daemon[787]: ts=2026-09-21T12:08:22Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14467worker1 # [ 33.281216] nix-grpc-daemon[763]: ts=2026-09-21T12:08:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864468worker1 # [ 33.302199] nix-grpc-daemon[763]: ts=2026-09-21T12:08:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371864469client # copying 1 paths to 'lb:50051'...4470worker1 # [ 33.315970] nix-grpc-daemon[763]: ts=2026-09-21T12:08:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:371864471client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4472worker1 # [ 33.349896] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4473worker1 # [ 33.376875] nix-grpc-daemon[763]: ts=2026-09-21T12:08:21Z level=info event=niks3_push_started pid=8614474worker1 # [ 33.646137] nix-grpc-daemon[861]: time=2026-09-21T12:08:21.864Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""4475worker1 # [ 33.842109] nix-daemon[848]: accepted connection from pid 874, user nix-grpc-daemon (trusted)4476worker1 # [ 33.856700] nix-daemon[848]: reaped child process 877, status = succeeded4477worker1 # [ 34.023615] nix-daemon[848]: accepted connection from pid 879, user nix-grpc-daemon (trusted)4478worker1 # [ 34.035884] nix-daemon[848]: reaped child process 882, status = succeeded4479lb # [ 34.057291] niks3-server[1029]: 2026/09/21 12:08:23 INFO Received uploads request method=POST path=/api/pending_closures4480worker1 # [ 34.082056] nix-grpc-daemon[861]: time=2026-09-21T12:08:22.304Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4481worker1 # [ 34.084622] nix-grpc-daemon[861]: time=2026-09-21T12:08:22.307Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4482lb # [ 34.135847] niks3-server[1029]: 2026/09/21 12:08:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4483lb # [ 34.147063] niks3-server[1029]: 2026/09/21 12:08:23 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4484lb # [ 34.159642] niks3-server[1029]: 2026/09/21 12:08:23 INFO Signed narinfos id=1 count=14485worker1 # [ 34.156024] nix-grpc-daemon[861]: time=2026-09-21T12:08:22.378Z level=INFO msg="Uploading 1 narinfos"4486lb # [ 34.170882] niks3-server[1029]: 2026/09/21 12:08:23 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4487lb # [ 34.176186] niks3-server[1029]: 2026/09/21 12:08:23 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4488lb # [ 34.184472] niks3-server[1029]: 2026/09/21 12:08:23 INFO Completed upload id=14489worker1 # [ 34.179630] nix-grpc-daemon[861]: time=2026-09-21T12:08:22.401Z level=INFO msg="Upload complete. (518ms)"4490worker1 # [ 34.184716] nix-grpc-daemon[763]: ts=2026-09-21T12:08:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=0 paths=1 nar_bytes_in=5044491lb # [ 34.193909] niks3-server[1029]: 2026/09/21 12:08:23 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4492worker1 # [ 34.192325] nix-grpc-daemon[763]: ts=2026-09-21T12:08:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864493worker1 # [ 34.202622] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4494worker1 # [ 34.212689] nix-daemon[848]: reaped child process 859, status = succeeded4495client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4496worker1 # [ 34.221056] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4497client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4498client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4499client # farm-a> LOG-farm-a-t14500worker1 # [ 35.437416] nix-daemon[848]: reaped child process 886, status = succeeded4501worker1 # [ 35.602853] nix-daemon[848]: accepted connection from pid 903, user nix-grpc-daemon (trusted)4502worker1 # [ 35.615661] nix-daemon[848]: reaped child process 906, status = succeeded4503lb # [ 35.637352] niks3-server[1029]: 2026/09/21 12:08:24 INFO Received uploads request method=POST path=/api/pending_closures4504worker1 # [ 35.637925] nix-grpc-daemon[861]: time=2026-09-21T12:08:23.860Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4505worker1 # [ 35.639530] nix-grpc-daemon[861]: time=2026-09-21T12:08:23.861Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4506lb # [ 35.664302] niks3-server[1029]: 2026/09/21 12:08:24 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4507lb # [ 35.668525] niks3-server[1029]: 2026/09/21 12:08:24 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4508lb # [ 35.670601] niks3-server[1029]: 2026/09/21 12:08:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4509lb # [ 35.672632] niks3-server[1029]: 2026/09/21 12:08:24 INFO Signed narinfos id=2 count=14510worker1 # [ 35.669055] nix-grpc-daemon[861]: time=2026-09-21T12:08:23.890Z level=INFO msg="Uploading 1 narinfos"4511lb # [ 35.676504] niks3-server[1029]: 2026/09/21 12:08:24 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4512lb # [ 35.680825] niks3-server[1029]: 2026/09/21 12:08:24 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4513lb # [ 35.687152] niks3-server[1029]: 2026/09/21 12:08:24 INFO Completed upload id=24514worker1 # [ 35.681926] nix-grpc-daemon[861]: time=2026-09-21T12:08:23.904Z level=INFO msg="Upload complete. (250ms)"4515lb # [ 35.688930] niks3-server[1029]: 2026/09/21 12:08:24 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4516worker1 # [ 35.686149] nix-grpc-daemon[763]: ts=2026-09-21T12:08:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14517worker1 # [ 35.691284] nix-daemon[848]: reaped child process 884, status = succeeded4518worker2 # [ 35.703936] nix-grpc-daemon[787]: ts=2026-09-21T12:08:24Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04519worker2 # [ 35.708085] nix-grpc-daemon[787]: ts=2026-09-21T12:08:24Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14520worker1 # [ 35.757778] nix-grpc-daemon[763]: ts=2026-09-21T12:08:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864521worker1 # [ 35.770698] nix-grpc-daemon[763]: ts=2026-09-21T12:08:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371864522worker1 # [ 35.784574] nix-grpc-daemon[861]: time=2026-09-21T12:08:24.006Z level=INFO msg="All 1 paths already cached"4523client # copying 1 paths to 'lb:50051'...4524worker1 # [ 35.792586] nix-grpc-daemon[763]: ts=2026-09-21T12:08:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:371864525client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4526worker1 # [ 35.811289] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4527worker1 # [ 36.019711] nix-daemon[848]: accepted connection from pid 910, user nix-grpc-daemon (trusted)4528worker1 # [ 36.033081] nix-daemon[848]: reaped child process 913, status = succeeded4529worker1 # [ 36.203108] nix-daemon[848]: accepted connection from pid 915, user nix-grpc-daemon (trusted)4530worker1 # [ 36.215082] nix-daemon[848]: reaped child process 918, status = succeeded4531lb # [ 36.237906] niks3-server[1029]: 2026/09/21 12:08:25 INFO Received uploads request method=POST path=/api/pending_closures4532worker1 # [ 36.244317] nix-grpc-daemon[861]: time=2026-09-21T12:08:24.466Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4533worker1 # [ 36.245823] nix-grpc-daemon[861]: time=2026-09-21T12:08:24.468Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4534lb # [ 36.270576] niks3-server[1029]: 2026/09/21 12:08:25 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4535lb # [ 36.282376] niks3-server[1029]: 2026/09/21 12:08:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4536lb # [ 36.287658] niks3-server[1029]: 2026/09/21 12:08:25 INFO Signed narinfos id=3 count=14537worker1 # [ 36.285492] nix-grpc-daemon[861]: time=2026-09-21T12:08:24.507Z level=INFO msg="Uploading 1 narinfos"4538lb # [ 36.295333] niks3-server[1029]: 2026/09/21 12:08:25 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4539lb # [ 36.306521] niks3-server[1029]: 2026/09/21 12:08:25 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4540lb # [ 36.311379] niks3-server[1029]: 2026/09/21 12:08:25 INFO Completed upload id=34541lb # [ 36.315078] niks3-server[1029]: 2026/09/21 12:08:25 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4542worker1 # [ 36.308808] nix-grpc-daemon[861]: time=2026-09-21T12:08:24.531Z level=INFO msg="Upload complete. (482ms)"4543worker1 # [ 36.312966] nix-grpc-daemon[763]: ts=2026-09-21T12:08:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=0 paths=1 nar_bytes_in=6244544worker1 # [ 36.319521] nix-grpc-daemon[763]: ts=2026-09-21T12:08:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864545worker1 # [ 36.328497] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4546worker1 # [ 36.336742] nix-daemon[848]: reaped child process 908, status = succeeded4547worker1 # [ 36.342274] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4548client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4549client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4550client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4551lb # [ 37.376955] envoy[909]: conn peer=[2001:db8:1::1]:50560 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=407 ms=24134552lb # [ 37.380772] envoy[909]: conn peer=[2001:db8:1::1]:50574 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=744553lb # [ 37.384133] envoy[909]: conn peer=[2001:db8:1::1]:50590 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=144554lb # [ 37.387579] envoy[909]: conn peer=[2001:db8:1::1]:50658 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=84555client # farm-b> LOG-farm-b-t14556worker1 # [ 37.567769] nix-daemon[848]: reaped child process 922, status = succeeded4557worker1 # [ 37.752079] nix-daemon[848]: accepted connection from pid 939, user nix-grpc-daemon (trusted)4558worker1 # [ 37.764054] nix-daemon[848]: reaped child process 942, status = succeeded4559lb # [ 37.788424] niks3-server[1029]: 2026/09/21 12:08:26 INFO Received uploads request method=POST path=/api/pending_closures4560worker1 # [ 37.792648] nix-grpc-daemon[861]: time=2026-09-21T12:08:26.014Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4561worker1 # [ 37.794136] nix-grpc-daemon[861]: time=2026-09-21T12:08:26.016Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4562lb # [ 37.832440] niks3-server[1029]: 2026/09/21 12:08:26 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4563lb # [ 37.838134] niks3-server[1029]: 2026/09/21 12:08:26 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4564lb # [ 37.846901] niks3-server[1029]: 2026/09/21 12:08:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4565lb # [ 37.853642] niks3-server[1029]: 2026/09/21 12:08:26 INFO Signed narinfos id=4 count=14566worker1 # [ 37.849248] nix-grpc-daemon[861]: time=2026-09-21T12:08:26.071Z level=INFO msg="Uploading 1 narinfos"4567lb # [ 37.857356] niks3-server[1029]: 2026/09/21 12:08:26 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4568lb # [ 37.867752] niks3-server[1029]: 2026/09/21 12:08:26 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4569lb # [ 37.873297] niks3-server[1029]: 2026/09/21 12:08:27 INFO Completed upload id=44570worker1 # [ 37.868297] nix-grpc-daemon[861]: time=2026-09-21T12:08:26.090Z level=INFO msg="Upload complete. (317ms)"4571lb # [ 37.875108] niks3-server[1029]: 2026/09/21 12:08:27 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4572worker1 # [ 37.873901] nix-grpc-daemon[763]: ts=2026-09-21T12:08:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14573worker2 # [ 37.876777] nix-grpc-daemon[787]: ts=2026-09-21T12:08:26Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04574worker2 # [ 37.881347] nix-grpc-daemon[787]: ts=2026-09-21T12:08:26Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14575worker1 # [ 37.888590] nix-daemon[848]: reaped child process 920, status = succeeded4576worker1 # [ 37.925491] nix-grpc-daemon[763]: ts=2026-09-21T12:08:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864577worker1 # [ 37.937368] nix-grpc-daemon[763]: ts=2026-09-21T12:08:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371864578worker1 # [ 37.951527] nix-grpc-daemon[861]: time=2026-09-21T12:08:26.173Z level=INFO msg="All 2 paths already cached"4579client # copying 1 paths to 'lb:50051'...4580worker1 # [ 37.959496] nix-grpc-daemon[763]: ts=2026-09-21T12:08:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:371864581client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4582worker1 # [ 37.979172] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4583worker1 # [ 38.178091] nix-daemon[848]: accepted connection from pid 946, user nix-grpc-daemon (trusted)4584worker1 # [ 38.190732] nix-daemon[848]: reaped child process 949, status = succeeded4585worker1 # [ 38.368526] nix-daemon[848]: accepted connection from pid 951, user nix-grpc-daemon (trusted)4586worker1 # [ 38.381638] nix-daemon[848]: reaped child process 954, status = succeeded4587lb # [ 38.403950] niks3-server[1029]: 2026/09/21 12:08:27 INFO Received uploads request method=POST path=/api/pending_closures4588worker1 # [ 38.407981] nix-grpc-daemon[861]: time=2026-09-21T12:08:26.630Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4589worker1 # [ 38.409368] nix-grpc-daemon[861]: time=2026-09-21T12:08:26.630Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4590lb # [ 38.431584] niks3-server[1029]: 2026/09/21 12:08:27 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4591lb # [ 38.444656] niks3-server[1029]: 2026/09/21 12:08:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4592lb # [ 38.450298] niks3-server[1029]: 2026/09/21 12:08:27 INFO Signed narinfos id=5 count=14593worker1 # [ 38.448709] nix-grpc-daemon[861]: time=2026-09-21T12:08:26.670Z level=INFO msg="Uploading 1 narinfos"4594lb # [ 38.459142] niks3-server[1029]: 2026/09/21 12:08:27 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4595lb # [ 38.467447] niks3-server[1029]: 2026/09/21 12:08:27 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4596lb # [ 38.474213] niks3-server[1029]: 2026/09/21 12:08:27 INFO Completed upload id=54597worker1 # [ 38.469158] nix-grpc-daemon[861]: time=2026-09-21T12:08:26.691Z level=INFO msg="Upload complete. (476ms)"4598lb # [ 38.478841] niks3-server[1029]: 2026/09/21 12:08:27 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4599worker1 # [ 38.473123] nix-grpc-daemon[763]: ts=2026-09-21T12:08:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=0 paths=1 nar_bytes_in=7524600worker1 # [ 38.477851] nix-grpc-daemon[763]: ts=2026-09-21T12:08:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864601worker1 # [ 38.488113] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4602worker1 # [ 38.492157] nix-daemon[848]: reaped child process 944, status = succeeded4603worker1 # [ 38.497433] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4604client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4605client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4606client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4607client # farm-top> LOG-farm-top-t14608worker1 # [ 40.123713] nix-daemon[848]: reaped child process 958, status = succeeded4609worker1 # [ 40.325090] nix-daemon[848]: accepted connection from pid 975, user nix-grpc-daemon (trusted)4610worker1 # [ 40.336129] nix-daemon[848]: reaped child process 978, status = succeeded4611lb # [ 40.362339] niks3-server[1029]: 2026/09/21 12:08:29 INFO Received uploads request method=POST path=/api/pending_closures4612worker1 # [ 40.366182] nix-grpc-daemon[861]: time=2026-09-21T12:08:28.588Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4613worker1 # [ 40.369054] nix-grpc-daemon[861]: time=2026-09-21T12:08:28.590Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4614lb # [ 40.403746] niks3-server[1029]: 2026/09/21 12:08:29 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4615lb # [ 40.408742] niks3-server[1029]: 2026/09/21 12:08:29 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4616lb # [ 40.420899] niks3-server[1029]: 2026/09/21 12:08:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4617lb # [ 40.424436] niks3-server[1029]: 2026/09/21 12:08:29 INFO Signed narinfos id=6 count=14618worker1 # [ 40.419760] nix-grpc-daemon[861]: time=2026-09-21T12:08:28.641Z level=INFO msg="Uploading 1 narinfos"4619lb # [ 40.430210] niks3-server[1029]: 2026/09/21 12:08:29 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4620lb # [ 40.438779] niks3-server[1029]: 2026/09/21 12:08:29 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4621lb # [ 40.442374] niks3-server[1029]: 2026/09/21 12:08:29 INFO Completed upload id=64622lb # [ 40.444341] niks3-server[1029]: 2026/09/21 12:08:29 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4623worker1 # [ 40.437372] nix-grpc-daemon[861]: time=2026-09-21T12:08:28.659Z level=INFO msg="Upload complete. (331ms)"4624worker1 # [ 40.444392] nix-grpc-daemon[763]: ts=2026-09-21T12:08:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14625worker1 # [ 40.456821] nix-daemon[848]: reaped child process 956, status = succeeded4626client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 7.57 seconds)4627client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4628client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.23 seconds)4629client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4630client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.02 seconds)4631client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4632client # copying 3 paths...4633client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4634client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4635client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4636client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.29 seconds)4637(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 8.20 seconds)4638subtest: repeat is answered Cached by the scheduler without building4639worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4640worker1 # finding garbage collector roots...4641worker1 # deleting specified paths...4642worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4643worker1 # deleting unused links...4644worker1 # note: hard linking is currently saving 0.0 KiB4645worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.29 seconds)4646worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4647worker2 # finding garbage collector roots...4648worker2 # deleting specified paths...4649worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4650worker2 # deleting unused links...4651worker2 # note: hard linking is currently saving 0.0 KiB4652worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.30 seconds)4653client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24654worker2 # [ 41.965258] nix-grpc-daemon[787]: ts=2026-09-21T12:08:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:567804655client # don't know how to build these paths:4656client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4657worker1 # [ 41.978818] nix-grpc-daemon[763]: ts=2026-09-21T12:08:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371864658worker1 # [ 41.987550] nix-grpc-daemon[861]: time=2026-09-21T12:08:30.209Z level=INFO msg="All 2 paths already cached"4659worker2 # [ 41.997166] nix-grpc-daemon[787]: ts=2026-09-21T12:08:30Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14660client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.31 seconds)4661client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4662client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.20 seconds)4663lb: must succeed: curl -sf localhost:9901/clusters4664lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)4665lb: must succeed: curl -sf localhost:9901/clusters4666lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4667worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4668worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4669(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.40 seconds)4670subtest: inputs already in the cache are not sent to the scheduler4671lb: must succeed: curl -sf localhost:9901/clusters4672lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4673lb: must succeed: curl -sf localhost:9901/clusters4674lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4675worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4676worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4677client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24678client # don't know how to build these paths:4679client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4680worker1 # [ 42.928788] nix-grpc-daemon[763]: ts=2026-09-21T12:08:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:371864681worker1 # [ 42.933576] nix-grpc-daemon[763]: ts=2026-09-21T12:08:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371864682worker1 # [ 42.943677] nix-grpc-daemon[861]: time=2026-09-21T12:08:31.164Z level=INFO msg="All 2 paths already cached"4683worker2 # [ 42.949749] nix-grpc-daemon[787]: ts=2026-09-21T12:08:31Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04684worker2 # [ 42.954101] nix-grpc-daemon[787]: ts=2026-09-21T12:08:31Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14685worker1 # [ 42.998980] nix-grpc-daemon[763]: ts=2026-09-21T12:08:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864686worker1 # [ 43.006098] nix-grpc-daemon[763]: ts=2026-09-21T12:08:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371864687worker1 # [ 43.015968] nix-grpc-daemon[861]: time=2026-09-21T12:08:31.238Z level=INFO msg="All 2 paths already cached"4688client # copying 1 paths to 'lb:50051'...4689worker1 # [ 43.021471] nix-grpc-daemon[763]: ts=2026-09-21T12:08:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:371864690client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4691worker1 # [ 43.033749] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4692worker1 # [ 43.201519] nix-daemon[848]: accepted connection from pid 995, user nix-grpc-daemon (trusted)4693worker1 # [ 43.215129] nix-daemon[848]: reaped child process 999, status = succeeded4694worker1 # [ 43.386258] nix-daemon[848]: accepted connection from pid 1001, user nix-grpc-daemon (trusted)4695worker1 # [ 43.398919] nix-daemon[848]: reaped child process 1004, status = succeeded4696lb # [ 43.421879] niks3-server[1029]: 2026/09/21 12:08:32 INFO Received uploads request method=POST path=/api/pending_closures4697worker1 # [ 43.425660] nix-grpc-daemon[861]: time=2026-09-21T12:08:31.647Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4698worker1 # [ 43.427057] nix-grpc-daemon[861]: time=2026-09-21T12:08:31.647Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4699lb # [ 43.448644] niks3-server[1029]: 2026/09/21 12:08:32 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4700lb # [ 43.459394] niks3-server[1029]: 2026/09/21 12:08:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4701worker1 # [ 43.459701] nix-grpc-daemon[861]: time=2026-09-21T12:08:31.681Z level=INFO msg="Uploading 1 narinfos"4702lb # [ 43.464819] niks3-server[1029]: 2026/09/21 12:08:32 INFO Signed narinfos id=7 count=14703lb # [ 43.470796] niks3-server[1029]: 2026/09/21 12:08:32 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4704lb # [ 43.481890] niks3-server[1029]: 2026/09/21 12:08:32 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4705worker1 # [ 43.483450] nix-grpc-daemon[861]: time=2026-09-21T12:08:31.705Z level=INFO msg="Upload complete. (443ms)"4706lb # [ 43.488650] niks3-server[1029]: 2026/09/21 12:08:32 INFO Completed upload id=74707worker1 # [ 43.485452] nix-grpc-daemon[763]: ts=2026-09-21T12:08:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=0 paths=1 nar_bytes_in=7524708lb # [ 43.491691] niks3-server[1029]: 2026/09/21 12:08:32 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4709worker1 # [ 43.490627] nix-grpc-daemon[763]: ts=2026-09-21T12:08:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864710worker1 # [ 43.501415] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4711worker1 # [ 43.502968] nix-daemon[848]: reaped child process 993, status = succeeded4712worker1 # [ 43.508508] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4713client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4714client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4715client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4716client # farm-top> LOG-farm-top-t1b4717worker1 # [ 45.135866] nix-daemon[848]: reaped child process 1008, status = succeeded4718worker1 # [ 45.319589] nix-daemon[848]: accepted connection from pid 1024, user nix-grpc-daemon (trusted)4719worker1 # [ 45.331986] nix-daemon[848]: reaped child process 1027, status = succeeded4720lb # [ 45.354815] niks3-server[1029]: 2026/09/21 12:08:34 INFO Received uploads request method=POST path=/api/pending_closures4721worker1 # [ 45.357993] nix-grpc-daemon[861]: time=2026-09-21T12:08:33.580Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4722worker1 # [ 45.359515] nix-grpc-daemon[861]: time=2026-09-21T12:08:33.581Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4723lb # [ 45.394137] niks3-server[1029]: 2026/09/21 12:08:34 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4724lb # [ 45.399641] niks3-server[1029]: 2026/09/21 12:08:34 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4725lb # [ 45.411848] niks3-server[1029]: 2026/09/21 12:08:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4726lb # [ 45.417438] niks3-server[1029]: 2026/09/21 12:08:34 INFO Signed narinfos id=8 count=14727worker1 # [ 45.414770] nix-grpc-daemon[861]: time=2026-09-21T12:08:33.636Z level=INFO msg="Uploading 1 narinfos"4728lb # [ 45.423208] niks3-server[1029]: 2026/09/21 12:08:34 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4729lb # [ 45.432101] niks3-server[1029]: 2026/09/21 12:08:34 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4730lb # [ 45.436574] niks3-server[1029]: 2026/09/21 12:08:34 INFO Completed upload id=84731worker1 # [ 45.432678] nix-grpc-daemon[861]: time=2026-09-21T12:08:33.655Z level=INFO msg="Upload complete. (315ms)"4732lb # [ 45.439287] niks3-server[1029]: 2026/09/21 12:08:34 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4733worker1 # [ 45.437857] nix-grpc-daemon[763]: ts=2026-09-21T12:08:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14734worker1 # [ 45.451402] nix-daemon[848]: reaped child process 1006, status = succeeded4735client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.77 seconds)4736client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4737client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.20 seconds)4738lb: must succeed: curl -sf localhost:9901/clusters4739lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4740lb: must succeed: curl -sf localhost:9901/clusters4741lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4742worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4743worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4744(finished: subtest: inputs already in the cache are not sent to the scheduler, in 3.33 seconds)4745subtest: two clients wanting the same drv build it once4746client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4747client # [ 45.926745] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4748client # Running as unit: dedup1.service; invocation ID: 247b89c193944c399e85b00c32f28b824749client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.07 seconds)4750client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4751worker2 # [ 46.126750] nix-grpc-daemon[787]: ts=2026-09-21T12:08:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:567804752client # [ 46.147619] nix[887]: don't know how to build these paths:4753client # [ 46.148489] nix[887]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4754worker2 # [ 46.139785] nix-grpc-daemon[787]: ts=2026-09-21T12:08:35Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04755worker2 # [ 46.146215] nix-grpc-daemon[787]: ts=2026-09-21T12:08:35Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=14756worker2 # [ 46.150935] nix-grpc-daemon[787]: ts=2026-09-21T12:08:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:567804757worker2 # [ 46.163412] nix-grpc-daemon[787]: ts=2026-09-21T12:08:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:567804758client # [ 46.188121] nix[887]: copying 1 paths to 'lb:50051'...4759client # [ 46.188892] nix[887]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4760worker2 # [ 46.173980] nix-grpc-daemon[787]: ts=2026-09-21T12:08:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:567804761worker2 # [ 46.185374] nix-daemon[870]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4762worker2 # [ 46.207391] nix-grpc-daemon[787]: ts=2026-09-21T12:08:35Z level=info event=niks3_push_started pid=9254763worker2 # [ 46.486907] nix-grpc-daemon[925]: time=2026-09-21T12:08:35.482Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""4764worker2 # [ 46.698832] nix-daemon[870]: accepted connection from pid 938, user nix-grpc-daemon (trusted)4765worker2 # [ 46.712477] nix-daemon[870]: reaped child process 941, status = succeeded4766worker2 # [ 46.876243] nix-daemon[870]: accepted connection from pid 943, user nix-grpc-daemon (trusted)4767worker2 # [ 46.889248] nix-daemon[870]: reaped child process 946, status = succeeded4768lb # [ 46.916418] niks3-server[1029]: 2026/09/21 12:08:36 INFO Received uploads request method=POST path=/api/pending_closures4769worker2 # [ 46.915876] nix-grpc-daemon[925]: time=2026-09-21T12:08:35.915Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4770worker2 # [ 46.918632] nix-grpc-daemon[925]: time=2026-09-21T12:08:35.918Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4771lb # [ 46.960700] niks3-server[1029]: 2026/09/21 12:08:36 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4772lb # [ 46.969826] niks3-server[1029]: 2026/09/21 12:08:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4773worker2 # [ 46.967475] nix-grpc-daemon[925]: time=2026-09-21T12:08:35.966Z level=INFO msg="Uploading 1 narinfos"4774lb # [ 46.977545] niks3-server[1029]: 2026/09/21 12:08:36 INFO Signed narinfos id=9 count=14775lb # [ 46.983279] niks3-server[1029]: 2026/09/21 12:08:36 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4776lb # [ 46.991940] niks3-server[1029]: 2026/09/21 12:08:36 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4777lb # [ 46.999210] niks3-server[1029]: 2026/09/21 12:08:36 INFO Completed upload id=94778worker2 # [ 46.989817] nix-grpc-daemon[925]: time=2026-09-21T12:08:35.989Z level=INFO msg="Upload complete. (477ms)"4779lb # [ 47.000975] niks3-server[1029]: 2026/09/21 12:08:36 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4780worker2 # [ 46.991959] nix-grpc-daemon[787]: ts=2026-09-21T12:08:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56780 duration_s=0 paths=1 nar_bytes_in=4324781worker2 # [ 46.997902] nix-daemon[870]: reaped child process 923, status = succeeded4782client # [ 47.004864] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4783worker2 # [ 47.001992] nix-grpc-daemon[787]: ts=2026-09-21T12:08:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:567804784client # Running as unit: dedup2.service; invocation ID: 722534ed0c684ac993dd1e622a7327854785worker2 # [ 47.005846] nix-daemon[870]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4786client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.08 seconds)4787worker2 # [ 47.012872] nix-daemon[870]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4788client # [ 47.027554] nix[887]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4789client # [ 47.285482] nix[903]: don't know how to build these paths:4790worker1 # [ 47.274230] nix-grpc-daemon[763]: ts=2026-09-21T12:08:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:371864791client # [ 47.288055] nix[903]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4792worker2 # [ 47.284921] nix-grpc-daemon[787]: ts=2026-09-21T12:08:36Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04793client # [ 47.306051] nix[903]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4794worker2 # [ 47.290654] nix-grpc-daemon[787]: ts=2026-09-21T12:08:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:567804795lb # [ 47.378381] envoy[909]: conn peer=[2001:db8:1::1]:50662 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9567 tx=4088 ms=73134796lb # [ 47.382119] envoy[909]: conn peer=[2001:db8:1::1]:50670 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=434797lb # [ 47.385972] envoy[909]: conn peer=[2001:db8:1::1]:37464 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4182 tx=1779 ms=25194798client: must succeed: sleep 24799client: (finished: must succeed: sleep 2, in 2.05 seconds)4800client: waiting for success: ! systemctl is-active dedup1 dedup24801worker2 # [ 50.639970] nix-daemon[870]: reaped child process 950, status = succeeded4802worker2 # [ 50.814327] nix-daemon[870]: accepted connection from pid 982, user nix-grpc-daemon (trusted)4803worker2 # [ 50.826566] nix-daemon[870]: reaped child process 985, status = succeeded4804lb # [ 50.853355] niks3-server[1029]: 2026/09/21 12:08:39 INFO Received uploads request method=POST path=/api/pending_closures4805worker2 # [ 50.852031] nix-grpc-daemon[925]: time=2026-09-21T12:08:39.851Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4806worker2 # [ 50.853742] nix-grpc-daemon[925]: time=2026-09-21T12:08:39.853Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4807lb # [ 50.888212] niks3-server[1029]: 2026/09/21 12:08:40 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4808lb # [ 50.902688] niks3-server[1029]: 2026/09/21 12:08:40 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4809lb # [ 50.915092] niks3-server[1029]: 2026/09/21 12:08:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4810lb # [ 50.919361] niks3-server[1029]: 2026/09/21 12:08:40 INFO Signed narinfos id=10 count=14811worker2 # [ 50.910957] nix-grpc-daemon[925]: time=2026-09-21T12:08:39.909Z level=INFO msg="Uploading 1 narinfos"4812lb # [ 50.924582] niks3-server[1029]: 2026/09/21 12:08:40 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4813lb # [ 50.933798] niks3-server[1029]: 2026/09/21 12:08:40 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4814lb # [ 50.937178] niks3-server[1029]: 2026/09/21 12:08:40 INFO Completed upload id=104815worker2 # [ 50.929127] nix-grpc-daemon[925]: time=2026-09-21T12:08:39.928Z level=INFO msg="Upload complete. (303ms)"4816lb # [ 50.939650] niks3-server[1029]: 2026/09/21 12:08:40 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo4817worker2 # [ 50.935564] nix-grpc-daemon[787]: ts=2026-09-21T12:08:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56780 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14818worker2 # [ 50.942290] nix-grpc-daemon[787]: ts=2026-09-21T12:08:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56780 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14819worker2 # [ 50.951788] nix-daemon[870]: reaped child process 948, status = succeeded4820client # [ 50.992224] systemd[1]: dedup2.service: Deactivated successfully.4821client # [ 50.996757] systemd[1]: dedup2.service: Consumed 143ms CPU time over 3.990s wall clock time, 35.2M memory peak, 6K incoming IP traffic, 5.7K outgoing IP traffic.4822client # [ 51.033876] systemd[1]: dedup1.service: Deactivated successfully.4823client # [ 51.035342] systemd[1]: dedup1.service: Consumed 118ms CPU time over 5.108s wall clock time, 41.4M memory peak, 7.3K incoming IP traffic, 8.5K outgoing IP traffic.4824client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.12 seconds)4825client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4826client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)4827worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4828worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)4829worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4830worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)4831worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4832worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4833worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4834worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4835(finished: subtest: two clients wanting the same drv build it once, in 6.03 seconds)4836subtest: build hook: nix-daemon with builders = grpc://lb4837client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log4838client # [ 52.103742] systemd[1]: Started Nix Daemon.4839client # [ 52.276052] nix-daemon[945]: accepted connection from pid 942, user root (trusted)4840worker2 # [ 52.459824] nix-grpc-daemon[787]: ts=2026-09-21T12:08:41Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:567804841worker1 # [ 52.468657] nix-grpc-daemon[763]: ts=2026-09-21T12:08:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371864842worker2 # [ 52.480611] nix-grpc-daemon[787]: ts=2026-09-21T12:08:41Z level=debug event=want client=6 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=04843worker2 # [ 52.484806] nix-grpc-daemon[787]: ts=2026-09-21T12:08:41Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=14844worker1 # [ 52.529533] nix-grpc-daemon[763]: ts=2026-09-21T12:08:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864845worker1 # [ 52.544959] nix-grpc-daemon[763]: ts=2026-09-21T12:08:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371864846worker1 # [ 52.561342] nix-grpc-daemon[763]: ts=2026-09-21T12:08:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:371864847worker1 # [ 52.573100] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4848worker1 # [ 52.769300] nix-daemon[848]: accepted connection from pid 1052, user nix-grpc-daemon (trusted)4849worker1 # [ 52.783128] nix-daemon[848]: reaped child process 1055, status = succeeded4850worker1 # [ 52.951119] nix-daemon[848]: accepted connection from pid 1057, user nix-grpc-daemon (trusted)4851worker1 # [ 52.962792] nix-daemon[848]: reaped child process 1060, status = succeeded4852lb # [ 52.985384] niks3-server[1029]: 2026/09/21 12:08:42 INFO Received uploads request method=POST path=/api/pending_closures4853worker1 # [ 52.988327] nix-grpc-daemon[861]: time=2026-09-21T12:08:41.210Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4854worker1 # [ 52.989719] nix-grpc-daemon[861]: time=2026-09-21T12:08:41.210Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"4855lb # [ 53.008988] niks3-server[1029]: 2026/09/21 12:08:42 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst4856lb # [ 53.020531] niks3-server[1029]: 2026/09/21 12:08:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4857lb # [ 53.027717] niks3-server[1029]: 2026/09/21 12:08:42 INFO Signed narinfos id=11 count=14858worker1 # [ 53.026219] nix-grpc-daemon[861]: time=2026-09-21T12:08:41.248Z level=INFO msg="Uploading 1 narinfos"4859lb # [ 53.035651] niks3-server[1029]: 2026/09/21 12:08:42 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls4860lb # [ 53.046542] niks3-server[1029]: 2026/09/21 12:08:42 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4861lb # [ 53.052045] niks3-server[1029]: 2026/09/21 12:08:42 INFO Completed upload id=114862worker1 # [ 53.049348] nix-grpc-daemon[861]: time=2026-09-21T12:08:41.271Z level=INFO msg="Upload complete. (464ms)"4863lb # [ 53.056049] niks3-server[1029]: 2026/09/21 12:08:42 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo4864worker1 # [ 53.051685] nix-grpc-daemon[763]: ts=2026-09-21T12:08:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=0 paths=1 nar_bytes_in=5124865worker1 # [ 53.057322] nix-grpc-daemon[763]: ts=2026-09-21T12:08:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864866worker1 # [ 53.066237] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4867worker1 # [ 53.070799] nix-daemon[848]: reaped child process 1050, status = succeeded4868worker1 # [ 53.075743] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4869worker1 # [ 54.089689] nix-daemon[848]: reaped child process 1064, status = succeeded4870worker1 # [ 54.255789] nix-daemon[848]: accepted connection from pid 1080, user nix-grpc-daemon (trusted)4871worker1 # [ 54.266844] nix-daemon[848]: reaped child process 1083, status = succeeded4872lb # [ 54.287900] niks3-server[1029]: 2026/09/21 12:08:43 INFO Received uploads request method=POST path=/api/pending_closures4873worker1 # [ 54.292155] nix-grpc-daemon[861]: time=2026-09-21T12:08:42.514Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4874worker1 # [ 54.294790] nix-grpc-daemon[861]: time=2026-09-21T12:08:42.516Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"4875lb # [ 54.330725] niks3-server[1029]: 2026/09/21 12:08:43 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4876lb # [ 54.337538] niks3-server[1029]: 2026/09/21 12:08:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4877worker1 # [ 54.339984] nix-grpc-daemon[861]: time=2026-09-21T12:08:42.562Z level=INFO msg="Uploading 1 narinfos"4878lb # [ 54.344963] niks3-server[1029]: 2026/09/21 12:08:43 INFO Signed narinfos id=12 count=14879lb # [ 54.350212] niks3-server[1029]: 2026/09/21 12:08:43 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv4880lb # [ 54.354697] niks3-server[1029]: 2026/09/21 12:08:43 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls4881lb # [ 54.366203] niks3-server[1029]: 2026/09/21 12:08:43 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4882lb # [ 54.372514] niks3-server[1029]: 2026/09/21 12:08:43 INFO Completed upload id=124883worker1 # [ 54.367386] nix-grpc-daemon[861]: time=2026-09-21T12:08:42.589Z level=INFO msg="Upload complete. (285ms)"4884lb # [ 54.375292] niks3-server[1029]: 2026/09/21 12:08:43 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo4885worker1 # [ 54.373169] nix-grpc-daemon[763]: ts=2026-09-21T12:08:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=6 outputs=14886worker2 # [ 54.375411] nix-grpc-daemon[787]: ts=2026-09-21T12:08:43Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:567804887worker1 # [ 54.386405] nix-daemon[848]: reaped child process 1062, status = succeeded4888worker2 # [ 54.393228] nix-grpc-daemon[787]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...4889worker1 # [ 54.429264] nix-grpc-daemon[763]: ts=2026-09-21T12:08:42Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:371864890worker1 # [ 54.434630] nix-grpc-daemon[763]: ts=2026-09-21T12:08:42Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=0 paths=1 nar_bytes_out=1284891worker2 # [ 54.666509] nix-grpc-daemon[787]: ts=2026-09-21T12:08:43Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:567804892worker1 # [ 54.674144] nix-grpc-daemon[763]: ts=2026-09-21T12:08:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371864893worker1 # [ 54.683075] nix-grpc-daemon[861]: time=2026-09-21T12:08:42.905Z level=INFO msg="All 1 paths already cached"4894worker2 # [ 54.693087] nix-grpc-daemon[787]: ts=2026-09-21T12:08:43Z level=debug event=want client=7 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=04895worker1 # [ 54.699557] nix-grpc-daemon[763]: ts=2026-09-21T12:08:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864896worker2 # [ 54.697346] nix-grpc-daemon[787]: ts=2026-09-21T12:08:43Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=14897worker1 # [ 54.710931] nix-grpc-daemon[763]: ts=2026-09-21T12:08:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371864898worker1 # [ 54.722559] nix-grpc-daemon[861]: time=2026-09-21T12:08:42.944Z level=INFO msg="All 1 paths already cached"4899worker1 # [ 54.729079] nix-grpc-daemon[763]: ts=2026-09-21T12:08:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:371864900worker1 # [ 54.739251] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4901worker1 # [ 54.939548] nix-daemon[848]: accepted connection from pid 1087, user nix-grpc-daemon (trusted)4902worker1 # [ 54.953134] nix-daemon[848]: reaped child process 1090, status = succeeded4903worker1 # [ 55.124363] nix-daemon[848]: accepted connection from pid 1092, user nix-grpc-daemon (trusted)4904worker1 # [ 55.136174] nix-daemon[848]: reaped child process 1095, status = succeeded4905lb # [ 55.157684] niks3-server[1029]: 2026/09/21 12:08:44 INFO Received uploads request method=POST path=/api/pending_closures4906worker1 # [ 55.159892] nix-grpc-daemon[861]: time=2026-09-21T12:08:43.382Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4907worker1 # [ 55.161683] nix-grpc-daemon[861]: time=2026-09-21T12:08:43.383Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"4908lb # [ 55.187752] niks3-server[1029]: 2026/09/21 12:08:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4909worker1 # [ 55.189352] nix-grpc-daemon[861]: time=2026-09-21T12:08:43.411Z level=INFO msg="Uploading 1 narinfos"4910lb # [ 55.195119] niks3-server[1029]: 2026/09/21 12:08:44 INFO Signed narinfos id=13 count=14911lb # [ 55.199394] niks3-server[1029]: 2026/09/21 12:08:44 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst4912lb # [ 55.204889] niks3-server[1029]: 2026/09/21 12:08:44 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls4913lb # [ 55.216559] niks3-server[1029]: 2026/09/21 12:08:44 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4914lb # [ 55.224940] niks3-server[1029]: 2026/09/21 12:08:44 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo4915lb # [ 55.229526] niks3-server[1029]: 2026/09/21 12:08:44 INFO Completed upload id=134916worker1 # [ 55.224299] nix-grpc-daemon[861]: time=2026-09-21T12:08:43.446Z level=INFO msg="Upload complete. (472ms)"4917worker1 # [ 55.228308] nix-grpc-daemon[763]: ts=2026-09-21T12:08:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=0 paths=1 nar_bytes_in=6244918worker1 # [ 55.234280] nix-grpc-daemon[763]: ts=2026-09-21T12:08:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864919worker1 # [ 55.244111] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4920worker1 # [ 55.252118] nix-daemon[848]: reaped child process 1085, status = succeeded4921worker1 # [ 55.256073] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4922worker1 # [ 56.068102] nix-daemon[848]: reaped child process 1099, status = succeeded4923worker1 # [ 56.237961] nix-daemon[848]: accepted connection from pid 1115, user nix-grpc-daemon (trusted)4924worker1 # [ 56.249980] nix-daemon[848]: reaped child process 1118, status = succeeded4925lb # [ 56.271395] niks3-server[1029]: 2026/09/21 12:08:45 INFO Received uploads request method=POST path=/api/pending_closures4926worker1 # [ 56.274926] nix-grpc-daemon[861]: time=2026-09-21T12:08:44.497Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4927worker1 # [ 56.277391] nix-grpc-daemon[861]: time=2026-09-21T12:08:44.498Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"4928lb # [ 56.310258] niks3-server[1029]: 2026/09/21 12:08:45 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst4929lb # [ 56.316091] niks3-server[1029]: 2026/09/21 12:08:45 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv4930lb # [ 56.325812] niks3-server[1029]: 2026/09/21 12:08:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4931lb # [ 56.331653] niks3-server[1029]: 2026/09/21 12:08:45 INFO Signed narinfos id=14 count=14932worker1 # [ 56.329452] nix-grpc-daemon[861]: time=2026-09-21T12:08:44.551Z level=INFO msg="Uploading 1 narinfos"4933lb # [ 56.336097] niks3-server[1029]: 2026/09/21 12:08:45 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls4934lb # [ 56.347394] niks3-server[1029]: 2026/09/21 12:08:45 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4935lb # [ 56.352762] niks3-server[1029]: 2026/09/21 12:08:45 INFO Completed upload id=144936lb # [ 56.355732] niks3-server[1029]: 2026/09/21 12:08:45 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo4937worker1 # [ 56.349936] nix-grpc-daemon[861]: time=2026-09-21T12:08:44.572Z level=INFO msg="Upload complete. (288ms)"4938worker1 # [ 56.354739] nix-grpc-daemon[763]: ts=2026-09-21T12:08:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=7 outputs=14939worker1 # [ 56.368439] nix-daemon[848]: reaped child process 1097, status = succeeded4940worker2 # [ 56.396763] nix-grpc-daemon[787]: ts=2026-09-21T12:08:45Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:567804941worker2 # [ 56.407719] nix-grpc-daemon[787]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...4942worker1 # [ 56.442213] nix-grpc-daemon[763]: ts=2026-09-21T12:08:44Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:371864943worker1 # [ 56.446698] nix-grpc-daemon[763]: ts=2026-09-21T12:08:44Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=0 paths=1 nar_bytes_out=1764944worker1 # [ 56.680705] nix-grpc-daemon[763]: ts=2026-09-21T12:08:44Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:371864945worker2 # [ 56.680269] nix-grpc-daemon[787]: ts=2026-09-21T12:08:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:567804946worker2 # [ 56.689843] nix-grpc-daemon[925]: time=2026-09-21T12:08:45.688Z level=INFO msg="All 2 paths already cached"4947worker2 # [ 56.703703] nix-grpc-daemon[787]: ts=2026-09-21T12:08:45Z level=debug event=want client=8 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=04948worker1 # [ 56.710417] nix-grpc-daemon[763]: ts=2026-09-21T12:08:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864949worker2 # [ 56.708142] nix-grpc-daemon[787]: ts=2026-09-21T12:08:45Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=14950worker1 # [ 56.721870] nix-grpc-daemon[763]: ts=2026-09-21T12:08:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371864951worker1 # [ 56.734309] nix-grpc-daemon[861]: time=2026-09-21T12:08:44.956Z level=INFO msg="All 2 paths already cached"4952worker1 # [ 56.741160] nix-grpc-daemon[763]: ts=2026-09-21T12:08:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:371864953worker1 # [ 56.751172] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4954worker1 # [ 56.944454] nix-daemon[848]: accepted connection from pid 1122, user nix-grpc-daemon (trusted)4955worker1 # [ 56.956681] nix-daemon[848]: reaped child process 1125, status = succeeded4956worker1 # [ 57.119738] nix-daemon[848]: accepted connection from pid 1127, user nix-grpc-daemon (trusted)4957worker1 # [ 57.132135] nix-daemon[848]: reaped child process 1130, status = succeeded4958lb # [ 57.152470] niks3-server[1029]: 2026/09/21 12:08:46 INFO Received uploads request method=POST path=/api/pending_closures4959worker1 # [ 57.170148] nix-grpc-daemon[861]: time=2026-09-21T12:08:45.392Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4960worker1 # [ 57.171517] nix-grpc-daemon[861]: time=2026-09-21T12:08:45.392Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"4961lb # [ 57.183132] niks3-server[1029]: 2026/09/21 12:08:46 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst4962lb # [ 57.186144] niks3-server[1029]: 2026/09/21 12:08:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4963worker1 # [ 57.183960] nix-grpc-daemon[861]: time=2026-09-21T12:08:45.406Z level=INFO msg="Uploading 1 narinfos"4964lb # [ 57.189086] niks3-server[1029]: 2026/09/21 12:08:46 INFO Signed narinfos id=15 count=14965lb # [ 57.192597] niks3-server[1029]: 2026/09/21 12:08:46 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls4966lb # [ 57.195738] niks3-server[1029]: 2026/09/21 12:08:46 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4967lb # [ 57.198830] niks3-server[1029]: 2026/09/21 12:08:46 INFO Completed upload id=154968worker1 # [ 57.193597] nix-grpc-daemon[861]: time=2026-09-21T12:08:45.415Z level=INFO msg="Upload complete. (429ms)"4969worker1 # [ 57.195506] nix-grpc-daemon[763]: ts=2026-09-21T12:08:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=0 paths=1 nar_bytes_in=7524970lb # [ 57.201206] niks3-server[1029]: 2026/09/21 12:08:46 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo4971worker1 # [ 57.198409] nix-grpc-daemon[763]: ts=2026-09-21T12:08:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371864972worker1 # [ 57.205554] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4973worker1 # [ 57.207797] nix-daemon[848]: reaped child process 1120, status = succeeded4974worker1 # [ 57.213467] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)4975lb # [ 57.378787] envoy[909]: conn peer=[2001:db8:1::1]:37476 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1427 tx=1085 ms=36874976lb # [ 57.382448] envoy[909]: conn peer=[2001:db8:1::1]:37474 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2998 tx=1447 ms=48754977lb # [ 57.385940] envoy[909]: conn peer=[2001:db8:1::1]:37478 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3423 tx=1853 ms=19934978lb # [ 57.389887] envoy[909]: conn peer=[2001:db8:1::1]:41470 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=284979lb # [ 57.393335] envoy[909]: conn peer=[2001:db8:1::1]:41482 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=174980lb # [ 57.396685] envoy[909]: conn peer=[2001:db8:1::1]:41476 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3893 tx=2070 ms=17894981worker1 # [ 58.435717] nix-daemon[848]: reaped child process 1134, status = succeeded4982worker1 # [ 58.600866] nix-daemon[848]: accepted connection from pid 1150, user nix-grpc-daemon (trusted)4983worker1 # [ 58.612915] nix-daemon[848]: reaped child process 1153, status = succeeded4984lb # [ 58.637607] niks3-server[1029]: 2026/09/21 12:08:47 INFO Received uploads request method=POST path=/api/pending_closures4985worker1 # [ 58.641506] nix-grpc-daemon[861]: time=2026-09-21T12:08:46.863Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4986worker1 # [ 58.642953] nix-grpc-daemon[861]: time=2026-09-21T12:08:46.865Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"4987lb # [ 58.673405] niks3-server[1029]: 2026/09/21 12:08:47 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv4988lb # [ 58.683921] niks3-server[1029]: 2026/09/21 12:08:47 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst4989lb # [ 58.692242] niks3-server[1029]: 2026/09/21 12:08:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4990worker1 # [ 58.691075] nix-grpc-daemon[861]: time=2026-09-21T12:08:46.913Z level=INFO msg="Uploading 1 narinfos"4991lb # [ 58.696430] niks3-server[1029]: 2026/09/21 12:08:47 INFO Signed narinfos id=16 count=14992lb # [ 58.701138] niks3-server[1029]: 2026/09/21 12:08:47 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls4993lb # [ 58.707854] niks3-server[1029]: 2026/09/21 12:08:47 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4994lb # [ 58.710875] niks3-server[1029]: 2026/09/21 12:08:47 INFO Completed upload id=164995worker1 # [ 58.705866] nix-grpc-daemon[861]: time=2026-09-21T12:08:46.928Z level=INFO msg="Upload complete. (285ms)"4996lb # [ 58.712910] niks3-server[1029]: 2026/09/21 12:08:47 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo4997worker1 # [ 58.711466] nix-grpc-daemon[763]: ts=2026-09-21T12:08:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=8 outputs=14998worker1 # [ 58.716765] nix-grpc-daemon[763]: ts=2026-09-21T12:08:46Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:371864999worker1 # [ 58.725655] nix-daemon[848]: reaped child process 1132, status = succeeded5000worker2 # [ 58.731782] nix-grpc-daemon[787]: ts=2026-09-21T12:08:47Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:567805001worker2 # [ 58.737396] nix-grpc-daemon[787]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...5002worker2 # [ 58.748860] nix-grpc-daemon[787]: ts=2026-09-21T12:08:47Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56780 duration_s=0 paths=1 nar_bytes_out=2325003client # [ 58.875125] nix-daemon[945]: reaped child process 947, status = succeeded5004client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.99 seconds)5005client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5006client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.03 seconds)5007client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5008client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.04 seconds)5009client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5010client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)5011(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.08 seconds)5012subtest: an input only one worker has still reaches the builder5013client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only5014client # this derivation will be built:5015client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5016client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5017client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.44 seconds)5018client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5019client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.18 seconds)5020worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5021worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.23 seconds)5022worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5023worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)5024client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&25025client # [ 60.060548] nix-daemon[945]: accepted connection from pid 1028, user root (trusted)5026client # this derivation will be built:5027client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5028worker1 # [ 60.237827] nix-grpc-daemon[763]: ts=2026-09-21T12:08:48Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:371865029client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5030worker2 # [ 60.236743] nix-grpc-daemon[787]: ts=2026-09-21T12:08:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:567805031client # copying 1 paths...5032client # copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' to 'grpc://lb:50051'...5033worker1 # [ 60.252518] nix-grpc-daemon[763]: ts=2026-09-21T12:08:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:371865034worker1 # [ 60.263356] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)5035worker1 # [ 60.470177] nix-daemon[848]: accepted connection from pid 1162, user nix-grpc-daemon (trusted)5036worker1 # [ 60.482967] nix-daemon[848]: reaped child process 1165, status = succeeded5037lb # [ 60.506156] niks3-server[1029]: 2026/09/21 12:08:49 INFO Received uploads request method=POST path=/api/pending_closures5038worker1 # [ 60.509527] nix-grpc-daemon[861]: time=2026-09-21T12:08:48.731Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5039worker1 # [ 60.511329] nix-grpc-daemon[861]: time=2026-09-21T12:08:48.733Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5040lb # [ 60.533496] niks3-server[1029]: 2026/09/21 12:08:49 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5041lb # [ 60.544619] niks3-server[1029]: 2026/09/21 12:08:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5042lb # [ 60.551445] niks3-server[1029]: 2026/09/21 12:08:49 INFO Signed narinfos id=17 count=15043worker1 # [ 60.549438] nix-grpc-daemon[861]: time=2026-09-21T12:08:48.771Z level=INFO msg="Uploading 1 narinfos"5044lb # [ 60.555073] niks3-server[1029]: 2026/09/21 12:08:49 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5045lb # [ 60.569280] niks3-server[1029]: 2026/09/21 12:08:49 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5046lb # [ 60.573714] niks3-server[1029]: 2026/09/21 12:08:49 INFO Completed upload id=175047worker1 # [ 60.571041] nix-grpc-daemon[861]: time=2026-09-21T12:08:48.793Z level=INFO msg="Upload complete. (297ms)"5048lb # [ 60.577341] niks3-server[1029]: 2026/09/21 12:08:49 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5049worker1 # [ 60.574322] nix-grpc-daemon[763]: ts=2026-09-21T12:08:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=0 paths=1 nar_bytes_in=1205050worker1 # [ 60.585194] nix-daemon[848]: reaped child process 1160, status = succeeded5051worker2 # [ 60.586307] nix-grpc-daemon[787]: ts=2026-09-21T12:08:49Z level=debug event=want client=9 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05052worker2 # [ 60.591625] nix-grpc-daemon[787]: ts=2026-09-21T12:08:49Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=15053worker2 # [ 60.635772] nix-grpc-daemon[787]: ts=2026-09-21T12:08:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:567805054worker2 # [ 60.648354] nix-grpc-daemon[787]: ts=2026-09-21T12:08:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:567805055client # copying 2 paths to 'lb:50051'...5056client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5057worker2 # [ 60.664633] nix-grpc-daemon[787]: ts=2026-09-21T12:08:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:567805058client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5059worker2 # [ 60.676192] nix-daemon[870]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5060worker2 # [ 60.875121] nix-daemon[870]: accepted connection from pid 1012, user nix-grpc-daemon (trusted)5061worker2 # [ 60.888361] nix-daemon[870]: reaped child process 1015, status = succeeded5062worker2 # [ 61.024369] nix-daemon[870]: accepted connection from pid 1017, user nix-grpc-daemon (trusted)5063worker2 # [ 61.036211] nix-daemon[870]: reaped child process 1020, status = succeeded5064lb # [ 61.056137] niks3-server[1029]: 2026/09/21 12:08:50 INFO Received uploads request method=POST path=/api/pending_closures5065lb # [ 61.066637] niks3-server[1029]: 2026/09/21 12:08:50 INFO Received uploads request method=POST path=/api/pending_closures5066worker2 # [ 61.065452] nix-grpc-daemon[925]: time=2026-09-21T12:08:50.064Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5067worker2 # [ 61.067710] nix-grpc-daemon[925]: time=2026-09-21T12:08:50.066Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5068worker2 # [ 61.069331] nix-grpc-daemon[925]: time=2026-09-21T12:08:50.066Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5069lb # [ 61.102372] niks3-server[1029]: 2026/09/21 12:08:50 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5070lb # [ 61.109081] niks3-server[1029]: 2026/09/21 12:08:50 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5071lb # [ 61.124558] niks3-server[1029]: 2026/09/21 12:08:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5072lb # [ 61.131242] niks3-server[1029]: 2026/09/21 12:08:50 INFO Signed narinfos id=19 count=25073lb # [ 61.134143] niks3-server[1029]: 2026/09/21 12:08:50 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5074lb # [ 61.136983] niks3-server[1029]: 2026/09/21 12:08:50 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5075lb # [ 61.138706] niks3-server[1029]: 2026/09/21 12:08:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5076lb # [ 61.140466] niks3-server[1029]: 2026/09/21 12:08:50 INFO Signed narinfos id=18 count=15077worker2 # [ 61.132253] nix-grpc-daemon[925]: time=2026-09-21T12:08:50.130Z level=INFO msg="Uploading 3 narinfos"5078lb # [ 61.152766] niks3-server[1029]: 2026/09/21 12:08:50 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5079lb # [ 61.155473] niks3-server[1029]: 2026/09/21 12:08:50 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5080lb # [ 61.157428] niks3-server[1029]: 2026/09/21 12:08:50 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5081lb # [ 61.161056] niks3-server[1029]: 2026/09/21 12:08:50 INFO Completed upload id=195082lb # [ 61.162345] niks3-server[1029]: 2026/09/21 12:08:50 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5083lb # [ 61.164583] niks3-server[1029]: 2026/09/21 12:08:50 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5084lb # [ 61.166843] niks3-server[1029]: 2026/09/21 12:08:50 INFO Completed upload id=185085worker2 # [ 61.158736] nix-grpc-daemon[925]: time=2026-09-21T12:08:50.158Z level=INFO msg="Upload complete. (468ms)"5086worker2 # [ 61.164161] nix-grpc-daemon[787]: ts=2026-09-21T12:08:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56780 duration_s=0 paths=2 nar_bytes_in=9765087worker2 # [ 61.169491] nix-grpc-daemon[787]: ts=2026-09-21T12:08:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:567805088worker2 # [ 61.179634] nix-daemon[870]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5089worker2 # [ 61.190284] nix-daemon[870]: reaped child process 1010, status = succeeded5090worker2 # [ 61.200188] nix-grpc-daemon[787]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5091client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5092worker2 # [ 61.212884] nix-daemon[870]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5093client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5094client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5095worker2 # [ 62.240763] nix-daemon[870]: reaped child process 1025, status = succeeded5096worker2 # [ 62.400773] nix-daemon[870]: accepted connection from pid 1041, user nix-grpc-daemon (trusted)5097worker2 # [ 62.411228] nix-daemon[870]: reaped child process 1044, status = succeeded5098lb # [ 62.433158] niks3-server[1029]: 2026/09/21 12:08:51 INFO Received uploads request method=POST path=/api/pending_closures5099worker2 # [ 62.432925] nix-grpc-daemon[925]: time=2026-09-21T12:08:51.432Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5100lb # [ 62.463269] niks3-server[1029]: 2026/09/21 12:08:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5101lb # [ 62.469260] niks3-server[1029]: 2026/09/21 12:08:51 INFO Signed narinfos id=20 count=15102worker2 # [ 62.462881] nix-grpc-daemon[925]: time=2026-09-21T12:08:51.462Z level=INFO msg="Uploading 1 narinfos"5103lb # [ 62.477715] niks3-server[1029]: 2026/09/21 12:08:51 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5104lb # [ 62.486297] niks3-server[1029]: 2026/09/21 12:08:51 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5105lb # [ 62.494700] niks3-server[1029]: 2026/09/21 12:08:51 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5106worker2 # [ 62.491626] nix-grpc-daemon[925]: time=2026-09-21T12:08:51.490Z level=INFO msg="Upload complete. (265ms)"5107lb # [ 62.501100] niks3-server[1029]: 2026/09/21 12:08:51 INFO Completed upload id=205108worker2 # [ 62.495155] nix-grpc-daemon[787]: ts=2026-09-21T12:08:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56780 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=9 outputs=15109lb # [ 62.504458] niks3-server[1029]: 2026/09/21 12:08:51 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5110client # copying 1 paths...5111worker2 # [ 62.501467] nix-grpc-daemon[787]: ts=2026-09-21T12:08:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:567805112worker2 # [ 62.503332] nix-daemon[870]: reaped child process 1022, status = succeeded5113client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5114worker2 # [ 62.517589] nix-grpc-daemon[787]: ts=2026-09-21T12:08:51Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:567805115worker2 # [ 62.520651] nix-grpc-daemon[787]: ts=2026-09-21T12:08:51Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56780 duration_s=0 paths=1 nar_bytes_out=1205116client # [ 62.642989] nix-daemon[945]: reaped child process 1031, status = succeeded5117client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.79 seconds)5118client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25119client # [ 62.812862] nix-daemon[945]: accepted connection from pid 1050, user root (trusted)5120client # this derivation will be built:5121client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5122client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5123worker1 # [ 62.986698] nix-grpc-daemon[763]: ts=2026-09-21T12:08:51Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:371865124worker1 # [ 62.992460] nix-grpc-daemon[763]: ts=2026-09-21T12:08:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371865125worker1 # [ 62.997679] nix-grpc-daemon[861]: time=2026-09-21T12:08:51.218Z level=INFO msg="All 1 paths already cached"5126worker2 # [ 63.003176] nix-grpc-daemon[787]: ts=2026-09-21T12:08:52Z level=debug event=want client=10 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05127worker1 # [ 63.010179] nix-grpc-daemon[763]: ts=2026-09-21T12:08:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371865128worker2 # [ 63.007604] nix-grpc-daemon[787]: ts=2026-09-21T12:08:52Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15129worker1 # [ 63.022100] nix-grpc-daemon[763]: ts=2026-09-21T12:08:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:371865130client # copying 1 paths to 'lb:50051'...5131client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5132worker1 # [ 63.040150] nix-grpc-daemon[763]: ts=2026-09-21T12:08:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:371865133worker1 # [ 63.051328] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)5134worker1 # [ 63.067444] nix-grpc-daemon[763]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 92 ms (attempt 1/5)5135worker1 # [ 63.164247] nix-grpc-daemon[763]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 13 ms (attempt 2/5)5136worker1 # [ 63.183077] nix-grpc-daemon[763]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 305 ms (attempt 3/5)5137worker1 # [ 63.494299] nix-grpc-daemon[763]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 620 ms (attempt 4/5)5138worker1 # [ 64.115720] nix-grpc-daemon[763]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5139worker1 # [ 64.119691] nix-grpc-daemon[763]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5140worker1 # [ 64.124943] nix-grpc-daemon[763]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5141worker1 # [ 64.343672] nix-daemon[848]: accepted connection from pid 1174, user nix-grpc-daemon (trusted)5142worker1 # [ 64.357326] nix-daemon[848]: reaped child process 1177, status = succeeded5143worker1 # [ 64.535079] nix-daemon[848]: accepted connection from pid 1179, user nix-grpc-daemon (trusted)5144worker1 # [ 64.547228] nix-daemon[848]: reaped child process 1182, status = succeeded5145lb # [ 64.570722] niks3-server[1029]: 2026/09/21 12:08:53 INFO Received uploads request method=POST path=/api/pending_closures5146worker1 # [ 64.575380] nix-grpc-daemon[861]: time=2026-09-21T12:08:52.797Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5147worker1 # [ 64.577359] nix-grpc-daemon[861]: time=2026-09-21T12:08:52.797Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5148lb # [ 64.602397] niks3-server[1029]: 2026/09/21 12:08:53 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5149lb # [ 64.612861] niks3-server[1029]: 2026/09/21 12:08:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5150worker1 # [ 64.613558] nix-grpc-daemon[861]: time=2026-09-21T12:08:52.835Z level=INFO msg="Uploading 1 narinfos"5151lb # [ 64.618927] niks3-server[1029]: 2026/09/21 12:08:53 INFO Signed narinfos id=21 count=15152lb # [ 64.624281] niks3-server[1029]: 2026/09/21 12:08:53 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5153lb # [ 64.635464] niks3-server[1029]: 2026/09/21 12:08:53 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5154lb # [ 64.643298] niks3-server[1029]: 2026/09/21 12:08:53 INFO Completed upload id=215155worker1 # [ 64.638110] nix-grpc-daemon[861]: time=2026-09-21T12:08:52.860Z level=INFO msg="Upload complete. (490ms)"5156lb # [ 64.645678] niks3-server[1029]: 2026/09/21 12:08:53 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5157worker1 # [ 64.642974] nix-grpc-daemon[763]: ts=2026-09-21T12:08:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=1 paths=1 nar_bytes_in=5605158worker1 # [ 64.649883] nix-grpc-daemon[763]: ts=2026-09-21T12:08:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:371865159worker1 # [ 64.659780] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)5160worker1 # [ 64.669970] nix-daemon[848]: reaped child process 1167, status = succeeded5161client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5162worker1 # [ 64.674094] nix-daemon[848]: accepted connection from pid 763, user nix-grpc-daemon (trusted)5163client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5164client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5165worker1 # [ 66.100658] nix-daemon[848]: reaped child process 1186, status = succeeded5166worker1 # [ 66.268060] nix-daemon[848]: accepted connection from pid 1202, user nix-grpc-daemon (trusted)5167worker1 # [ 66.280745] nix-daemon[848]: reaped child process 1205, status = succeeded5168lb # [ 66.301534] niks3-server[1029]: 2026/09/21 12:08:55 INFO Received uploads request method=POST path=/api/pending_closures5169worker1 # [ 66.328366] nix-grpc-daemon[861]: time=2026-09-21T12:08:54.550Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5170lb # [ 66.355459] niks3-server[1029]: 2026/09/21 12:08:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5171lb # [ 66.363434] niks3-server[1029]: 2026/09/21 12:08:55 INFO Signed narinfos id=22 count=15172worker1 # [ 66.361447] nix-grpc-daemon[861]: time=2026-09-21T12:08:54.583Z level=INFO msg="Uploading 1 narinfos"5173lb # [ 66.368808] niks3-server[1029]: 2026/09/21 12:08:55 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5174lb # [ 66.377095] niks3-server[1029]: 2026/09/21 12:08:55 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5175lb # [ 66.387438] niks3-server[1029]: 2026/09/21 12:08:55 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5176lb # [ 66.394907] niks3-server[1029]: 2026/09/21 12:08:55 INFO Completed upload id=225177worker1 # [ 66.389959] nix-grpc-daemon[861]: time=2026-09-21T12:08:54.612Z level=INFO msg="Upload complete. (308ms)"5178lb # [ 66.397156] niks3-server[1029]: 2026/09/21 12:08:55 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5179worker1 # [ 66.395883] nix-grpc-daemon[763]: ts=2026-09-21T12:08:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37186 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=10 outputs=15180worker1 # [ 66.409553] nix-daemon[848]: reaped child process 1184, status = succeeded5181client # copying 1 paths...5182worker2 # [ 66.438663] nix-grpc-daemon[787]: ts=2026-09-21T12:08:55Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:567805183worker2 # [ 66.450618] nix-grpc-daemon[787]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5184client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5185worker2 # [ 66.481621] nix-grpc-daemon[787]: ts=2026-09-21T12:08:55Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:567805186worker2 # [ 66.486463] nix-grpc-daemon[787]: ts=2026-09-21T12:08:55Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56780 duration_s=0 paths=1 nar_bytes_out=1205187client # [ 66.611207] nix-daemon[945]: reaped child process 1053, status = succeeded5188client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.96 seconds)5189client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25190client # [ 66.790181] nix-daemon[945]: accepted connection from pid 1072, user root (trusted)5191client # this derivation will be built:5192client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5193worker1 # [ 66.971730] nix-grpc-daemon[763]: ts=2026-09-21T12:08:55Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:371865194client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5195worker2 # [ 66.971274] nix-grpc-daemon[787]: ts=2026-09-21T12:08:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:567805196worker2 # [ 66.980942] nix-grpc-daemon[925]: time=2026-09-21T12:08:55.979Z level=INFO msg="All 1 paths already cached"5197worker2 # [ 66.997214] nix-grpc-daemon[787]: ts=2026-09-21T12:08:55Z level=debug event=want client=11 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05198worker2 # [ 67.003545] nix-grpc-daemon[787]: ts=2026-09-21T12:08:55Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15199worker2 # [ 67.008674] nix-grpc-daemon[787]: ts=2026-09-21T12:08:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:567805200worker2 # [ 67.019428] nix-grpc-daemon[787]: ts=2026-09-21T12:08:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:567805201worker2 # [ 67.032715] nix-grpc-daemon[925]: time=2026-09-21T12:08:56.032Z level=INFO msg="All 1 paths already cached"5202client # copying 1 paths to 'lb:50051'...5203client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5204worker2 # [ 67.040914] nix-grpc-daemon[787]: ts=2026-09-21T12:08:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:567805205worker2 # [ 67.052142] nix-daemon[870]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5206worker2 # [ 67.242230] nix-daemon[870]: accepted connection from pid 1053, user nix-grpc-daemon (trusted)5207worker2 # [ 67.255161] nix-daemon[870]: reaped child process 1056, status = succeeded5208lb # [ 67.380197] envoy[909]: conn peer=[2001:db8:1::1]:41490 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4341 tx=2188 ms=21245209lb # [ 67.384605] envoy[909]: conn peer=[2001:db8:1::1]:41496 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=411 ms=705210lb # [ 67.388113] envoy[909]: conn peer=[2001:db8:1::1]:41498 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4259 tx=1850 ms=23405211lb # [ 67.391591] envoy[909]: conn peer=[2001:db8:1::1]:41512 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=545212lb # [ 67.394943] envoy[909]: conn peer=[2001:db8:1::1]:58952 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3741 tx=2012 ms=35565213lb # [ 67.398608] envoy[909]: conn peer=[2001:db8:1::1]:58958 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=575214worker2 # [ 67.416237] nix-daemon[870]: accepted connection from pid 1058, user nix-grpc-daemon (trusted)5215worker2 # [ 67.427896] nix-daemon[870]: reaped child process 1061, status = succeeded5216lb # [ 67.453759] niks3-server[1029]: 2026/09/21 12:08:56 INFO Received uploads request method=POST path=/api/pending_closures5217worker2 # [ 67.453484] nix-grpc-daemon[925]: time=2026-09-21T12:08:56.452Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5218worker2 # [ 67.454851] nix-grpc-daemon[925]: time=2026-09-21T12:08:56.452Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5219lb # [ 67.481111] niks3-server[1029]: 2026/09/21 12:08:56 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5220lb # [ 67.490780] niks3-server[1029]: 2026/09/21 12:08:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5221worker2 # [ 67.486735] nix-grpc-daemon[925]: time=2026-09-21T12:08:56.485Z level=INFO msg="Uploading 1 narinfos"5222lb # [ 67.497362] niks3-server[1029]: 2026/09/21 12:08:56 INFO Signed narinfos id=23 count=15223lb # [ 67.501929] niks3-server[1029]: 2026/09/21 12:08:56 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5224lb # [ 67.512213] niks3-server[1029]: 2026/09/21 12:08:56 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5225worker2 # [ 67.509277] nix-grpc-daemon[925]: time=2026-09-21T12:08:56.508Z level=INFO msg="Upload complete. (445ms)"5226lb # [ 67.518685] niks3-server[1029]: 2026/09/21 12:08:56 INFO Completed upload id=235227worker2 # [ 67.512334] nix-grpc-daemon[787]: ts=2026-09-21T12:08:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56780 duration_s=0 paths=1 nar_bytes_in=5605228lb # [ 67.521917] niks3-server[1029]: 2026/09/21 12:08:56 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5229worker2 # [ 67.515489] nix-grpc-daemon[787]: ts=2026-09-21T12:08:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:567805230worker2 # [ 67.522576] nix-daemon[870]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5231worker2 # [ 67.523857] nix-daemon[870]: reaped child process 1051, status = succeeded5232worker2 # [ 67.529232] nix-daemon[870]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5233client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5234client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5235client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5236worker2 # [ 68.753811] nix-daemon[870]: reaped child process 1065, status = succeeded5237worker2 # [ 68.937531] nix-daemon[870]: accepted connection from pid 1081, user nix-grpc-daemon (trusted)5238worker2 # [ 68.949156] nix-daemon[870]: reaped child process 1084, status = succeeded5239lb # [ 68.976371] niks3-server[1029]: 2026/09/21 12:08:58 INFO Received uploads request method=POST path=/api/pending_closures5240worker2 # [ 69.009954] nix-grpc-daemon[925]: time=2026-09-21T12:08:58.009Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5241lb # [ 69.039273] niks3-server[1029]: 2026/09/21 12:08:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5242lb # [ 69.047933] niks3-server[1029]: 2026/09/21 12:08:58 INFO Signed narinfos id=24 count=15243worker2 # [ 69.041809] nix-grpc-daemon[925]: time=2026-09-21T12:08:58.040Z level=INFO msg="Uploading 1 narinfos"5244lb # [ 69.052109] niks3-server[1029]: 2026/09/21 12:08:58 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5245lb # [ 69.065865] niks3-server[1029]: 2026/09/21 12:08:58 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5246lb # [ 69.074217] niks3-server[1029]: 2026/09/21 12:08:58 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5247lb # [ 69.080123] niks3-server[1029]: 2026/09/21 12:08:58 INFO Completed upload id=245248worker2 # [ 69.070784] nix-grpc-daemon[925]: time=2026-09-21T12:08:58.070Z level=INFO msg="Upload complete. (334ms)"5249lb # [ 69.085319] niks3-server[1029]: 2026/09/21 12:08:58 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5250worker2 # [ 69.078307] nix-grpc-daemon[787]: ts=2026-09-21T12:08:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56780 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=11 outputs=15251client # copying 1 paths...5252worker2 # [ 69.091754] nix-daemon[870]: reaped child process 1063, status = succeeded5253worker1 # [ 69.094455] nix-grpc-daemon[763]: ts=2026-09-21T12:08:57Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:371865254worker1 # [ 69.107497] nix-grpc-daemon[763]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5255client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5256worker2 # [ 69.136147] nix-grpc-daemon[787]: ts=2026-09-21T12:08:58Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:567805257worker2 # [ 69.140115] nix-grpc-daemon[787]: ts=2026-09-21T12:08:58Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56780 duration_s=0 paths=1 nar_bytes_out=1205258client # [ 69.214085] nix-daemon[945]: reaped child process 1075, status = succeeded5259client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.61 seconds)5260(finished: subtest: an input only one worker has still reaches the builder, in 10.24 seconds)5261subtest: upload whose reference the worker lacks is completed from the cache5262client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5263client # this derivation will be built:5264client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5265client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5266client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.40 seconds)5267client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5268client # this derivation will be built:5269client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5270client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5271client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.40 seconds)5272client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5273client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.19 seconds)5274worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5275worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.22 seconds)5276worker2 # [ 70.479512] systemd[1]: Stopping Nix worker-protocol over gRPC...5277worker2 # [ 70.880250] nix-grpc-daemon[787]: ts=2026-09-21T12:08:59Z level=info event=signal_exit5278worker2 # [ 71.148240] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=scheduler_yield5279worker2 # [ 71.152384] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=scheduler_restarting addr=lb:500515280lb # [ 71.161210] niks3-server[1029]: 2026/09/21 12:09:00 INFO lead: released remote=192.168.1.4:426745281worker2 # [ 71.156779] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:567685282worker1 # [ 71.163476] nix-grpc-daemon[763]: ts=2026-09-21T12:08:59Z level=info event=scheduler_restarting addr=lb:500515283worker2 # [ 71.161591] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=worker_gone addr=192.168.1.4:500515284worker2 # [ 71.164588] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:567685285worker2 # [ 71.167886] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=worker_gone addr=192.168.1.3:500515286worker2 # [ 71.353524] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500515287worker2 # [ 71.357419] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5288worker1 # [ 71.363613] nix-grpc-daemon[763]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500515289worker1 # [ 71.367704] nix-grpc-daemon[763]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5290worker2 # [ 71.558675] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500515291worker2 # [ 71.562532] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5292worker1 # [ 71.568597] nix-grpc-daemon[763]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500515293worker1 # [ 71.572527] nix-grpc-daemon[763]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5294worker2 # [ 71.763623] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500515295worker2 # [ 71.767516] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5296worker1 # [ 71.773541] nix-grpc-daemon[763]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500515297worker1 # [ 71.777435] nix-grpc-daemon[763]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5298worker2 # [ 71.969475] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500515299worker2 # [ 71.973410] nix-grpc-daemon[787]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5300worker1 # [ 71.978246] nix-grpc-daemon[763]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500515301worker1 # [ 71.982214] nix-grpc-daemon[763]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5302lb # [ 72.041330] niks3-server[1029]: 2026/09/21 12:09:01 INFO lead: acquired remote=192.168.1.3:402965303worker1 # [ 72.039879] nix-grpc-daemon[763]: ts=2026-09-21T12:09:00Z level=info event=scheduler_take_over5304worker2 # [ 72.152539] nix-grpc-daemon[787]: ts=2026-09-21T12:09:01Z level=info event=scheduler_restarting5305worker1 # [ 72.184567] nix-grpc-daemon[763]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500515306worker1 # [ 72.187626] nix-grpc-daemon[763]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5307worker2 # [ 72.190442] nix-grpc-daemon[787]: ts=2026-09-21T12:09:01Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5308worker2 # [ 72.196755] nix-daemon[870]: reaped child process 873, status = succeeded5309worker1 # [ 72.385621] nix-grpc-daemon[763]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500515310worker1 # [ 72.389542] nix-grpc-daemon[763]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5311worker1 # [ 72.590687] nix-grpc-daemon[763]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500515312worker1 # [ 72.594722] nix-grpc-daemon[763]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5313worker1 # [ 72.795498] nix-grpc-daemon[763]: ts=2026-09-21T12:09:01Z level=info event=scheduler_connected addr=lb:500515314worker1 # [ 72.799492] nix-grpc-daemon[763]: ts=2026-09-21T12:09:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5315worker1 # [ 73.000408] nix-grpc-daemon[763]: ts=2026-09-21T12:09:01Z level=info event=scheduler_connected addr=lb:500515316worker1 # [ 73.012732] nix-grpc-daemon[763]: ts=2026-09-21T12:09:01Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:489485317worker2 # [ 73.008715] nix-grpc-daemon[787]: E0000 00:00:1789992542.007155 832 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5318worker2 # [ 73.013722] nix-grpc-daemon[787]: E0000 00:00:1789992542.007301 832 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5319worker1 # [ 73.017290] nix-grpc-daemon[763]: ts=2026-09-21T12:09:01Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05320worker2 # [ 73.949549] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5321worker2 # [ 73.953093] systemd[1]: Stopped Nix worker-protocol over gRPC.5322worker2 # [ 73.957329] systemd[1]: nix-grpc-daemon.service: Consumed 1.498s CPU time over 56.365s wall clock time, 53.1M memory peak, 54.2K incoming IP traffic, 167.6K outgoing IP traffic.5323worker2 # [ 73.968195] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5324worker2 # [ 73.971706] systemd[1]: Closed Nix worker-protocol over gRPC.5325worker2 # [ 73.974307] systemd[1]: nix-grpc-daemon.socket: Consumed 289us CPU time over 1min 3.586s wall clock time, 256K memory peak, 75.2K incoming IP traffic, 64.2K outgoing IP traffic.5326lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5327lb: must succeed: curl -sf localhost:9901/clusters5328lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5329lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.08 seconds)5330client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25331worker1 # [ 74.312927] nix-grpc-daemon[763]: ts=2026-09-21T12:09:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:371865332client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5333worker1 # [ 74.318303] nix-grpc-daemon[763]: ts=2026-09-21T12:09:02Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:371865334client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.28 seconds)5335worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5336worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.03 seconds)5337worker2 # [ 74.429658] systemd[1]: Listening on Nix worker-protocol over gRPC.5338worker1 # [ 74.504276] systemd[1]: Stopping Nix worker-protocol over gRPC...5339worker1 # [ 74.668749] nix-grpc-daemon[763]: ts=2026-09-21T12:09:02Z level=info event=signal_exit5340worker1 # [ 74.937932] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=scheduler_yield5341worker2 # [ 74.935765] systemd[1]: Starting scheduler-token.service...5342lb # [ 74.944787] niks3-server[1029]: 2026/09/21 12:09:04 INFO lead: released remote=192.168.1.3:402965343worker1 # [ 74.943575] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=scheduler_restarting addr=lb:500515344worker1 # [ 74.948342] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:489485345worker1 # [ 74.951645] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=worker_gone addr=192.168.1.3:500515346worker2 # [ 75.041099] systemd[1]: scheduler-token.service: Deactivated successfully.5347worker2 # [ 75.042576] systemd[1]: Finished scheduler-token.service.5348worker2 # [ 75.043880] systemd[1]: scheduler-token.service: Consumed 37ms CPU time over 104ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5349worker2 # [ 75.049477] systemd[1]: Starting Nix worker-protocol over gRPC...5350worker1 # [ 75.144775] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500515351worker1 # [ 75.148760] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5352worker2 # [ 75.215820] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5353worker2 # [ 75.218212] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5354worker2 # [ 75.221857] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15355worker2 # [ 75.230068] nix-grpc-daemon[1108]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5356worker2 # [ 75.231714] nix-grpc-daemon[1108]: E0000 00:00:1789992544.228741 1108 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5357worker2 # [ 75.236899] nix-grpc-daemon[1108]: E0000 00:00:1789992544.234057 1108 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5358worker2 # [ 75.240338] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5359worker2 # [ 75.243378] systemd[1]: Started Nix worker-protocol over gRPC.5360worker2 # [ 75.249762] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=scheduler_take_over5361lb # [ 75.259934] niks3-server[1029]: 2026/09/21 12:09:04 INFO lead: acquired remote=192.168.1.4:521325362worker2 # [ 75.253142] nix-grpc-daemon[1108]: E0000 00:00:1789992544.250878 1150 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5363worker2 # [ 75.264052] nix-grpc-daemon[1108]: E0000 00:00:1789992544.263441 1150 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5364worker2 # [ 75.268053] nix-grpc-daemon[1108]: E0000 00:00:1789992544.265769 1150 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5365worker2 # [ 75.278118] nix-grpc-daemon[1108]: E0000 00:00:1789992544.277492 1150 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5366worker2 # [ 75.287073] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500515367worker2 # [ 75.288578] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5368worker1 # [ 75.351253] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500515369worker1 # [ 75.354400] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5370worker2 # [ 75.489928] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500515371worker2 # [ 75.493927] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5372worker1 # [ 75.551917] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500515373worker1 # [ 75.555924] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5374worker2 # [ 75.695067] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500515375worker2 # [ 75.698870] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5376worker1 # [ 75.756866] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500515377worker1 # [ 75.760877] nix-grpc-daemon[763]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5378worker2 # [ 75.900327] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500515379worker2 # [ 75.904109] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5380worker1 # [ 75.939217] nix-grpc-daemon[763]: ts=2026-09-21T12:09:04Z level=info event=scheduler_restarting5381worker1 # [ 75.977369] nix-grpc-daemon[763]: ts=2026-09-21T12:09:04Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5382worker1 # [ 75.981849] nix-daemon[848]: reaped child process 850, status = succeeded5383worker2 # [ 76.105273] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:05Z level=info event=scheduler_connected addr=lb:500515384worker2 # [ 76.109218] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5385worker2 # [ 76.310790] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:05Z level=info event=scheduler_connected addr=lb:500515386worker2 # [ 76.323134] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:05Z level=info event=oidc_jwks provider=mock keys=15387worker2 # [ 76.327108] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:05Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:348165388worker2 # [ 76.330907] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:05Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05389worker1 # [ 76.817347] nix-grpc-daemon[763]: E0000 00:00:1789992545.038726 830 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5390worker1 # [ 76.822380] nix-grpc-daemon[763]: E0000 00:00:1789992545.044228 830 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5391lb # [ 77.380964] envoy[909]: conn peer=[2001:db8:1::1]:58970 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3742 tx=1988 ms=21815392lb # [ 77.384828] envoy[909]: conn peer=[2001:db8:1::1]:58972 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=473 tx=307 ms=135393lb # [ 77.388291] envoy[909]: conn peer=[2001:db8:1::4]:36780 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=45341 tx=2248 ms=542315394lb # [ 77.391881] envoy[909]: conn peer=[2001:db8:1::1]:50286 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=165395lb # [ 77.395294] envoy[909]: conn peer=[2001:db8:1::3]:54952 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=14709 tx=2995 ms=582515396worker1 # [ 77.876594] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5397worker1 # [ 77.879747] systemd[1]: Stopped Nix worker-protocol over gRPC.5398worker1 # [ 77.883981] systemd[1]: nix-grpc-daemon.service: Consumed 2.701s CPU time over 1min 517ms wall clock time, 61.7M memory peak, 90.4K incoming IP traffic, 168.8K outgoing IP traffic.5399worker1 # [ 77.895298] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5400worker1 # [ 77.898617] systemd[1]: Closed Nix worker-protocol over gRPC.5401worker1 # [ 77.901366] systemd[1]: nix-grpc-daemon.socket: Consumed 292us CPU time over 1min 7.678s wall clock time, 256K memory peak, 81.3K incoming IP traffic, 67.8K outgoing IP traffic.5402lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5403lb: must succeed: curl -sf localhost:9901/clusters5404lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5405lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.08 seconds)5406client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25407worker2 # [ 78.236584] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:348185408worker2 # [ 78.248797] nix-daemon[870]: accepted connection from pid 1108, user nix-grpc-daemon (trusted)5409worker2 # [ 78.268300] nix-grpc-daemon[1108]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 1 ms (attempt 1/5)5410worker2 # [ 78.274898] nix-grpc-daemon[1108]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 158 ms (attempt 2/5)5411worker2 # [ 78.438895] nix-grpc-daemon[1108]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 316 ms (attempt 3/5)5412worker2 # [ 78.760911] nix-grpc-daemon[1108]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 718 ms (attempt 4/5)5413worker2 # [ 79.485863] nix-grpc-daemon[1108]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5414worker2 # [ 79.489835] nix-grpc-daemon[1108]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5415client # copying 2 paths...5416client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5417worker2 # [ 79.511979] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:348185418client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5419worker2 # [ 79.525078] nix-daemon[870]: accepted connection from pid 1108, user nix-grpc-daemon (trusted)5420worker2 # [ 79.546980] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:08Z level=info event=niks3_push_started pid=11695421worker2 # [ 79.762982] nix-grpc-daemon[1169]: time=2026-09-21T12:09:08.761Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5422worker2 # [ 79.924089] nix-daemon[870]: accepted connection from pid 1182, user nix-grpc-daemon (trusted)5423worker2 # [ 79.934673] nix-daemon[870]: reaped child process 1185, status = succeeded5424lb # [ 79.961862] niks3-server[1029]: 2026/09/21 12:09:09 INFO Received uploads request method=POST path=/api/pending_closures5425lb # [ 79.972519] niks3-server[1029]: 2026/09/21 12:09:09 INFO Received uploads request method=POST path=/api/pending_closures5426worker2 # [ 79.970477] nix-grpc-daemon[1169]: time=2026-09-21T12:09:08.969Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5427worker2 # [ 79.972039] nix-grpc-daemon[1169]: time=2026-09-21T12:09:08.971Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5428worker2 # [ 79.973956] nix-grpc-daemon[1169]: time=2026-09-21T12:09:08.973Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5429lb # [ 80.020554] niks3-server[1029]: 2026/09/21 12:09:09 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5430lb # [ 80.037657] niks3-server[1029]: 2026/09/21 12:09:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5431lb # [ 80.043284] niks3-server[1029]: 2026/09/21 12:09:09 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5432lb # [ 80.049632] niks3-server[1029]: 2026/09/21 12:09:09 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5433lb # [ 80.055319] niks3-server[1029]: 2026/09/21 12:09:09 INFO Signed narinfos id=25 count=15434lb # [ 80.056937] niks3-server[1029]: 2026/09/21 12:09:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5435lb # [ 80.058610] niks3-server[1029]: 2026/09/21 12:09:09 INFO Signed narinfos id=26 count=25436worker2 # [ 80.052258] nix-grpc-daemon[1169]: time=2026-09-21T12:09:09.049Z level=INFO msg="Uploading 3 narinfos"5437lb # [ 80.067560] niks3-server[1029]: 2026/09/21 12:09:09 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5438lb # [ 80.076272] niks3-server[1029]: 2026/09/21 12:09:09 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5439lb # [ 80.080056] niks3-server[1029]: 2026/09/21 12:09:09 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5440lb # [ 80.083033] niks3-server[1029]: 2026/09/21 12:09:09 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5441lb # [ 80.085052] niks3-server[1029]: 2026/09/21 12:09:09 INFO Completed upload id=255442lb # [ 80.086314] niks3-server[1029]: 2026/09/21 12:09:09 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5443lb # [ 80.090026] niks3-server[1029]: 2026/09/21 12:09:09 INFO Completed upload id=265444lb # [ 80.091864] niks3-server[1029]: 2026/09/21 12:09:09 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5445worker2 # [ 80.080904] nix-grpc-daemon[1169]: time=2026-09-21T12:09:09.080Z level=INFO msg="Upload complete. (314ms)"5446worker2 # [ 80.086316] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34818 duration_s=0 paths=2 nar_bytes_in=3125447worker2 # [ 80.098950] nix-daemon[870]: reaped child process 1167, status = succeeded5448client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 2.14 seconds)5449worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5450worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.03 seconds)5451worker1 # [ 80.229702] systemd[1]: Listening on Nix worker-protocol over gRPC.5452worker1 # [ 80.237738] systemd[1]: Starting Nix worker-protocol over gRPC...5453worker1 # [ 80.441443] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:08Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5454worker1 # [ 80.443334] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:08Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5455worker1 # [ 80.446965] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:08Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15456worker1 # [ 80.455067] nix-grpc-daemon[1229]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5457worker1 # [ 80.456596] nix-grpc-daemon[1229]: E0000 00:00:1789992548.676695 1229 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5458worker1 # [ 80.462535] nix-grpc-daemon[1229]: E0000 00:00:1789992548.681865 1229 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5459worker1 # [ 80.465417] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:08Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5460worker1 # [ 80.468631] systemd[1]: Started Nix worker-protocol over gRPC.5461worker1 # [ 80.473960] nix-grpc-daemon[1229]: E0000 00:00:1789992548.695553 1271 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5462lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5463lb: must succeed: curl -sf localhost:9901/clusters5464worker1 # [ 80.486275] nix-grpc-daemon[1229]: E0000 00:00:1789992548.708584 1271 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5465worker1 # [ 80.490134] nix-grpc-daemon[1229]: E0000 00:00:1789992548.710810 1271 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5466worker1 # [ 80.500389] nix-grpc-daemon[1229]: E0000 00:00:1789992548.722752 1271 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5467worker1 # [ 80.513953] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:08Z level=info event=scheduler_connected addr=lb:500515468worker2 # [ 80.512923] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:09Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:348165469worker2 # [ 80.516699] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:09Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05470lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5471lb: must succeed: curl -sf localhost:9901/clusters5472lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5473lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.17 seconds)5474(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 12.43 seconds)5475subtest: scheduler down: the other node takes the lock and keeps it5476lb: must succeed: curl -sf localhost:9901/clusters5477lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5478lb: must succeed: curl -sf localhost:9901/clusters5479lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5480lb: must succeed: curl -sf localhost:9901/clusters5481lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5482lb: must succeed: curl -sf localhost:9901/clusters5483lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5484worker2 # [ 81.929293] systemd[1]: Stopping Nix worker-protocol over gRPC...5485worker2 # [ 82.244484] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:11Z level=info event=signal_exit5486worker2 # [ 82.456676] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:11Z level=info event=scheduler_yield5487lb # [ 82.469501] niks3-server[1029]: 2026/09/21 12:09:11 INFO lead: released remote=192.168.1.4:521325488worker1 # [ 82.465499] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:10Z level=info event=scheduler_restarting addr=lb:500515489worker2 # [ 82.461926] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:11Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:348165490worker2 # [ 82.466488] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:11Z level=info event=worker_gone addr=192.168.1.3:500515491worker2 # [ 82.471172] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:11Z level=info event=scheduler_restarting addr=lb:500515492worker2 # [ 82.474193] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:11Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:348165493worker2 # [ 82.477412] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:11Z level=info event=worker_gone addr=192.168.1.4:500515494worker1 # [ 82.665703] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:10Z level=info event=scheduler_connected addr=lb:500515495worker1 # [ 82.669738] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5496worker2 # [ 82.667669] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:11Z level=info event=scheduler_connected addr=lb:500515497worker2 # [ 82.671469] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5498worker1 # [ 82.873221] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:11Z level=info event=scheduler_connected addr=lb:500515499worker2 # [ 82.872475] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:11Z level=info event=scheduler_connected addr=lb:500515500worker1 # [ 82.876377] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5501worker2 # [ 82.876314] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5502worker1 # [ 83.073477] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:11Z level=info event=scheduler_connected addr=lb:500515503worker1 # [ 83.077397] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5504worker2 # [ 83.077545] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:12Z level=info event=scheduler_connected addr=lb:500515505worker2 # [ 83.081314] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5506worker1 # [ 83.278359] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:11Z level=info event=scheduler_connected addr=lb:500515507worker1 # [ 83.282293] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5508worker2 # [ 83.283311] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:12Z level=info event=scheduler_connected addr=lb:500515509worker2 # [ 83.286518] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5510worker2 # [ 83.461199] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:12Z level=info event=scheduler_restarting5511worker1 # [ 83.483785] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:11Z level=info event=scheduler_connected addr=lb:500515512worker1 # [ 83.487614] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5513worker2 # [ 83.491852] nix-daemon[870]: reaped child process 1160, status = succeeded5514worker2 # [ 83.497713] nix-grpc-daemon[1108]: ts=2026-09-21T12:09:12Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5515worker1 # [ 83.688553] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:11Z level=info event=scheduler_connected addr=lb:500515516worker1 # [ 83.692578] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5517worker1 # [ 83.893229] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:12Z level=info event=scheduler_connected addr=lb:500515518worker1 # [ 83.897246] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5519worker1 # [ 84.098095] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:12Z level=info event=scheduler_connected addr=lb:500515520worker1 # [ 84.102187] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5521worker2 # [ 84.274654] nix-grpc-daemon[1108]: E0000 00:00:1789992553.273825 1153 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5522worker2 # [ 84.279524] nix-grpc-daemon[1108]: E0000 00:00:1789992553.273962 1153 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5523worker1 # [ 84.303084] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:12Z level=info event=scheduler_connected addr=lb:500515524worker1 # [ 84.307149] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5525worker1 # [ 84.508616] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:12Z level=info event=scheduler_connected addr=lb:500515526worker1 # [ 84.512333] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5527worker1 # [ 84.712972] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:12Z level=info event=scheduler_connected addr=lb:500515528worker1 # [ 84.716926] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5529worker1 # [ 84.918142] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:13Z level=info event=scheduler_connected addr=lb:500515530worker1 # [ 84.922248] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5531worker1 # [ 85.123405] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:13Z level=info event=scheduler_connected addr=lb:500515532worker1 # [ 85.127405] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5533worker2 # [ 85.262161] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5534worker2 # [ 85.266658] systemd[1]: Stopped Nix worker-protocol over gRPC.5535worker2 # [ 85.268679] systemd[1]: nix-grpc-daemon.service: Consumed 366ms CPU time over 10.211s wall clock time, 33.1M memory peak, 20.1K incoming IP traffic, 31.4K outgoing IP traffic.5536worker2 # [ 85.279777] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5537worker2 # [ 85.283782] systemd[1]: Closed Nix worker-protocol over gRPC.5538worker2 # [ 85.286565] systemd[1]: nix-grpc-daemon.socket: Consumed 302us CPU time over 10.859s wall clock time, 256K memory peak, 20.7K incoming IP traffic, 15.5K outgoing IP traffic.5539worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5540worker1 # [ 85.328282] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:13Z level=info event=scheduler_connected addr=lb:500515541worker1 # [ 85.332775] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5542worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.09 seconds)5543worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5544worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5545lb # [ 85.477939] niks3-server[1029]: 2026/09/21 12:09:14 INFO lead: acquired remote=192.168.1.3:382485546worker1 # [ 85.475740] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:13Z level=info event=scheduler_take_over5547worker1 # [ 85.533071] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:13Z level=info event=scheduler_connected addr=lb:500515548worker1 # [ 85.535746] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5549worker1 # [ 85.736540] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:13Z level=info event=scheduler_connected addr=lb:500515550worker1 # [ 85.740444] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5551worker1 # [ 85.941170] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:14Z level=info event=scheduler_connected addr=lb:500515552worker1 # [ 85.948494] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:14Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:585925553worker1 # [ 85.952087] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:14Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05554worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5555worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5556client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&25557worker1 # [ 86.782278] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:15Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:585965558worker1 # [ 86.789859] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5559client # don't know how to build these paths:5560client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5561worker1 # [ 86.806790] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:585965562worker1 # [ 86.818451] nix-grpc-daemon[1229]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 43 ms (attempt 1/5)5563worker1 # [ 86.867423] nix-grpc-daemon[1229]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 193 ms (attempt 2/5)5564worker1 # [ 87.066178] nix-grpc-daemon[1229]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 144 ms (attempt 3/5)5565worker1 # [ 87.216181] nix-grpc-daemon[1229]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 118 ms (attempt 4/5)5566worker1 # [ 87.340500] nix-grpc-daemon[1229]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5567worker1 # [ 87.343429] nix-grpc-daemon[1229]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5568worker1 # [ 87.373170] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:15Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=05569lb # [ 87.381837] envoy[909]: conn peer=[2001:db8:1::1]:50298 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1005 tx=305 ms=18635570worker1 # [ 87.378170] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:15Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=15571lb # [ 87.385934] envoy[909]: conn peer=[2001:db8:1::4]:47442 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=8603 tx=535 ms=82025572worker1 # [ 87.417810] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:585965573worker1 # [ 87.430666] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:585965574client # copying 1 paths to 'lb:50051'...5575client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...5576worker1 # [ 87.443161] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:585965577worker1 # [ 87.451762] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5578worker1 # [ 87.474427] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:15Z level=info event=niks3_push_started pid=13035579worker1 # [ 87.721594] nix-grpc-daemon[1303]: time=2026-09-21T12:09:15.943Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""5580worker1 # [ 87.932670] nix-daemon[848]: accepted connection from pid 1316, user nix-grpc-daemon (trusted)5581worker1 # [ 87.944976] nix-daemon[848]: reaped child process 1319, status = succeeded5582worker1 # [ 88.158342] nix-daemon[848]: accepted connection from pid 1321, user nix-grpc-daemon (trusted)5583worker1 # [ 88.170468] nix-daemon[848]: reaped child process 1324, status = succeeded5584lb # [ 88.194222] niks3-server[1029]: 2026/09/21 12:09:17 INFO Received uploads request method=POST path=/api/pending_closures5585worker1 # [ 88.197722] nix-grpc-daemon[1303]: time=2026-09-21T12:09:16.419Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5586worker1 # [ 88.199159] nix-grpc-daemon[1303]: time=2026-09-21T12:09:16.419Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"5587lb # [ 88.226070] niks3-server[1029]: 2026/09/21 12:09:17 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst5588lb # [ 88.238950] niks3-server[1029]: 2026/09/21 12:09:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5589lb # [ 88.244346] niks3-server[1029]: 2026/09/21 12:09:17 INFO Signed narinfos id=27 count=15590worker1 # [ 88.242262] nix-grpc-daemon[1303]: time=2026-09-21T12:09:16.464Z level=INFO msg="Uploading 1 narinfos"5591lb # [ 88.256756] niks3-server[1029]: 2026/09/21 12:09:17 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls5592lb # [ 88.264179] niks3-server[1029]: 2026/09/21 12:09:17 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5593lb # [ 88.271913] niks3-server[1029]: 2026/09/21 12:09:17 INFO Completed upload id=275594worker1 # [ 88.266797] nix-grpc-daemon[1303]: time=2026-09-21T12:09:16.489Z level=INFO msg="Upload complete. (541ms)"5595lb # [ 88.276961] niks3-server[1029]: 2026/09/21 12:09:17 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo5596worker1 # [ 88.271132] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58596 duration_s=0 paths=1 nar_bytes_in=5205597worker1 # [ 88.276372] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:585965598worker1 # [ 88.286111] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5599worker1 # [ 88.290521] nix-daemon[848]: reaped child process 1301, status = succeeded5600worker1 # [ 88.295533] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5601client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...5602client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5603client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5604client # farm-a> LOG-farm-a-on-standby5605worker1 # [ 89.321309] nix-daemon[848]: reaped child process 1328, status = succeeded5606worker1 # [ 89.513382] nix-daemon[848]: accepted connection from pid 1344, user nix-grpc-daemon (trusted)5607worker1 # [ 89.523895] nix-daemon[848]: reaped child process 1347, status = succeeded5608lb # [ 89.547706] niks3-server[1029]: 2026/09/21 12:09:18 INFO Received uploads request method=POST path=/api/pending_closures5609worker1 # [ 89.553253] nix-grpc-daemon[1303]: time=2026-09-21T12:09:17.775Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5610worker1 # [ 89.554778] nix-grpc-daemon[1303]: time=2026-09-21T12:09:17.777Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"5611lb # [ 89.593838] niks3-server[1029]: 2026/09/21 12:09:18 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5612lb # [ 89.599931] niks3-server[1029]: 2026/09/21 12:09:18 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5613lb # [ 89.606382] niks3-server[1029]: 2026/09/21 12:09:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5614lb # [ 89.611317] niks3-server[1029]: 2026/09/21 12:09:18 INFO Signed narinfos id=28 count=15615worker1 # [ 89.608955] nix-grpc-daemon[1303]: time=2026-09-21T12:09:17.830Z level=INFO msg="Uploading 1 narinfos"5616lb # [ 89.616750] niks3-server[1029]: 2026/09/21 12:09:18 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls5617lb # [ 89.627350] niks3-server[1029]: 2026/09/21 12:09:18 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5618lb # [ 89.632302] niks3-server[1029]: 2026/09/21 12:09:18 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo5619lb # [ 89.634627] niks3-server[1029]: 2026/09/21 12:09:18 INFO Completed upload id=285620worker1 # [ 89.629619] nix-grpc-daemon[1303]: time=2026-09-21T12:09:17.851Z level=INFO msg="Upload complete. (325ms)"5621worker1 # [ 89.637676] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58596 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=15622worker1 # [ 89.658299] nix-daemon[848]: reaped child process 1326, status = succeeded5623worker1 # [ 89.659951] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:17Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=05624worker1 # [ 89.661933] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:17Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=15625worker1 # [ 89.706260] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:585965626worker1 # [ 89.719168] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:585965627worker1 # [ 89.732366] nix-grpc-daemon[1303]: time=2026-09-21T12:09:17.954Z level=INFO msg="All 1 paths already cached"5628client # copying 1 paths to 'lb:50051'...5629client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...5630worker1 # [ 89.740562] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:585965631worker1 # [ 89.751162] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5632worker1 # [ 89.955254] nix-daemon[848]: accepted connection from pid 1353, user nix-grpc-daemon (trusted)5633worker1 # [ 89.967269] nix-daemon[848]: reaped child process 1356, status = succeeded5634worker1 # [ 90.146363] nix-daemon[848]: accepted connection from pid 1358, user nix-grpc-daemon (trusted)5635worker1 # [ 90.158951] nix-daemon[848]: reaped child process 1361, status = succeeded5636lb # [ 90.180627] niks3-server[1029]: 2026/09/21 12:09:19 INFO Received uploads request method=POST path=/api/pending_closures5637worker1 # [ 90.183422] nix-grpc-daemon[1303]: time=2026-09-21T12:09:18.405Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5638worker1 # [ 90.185444] nix-grpc-daemon[1303]: time=2026-09-21T12:09:18.407Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"5639lb # [ 90.212755] niks3-server[1029]: 2026/09/21 12:09:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5640lb # [ 90.218173] niks3-server[1029]: 2026/09/21 12:09:19 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst5641lb # [ 90.223890] niks3-server[1029]: 2026/09/21 12:09:19 INFO Signed narinfos id=29 count=15642worker1 # [ 90.223808] nix-grpc-daemon[1303]: time=2026-09-21T12:09:18.446Z level=INFO msg="Uploading 1 narinfos"5643lb # [ 90.233377] niks3-server[1029]: 2026/09/21 12:09:19 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls5644lb # [ 90.244504] niks3-server[1029]: 2026/09/21 12:09:19 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5645lb # [ 90.248741] niks3-server[1029]: 2026/09/21 12:09:19 INFO Completed upload id=295646worker1 # [ 90.246058] nix-grpc-daemon[1303]: time=2026-09-21T12:09:18.468Z level=INFO msg="Upload complete. (483ms)"5647lb # [ 90.251890] niks3-server[1029]: 2026/09/21 12:09:19 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo5648worker1 # [ 90.248636] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58596 duration_s=0 paths=1 nar_bytes_in=6405649worker1 # [ 90.252760] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:585965650worker1 # [ 90.259123] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5651worker1 # [ 90.261464] nix-daemon[848]: reaped child process 1351, status = succeeded5652worker1 # [ 90.266893] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5653client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...5654client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5655client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5656client # farm-b> LOG-farm-b-on-standby5657worker1 # [ 91.480860] nix-daemon[848]: reaped child process 1365, status = succeeded5658worker1 # [ 91.627664] nix-daemon[848]: accepted connection from pid 1381, user nix-grpc-daemon (trusted)5659worker1 # [ 91.639993] nix-daemon[848]: reaped child process 1384, status = succeeded5660lb # [ 91.663734] niks3-server[1029]: 2026/09/21 12:09:20 INFO Received uploads request method=POST path=/api/pending_closures5661worker1 # [ 91.666757] nix-grpc-daemon[1303]: time=2026-09-21T12:09:19.888Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5662worker1 # [ 91.668455] nix-grpc-daemon[1303]: time=2026-09-21T12:09:19.890Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"5663lb # [ 91.702924] niks3-server[1029]: 2026/09/21 12:09:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5664worker1 # [ 91.703537] nix-grpc-daemon[1303]: time=2026-09-21T12:09:19.925Z level=INFO msg="Uploading 1 narinfos"5665lb # [ 91.708672] niks3-server[1029]: 2026/09/21 12:09:20 INFO Signed narinfos id=30 count=15666lb # [ 91.723926] niks3-server[1029]: 2026/09/21 12:09:20 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls5667lb # [ 91.730117] niks3-server[1029]: 2026/09/21 12:09:20 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5668lb # [ 91.740878] niks3-server[1029]: 2026/09/21 12:09:20 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5669lb # [ 91.745280] niks3-server[1029]: 2026/09/21 12:09:20 INFO Completed upload id=305670worker1 # [ 91.741275] nix-grpc-daemon[1303]: time=2026-09-21T12:09:19.963Z level=INFO msg="Upload complete. (267ms)"5671lb # [ 91.747180] niks3-server[1029]: 2026/09/21 12:09:20 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo5672worker1 # [ 91.744717] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58596 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=15673lb # [ 91.752033] niks3-server[1029]: 2026/09/21 12:09:20 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst5674worker1 # [ 91.752549] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:19Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=05675worker1 # [ 91.754493] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:19Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=15676worker1 # [ 91.757057] nix-daemon[848]: reaped child process 1363, status = succeeded5677worker1 # [ 91.796152] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:585965678worker1 # [ 91.804773] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:585965679worker1 # [ 91.814856] nix-grpc-daemon[1303]: time=2026-09-21T12:09:20.037Z level=INFO msg="All 2 paths already cached"5680client # copying 1 paths to 'lb:50051'...5681worker1 # [ 91.821738] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:585965682client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...5683worker1 # [ 91.827402] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5684worker1 # [ 91.983646] nix-daemon[848]: accepted connection from pid 1390, user nix-grpc-daemon (trusted)5685worker1 # [ 91.995984] nix-daemon[848]: reaped child process 1393, status = succeeded5686worker1 # [ 92.154689] nix-daemon[848]: accepted connection from pid 1395, user nix-grpc-daemon (trusted)5687worker1 # [ 92.167161] nix-daemon[848]: reaped child process 1398, status = succeeded5688lb # [ 92.189638] niks3-server[1029]: 2026/09/21 12:09:21 INFO Received uploads request method=POST path=/api/pending_closures5689worker1 # [ 92.195366] nix-grpc-daemon[1303]: time=2026-09-21T12:09:20.417Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5690worker1 # [ 92.196789] nix-grpc-daemon[1303]: time=2026-09-21T12:09:20.417Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"5691lb # [ 92.220923] niks3-server[1029]: 2026/09/21 12:09:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5692lb # [ 92.225478] niks3-server[1029]: 2026/09/21 12:09:21 INFO Signed narinfos id=31 count=15693worker1 # [ 92.223366] nix-grpc-daemon[1303]: time=2026-09-21T12:09:20.445Z level=INFO msg="Uploading 1 narinfos"5694lb # [ 92.236687] niks3-server[1029]: 2026/09/21 12:09:21 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst5695lb # [ 92.250843] niks3-server[1029]: 2026/09/21 12:09:21 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5696lb # [ 92.253734] niks3-server[1029]: 2026/09/21 12:09:21 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls5697worker1 # [ 92.253908] nix-grpc-daemon[1303]: time=2026-09-21T12:09:20.476Z level=INFO msg="Upload complete. (420ms)"5698lb # [ 92.258948] niks3-server[1029]: 2026/09/21 12:09:21 INFO Completed upload id=315699lb # [ 92.262201] niks3-server[1029]: 2026/09/21 12:09:21 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo5700worker1 # [ 92.257104] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58596 duration_s=0 paths=1 nar_bytes_in=7685701worker1 # [ 92.262420] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:585965702worker1 # [ 92.273805] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5703worker1 # [ 92.276149] nix-daemon[848]: reaped child process 1388, status = succeeded5704worker1 # [ 92.281754] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5705client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...5706client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5707client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5708client # farm-top> LOG-farm-top-on-standby5709worker1 # [ 92.894703] nix-daemon[848]: reaped child process 1402, status = succeeded5710worker1 # [ 93.045611] nix-daemon[848]: accepted connection from pid 1418, user nix-grpc-daemon (trusted)5711worker1 # [ 93.056371] nix-daemon[848]: reaped child process 1421, status = succeeded5712lb # [ 93.079434] niks3-server[1029]: 2026/09/21 12:09:22 INFO Received uploads request method=POST path=/api/pending_closures5713worker1 # [ 93.084079] nix-grpc-daemon[1303]: time=2026-09-21T12:09:21.306Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5714worker1 # [ 93.085649] nix-grpc-daemon[1303]: time=2026-09-21T12:09:21.307Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"5715lb # [ 93.117426] niks3-server[1029]: 2026/09/21 12:09:22 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst5716lb # [ 93.127370] niks3-server[1029]: 2026/09/21 12:09:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5717lb # [ 93.132825] niks3-server[1029]: 2026/09/21 12:09:22 INFO Signed narinfos id=32 count=15718worker1 # [ 93.130633] nix-grpc-daemon[1303]: time=2026-09-21T12:09:21.352Z level=INFO msg="Uploading 1 narinfos"5719lb # [ 93.141448] niks3-server[1029]: 2026/09/21 12:09:22 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5720lb # [ 93.148888] niks3-server[1029]: 2026/09/21 12:09:22 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls5721lb # [ 93.153950] niks3-server[1029]: 2026/09/21 12:09:22 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5722lb # [ 93.159900] niks3-server[1029]: 2026/09/21 12:09:22 INFO Completed upload id=325723worker1 # [ 93.154966] nix-grpc-daemon[1303]: time=2026-09-21T12:09:21.377Z level=INFO msg="Upload complete. (267ms)"5724lb # [ 93.162622] niks3-server[1029]: 2026/09/21 12:09:22 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo5725worker1 # [ 93.163326] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58596 duration_s=0 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=15726worker1 # [ 93.175311] nix-daemon[848]: reaped child process 1400, status = succeeded5727client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 6.69 seconds)5728client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath5729client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.23 seconds)5730worker2 # [ 93.501545] systemd[1]: Listening on Nix worker-protocol over gRPC.5731worker2 # [ 93.508250] systemd[1]: Starting scheduler-token.service...5732worker2 # [ 93.586859] systemd[1]: scheduler-token.service: Deactivated successfully.5733worker2 # [ 93.588074] systemd[1]: Finished scheduler-token.service.5734worker2 # [ 93.589494] systemd[1]: scheduler-token.service: Consumed 18ms CPU time over 79ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5735worker2 # [ 93.594985] systemd[1]: Starting Nix worker-protocol over gRPC...5736worker2 # [ 93.760244] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:22Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5737worker2 # [ 93.762230] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:22Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5738worker2 # [ 93.765933] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:22Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15739worker2 # [ 93.774069] nix-grpc-daemon[1209]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5740worker2 # [ 93.775499] nix-grpc-daemon[1209]: E0000 00:00:1789992562.772558 1209 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5741worker2 # [ 93.781053] nix-grpc-daemon[1209]: E0000 00:00:1789992562.777850 1209 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5742worker2 # [ 93.784464] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:22Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5743worker2 # [ 93.786611] systemd[1]: Started Nix worker-protocol over gRPC.5744lb: must succeed: curl -sf localhost:9901/clusters5745worker2 # [ 93.792861] nix-grpc-daemon[1209]: E0000 00:00:1789992562.792271 1251 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5746worker2 # [ 93.807624] nix-grpc-daemon[1209]: E0000 00:00:1789992562.806974 1251 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5747worker2 # [ 93.813064] nix-grpc-daemon[1209]: E0000 00:00:1789992562.809595 1251 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5748worker2 # [ 93.823709] nix-grpc-daemon[1209]: E0000 00:00:1789992562.823097 1251 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5749worker2 # [ 93.833943] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:22Z level=info event=scheduler_connected addr=lb:500515750worker1 # [ 93.846166] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:22Z level=info event=oidc_jwks provider=mock keys=15751worker1 # [ 93.849844] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:22Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:585925752worker1 # [ 93.853647] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:22Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05753lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5754worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5755worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5756lb: must succeed: curl -sf localhost:9901/clusters5757lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5758lb: must succeed: curl -sf localhost:9901/clusters5759lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5760worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5761worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5762(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.51 seconds)5763subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5764client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs5765client # [ 94.237698] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.5766client # Running as unit: rs.service; invocation ID: 6a3dc738f43a4876a026b8d3161ca12b5767client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.08 seconds)5768worker2 # [ 94.457565] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:503925769worker2 # [ 94.469329] nix-daemon[870]: accepted connection from pid 1209, user nix-grpc-daemon (trusted)5770client # [ 94.496326] nix[1176]: don't know how to build these paths:5771client # [ 94.497729] nix[1176]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5772worker1 # [ 94.495768] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:22Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=05773worker1 # [ 94.501254] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:22Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=15774worker1 # [ 94.545785] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:585965775worker1 # [ 94.558678] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:585965776client # [ 94.579260] nix[1176]: copying 1 paths to 'lb:50051'...5777client # [ 94.582719] nix[1176]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...5778worker1 # [ 94.572477] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:585965779worker1 # [ 94.584138] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5780worker1 # [ 94.774682] nix-daemon[848]: accepted connection from pid 1434, user nix-grpc-daemon (trusted)5781worker1 # [ 94.786813] nix-daemon[848]: reaped child process 1437, status = succeeded5782worker1 # [ 94.963334] nix-daemon[848]: accepted connection from pid 1439, user nix-grpc-daemon (trusted)5783worker1 # [ 94.975170] nix-daemon[848]: reaped child process 1442, status = succeeded5784lb # [ 94.999443] niks3-server[1029]: 2026/09/21 12:09:24 INFO Received uploads request method=POST path=/api/pending_closures5785worker1 # [ 95.004728] nix-grpc-daemon[1303]: time=2026-09-21T12:09:23.226Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5786worker1 # [ 95.006183] nix-grpc-daemon[1303]: time=2026-09-21T12:09:23.226Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"5787lb # [ 95.026878] niks3-server[1029]: 2026/09/21 12:09:24 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst5788lb # [ 95.040709] niks3-server[1029]: 2026/09/21 12:09:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5789worker1 # [ 95.040920] nix-grpc-daemon[1303]: time=2026-09-21T12:09:23.263Z level=INFO msg="Uploading 1 narinfos"5790lb # [ 95.046348] niks3-server[1029]: 2026/09/21 12:09:24 INFO Signed narinfos id=33 count=15791lb # [ 95.052263] niks3-server[1029]: 2026/09/21 12:09:24 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls5792lb # [ 95.062482] niks3-server[1029]: 2026/09/21 12:09:24 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5793lb # [ 95.069203] niks3-server[1029]: 2026/09/21 12:09:24 INFO Completed upload id=335794worker1 # [ 95.064138] nix-grpc-daemon[1303]: time=2026-09-21T12:09:23.286Z level=INFO msg="Upload complete. (468ms)"5795lb # [ 95.073944] niks3-server[1029]: 2026/09/21 12:09:24 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo5796worker1 # [ 95.068135] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58596 duration_s=0 paths=1 nar_bytes_in=4165797worker1 # [ 95.072850] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:585965798worker1 # [ 95.082153] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5799worker1 # [ 95.090122] nix-daemon[848]: reaped child process 1432, status = succeeded5800worker1 # [ 95.094073] nix-daemon[848]: accepted connection from pid 1229, user nix-grpc-daemon (trusted)5801client # [ 95.104421] nix[1176]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...5802lb: must succeed: curl -sf localhost:9901/clusters5803lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5804lb: must succeed: curl -sf localhost:9901/clusters5805lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5806lb: must succeed: curl -sf localhost:9901/clusters5807lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5808lb: must succeed: curl -sf localhost:9901/clusters5809lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5810worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5811worker2 # [ 97.199574] systemd-journald[288]: Received client request to rotate journal, rotating.5812worker2 # [ 97.217060] systemd-journald[288]: Vacuuming done, freed 0B of archived journals from /var/log/journal/fc845428e8c84568a79c4aa5148befd7.5813worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.07 seconds)5814worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5815worker1 # [ 97.146190] systemd[1]: Stopping Nix worker-protocol over gRPC...5816worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)5817client: must succeed: sleep 5.05818lb # [ 97.382259] envoy[909]: conn peer=[2001:db8:1::1]:38414 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9620 tx=4146 ms=63985819worker1 # [ 97.467355] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:25Z level=info event=draining5820client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5821worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5822worker1 # [ 102.524938] nix-daemon[848]: reaped child process 1446, status = succeeded5823worker1 # [ 102.709077] nix-daemon[848]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5824worker1 # [ 102.720077] nix-daemon[848]: reaped child process 1490, status = succeeded5825lb # [ 102.745260] niks3-server[1029]: 2026/09/21 12:09:31 INFO Received uploads request method=POST path=/api/pending_closures5826worker1 # [ 102.749509] nix-grpc-daemon[1303]: time=2026-09-21T12:09:30.971Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5827worker1 # [ 102.754718] nix-grpc-daemon[1303]: time=2026-09-21T12:09:30.973Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"5828lb # [ 102.788967] niks3-server[1029]: 2026/09/21 12:09:31 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5829lb # [ 102.800879] niks3-server[1029]: 2026/09/21 12:09:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5830lb # [ 102.806820] niks3-server[1029]: 2026/09/21 12:09:31 INFO Signed narinfos id=34 count=15831worker1 # [ 102.805221] nix-grpc-daemon[1303]: time=2026-09-21T12:09:31.027Z level=INFO msg="Uploading 1 narinfos"5832lb # [ 102.812313] niks3-server[1029]: 2026/09/21 12:09:31 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5833lb # [ 102.822293] niks3-server[1029]: 2026/09/21 12:09:31 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls5834lb # [ 102.825474] niks3-server[1029]: 2026/09/21 12:09:31 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5835lb # [ 102.831149] niks3-server[1029]: 2026/09/21 12:09:31 INFO Completed upload id=345836worker1 # [ 102.826154] nix-grpc-daemon[1303]: time=2026-09-21T12:09:31.048Z level=INFO msg="Upload complete. (318ms)"5837lb # [ 102.833663] niks3-server[1029]: 2026/09/21 12:09:31 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo5838worker1 # [ 102.834619] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58596 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=15839worker1 # [ 102.847569] nix-daemon[848]: reaped child process 1444, status = succeeded5840client # [ 102.887591] systemd[1]: rs.service: Deactivated successfully.5841client # [ 102.891190] systemd[1]: rs.service: Consumed 147ms CPU time over 8.652s wall clock time, 41.4M memory peak, 7.3K incoming IP traffic, 8.4K outgoing IP traffic.5842worker1 # [ 103.468663] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:31Z level=info event=signal_exit5843worker1 # [ 103.698206] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:31Z level=info event=scheduler_yield5844lb # [ 103.706720] niks3-server[1029]: 2026/09/21 12:09:32 INFO lead: released remote=192.168.1.3:382485845worker2 # [ 103.697932] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:32Z level=info event=scheduler_restarting addr=lb:500515846worker1 # [ 103.702541] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:31Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:585925847worker1 # [ 103.707312] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:31Z level=info event=worker_gone addr=192.168.1.4:500515848worker1 # [ 103.711282] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:31Z level=info event=scheduler_restarting addr=lb:500515849worker1 # [ 103.714304] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:31Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:585925850worker1 # [ 103.717537] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:31Z level=info event=worker_gone addr=192.168.1.3:500515851lb # [ 103.801247] niks3-server[1029]: 2026/09/21 12:09:32 INFO lead: acquired remote=192.168.1.4:410065852worker2 # [ 103.795088] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:32Z level=info event=scheduler_take_over5853worker2 # [ 103.898442] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:32Z level=info event=scheduler_connected addr=lb:500515854worker2 # [ 103.902441] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:32Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5855worker1 # [ 103.911981] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:32Z level=info event=scheduler_connected addr=lb:500515856worker1 # [ 103.915376] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:32Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5857worker2 # [ 104.103998] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:33Z level=info event=scheduler_connected addr=lb:500515858worker1 # [ 104.111179] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:32Z level=info event=scheduler_connected addr=lb:500515859worker2 # [ 104.107968] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5860worker1 # [ 104.115133] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5861worker2 # [ 104.308487] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:33Z level=info event=scheduler_connected addr=lb:500515862worker1 # [ 104.316171] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:32Z level=info event=scheduler_connected addr=lb:500515863worker2 # [ 104.312355] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5864worker1 # [ 104.320135] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5865worker2 # [ 104.513281] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:33Z level=info event=scheduler_connected addr=lb:500515866worker1 # [ 104.520908] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:32Z level=info event=scheduler_connected addr=lb:500515867worker2 # [ 104.517236] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5868worker1 # [ 104.524815] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5869worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.17 seconds)5870client: waiting for success: ! systemctl is-active rs5871worker1 # [ 104.703213] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:32Z level=info event=scheduler_restarting5872client: (finished: waiting for success: ! systemctl is-active rs, in 0.07 seconds)5873client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5874worker2 # [ 104.719489] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:33Z level=info event=scheduler_connected addr=lb:500515875worker2 # [ 104.723373] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5876worker1 # [ 104.739597] nix-grpc-daemon[1229]: ts=2026-09-21T12:09:32Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5877worker1 # [ 104.744643] nix-daemon[848]: reaped child process 1293, status = succeeded5878client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.04 seconds)5879client: must fail: journalctl -u rs | grep -q 'reconnecting'5880client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5881lb: must succeed: curl -sf localhost:9901/clusters5882lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5883worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5884worker1 # [ 104.913120] nix-grpc-daemon[1229]: E0000 00:00:1789992573.134308 1274 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5885worker1 # [ 104.918738] nix-grpc-daemon[1229]: E0000 00:00:1789992573.134628 1274 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5886worker2 # [ 104.924995] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:33Z level=info event=scheduler_connected addr=lb:500515887worker2 # [ 104.928262] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5888worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5889worker2 # [ 105.129288] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:34Z level=info event=scheduler_connected addr=lb:500515890worker2 # [ 105.133207] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5891worker2 # [ 105.334256] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:34Z level=info event=scheduler_connected addr=lb:500515892worker2 # [ 105.347145] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:34Z level=info event=oidc_jwks provider=mock keys=15893worker2 # [ 105.350954] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:34Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:549145894worker2 # [ 105.354777] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:34Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05895lb: must succeed: curl -sf localhost:9901/clusters5896worker1 # [ 106.001745] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5897worker1 # [ 106.004863] systemd[1]: Stopped Nix worker-protocol over gRPC.5898worker1 # [ 106.008364] systemd[1]: nix-grpc-daemon.service: Consumed 1.307s CPU time over 25.762s wall clock time, 35.6M memory peak, 49K incoming IP traffic, 96.9K outgoing IP traffic.5899worker1 # [ 106.023327] systemd[1]: Starting Nix worker-protocol over gRPC...5900lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5901worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5902worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5903worker1 # [ 106.212804] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:34Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5904worker1 # [ 106.214877] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:34Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5905worker1 # [ 106.218844] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:34Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15906worker1 # [ 106.227064] nix-grpc-daemon[1497]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5907worker1 # [ 106.230100] nix-grpc-daemon[1497]: E0000 00:00:1789992574.449135 1497 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5908worker1 # [ 106.233815] nix-grpc-daemon[1497]: E0000 00:00:1789992574.452386 1497 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5909worker1 # [ 106.237118] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:34Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5910worker1 # [ 106.239850] systemd[1]: Started Nix worker-protocol over gRPC.5911worker1 # [ 106.247071] nix-grpc-daemon[1497]: E0000 00:00:1789992574.468861 1539 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5912worker1 # [ 106.258696] nix-grpc-daemon[1497]: E0000 00:00:1789992574.481059 1539 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5913worker1 # [ 106.264054] nix-grpc-daemon[1497]: E0000 00:00:1789992574.483730 1539 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5914worker1 # [ 106.273507] nix-grpc-daemon[1497]: E0000 00:00:1789992574.495867 1539 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5915worker1 # [ 106.283323] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:34Z level=info event=scheduler_connected addr=lb:500515916worker2 # [ 106.281503] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:35Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:549145917worker2 # [ 106.285584] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:35Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05918lb: must succeed: curl -sf localhost:9901/clusters5919lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5920worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5921worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5922(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 13.10 seconds)5923subtest: scheduler restart mid-build: build finishes, no double build5924client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid5925client # [ 107.329271] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.5926client # Running as unit: mid.service; invocation ID: 469d5b338dc14086a804a4ba77a000e15927client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)5928lb # [ 107.382551] envoy[909]: conn peer=[2001:db8:1::1]:53270 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2973 tx=1428 ms=83885929lb # [ 107.386339] envoy[909]: conn peer=[2001:db8:1::3]:53406 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=5568 tx=1321 ms=242215930worker2 # [ 107.574323] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:36Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:503925931client # [ 107.595770] nix[1212]: don't know how to build these paths:5932client # [ 107.596807] nix[1212]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv5933worker2 # [ 107.591160] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:36Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=05934worker2 # [ 107.596439] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:36Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=15935worker2 # [ 107.635889] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503925936worker2 # [ 107.649369] nix-grpc-daemon[1209]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 42 ms (attempt 1/5)5937worker2 # [ 107.697877] nix-grpc-daemon[1209]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 149 ms (attempt 2/5)5938worker2 # [ 107.852839] nix-grpc-daemon[1209]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 263 ms (attempt 3/5)5939worker2 # [ 108.121931] nix-grpc-daemon[1209]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 509 ms (attempt 4/5)5940worker2 # [ 108.637521] nix-grpc-daemon[1209]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5941worker2 # [ 108.639770] nix-grpc-daemon[1209]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5942worker2 # [ 108.655769] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:503925943client # [ 108.680369] nix[1212]: copying 1 paths to 'lb:50051'...5944worker2 # [ 108.669611] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:503925945client # [ 108.685180] nix[1212]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...5946worker2 # [ 108.680403] nix-daemon[870]: accepted connection from pid 1209, user nix-grpc-daemon (trusted)5947worker2 # [ 108.695386] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:37Z level=info event=niks3_push_started pid=13385948worker2 # [ 108.907079] nix-grpc-daemon[1338]: time=2026-09-21T12:09:37.905Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5949worker2 # [ 109.084488] nix-daemon[870]: accepted connection from pid 1351, user nix-grpc-daemon (trusted)5950worker2 # [ 109.095794] nix-daemon[870]: reaped child process 1354, status = succeeded5951worker2 # [ 109.272823] nix-daemon[870]: accepted connection from pid 1356, user nix-grpc-daemon (trusted)5952worker2 # [ 109.284396] nix-daemon[870]: reaped child process 1359, status = succeeded5953lb # [ 109.313179] niks3-server[1029]: 2026/09/21 12:09:38 INFO Received uploads request method=POST path=/api/pending_closures5954worker2 # [ 109.313792] nix-grpc-daemon[1338]: time=2026-09-21T12:09:38.313Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5955worker2 # [ 109.316828] nix-grpc-daemon[1338]: time=2026-09-21T12:09:38.314Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"5956lb # [ 109.357551] niks3-server[1029]: 2026/09/21 12:09:38 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst5957lb # [ 109.368099] niks3-server[1029]: 2026/09/21 12:09:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5958lb # [ 109.372502] niks3-server[1029]: 2026/09/21 12:09:38 INFO Signed narinfos id=35 count=15959worker2 # [ 109.365723] nix-grpc-daemon[1338]: time=2026-09-21T12:09:38.364Z level=INFO msg="Uploading 1 narinfos"5960lb # [ 109.382595] niks3-server[1029]: 2026/09/21 12:09:38 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5961lb # [ 109.386677] niks3-server[1029]: 2026/09/21 12:09:38 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls5962lb # [ 109.390888] niks3-server[1029]: 2026/09/21 12:09:38 INFO Completed upload id=355963worker2 # [ 109.381510] nix-grpc-daemon[1338]: time=2026-09-21T12:09:38.380Z level=INFO msg="Upload complete. (470ms)"5964worker2 # [ 109.385373] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50392 duration_s=0 paths=1 nar_bytes_in=4245965worker2 # [ 109.391368] nix-grpc-daemon[1209]: ts=2026-09-21T12:09:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:503925966worker2 # [ 109.394356] nix-grpc-daemon[1338]: time=2026-09-21T12:09:38.389Z level=WARN msg="Failed to register uploaded object" key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo error="server returned 404: object is not pending upload\n"5967worker2 # [ 109.400321] nix-daemon[870]: accepted connection from pid 1209, user nix-grpc-daemon (trusted)5968worker2 # [ 109.406777] nix-daemon[870]: reaped child process 1336, status = succeeded5969worker2 # [ 109.410817] nix-daemon[870]: accepted connection from pid 1209, user nix-grpc-daemon (trusted)5970client # [ 109.424355] nix[1212]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...5971lb: must succeed: curl -sf localhost:9901/clusters5972lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5973lb: must succeed: curl -sf localhost:9901/clusters5974lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5975worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5976worker2 # [ 111.206775] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1209 (nix-grpc-daemon) on client request.5977worker2 # [ 111.210305] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5978lb # [ 111.225726] niks3-server[1029]: 2026/09/21 12:09:40 INFO lead: released remote=192.168.1.4:410065979client # [ 111.240601] nix[1212]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5980worker1 # [ 111.228737] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:39Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5981worker2 # [ 111.230493] systemd[1]: Starting scheduler-token.service...5982client # [ 111.245595] nix[1212]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5983worker2 # [ 111.240644] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5984worker2 # [ 111.242048] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5985lb # [ 111.250972] niks3-server[1029]: 2026/09/21 12:09:40 INFO lead: acquired remote=192.168.1.3:339865986worker2 # [ 111.245299] systemd[1]: nix-grpc-daemon.service: Consumed 455ms CPU time over 17.645s wall clock time, 33.3M memory peak, 17.4K incoming IP traffic, 37.3K outgoing IP traffic.5987worker1 # [ 111.249514] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:39Z level=info event=scheduler_take_over5988worker2 # [ 111.252839] nix-daemon[870]: reaped child process 1361, status = succeeded5989worker2 # [ 111.254161] nix-daemon[870]: reaped child process 1268, status = succeeded5990worker2 # [ 111.264251] nix-daemon[1363]: unexpected Nix daemon error: error: interrupted by the user5991worker2 # [ 111.269830] nix-daemon[870]: reaped child process 1363, status = failed with exit code 15992worker2 # [ 111.324112] systemd[1]: scheduler-token.service: Deactivated successfully.5993worker2 # [ 111.325720] systemd[1]: Finished scheduler-token.service.5994worker2 # [ 111.326581] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 93ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5995worker2 # [ 111.329299] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.5996worker2 # [ 111.333886] systemd[1]: Starting scheduler-token.service...5997worker2 # [ 111.405067] systemd[1]: scheduler-token.service: Deactivated successfully.5998worker2 # [ 111.407305] systemd[1]: Finished scheduler-token.service.5999worker2 # [ 111.408089] systemd[1]: scheduler-token.service: Consumed 16ms CPU time over 71ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6000worker2 # [ 111.412888] systemd[1]: Starting Nix worker-protocol over gRPC...6001worker1 # [ 111.429235] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:39Z level=info event=scheduler_connected addr=lb:500516002worker2 # [ 111.576852] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:40Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6003worker2 # [ 111.579133] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:40Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6004worker2 # [ 111.582758] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:40Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16005worker2 # [ 111.590065] nix-grpc-daemon[1406]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6006worker2 # [ 111.593073] nix-grpc-daemon[1406]: E0000 00:00:1789992580.589267 1406 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6007worker2 # [ 111.597155] nix-grpc-daemon[1406]: E0000 00:00:1789992580.592273 1406 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6008worker2 # [ 111.602116] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:40Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6009worker2 # [ 111.605746] systemd[1]: Started Nix worker-protocol over gRPC.6010worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.45 seconds)6011client: must succeed: sleep 5.06012worker2 # [ 111.612074] nix-grpc-daemon[1406]: E0000 00:00:1789992580.611048 1447 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6013worker1 # [ 111.624441] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:39Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6014worker2 # [ 111.629816] nix-grpc-daemon[1406]: E0000 00:00:1789992580.629161 1447 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6015worker2 # [ 111.634063] nix-grpc-daemon[1406]: E0000 00:00:1789992580.631539 1447 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6016worker2 # [ 111.646458] nix-grpc-daemon[1406]: E0000 00:00:1789992580.645773 1447 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6017worker2 # [ 111.656979] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:40Z level=info event=scheduler_connected addr=lb:500516018worker2 # [ 111.658291] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6019client # [ 111.747896] nix[1212]: scheduler at lb:50051: no healthy upstream, reconnecting6020worker1 # [ 111.825450] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:40Z level=info event=scheduler_connected addr=lb:500516021worker1 # [ 111.829952] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6022worker2 # [ 111.858728] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:40Z level=info event=scheduler_connected addr=lb:500516023worker2 # [ 111.862581] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6024worker1 # [ 112.030644] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:40Z level=info event=scheduler_connected addr=lb:500516025worker1 # [ 112.035095] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6026worker2 # [ 112.063699] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:41Z level=info event=scheduler_connected addr=lb:500516027worker2 # [ 112.067548] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6028worker1 # [ 112.236671] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:40Z level=info event=scheduler_connected addr=lb:500516029worker1 # [ 112.241503] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6030worker2 # [ 112.268677] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:41Z level=info event=scheduler_connected addr=lb:500516031worker2 # [ 112.272530] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6032worker1 # [ 112.442540] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:40Z level=info event=scheduler_connected addr=lb:500516033worker1 # [ 112.446963] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6034worker2 # [ 112.473710] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:41Z level=info event=scheduler_connected addr=lb:500516035worker2 # [ 112.477511] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6036worker1 # [ 112.647644] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:40Z level=info event=scheduler_connected addr=lb:500516037worker1 # [ 112.652169] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6038worker2 # [ 112.679695] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:41Z level=info event=scheduler_connected addr=lb:500516039worker2 # [ 112.683267] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6040client # [ 112.751687] nix[1212]: scheduler at lb:50051: no healthy upstream, reconnecting6041worker1 # [ 112.852907] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:41Z level=info event=scheduler_connected addr=lb:500516042worker1 # [ 112.857374] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6043worker2 # [ 112.884382] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:41Z level=info event=scheduler_connected addr=lb:500516044worker2 # [ 112.888433] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6045worker1 # [ 113.058085] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:41Z level=info event=scheduler_connected addr=lb:500516046worker1 # [ 113.069597] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:41Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:586506047worker1 # [ 113.074865] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:41Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06048worker2 # [ 113.089366] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:42Z level=info event=scheduler_connected addr=lb:500516049worker1 # [ 113.101154] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:41Z level=info event=oidc_jwks provider=mock keys=16050worker1 # [ 113.105087] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:41Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:586506051worker1 # [ 113.108934] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:41Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06052worker1 # [ 114.754219] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:42Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06053worker1 # [ 114.758283] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:42Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16054worker1 # [ 114.804644] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:586606055worker1 # [ 114.815728] nix-daemon[848]: accepted connection from pid 1497, user nix-grpc-daemon (trusted)6056worker1 # [ 114.833414] nix-grpc-daemon[1497]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 79 ms (attempt 1/5)6057worker1 # [ 114.918240] nix-grpc-daemon[1497]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 159 ms (attempt 2/5)6058worker1 # [ 115.082600] nix-grpc-daemon[1497]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 306 ms (attempt 3/5)6059worker1 # [ 115.391403] nix-grpc-daemon[1497]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 222 ms (attempt 4/5)6060worker1 # [ 115.613753] nix-grpc-daemon[1497]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6061worker1 # [ 115.615036] nix-grpc-daemon[1497]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6062worker1 # [ 115.626201] nix-grpc-daemon[1497]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' from 'http://lb:5751'...6063worker1 # [ 115.641461] nix-daemon[848]: accepted connection from pid 1497, user nix-grpc-daemon (trusted)6064worker1 # [ 115.648544] nix-daemon[848]: accepted connection from pid 1497, user nix-grpc-daemon (trusted)6065client # [ 115.660625] nix[1212]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6066client: (finished: must succeed: sleep 5.0, in 5.05 seconds)6067client: waiting for success: ! systemctl is-active mid6068worker1 # [ 117.063260] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:45Z level=info event=niks3_push_started pid=16056069worker1 # [ 117.089673] nix-daemon[848]: reaped child process 1580, status = succeeded6070worker1 # [ 117.286578] nix-grpc-daemon[1605]: time=2026-09-21T12:09:45.508Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6071lb # [ 117.383214] envoy[909]: conn peer=[2001:db8:1::4]:37386 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=7821 tx=618 ms=173946072worker1 # [ 117.462079] nix-daemon[848]: accepted connection from pid 1618, user nix-grpc-daemon (trusted)6073worker1 # [ 117.473436] nix-daemon[848]: reaped child process 1621, status = succeeded6074lb # [ 117.496420] niks3-server[1029]: 2026/09/21 12:09:46 INFO Received uploads request method=POST path=/api/pending_closures6075worker1 # [ 117.499768] nix-grpc-daemon[1605]: time=2026-09-21T12:09:45.722Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6076worker1 # [ 117.501306] nix-grpc-daemon[1605]: time=2026-09-21T12:09:45.723Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6077lb # [ 117.541117] niks3-server[1029]: 2026/09/21 12:09:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign6078lb # [ 117.549578] niks3-server[1029]: 2026/09/21 12:09:46 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6079worker1 # [ 117.554269] nix-grpc-daemon[1605]: time=2026-09-21T12:09:45.776Z level=INFO msg="Uploading 1 narinfos"6080lb # [ 117.559627] niks3-server[1029]: 2026/09/21 12:09:46 INFO Signed narinfos id=36 count=16081lb # [ 117.567597] niks3-server[1029]: 2026/09/21 12:09:46 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6082lb # [ 117.580119] niks3-server[1029]: 2026/09/21 12:09:46 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete6083lb # [ 117.587022] niks3-server[1029]: 2026/09/21 12:09:46 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6084lb # [ 117.590267] niks3-server[1029]: 2026/09/21 12:09:46 INFO Completed upload id=366085lb # [ 117.592033] niks3-server[1029]: 2026/09/21 12:09:46 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6086worker1 # [ 117.585250] nix-grpc-daemon[1605]: time=2026-09-21T12:09:45.807Z level=INFO msg="Upload complete. (294ms)"6087worker1 # [ 117.593362] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58660 duration_s=2 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16088worker1 # [ 117.607701] nix-daemon[848]: reaped child process 1578, status = succeeded6089client # [ 117.653687] systemd[1]: mid.service: Deactivated successfully.6090client # [ 117.655799] systemd[1]: mid.service: Consumed 166ms CPU time over 10.324s wall clock time, 41.4M memory peak, 9.1K incoming IP traffic, 11.2K outgoing IP traffic.6091client: (finished: waiting for success: ! systemctl is-active mid, in 1.13 seconds)6092client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6093client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds)6094(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.85 seconds)6095subtest: interrupting the client stops the build on the worker6096client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6097client # [ 118.148978] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6098client # Running as unit: intr.service; invocation ID: c2f65988ecec4f0085277b1e817baefc6099client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.04 seconds)6100worker1 # [ 118.369633] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:586606101client # [ 118.387102] nix[1248]: don't know how to build these paths:6102client # [ 118.388099] nix[1248]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6103worker1 # [ 118.384193] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:46Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06104worker1 # [ 118.388521] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:46Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16105worker2 # [ 118.433814] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402966106worker2 # [ 118.445613] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6107worker2 # [ 118.463226] nix-grpc-daemon[1406]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 6 ms (attempt 1/5)6108worker2 # [ 118.474870] nix-grpc-daemon[1406]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 121 ms (attempt 2/5)6109worker2 # [ 118.601822] nix-grpc-daemon[1406]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 282 ms (attempt 3/5)6110worker2 # [ 118.889754] nix-grpc-daemon[1406]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 181 ms (attempt 4/5)6111worker2 # [ 119.076857] nix-grpc-daemon[1406]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6112worker2 # [ 119.080710] nix-grpc-daemon[1406]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6113worker2 # [ 119.098259] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:402966114client # [ 119.124192] nix[1248]: copying 1 paths to 'lb:50051'...6115client # [ 119.128158] nix[1248]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6116worker2 # [ 119.113799] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:402966117worker2 # [ 119.122283] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6118worker2 # [ 119.142834] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:48Z level=info event=niks3_push_started pid=14786119worker2 # [ 119.360474] nix-grpc-daemon[1478]: time=2026-09-21T12:09:48.358Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6120worker2 # [ 119.523866] nix-daemon[870]: accepted connection from pid 1495, user nix-grpc-daemon (trusted)6121worker2 # [ 119.536284] nix-daemon[870]: reaped child process 1498, status = succeeded6122worker2 # [ 119.708474] nix-daemon[870]: accepted connection from pid 1500, user nix-grpc-daemon (trusted)6123worker2 # [ 119.720302] nix-daemon[870]: reaped child process 1503, status = succeeded6124lb # [ 119.748662] niks3-server[1029]: 2026/09/21 12:09:48 INFO Received uploads request method=POST path=/api/pending_closures6125worker2 # [ 119.748830] nix-grpc-daemon[1478]: time=2026-09-21T12:09:48.748Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6126worker2 # [ 119.751886] nix-grpc-daemon[1478]: time=2026-09-21T12:09:48.749Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6127lb # [ 119.781931] niks3-server[1029]: 2026/09/21 12:09:48 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6128lb # [ 119.791952] niks3-server[1029]: 2026/09/21 12:09:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6129lb # [ 119.796936] niks3-server[1029]: 2026/09/21 12:09:48 INFO Signed narinfos id=37 count=16130worker2 # [ 119.792483] nix-grpc-daemon[1478]: time=2026-09-21T12:09:48.791Z level=INFO msg="Uploading 1 narinfos"6131lb # [ 119.805455] niks3-server[1029]: 2026/09/21 12:09:48 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6132lb # [ 119.817594] niks3-server[1029]: 2026/09/21 12:09:48 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6133lb # [ 119.821065] niks3-server[1029]: 2026/09/21 12:09:48 INFO Completed upload id=376134worker2 # [ 119.811739] nix-grpc-daemon[1478]: time=2026-09-21T12:09:48.811Z level=INFO msg="Upload complete. (448ms)"6135lb # [ 119.823625] niks3-server[1029]: 2026/09/21 12:09:48 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6136worker2 # [ 119.815556] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=0 paths=1 nar_bytes_in=4246137worker2 # [ 119.821612] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402966138worker2 # [ 119.828506] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6139worker2 # [ 119.833877] nix-daemon[870]: reaped child process 1476, status = succeeded6140worker2 # [ 119.839469] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6141client # [ 119.853793] nix[1248]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6142client: must succeed: systemctl kill -s INT intr6143client # [ 121.737745] systemd[1]: intr.service: Sent signal SIGINT to main process 1248 (nix) on client request.6144worker2 # [ 121.731212] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:50Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6145client # [ 121.747306] nix[1248]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6146client: (finished: must succeed: systemctl kill -s INT intr, in 0.06 seconds)6147client # [ 121.753799] nix[1248]: error: interrupted by the user6148client # [ 121.788845] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6149client # [ 121.790305] systemd[1]: intr.service: Failed with result 'exit-code'.6150client # [ 121.792376] systemd[1]: intr.service: Consumed 131ms CPU time over 3.642s wall clock time, 43.4M memory peak, 6.4K incoming IP traffic, 8.3K outgoing IP traffic.6151worker2 # [ 121.861569] nix-daemon[870]: reaped child process 1505, status = succeeded6152worker2 # [ 121.864325] nix-daemon[1507]: unexpected Nix daemon error: error: interrupted by the user6153worker2 # [ 121.869896] nix-daemon[870]: reaped child process 1507, status = failed with exit code 16154(finished: subtest: interrupting the client stops the build on the worker, in 3.77 seconds)6155subtest: a deploy mid-build drains: the build finishes, the new generation takes over6156client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6157client # [ 121.934843] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6158client # Running as unit: sw.service; invocation ID: ddf77193ae5e4001a8d1d4dff271b3746159client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.06 seconds)6160worker2 # [ 122.151489] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:402966161client # [ 122.173311] nix[1269]: don't know how to build these paths:6162client # [ 122.174277] nix[1269]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6163worker1 # [ 122.172573] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:50Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06164worker1 # [ 122.176522] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:50Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16165worker1 # [ 122.214869] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:586606166worker1 # [ 122.227242] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:586606167client # [ 122.248356] nix[1269]: copying 1 paths to 'lb:50051'...6168worker1 # [ 122.238799] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:586606169client # [ 122.250623] nix[1269]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6170worker1 # [ 122.251777] nix-daemon[848]: accepted connection from pid 1497, user nix-grpc-daemon (trusted)6171worker1 # [ 122.444501] nix-daemon[848]: accepted connection from pid 1653, user nix-grpc-daemon (trusted)6172worker1 # [ 122.457069] nix-daemon[848]: reaped child process 1656, status = succeeded6173worker1 # [ 122.630517] nix-daemon[848]: accepted connection from pid 1658, user nix-grpc-daemon (trusted)6174worker1 # [ 122.642338] nix-daemon[848]: reaped child process 1661, status = succeeded6175lb # [ 122.665481] niks3-server[1029]: 2026/09/21 12:09:51 INFO Received uploads request method=POST path=/api/pending_closures6176worker1 # [ 122.670890] nix-grpc-daemon[1605]: time=2026-09-21T12:09:50.893Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6177worker1 # [ 122.672988] nix-grpc-daemon[1605]: time=2026-09-21T12:09:50.894Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6178lb # [ 122.696891] niks3-server[1029]: 2026/09/21 12:09:51 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6179lb # [ 122.705239] niks3-server[1029]: 2026/09/21 12:09:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6180lb # [ 122.711872] niks3-server[1029]: 2026/09/21 12:09:51 INFO Signed narinfos id=38 count=16181worker1 # [ 122.709902] nix-grpc-daemon[1605]: time=2026-09-21T12:09:50.932Z level=INFO msg="Uploading 1 narinfos"6182lb # [ 122.719604] niks3-server[1029]: 2026/09/21 12:09:51 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6183lb # [ 122.731044] niks3-server[1029]: 2026/09/21 12:09:51 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6184lb # [ 122.736152] niks3-server[1029]: 2026/09/21 12:09:51 INFO Completed upload id=386185worker1 # [ 122.733362] nix-grpc-daemon[1605]: time=2026-09-21T12:09:50.955Z level=INFO msg="Upload complete. (469ms)"6186lb # [ 122.739108] niks3-server[1029]: 2026/09/21 12:09:51 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6187worker1 # [ 122.737161] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58660 duration_s=0 paths=1 nar_bytes_in=4166188worker1 # [ 122.742089] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:586606189worker1 # [ 122.752109] nix-daemon[848]: accepted connection from pid 1497, user nix-grpc-daemon (trusted)6190worker1 # [ 122.756393] nix-daemon[848]: reaped child process 1651, status = succeeded6191worker1 # [ 122.762240] nix-daemon[848]: accepted connection from pid 1497, user nix-grpc-daemon (trusted)6192client # [ 122.771623] nix[1269]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6193worker1: must succeed: systemctl reload nix-grpc-daemon.service6194worker1 # [ 124.528627] systemd[1]: Reloading Nix worker-protocol over gRPC...6195worker1 # [ 124.573733] systemd[1]: Reloaded Nix worker-protocol over gRPC.6196worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.09 seconds)6197worker1: must succeed: pgrep -f 'read -t [9]0 x'6198worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)6199client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26200worker1 # [ 124.888346] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:586606201client # don't know how to build these paths:6202client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6203worker2 # [ 124.892518] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:402966204worker1 # [ 124.918062] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:53Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06205worker1 # [ 124.919806] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:53Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16206worker2 # [ 124.964775] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402966207worker2 # [ 124.977664] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:402966208client # copying 1 paths to 'lb:50051'...6209worker2 # [ 124.990797] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:402966210client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6211worker2 # [ 125.011495] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6212worker2 # [ 125.223148] nix-daemon[870]: accepted connection from pid 1554, user nix-grpc-daemon (trusted)6213worker2 # [ 125.235085] nix-daemon[870]: reaped child process 1557, status = succeeded6214worker1 # [ 125.244969] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:53Z level=info event=draining6215worker2 # [ 125.407201] nix-daemon[870]: accepted connection from pid 1559, user nix-grpc-daemon (trusted)6216worker2 # [ 125.418863] nix-daemon[870]: reaped child process 1562, status = succeeded6217lb # [ 125.447926] niks3-server[1029]: 2026/09/21 12:09:54 INFO Received uploads request method=POST path=/api/pending_closures6218worker2 # [ 125.449993] nix-grpc-daemon[1478]: time=2026-09-21T12:09:54.449Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6219worker2 # [ 125.451577] nix-grpc-daemon[1478]: time=2026-09-21T12:09:54.450Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6220lb # [ 125.479666] niks3-server[1029]: 2026/09/21 12:09:54 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6221lb # [ 125.489285] niks3-server[1029]: 2026/09/21 12:09:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6222lb # [ 125.494324] niks3-server[1029]: 2026/09/21 12:09:54 INFO Signed narinfos id=39 count=16223worker2 # [ 125.487807] nix-grpc-daemon[1478]: time=2026-09-21T12:09:54.486Z level=INFO msg="Uploading 1 narinfos"6224lb # [ 125.499981] niks3-server[1029]: 2026/09/21 12:09:54 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6225lb # [ 125.512674] niks3-server[1029]: 2026/09/21 12:09:54 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6226lb # [ 125.518879] niks3-server[1029]: 2026/09/21 12:09:54 INFO Completed upload id=396227worker2 # [ 125.511930] nix-grpc-daemon[1478]: time=2026-09-21T12:09:54.511Z level=INFO msg="Upload complete. (488ms)"6228lb # [ 125.522142] niks3-server[1029]: 2026/09/21 12:09:54 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6229worker2 # [ 125.515740] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=0 paths=1 nar_bytes_in=5286230worker2 # [ 125.522853] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402966231worker2 # [ 125.531888] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6232worker2 # [ 125.539429] nix-daemon[870]: reaped child process 1552, status = succeeded6233worker2 # [ 125.544503] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6234client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6235client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6236client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6237client # farm-a> LOG-farm-a-during-drain6238worker2 # [ 127.169587] nix-daemon[870]: reaped child process 1566, status = succeeded6239worker2 # [ 127.358058] nix-daemon[870]: accepted connection from pid 1582, user nix-grpc-daemon (trusted)6240worker2 # [ 127.368681] nix-daemon[870]: reaped child process 1585, status = succeeded6241lb # [ 127.383478] envoy[909]: conn peer=[2001:db8:1::1]:49304 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4291 tx=1998 ms=100356242lb # [ 127.387663] envoy[909]: conn peer=[2001:db8:1::1]:42502 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2980 tx=829 ms=33856243lb # [ 127.397436] niks3-server[1029]: 2026/09/21 12:09:56 INFO Received uploads request method=POST path=/api/pending_closures6244worker2 # [ 127.399336] nix-grpc-daemon[1478]: time=2026-09-21T12:09:56.398Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6245worker2 # [ 127.403384] nix-grpc-daemon[1478]: time=2026-09-21T12:09:56.400Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6246lb # [ 127.432170] niks3-server[1029]: 2026/09/21 12:09:56 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6247lb # [ 127.451283] niks3-server[1029]: 2026/09/21 12:09:56 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6248lb # [ 127.458627] niks3-server[1029]: 2026/09/21 12:09:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6249lb # [ 127.463276] niks3-server[1029]: 2026/09/21 12:09:56 INFO Signed narinfos id=40 count=16250worker2 # [ 127.457059] nix-grpc-daemon[1478]: time=2026-09-21T12:09:56.455Z level=INFO msg="Uploading 1 narinfos"6251lb # [ 127.471577] niks3-server[1029]: 2026/09/21 12:09:56 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6252lb # [ 127.480849] niks3-server[1029]: 2026/09/21 12:09:56 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6253lb # [ 127.485791] niks3-server[1029]: 2026/09/21 12:09:56 INFO Completed upload id=406254lb # [ 127.488557] niks3-server[1029]: 2026/09/21 12:09:56 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6255worker2 # [ 127.477775] nix-grpc-daemon[1478]: time=2026-09-21T12:09:56.477Z level=INFO msg="Upload complete. (325ms)"6256worker1 # [ 127.491815] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:55Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06257worker2 # [ 127.484612] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16258worker1 # [ 127.494279] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:55Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16259worker2 # [ 127.497581] nix-daemon[870]: reaped child process 1564, status = succeeded6260worker2 # [ 127.533949] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402966261worker2 # [ 127.546445] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:402966262worker2 # [ 127.560884] nix-grpc-daemon[1478]: time=2026-09-21T12:09:56.560Z level=INFO msg="All 1 paths already cached"6263client # copying 1 paths to 'lb:50051'...6264worker2 # [ 127.574919] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:402966265client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6266worker2 # [ 127.594328] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6267worker2 # [ 127.809336] nix-daemon[870]: accepted connection from pid 1589, user nix-grpc-daemon (trusted)6268worker2 # [ 127.821903] nix-daemon[870]: reaped child process 1592, status = succeeded6269worker2 # [ 127.996537] nix-daemon[870]: accepted connection from pid 1594, user nix-grpc-daemon (trusted)6270worker2 # [ 128.008136] nix-daemon[870]: reaped child process 1597, status = succeeded6271lb # [ 128.035810] niks3-server[1029]: 2026/09/21 12:09:57 INFO Received uploads request method=POST path=/api/pending_closures6272worker2 # [ 128.035525] nix-grpc-daemon[1478]: time=2026-09-21T12:09:57.034Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6273worker2 # [ 128.037026] nix-grpc-daemon[1478]: time=2026-09-21T12:09:57.036Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6274lb # [ 128.064516] niks3-server[1029]: 2026/09/21 12:09:57 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6275lb # [ 128.076594] niks3-server[1029]: 2026/09/21 12:09:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6276lb # [ 128.082068] niks3-server[1029]: 2026/09/21 12:09:57 INFO Signed narinfos id=41 count=16277worker2 # [ 128.075768] nix-grpc-daemon[1478]: time=2026-09-21T12:09:57.074Z level=INFO msg="Uploading 1 narinfos"6278lb # [ 128.089325] niks3-server[1029]: 2026/09/21 12:09:57 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6279lb # [ 128.099743] niks3-server[1029]: 2026/09/21 12:09:57 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6280lb # [ 128.106149] niks3-server[1029]: 2026/09/21 12:09:57 INFO Completed upload id=416281worker2 # [ 128.096657] nix-grpc-daemon[1478]: time=2026-09-21T12:09:57.096Z level=INFO msg="Upload complete. (489ms)"6282lb # [ 128.109900] niks3-server[1029]: 2026/09/21 12:09:57 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6283worker2 # [ 128.099551] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=0 paths=1 nar_bytes_in=6406284worker2 # [ 128.105259] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402966285worker2 # [ 128.113846] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6286worker2 # [ 128.120697] nix-daemon[870]: reaped child process 1587, status = succeeded6287worker2 # [ 128.125557] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6288client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6289client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6290client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6291client # farm-b> LOG-farm-b-during-drain6292worker2 # [ 129.149566] nix-daemon[870]: reaped child process 1601, status = succeeded6293worker2 # [ 129.333376] nix-daemon[870]: accepted connection from pid 1617, user nix-grpc-daemon (trusted)6294worker2 # [ 129.345309] nix-daemon[870]: reaped child process 1620, status = succeeded6295lb # [ 129.373726] niks3-server[1029]: 2026/09/21 12:09:58 INFO Received uploads request method=POST path=/api/pending_closures6296worker2 # [ 129.372897] nix-grpc-daemon[1478]: time=2026-09-21T12:09:58.372Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6297worker2 # [ 129.374346] nix-grpc-daemon[1478]: time=2026-09-21T12:09:58.373Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6298lb # [ 129.404672] niks3-server[1029]: 2026/09/21 12:09:58 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6299lb # [ 129.419259] niks3-server[1029]: 2026/09/21 12:09:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6300worker2 # [ 129.418331] nix-grpc-daemon[1478]: time=2026-09-21T12:09:58.417Z level=INFO msg="Uploading 1 narinfos"6301lb # [ 129.428715] niks3-server[1029]: 2026/09/21 12:09:58 INFO Signed narinfos id=42 count=16302lb # [ 129.434665] niks3-server[1029]: 2026/09/21 12:09:58 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6303lb # [ 129.439704] niks3-server[1029]: 2026/09/21 12:09:58 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6304lb # [ 129.448983] niks3-server[1029]: 2026/09/21 12:09:58 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6305lb # [ 129.453219] niks3-server[1029]: 2026/09/21 12:09:58 INFO Completed upload id=426306worker2 # [ 129.444083] nix-grpc-daemon[1478]: time=2026-09-21T12:09:58.443Z level=INFO msg="Upload complete. (311ms)"6307lb # [ 129.455853] niks3-server[1029]: 2026/09/21 12:09:58 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6308worker2 # [ 129.449491] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16309worker1 # [ 129.457550] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:57Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06310worker1 # [ 129.460207] nix-grpc-daemon[1497]: ts=2026-09-21T12:09:57Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16311worker2 # [ 129.463541] nix-daemon[870]: reaped child process 1599, status = succeeded6312worker2 # [ 129.498226] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402966313worker2 # [ 129.510240] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:402966314worker2 # [ 129.523719] nix-grpc-daemon[1478]: time=2026-09-21T12:09:58.523Z level=INFO msg="All 2 paths already cached"6315client # copying 1 paths to 'lb:50051'...6316client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6317worker2 # [ 129.532903] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:402966318worker2 # [ 129.543156] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6319worker2 # [ 129.733950] nix-daemon[870]: accepted connection from pid 1624, user nix-grpc-daemon (trusted)6320worker2 # [ 129.745956] nix-daemon[870]: reaped child process 1627, status = succeeded6321worker2 # [ 129.915496] nix-daemon[870]: accepted connection from pid 1629, user nix-grpc-daemon (trusted)6322worker2 # [ 129.927709] nix-daemon[870]: reaped child process 1632, status = succeeded6323lb # [ 129.955778] niks3-server[1029]: 2026/09/21 12:09:59 INFO Received uploads request method=POST path=/api/pending_closures6324worker2 # [ 129.953707] nix-grpc-daemon[1478]: time=2026-09-21T12:09:58.953Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6325worker2 # [ 129.955575] nix-grpc-daemon[1478]: time=2026-09-21T12:09:58.954Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6326lb # [ 129.982075] niks3-server[1029]: 2026/09/21 12:09:59 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6327lb # [ 129.994113] niks3-server[1029]: 2026/09/21 12:09:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6328worker2 # [ 129.989161] nix-grpc-daemon[1478]: time=2026-09-21T12:09:58.988Z level=INFO msg="Uploading 1 narinfos"6329lb # [ 129.998457] niks3-server[1029]: 2026/09/21 12:09:59 INFO Signed narinfos id=43 count=16330lb # [ 130.005321] niks3-server[1029]: 2026/09/21 12:09:59 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6331lb # [ 130.016082] niks3-server[1029]: 2026/09/21 12:09:59 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6332lb # [ 130.022534] niks3-server[1029]: 2026/09/21 12:09:59 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6333lb # [ 130.026439] niks3-server[1029]: 2026/09/21 12:09:59 INFO Completed upload id=436334worker2 # [ 130.019322] nix-grpc-daemon[1478]: time=2026-09-21T12:09:59.018Z level=INFO msg="Upload complete. (464ms)"6335worker2 # [ 130.023347] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=0 paths=1 nar_bytes_in=7686336worker2 # [ 130.028262] nix-grpc-daemon[1406]: ts=2026-09-21T12:09:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402966337worker2 # [ 130.038105] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6338worker2 # [ 130.047632] nix-daemon[870]: reaped child process 1622, status = succeeded6339worker2 # [ 130.050918] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6340client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6341client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6342client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6343client # farm-top> LOG-farm-top-during-drain6344worker2 # [ 130.864052] nix-daemon[870]: reaped child process 1636, status = succeeded6345worker2 # [ 131.031514] nix-daemon[870]: accepted connection from pid 1652, user nix-grpc-daemon (trusted)6346worker2 # [ 131.042207] nix-daemon[870]: reaped child process 1655, status = succeeded6347lb # [ 131.070736] niks3-server[1029]: 2026/09/21 12:10:00 INFO Received uploads request method=POST path=/api/pending_closures6348worker2 # [ 131.072833] nix-grpc-daemon[1478]: time=2026-09-21T12:10:00.072Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6349worker2 # [ 131.074347] nix-grpc-daemon[1478]: time=2026-09-21T12:10:00.073Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6350lb # [ 131.111299] niks3-server[1029]: 2026/09/21 12:10:00 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6351lb # [ 131.117062] niks3-server[1029]: 2026/09/21 12:10:00 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6352lb # [ 131.130073] niks3-server[1029]: 2026/09/21 12:10:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6353worker2 # [ 131.125404] nix-grpc-daemon[1478]: time=2026-09-21T12:10:00.124Z level=INFO msg="Uploading 1 narinfos"6354lb # [ 131.135092] niks3-server[1029]: 2026/09/21 12:10:00 INFO Signed narinfos id=44 count=16355lb # [ 131.140700] niks3-server[1029]: 2026/09/21 12:10:00 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6356lb # [ 131.149462] niks3-server[1029]: 2026/09/21 12:10:00 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6357lb # [ 131.154831] niks3-server[1029]: 2026/09/21 12:10:00 INFO Completed upload id=446358worker2 # [ 131.145594] nix-grpc-daemon[1478]: time=2026-09-21T12:10:00.144Z level=INFO msg="Upload complete. (288ms)"6359lb # [ 131.157556] niks3-server[1029]: 2026/09/21 12:10:00 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6360worker2 # [ 131.152598] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16361worker2 # [ 131.165084] nix-daemon[870]: reaped child process 1634, status = succeeded6362client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 6.58 seconds)6363client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6364client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.22 seconds)6365worker1: must succeed: pkill -f 'read -t [9]0 x'6366worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6367client: waiting for success: ! systemctl is-active sw6368worker1 # [ 131.590706] nix-daemon[848]: reaped child process 1665, status = succeeded6369worker1 # [ 131.760528] nix-daemon[848]: accepted connection from pid 1713, user nix-grpc-daemon (trusted)6370worker1 # [ 131.772252] nix-daemon[848]: reaped child process 1716, status = succeeded6371lb # [ 131.795707] niks3-server[1029]: 2026/09/21 12:10:00 INFO Received uploads request method=POST path=/api/pending_closures6372worker1 # [ 131.798524] nix-grpc-daemon[1605]: time=2026-09-21T12:10:00.020Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6373worker1 # [ 131.800047] nix-grpc-daemon[1605]: time=2026-09-21T12:10:00.022Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6374lb # [ 131.836701] niks3-server[1029]: 2026/09/21 12:10:00 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6375lb # [ 131.841817] niks3-server[1029]: 2026/09/21 12:10:00 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6376lb # [ 131.848451] niks3-server[1029]: 2026/09/21 12:10:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6377lb # [ 131.852706] niks3-server[1029]: 2026/09/21 12:10:00 INFO Signed narinfos id=45 count=16378worker1 # [ 131.851095] nix-grpc-daemon[1605]: time=2026-09-21T12:10:00.072Z level=INFO msg="Uploading 1 narinfos"6379lb # [ 131.862321] niks3-server[1029]: 2026/09/21 12:10:00 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6380lb # [ 131.868546] niks3-server[1029]: 2026/09/21 12:10:00 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6381lb # [ 131.874840] niks3-server[1029]: 2026/09/21 12:10:01 INFO Completed upload id=456382lb # [ 131.878895] niks3-server[1029]: 2026/09/21 12:10:01 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6383worker1 # [ 131.869817] nix-grpc-daemon[1605]: time=2026-09-21T12:10:00.092Z level=INFO msg="Upload complete. (293ms)"6384worker1 # [ 131.877649] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58660 duration_s=9 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16385worker1 # [ 131.890549] nix-daemon[848]: reaped child process 1663, status = succeeded6386client # [ 131.929503] systemd[1]: sw.service: Deactivated successfully.6387client # [ 131.933763] systemd[1]: sw.service: Consumed 144ms CPU time over 9.996s wall clock time, 43.4M memory peak, 7K incoming IP traffic, 8.5K outgoing IP traffic.6388worker1 # [ 132.247166] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:00Z level=info event=signal_exit6389worker1 # [ 132.470771] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:00Z level=info event=scheduler_yield6390lb # [ 132.479649] niks3-server[1029]: 2026/09/21 12:10:01 INFO lead: released remote=192.168.1.3:339866391worker2 # [ 132.469842] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:01Z level=info event=scheduler_restarting addr=lb:500516392worker1 # [ 132.475114] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:00Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:586506393worker1 # [ 132.479481] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:00Z level=info event=worker_gone addr=192.168.1.4:500516394worker1 # [ 132.483616] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:00Z level=info event=scheduler_restarting addr=lb:500516395worker1 # [ 132.486565] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:00Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:586506396worker1 # [ 132.489783] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:00Z level=info event=worker_gone addr=192.168.1.3:500516397client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)6398client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6399client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds)6400worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6401worker2 # [ 132.670127] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:01Z level=info event=scheduler_connected addr=lb:500516402worker2 # [ 132.674171] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:01Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6403worker1 # [ 132.680277] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:00Z level=info event=scheduler_connected addr=lb:500516404worker1 # [ 132.684612] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:00Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6405worker2 # [ 132.875431] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:01Z level=info event=scheduler_connected addr=lb:500516406worker2 # [ 132.879421] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:01Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6407worker1 # [ 132.886778] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:01Z level=info event=scheduler_connected addr=lb:500516408worker1 # [ 132.890586] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:01Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6409worker2 # [ 133.080626] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500516410worker2 # [ 133.084596] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6411worker1 # [ 133.091418] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:01Z level=info event=scheduler_connected addr=lb:500516412worker1 # [ 133.095326] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6413worker2 # [ 133.285791] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500516414worker2 # [ 133.289751] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6415worker1 # [ 133.296144] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:01Z level=info event=scheduler_connected addr=lb:500516416worker1 # [ 133.300337] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6417worker1 # [ 133.475164] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:01Z level=info event=scheduler_restarting6418worker2 # [ 133.491700] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500516419worker1 # [ 133.499250] nix-daemon[848]: reaped child process 1571, status = succeeded6420worker2 # [ 133.495584] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6421worker1 # [ 133.513412] nix-grpc-daemon[1497]: ts=2026-09-21T12:10:01Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6422worker2 # [ 133.696572] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500516423worker2 # [ 133.700515] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6424worker2 # [ 133.901460] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500516425worker2 # [ 133.905317] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6426worker1 # [ 134.047234] nix-grpc-daemon[1497]: E0000 00:00:1789992602.268644 1542 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true6427worker1 # [ 134.052163] nix-grpc-daemon[1497]: E0000 00:00:1789992602.268776 1542 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6428worker2 # [ 134.106733] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:03Z level=info event=scheduler_connected addr=lb:500516429worker2 # [ 134.110691] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6430worker1 # [ 134.278497] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6431worker1 # [ 134.282353] systemd[1]: nix-grpc-daemon.service: Consumed 738ms CPU time over 28.254s wall clock time, 33.2M memory peak, 27.7K incoming IP traffic, 37.5K outgoing IP traffic.6432worker1 # [ 134.298568] systemd[1]: Starting Nix worker-protocol over gRPC...6433worker2 # [ 134.311672] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:03Z level=info event=scheduler_connected addr=lb:500516434worker2 # [ 134.315585] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6435worker1 # [ 134.487800] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:02Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6436worker1 # [ 134.489590] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:02Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6437worker1 # [ 134.493452] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:02Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16438worker1 # [ 134.501062] nix-grpc-daemon[1730]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6439worker1 # [ 134.502455] nix-grpc-daemon[1730]: E0000 00:00:1789992602.723145 1730 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6440worker1 # [ 134.508486] nix-grpc-daemon[1730]: E0000 00:00:1789992602.728337 1730 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6441worker1 # [ 134.511730] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:02Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6442worker1 # [ 134.513931] systemd[1]: Started Nix worker-protocol over gRPC.6443worker1 # [ 134.518363] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:02Z level=info event=scheduler_take_over6444lb # [ 134.524507] niks3-server[1029]: 2026/09/21 12:10:03 INFO lead: acquired remote=192.168.1.3:344766445worker1 # [ 134.522333] nix-grpc-daemon[1730]: E0000 00:00:1789992602.743905 1771 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6446worker2 # [ 134.517451] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:03Z level=info event=scheduler_connected addr=lb:500516447worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 1.91 seconds)6448worker2 # [ 134.521312] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6449lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6450lb: must succeed: curl -sf localhost:9901/clusters6451worker1 # [ 134.534245] nix-grpc-daemon[1730]: E0000 00:00:1789992602.756599 1771 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6452worker1 # [ 134.538122] nix-grpc-daemon[1730]: E0000 00:00:1789992602.758765 1771 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6453worker1 # [ 134.548316] nix-grpc-daemon[1730]: E0000 00:00:1789992602.770680 1771 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6454worker1 # [ 134.560597] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500516455worker1 # [ 134.562093] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6456lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6457worker2 # [ 134.722290] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:03Z level=info event=scheduler_connected addr=lb:500516458worker2 # [ 134.726275] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6459worker1 # [ 134.761959] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500516460worker1 # [ 134.766086] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6461worker2 # [ 134.927268] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:03Z level=info event=scheduler_connected addr=lb:500516462worker2 # [ 134.931166] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6463worker1 # [ 134.966947] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:03Z level=info event=scheduler_connected addr=lb:500516464worker1 # [ 134.970938] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6465worker2 # [ 135.132309] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:04Z level=info event=scheduler_connected addr=lb:500516466worker2 # [ 135.136354] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6467worker1 # [ 135.171790] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:03Z level=info event=scheduler_connected addr=lb:500516468worker1 # [ 135.175774] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6469worker2 # [ 135.337368] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:04Z level=info event=scheduler_connected addr=lb:500516470worker1 # [ 135.352263] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:03Z level=info event=oidc_jwks provider=mock keys=16471worker1 # [ 135.356544] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:03Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:395066472worker1 # [ 135.360397] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:03Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06473worker1 # [ 135.376328] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:03Z level=info event=scheduler_connected addr=lb:500516474worker1 # [ 135.380294] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:03Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:395066475worker1 # [ 135.383874] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:03Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06476lb: must succeed: curl -sf localhost:9901/clusters6477lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6478lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.17 seconds)6479lb: must succeed: curl -sf localhost:9901/clusters6480lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6481worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6482worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6483(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 13.96 seconds)6484subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6485client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16486worker1 # [ 136.092555] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:04Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:395206487worker1 # [ 136.103823] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6488worker2 # [ 136.115692] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:402966489worker1 # [ 136.165250] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:04Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06490worker1 # [ 136.169251] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:04Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16491worker1 # [ 136.173417] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:04Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06492worker1 # [ 136.177919] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:04Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06493worker1 # [ 136.222401] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206494worker1 # [ 136.235197] nix-grpc-daemon[1730]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 75 ms (attempt 1/5)6495worker1 # [ 136.316075] nix-grpc-daemon[1730]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 127 ms (attempt 2/5)6496worker1 # [ 136.448848] nix-grpc-daemon[1730]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 170 ms (attempt 3/5)6497worker1 # [ 136.624904] nix-grpc-daemon[1730]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 267 ms (attempt 4/5)6498worker1 # [ 136.898214] nix-grpc-daemon[1730]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6499worker1 # [ 136.902182] nix-grpc-daemon[1730]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6500worker1 # [ 136.920679] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395206501worker1 # [ 136.932441] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395206502worker1 # [ 136.943165] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6503worker1 # [ 136.963867] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:05Z level=info event=niks3_push_started pid=17956504worker1 # [ 137.192175] nix-grpc-daemon[1795]: time=2026-09-21T12:10:05.414Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6505lb # [ 137.383765] envoy[909]: conn peer=[2001:db8:1::1]:58204 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9641 tx=4122 ms=62876506lb # [ 137.387681] envoy[909]: conn peer=[2001:db8:1::1]:42508 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2962 tx=1404 ms=97306507lb # [ 137.391208] envoy[909]: conn peer=[2001:db8:1::3]:53894 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=3338 tx=904 ms=272246508worker1 # [ 137.388741] nix-daemon[848]: accepted connection from pid 1808, user nix-grpc-daemon (trusted)6509worker1 # [ 137.399500] nix-daemon[848]: reaped child process 1811, status = succeeded6510worker1 # [ 137.597050] nix-daemon[848]: accepted connection from pid 1813, user nix-grpc-daemon (trusted)6511worker1 # [ 137.608862] nix-daemon[848]: reaped child process 1816, status = succeeded6512lb # [ 137.635072] niks3-server[1029]: 2026/09/21 12:10:06 INFO Received uploads request method=POST path=/api/pending_closures6513worker1 # [ 137.638643] nix-grpc-daemon[1795]: time=2026-09-21T12:10:05.860Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6514worker1 # [ 137.640063] nix-grpc-daemon[1795]: time=2026-09-21T12:10:05.860Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6515lb # [ 137.669130] niks3-server[1029]: 2026/09/21 12:10:06 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6516lb # [ 137.676591] niks3-server[1029]: 2026/09/21 12:10:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6517lb # [ 137.681997] niks3-server[1029]: 2026/09/21 12:10:06 INFO Signed narinfos id=46 count=16518worker1 # [ 137.680905] nix-grpc-daemon[1795]: time=2026-09-21T12:10:05.902Z level=INFO msg="Uploading 1 narinfos"6519lb # [ 137.693651] niks3-server[1029]: 2026/09/21 12:10:06 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6520lb # [ 137.701295] niks3-server[1029]: 2026/09/21 12:10:06 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6521lb # [ 137.708445] niks3-server[1029]: 2026/09/21 12:10:06 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6522worker1 # [ 137.708518] nix-grpc-daemon[1795]: time=2026-09-21T12:10:05.930Z level=INFO msg="Upload complete. (511ms)"6523lb # [ 137.713751] niks3-server[1029]: 2026/09/21 12:10:06 INFO Completed upload id=466524worker1 # [ 137.711989] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=0 paths=1 nar_bytes_in=5046525worker1 # [ 137.717773] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206526worker1 # [ 137.728258] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6527worker1 # [ 137.731806] nix-daemon[848]: reaped child process 1793, status = succeeded6528worker1 # [ 137.737110] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6529worker1 # [ 138.949878] nix-daemon[848]: reaped child process 1820, status = succeeded6530worker1 # [ 139.133752] nix-daemon[848]: accepted connection from pid 1836, user nix-grpc-daemon (trusted)6531worker1 # [ 139.144658] nix-daemon[848]: reaped child process 1839, status = succeeded6532lb # [ 139.171612] niks3-server[1029]: 2026/09/21 12:10:08 INFO Received uploads request method=POST path=/api/pending_closures6533worker1 # [ 139.177222] nix-grpc-daemon[1795]: time=2026-09-21T12:10:07.399Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6534worker1 # [ 139.178741] nix-grpc-daemon[1795]: time=2026-09-21T12:10:07.401Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6535lb # [ 139.221127] niks3-server[1029]: 2026/09/21 12:10:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6536lb # [ 139.225171] niks3-server[1029]: 2026/09/21 12:10:08 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6537lb # [ 139.231411] niks3-server[1029]: 2026/09/21 12:10:08 INFO Signed narinfos id=47 count=16538worker1 # [ 139.228930] nix-grpc-daemon[1795]: time=2026-09-21T12:10:07.451Z level=INFO msg="Uploading 1 narinfos"6539lb # [ 139.241587] niks3-server[1029]: 2026/09/21 12:10:08 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6540lb # [ 139.252605] niks3-server[1029]: 2026/09/21 12:10:08 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6541lb # [ 139.254518] niks3-server[1029]: 2026/09/21 12:10:08 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6542lb # [ 139.258900] niks3-server[1029]: 2026/09/21 12:10:08 INFO Completed upload id=476543lb # [ 139.260890] niks3-server[1029]: 2026/09/21 12:10:08 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6544worker1 # [ 139.253559] nix-grpc-daemon[1795]: time=2026-09-21T12:10:07.475Z level=INFO msg="Upload complete. (309ms)"6545worker1 # [ 139.260865] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=16546worker1 # [ 139.272628] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:07Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16547worker1 # [ 139.277269] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:07Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06548worker1 # [ 139.284572] nix-daemon[848]: reaped child process 1818, status = succeeded6549worker1 # [ 139.312785] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206550worker1 # [ 139.325156] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395206551worker1 # [ 139.336475] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395206552worker1 # [ 139.344771] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6553worker1 # [ 139.551176] nix-daemon[848]: accepted connection from pid 1845, user nix-grpc-daemon (trusted)6554worker1 # [ 139.563154] nix-daemon[848]: reaped child process 1848, status = succeeded6555worker1 # [ 139.771123] nix-daemon[848]: accepted connection from pid 1850, user nix-grpc-daemon (trusted)6556worker1 # [ 139.782897] nix-daemon[848]: reaped child process 1853, status = succeeded6557lb # [ 139.809091] niks3-server[1029]: 2026/09/21 12:10:08 INFO Received uploads request method=POST path=/api/pending_closures6558worker1 # [ 139.819135] nix-grpc-daemon[1795]: time=2026-09-21T12:10:08.041Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6559worker1 # [ 139.820515] nix-grpc-daemon[1795]: time=2026-09-21T12:10:08.041Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6560lb # [ 139.842497] niks3-server[1029]: 2026/09/21 12:10:08 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6561lb # [ 139.854676] niks3-server[1029]: 2026/09/21 12:10:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6562worker1 # [ 139.854077] nix-grpc-daemon[1795]: time=2026-09-21T12:10:08.076Z level=INFO msg="Uploading 1 narinfos"6563lb # [ 139.859488] niks3-server[1029]: 2026/09/21 12:10:08 INFO Signed narinfos id=48 count=16564lb # [ 139.865151] niks3-server[1029]: 2026/09/21 12:10:08 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6565lb # [ 139.873511] niks3-server[1029]: 2026/09/21 12:10:09 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6566lb # [ 139.878763] niks3-server[1029]: 2026/09/21 12:10:09 INFO Completed upload id=486567worker1 # [ 139.873731] nix-grpc-daemon[1795]: time=2026-09-21T12:10:08.096Z level=INFO msg="Upload complete. (517ms)"6568lb # [ 139.880813] niks3-server[1029]: 2026/09/21 12:10:09 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6569worker1 # [ 139.877704] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=0 paths=1 nar_bytes_in=5046570worker1 # [ 139.882508] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206571worker1 # [ 139.891472] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6572worker1 # [ 139.896935] nix-daemon[848]: reaped child process 1843, status = succeeded6573worker1 # [ 139.902025] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6574worker1 # [ 141.123519] nix-daemon[848]: reaped child process 1857, status = succeeded6575worker1 # [ 141.315333] nix-daemon[848]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6576worker1 # [ 141.325982] nix-daemon[848]: reaped child process 1876, status = succeeded6577lb # [ 141.351660] niks3-server[1029]: 2026/09/21 12:10:10 INFO Received uploads request method=POST path=/api/pending_closures6578worker1 # [ 141.355640] nix-grpc-daemon[1795]: time=2026-09-21T12:10:09.577Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6579worker1 # [ 141.357480] nix-grpc-daemon[1795]: time=2026-09-21T12:10:09.579Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6580lb # [ 141.389690] niks3-server[1029]: 2026/09/21 12:10:10 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6581lb # [ 141.395618] niks3-server[1029]: 2026/09/21 12:10:10 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6582lb # [ 141.406103] niks3-server[1029]: 2026/09/21 12:10:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6583lb # [ 141.410222] niks3-server[1029]: 2026/09/21 12:10:10 INFO Signed narinfos id=49 count=16584worker1 # [ 141.408106] nix-grpc-daemon[1795]: time=2026-09-21T12:10:09.630Z level=INFO msg="Uploading 1 narinfos"6585lb # [ 141.420561] niks3-server[1029]: 2026/09/21 12:10:10 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6586lb # [ 141.426612] niks3-server[1029]: 2026/09/21 12:10:10 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6587lb # [ 141.433505] niks3-server[1029]: 2026/09/21 12:10:10 INFO Completed upload id=496588lb # [ 141.435051] niks3-server[1029]: 2026/09/21 12:10:10 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6589worker1 # [ 141.429967] nix-grpc-daemon[1795]: time=2026-09-21T12:10:09.652Z level=INFO msg="Upload complete. (320ms)"6590worker1 # [ 141.438600] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=16591worker1 # [ 141.447742] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:09Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16592worker1 # [ 141.452270] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206593worker1 # [ 141.456421] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:09Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06594worker1 # [ 141.465734] nix-daemon[848]: reaped child process 1855, status = succeeded6595worker1 # [ 141.468319] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395206596worker1 # [ 141.476399] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395206597worker1 # [ 141.481543] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6598worker1 # [ 141.668122] nix-daemon[848]: accepted connection from pid 1882, user nix-grpc-daemon (trusted)6599worker1 # [ 141.680221] nix-daemon[848]: reaped child process 1885, status = succeeded6600worker1 # [ 141.876264] nix-daemon[848]: accepted connection from pid 1887, user nix-grpc-daemon (trusted)6601worker1 # [ 141.888377] nix-daemon[848]: reaped child process 1890, status = succeeded6602lb # [ 141.910287] niks3-server[1029]: 2026/09/21 12:10:11 INFO Received uploads request method=POST path=/api/pending_closures6603worker1 # [ 141.912676] nix-grpc-daemon[1795]: time=2026-09-21T12:10:10.135Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6604worker1 # [ 141.914306] nix-grpc-daemon[1795]: time=2026-09-21T12:10:10.136Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6605lb # [ 141.937726] niks3-server[1029]: 2026/09/21 12:10:11 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6606lb # [ 141.948813] niks3-server[1029]: 2026/09/21 12:10:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6607worker1 # [ 141.948733] nix-grpc-daemon[1795]: time=2026-09-21T12:10:10.171Z level=INFO msg="Uploading 1 narinfos"6608lb # [ 141.954126] niks3-server[1029]: 2026/09/21 12:10:11 INFO Signed narinfos id=50 count=16609lb # [ 141.960069] niks3-server[1029]: 2026/09/21 12:10:11 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6610lb # [ 141.970049] niks3-server[1029]: 2026/09/21 12:10:11 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6611worker1 # [ 141.972199] nix-grpc-daemon[1795]: time=2026-09-21T12:10:10.194Z level=INFO msg="Upload complete. (485ms)"6612lb # [ 141.977498] niks3-server[1029]: 2026/09/21 12:10:11 INFO Completed upload id=506613lb # [ 141.980859] niks3-server[1029]: 2026/09/21 12:10:11 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6614worker1 # [ 141.974764] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=0 paths=1 nar_bytes_in=5046615worker1 # [ 141.979870] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206616worker1 # [ 141.989130] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6617worker1 # [ 141.994503] nix-daemon[848]: reaped child process 1880, status = succeeded6618worker1 # [ 141.999448] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6619worker1 # [ 143.223945] nix-daemon[848]: reaped child process 1894, status = succeeded6620worker1 # [ 143.420867] nix-daemon[848]: accepted connection from pid 1910, user nix-grpc-daemon (trusted)6621worker1 # [ 143.431620] nix-daemon[848]: reaped child process 1913, status = succeeded6622lb # [ 143.456344] niks3-server[1029]: 2026/09/21 12:10:12 INFO Received uploads request method=POST path=/api/pending_closures6623worker1 # [ 143.460044] nix-grpc-daemon[1795]: time=2026-09-21T12:10:11.682Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6624worker1 # [ 143.463347] nix-grpc-daemon[1795]: time=2026-09-21T12:10:11.683Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"6625lb # [ 143.487772] niks3-server[1029]: 2026/09/21 12:10:12 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv6626lb # [ 143.500256] niks3-server[1029]: 2026/09/21 12:10:12 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6627lb # [ 143.511562] niks3-server[1029]: 2026/09/21 12:10:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6628worker1 # [ 143.511069] nix-grpc-daemon[1795]: time=2026-09-21T12:10:11.733Z level=INFO msg="Uploading 1 narinfos"6629lb # [ 143.517342] niks3-server[1029]: 2026/09/21 12:10:12 INFO Signed narinfos id=51 count=16630lb # [ 143.521624] niks3-server[1029]: 2026/09/21 12:10:12 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls6631lb # [ 143.531988] niks3-server[1029]: 2026/09/21 12:10:12 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6632lb # [ 143.538239] niks3-server[1029]: 2026/09/21 12:10:12 INFO Completed upload id=516633worker1 # [ 143.533286] nix-grpc-daemon[1795]: time=2026-09-21T12:10:11.755Z level=INFO msg="Upload complete. (326ms)"6634lb # [ 143.540560] niks3-server[1029]: 2026/09/21 12:10:12 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo6635worker1 # [ 143.544595] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=16636worker1 # [ 143.551897] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:11Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=16637worker1 # [ 143.555337] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206638worker1 # [ 143.558690] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:11Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=06639worker1 # [ 143.569098] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395206640worker1 # [ 143.570965] nix-daemon[848]: reaped child process 1892, status = succeeded6641worker1 # [ 143.579094] nix-grpc-daemon[1795]: time=2026-09-21T12:10:11.801Z level=INFO msg="All 1 paths already cached"6642worker1 # [ 143.583434] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395206643worker1 # [ 143.588433] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6644worker1 # [ 143.770917] nix-daemon[848]: accepted connection from pid 1919, user nix-grpc-daemon (trusted)6645worker1 # [ 143.782982] nix-daemon[848]: reaped child process 1922, status = succeeded6646worker1 # [ 143.970779] nix-daemon[848]: accepted connection from pid 1924, user nix-grpc-daemon (trusted)6647worker1 # [ 143.982546] nix-daemon[848]: reaped child process 1927, status = succeeded6648lb # [ 144.006508] niks3-server[1029]: 2026/09/21 12:10:13 INFO Received uploads request method=POST path=/api/pending_closures6649worker1 # [ 144.010034] nix-grpc-daemon[1795]: time=2026-09-21T12:10:12.232Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6650worker1 # [ 144.011670] nix-grpc-daemon[1795]: time=2026-09-21T12:10:12.234Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"6651lb # [ 144.036658] niks3-server[1029]: 2026/09/21 12:10:13 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst6652lb # [ 144.049727] niks3-server[1029]: 2026/09/21 12:10:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6653worker1 # [ 144.050282] nix-grpc-daemon[1795]: time=2026-09-21T12:10:12.272Z level=INFO msg="Uploading 1 narinfos"6654lb # [ 144.055678] niks3-server[1029]: 2026/09/21 12:10:13 INFO Signed narinfos id=52 count=16655lb # [ 144.061319] niks3-server[1029]: 2026/09/21 12:10:13 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls6656lb # [ 144.075430] niks3-server[1029]: 2026/09/21 12:10:13 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6657lb # [ 144.080558] niks3-server[1029]: 2026/09/21 12:10:13 INFO Completed upload id=526658worker1 # [ 144.075459] nix-grpc-daemon[1795]: time=2026-09-21T12:10:12.297Z level=INFO msg="Upload complete. (482ms)"6659lb # [ 144.083349] niks3-server[1029]: 2026/09/21 12:10:13 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo6660worker1 # [ 144.078968] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=0 paths=1 nar_bytes_in=6246661worker1 # [ 144.085122] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206662worker1 # [ 144.094248] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6663worker1 # [ 144.102250] nix-daemon[848]: reaped child process 1917, status = succeeded6664worker1 # [ 144.107537] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6665worker1 # [ 145.131288] nix-daemon[848]: reaped child process 1931, status = succeeded6666worker1 # [ 145.320754] nix-daemon[848]: accepted connection from pid 1947, user nix-grpc-daemon (trusted)6667worker1 # [ 145.332784] nix-daemon[848]: reaped child process 1950, status = succeeded6668lb # [ 145.358954] niks3-server[1029]: 2026/09/21 12:10:14 INFO Received uploads request method=POST path=/api/pending_closures6669worker1 # [ 145.362131] nix-grpc-daemon[1795]: time=2026-09-21T12:10:13.584Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6670worker1 # [ 145.363754] nix-grpc-daemon[1795]: time=2026-09-21T12:10:13.586Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"6671lb # [ 145.399631] niks3-server[1029]: 2026/09/21 12:10:14 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv6672lb # [ 145.404974] niks3-server[1029]: 2026/09/21 12:10:14 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst6673lb # [ 145.416384] niks3-server[1029]: 2026/09/21 12:10:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6674lb # [ 145.420340] niks3-server[1029]: 2026/09/21 12:10:14 INFO Signed narinfos id=53 count=16675worker1 # [ 145.415412] nix-grpc-daemon[1795]: time=2026-09-21T12:10:13.637Z level=INFO msg="Uploading 1 narinfos"6676lb # [ 145.426488] niks3-server[1029]: 2026/09/21 12:10:14 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls6677lb # [ 145.435131] niks3-server[1029]: 2026/09/21 12:10:14 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6678lb # [ 145.439233] niks3-server[1029]: 2026/09/21 12:10:14 INFO Completed upload id=536679worker1 # [ 145.434261] nix-grpc-daemon[1795]: time=2026-09-21T12:10:13.656Z level=INFO msg="Upload complete. (318ms)"6680lb # [ 145.441085] niks3-server[1029]: 2026/09/21 12:10:14 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo6681worker1 # [ 145.441917] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=16682worker1 # [ 145.450605] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:13Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=16683worker1 # [ 145.455424] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206684worker1 # [ 145.459980] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:13Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=06685worker1 # [ 145.469434] nix-daemon[848]: reaped child process 1929, status = succeeded6686worker1 # [ 145.472078] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395206687worker1 # [ 145.480207] nix-grpc-daemon[1795]: time=2026-09-21T12:10:13.702Z level=INFO msg="All 1 paths already cached"6688worker1 # [ 145.484570] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395206689worker1 # [ 145.489532] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6690worker1 # [ 145.672288] nix-daemon[848]: accepted connection from pid 1956, user nix-grpc-daemon (trusted)6691worker1 # [ 145.684517] nix-daemon[848]: reaped child process 1959, status = succeeded6692worker1 # [ 145.869455] nix-daemon[848]: accepted connection from pid 1961, user nix-grpc-daemon (trusted)6693worker1 # [ 145.881140] nix-daemon[848]: reaped child process 1964, status = succeeded6694lb # [ 145.904975] niks3-server[1029]: 2026/09/21 12:10:15 INFO Received uploads request method=POST path=/api/pending_closures6695worker1 # [ 145.908949] nix-grpc-daemon[1795]: time=2026-09-21T12:10:14.131Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6696worker1 # [ 145.910421] nix-grpc-daemon[1795]: time=2026-09-21T12:10:14.131Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"6697lb # [ 145.931739] niks3-server[1029]: 2026/09/21 12:10:15 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst6698lb # [ 145.941198] niks3-server[1029]: 2026/09/21 12:10:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6699worker1 # [ 145.941480] nix-grpc-daemon[1795]: time=2026-09-21T12:10:14.163Z level=INFO msg="Uploading 1 narinfos"6700lb # [ 145.946710] niks3-server[1029]: 2026/09/21 12:10:15 INFO Signed narinfos id=54 count=16701lb # [ 145.953441] niks3-server[1029]: 2026/09/21 12:10:15 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls6702lb # [ 145.965888] niks3-server[1029]: 2026/09/21 12:10:15 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6703lb # [ 145.970963] niks3-server[1029]: 2026/09/21 12:10:15 INFO Completed upload id=546704worker1 # [ 145.967882] nix-grpc-daemon[1795]: time=2026-09-21T12:10:14.190Z level=INFO msg="Upload complete. (473ms)"6705lb # [ 145.974603] niks3-server[1029]: 2026/09/21 12:10:15 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo6706worker1 # [ 145.970923] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=0 paths=1 nar_bytes_in=6246707worker1 # [ 145.975659] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206708worker1 # [ 145.984479] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6709worker1 # [ 145.991123] nix-daemon[848]: reaped child process 1954, status = succeeded6710worker1 # [ 145.995661] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6711worker1 # [ 147.621513] nix-daemon[848]: reaped child process 1968, status = succeeded6712worker1 # [ 147.789708] nix-daemon[848]: accepted connection from pid 1984, user nix-grpc-daemon (trusted)6713worker1 # [ 147.800512] nix-daemon[848]: reaped child process 1987, status = succeeded6714lb # [ 147.825473] niks3-server[1029]: 2026/09/21 12:10:16 INFO Received uploads request method=POST path=/api/pending_closures6715worker1 # [ 147.828307] nix-grpc-daemon[1795]: time=2026-09-21T12:10:16.050Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6716worker1 # [ 147.829806] nix-grpc-daemon[1795]: time=2026-09-21T12:10:16.052Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"6717lb # [ 147.860824] niks3-server[1029]: 2026/09/21 12:10:16 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst6718lb # [ 147.874158] niks3-server[1029]: 2026/09/21 12:10:17 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv6719lb # [ 147.881250] niks3-server[1029]: 2026/09/21 12:10:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6720lb # [ 147.886456] niks3-server[1029]: 2026/09/21 12:10:17 INFO Signed narinfos id=55 count=16721worker1 # [ 147.884656] nix-grpc-daemon[1795]: time=2026-09-21T12:10:16.106Z level=INFO msg="Uploading 1 narinfos"6722lb # [ 147.893125] niks3-server[1029]: 2026/09/21 12:10:17 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls6723lb # [ 147.903495] niks3-server[1029]: 2026/09/21 12:10:17 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6724lb # [ 147.909221] niks3-server[1029]: 2026/09/21 12:10:17 INFO Completed upload id=556725worker1 # [ 147.903940] nix-grpc-daemon[1795]: time=2026-09-21T12:10:16.126Z level=INFO msg="Upload complete. (300ms)"6726lb # [ 147.912701] niks3-server[1029]: 2026/09/21 12:10:17 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo6727worker1 # [ 147.915900] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=16728worker1 # [ 147.921374] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:16Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=16729worker1 # [ 147.925504] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206730worker1 # [ 147.928775] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:16Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=06731worker1 # [ 147.937917] nix-daemon[848]: reaped child process 1966, status = succeeded6732worker1 # [ 147.940332] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395206733worker1 # [ 147.948068] nix-grpc-daemon[1795]: time=2026-09-21T12:10:16.170Z level=INFO msg="All 1 paths already cached"6734worker1 # [ 147.952778] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395206735worker1 # [ 147.956540] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6736worker1 # [ 148.145351] nix-daemon[848]: accepted connection from pid 1993, user nix-grpc-daemon (trusted)6737worker1 # [ 148.155950] nix-daemon[848]: reaped child process 1996, status = succeeded6738worker1 # [ 148.343262] nix-daemon[848]: accepted connection from pid 1998, user nix-grpc-daemon (trusted)6739worker1 # [ 148.355256] nix-daemon[848]: reaped child process 2001, status = succeeded6740lb # [ 148.378992] niks3-server[1029]: 2026/09/21 12:10:17 INFO Received uploads request method=POST path=/api/pending_closures6741worker1 # [ 148.383633] nix-grpc-daemon[1795]: time=2026-09-21T12:10:16.606Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6742worker1 # [ 148.385482] nix-grpc-daemon[1795]: time=2026-09-21T12:10:16.607Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"6743lb # [ 148.407071] niks3-server[1029]: 2026/09/21 12:10:17 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst6744lb # [ 148.417346] niks3-server[1029]: 2026/09/21 12:10:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6745lb # [ 148.421803] niks3-server[1029]: 2026/09/21 12:10:17 INFO Signed narinfos id=56 count=16746worker1 # [ 148.421447] nix-grpc-daemon[1795]: time=2026-09-21T12:10:16.643Z level=INFO msg="Uploading 1 narinfos"6747lb # [ 148.430998] niks3-server[1029]: 2026/09/21 12:10:17 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls6748lb # [ 148.440800] niks3-server[1029]: 2026/09/21 12:10:17 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6749lb # [ 148.447208] niks3-server[1029]: 2026/09/21 12:10:17 INFO Completed upload id=566750worker1 # [ 148.444317] nix-grpc-daemon[1795]: time=2026-09-21T12:10:16.666Z level=INFO msg="Upload complete. (481ms)"6751lb # [ 148.451040] niks3-server[1029]: 2026/09/21 12:10:17 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo6752worker1 # [ 148.447886] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=0 paths=1 nar_bytes_in=6246753worker1 # [ 148.453297] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206754worker1 # [ 148.464114] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6755worker1 # [ 148.468132] nix-daemon[848]: reaped child process 1991, status = succeeded6756worker1 # [ 148.473543] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6757worker1 # [ 149.498518] nix-daemon[848]: reaped child process 2005, status = succeeded6758worker1 # [ 149.693068] nix-daemon[848]: accepted connection from pid 2021, user nix-grpc-daemon (trusted)6759worker1 # [ 149.705124] nix-daemon[848]: reaped child process 2024, status = succeeded6760lb # [ 149.729914] niks3-server[1029]: 2026/09/21 12:10:18 INFO Received uploads request method=POST path=/api/pending_closures6761worker1 # [ 149.733089] nix-grpc-daemon[1795]: time=2026-09-21T12:10:17.955Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6762worker1 # [ 149.734739] nix-grpc-daemon[1795]: time=2026-09-21T12:10:17.957Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"6763lb # [ 149.769637] niks3-server[1029]: 2026/09/21 12:10:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6764worker1 # [ 149.773966] nix-grpc-daemon[1795]: time=2026-09-21T12:10:17.996Z level=INFO msg="Uploading 1 narinfos"6765lb # [ 149.779058] niks3-server[1029]: 2026/09/21 12:10:18 INFO Signed narinfos id=57 count=16766lb # [ 149.782724] niks3-server[1029]: 2026/09/21 12:10:18 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv6767lb # [ 149.789349] niks3-server[1029]: 2026/09/21 12:10:18 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst6768lb # [ 149.795269] niks3-server[1029]: 2026/09/21 12:10:18 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls6769lb # [ 149.804492] niks3-server[1029]: 2026/09/21 12:10:18 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6770lb # [ 149.810855] niks3-server[1029]: 2026/09/21 12:10:18 INFO Completed upload id=576771lb # [ 149.812552] niks3-server[1029]: 2026/09/21 12:10:18 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo6772worker1 # [ 149.805848] nix-grpc-daemon[1795]: time=2026-09-21T12:10:18.028Z level=INFO msg="Upload complete. (325ms)"6773worker1 # [ 149.813445] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=16774worker1 # [ 149.822749] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:18Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=16775worker1 # [ 149.828321] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206776worker1 # [ 149.832218] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:18Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=06777worker1 # [ 149.841856] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395206778worker1 # [ 149.843688] nix-daemon[848]: reaped child process 2003, status = succeeded6779worker1 # [ 149.850266] nix-grpc-daemon[1795]: time=2026-09-21T12:10:18.072Z level=INFO msg="All 2 paths already cached"6780worker1 # [ 149.854545] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395206781worker1 # [ 149.858616] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6782worker1 # [ 150.038687] nix-daemon[848]: accepted connection from pid 2030, user nix-grpc-daemon (trusted)6783worker1 # [ 150.050733] nix-daemon[848]: reaped child process 2033, status = succeeded6784worker1 # [ 150.237409] nix-daemon[848]: accepted connection from pid 2035, user nix-grpc-daemon (trusted)6785worker1 # [ 150.249429] nix-daemon[848]: reaped child process 2038, status = succeeded6786lb # [ 150.272892] niks3-server[1029]: 2026/09/21 12:10:19 INFO Received uploads request method=POST path=/api/pending_closures6787worker1 # [ 150.275812] nix-grpc-daemon[1795]: time=2026-09-21T12:10:18.497Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6788worker1 # [ 150.277215] nix-grpc-daemon[1795]: time=2026-09-21T12:10:18.497Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"6789lb # [ 150.298681] niks3-server[1029]: 2026/09/21 12:10:19 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst6790lb # [ 150.310351] niks3-server[1029]: 2026/09/21 12:10:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6791worker1 # [ 150.311908] nix-grpc-daemon[1795]: time=2026-09-21T12:10:18.534Z level=INFO msg="Uploading 1 narinfos"6792lb # [ 150.316921] niks3-server[1029]: 2026/09/21 12:10:19 INFO Signed narinfos id=58 count=16793lb # [ 150.321397] niks3-server[1029]: 2026/09/21 12:10:19 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls6794lb # [ 150.333365] niks3-server[1029]: 2026/09/21 12:10:19 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6795lb # [ 150.337692] niks3-server[1029]: 2026/09/21 12:10:19 INFO Completed upload id=586796worker1 # [ 150.334958] nix-grpc-daemon[1795]: time=2026-09-21T12:10:18.557Z level=INFO msg="Upload complete. (470ms)"6797lb # [ 150.341840] niks3-server[1029]: 2026/09/21 12:10:19 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo6798worker1 # [ 150.338418] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=0 paths=1 nar_bytes_in=7526799worker1 # [ 150.343469] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206800worker1 # [ 150.352138] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6801worker1 # [ 150.357733] nix-daemon[848]: reaped child process 2028, status = succeeded6802worker1 # [ 150.362949] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6803worker1 # [ 150.974406] nix-daemon[848]: reaped child process 2042, status = succeeded6804worker1 # [ 151.139869] nix-daemon[848]: accepted connection from pid 2058, user nix-grpc-daemon (trusted)6805worker1 # [ 151.151826] nix-daemon[848]: reaped child process 2061, status = succeeded6806lb # [ 151.175890] niks3-server[1029]: 2026/09/21 12:10:20 INFO Received uploads request method=POST path=/api/pending_closures6807worker1 # [ 151.180290] nix-grpc-daemon[1795]: time=2026-09-21T12:10:19.401Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6808worker1 # [ 151.181651] nix-grpc-daemon[1795]: time=2026-09-21T12:10:19.401Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"6809lb # [ 151.218204] niks3-server[1029]: 2026/09/21 12:10:20 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv6810lb # [ 151.224388] niks3-server[1029]: 2026/09/21 12:10:20 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst6811lb # [ 151.234651] niks3-server[1029]: 2026/09/21 12:10:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6812worker1 # [ 151.234163] nix-grpc-daemon[1795]: time=2026-09-21T12:10:19.456Z level=INFO msg="Uploading 1 narinfos"6813lb # [ 151.239140] niks3-server[1029]: 2026/09/21 12:10:20 INFO Signed narinfos id=59 count=16814lb # [ 151.245823] niks3-server[1029]: 2026/09/21 12:10:20 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls6815lb # [ 151.255216] niks3-server[1029]: 2026/09/21 12:10:20 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6816lb # [ 151.261195] niks3-server[1029]: 2026/09/21 12:10:20 INFO Completed upload id=596817worker1 # [ 151.256047] nix-grpc-daemon[1795]: time=2026-09-21T12:10:19.478Z level=INFO msg="Upload complete. (285ms)"6818lb # [ 151.263900] niks3-server[1029]: 2026/09/21 12:10:20 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo6819worker1 # [ 151.263726] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=0 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=16820worker1 # [ 151.270349] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:19Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=16821worker1 # [ 151.280430] nix-daemon[848]: reaped child process 2040, status = succeeded6822worker1 # [ 151.307191] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206823worker1 # [ 151.319737] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395206824worker1 # [ 151.331848] nix-grpc-daemon[1795]: time=2026-09-21T12:10:19.554Z level=INFO msg="All 2 paths already cached"6825worker1 # [ 151.339104] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395206826worker1 # [ 151.349152] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6827worker1 # [ 151.531624] nix-daemon[848]: accepted connection from pid 2065, user nix-grpc-daemon (trusted)6828worker1 # [ 151.544802] nix-daemon[848]: reaped child process 2068, status = succeeded6829worker1 # [ 151.724172] nix-daemon[848]: accepted connection from pid 2070, user nix-grpc-daemon (trusted)6830worker1 # [ 151.736428] nix-daemon[848]: reaped child process 2073, status = succeeded6831lb # [ 151.759308] niks3-server[1029]: 2026/09/21 12:10:20 INFO Received uploads request method=POST path=/api/pending_closures6832worker1 # [ 151.761637] nix-grpc-daemon[1795]: time=2026-09-21T12:10:19.984Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6833worker1 # [ 151.763445] nix-grpc-daemon[1795]: time=2026-09-21T12:10:19.985Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"6834lb # [ 151.786139] niks3-server[1029]: 2026/09/21 12:10:20 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst6835lb # [ 151.796489] niks3-server[1029]: 2026/09/21 12:10:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6836worker1 # [ 151.796354] nix-grpc-daemon[1795]: time=2026-09-21T12:10:20.018Z level=INFO msg="Uploading 1 narinfos"6837lb # [ 151.801965] niks3-server[1029]: 2026/09/21 12:10:20 INFO Signed narinfos id=60 count=16838lb # [ 151.807114] niks3-server[1029]: 2026/09/21 12:10:20 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls6839lb # [ 151.818889] niks3-server[1029]: 2026/09/21 12:10:20 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6840lb # [ 151.824356] niks3-server[1029]: 2026/09/21 12:10:20 INFO Completed upload id=606841worker1 # [ 151.819136] nix-grpc-daemon[1795]: time=2026-09-21T12:10:20.041Z level=INFO msg="Upload complete. (458ms)"6842lb # [ 151.827290] niks3-server[1029]: 2026/09/21 12:10:20 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo6843worker1 # [ 151.822723] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=0 paths=1 nar_bytes_in=7526844worker1 # [ 151.828884] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206845worker1 # [ 151.838780] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6846worker1 # [ 151.841243] nix-daemon[848]: reaped child process 2063, status = succeeded6847worker1 # [ 151.846785] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6848worker1 # [ 153.473679] nix-daemon[848]: reaped child process 2077, status = succeeded6849worker1 # [ 153.659480] nix-daemon[848]: accepted connection from pid 2093, user nix-grpc-daemon (trusted)6850worker1 # [ 153.671600] nix-daemon[848]: reaped child process 2096, status = succeeded6851lb # [ 153.695919] niks3-server[1029]: 2026/09/21 12:10:22 INFO Received uploads request method=POST path=/api/pending_closures6852worker1 # [ 153.699572] nix-grpc-daemon[1795]: time=2026-09-21T12:10:21.921Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6853worker1 # [ 153.702068] nix-grpc-daemon[1795]: time=2026-09-21T12:10:21.923Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"6854lb # [ 153.732691] niks3-server[1029]: 2026/09/21 12:10:22 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst6855lb # [ 153.742546] niks3-server[1029]: 2026/09/21 12:10:22 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv6856lb # [ 153.750632] niks3-server[1029]: 2026/09/21 12:10:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6857worker1 # [ 153.750258] nix-grpc-daemon[1795]: time=2026-09-21T12:10:21.972Z level=INFO msg="Uploading 1 narinfos"6858lb # [ 153.755634] niks3-server[1029]: 2026/09/21 12:10:22 INFO Signed narinfos id=61 count=16859lb # [ 153.761449] niks3-server[1029]: 2026/09/21 12:10:22 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls6860lb # [ 153.770936] niks3-server[1029]: 2026/09/21 12:10:22 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6861lb # [ 153.776895] niks3-server[1029]: 2026/09/21 12:10:22 INFO Completed upload id=616862worker1 # [ 153.771807] nix-grpc-daemon[1795]: time=2026-09-21T12:10:21.994Z level=INFO msg="Upload complete. (315ms)"6863lb # [ 153.778662] niks3-server[1029]: 2026/09/21 12:10:22 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo6864worker1 # [ 153.779975] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=16865worker1 # [ 153.786613] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:22Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=16866worker1 # [ 153.797119] nix-daemon[848]: reaped child process 2075, status = succeeded6867worker1 # [ 153.822610] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206868worker1 # [ 153.834713] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395206869worker1 # [ 153.847412] nix-grpc-daemon[1795]: time=2026-09-21T12:10:22.069Z level=INFO msg="All 2 paths already cached"6870worker1 # [ 153.854780] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395206871worker1 # [ 153.865170] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6872worker1 # [ 154.052568] nix-daemon[848]: accepted connection from pid 2100, user nix-grpc-daemon (trusted)6873worker1 # [ 154.064649] nix-daemon[848]: reaped child process 2103, status = succeeded6874worker1 # [ 154.245161] nix-daemon[848]: accepted connection from pid 2105, user nix-grpc-daemon (trusted)6875worker1 # [ 154.256920] nix-daemon[848]: reaped child process 2108, status = succeeded6876lb # [ 154.280363] niks3-server[1029]: 2026/09/21 12:10:23 INFO Received uploads request method=POST path=/api/pending_closures6877worker1 # [ 154.284038] nix-grpc-daemon[1795]: time=2026-09-21T12:10:22.506Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6878worker1 # [ 154.285577] nix-grpc-daemon[1795]: time=2026-09-21T12:10:22.507Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"6879lb # [ 154.309248] niks3-server[1029]: 2026/09/21 12:10:23 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst6880lb # [ 154.321701] niks3-server[1029]: 2026/09/21 12:10:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6881lb # [ 154.325766] niks3-server[1029]: 2026/09/21 12:10:23 INFO Signed narinfos id=62 count=16882worker1 # [ 154.324132] nix-grpc-daemon[1795]: time=2026-09-21T12:10:22.546Z level=INFO msg="Uploading 1 narinfos"6883lb # [ 154.332812] niks3-server[1029]: 2026/09/21 12:10:23 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls6884lb # [ 154.345601] niks3-server[1029]: 2026/09/21 12:10:23 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6885lb # [ 154.350944] niks3-server[1029]: 2026/09/21 12:10:23 INFO Completed upload id=626886worker1 # [ 154.346595] nix-grpc-daemon[1795]: time=2026-09-21T12:10:22.568Z level=INFO msg="Upload complete. (469ms)"6887lb # [ 154.355065] niks3-server[1029]: 2026/09/21 12:10:23 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo6888worker1 # [ 154.348984] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=0 paths=1 nar_bytes_in=7526889worker1 # [ 154.354578] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395206890worker1 # [ 154.363475] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6891worker1 # [ 154.368571] nix-daemon[848]: reaped child process 2098, status = succeeded6892worker1 # [ 154.373784] nix-daemon[848]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6893worker1 # [ 155.387047] nix-daemon[848]: reaped child process 2112, status = succeeded6894worker1 # [ 155.553878] nix-daemon[848]: accepted connection from pid 2128, user nix-grpc-daemon (trusted)6895worker1 # [ 155.565873] nix-daemon[848]: reaped child process 2131, status = succeeded6896lb # [ 155.590455] niks3-server[1029]: 2026/09/21 12:10:24 INFO Received uploads request method=POST path=/api/pending_closures6897worker1 # [ 155.594951] nix-grpc-daemon[1795]: time=2026-09-21T12:10:23.817Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6898worker1 # [ 155.597937] nix-grpc-daemon[1795]: time=2026-09-21T12:10:23.818Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"6899lb # [ 155.629331] niks3-server[1029]: 2026/09/21 12:10:24 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv6900lb # [ 155.635838] niks3-server[1029]: 2026/09/21 12:10:24 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst6901lb # [ 155.644390] niks3-server[1029]: 2026/09/21 12:10:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6902lb # [ 155.648600] niks3-server[1029]: 2026/09/21 12:10:24 INFO Signed narinfos id=63 count=16903worker1 # [ 155.646197] nix-grpc-daemon[1795]: time=2026-09-21T12:10:23.868Z level=INFO msg="Uploading 1 narinfos"6904lb # [ 155.657862] niks3-server[1029]: 2026/09/21 12:10:24 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls6905lb # [ 155.663363] niks3-server[1029]: 2026/09/21 12:10:24 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6906lb # [ 155.669216] niks3-server[1029]: 2026/09/21 12:10:24 INFO Completed upload id=636907worker1 # [ 155.664142] nix-grpc-daemon[1795]: time=2026-09-21T12:10:23.886Z level=INFO msg="Upload complete. (283ms)"6908lb # [ 155.671711] niks3-server[1029]: 2026/09/21 12:10:24 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo6909worker1 # [ 155.669715] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39520 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=16910worker1 # [ 155.684124] nix-daemon[848]: reaped child process 2110, status = succeeded6911client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 19.89 seconds)6912worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6913worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6914client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'6915client # [ 155.864065] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6916client # Running as unit: gpu.service; invocation ID: c254b44653204af6a69874d979fb5f4c6917client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.07 seconds)6918lb: must succeed: curl -sf localhost:9901/clusters6919lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6920lb: must succeed: curl -sf localhost:9901/clusters6921lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6922worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6923worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6924client # [ 156.097188] nix[1348]: don't know how to build these paths:6925client # [ 156.098122] nix[1348]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv6926worker1 # [ 156.085163] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:395206927worker1 # [ 156.089581] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395206928client # [ 156.121670] nix[1348]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker6929worker1 # [ 156.110300] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:24Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=06930lb: must succeed: curl -sf localhost:9901/clusters6931lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6932lb: must succeed: curl -sf localhost:9901/clusters6933lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6934worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6935worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6936client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6937lb # [ 157.385631] envoy[909]: conn peer=[2001:db8:1::1]:33068 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27059 tx=11411 ms=196036938client # [ 186.123321] nix[1348]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}6939client # [ 186.133195] nix[1348]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed6940client # [ 186.180367] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6941client # [ 186.181831] systemd[1]: gpu.service: Failed with result 'exit-code'.6942client # [ 186.184286] systemd[1]: gpu.service: Consumed 147ms CPU time over 30.319s wall clock time, 37.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.6943client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 28.90 seconds)6944client: must succeed: journalctl -u gpu -o cat6945client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)6946lb: must succeed: curl -sf localhost:9901/clusters6947lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6948lb: must succeed: curl -sf localhost:9901/clusters6949lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6950worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6951worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6952(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 50.59 seconds)6953subtest: low disk drains a worker and builds go to the other6954worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6955worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.92 seconds)6956worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6957lb # [ 187.387814] envoy[909]: conn peer=[2001:db8:1::1]:34774 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300536958worker1 # [ 187.530982] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:55Z level=info event=unhealthy reason=min_free6959worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.13 seconds)6960client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&26961client # don't know how to build these paths:6962client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv6963worker1 # [ 188.726506] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:56Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:395206964worker1 # [ 188.732180] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395206965worker1 # [ 188.752231] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:56Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=06966worker1 # [ 188.756088] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:56Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=16967worker2 # [ 188.803852] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402966968worker2 # [ 188.816937] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:402966969client # copying 1 paths to 'lb:50051'...6970client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...6971worker2 # [ 188.830129] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:402966972worker2 # [ 188.841369] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6973worker2 # [ 189.055748] nix-daemon[870]: accepted connection from pid 1659, user nix-grpc-daemon (trusted)6974worker2 # [ 189.067543] nix-daemon[870]: reaped child process 1662, status = succeeded6975worker2 # [ 189.248451] nix-daemon[870]: accepted connection from pid 1664, user nix-grpc-daemon (trusted)6976worker2 # [ 189.260118] nix-daemon[870]: reaped child process 1667, status = succeeded6977lb # [ 189.288370] niks3-server[1029]: 2026/09/21 12:10:58 INFO Received uploads request method=POST path=/api/pending_closures6978worker2 # [ 189.288833] nix-grpc-daemon[1478]: time=2026-09-21T12:10:58.287Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6979worker2 # [ 189.290226] nix-grpc-daemon[1478]: time=2026-09-21T12:10:58.287Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"6980lb # [ 189.320163] niks3-server[1029]: 2026/09/21 12:10:58 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst6981lb # [ 189.336748] niks3-server[1029]: 2026/09/21 12:10:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6982lb # [ 189.341631] niks3-server[1029]: 2026/09/21 12:10:58 INFO Signed narinfos id=64 count=16983worker2 # [ 189.335616] nix-grpc-daemon[1478]: time=2026-09-21T12:10:58.334Z level=INFO msg="Uploading 1 narinfos"6984lb # [ 189.349790] niks3-server[1029]: 2026/09/21 12:10:58 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls6985lb # [ 189.361460] niks3-server[1029]: 2026/09/21 12:10:58 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6986worker2 # [ 189.357925] nix-grpc-daemon[1478]: time=2026-09-21T12:10:58.357Z level=INFO msg="Upload complete. (504ms)"6987lb # [ 189.367462] niks3-server[1029]: 2026/09/21 12:10:58 INFO Completed upload id=646988worker2 # [ 189.360068] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=0 paths=1 nar_bytes_in=5126989lb # [ 189.371925] niks3-server[1029]: 2026/09/21 12:10:58 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo6990worker2 # [ 189.366440] nix-grpc-daemon[1406]: ts=2026-09-21T12:10:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402966991worker2 # [ 189.371548] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6992worker2 # [ 189.373809] nix-daemon[870]: reaped child process 1657, status = succeeded6993worker2 # [ 189.379410] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)6994client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...6995client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv6996client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6997client # farm-a> LOG-farm-a-drain6998worker2 # [ 190.804355] nix-daemon[870]: reaped child process 1671, status = succeeded6999worker2 # [ 191.002320] nix-daemon[870]: accepted connection from pid 1687, user nix-grpc-daemon (trusted)7000worker2 # [ 191.012809] nix-daemon[870]: reaped child process 1690, status = succeeded7001lb # [ 191.039859] niks3-server[1029]: 2026/09/21 12:11:00 INFO Received uploads request method=POST path=/api/pending_closures7002worker2 # [ 191.038913] nix-grpc-daemon[1478]: time=2026-09-21T12:11:00.038Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7003worker2 # [ 191.040523] nix-grpc-daemon[1478]: time=2026-09-21T12:11:00.039Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7004lb # [ 191.073686] niks3-server[1029]: 2026/09/21 12:11:00 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7005lb # [ 191.090357] niks3-server[1029]: 2026/09/21 12:11:00 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7006lb # [ 191.094534] niks3-server[1029]: 2026/09/21 12:11:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7007worker2 # [ 191.092418] nix-grpc-daemon[1478]: time=2026-09-21T12:11:00.091Z level=INFO msg="Uploading 1 narinfos"7008lb # [ 191.102162] niks3-server[1029]: 2026/09/21 12:11:00 INFO Signed narinfos id=65 count=17009lb # [ 191.108861] niks3-server[1029]: 2026/09/21 12:11:00 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7010lb # [ 191.118039] niks3-server[1029]: 2026/09/21 12:11:00 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7011lb # [ 191.123902] niks3-server[1029]: 2026/09/21 12:11:00 INFO Completed upload id=657012worker2 # [ 191.114749] nix-grpc-daemon[1478]: time=2026-09-21T12:11:00.114Z level=INFO msg="Upload complete. (327ms)"7013lb # [ 191.126316] niks3-server[1029]: 2026/09/21 12:11:00 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7014worker2 # [ 191.119237] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=17015worker1 # [ 191.131429] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:59Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07016worker2 # [ 191.131501] nix-daemon[870]: reaped child process 1669, status = succeeded7017worker1 # [ 191.135448] nix-grpc-daemon[1730]: ts=2026-09-21T12:10:59Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17018worker2 # [ 191.170437] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402967019worker2 # [ 191.183126] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:402967020worker2 # [ 191.196692] nix-grpc-daemon[1478]: time=2026-09-21T12:11:00.195Z level=INFO msg="All 1 paths already cached"7021client # copying 1 paths to 'lb:50051'...7022client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7023worker2 # [ 191.205347] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:402967024worker2 # [ 191.213382] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)7025worker2 # [ 191.421519] nix-daemon[870]: accepted connection from pid 1694, user nix-grpc-daemon (trusted)7026worker2 # [ 191.434824] nix-daemon[870]: reaped child process 1697, status = succeeded7027worker2 # [ 191.614610] nix-daemon[870]: accepted connection from pid 1699, user nix-grpc-daemon (trusted)7028worker2 # [ 191.626773] nix-daemon[870]: reaped child process 1702, status = succeeded7029lb # [ 191.651691] niks3-server[1029]: 2026/09/21 12:11:00 INFO Received uploads request method=POST path=/api/pending_closures7030worker2 # [ 191.649721] nix-grpc-daemon[1478]: time=2026-09-21T12:11:00.649Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7031worker2 # [ 191.651229] nix-grpc-daemon[1478]: time=2026-09-21T12:11:00.650Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7032lb # [ 191.677894] niks3-server[1029]: 2026/09/21 12:11:00 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7033lb # [ 191.690741] niks3-server[1029]: 2026/09/21 12:11:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7034worker2 # [ 191.687414] nix-grpc-daemon[1478]: time=2026-09-21T12:11:00.686Z level=INFO msg="Uploading 1 narinfos"7035lb # [ 191.697338] niks3-server[1029]: 2026/09/21 12:11:00 INFO Signed narinfos id=66 count=17036lb # [ 191.702208] niks3-server[1029]: 2026/09/21 12:11:00 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7037lb # [ 191.715074] niks3-server[1029]: 2026/09/21 12:11:00 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7038worker2 # [ 191.710347] nix-grpc-daemon[1478]: time=2026-09-21T12:11:00.709Z level=INFO msg="Upload complete. (483ms)"7039lb # [ 191.720082] niks3-server[1029]: 2026/09/21 12:11:00 INFO Completed upload id=667040worker2 # [ 191.712581] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=0 paths=1 nar_bytes_in=6247041lb # [ 191.723333] niks3-server[1029]: 2026/09/21 12:11:00 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7042worker2 # [ 191.717997] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402967043worker2 # [ 191.728108] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)7044worker2 # [ 191.732323] nix-daemon[870]: reaped child process 1692, status = succeeded7045worker2 # [ 191.737420] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)7046client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7047client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7048client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7049client # farm-b> LOG-farm-b-drain7050worker2 # [ 192.762742] nix-daemon[870]: reaped child process 1706, status = succeeded7051worker2 # [ 192.933279] nix-daemon[870]: accepted connection from pid 1722, user nix-grpc-daemon (trusted)7052worker2 # [ 192.945290] nix-daemon[870]: reaped child process 1725, status = succeeded7053lb # [ 192.969915] niks3-server[1029]: 2026/09/21 12:11:02 INFO Received uploads request method=POST path=/api/pending_closures7054worker2 # [ 192.968616] nix-grpc-daemon[1478]: time=2026-09-21T12:11:01.967Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7055worker2 # [ 192.970138] nix-grpc-daemon[1478]: time=2026-09-21T12:11:01.969Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7056lb # [ 193.016592] niks3-server[1029]: 2026/09/21 12:11:02 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7057lb # [ 193.028511] niks3-server[1029]: 2026/09/21 12:11:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7058worker2 # [ 193.022945] nix-grpc-daemon[1478]: time=2026-09-21T12:11:02.022Z level=INFO msg="Uploading 1 narinfos"7059lb # [ 193.033300] niks3-server[1029]: 2026/09/21 12:11:02 INFO Signed narinfos id=67 count=17060lb # [ 193.036712] niks3-server[1029]: 2026/09/21 12:11:02 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7061lb # [ 193.040861] niks3-server[1029]: 2026/09/21 12:11:02 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7062lb # [ 193.053512] niks3-server[1029]: 2026/09/21 12:11:02 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7063lb # [ 193.056699] niks3-server[1029]: 2026/09/21 12:11:02 INFO Completed upload id=677064worker2 # [ 193.047507] nix-grpc-daemon[1478]: time=2026-09-21T12:11:02.046Z level=INFO msg="Upload complete. (303ms)"7065lb # [ 193.058381] niks3-server[1029]: 2026/09/21 12:11:02 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7066worker2 # [ 193.052890] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=17067worker1 # [ 193.064751] nix-grpc-daemon[1730]: ts=2026-09-21T12:11:01Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07068worker1 # [ 193.068841] nix-grpc-daemon[1730]: ts=2026-09-21T12:11:01Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17069worker2 # [ 193.067521] nix-daemon[870]: reaped child process 1704, status = succeeded7070worker2 # [ 193.104285] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402967071worker2 # [ 193.116589] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:402967072worker2 # [ 193.129471] nix-grpc-daemon[1478]: time=2026-09-21T12:11:02.128Z level=INFO msg="All 2 paths already cached"7073client # copying 1 paths to 'lb:50051'...7074client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7075worker2 # [ 193.138109] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:402967076worker2 # [ 193.146285] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)7077worker2 # [ 193.337951] nix-daemon[870]: accepted connection from pid 1729, user nix-grpc-daemon (trusted)7078worker2 # [ 193.351424] nix-daemon[870]: reaped child process 1732, status = succeeded7079worker2 # [ 193.524754] nix-daemon[870]: accepted connection from pid 1734, user nix-grpc-daemon (trusted)7080worker2 # [ 193.536857] nix-daemon[870]: reaped child process 1737, status = succeeded7081lb # [ 193.564734] niks3-server[1029]: 2026/09/21 12:11:02 INFO Received uploads request method=POST path=/api/pending_closures7082worker2 # [ 193.562895] nix-grpc-daemon[1478]: time=2026-09-21T12:11:02.562Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7083worker2 # [ 193.564273] nix-grpc-daemon[1478]: time=2026-09-21T12:11:02.562Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7084lb # [ 193.594172] niks3-server[1029]: 2026/09/21 12:11:02 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7085lb # [ 193.602800] niks3-server[1029]: 2026/09/21 12:11:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7086worker2 # [ 193.600324] nix-grpc-daemon[1478]: time=2026-09-21T12:11:02.599Z level=INFO msg="Uploading 1 narinfos"7087lb # [ 193.609057] niks3-server[1029]: 2026/09/21 12:11:02 INFO Signed narinfos id=68 count=17088lb # [ 193.613745] niks3-server[1029]: 2026/09/21 12:11:02 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7089lb # [ 193.625541] niks3-server[1029]: 2026/09/21 12:11:02 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7090worker2 # [ 193.620586] nix-grpc-daemon[1478]: time=2026-09-21T12:11:02.620Z level=INFO msg="Upload complete. (460ms)"7091lb # [ 193.630116] niks3-server[1029]: 2026/09/21 12:11:02 INFO Completed upload id=687092worker2 # [ 193.622517] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=0 paths=1 nar_bytes_in=7527093lb # [ 193.635475] niks3-server[1029]: 2026/09/21 12:11:02 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7094worker2 # [ 193.628313] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:402967095worker2 # [ 193.635079] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)7096worker2 # [ 193.640186] nix-daemon[870]: reaped child process 1727, status = succeeded7097worker2 # [ 193.645314] nix-daemon[870]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)7098client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7099client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7100client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7101client # farm-top> LOG-farm-top-drain7102worker2 # [ 194.669575] nix-daemon[870]: reaped child process 1741, status = succeeded7103worker2 # [ 194.844386] nix-daemon[870]: accepted connection from pid 1757, user nix-grpc-daemon (trusted)7104worker2 # [ 194.855176] nix-daemon[870]: reaped child process 1760, status = succeeded7105lb # [ 194.883900] niks3-server[1029]: 2026/09/21 12:11:04 INFO Received uploads request method=POST path=/api/pending_closures7106worker2 # [ 194.884604] nix-grpc-daemon[1478]: time=2026-09-21T12:11:03.883Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7107worker2 # [ 194.886077] nix-grpc-daemon[1478]: time=2026-09-21T12:11:03.885Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7108lb # [ 194.922751] niks3-server[1029]: 2026/09/21 12:11:04 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7109lb # [ 194.928751] niks3-server[1029]: 2026/09/21 12:11:04 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7110lb # [ 194.941842] niks3-server[1029]: 2026/09/21 12:11:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7111lb # [ 194.947476] niks3-server[1029]: 2026/09/21 12:11:04 INFO Signed narinfos id=69 count=17112worker2 # [ 194.940802] nix-grpc-daemon[1478]: time=2026-09-21T12:11:03.939Z level=INFO msg="Uploading 1 narinfos"7113lb # [ 194.951518] niks3-server[1029]: 2026/09/21 12:11:04 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7114lb # [ 194.960163] niks3-server[1029]: 2026/09/21 12:11:04 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7115lb # [ 194.966631] niks3-server[1029]: 2026/09/21 12:11:04 INFO Completed upload id=697116worker2 # [ 194.957282] nix-grpc-daemon[1478]: time=2026-09-21T12:11:03.956Z level=INFO msg="Upload complete. (305ms)"7117lb # [ 194.968735] niks3-server[1029]: 2026/09/21 12:11:04 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7118worker2 # [ 194.962525] nix-grpc-daemon[1406]: ts=2026-09-21T12:11:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40296 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=17119worker2 # [ 194.975817] nix-daemon[870]: reaped child process 1739, status = succeeded7120client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 6.58 seconds)7121client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7122client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.22 seconds)7123worker1: must succeed: rm /nix/.rw-store/fill7124worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.13 seconds)7125worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7126worker1 # [ 195.533365] nix-grpc-daemon[1730]: ts=2026-09-21T12:11:03Z level=info event=healthy reason=min_free7127worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.13 seconds)7128(finished: subtest: low disk drains a worker and builds go to the other, in 10.19 seconds)7129(finished: run the VM test script, in 197.49 seconds)7130test script finished in 197.77s7131cleanup7132kill QemuMachine (pid 49)7133client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7134kill QemuMachine (pid 51)7135lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7136kill QemuMachine (pid 50)7137worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7138kill QemuMachine (pid 52)7139worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7140(finished: cleanup, in 1.01 seconds)7141additionally exposed symbols:7142 client, lb, worker1, worker2,7143 vlan1,7144 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh