vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #471
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker1 # Disk image does not exist, creating the virtualisation disk image...13worker1: QEMU running (pid 51)14worker1 # Formatting '/build/vm-state-worker1/tmp.pnF3b1Bsp3', fmt=raw size=107374182415worker1 # mke2fs 1.47.4 (6-Mar-2025)16worker1 # Discarding device blocks: 0/262144 done17worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes18worker1 # Filesystem UUID: 5e083b31-5642-4f37-bb08-5fd602cc04c319worker1 # Superblock backups stored on blocks:20worker1 # 32768, 98304, 163840, 22937621worker1 # 22worker1 # Allocating group tables: 0/8 done23worker2 # Disk image does not exist, creating the virtualisation disk image...24worker2: QEMU running (pid 52)25client # Disk image does not exist, creating the virtualisation disk image...26worker2 # Formatting '/build/vm-state-worker2/tmp.PFdGoYLJ77', fmt=raw size=107374182427worker1 # Writing inode tables: 0/8 done28client: QEMU running (pid 49)29client # Formatting '/build/vm-state-client/tmp.ndP2vWVKOi', fmt=raw size=107374182430worker2 # mke2fs 1.47.4 (6-Mar-2025)31client # mke2fs 1.47.4 (6-Mar-2025)32worker2 # Discarding device blocks: 0/262144 done33client # Discarding device blocks: 0/262144 done34worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes35client # Creating filesystem with 262144 4k blocks and 65536 inodes36worker2 # Filesystem UUID: a55e5c61-4b6b-41fd-af43-7db9df9a5d1e37client # Filesystem UUID: b1ad2766-4142-414e-9430-a41bb171bb9838worker2 # Superblock backups stored on blocks:39client # Superblock backups stored on blocks:40worker2 # 32768, 98304, 163840, 22937641client # 32768, 98304, 163840, 22937642worker2 # 43client # 44worker2 # Allocating group tables: 0/8 done45client # Allocating group tables: 0/8 done46worker2 # Writing inode tables: 0/8 done47client # Writing inode tables: 0/8 done48worker2 # Creating journal (8192 blocks): done49client # Creating journal (8192 blocks): done50worker2 # Writing superblocks and filesystem accounting information: 0/8 done51client # Writing superblocks and filesystem accounting information: 0/8 done52worker2 # 53client # 54worker2 # Virtualisation disk image created.55client # Virtualisation disk image created.56worker1 # Creating journal (8192 blocks): done57worker1 # Writing superblocks and filesystem accounting information: 0/8 done58worker1 # 59worker1 # Virtualisation disk image created.60lb: QEMU running (pid 50)61lb # Disk image does not exist, creating the virtualisation disk image...62lb # Formatting '/build/vm-state-lb/tmp.okghjkp1ji', fmt=raw size=107374182463lb # mke2fs 1.47.4 (6-Mar-2025)64lb # Discarding device blocks: 0/262144 done65lb # Creating filesystem with 262144 4k blocks and 65536 inodes66lb # Filesystem UUID: 9b71a840-d8c5-4223-ac16-c11d8f2e820e67lb # Superblock backups stored on blocks:68lb # 32768, 98304, 163840, 22937669lb # 70lb # Allocating group tables: 0/8 done71(finished: start all VMs, in 0.25 seconds)72client: waiting for unit network-addresses-eth1.service73lb # Writing inode tables: 0/8 done74lb # Creating journal (8192 blocks): done75lb # Writing superblocks and filesystem accounting information: 0/8 done76lb # 77lb # Virtualisation disk image created.78client: waiting for the VM to finish booting79worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83worker2 # 84worker2 # 85worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0086worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...87worker2 # 88worker2 # 89worker2 # 90worker2 # 91worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB0092worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...93worker2 # 94worker2 # 95worker1 # 96worker1 # 97worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0098worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...99worker1 # 100worker1 # 101worker1 # 102lb # 103worker1 # 104lb # 105lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00106worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00107client # 108client # 109client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00110lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...111lb # 112lb # 113lb # 114lb # 115worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...116worker1 # 117worker1 # 118lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00119client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...120client # 121client # 122client # 123client # 124client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00125lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...126lb # 127lb # 128client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129client # 130client # 131worker2 # Booting from ROM...132worker1 # Booting from ROM...133lb # Booting from ROM...134client # Booting from ROM...135worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/q6x897783vrx1j21yalizihcz5pshnmx-nixos-system-worker2-test/init regInfo=/nix/store/sdbl9k12km9rsxdcvhj0g4kca5f47ysa-closure-info/registration console=ttyS0,115200n8 console=tty0137worker2 # [ 0.000000] BIOS-provided physical RAM map:138worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026141worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved142worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable143worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved144worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved145worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/m0k8shb6bvnyyb7vss69gd05kh2ps8cp-nixos-system-worker1-test/init regInfo=/nix/store/9pdhxfrd2i3gvvk5rwjv04jkk85gy654-closure-info/registration console=ttyS0,115200n8 console=tty0146worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved147worker1 # [ 0.000000] BIOS-provided physical RAM map:148worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved149worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable150worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved151worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved152worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved153worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved154worker2 # [ 0.000000] NX (Execute Disable) protection: active155worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable156worker2 # [ 0.000000] APIC: Static calls initialized157worker2 # [ 0.000000] SMBIOS 2.8 present.158worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved159worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014160worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved161worker2 # [ 0.000000] DMI: Memory slots populated: 1/1162worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved163worker2 # [ 0.000000] Hypervisor detected: KVM164worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000165worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved166worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00167worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved168worker2 # [ 0.000000] kvm-clock: using sched offset of 485348934 cycles169worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved170worker1 # [ 0.000000] NX (Execute Disable) protection: active171worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns172worker1 # [ 0.000000] APIC: Static calls initialized173worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor174worker1 # [ 0.000000] SMBIOS 2.8 present.175worker2 # [ 0.000813] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000176worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014177client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026178worker2 # [ 0.000840] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs179worker1 # [ 0.000000] DMI: Memory slots populated: 1/1180worker1 # [ 0.000000] Hypervisor detected: KVM181worker2 # [ 0.000843] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT182worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000183worker2 # [ 0.002736] found SMP MP-table at [mem 0x000f5450-0x000f545f]184worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00185worker2 # [ 0.002747] Using GB pages for direct mapping186worker1 # [ 0.000000] kvm-clock: using sched offset of 500162220 cycles187worker2 # [ 0.002884] RAMDISK: [mem 0x5e367000-0x5ffcffff]188worker2 # [ 0.002892] ACPI: Early table checksum verification disabled189lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026190client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/lgjx0fgzgnqvb086gw02890w1401r9xx-nixos-system-client-test/init regInfo=/nix/store/9f9i8jsph8f3c9h91zzs864hg65vz5v7-closure-info/registration console=ttyS0,115200n8 console=tty0191worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns192worker2 # [ 0.002895] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )193client # [ 0.000000] BIOS-provided physical RAM map:194worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor195client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable196worker2 # [ 0.002899] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)197worker1 # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000198client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved199worker1 # [ 0.000839] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs200worker2 # [ 0.002902] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)201client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved202worker1 # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT203worker2 # [ 0.002909] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)204lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/fj20bcn47ir2axl7b7sxwa0fvig2ds2j-nixos-system-lb-test/init regInfo=/nix/store/l5iprc9wdxb7nl0q1bx2grj7psaadl34-closure-info/registration console=ttyS0,115200n8 console=tty0205client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable206worker1 # [ 0.002733] found SMP MP-table at [mem 0x000f5450-0x000f545f]207lb # [ 0.000000] BIOS-provided physical RAM map:208worker2 # [ 0.002911] ACPI: FACS 0x000000005FFE0000 000040209worker1 # [ 0.002744] Using GB pages for direct mapping210client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved211lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable212worker1 # [ 0.002868] RAMDISK: [mem 0x5e367000-0x5ffcffff]213worker2 # [ 0.002912] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)214worker1 # [ 0.002876] ACPI: Early table checksum verification disabled215lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved216client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved217worker2 # [ 0.002914] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)218worker1 # [ 0.002878] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )219client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved220lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved221worker2 # [ 0.002915] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)222client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved223worker1 # [ 0.002882] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)224lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable225client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved226worker2 # [ 0.002917] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)227lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved228worker1 # [ 0.002886] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)229client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved230worker2 # [ 0.002918] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]231lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved232worker1 # [ 0.002893] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)233client # [ 0.000000] NX (Execute Disable) protection: active234lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved235worker2 # [ 0.002919] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]236worker1 # [ 0.002895] ACPI: FACS 0x000000005FFE0000 000040237client # [ 0.000000] APIC: Static calls initialized238client # [ 0.000000] SMBIOS 2.8 present.239worker2 # [ 0.002920] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]240lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved241worker1 # [ 0.002897] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)242lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved243worker2 # [ 0.002920] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]244client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014245worker1 # [ 0.002898] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)246client # [ 0.000000] DMI: Memory slots populated: 1/1247lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved248worker2 # [ 0.002921] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]249client # [ 0.000000] Hypervisor detected: KVM250lb # [ 0.000000] NX (Execute Disable) protection: active251worker1 # [ 0.002900] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)252worker2 # [ 0.002921] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]253client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000254lb # [ 0.000000] APIC: Static calls initialized255lb # [ 0.000000] SMBIOS 2.8 present.256client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00257worker1 # [ 0.002901] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)258worker2 # [ 0.002922] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]259client # [ 0.000000] kvm-clock: using sched offset of 511755142 cycles260worker2 # [ 0.003148] No NUMA configuration found261worker1 # [ 0.002903] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]262lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014263worker2 # [ 0.003149] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]264lb # [ 0.000000] DMI: Memory slots populated: 1/1265worker1 # [ 0.002904] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]266client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns267lb # [ 0.000000] Hypervisor detected: KVM268worker2 # [ 0.003151] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]269client # [ 0.000005] tsc: Detected 2400.012 MHz processor270worker2 # [ 0.005541] Zone ranges:271lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000272worker1 # [ 0.002904] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]273client # [ 0.000809] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000274worker2 # [ 0.005542] DMA [mem 0x0000000000001000-0x0000000000ffffff]275lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00276worker1 # [ 0.002905] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]277worker2 # [ 0.005543] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]278lb # [ 0.000000] kvm-clock: using sched offset of 506540598 cycles279client # [ 0.000837] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs280worker2 # [ 0.005544] Normal empty281worker1 # [ 0.002905] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]282worker2 # [ 0.005545] Device empty283worker2 # [ 0.005546] Movable zone start for each node284client # [ 0.000840] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT285worker1 # [ 0.002906] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]286lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns287worker2 # [ 0.005546] Early memory node ranges288client # [ 0.002748] found SMP MP-table at [mem 0x000f5450-0x000f545f]289lb # [ 0.000005] tsc: Detected 2400.012 MHz processor290worker1 # [ 0.002906] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]291client # [ 0.002759] Using GB pages for direct mapping292worker2 # [ 0.005547] node 0: [mem 0x0000000000001000-0x000000000009efff]293worker1 # [ 0.003128] No NUMA configuration found294lb # [ 0.000810] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000295client # [ 0.002827] RAMDISK: [mem 0x5e366000-0x5ffcffff]296worker2 # [ 0.005548] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]297worker1 # [ 0.003129] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]298client # [ 0.002835] ACPI: Early table checksum verification disabled299lb # [ 0.000837] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs300worker2 # [ 0.005549] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]301worker1 # [ 0.003131] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]302client # [ 0.002838] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )303lb # [ 0.000839] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT304worker1 # [ 0.005407] Zone ranges:305worker2 # [ 0.005572] On node 0, zone DMA: 1 pages in unavailable ranges306worker1 # [ 0.005408] DMA [mem 0x0000000000001000-0x0000000000ffffff]307client # [ 0.002842] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)308lb # [ 0.002730] found SMP MP-table at [mem 0x000f5450-0x000f545f]309worker2 # [ 0.005847] On node 0, zone DMA: 97 pages in unavailable ranges310lb # [ 0.002741] Using GB pages for direct mapping311worker1 # [ 0.005409] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]312worker2 # [ 0.033920] On node 0, zone DMA32: 40 pages in unavailable ranges313client # [ 0.002846] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)314lb # [ 0.002854] RAMDISK: [mem 0x5e367000-0x5ffcffff]315worker1 # [ 0.005411] Normal empty316worker2 # [ 0.034383] ACPI: PM-Timer IO Port: 0x608317worker1 # [ 0.005411] Device empty318lb # [ 0.002862] ACPI: Early table checksum verification disabled319worker1 # [ 0.005412] Movable zone start for each node320client # [ 0.002853] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)321worker2 # [ 0.034395] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])322worker1 # [ 0.005413] Early memory node ranges323lb # [ 0.002866] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )324client # [ 0.002855] ACPI: FACS 0x000000005FFE0000 000040325worker2 # [ 0.034423] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23326worker1 # [ 0.005413] node 0: [mem 0x0000000000001000-0x000000000009efff]327lb # [ 0.002870] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)328client # [ 0.002857] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)329worker2 # [ 0.034425] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)330worker1 # [ 0.005414] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]331lb # [ 0.002874] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)332worker2 # [ 0.034427] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)333client # [ 0.002859] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)334worker1 # [ 0.005415] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]335worker2 # [ 0.034428] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)336worker1 # [ 0.005434] On node 0, zone DMA: 1 pages in unavailable ranges337lb # [ 0.002881] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)338client # [ 0.002860] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)339lb # [ 0.002883] ACPI: FACS 0x000000005FFE0000 000040340worker2 # [ 0.034429] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)341worker1 # [ 0.005698] On node 0, zone DMA: 97 pages in unavailable ranges342client # [ 0.002862] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)343worker1 # [ 0.032677] On node 0, zone DMA32: 40 pages in unavailable ranges344worker2 # [ 0.034430] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)345lb # [ 0.002884] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)346worker1 # [ 0.033132] ACPI: PM-Timer IO Port: 0x608347client # [ 0.002863] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]348worker2 # [ 0.034432] ACPI: Using ACPI (MADT) for SMP configuration information349worker1 # [ 0.033142] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])350lb # [ 0.002886] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)351client # [ 0.002864] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]352worker2 # [ 0.034433] ACPI: HPET id: 0x8086a201 base: 0xfed00000353worker1 # [ 0.033168] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23354worker2 # [ 0.034436] TSC deadline timer available355lb # [ 0.002887] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)356client # [ 0.002865] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]357worker2 # [ 0.034440] CPU topo: Max. logical packages: 1358worker1 # [ 0.033171] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)359worker2 # [ 0.034441] CPU topo: Max. logical dies: 1360client # [ 0.002865] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]361lb # [ 0.002889] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)362worker1 # [ 0.033172] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)363worker2 # [ 0.034441] CPU topo: Max. dies per package: 1364lb # [ 0.002890] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]365worker2 # [ 0.034445] CPU topo: Max. threads per core: 1366worker1 # [ 0.033173] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)367client # [ 0.002866] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]368worker2 # [ 0.034445] CPU topo: Num. cores per package: 1369client # [ 0.002866] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]370lb # [ 0.002891] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]371worker1 # [ 0.033175] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)372worker2 # [ 0.034446] CPU topo: Num. threads per package: 1373client # [ 0.002866] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]374lb # [ 0.002892] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]375worker1 # [ 0.033175] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)376worker2 # [ 0.034446] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs377client # [ 0.003095] No NUMA configuration found378lb # [ 0.002892] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]379worker1 # [ 0.033178] ACPI: Using ACPI (MADT) for SMP configuration information380worker2 # [ 0.034466] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()381client # [ 0.003096] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]382worker1 # [ 0.033179] ACPI: HPET id: 0x8086a201 base: 0xfed00000383lb # [ 0.002893] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]384worker1 # [ 0.033182] TSC deadline timer available385client # [ 0.003098] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]386worker2 # [ 0.034498] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]387client # [ 0.005855] Zone ranges:388worker1 # [ 0.033186] CPU topo: Max. logical packages: 1389lb # [ 0.002893] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]390client # [ 0.005856] DMA [mem 0x0000000000001000-0x0000000000ffffff]391worker2 # [ 0.034499] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]392worker1 # [ 0.033187] CPU topo: Max. logical dies: 1393lb # [ 0.002894] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]394worker2 # [ 0.034501] [mem 0x60000000-0xafffffff] available for PCI devices395client # [ 0.005858] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]396worker1 # [ 0.033187] CPU topo: Max. dies per package: 1397lb # [ 0.003126] No NUMA configuration found398client # [ 0.005859] Normal empty399worker1 # [ 0.033191] CPU topo: Max. threads per core: 1400worker2 # [ 0.034503] Booting paravirtualized kernel on KVM401client # [ 0.005860] Device empty402lb # [ 0.003127] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]403worker1 # [ 0.033191] CPU topo: Num. cores per package: 1404client # [ 0.005860] Movable zone start for each node405worker1 # [ 0.033192] CPU topo: Num. threads per package: 1406client # [ 0.005861] Early memory node ranges407worker2 # [ 0.034506] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns408lb # [ 0.003129] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]409lb # [ 0.005525] Zone ranges:410worker1 # [ 0.033192] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs411client # [ 0.005862] node 0: [mem 0x0000000000001000-0x000000000009efff]412worker2 # [ 0.038972] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1413lb # [ 0.005527] DMA [mem 0x0000000000001000-0x0000000000ffffff]414client # [ 0.005863] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]415worker1 # [ 0.033209] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()416worker2 # [ 0.041248] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152417lb # [ 0.005528] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]418lb # [ 0.005529] Normal empty419worker2 # [ 0.041293] kvm-guest: PV spinlocks disabled, single CPU420worker1 # [ 0.033239] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]421lb # [ 0.005530] Device empty422client # [ 0.005864] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]423lb # [ 0.005531] Movable zone start for each node424client # [ 0.005888] On node 0, zone DMA: 1 pages in unavailable ranges425worker1 # [ 0.033247] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]426lb # [ 0.005532] Early memory node ranges427client # [ 0.006168] On node 0, zone DMA: 97 pages in unavailable ranges428worker1 # [ 0.033249] [mem 0x60000000-0xafffffff] available for PCI devices429lb # [ 0.005532] node 0: [mem 0x0000000000001000-0x000000000009efff]430client # [ 0.034506] On node 0, zone DMA32: 40 pages in unavailable ranges431worker1 # [ 0.033250] Booting paravirtualized kernel on KVM432client # [ 0.034971] ACPI: PM-Timer IO Port: 0x608433lb # [ 0.005533] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]434client # [ 0.034981] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])435lb # [ 0.005534] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]436worker1 # [ 0.033253] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns437worker2 # [ 0.041295] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/q6x897783vrx1j21yalizihcz5pshnmx-nixos-system-worker2-test/init regInfo=/nix/store/sdbl9k12km9rsxdcvhj0g4kca5f47ysa-closure-info/registration console=ttyS0,115200n8 console=tty0438client # [ 0.035012] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23439lb # [ 0.005555] On node 0, zone DMA: 1 pages in unavailable ranges440worker1 # [ 0.037737] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1441client # [ 0.035014] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)442lb # [ 0.005833] On node 0, zone DMA: 97 pages in unavailable ranges443worker1 # [ 0.039912] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152444worker2 # [ 0.041399] Unknown kernel command line parameters "regInfo=/nix/store/sdbl9k12km9rsxdcvhj0g4kca5f47ysa-closure-info/registration", will be passed to user space.445client # [ 0.035016] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)446lb # [ 0.034049] On node 0, zone DMA32: 40 pages in unavailable ranges447worker1 # [ 0.039955] kvm-guest: PV spinlocks disabled, single CPU448lb # [ 0.034515] ACPI: PM-Timer IO Port: 0x608449worker2 # [ 0.041411] random: crng init done450client # [ 0.035017] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)451lb # [ 0.034527] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])452worker2 # [ 0.041412] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes453client # [ 0.035019] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)454lb # [ 0.034556] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23455worker2 # [ 0.043642] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)456client # [ 0.035019] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)457lb # [ 0.034559] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)458worker2 # [ 0.045069] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)459client # [ 0.035022] ACPI: Using ACPI (MADT) for SMP configuration information460lb # [ 0.034561] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)461worker2 # [ 0.045101] Fallback order for Node 0: 0462client # [ 0.035023] ACPI: HPET id: 0x8086a201 base: 0xfed00000463worker1 # [ 0.039957] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/m0k8shb6bvnyyb7vss69gd05kh2ps8cp-nixos-system-worker1-test/init regInfo=/nix/store/9pdhxfrd2i3gvvk5rwjv04jkk85gy654-closure-info/registration console=ttyS0,115200n8 console=tty0464client # [ 0.035026] TSC deadline timer available465lb # [ 0.034562] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)466worker2 # [ 0.045104] Built 1 zonelists, mobility grouping on. Total pages: 393078467client # [ 0.035030] CPU topo: Max. logical packages: 1468worker2 # [ 0.045105] Policy zone: DMA32469lb # [ 0.034563] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)470client # [ 0.035031] CPU topo: Max. logical dies: 1471worker2 # [ 0.047719] mem auto-init: stack:all(zero), heap alloc:on, heap free:off472worker1 # [ 0.040049] Unknown kernel command line parameters "regInfo=/nix/store/9pdhxfrd2i3gvvk5rwjv04jkk85gy654-closure-info/registration", will be passed to user space.473lb # [ 0.034564] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)474client # [ 0.035031] CPU topo: Max. dies per package: 1475worker1 # [ 0.040061] random: crng init done476worker2 # [ 0.051040] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1477client # [ 0.035034] CPU topo: Max. threads per core: 1478lb # [ 0.034566] ACPI: Using ACPI (MADT) for SMP configuration information479worker2 # [ 0.054520] allocated 3145728 bytes of page_ext480worker1 # [ 0.040062] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes481client # [ 0.035035] CPU topo: Num. cores per package: 1482lb # [ 0.034567] ACPI: HPET id: 0x8086a201 base: 0xfed00000483worker2 # [ 0.064764] ftrace: allocating 48731 entries in 192 pages484client # [ 0.035035] CPU topo: Num. threads per package: 1485lb # [ 0.034570] TSC deadline timer available486worker1 # [ 0.042489] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)487worker2 # [ 0.064767] ftrace: allocated 192 pages with 2 groups488lb # [ 0.034574] CPU topo: Max. logical packages: 1489client # [ 0.035036] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs490worker2 # [ 0.065643] Dynamic Preempt: lazy491worker1 # [ 0.043557] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)492lb # [ 0.034575] CPU topo: Max. logical dies: 1493client # [ 0.035055] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()494worker2 # [ 0.065780] rcu: Preemptible hierarchical RCU implementation.495worker1 # [ 0.043588] Fallback order for Node 0: 0496lb # [ 0.034575] CPU topo: Max. dies per package: 1497worker2 # [ 0.065781] rcu: RCU event tracing is enabled.498lb # [ 0.034579] CPU topo: Max. threads per core: 1499worker1 # [ 0.043591] Built 1 zonelists, mobility grouping on. Total pages: 393078500client # [ 0.035088] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]501worker1 # [ 0.043592] Policy zone: DMA32502lb # [ 0.034579] CPU topo: Num. cores per package: 1503worker2 # [ 0.065781] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.504client # [ 0.035090] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]505lb # [ 0.034580] CPU topo: Num. threads per package: 1506worker2 # [ 0.065783] Trampoline variant of Tasks RCU enabled.507worker1 # [ 0.046087] mem auto-init: stack:all(zero), heap alloc:on, heap free:off508worker2 # [ 0.065784] Rude variant of Tasks RCU enabled.509client # [ 0.035091] [mem 0x60000000-0xafffffff] available for PCI devices510lb # [ 0.034580] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs511worker1 # [ 0.049216] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1512client # [ 0.035093] Booting paravirtualized kernel on KVM513worker2 # [ 0.065784] Tracing variant of Tasks RCU enabled.514worker1 # [ 0.052838] allocated 3145728 bytes of page_ext515lb # [ 0.034600] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()516worker1 # [ 0.062552] ftrace: allocating 48731 entries in 192 pages517worker2 # [ 0.065785] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.518client # [ 0.035096] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns519lb # [ 0.034633] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]520worker1 # [ 0.062554] ftrace: allocated 192 pages with 2 groups521worker2 # [ 0.065786] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1522worker1 # [ 0.063401] Dynamic Preempt: lazy523client # [ 0.039557] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1524lb # [ 0.034635] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]525worker1 # [ 0.063524] rcu: Preemptible hierarchical RCU implementation.526worker2 # [ 0.065846] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.527client # [ 0.040607] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152528lb # [ 0.034636] [mem 0x60000000-0xafffffff] available for PCI devices529worker1 # [ 0.063524] rcu: RCU event tracing is enabled.530lb # [ 0.034638] Booting paravirtualized kernel on KVM531client # [ 0.040655] kvm-guest: PV spinlocks disabled, single CPU532worker2 # [ 0.065848] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.533worker1 # [ 0.063525] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.534worker1 # [ 0.063526] Trampoline variant of Tasks RCU enabled.535lb # [ 0.034641] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns536worker2 # [ 0.065849] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.537worker1 # [ 0.063527] Rude variant of Tasks RCU enabled.538worker2 # [ 0.070625] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16539worker1 # [ 0.063527] Tracing variant of Tasks RCU enabled.540lb # [ 0.039168] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1541lb # [ 0.041454] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152542worker2 # [ 0.070912] rcu: srcu_init: Setting srcu_struct sizes based on contention.543worker1 # [ 0.063528] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.544lb # [ 0.041504] kvm-guest: PV spinlocks disabled, single CPU545client # [ 0.040656] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/lgjx0fgzgnqvb086gw02890w1401r9xx-nixos-system-client-test/init regInfo=/nix/store/9f9i8jsph8f3c9h91zzs864hg65vz5v7-closure-info/registration console=ttyS0,115200n8 console=tty0546worker1 # [ 0.063529] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1547worker2 # [ 0.070920] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns548worker1 # [ 0.063584] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.549worker2 # [ 0.071030] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)550client # [ 0.040758] Unknown kernel command line parameters "regInfo=/nix/store/9f9i8jsph8f3c9h91zzs864hg65vz5v7-closure-info/registration", will be passed to user space.551worker2 # [ 0.074574] Console: colour VGA+ 80x25552client # [ 0.040771] random: crng init done553worker1 # [ 0.063585] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.554worker2 # [ 0.074577] printk: legacy console [tty0] enabled555client # [ 0.040772] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes556worker2 # [ 0.104000] printk: legacy console [ttyS0] enabled557worker1 # [ 0.063586] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.558lb # [ 0.041506] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/fj20bcn47ir2axl7b7sxwa0fvig2ds2j-nixos-system-lb-test/init regInfo=/nix/store/l5iprc9wdxb7nl0q1bx2grj7psaadl34-closure-info/registration console=ttyS0,115200n8 console=tty0559client # [ 0.040894] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)560worker1 # [ 0.067929] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16561worker2 # [ 0.200227] ACPI: Core revision 20250807562worker1 # [ 0.068197] rcu: srcu_init: Setting srcu_struct sizes based on contention.563client # [ 0.042033] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)564lb # [ 0.041603] Unknown kernel command line parameters "regInfo=/nix/store/l5iprc9wdxb7nl0q1bx2grj7psaadl34-closure-info/registration", will be passed to user space.565client # [ 0.042063] Fallback order for Node 0: 0566worker2 # [ 0.201137] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns567lb # [ 0.041615] random: crng init done568worker1 # [ 0.068204] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns569client # [ 0.042066] Built 1 zonelists, mobility grouping on. Total pages: 393078570worker2 # [ 0.202634] APIC: Switch to symmetric I/O mode setup571lb # [ 0.041616] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes572client # [ 0.042067] Policy zone: DMA32573worker1 # [ 0.068309] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)574worker2 # [ 0.203572] x2apic enabled575client # [ 0.042513] mem auto-init: stack:all(zero), heap alloc:on, heap free:off576lb # [ 0.043848] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)577worker1 # [ 0.071813] Console: colour VGA+ 80x25578client # [ 0.045834] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1579worker1 # [ 0.071816] printk: legacy console [tty0] enabled580worker2 # [ 0.204323] APIC: Switched APIC routing to: physical x2apic581lb # [ 0.045247] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)582client # [ 0.049359] allocated 3145728 bytes of page_ext583worker1 # [ 0.101133] printk: legacy console [ttyS0] enabled584lb # [ 0.045281] Fallback order for Node 0: 0585client # [ 0.059966] ftrace: allocating 48731 entries in 192 pages586worker1 # [ 0.196553] ACPI: Core revision 20250807587lb # [ 0.045284] Built 1 zonelists, mobility grouping on. Total pages: 393078588client # [ 0.059968] ftrace: allocated 192 pages with 2 groups589lb # [ 0.045285] Policy zone: DMA32590worker2 # [ 0.206113] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1591client # [ 0.060838] Dynamic Preempt: lazy592lb # [ 0.047899] mem auto-init: stack:all(zero), heap alloc:on, heap free:off593client # [ 0.061022] rcu: Preemptible hierarchical RCU implementation.594worker1 # [ 0.197470] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns595client # [ 0.061022] rcu: RCU event tracing is enabled.596lb # [ 0.051184] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1597worker2 # [ 0.207061] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns598worker1 # [ 0.198948] APIC: Switch to symmetric I/O mode setup599lb # [ 0.054667] allocated 3145728 bytes of page_ext600client # [ 0.061023] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.601worker1 # [ 0.199921] x2apic enabled602client # [ 0.061025] Trampoline variant of Tasks RCU enabled.603lb # [ 0.064968] ftrace: allocating 48731 entries in 192 pages604worker2 # [ 0.208661] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)605lb # [ 0.064969] ftrace: allocated 192 pages with 2 groups606client # [ 0.061025] Rude variant of Tasks RCU enabled.607lb # [ 0.065855] Dynamic Preempt: lazy608worker1 # [ 0.200694] APIC: Switched APIC routing to: physical x2apic609client # [ 0.061026] Tracing variant of Tasks RCU enabled.610worker2 # [ 0.209748] x86/cpu: User Mode Instruction Prevention (UMIP) activated611lb # [ 0.066006] rcu: Preemptible hierarchical RCU implementation.612client # [ 0.061026] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.613lb # [ 0.066006] rcu: RCU event tracing is enabled.614worker2 # [ 0.210789] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127615client # [ 0.061027] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1616lb # [ 0.066007] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.617worker1 # [ 0.202486] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1618worker2 # [ 0.211659] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0619lb # [ 0.066009] Trampoline variant of Tasks RCU enabled.620client # [ 0.061043] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.621lb # [ 0.066009] Rude variant of Tasks RCU enabled.622worker1 # [ 0.203448] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns623lb # [ 0.066010] Tracing variant of Tasks RCU enabled.624client # [ 0.061045] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.625worker2 # [ 0.212662] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto626lb # [ 0.066010] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.627worker1 # [ 0.205074] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)628client # [ 0.061046] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.629lb # [ 0.066011] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1630worker2 # [ 0.213659] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl631client # [ 0.065490] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16632worker1 # [ 0.206159] x86/cpu: User Mode Instruction Prevention (UMIP) activated633lb # [ 0.066063] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.634worker2 # [ 0.214659] Transient Scheduler Attacks: Vulnerable: No microcode635client # [ 0.065781] rcu: srcu_init: Setting srcu_struct sizes based on contention.636worker1 # [ 0.207199] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127637worker2 # [ 0.215658] Spectre V2 : Mitigation: Enhanced / Automatic IBRS638lb # [ 0.066065] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.639client # [ 0.065790] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns640worker1 # [ 0.208072] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0641worker2 # [ 0.216658] Speculative Return Stack Overflow: Mitigation: Safe RET642lb # [ 0.066065] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.643client # [ 0.065896] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)644lb # [ 0.070489] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16645client # [ 0.069454] Console: colour VGA+ 80x25646client # [ 0.069457] printk: legacy console [tty0] enabled647worker1 # [ 0.209074] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto648lb # [ 0.071057] rcu: srcu_init: Setting srcu_struct sizes based on contention.649client # [ 0.099063] printk: legacy console [ttyS0] enabled650client # [ 0.194333] ACPI: Core revision 20250807651lb # [ 0.071064] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns652worker1 # [ 0.211071] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl653worker2 # [ 0.217658] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization654worker1 # [ 0.212071] Transient Scheduler Attacks: Vulnerable: No microcode655lb # [ 0.071173] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)656lb # [ 0.074707] Console: colour VGA+ 80x25657client # [ 0.195266] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns658worker2 # [ 0.219666] Spectre V2 : Enabling IBPB for BPF659worker1 # [ 0.213071] Spectre V2 : Mitigation: Enhanced / Automatic IBRS660lb # [ 0.074711] printk: legacy console [tty0] enabled661client # [ 0.196760] APIC: Switch to symmetric I/O mode setup662lb # [ 0.104108] printk: legacy console [ttyS0] enabled663worker2 # [ 0.221171] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier664worker1 # [ 0.214071] Speculative Return Stack Overflow: Mitigation: Safe RET665client # [ 0.197730] x2apic enabled666lb # [ 0.199502] ACPI: Core revision 20250807667worker2 # [ 0.222659] active return thunk: srso_alias_return_thunk668client # [ 0.198501] APIC: Switched APIC routing to: physical x2apic669worker2 # [ 0.223547] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'670lb # [ 0.200423] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns671lb # [ 0.201956] APIC: Switch to symmetric I/O mode setup672worker2 # [ 0.224658] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'673client # [ 0.200371] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1674lb # [ 0.202921] x2apic enabled675worker1 # [ 0.215071] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization676worker2 # [ 0.225659] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'677worker1 # [ 0.217078] Spectre V2 : Enabling IBPB for BPF678lb # [ 0.203692] APIC: Switched APIC routing to: physical x2apic679client # [ 0.201327] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns680worker2 # [ 0.226659] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'681worker2 # [ 0.227658] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'682worker1 # [ 0.218072] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier683client # [ 0.202972] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)684lb # [ 0.205469] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1685worker1 # [ 0.219072] active return thunk: srso_alias_return_thunk686worker2 # [ 0.228602] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'687client # [ 0.204058] x86/cpu: User Mode Instruction Prevention (UMIP) activated688worker1 # [ 0.220093] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'689worker2 # [ 0.229419] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'690lb # [ 0.206424] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns691client # [ 0.205095] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127692worker1 # [ 0.221071] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'693worker2 # [ 0.230659] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256694client # [ 0.205970] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0695lb # [ 0.208030] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)696worker2 # [ 0.231658] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64697worker1 # [ 0.222071] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'698lb # [ 0.209123] x86/cpu: User Mode Instruction Prevention (UMIP) activated699worker2 # [ 0.232658] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512700worker1 # [ 0.223071] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'701client # [ 0.206974] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto702lb # [ 0.210148] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127703worker2 # [ 0.233569] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024704worker1 # [ 0.224071] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'705client # [ 0.208970] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl706lb # [ 0.211028] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0707worker2 # [ 0.234356] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8708worker1 # [ 0.225071] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'709client # [ 0.209970] Transient Scheduler Attacks: Vulnerable: No microcode710worker1 # [ 0.226071] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'711worker2 # [ 0.235353] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.712lb # [ 0.212031] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto713client # [ 0.210969] Spectre V2 : Mitigation: Enhanced / Automatic IBRS714worker1 # [ 0.227071] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256715client # [ 0.211970] Speculative Return Stack Overflow: Mitigation: Safe RET716lb # [ 0.213028] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl717worker1 # [ 0.228793] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64718lb # [ 0.214028] Transient Scheduler Attacks: Vulnerable: No microcode719worker1 # [ 0.229071] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512720lb # [ 0.215027] Spectre V2 : Mitigation: Enhanced / Automatic IBRS721worker1 # [ 0.230071] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024722lb # [ 0.216027] Speculative Return Stack Overflow: Mitigation: Safe RET723worker1 # [ 0.231071] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8724client # [ 0.212969] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization725client # [ 0.214979] Spectre V2 : Enabling IBPB for BPF726worker1 # [ 0.232071] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.727client # [ 0.215970] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier728client # [ 0.216970] active return thunk: srso_alias_return_thunk729lb # [ 0.217027] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization730lb # [ 0.219035] Spectre V2 : Enabling IBPB for BPF731client # [ 0.217993] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'732client # [ 0.218969] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'733lb # [ 0.220028] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier734lb # [ 0.221028] active return thunk: srso_alias_return_thunk735client # [ 0.220744] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'736client # [ 0.221747] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'737lb # [ 0.222050] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'738client # [ 0.222748] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'739lb # [ 0.223027] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'740client # [ 0.223737] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'741lb # [ 0.224027] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'742lb # [ 0.225027] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'743client # [ 0.224786] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'744lb # [ 0.226028] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'745client # [ 0.225970] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256746lb # [ 0.227027] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'747client # [ 0.226969] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64748client # [ 0.227969] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512749lb # [ 0.228028] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'750client # [ 0.228969] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024751lb # [ 0.229028] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256752lb # [ 0.230751] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64753client # [ 0.229969] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8754lb # [ 0.231764] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512755client # [ 0.230970] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.756lb # [ 0.232765] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024757lb # [ 0.233760] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8758lb # [ 0.234027] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759worker2 # [ 0.270044] Freeing SMP alternatives memory: 44K760worker2 # [ 0.270662] pid_max: default: 32768 minimum: 301761worker2 # [ 0.271682] LSM: initializing lsm=capability,landlock,yama,bpf,ima762worker2 # [ 0.272759] landlock: Up and running.763worker2 # [ 0.273379] Yama: becoming mindful.764worker2 # [ 0.273874] LSM support for eBPF active765worker2 # [ 0.274770] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)766worker1 # [ 0.266980] Freeing SMP alternatives memory: 44K767worker2 # [ 0.275704] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)768worker1 # [ 0.267628] pid_max: default: 32768 minimum: 301769worker1 # [ 0.268170] LSM: initializing lsm=capability,landlock,yama,bpf,ima770worker1 # [ 0.269172] landlock: Up and running.771worker2 # [ 0.278509] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)772worker1 # [ 0.270072] Yama: becoming mindful.773worker1 # [ 0.271157] LSM support for eBPF active774worker2 # [ 0.279202] Performance Events: Fam17h+ core perfctr, AMD PMU driver.775worker1 # [ 0.271910] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)776worker2 # [ 0.279663] ... version: 2777worker2 # [ 0.280347] ... bit width: 48778worker1 # [ 0.273109] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)779worker2 # [ 0.280662] ... generic counters: 6780worker2 # [ 0.281355] ... generic bitmap: 000000000000003f781worker2 # [ 0.281717] ... fixed-purpose counters: 0782worker1 # [ 0.275362] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)783worker2 # [ 0.282405] ... fixed-purpose bitmap: 0000000000000000784worker2 # [ 0.282662] ... value mask: 0000ffffffffffff785worker1 # [ 0.276632] Performance Events: Fam17h+ core perfctr, AMD PMU driver.786worker2 # [ 0.283537] ... max period: 00007fffffffffff787worker1 # [ 0.277080] ... version: 2788worker2 # [ 0.284337] ... global_ctrl mask: 000000000000003f789worker1 # [ 0.277785] ... bit width: 48790worker2 # [ 0.284790] signal: max sigframe size: 3376791worker1 # [ 0.278073] ... generic counters: 6792worker2 # [ 0.285554] rcu: Hierarchical SRCU implementation.793worker1 # [ 0.278837] ... generic bitmap: 000000000000003f794worker2 # [ 0.286229] rcu: Max phase no-delay instances is 400.795worker1 # [ 0.279073] ... fixed-purpose counters: 0796worker1 # [ 0.279780] ... fixed-purpose bitmap: 0000000000000000797worker1 # [ 0.280073] ... value mask: 0000ffffffffffff798client # [ 0.266078] Freeing SMP alternatives memory: 44K799worker1 # [ 0.280941] ... max period: 00007fffffffffff800client # [ 0.266874] pid_max: default: 32768 minimum: 301801worker1 # [ 0.281755] ... global_ctrl mask: 000000000000003f802client # [ 0.267626] LSM: initializing lsm=capability,landlock,yama,bpf,ima803worker1 # [ 0.282181] signal: max sigframe size: 3376804client # [ 0.268831] landlock: Up and running.805client # [ 0.269416] Yama: becoming mindful.806worker2 # [ 0.291235] smp: Bringing up secondary CPUs ...807worker1 # [ 0.282989] rcu: Hierarchical SRCU implementation.808lb # [ 0.270049] Freeing SMP alternatives memory: 44K809worker2 # [ 0.291675] smp: Brought up 1 node, 1 CPU810worker1 # [ 0.283659] rcu: Max phase no-delay instances is 400.811client # [ 0.270185] LSM support for eBPF active812lb # [ 0.270820] pid_max: default: 32768 minimum: 301813worker2 # [ 0.292316] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)814lb # [ 0.271141] LSM: initializing lsm=capability,landlock,yama,bpf,ima815client # [ 0.271077] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)816lb # [ 0.272148] landlock: Up and running.817lb # [ 0.273028] Yama: becoming mindful.818client # [ 0.272010] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)819worker2 # [ 0.292809] Memory: 1454544K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)820lb # [ 0.274237] LSM support for eBPF active821worker2 # [ 0.293916] devtmpfs: initialized822lb # [ 0.275012] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)823worker2 # [ 0.294760] x86/mm: Memory block size: 128MB824client # [ 0.274539] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)825worker1 # [ 0.288609] smp: Bringing up secondary CPUs ...826worker1 # [ 0.289088] smp: Brought up 1 node, 1 CPU827lb # [ 0.276067] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)828client # [ 0.275528] Performance Events: Fam17h+ core perfctr, AMD PMU driver.829worker1 # [ 0.289783] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)830worker2 # [ 0.296527] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)831client # [ 0.275974] ... version: 2832client # [ 0.276705] ... bit width: 48833lb # [ 0.278546] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)834worker2 # [ 0.297527] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).835client # [ 0.276972] ... generic counters: 6836worker1 # [ 0.290243] Memory: 1454572K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)837worker2 # [ 0.298750] pinctrl core: initialized pinctrl subsystem838client # [ 0.277645] ... generic bitmap: 000000000000003f839worker1 # [ 0.291328] devtmpfs: initialized840lb # [ 0.279584] Performance Events: Fam17h+ core perfctr, AMD PMU driver.841client # [ 0.277983] ... fixed-purpose counters: 0842worker2 # [ 0.299868] PM: RTC time: 12:32:36, date: 2026-09-21843lb # [ 0.280032] ... version: 2844worker1 # [ 0.292259] x86/mm: Memory block size: 128MB845client # [ 0.278711] ... fixed-purpose bitmap: 0000000000000000846lb # [ 0.280740] ... bit width: 48847lb # [ 0.281030] ... generic counters: 6848client # [ 0.278972] ... value mask: 0000ffffffffffff849lb # [ 0.281730] ... generic bitmap: 000000000000003f850client # [ 0.279874] ... max period: 00007fffffffffff851worker1 # [ 0.294054] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)852lb # [ 0.282042] ... fixed-purpose counters: 0853client # [ 0.280654] ... global_ctrl mask: 000000000000003f854worker2 # [ 0.303585] NET: Registered PF_NETLINK/PF_ROUTE protocol family855lb # [ 0.282747] ... fixed-purpose bitmap: 0000000000000000856worker1 # [ 0.294972] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).857client # [ 0.281101] signal: max sigframe size: 3376858lb # [ 0.283030] ... value mask: 0000ffffffffffff859client # [ 0.281841] rcu: Hierarchical SRCU implementation.860worker1 # [ 0.296145] pinctrl core: initialized pinctrl subsystem861worker2 # [ 0.304868] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations862lb # [ 0.283875] ... max period: 00007fffffffffff863client # [ 0.282513] rcu: Max phase no-delay instances is 400.864worker1 # [ 0.297178] PM: RTC time: 12:32:36, date: 2026-09-21865worker2 # [ 0.305689] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations866lb # [ 0.284669] ... global_ctrl mask: 000000000000003f867lb # [ 0.285154] signal: max sigframe size: 3376868worker2 # [ 0.306955] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations869lb # [ 0.285958] rcu: Hierarchical SRCU implementation.870worker2 # [ 0.307672] audit: initializing netlink subsys (disabled)871lb # [ 0.286630] rcu: Max phase no-delay instances is 400.872worker1 # [ 0.300725] NET: Registered PF_NETLINK/PF_ROUTE protocol family873worker2 # [ 0.308663] audit: type=2000 audit(1789993957.131:1): state=initialized audit_enabled=0 res=1874client # [ 0.287693] smp: Bringing up secondary CPUs ...875worker1 # [ 0.301569] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations876worker2 # [ 0.309820] thermal_sys: Registered thermal governor 'fair_share'877client # [ 0.287985] smp: Brought up 1 node, 1 CPU878worker2 # [ 0.309822] thermal_sys: Registered thermal governor 'bang_bang'879worker1 # [ 0.302096] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations880client # [ 0.288645] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)881worker2 # [ 0.310663] thermal_sys: Registered thermal governor 'step_wise'882worker1 # [ 0.303353] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations883worker2 # [ 0.311578] thermal_sys: Registered thermal governor 'user_space'884lb # [ 0.291746] smp: Bringing up secondary CPUs ...885worker1 # [ 0.304083] audit: initializing netlink subsys (disabled)886client # [ 0.289162] Memory: 1454528K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)887worker2 # [ 0.312379] thermal_sys: Registered thermal governor 'power_allocator'888lb # [ 0.292044] smp: Brought up 1 node, 1 CPU889worker2 # [ 0.312674] cpuidle: using governor menu890client # [ 0.290253] devtmpfs: initialized891worker1 # [ 0.305147] thermal_sys: Registered thermal governor 'fair_share'892lb # [ 0.292702] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)893client # [ 0.291031] x86/mm: Memory block size: 128MB894worker1 # [ 0.305149] thermal_sys: Registered thermal governor 'bang_bang'895worker2 # [ 0.314685] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5896worker1 # [ 0.306074] thermal_sys: Registered thermal governor 'step_wise'897lb # [ 0.293284] Memory: 1454584K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)898worker1 # [ 0.307021] thermal_sys: Registered thermal governor 'user_space'899worker2 # [ 0.315898] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]900client # [ 0.292856] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)901lb # [ 0.294288] devtmpfs: initialized902worker2 # [ 0.316665] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry903lb # [ 0.295091] x86/mm: Memory block size: 128MB904worker1 # [ 0.307788] audit: type=2000 audit(1789993957.137:1): state=initialized audit_enabled=0 res=1905client # [ 0.293818] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).906worker2 # [ 0.317674] PCI: Using configuration type 1 for base access907worker1 # [ 0.309077] thermal_sys: Registered thermal governor 'power_allocator'908client # [ 0.295064] pinctrl core: initialized pinctrl subsystem909worker1 # [ 0.309090] cpuidle: using governor menu910worker2 # [ 0.318728] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.911lb # [ 0.296864] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)912client # [ 0.296219] PM: RTC time: 12:32:36, date: 2026-09-21913worker1 # [ 0.311273] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5914lb # [ 0.297918] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).915lb # [ 0.299123] pinctrl core: initialized pinctrl subsystem916worker1 # [ 0.312305] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]917lb # [ 0.300222] PM: RTC time: 12:32:36, date: 2026-09-21918worker1 # [ 0.313077] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry919client # [ 0.299876] NET: Registered PF_NETLINK/PF_ROUTE protocol family920worker1 # [ 0.314085] PCI: Using configuration type 1 for base access921client # [ 0.301200] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations922worker1 # [ 0.315226] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.923worker2 # [ 0.325688] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages924client # [ 0.301996] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations925lb # [ 0.303884] NET: Registered PF_NETLINK/PF_ROUTE protocol family926worker2 # [ 0.326662] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page927client # [ 0.303264] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations928lb # [ 0.305280] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations929client # [ 0.303981] audit: initializing netlink subsys (disabled)930lb # [ 0.306056] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations931client # [ 0.305089] thermal_sys: Registered thermal governor 'fair_share'932client # [ 0.305091] thermal_sys: Registered thermal governor 'bang_bang'933lb # [ 0.307323] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations934worker1 # [ 0.320341] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages935client # [ 0.305973] thermal_sys: Registered thermal governor 'step_wise'936lb # [ 0.308039] audit: initializing netlink subsys (disabled)937worker1 # [ 0.321076] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page938worker2 # [ 0.331662] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages939client # [ 0.306898] thermal_sys: Registered thermal governor 'user_space'940lb # [ 0.309244] thermal_sys: Registered thermal governor 'fair_share'941worker2 # [ 0.332662] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page942lb # [ 0.309246] thermal_sys: Registered thermal governor 'bang_bang'943client # [ 0.307672] audit: type=2000 audit(1789993957.154:1): state=initialized audit_enabled=0 res=1944lb # [ 0.310031] thermal_sys: Registered thermal governor 'step_wise'945client # [ 0.308974] thermal_sys: Registered thermal governor 'power_allocator'946client # [ 0.308988] cpuidle: using governor menu947lb # [ 0.310955] thermal_sys: Registered thermal governor 'user_space'948worker1 # [ 0.326075] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages949lb # [ 0.311751] audit: type=2000 audit(1789993957.157:1): state=initialized audit_enabled=0 res=1950client # [ 0.311215] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5951worker1 # [ 0.327073] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page952lb # [ 0.313033] thermal_sys: Registered thermal governor 'power_allocator'953lb # [ 0.313046] cpuidle: using governor menu954client # [ 0.312211] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]955client # [ 0.312975] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry956lb # [ 0.315229] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5957client # [ 0.313984] PCI: Using configuration type 1 for base access958lb # [ 0.316274] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]959client # [ 0.315122] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.960lb # [ 0.317032] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry961lb # [ 0.318042] PCI: Using configuration type 1 for base access962worker2 # [ 0.343903] ACPI: Added _OSI(Module Device)963worker2 # [ 0.344623] ACPI: Added _OSI(Processor Device)964lb # [ 0.319139] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.965worker2 # [ 0.345201] ACPI: Added _OSI(Processor Aggregator Device)966worker1 # [ 0.337460] ACPI: Added _OSI(Module Device)967worker1 # [ 0.338075] ACPI: Added _OSI(Processor Device)968worker1 # [ 0.338849] ACPI: Added _OSI(Processor Aggregator Device)969client # [ 0.322286] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages970client # [ 0.322974] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page971worker2 # [ 0.351447] ACPI: 1 ACPI AML tables successfully acquired and loaded972lb # [ 0.326324] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages973lb # [ 0.327032] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page974worker2 # [ 0.355414] ACPI: Interpreter enabled975client # [ 0.327973] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages976worker2 # [ 0.355679] ACPI: PM: (supports S0 S3 S4 S5)977client # [ 0.328973] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page978worker2 # [ 0.356380] ACPI: Using IOAPIC for interrupt routing979lb # [ 0.332032] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages980worker2 # [ 0.356711] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug981worker1 # [ 0.347619] ACPI: 1 ACPI AML tables successfully acquired and loaded982lb # [ 0.333031] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page983worker1 # [ 0.351485] ACPI: Interpreter enabled984worker2 # [ 0.361662] PCI: Using E820 reservations for host bridge windows985worker1 # [ 0.352089] ACPI: PM: (supports S0 S3 S4 S5)986worker2 # [ 0.362740] ACPI: Enabled 2 GPEs in block 00 to 3F987worker1 # [ 0.352785] ACPI: Using IOAPIC for interrupt routing988client # [ 0.337575] ACPI: Added _OSI(Module Device)989client # [ 0.337974] ACPI: Added _OSI(Processor Device)990worker1 # [ 0.353134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug991client # [ 0.338712] ACPI: Added _OSI(Processor Aggregator Device)992worker1 # [ 0.354075] PCI: Using E820 reservations for host bridge windows993lb # [ 0.341627] ACPI: Added _OSI(Module Device)994lb # [ 0.342031] ACPI: Added _OSI(Processor Device)995lb # [ 0.342759] ACPI: Added _OSI(Processor Aggregator Device)996worker1 # [ 0.357210] ACPI: Enabled 2 GPEs in block 00 to 3F997worker2 # [ 0.370681] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])998worker2 # [ 0.371667] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]999client # [ 0.345528] ACPI: 1 ACPI AML tables successfully acquired and loaded1000worker2 # [ 0.372750] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1001worker2 # [ 0.373786] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1002lb # [ 0.349628] ACPI: 1 ACPI AML tables successfully acquired and loaded1003worker2 # [ 0.374996] PCI host bridge to bus 0000:001004client # [ 0.348927] ACPI: Interpreter enabled1005worker2 # [ 0.375666] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1006client # [ 0.349415] ACPI: PM: (supports S0 S3 S4 S5)1007worker1 # [ 0.365880] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1008client # [ 0.349973] ACPI: Using IOAPIC for interrupt routing1009worker2 # [ 0.376661] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1010lb # [ 0.352981] ACPI: Interpreter enabled1011worker1 # [ 0.366835] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1012worker2 # [ 0.377661] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1013lb # [ 0.353472] ACPI: PM: (supports S0 S3 S4 S5)1014client # [ 0.350818] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1015lb # [ 0.354031] ACPI: Using IOAPIC for interrupt routing1016worker2 # [ 0.378662] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1017worker1 # [ 0.368800] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1018worker2 # [ 0.379662] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1019lb # [ 0.354872] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1020worker1 # [ 0.370077] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1021worker2 # [ 0.380662] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1022worker1 # [ 0.371406] PCI host bridge to bus 0000:001023worker2 # [ 0.381662] pci_bus 0000:00: root bus resource [bus 00-ff]1024worker1 # [ 0.372078] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1025client # [ 0.355972] PCI: Using E820 reservations for host bridge windows1026worker2 # [ 0.382604] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1027worker1 # [ 0.373074] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1028client # [ 0.357041] ACPI: Enabled 2 GPEs in block 00 to 3F1029worker1 # [ 0.374074] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1030lb # [ 0.360030] PCI: Using E820 reservations for host bridge windows1031worker2 # [ 0.384073] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1032lb # [ 0.361100] ACPI: Enabled 2 GPEs in block 00 to 3F1033worker1 # [ 0.375074] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1034worker1 # [ 0.376074] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1035worker1 # [ 0.377079] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1036worker1 # [ 0.378099] pci_bus 0000:00: root bus resource [bus 00-ff]1037worker1 # [ 0.379131] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1038client # [ 0.365660] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1039worker1 # [ 0.380500] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1040worker2 # [ 0.386726] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1041lb # [ 0.369710] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1042client # [ 0.366622] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1043worker2 # [ 0.387673] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1044worker2 # [ 0.388610] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1045client # [ 0.367065] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1046lb # [ 0.370654] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1047client # [ 0.368097] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1048worker2 # [ 0.389834] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1049lb # [ 0.371107] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1050client # [ 0.369305] PCI host bridge to bus 0000:001051lb # [ 0.372158] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1052client # [ 0.369977] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1053worker2 # [ 0.391590] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1054lb # [ 0.373374] PCI host bridge to bus 0000:001055client # [ 0.370973] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1056lb # [ 0.374035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1057worker1 # [ 0.383158] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1058client # [ 0.371972] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1059lb # [ 0.375030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1060worker1 # [ 0.384086] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1061client # [ 0.372972] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1062worker2 # [ 0.394669] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1063worker1 # [ 0.385041] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1064lb # [ 0.376030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1065worker2 # [ 0.395485] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1066client # [ 0.373972] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1067lb # [ 0.377030] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1068worker2 # [ 0.396364] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1069worker1 # [ 0.386111] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1070client # [ 0.374972] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1071lb # [ 0.378030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1072worker2 # [ 0.396667] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1073client # [ 0.375973] pci_bus 0000:00: root bus resource [bus 00-ff]1074lb # [ 0.379030] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1075worker1 # [ 0.387793] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1076lb # [ 0.380030] pci_bus 0000:00: root bus resource [bus 00-ff]1077client # [ 0.376974] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1078worker2 # [ 0.398639] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1079lb # [ 0.380983] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1080client # [ 0.378445] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1081worker1 # [ 0.390105] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1082lb # [ 0.382449] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1083worker2 # [ 0.400670] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1084worker1 # [ 0.390991] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1085worker2 # [ 0.401492] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1086worker1 # [ 0.391831] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1087worker2 # [ 0.402446] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1088worker1 # [ 0.392080] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1089worker2 # [ 0.404033] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1090worker1 # [ 0.393608] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1091worker1 # [ 0.396100] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1092client # [ 0.381061] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1093worker1 # [ 0.396994] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1094worker2 # [ 0.406669] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1095client # [ 0.381984] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1096worker2 # [ 0.407668] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1097lb # [ 0.385112] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1098worker1 # [ 0.397816] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1099lb # [ 0.386042] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1100worker2 # [ 0.408587] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1101client # [ 0.382921] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1102worker1 # [ 0.398640] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1103lb # [ 0.386967] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1104worker2 # [ 0.410024] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1105client # [ 0.384395] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1106lb # [ 0.388335] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1107client # [ 0.385701] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1108worker2 # [ 0.411682] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1109lb # [ 0.389979] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1110worker2 # [ 0.412523] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1111worker1 # [ 0.401082] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1112worker1 # [ 0.402706] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1113worker2 # [ 0.413384] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1114client # [ 0.387997] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1115worker1 # [ 0.403096] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1116client # [ 0.388872] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1117lb # [ 0.393056] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1118worker2 # [ 0.414213] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1119client # [ 0.389747] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1120worker1 # [ 0.404649] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1121lb # [ 0.393892] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1122client # [ 0.390789] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1123lb # [ 0.394750] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1124worker2 # [ 0.415678] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1125worker1 # [ 0.406105] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1126lb # [ 0.395036] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1127client # [ 0.392562] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1128worker2 # [ 0.416518] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1129worker1 # [ 0.406960] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1130worker2 # [ 0.417368] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1131worker1 # [ 0.407818] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1132lb # [ 0.396633] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1133client # [ 0.393997] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1134worker2 # [ 0.418247] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1135client # [ 0.394867] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1136worker1 # [ 0.408606] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1137client # [ 0.395744] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1138lb # [ 0.398954] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1139lb # [ 0.399642] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1140client # [ 0.397346] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1141worker1 # [ 0.410081] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1142lb # [ 0.400052] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1143worker2 # [ 0.420678] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1144worker1 # [ 0.410937] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1145worker2 # [ 0.421669] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1146lb # [ 0.401612] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1147worker1 # [ 0.411845] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1148worker2 # [ 0.422603] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1149client # [ 0.399980] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1150worker1 # [ 0.413470] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1151worker2 # [ 0.424039] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1152client # [ 0.400824] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1153client # [ 0.401722] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1154lb # [ 0.404992] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1155worker2 # [ 0.425669] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1156lb # [ 0.405641] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1157client # [ 0.402558] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1158worker2 # [ 0.426608] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1159lb # [ 0.406052] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1160worker1 # [ 0.416221] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1161worker2 # [ 0.427395] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1162worker1 # [ 0.417087] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1163worker2 # [ 0.427668] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1164client # [ 0.403999] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1165lb # [ 0.407627] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1166worker1 # [ 0.418124] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1167client # [ 0.404850] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1168worker2 # [ 0.429188] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1169client # [ 0.405745] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1170worker1 # [ 0.419688] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1171lb # [ 0.410010] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1172lb # [ 0.410668] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1173worker2 # [ 0.431669] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1174client # [ 0.407542] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1175worker2 # [ 0.432629] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1176worker1 # [ 0.421083] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1177lb # [ 0.411059] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1178worker1 # [ 0.422008] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1179client # [ 0.408979] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1180worker2 # [ 0.434038] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1181client # [ 0.409801] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1182worker1 # [ 0.422908] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1183lb # [ 0.412592] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1184client # [ 0.410689] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1185worker1 # [ 0.424032] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1186worker2 # [ 0.435689] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1187lb # [ 0.414761] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1188client # [ 0.411566] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1189worker2 # [ 0.436542] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1190lb # [ 0.415037] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1191worker1 # [ 0.425669] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1192lb # [ 0.415945] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1193worker2 # [ 0.437366] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1194worker1 # [ 0.427110] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1195lb # [ 0.417448] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1196worker2 # [ 0.438218] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1197client # [ 0.414259] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1198worker1 # [ 0.428103] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1199client # [ 0.414979] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1200worker2 # [ 0.439668] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1201client # [ 0.415987] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1202worker1 # [ 0.429636] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1203worker2 # [ 0.440503] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1204lb # [ 0.420063] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1205worker2 # [ 0.441437] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1206client # [ 0.417594] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1207lb # [ 0.420869] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1208worker1 # [ 0.431774] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1209lb # [ 0.421734] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1210worker2 # [ 0.443057] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1211client # [ 0.418979] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1212worker1 # [ 0.432813] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1213client # [ 0.419793] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1214worker2 # [ 0.444290] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1215lb # [ 0.422601] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1216worker1 # [ 0.433101] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1217client # [ 0.420699] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1218client # [ 0.420979] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1219worker2 # [ 0.445611] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1220lb # [ 0.424723] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1221worker1 # [ 0.434648] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1222lb # [ 0.425037] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1223worker2 # [ 0.447331] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1224client # [ 0.423030] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1225lb # [ 0.425970] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1226worker1 # [ 0.436083] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1227client # [ 0.424750] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1228worker1 # [ 0.437075] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1229lb # [ 0.427030] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1230client # [ 0.425721] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1231worker1 # [ 0.438103] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1232lb # [ 0.428580] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1233client # [ 0.426522] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1234worker1 # [ 0.440563] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1235lb # [ 0.430062] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1236client # [ 0.428015] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1237lb # [ 0.431006] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1238worker1 # [ 0.441834] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1239client # [ 0.428922] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1240lb # [ 0.432372] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1241worker1 # [ 0.442398] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1242client # [ 0.429730] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1243worker1 # [ 0.443759] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1244client # [ 0.431416] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1245worker2 # [ 0.447875] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1246lb # [ 0.434063] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1247worker2 # [ 0.450319] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1248client # [ 0.433002] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1249lb # [ 0.435037] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1250client # [ 0.433860] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1251lb # [ 0.435950] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1252worker2 # [ 0.450898] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1253client # [ 0.434711] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1254worker2 # [ 0.452295] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1255lb # [ 0.437391] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1256client # [ 0.435529] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1257worker2 # [ 0.452930] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1258lb # [ 0.439709] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1259client # [ 0.436552] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1260worker2 # [ 0.454954] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1261lb # [ 0.440037] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1262client # [ 0.437181] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1263lb # [ 0.440973] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1264worker2 # [ 0.455931] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1265client # [ 0.438510] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1266worker2 # [ 0.457691] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1267lb # [ 0.442394] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1268worker1 # [ 0.444296] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1269worker2 # [ 0.458524] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1270lb # [ 0.443829] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1271worker1 # [ 0.447032] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1272worker2 # [ 0.459721] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1273lb # [ 0.444854] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1274worker1 # [ 0.448103] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1275worker2 # [ 0.461318] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1276lb # [ 0.446690] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1277worker1 # [ 0.449755] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1278worker1 # [ 0.450356] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1279worker2 # [ 0.464613] ACPI: PCI: Interrupt link LNKA configured for IRQ 101280worker1 # [ 0.451372] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1281worker2 # [ 0.465544] ACPI: PCI: Interrupt link LNKB configured for IRQ 101282worker2 # [ 0.466506] ACPI: PCI: Interrupt link LNKC configured for IRQ 111283worker1 # [ 0.452358] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1284worker2 # [ 0.467487] ACPI: PCI: Interrupt link LNKD configured for IRQ 111285client # [ 0.439180] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1286worker2 # [ 0.468476] ACPI: PCI: Interrupt link LNKE configured for IRQ 101287client # [ 0.441523] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1288worker1 # [ 0.454569] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1289worker2 # [ 0.469512] ACPI: PCI: Interrupt link LNKF configured for IRQ 101290worker1 # [ 0.455083] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1291client # [ 0.442228] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1292worker2 # [ 0.470488] ACPI: PCI: Interrupt link LNKG configured for IRQ 111293worker2 # [ 0.471759] ACPI: PCI: Interrupt link LNKH configured for IRQ 111294worker1 # [ 0.456408] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1295client # [ 0.443492] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1296worker2 # [ 0.472705] ACPI: PCI: Interrupt link GSIA configured for IRQ 161297lb # [ 0.447222] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1298worker1 # [ 0.457763] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1299client # [ 0.444256] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1300worker2 # [ 0.473677] ACPI: PCI: Interrupt link GSIB configured for IRQ 171301worker2 # [ 0.474610] ACPI: PCI: Interrupt link GSIC configured for IRQ 181302lb # [ 0.449776] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1303client # [ 0.445258] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1304worker2 # [ 0.475429] ACPI: PCI: Interrupt link GSID configured for IRQ 191305worker2 # [ 0.475676] ACPI: PCI: Interrupt link GSIE configured for IRQ 201306lb # [ 0.450305] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1307client # [ 0.446227] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1308worker2 # [ 0.476625] ACPI: PCI: Interrupt link GSIF configured for IRQ 211309lb # [ 0.451709] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1310worker1 # [ 0.461079] ACPI: PCI: Interrupt link LNKA configured for IRQ 101311client # [ 0.447772] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1312worker2 # [ 0.477448] ACPI: PCI: Interrupt link GSIG configured for IRQ 221313client # [ 0.448620] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1314worker2 # [ 0.478378] ACPI: PCI: Interrupt link GSIH configured for IRQ 231315worker1 # [ 0.462210] ACPI: PCI: Interrupt link LNKB configured for IRQ 101316lb # [ 0.452259] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1317worker1 # [ 0.463200] ACPI: PCI: Interrupt link LNKC configured for IRQ 111318client # [ 0.449288] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1319lb # [ 0.453313] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1320worker2 # [ 0.479765] iommu: Default domain type: Translated1321worker1 # [ 0.464219] ACPI: PCI: Interrupt link LNKD configured for IRQ 111322worker2 # [ 0.480553] iommu: DMA domain TLB invalidation policy: lazy mode1323lb # [ 0.454276] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1324client # [ 0.450572] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1325worker1 # [ 0.465195] ACPI: PCI: Interrupt link LNKE configured for IRQ 101326worker2 # [ 0.481680] ACPI: bus type USB registered1327worker2 # [ 0.482411] usbcore: registered new interface driver usbfs1328worker1 # [ 0.466196] ACPI: PCI: Interrupt link LNKF configured for IRQ 101329lb # [ 0.456076] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1330worker2 # [ 0.482679] usbcore: registered new interface driver hub1331worker1 # [ 0.467194] ACPI: PCI: Interrupt link LNKG configured for IRQ 111332lb # [ 0.456920] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1333worker2 # [ 0.483526] usbcore: registered new device driver usb1334worker1 # [ 0.468192] ACPI: PCI: Interrupt link LNKH configured for IRQ 111335client # [ 0.454576] ACPI: PCI: Interrupt link LNKA configured for IRQ 101336worker2 # [ 0.485192] NetLabel: Initializing1337worker1 # [ 0.469122] ACPI: PCI: Interrupt link GSIA configured for IRQ 161338client # [ 0.455078] ACPI: PCI: Interrupt link LNKB configured for IRQ 101339lb # [ 0.458558] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1340worker2 # [ 0.485662] NetLabel: domain hash size = 1281341client # [ 0.456068] ACPI: PCI: Interrupt link LNKC configured for IRQ 111342worker2 # [ 0.486350] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1343worker1 # [ 0.470097] ACPI: PCI: Interrupt link GSIB configured for IRQ 171344lb # [ 0.459822] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1345client # [ 0.457068] ACPI: PCI: Interrupt link LNKD configured for IRQ 111346worker2 # [ 0.486696] NetLabel: unlabeled traffic allowed by default1347worker1 # [ 0.471094] ACPI: PCI: Interrupt link GSIC configured for IRQ 181348worker2 # [ 0.487566] PCI: Using ACPI for IRQ routing1349client # [ 0.458070] ACPI: PCI: Interrupt link LNKE configured for IRQ 101350worker1 # [ 0.472100] ACPI: PCI: Interrupt link GSID configured for IRQ 191351client # [ 0.459075] ACPI: PCI: Interrupt link LNKF configured for IRQ 101352worker1 # [ 0.473097] ACPI: PCI: Interrupt link GSIE configured for IRQ 201353client # [ 0.460076] ACPI: PCI: Interrupt link LNKG configured for IRQ 111354worker1 # [ 0.474092] ACPI: PCI: Interrupt link GSIF configured for IRQ 211355lb # [ 0.463642] ACPI: PCI: Interrupt link LNKA configured for IRQ 101356client # [ 0.461082] ACPI: PCI: Interrupt link LNKH configured for IRQ 111357worker1 # [ 0.475097] ACPI: PCI: Interrupt link GSIG configured for IRQ 221358lb # [ 0.464138] ACPI: PCI: Interrupt link LNKB configured for IRQ 101359client # [ 0.462011] ACPI: PCI: Interrupt link GSIA configured for IRQ 161360lb # [ 0.465132] ACPI: PCI: Interrupt link LNKC configured for IRQ 111361worker1 # [ 0.476093] ACPI: PCI: Interrupt link GSIH configured for IRQ 231362client # [ 0.462988] ACPI: PCI: Interrupt link GSIB configured for IRQ 171363lb # [ 0.466128] ACPI: PCI: Interrupt link LNKD configured for IRQ 111364client # [ 0.463980] ACPI: PCI: Interrupt link GSIC configured for IRQ 181365worker1 # [ 0.478223] iommu: Default domain type: Translated1366lb # [ 0.467420] ACPI: PCI: Interrupt link LNKE configured for IRQ 101367client # [ 0.464977] ACPI: PCI: Interrupt link GSID configured for IRQ 191368worker1 # [ 0.479090] iommu: DMA domain TLB invalidation policy: lazy mode1369lb # [ 0.468129] ACPI: PCI: Interrupt link LNKF configured for IRQ 101370client # [ 0.465950] ACPI: PCI: Interrupt link GSIE configured for IRQ 201371worker1 # [ 0.480363] ACPI: bus type USB registered1372lb # [ 0.469136] ACPI: PCI: Interrupt link LNKG configured for IRQ 111373client # [ 0.466742] ACPI: PCI: Interrupt link GSIF configured for IRQ 211374lb # [ 0.470131] ACPI: PCI: Interrupt link LNKH configured for IRQ 111375client # [ 0.466986] ACPI: PCI: Interrupt link GSIG configured for IRQ 221376lb # [ 0.471063] ACPI: PCI: Interrupt link GSIA configured for IRQ 161377client # [ 0.467983] ACPI: PCI: Interrupt link GSIH configured for IRQ 231378worker1 # [ 0.482275] usbcore: registered new interface driver usbfs1379lb # [ 0.472005] ACPI: PCI: Interrupt link GSIB configured for IRQ 171380lb # [ 0.472754] ACPI: PCI: Interrupt link GSIC configured for IRQ 181381worker1 # [ 0.483100] usbcore: registered new interface driver hub1382client # [ 0.470011] iommu: Default domain type: Translated1383lb # [ 0.473045] ACPI: PCI: Interrupt link GSID configured for IRQ 191384worker1 # [ 0.484085] usbcore: registered new device driver usb1385client # [ 0.470810] iommu: DMA domain TLB invalidation policy: lazy mode1386lb # [ 0.473966] ACPI: PCI: Interrupt link GSIE configured for IRQ 201387worker1 # [ 0.485980] NetLabel: Initializing1388lb # [ 0.474753] ACPI: PCI: Interrupt link GSIF configured for IRQ 211389client # [ 0.471942] ACPI: bus type USB registered1390worker1 # [ 0.486548] NetLabel: domain hash size = 1281391lb # [ 0.475045] ACPI: PCI: Interrupt link GSIG configured for IRQ 221392client # [ 0.473045] usbcore: registered new interface driver usbfs1393worker1 # [ 0.487075] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1394lb # [ 0.476046] ACPI: PCI: Interrupt link GSIH configured for IRQ 231395client # [ 0.473943] usbcore: registered new interface driver hub1396worker1 # [ 0.488122] NetLabel: unlabeled traffic allowed by default1397client # [ 0.474646] usbcore: registered new device driver usb1398worker1 # [ 0.489091] PCI: Using ACPI for IRQ routing1399lb # [ 0.478054] iommu: Default domain type: Translated1400client # [ 0.475829] NetLabel: Initializing1401lb # [ 0.478842] iommu: DMA domain TLB invalidation policy: lazy mode1402client # [ 0.476413] NetLabel: domain hash size = 1281403lb # [ 0.480040] ACPI: bus type USB registered1404client # [ 0.476972] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1405client # [ 0.477982] NetLabel: unlabeled traffic allowed by default1406client # [ 0.478965] PCI: Using ACPI for IRQ routing1407lb # [ 0.480778] usbcore: registered new interface driver usbfs1408lb # [ 0.481072] usbcore: registered new interface driver hub1409lb # [ 0.481928] usbcore: registered new device driver usb1410lb # [ 0.483556] NetLabel: Initializing1411lb # [ 0.484479] NetLabel: domain hash size = 1281412lb # [ 0.485031] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413lb # [ 0.486051] NetLabel: unlabeled traffic allowed by default1414lb # [ 0.487020] PCI: Using ACPI for IRQ routing1415worker2 # [ 0.530585] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416worker2 # [ 0.530657] pci 0000:00:01.0: vgaarb: bridge control possible1417worker2 # [ 0.530657] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418worker2 # [ 0.530666] vgaarb: loaded1419worker2 # [ 0.531410] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01420worker2 # [ 0.531661] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1421worker2 # [ 0.534756] clocksource: Switched to clocksource kvm-clock1422worker2 # [ 0.536488] VFS: Disk quotas dquot_6.6.01423worker2 # [ 0.539059] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1424worker2 # [ 0.540454] pnp: PnP ACPI init1425worker2 # [ 0.541288] ACPI: IRQ 4 override to edge(!), high(!)1426worker2 # [ 0.542289] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1427worker2 # [ 0.543845] pnp: PnP ACPI: found 6 devices1428worker1 # [ 0.533070] pci 0000:00:01.0: vgaarb: setting as boot VGA device1429worker1 # [ 0.533070] pci 0000:00:01.0: vgaarb: bridge control possible1430worker1 # [ 0.533070] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1431worker1 # [ 0.533081] vgaarb: loaded1432worker1 # [ 0.533914] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01433client # [ 0.522769] pci 0000:00:01.0: vgaarb: setting as boot VGA device1434worker1 # [ 0.534771] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1435client # [ 0.522968] pci 0000:00:01.0: vgaarb: bridge control possible1436worker2 # [ 0.551448] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1437client # [ 0.522968] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1438client # [ 0.522977] vgaarb: loaded1439worker2 # [ 0.552912] clocksource: Switched to clocksource acpi_pm1440worker2 # [ 0.553874] NET: Registered PF_INET protocol family1441client # [ 0.523743] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01442client # [ 0.523973] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1443worker1 # [ 0.538166] clocksource: Switched to clocksource kvm-clock1444worker2 # [ 0.555062] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1445worker1 # [ 0.541776] VFS: Disk quotas dquot_6.6.01446lb # [ 0.531357] pci 0000:00:01.0: vgaarb: setting as boot VGA device1447worker1 # [ 0.542576] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1448client # [ 0.528061] clocksource: Switched to clocksource kvm-clock1449lb # [ 0.532026] pci 0000:00:01.0: vgaarb: bridge control possible1450worker1 # [ 0.544073] pnp: PnP ACPI init1451lb # [ 0.532026] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1452worker1 # [ 0.544967] ACPI: IRQ 4 override to edge(!), high(!)1453lb # [ 0.532036] vgaarb: loaded1454lb # [ 0.532808] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01455worker1 # [ 0.546058] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1456client # [ 0.531643] VFS: Disk quotas dquot_6.6.01457lb # [ 0.533032] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1458worker1 # [ 0.547758] pnp: PnP ACPI: found 6 devices1459client # [ 0.532346] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1460client # [ 0.533676] pnp: PnP ACPI init1461client # [ 0.534488] ACPI: IRQ 4 override to edge(!), high(!)1462client # [ 0.535523] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1463lb # [ 0.536142] clocksource: Switched to clocksource kvm-clock1464client # [ 0.537074] pnp: PnP ACPI: found 6 devices1465lb # [ 0.539947] VFS: Disk quotas dquot_6.6.01466worker2 # [ 0.571944] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1467lb # [ 0.540693] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1468worker1 # [ 0.555461] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1469lb # [ 0.542095] pnp: PnP ACPI init1470worker1 # [ 0.557092] clocksource: Switched to clocksource acpi_pm1471worker2 # [ 0.573464] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1472lb # [ 0.542889] ACPI: IRQ 4 override to edge(!), high(!)1473worker1 # [ 0.558209] NET: Registered PF_INET protocol family1474worker2 # [ 0.574781] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1475lb # [ 0.543833] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1476worker2 # [ 0.576092] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1477worker1 # [ 0.559450] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1478lb # [ 0.545346] pnp: PnP ACPI: found 6 devices1479client # [ 0.544819] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1480worker2 # [ 0.577292] TCP: Hash tables configured (established 16384 bind 16384)1481client # [ 0.546217] clocksource: Switched to clocksource acpi_pm1482worker2 # [ 0.578389] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1483client # [ 0.547216] NET: Registered PF_INET protocol family1484worker2 # [ 0.579655] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1485client # [ 0.548432] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1486worker2 # [ 0.580784] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1487worker2 # [ 0.582002] NET: Registered PF_UNIX/PF_LOCAL protocol family1488worker2 # [ 0.583006] NET: Registered PF_XDP protocol family1489worker2 # [ 0.583869] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1490worker2 # [ 0.584897] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1491lb # [ 0.553003] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1492worker2 # [ 0.585900] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1493lb # [ 0.554396] clocksource: Switched to clocksource acpi_pm1494lb # [ 0.555361] NET: Registered PF_INET protocol family1495worker2 # [ 0.587002] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1496worker2 # [ 0.588116] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1497lb # [ 0.556529] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1498worker2 # [ 0.589219] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1499worker2 # [ 0.590931] ACPI: \_SB_.GSIA: Enabled at IRQ 161500worker2 # [ 0.593202] ACPI: \_SB_.GSIB: Enabled at IRQ 171501worker1 # [ 0.576077] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1502worker1 # [ 0.577774] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1503worker2 # [ 0.595631] ACPI: \_SB_.GSIC: Enabled at IRQ 181504worker1 # [ 0.579233] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1505worker2 # [ 0.597714] ACPI: \_SB_.GSID: Enabled at IRQ 191506client # [ 0.565344] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1507worker1 # [ 0.580710] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1508worker1 # [ 0.582100] TCP: Hash tables configured (established 16384 bind 16384)1509client # [ 0.566891] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1510worker2 # [ 0.599422] PCI: CLS 0 bytes, default 641511client # [ 0.568214] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1512worker1 # [ 0.583351] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1513worker2 # [ 0.600373] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1514client # [ 0.569533] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1515worker1 # [ 0.584746] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1516worker2 # [ 0.602035] Trying to unpack rootfs image as initramfs...1517client # [ 0.570778] TCP: Hash tables configured (established 16384 bind 16384)1518worker1 # [ 0.585979] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1519worker1 # [ 0.587368] NET: Registered PF_UNIX/PF_LOCAL protocol family1520client # [ 0.571941] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1521worker1 # [ 0.588448] NET: Registered PF_XDP protocol family1522client # [ 0.573243] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1523lb # [ 0.573427] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1524worker1 # [ 0.589353] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1525client # [ 0.574315] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1526lb # [ 0.574922] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1527client # [ 0.575488] NET: Registered PF_UNIX/PF_LOCAL protocol family1528worker1 # [ 0.590479] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1529client # [ 0.576414] NET: Registered PF_XDP protocol family1530lb # [ 0.576212] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1531worker1 # [ 0.591591] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1532client # [ 0.577228] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1533lb # [ 0.577499] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1534worker1 # [ 0.592839] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1535client # [ 0.578221] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1536lb # [ 0.578690] TCP: Hash tables configured (established 16384 bind 16384)1537worker1 # [ 0.594066] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1538client # [ 0.579218] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1539lb # [ 0.579790] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1540worker1 # [ 0.595274] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1541client # [ 0.580304] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1542lb # [ 0.581003] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1543client # [ 0.581373] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1544worker1 # [ 0.597125] ACPI: \_SB_.GSIA: Enabled at IRQ 161545lb # [ 0.582042] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1546client # [ 0.582449] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1547lb # [ 0.583218] NET: Registered PF_UNIX/PF_LOCAL protocol family1548client # [ 0.584108] ACPI: \_SB_.GSIA: Enabled at IRQ 161549lb # [ 0.584170] NET: Registered PF_XDP protocol family1550worker1 # [ 0.599698] ACPI: \_SB_.GSIB: Enabled at IRQ 171551lb # [ 0.584951] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1552lb # [ 0.585916] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1553client # [ 0.586472] ACPI: \_SB_.GSIB: Enabled at IRQ 171554worker1 # [ 0.601761] ACPI: \_SB_.GSIC: Enabled at IRQ 181555lb # [ 0.586870] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1556lb # [ 0.587904] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1557worker1 # [ 0.603804] ACPI: \_SB_.GSID: Enabled at IRQ 191558client # [ 0.588821] ACPI: \_SB_.GSIC: Enabled at IRQ 181559lb # [ 0.588945] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1560worker1 # [ 0.605486] PCI: CLS 0 bytes, default 641561lb # [ 0.590050] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1562client # [ 0.590787] ACPI: \_SB_.GSID: Enabled at IRQ 191563lb # [ 0.591763] ACPI: \_SB_.GSIA: Enabled at IRQ 161564worker1 # [ 0.606331] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1565client # [ 0.592574] PCI: CLS 0 bytes, default 641566worker1 # [ 0.608010] Trying to unpack rootfs image as initramfs...1567lb # [ 0.594306] ACPI: \_SB_.GSIB: Enabled at IRQ 171568client # [ 0.593482] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1569client # [ 0.595323] Trying to unpack rootfs image as initramfs...1570lb # [ 0.596242] ACPI: \_SB_.GSIC: Enabled at IRQ 181571lb # [ 0.598240] ACPI: \_SB_.GSID: Enabled at IRQ 191572lb # [ 0.599843] PCI: CLS 0 bytes, default 641573lb # [ 0.600699] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1574lb # [ 0.602392] Trying to unpack rootfs image as initramfs...1575worker2 # [ 0.648168] Initialise system trusted keyrings1576worker2 # [ 0.649147] workingset: timestamp_bits=40 max_order=19 bucket_order=01577client # [ 0.638141] Initialise system trusted keyrings1578worker1 # [ 0.653753] Initialise system trusted keyrings1579worker1 # [ 0.654712] workingset: timestamp_bits=40 max_order=19 bucket_order=01580worker2 # [ 0.672947] Key type asymmetric registered1581client # [ 0.641252] workingset: timestamp_bits=40 max_order=19 bucket_order=01582worker2 # [ 0.673707] Asymmetric key parser 'x509' registered1583worker2 # [ 0.674566] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1584worker2 # [ 0.677929] io scheduler mq-deadline registered1585worker2 # [ 0.678700] io scheduler kyber registered1586lb # [ 0.647190] Initialise system trusted keyrings1587lb # [ 0.650152] workingset: timestamp_bits=40 max_order=19 bucket_order=01588worker2 # [ 0.682951] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1589worker2 # [ 0.684248] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1590worker2 # [ 0.689599] Linux agpgart interface v0.1031591worker2 # [ 0.690396] ACPI: bus type drm_connector registered1592worker1 # [ 0.675849] Key type asymmetric registered1593worker2 # [ 0.693210] usbcore: registered new interface driver usbserial_generic1594client # [ 0.662452] Key type asymmetric registered1595worker2 # [ 0.694344] usbserial: USB Serial support registered for generic1596client # [ 0.663169] Asymmetric key parser 'x509' registered1597worker1 # [ 0.678922] Asymmetric key parser 'x509' registered1598worker1 # [ 0.679766] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1599worker2 # [ 0.697768] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1600worker2 # [ 0.697768] Please enable it if your BIOS has the CPPC option.1601client # [ 0.667259] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1602worker1 # [ 0.683105] io scheduler mq-deadline registered1603worker2 # [ 0.700010] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1604client # [ 0.668612] io scheduler mq-deadline registered1605worker1 # [ 0.683856] io scheduler kyber registered1606client # [ 0.669371] io scheduler kyber registered1607worker1 # [ 0.686447] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1608worker1 # [ 0.687725] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1609worker2 # [ 0.704906] drop_monitor: Initializing network drop monitor service1610client # [ 0.673252] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1611lb # [ 0.674096] Key type asymmetric registered1612worker2 # [ 0.706089] NET: Registered PF_INET6 protocol family1613lb # [ 0.674786] Asymmetric key parser 'x509' registered1614client # [ 0.674558] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1615lb # [ 0.675648] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1616worker2 # [ 0.709243] Segment Routing with IPv61617worker1 # [ 0.692700] Linux agpgart interface v0.1031618lb # [ 0.678162] io scheduler mq-deadline registered1619worker1 # [ 0.693478] ACPI: bus type drm_connector registered1620lb # [ 0.678883] io scheduler kyber registered1621worker2 # [ 0.711786] In-situ OAM (IOAM) with IPv61622client # [ 0.680841] Linux agpgart interface v0.1031623worker2 # [ 0.712878] IPI shorthand broadcast: enabled1624client # [ 0.681629] ACPI: bus type drm_connector registered1625worker1 # [ 0.697981] usbcore: registered new interface driver usbserial_generic1626lb # [ 0.683000] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1627worker1 # [ 0.699030] usbserial: USB Serial support registered for generic1628lb # [ 0.684254] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1629client # [ 0.685671] usbcore: registered new interface driver usbserial_generic1630client # [ 0.686764] usbserial: USB Serial support registered for generic1631worker1 # [ 0.701905] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1632worker1 # [ 0.701905] Please enable it if your BIOS has the CPPC option.1633client # [ 0.687761] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1634client # [ 0.687761] Please enable it if your BIOS has the CPPC option.1635worker1 # [ 0.704042] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1636worker2 # [ 0.721077] sched_clock: Marking stable (587014319, 133737601)->(791756551, -71004631)1637lb # [ 0.689840] Linux agpgart interface v0.1031638lb # [ 0.690585] ACPI: bus type drm_connector registered1639worker1 # [ 0.708047] drop_monitor: Initializing network drop monitor service1640client # [ 0.693215] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1641lb # [ 0.693501] usbcore: registered new interface driver usbserial_generic1642worker1 # [ 0.709199] NET: Registered PF_INET6 protocol family1643worker2 # [ 0.726021] registered taskstats version 11644client # [ 0.694576] drop_monitor: Initializing network drop monitor service1645lb # [ 0.694560] usbserial: USB Serial support registered for generic1646worker2 # [ 0.726966] Loading compiled-in X.509 certificates1647worker1 # [ 0.712326] Segment Routing with IPv61648client # [ 0.698340] NET: Registered PF_INET6 protocol family1649lb # [ 0.697993] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1650lb # [ 0.697993] Please enable it if your BIOS has the CPPC option.1651worker1 # [ 0.714917] In-situ OAM (IOAM) with IPv61652lb # [ 0.700162] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1653worker1 # [ 0.715979] IPI shorthand broadcast: enabled1654client # [ 0.701500] Segment Routing with IPv61655client # [ 0.702240] In-situ OAM (IOAM) with IPv61656lb # [ 0.704203] drop_monitor: Initializing network drop monitor service1657client # [ 0.705517] IPI shorthand broadcast: enabled1658lb # [ 0.705416] NET: Registered PF_INET6 protocol family1659lb # [ 0.708520] Segment Routing with IPv61660worker1 # [ 0.724191] sched_clock: Marking stable (591013915, 132866406)->(802479777, -78599456)1661lb # [ 0.711017] In-situ OAM (IOAM) with IPv61662lb # [ 0.712073] IPI shorthand broadcast: enabled1663worker1 # [ 0.729054] registered taskstats version 11664worker2 # [ 0.745769] Demotion targets for Node 0: null1665client # [ 0.713828] sched_clock: Marking stable (580017928, 133178584)->(789435512, -76239000)1666worker2 # [ 0.746633] Key type .fscrypt registered1667worker1 # [ 0.729995] Loading compiled-in X.509 certificates1668worker2 # [ 0.747283] Key type fscrypt-provisioning registered1669client # [ 0.718389] registered taskstats version 11670worker2 # [ 0.750876] ima: No TPM chip found, activating TPM-bypass!1671client # [ 0.719406] Loading compiled-in X.509 certificates1672worker2 # [ 0.751819] ima: Allocated hash algorithm: sha11673worker2 # [ 0.752599] ima: No architecture policies found1674lb # [ 0.720251] sched_clock: Marking stable (587013649, 132962786)->(797953850, -77977415)1675worker2 # [ 0.754954] PM: Magic number: 10:347:5321676worker2 # [ 0.756607] RAS: Correctable Errors collector initialized.1677lb # [ 0.725193] registered taskstats version 11678lb # [ 0.726135] Loading compiled-in X.509 certificates1679worker1 # [ 0.746379] Demotion targets for Node 0: null1680worker2 # [ 0.766436] clk: Disabling unused clocks1681worker1 # [ 0.749936] Key type .fscrypt registered1682worker1 # [ 0.750602] Key type fscrypt-provisioning registered1683worker1 # [ 0.751528] ima: No TPM chip found, activating TPM-bypass!1684worker2 # [ 0.768770] PM: genpd: Disabling unused power domains1685client # [ 0.737209] Demotion targets for Node 0: null1686client # [ 0.738098] Key type .fscrypt registered1687worker1 # [ 0.753893] ima: Allocated hash algorithm: sha11688client # [ 0.738854] Key type fscrypt-provisioning registered1689worker1 # [ 0.754675] ima: No architecture policies found1690client # [ 0.739875] ima: No TPM chip found, activating TPM-bypass!1691worker1 # [ 0.756893] PM: Magic number: 10:347:5321692client # [ 0.743225] ima: Allocated hash algorithm: sha11693worker1 # [ 0.758495] RAS: Correctable Errors collector initialized.1694client # [ 0.744035] ima: No architecture policies found1695lb # [ 0.744993] Demotion targets for Node 0: null1696lb # [ 0.745872] Key type .fscrypt registered1697lb # [ 0.746536] Key type fscrypt-provisioning registered1698client # [ 0.748207] PM: Magic number: 10:347:5321699lb # [ 0.749105] ima: No TPM chip found, activating TPM-bypass!1700lb # [ 0.749988] ima: Allocated hash algorithm: sha11701client # [ 0.749909] RAS: Correctable Errors collector initialized.1702lb # [ 0.750737] ima: No architecture policies found1703worker1 # [ 0.767551] clk: Disabling unused clocks1704lb # [ 0.755376] PM: Magic number: 10:347:5321705worker1 # [ 0.771904] PM: genpd: Disabling unused power domains1706client # [ 0.758883] clk: Disabling unused clocks1707lb # [ 0.759156] RAS: Correctable Errors collector initialized.1708client # [ 0.761208] PM: genpd: Disabling unused power domains1709lb # [ 0.767800] clk: Disabling unused clocks1710lb # [ 0.768496] PM: genpd: Disabling unused power domains1711worker2 # [ 0.934404] Freeing initrd memory: 29092K1712worker2 # [ 0.937632] Freeing unused decrypted memory: 2028K1713worker2 # [ 0.940340] Freeing unused kernel image (initmem) memory: 3644K1714worker2 # [ 0.941460] Write protecting the kernel read-only data: 32768k1715worker2 # [ 0.943377] Freeing unused kernel image (text/rodata gap) memory: 1208K1716worker2 # [ 0.944952] Freeing unused kernel image (rodata/data gap) memory: 752K1717worker1 # [ 0.934993] Freeing initrd memory: 29092K1718worker1 # [ 0.938152] Freeing unused decrypted memory: 2028K1719worker1 # [ 0.940736] Freeing unused kernel image (initmem) memory: 3644K1720worker1 # [ 0.941783] Write protecting the kernel read-only data: 32768k1721worker1 # [ 0.943613] Freeing unused kernel image (text/rodata gap) memory: 1208K1722client # [ 0.930018] Freeing initrd memory: 29096K1723worker1 # [ 0.945101] Freeing unused kernel image (rodata/data gap) memory: 752K1724client # [ 0.933292] Freeing unused decrypted memory: 2028K1725client # [ 0.936005] Freeing unused kernel image (initmem) memory: 3644K1726lb # [ 0.937296] Freeing initrd memory: 29092K1727client # [ 0.937153] Write protecting the kernel read-only data: 32768k1728client # [ 0.939075] Freeing unused kernel image (text/rodata gap) memory: 1208K1729lb # [ 0.940723] Freeing unused decrypted memory: 2028K1730client # [ 0.940732] Freeing unused kernel image (rodata/data gap) memory: 752K1731lb # [ 0.943663] Freeing unused kernel image (initmem) memory: 3644K1732lb # [ 0.944778] Write protecting the kernel read-only data: 32768k1733lb # [ 0.946734] Freeing unused kernel image (text/rodata gap) memory: 1208K1734lb # [ 0.948280] Freeing unused kernel image (rodata/data gap) memory: 752K1735worker2 # [ 0.996179] x86/mm: Checked W+X mappings: passed, no W+X pages found.1736worker2 # [ 0.997294] Run /init as init process1737worker2 # [ 1.007723] systemd[1]: Inserted module 'autofs4'1738worker1 # [ 0.996204] x86/mm: Checked W+X mappings: passed, no W+X pages found.1739worker1 # [ 0.997230] Run /init as init process1740client # [ 0.991841] x86/mm: Checked W+X mappings: passed, no W+X pages found.1741worker2 # [ 1.024130] fuse: init (API version 7.45)1742worker1 # [ 1.007650] systemd[1]: Inserted module 'autofs4'1743client # [ 0.992932] Run /init as init process1744worker2 # [ 1.030001] ACPI: \_SB_.GSIG: Enabled at IRQ 221745lb # [ 0.999518] x86/mm: Checked W+X mappings: passed, no W+X pages found.1746worker2 # [ 1.032304] ACPI: \_SB_.GSIH: Enabled at IRQ 231747lb # [ 1.000585] Run /init as init process1748worker2 # [ 1.035209] ACPI: \_SB_.GSIE: Enabled at IRQ 201749client # [ 1.003397] systemd[1]: Inserted module 'autofs4'1750worker2 # [ 1.037446] ACPI: \_SB_.GSIF: Enabled at IRQ 211751worker1 # [ 1.023828] fuse: init (API version 7.45)1752lb # [ 1.011333] systemd[1]: Inserted module 'autofs4'1753worker1 # [ 1.029606] ACPI: \_SB_.GSIG: Enabled at IRQ 221754worker1 # [ 1.032181] ACPI: \_SB_.GSIH: Enabled at IRQ 231755client # [ 1.019716] fuse: init (API version 7.45)1756worker1 # [ 1.035482] ACPI: \_SB_.GSIE: Enabled at IRQ 201757worker1 # [ 1.037944] ACPI: \_SB_.GSIF: Enabled at IRQ 211758client # [ 1.025700] ACPI: \_SB_.GSIG: Enabled at IRQ 221759lb # [ 1.028137] fuse: init (API version 7.45)1760client # [ 1.028117] ACPI: \_SB_.GSIH: Enabled at IRQ 231761worker2 # [ 1.062442] systemd[1]: Successfully made /usr/ read-only.1762client # [ 1.031050] ACPI: \_SB_.GSIE: Enabled at IRQ 201763client # [ 1.033366] ACPI: \_SB_.GSIF: Enabled at IRQ 211764lb # [ 1.034098] ACPI: \_SB_.GSIG: Enabled at IRQ 221765lb # [ 1.036703] ACPI: \_SB_.GSIH: Enabled at IRQ 231766lb # [ 1.040148] ACPI: \_SB_.GSIE: Enabled at IRQ 201767lb # [ 1.042815] ACPI: \_SB_.GSIF: Enabled at IRQ 211768worker1 # [ 1.067026] systemd[1]: Successfully made /usr/ read-only.1769client # [ 1.060056] systemd[1]: Successfully made /usr/ read-only.1770lb # [ 1.069813] systemd[1]: Successfully made /usr/ read-only.1771worker2 # [ 1.397579] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1772worker2 # [ 1.402753] systemd[1]: Detected virtualization kvm.1773worker2 # [ 1.403554] systemd[1]: Detected architecture x86-64.1774worker2 # [ 1.404401] systemd[1]: Running in initrd.1775worker2 # [ 1.405485] systemd[1]: Initializing machine ID from random generator.1776worker2 # [ 1.406678] systemd[1]: Hostname set to <worker2>.1777worker1 # [ 1.403742] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1778worker1 # [ 1.415615] systemd[1]: Detected virtualization kvm.1779worker1 # [ 1.417737] systemd[1]: Detected architecture x86-64.1780worker1 # [ 1.419978] systemd[1]: Running in initrd.1781client # [ 1.396628] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1782worker1 # [ 1.422628] systemd[1]: Initializing machine ID from random generator.1783lb # [ 1.405318] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1784client # [ 1.408582] systemd[1]: Detected virtualization kvm.1785lb # [ 1.410139] systemd[1]: Detected virtualization kvm.1786lb # [ 1.410942] systemd[1]: Detected architecture x86-64.1787worker1 # [ 1.425578] systemd[1]: Hostname set to <worker1>.1788lb # [ 1.411795] systemd[1]: Running in initrd.1789client # [ 1.410668] systemd[1]: Detected architecture x86-64.1790lb # [ 1.412915] systemd[1]: Initializing machine ID from random generator.1791client # [ 1.412795] systemd[1]: Running in initrd.1792lb # [ 1.414130] systemd[1]: Hostname set to <lb>.1793client # [ 1.415356] systemd[1]: Initializing machine ID from random generator.1794client # [ 1.418249] systemd[1]: Hostname set to <client>.1795worker2 # [ 1.598908] systemd[1]: bpf-restrict-fs: LSM BPF program attached1796worker2 # [ 1.637470] systemd[1]: Queued start job for default target Initrd Default Target.1797lb # [ 1.607380] systemd[1]: bpf-restrict-fs: LSM BPF program attached1798worker2 # [ 1.642108] systemd[1]: Created slice Slice /system/modprobe.1799worker2 # [ 1.643316] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1800worker2 # [ 1.644793] systemd[1]: Expecting device /dev/disk/by-label/nixos...1801worker2 # [ 1.645932] systemd[1]: Reached target Path Units.1802worker2 # [ 1.646848] systemd[1]: Reached target Slice Units.1803worker2 # [ 1.647762] systemd[1]: Reached target Swaps.1804worker2 # [ 1.648549] systemd[1]: Reached target Timer Units.1805worker2 # [ 1.649521] systemd[1]: Listening on D-Bus System Message Bus Socket.1806worker2 # [ 1.650782] systemd[1]: Listening on Journal Socket (/dev/log).1807worker2 # [ 1.651975] systemd[1]: Listening on Journal Sockets.1808worker2 # [ 1.653068] systemd[1]: Listening on udev Control Socket.1809worker2 # [ 1.654097] systemd[1]: Listening on udev Kernel Socket.1810worker2 # [ 1.655106] systemd[1]: Reached target Socket Units.1811worker2 # [ 1.656931] systemd[1]: Starting Create List of Static Device Nodes...1812worker2 # [ 1.660825] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1813worker2 # [ 1.667832] systemd[1]: Starting Load Kernel Module configfs...1814worker1 # [ 1.649860] systemd[1]: bpf-restrict-fs: LSM BPF program attached1815client # [ 1.639064] systemd[1]: bpf-restrict-fs: LSM BPF program attached1816lb # [ 1.646218] systemd[1]: Queued start job for default target Initrd Default Target.1817worker2 # [ 1.679938] systemd[1]: Starting Journal Service...1818lb # [ 1.650323] systemd[1]: Created slice Slice /system/modprobe.1819lb # [ 1.651574] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1820lb # [ 1.653025] systemd[1]: Expecting device /dev/disk/by-label/nixos...1821lb # [ 1.654161] systemd[1]: Reached target Path Units.1822lb # [ 1.655070] systemd[1]: Reached target Slice Units.1823lb # [ 1.655926] systemd[1]: Reached target Swaps.1824lb # [ 1.656767] systemd[1]: Reached target Timer Units.1825lb # [ 1.657769] systemd[1]: Listening on D-Bus System Message Bus Socket.1826lb # [ 1.659018] systemd[1]: Listening on Journal Socket (/dev/log).1827lb # [ 1.660173] systemd[1]: Listening on Journal Sockets.1828lb # [ 1.661190] systemd[1]: Listening on udev Control Socket.1829worker2 # [ 1.693847] systemd[1]: Starting Load Kernel Modules...1830lb # [ 1.662277] systemd[1]: Listening on udev Kernel Socket.1831lb # [ 1.663228] systemd[1]: Reached target Socket Units.1832lb # [ 1.665046] systemd[1]: Starting Create List of Static Device Nodes...1833lb # [ 1.668846] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1834lb # [ 1.675674] systemd[1]: Starting Load Kernel Module configfs...1835worker2 # [ 1.699875] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1836worker2 # [ 1.717275] netfs: FS-Cache loaded1837lb # [ 1.689613] systemd[1]: Starting Journal Service...1838worker2 # [ 1.722068] 9pnet: Installing 9P2000 support1839worker1 # [ 1.705561] systemd[1]: Queued start job for default target Initrd Default Target.1840client # [ 1.693277] systemd[1]: Queued start job for default target Initrd Default Target.1841worker1 # [ 1.710303] systemd[1]: Created slice Slice /system/modprobe.1842client # [ 1.696501] systemd[1]: Created slice Slice /system/modprobe.1843worker2 # [ 1.728318] systemd[1]: Starting Coldplug All udev Devices...1844worker1 # [ 1.711528] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1845worker1 # [ 1.712928] systemd[1]: Expecting device /dev/disk/by-label/nixos...1846client # [ 1.697616] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1847worker1 # [ 1.714060] systemd[1]: Reached target Path Units.1848client # [ 1.698993] systemd[1]: Expecting device /dev/disk/by-label/nixos...1849worker1 # [ 1.714951] systemd[1]: Reached target Slice Units.1850client # [ 1.700113] systemd[1]: Reached target Path Units.1851worker1 # [ 1.715839] systemd[1]: Reached target Swaps.1852client # [ 1.701008] systemd[1]: Reached target Slice Units.1853worker1 # [ 1.716667] systemd[1]: Reached target Timer Units.1854client # [ 1.701971] systemd[1]: Reached target Swaps.1855worker1 # [ 1.717694] systemd[1]: Listening on D-Bus System Message Bus Socket.1856client # [ 1.702839] systemd[1]: Reached target Timer Units.1857worker1 # [ 1.718958] systemd[1]: Listening on Journal Socket (/dev/log).1858client # [ 1.703820] systemd[1]: Listening on D-Bus System Message Bus Socket.1859lb # [ 1.704057] systemd[1]: Starting Load Kernel Modules...1860worker1 # [ 1.720202] systemd[1]: Listening on Journal Sockets.1861client # [ 1.704990] systemd[1]: Listening on Journal Socket (/dev/log).1862worker1 # [ 1.721221] systemd[1]: Listening on udev Control Socket.1863client # [ 1.706154] systemd[1]: Listening on Journal Sockets.1864worker1 # [ 1.722261] systemd[1]: Listening on udev Kernel Socket.1865client # [ 1.707223] systemd[1]: Listening on udev Control Socket.1866worker1 # [ 1.723212] systemd[1]: Reached target Socket Units.1867client # [ 1.708247] systemd[1]: Listening on udev Kernel Socket.1868client # [ 1.709221] systemd[1]: Reached target Socket Units.1869worker1 # [ 1.725969] systemd[1]: Starting Create List of Static Device Nodes...1870client # [ 1.711024] systemd[1]: Starting Create List of Static Device Nodes...1871worker2 # [ 1.742846] systemd[1]: Finished Create List of Static Device Nodes.1872worker1 # [ 1.728950] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1873client # [ 1.715264] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1874worker2 # [ 1.747493] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1875lb # [ 1.704950] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1876client # [ 1.722317] systemd[1]: Starting Load Kernel Module configfs...1877worker1 # [ 1.737953] systemd[1]: Starting Load Kernel Module configfs...1878worker2 # [ 1.756170] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1879lb # [ 1.725042] netfs: FS-Cache loaded1880lb # [ 1.729698] 9pnet: Installing 9P2000 support1881worker2 # [ 1.762175] systemd[1]: modprobe@configfs.service: Deactivated successfully.1882worker2 # [ 1.765192] systemd-journald[67]: Collecting audit messages is disabled.1883worker1 # [ 1.753071] systemd[1]: Starting Journal Service...1884client # [ 1.738270] systemd[1]: Starting Journal Service...1885worker2 # [ 1.770108] systemd[1]: Finished Load Kernel Module configfs.1886lb # [ 1.741139] systemd[1]: Starting Coldplug All udev Devices...1887worker2 # [ 1.775194] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1888client # [ 1.748267] systemd[1]: Starting Load Kernel Modules...1889worker2 # [ 1.778962] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1890worker1 # [ 1.765969] systemd[1]: Starting Load Kernel Modules...1891lb # [ 1.754469] systemd[1]: Finished Create List of Static Device Nodes.1892worker2 # [ 1.788826] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1893worker2 # [ 1.790454] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1894lb # [ 1.760587] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1895lb # [ 1.768052] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1896client # [ 1.756268] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1897worker1 # [ 1.772969] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1898client # [ 1.772102] netfs: FS-Cache loaded1899lb # [ 1.773406] systemd[1]: modprobe@configfs.service: Deactivated successfully.1900worker1 # [ 1.789152] netfs: FS-Cache loaded1901client # [ 1.777407] 9pnet: Installing 9P2000 support1902lb # [ 1.777203] systemd-journald[67]: Collecting audit messages is disabled.1903worker1 # [ 1.793976] 9pnet: Installing 9P2000 support1904lb # [ 1.782136] systemd[1]: Finished Load Kernel Module configfs.1905lb # [ 1.785682] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1906worker1 # [ 1.804192] systemd[1]: Starting Coldplug All udev Devices...1907lb # [ 1.788276] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1908worker2 # [ 1.821275] systemd[1]: Finished Load Kernel Modules.1909client # [ 1.792374] systemd[1]: Starting Coldplug All udev Devices...1910worker2 # [ 1.827935] systemd[1]: Starting Apply Kernel Variables...1911lb # [ 1.798149] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1912client # [ 1.799954] systemd-journald[67]: Collecting audit messages is disabled.1913lb # [ 1.799873] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1914client # [ 1.805349] systemd[1]: Finished Create List of Static Device Nodes.1915worker1 # [ 1.821149] systemd[1]: Finished Create List of Static Device Nodes.1916worker2 # [ 1.839305] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1917worker1 # [ 1.826522] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1918client # [ 1.811800] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1919worker2 # [ 1.849241] systemd[1]: Starting Create Static Device Nodes in /dev...1920worker1 # [ 1.834385] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1921client # [ 1.819601] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1922client # [ 1.825580] systemd[1]: modprobe@configfs.service: Deactivated successfully.1923worker1 # [ 1.841326] systemd[1]: modprobe@configfs.service: Deactivated successfully.1924worker1 # [ 1.843170] systemd-journald[67]: Collecting audit messages is disabled.1925lb # [ 1.828527] systemd[1]: Finished Load Kernel Modules.1926client # [ 1.832546] systemd[1]: Finished Load Kernel Module configfs.1927worker1 # [ 1.849269] systemd[1]: Finished Load Kernel Module configfs.1928worker2 # [ 1.867336] systemd[1]: Finished Apply Kernel Variables.1929lb # [ 1.837026] systemd[1]: Starting Apply Kernel Variables...1930client # [ 1.836608] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1931worker1 # [ 1.854424] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1932client # [ 1.840879] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1933worker1 # [ 1.857404] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1934lb # [ 1.847123] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1935worker2 # [ 1.880315] systemd[1]: Started Journal Service.1936client # [ 1.848284] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1937worker2 # [ 1.747726] systemd-modules-load[68]: Inserted module 'dm_mod'1938client # [ 1.851346] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1939worker2 # [ 1.750377] systemd-modules-load[68]: Inserted module 'virtio_balloon'1940worker1 # [ 1.866913] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1941worker1 # [ 1.868764] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1942worker2 # [ 1.752266] systemd-modules-load[68]: Inserted module 'virtio_gpu'1943worker2 # [ 1.754457] systemd[1]: Finished Create Static Device Nodes in /dev.1944lb # [ 1.858171] systemd[1]: Starting Create Static Device Nodes in /dev...1945worker2 # [ 1.763526] systemd[1]: Reached target Preparation for Local File Systems.1946worker2 # [ 1.764770] systemd[1]: Reached target Local File Systems.1947worker2 # [ 1.768126] systemd[1]: Starting Create System Files and Directories...1948worker2 # [ 1.772726] systemd[1]: Starting Rule-based Manager for Device Events and Files...1949lb # [ 1.877281] systemd[1]: Finished Apply Kernel Variables.1950worker1 # [ 1.895440] systemd[1]: Finished Load Kernel Modules.1951client # [ 1.881717] systemd[1]: Finished Load Kernel Modules.1952client # [ 1.888274] systemd[1]: Starting Apply Kernel Variables...1953worker1 # [ 1.904089] systemd[1]: Starting Apply Kernel Variables...1954lb # [ 1.889875] systemd[1]: Started Journal Service.1955lb # [ 1.759292] systemd-modules-load[68]: Inserted module 'dm_mod'1956lb # [ 1.761421] systemd-modules-load[68]: Inserted module 'virtio_balloon'1957lb # [ 1.764288] systemd-modules-load[68]: Inserted module 'virtio_gpu'1958lb # [ 1.765374] systemd[1]: Finished Create Static Device Nodes in /dev.1959lb # [ 1.767362] systemd[1]: Reached target Preparation for Local File Systems.1960client # [ 1.901266] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1961worker1 # [ 1.917051] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1962worker2 # [ 1.800866] systemd[1]: Finished Create System Files and Directories.1963lb # [ 1.770338] systemd[1]: Reached target Local File Systems.1964lb # [ 1.773922] systemd[1]: Starting Create System Files and Directories...1965lb # [ 1.775245] systemd[1]: Starting Rule-based Manager for Device Events and Files...1966client # [ 1.910389] systemd[1]: Starting Create Static Device Nodes in /dev...1967worker1 # [ 1.927597] systemd[1]: Starting Create Static Device Nodes in /dev...1968client # [ 1.920388] systemd[1]: Started Journal Service.1969client # [ 1.785925] systemd-modules-load[68]: Inserted module 'dm_mod'1970client # [ 1.789990] systemd-modules-load[68]: Inserted module 'virtio_balloon'1971worker2 # [ 1.822241] systemd-udevd[83]: Using default interface naming scheme 'v261'.1972client # [ 1.793072] systemd-modules-load[68]: Inserted module 'virtio_gpu'1973client # [ 1.797161] systemd[1]: Finished Apply Kernel Variables.1974worker1 # [ 1.950017] systemd[1]: Finished Apply Kernel Variables.1975lb # [ 1.806694] systemd[1]: Finished Create System Files and Directories.1976client # [ 1.812113] systemd[1]: Finished Create Static Device Nodes in /dev.1977client # [ 1.813093] systemd[1]: Reached target Preparation for Local File Systems.1978client # [ 1.814064] systemd[1]: Reached target Local File Systems.1979worker2 # [ 1.845261] systemd[1]: Started Rule-based Manager for Device Events and Files.1980client # [ 1.817061] systemd[1]: Starting Create System Files and Directories...1981worker1 # [ 1.965431] systemd[1]: Finished Create Static Device Nodes in /dev.1982worker1 # [ 1.834153] systemd-modules-load[68]: Inserted module 'dm_mod'1983worker1 # [ 1.836422] systemd-modules-load[68]: Inserted module 'virtio_balloon'1984client # [ 1.821888] systemd[1]: Starting Rule-based Manager for Device Events and Files...1985worker1 # [ 1.837612] systemd-modules-load[68]: Inserted module 'virtio_gpu'1986worker1 # [ 1.972108] systemd[1]: Started Journal Service.1987lb # [ 1.831257] systemd-udevd[77]: Using default interface naming scheme 'v261'.1988worker1 # [ 1.851117] systemd[1]: Reached target Preparation for Local File Systems.1989worker1 # [ 1.852486] systemd[1]: Reached target Local File Systems.1990worker1 # [ 1.853657] systemd[1]: Starting Create System Files and Directories...1991worker1 # [ 1.858804] systemd[1]: Starting Rule-based Manager for Device Events and Files...1992client # [ 1.845079] systemd[1]: Finished Create System Files and Directories.1993lb # [ 1.857361] systemd[1]: Started Rule-based Manager for Device Events and Files.1994worker2 # [ 1.893070] systemd[1]: Finished Coldplug All udev Devices.1995worker2 # [ 1.894241] systemd[1]: Reached target System Initialization.1996worker2 # [ 1.895199] systemd[1]: Reached target Basic System.1997client # [ 1.870389] systemd-udevd[82]: Using default interface naming scheme 'v261'.1998worker1 # [ 1.886554] systemd[1]: Finished Create System Files and Directories.1999client # [ 1.894355] systemd[1]: Started Rule-based Manager for Device Events and Files.2000worker1 # [ 1.911267] systemd-udevd[83]: Using default interface naming scheme 'v261'.
loading 2,153 hidden lines…
4154worker2 # [ 20.858542] nix-grpc-daemon[786]: ts=2026-09-21T12:32:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4155worker2 # [ 21.059695] nix-grpc-daemon[786]: ts=2026-09-21T12:32:58Z level=info event=scheduler_connected addr=lb:500514156worker2 # [ 21.063515] nix-grpc-daemon[786]: ts=2026-09-21T12:32:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4157worker1 # [ 21.133449] nix-grpc-daemon[788]: ts=2026-09-21T12:32:57Z level=info event=scheduler_connected addr=lb:500514158worker1 # [ 21.137859] nix-grpc-daemon[788]: ts=2026-09-21T12:32:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4159worker2 # [ 21.264704] nix-grpc-daemon[786]: ts=2026-09-21T12:32:58Z level=info event=scheduler_connected addr=lb:500514160worker2 # [ 21.268493] nix-grpc-daemon[786]: ts=2026-09-21T12:32:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4161worker1 # [ 21.338951] nix-grpc-daemon[788]: ts=2026-09-21T12:32:58Z level=info event=scheduler_connected addr=lb:500514162worker1 # [ 21.342882] nix-grpc-daemon[788]: ts=2026-09-21T12:32:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4163worker2 # [ 21.469604] nix-grpc-daemon[786]: ts=2026-09-21T12:32:58Z level=info event=scheduler_connected addr=lb:500514164worker2 # [ 21.473455] nix-grpc-daemon[786]: ts=2026-09-21T12:32:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4165worker1 # [ 21.543944] nix-grpc-daemon[788]: ts=2026-09-21T12:32:58Z level=info event=scheduler_connected addr=lb:500514166worker1 # [ 21.548271] nix-grpc-daemon[788]: ts=2026-09-21T12:32:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4167worker2 # [ 21.675601] nix-grpc-daemon[786]: ts=2026-09-21T12:32:58Z level=info event=scheduler_connected addr=lb:500514168worker2 # [ 21.679097] nix-grpc-daemon[786]: ts=2026-09-21T12:32:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4169worker1 # [ 21.748849] nix-grpc-daemon[788]: ts=2026-09-21T12:32:58Z level=info event=scheduler_connected addr=lb:500514170worker1 # [ 21.753131] nix-grpc-daemon[788]: ts=2026-09-21T12:32:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4171worker2 # [ 21.880400] nix-grpc-daemon[786]: ts=2026-09-21T12:32:58Z level=info event=scheduler_connected addr=lb:500514172worker2 # [ 21.884141] nix-grpc-daemon[786]: ts=2026-09-21T12:32:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4173worker1 # [ 21.953936] nix-grpc-daemon[788]: ts=2026-09-21T12:32:58Z level=info event=scheduler_connected addr=lb:500514174worker1 # [ 21.958319] nix-grpc-daemon[788]: ts=2026-09-21T12:32:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4175worker2 # [ 22.085265] nix-grpc-daemon[786]: ts=2026-09-21T12:32:59Z level=info event=scheduler_connected addr=lb:500514176worker2 # [ 22.089097] nix-grpc-daemon[786]: ts=2026-09-21T12:32:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4177worker1 # [ 22.161500] nix-grpc-daemon[788]: ts=2026-09-21T12:32:58Z level=info event=scheduler_connected addr=lb:500514178worker1 # [ 22.165433] nix-grpc-daemon[788]: ts=2026-09-21T12:32:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4179worker2 # [ 22.290263] nix-grpc-daemon[786]: ts=2026-09-21T12:32:59Z level=info event=scheduler_connected addr=lb:500514180worker2 # [ 22.294088] nix-grpc-daemon[786]: ts=2026-09-21T12:32:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4181worker1 # [ 22.362585] nix-grpc-daemon[788]: ts=2026-09-21T12:32:59Z level=info event=scheduler_connected addr=lb:500514182worker1 # [ 22.366484] nix-grpc-daemon[788]: ts=2026-09-21T12:32:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4183worker2 # [ 22.495345] nix-grpc-daemon[786]: ts=2026-09-21T12:32:59Z level=info event=scheduler_connected addr=lb:500514184worker2 # [ 22.499150] nix-grpc-daemon[786]: ts=2026-09-21T12:32:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4185worker1 # [ 22.567410] nix-grpc-daemon[788]: ts=2026-09-21T12:32:59Z level=info event=scheduler_connected addr=lb:500514186worker1 # [ 22.571236] nix-grpc-daemon[788]: ts=2026-09-21T12:32:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4187worker2 # [ 22.701154] nix-grpc-daemon[786]: ts=2026-09-21T12:32:59Z level=info event=scheduler_connected addr=lb:500514188worker2 # [ 22.704926] nix-grpc-daemon[786]: ts=2026-09-21T12:32:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4189worker1 # [ 22.772128] nix-grpc-daemon[788]: ts=2026-09-21T12:32:59Z level=info event=scheduler_connected addr=lb:500514190worker1 # [ 22.775842] nix-grpc-daemon[788]: ts=2026-09-21T12:32:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4191worker2 # [ 22.906072] nix-grpc-daemon[786]: ts=2026-09-21T12:32:59Z level=info event=scheduler_connected addr=lb:500514192worker2 # [ 22.909867] nix-grpc-daemon[786]: ts=2026-09-21T12:32:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4193worker1 # [ 22.976767] nix-grpc-daemon[788]: ts=2026-09-21T12:32:59Z level=info event=scheduler_connected addr=lb:500514194worker1 # [ 22.980670] nix-grpc-daemon[788]: ts=2026-09-21T12:32:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4195worker2 # [ 23.111099] nix-grpc-daemon[786]: ts=2026-09-21T12:33:00Z level=info event=scheduler_connected addr=lb:500514196worker2 # [ 23.114884] nix-grpc-daemon[786]: ts=2026-09-21T12:33:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4197worker1 # [ 23.183134] nix-grpc-daemon[788]: ts=2026-09-21T12:32:59Z level=info event=scheduler_connected addr=lb:500514198worker1 # [ 23.186220] nix-grpc-daemon[788]: ts=2026-09-21T12:32:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4199worker2 # [ 23.316124] nix-grpc-daemon[786]: ts=2026-09-21T12:33:00Z level=info event=scheduler_connected addr=lb:500514200worker2 # [ 23.319939] nix-grpc-daemon[786]: ts=2026-09-21T12:33:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4201worker1 # [ 23.383685] nix-grpc-daemon[788]: ts=2026-09-21T12:33:00Z level=info event=scheduler_connected addr=lb:500514202worker1 # [ 23.386914] nix-grpc-daemon[788]: ts=2026-09-21T12:33:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4203worker2 # [ 23.520915] nix-grpc-daemon[786]: ts=2026-09-21T12:33:00Z level=info event=scheduler_connected addr=lb:500514204worker2 # [ 23.524753] nix-grpc-daemon[786]: ts=2026-09-21T12:33:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4205worker1 # [ 23.587984] nix-grpc-daemon[788]: ts=2026-09-21T12:33:00Z level=info event=scheduler_connected addr=lb:500514206worker1 # [ 23.591945] nix-grpc-daemon[788]: ts=2026-09-21T12:33:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4207worker2 # [ 23.727100] nix-grpc-daemon[786]: ts=2026-09-21T12:33:00Z level=info event=scheduler_connected addr=lb:500514208worker2 # [ 23.730328] nix-grpc-daemon[786]: ts=2026-09-21T12:33:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4209worker1 # [ 23.792668] nix-grpc-daemon[788]: ts=2026-09-21T12:33:00Z level=info event=scheduler_connected addr=lb:500514210worker1 # [ 23.796445] nix-grpc-daemon[788]: ts=2026-09-21T12:33:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4211worker2 # [ 23.928316] nix-grpc-daemon[786]: ts=2026-09-21T12:33:00Z level=info event=scheduler_connected addr=lb:500514212worker2 # [ 23.932335] nix-grpc-daemon[786]: ts=2026-09-21T12:33:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4213worker1 # [ 23.997529] nix-grpc-daemon[788]: ts=2026-09-21T12:33:00Z level=info event=scheduler_connected addr=lb:500514214worker1 # [ 24.001351] nix-grpc-daemon[788]: ts=2026-09-21T12:33:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4215worker2 # [ 24.133551] nix-grpc-daemon[786]: ts=2026-09-21T12:33:01Z level=info event=scheduler_connected addr=lb:500514216worker2 # [ 24.137413] nix-grpc-daemon[786]: ts=2026-09-21T12:33:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4217worker1 # [ 24.202897] nix-grpc-daemon[788]: ts=2026-09-21T12:33:01Z level=info event=scheduler_connected addr=lb:500514218worker1 # [ 24.206597] nix-grpc-daemon[788]: ts=2026-09-21T12:33:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4219worker2 # [ 24.338546] nix-grpc-daemon[786]: ts=2026-09-21T12:33:01Z level=info event=scheduler_connected addr=lb:500514220worker2 # [ 24.342383] nix-grpc-daemon[786]: ts=2026-09-21T12:33:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4221worker1 # [ 24.407406] nix-grpc-daemon[788]: ts=2026-09-21T12:33:01Z level=info event=scheduler_connected addr=lb:500514222worker1 # [ 24.411314] nix-grpc-daemon[788]: ts=2026-09-21T12:33:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4223worker2 # [ 24.543615] nix-grpc-daemon[786]: ts=2026-09-21T12:33:01Z level=info event=scheduler_connected addr=lb:500514224worker2 # [ 24.547457] nix-grpc-daemon[786]: ts=2026-09-21T12:33:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4225worker1 # [ 24.612111] nix-grpc-daemon[788]: ts=2026-09-21T12:33:01Z level=info event=scheduler_connected addr=lb:500514226worker1 # [ 24.615995] nix-grpc-daemon[788]: ts=2026-09-21T12:33:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4227worker2 # [ 24.749810] nix-grpc-daemon[786]: ts=2026-09-21T12:33:01Z level=info event=scheduler_connected addr=lb:500514228worker2 # [ 24.752896] nix-grpc-daemon[786]: ts=2026-09-21T12:33:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4229worker1 # [ 24.816685] nix-grpc-daemon[788]: ts=2026-09-21T12:33:01Z level=info event=scheduler_connected addr=lb:500514230worker1 # [ 24.820538] nix-grpc-daemon[788]: ts=2026-09-21T12:33:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4231worker2 # [ 24.950973] nix-grpc-daemon[786]: ts=2026-09-21T12:33:01Z level=info event=scheduler_connected addr=lb:500514232worker2 # [ 24.954855] nix-grpc-daemon[786]: ts=2026-09-21T12:33:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4233worker1 # [ 25.021418] nix-grpc-daemon[788]: ts=2026-09-21T12:33:01Z level=info event=scheduler_connected addr=lb:500514234worker1 # [ 25.025296] nix-grpc-daemon[788]: ts=2026-09-21T12:33:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4235worker2 # [ 25.155940] nix-grpc-daemon[786]: ts=2026-09-21T12:33:02Z level=info event=scheduler_connected addr=lb:500514236worker2 # [ 25.159829] nix-grpc-daemon[786]: ts=2026-09-21T12:33:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4237worker1 # [ 25.227275] nix-grpc-daemon[788]: ts=2026-09-21T12:33:02Z level=info event=scheduler_connected addr=lb:500514238worker1 # [ 25.231117] nix-grpc-daemon[788]: ts=2026-09-21T12:33:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4239worker2 # [ 25.360946] nix-grpc-daemon[786]: ts=2026-09-21T12:33:02Z level=info event=scheduler_connected addr=lb:500514240worker2 # [ 25.364904] nix-grpc-daemon[786]: ts=2026-09-21T12:33:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4241worker1 # [ 25.432169] nix-grpc-daemon[788]: ts=2026-09-21T12:33:02Z level=info event=scheduler_connected addr=lb:500514242worker1 # [ 25.435986] nix-grpc-daemon[788]: ts=2026-09-21T12:33:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4243worker2 # [ 25.565940] nix-grpc-daemon[786]: ts=2026-09-21T12:33:02Z level=info event=scheduler_connected addr=lb:500514244worker2 # [ 25.569796] nix-grpc-daemon[786]: ts=2026-09-21T12:33:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4245worker1 # [ 25.636921] nix-grpc-daemon[788]: ts=2026-09-21T12:33:02Z level=info event=scheduler_connected addr=lb:500514246worker1 # [ 25.641247] nix-grpc-daemon[788]: ts=2026-09-21T12:33:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4247worker2 # [ 25.772585] nix-grpc-daemon[786]: ts=2026-09-21T12:33:02Z level=info event=scheduler_connected addr=lb:500514248worker2 # [ 25.775756] nix-grpc-daemon[786]: ts=2026-09-21T12:33:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4249worker1 # [ 25.841606] nix-grpc-daemon[788]: ts=2026-09-21T12:33:02Z level=info event=scheduler_connected addr=lb:500514250worker1 # [ 25.843414] nix-grpc-daemon[788]: ts=2026-09-21T12:33:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4251worker2 # [ 25.973541] nix-grpc-daemon[786]: ts=2026-09-21T12:33:02Z level=info event=scheduler_connected addr=lb:500514252worker2 # [ 25.977395] nix-grpc-daemon[786]: ts=2026-09-21T12:33:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4253lb # [ 25.948751] rustfs-bucket-start[1013]: mb s3://farm4254lb # [ 25.954409] systemd[1]: Finished rustfs-bucket.service.4255lb # [ 25.960698] systemd[1]: Starting niks3 server...4256worker1 # [ 26.044080] nix-grpc-daemon[788]: ts=2026-09-21T12:33:02Z level=info event=scheduler_connected addr=lb:500514257worker1 # [ 26.045827] nix-grpc-daemon[788]: ts=2026-09-21T12:33:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4258worker2 # [ 26.178465] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=scheduler_connected addr=lb:500514259worker2 # [ 26.182327] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4260worker1 # [ 26.247959] nix-grpc-daemon[788]: ts=2026-09-21T12:33:03Z level=info event=scheduler_connected addr=lb:500514261worker1 # [ 26.251122] nix-grpc-daemon[788]: ts=2026-09-21T12:33:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4262lb # [ 26.257870] postgres[1028]: [1028] ERROR: relation "goose_db_version" does not exist at character 364263lb # [ 26.259152] postgres[1028]: [1028] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4264lb # [ 26.304497] niks3-server[1023]: 2026/09/21 12:33:02 OK 20241026095416_initial_model.sql (24.12ms)4265lb # [ 26.308691] niks3-server[1023]: 2026/09/21 12:33:02 OK 20251210153512_drop_unused_gin_index.sql (2.88ms)4266lb # [ 26.314129] niks3-server[1023]: 2026/09/21 12:33:02 OK 20251218171726_add_pins.sql (3.97ms)4267lb # [ 26.319507] niks3-server[1023]: 2026/09/21 12:33:02 OK 20260628120000_add_object_size_and_stats.sql (5.49ms)4268lb # [ 26.324522] niks3-server[1023]: 2026/09/21 12:33:02 OK 20260905000000_add_claims.sql (3.69ms)4269lb # [ 26.329449] niks3-server[1023]: 2026/09/21 12:33:03 OK 20260920000000_drop_claims.sql (4.86ms)4270lb # [ 26.330616] niks3-server[1023]: 2026/09/21 12:33:03 goose: successfully migrated database to version: 202609200000004271lb # [ 26.337207] niks3-server[1023]: 2026/09/21 12:33:03 OK 1_commit_pending_closure.sql (7.73ms)4272lb # [ 26.339754] niks3-server[1023]: 2026/09/21 12:33:03 OK 2_object_stats_trigger.sql (2.59ms)4273lb # [ 26.340948] niks3-server[1023]: 2026/09/21 12:33:03 goose: up to current file version: 24274lb # [ 26.350378] niks3-server[1023]: 2026/09/21 12:33:03 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key4275worker2 # [ 26.383265] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=scheduler_connected addr=lb:500514276worker2 # [ 26.387112] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4277worker1 # [ 26.448932] nix-grpc-daemon[788]: ts=2026-09-21T12:33:03Z level=info event=scheduler_connected addr=lb:500514278worker1 # [ 26.452701] nix-grpc-daemon[788]: ts=2026-09-21T12:33:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4279lb # [ 26.447448] niks3-server[1023]: 2026/09/21 12:33:03 INFO Read proxy enabled — serving cache objects from S34280lb # [ 26.449236] niks3-server[1023]: 2026/09/21 12:33:03 INFO Using socket-activated listener address=0.0.0.0:57514281lb # [ 26.453939] niks3-server[1023]: 2026/09/21 12:33:03 INFO systemd watchdog enabled interval=15s4282lb # [ 26.455983] systemd[1]: Started niks3 server.4283lb # [ 26.457159] systemd[1]: Reached target Multi-User System.4284lb # [ 26.458833] systemd[1]: Startup finished in 871ms (kernel) + 4.169s (initrd) + 21.418s (userspace) = 26.458s.4285lb # [ 26.460810] niks3-server[1023]: 2026/09/21 12:33:03 INFO Starting HTTP server address=0.0.0.0:57514286lb # [ 26.466583] niks3-server[1023]: 2026/09/21 12:33:03 INFO lead: acquired remote=192.168.1.4:570064287worker2 # [ 26.504663] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=scheduler_take_over4288worker2 # [ 26.588239] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=scheduler_connected addr=lb:500514289worker2 # [ 26.592178] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4290worker1 # [ 26.653479] nix-grpc-daemon[788]: ts=2026-09-21T12:33:03Z level=info event=scheduler_connected addr=lb:500514291worker1 # [ 26.657273] nix-grpc-daemon[788]: ts=2026-09-21T12:33:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4292worker2 # [ 26.794494] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=scheduler_connected addr=lb:500514293worker2 # [ 26.797632] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4294worker1 # [ 26.858333] nix-grpc-daemon[788]: ts=2026-09-21T12:33:03Z level=info event=scheduler_connected addr=lb:500514295worker2 # [ 26.888117] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:419704296worker2 # [ 26.893276] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04297lb: (finished: waiting for unit niks3.service, in 15.61 seconds)4298lb: waiting for TCP port 5751 on localhost4299lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!4300lb: (finished: waiting for TCP port 5751 on localhost, in 0.08 seconds)4301lb: waiting for unit envoy.service4302worker2 # [ 26.995373] nix-grpc-daemon[786]: ts=2026-09-21T12:33:03Z level=info event=scheduler_connected addr=lb:500514303worker2 # [ 27.017087] nix-grpc-daemon[786]: ts=2026-09-21T12:33:04Z level=info event=oidc_jwks provider=mock keys=14304worker2 # [ 27.021401] nix-grpc-daemon[786]: ts=2026-09-21T12:33:04Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:419704305worker2 # [ 27.025649] nix-grpc-daemon[786]: ts=2026-09-21T12:33:04Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04306lb: (finished: waiting for unit envoy.service, in 0.07 seconds)4307lb: waiting for TCP port 50051 on localhost4308lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!4309lb # [ 27.066073] envoy[931]: conn peer=[::1]:35856 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=34310lb: (finished: waiting for TCP port 50051 on localhost, in 0.04 seconds)4311worker2 # [ 27.242537] systemd[1]: Starting scheduler-token.service...4312worker2 # [ 27.311919] systemd[1]: scheduler-token.service: Deactivated successfully.4313worker2 # [ 27.314448] systemd[1]: Finished scheduler-token.service.4314worker2 # [ 27.315479] systemd[1]: scheduler-token.service: Consumed 20ms CPU time over 74ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4315lb: waiting until exactly no endpoints of x86_64-linux are unhealthy4316lb: must succeed: curl -sf localhost:9901/clusters4317lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)4318lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.09 seconds)4319subtest: one node schedules and both builders hold a WorkerSession on it4320lb: must succeed: curl -sf localhost:9901/clusters4321lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4322worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4323worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4324lb: must succeed: curl -sf localhost:9901/clusters4325lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4326lb: must succeed: curl -sf localhost:9901/clusters4327lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4328lb: must succeed: curl -sf localhost:9901/clusters4329lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4330lb: must succeed: curl -sf localhost:9901/clusters4331lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4332worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4333worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.05 seconds)4334worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4335worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4336lb: must succeed: curl -sf localhost:9901/clusters4337lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4338lb: must succeed: curl -sf localhost:9901/clusters4339lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4340worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4341worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds)4342worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4343worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4344worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4345worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4346worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4347worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4348(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.82 seconds)4349subtest: balancer auth: client cert, bearer token, nothing, unknown CN4350worker2 # [ 28.616149] nix-grpc-daemon[786]: ts=2026-09-21T12:33:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:419864351worker2 # [ 28.629984] systemd[1]: Started Nix Daemon.4352worker2 # [ 28.893807] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4353worker2 # [ 28.920620] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 73 ms (attempt 1/5)4354worker2 # [ 28.996063] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 95 ms (attempt 2/5)4355worker2 # [ 29.093911] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 241 ms (attempt 3/5)4356worker2 # [ 29.341069] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 475 ms (attempt 4/5)4357worker2 # [ 29.827536] nix-grpc-daemon[786]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4358worker2 # [ 29.831533] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4359worker1 # [ 29.856473] nix-grpc-daemon[788]: ts=2026-09-21T12:33:06Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:360024360worker1 # [ 29.868990] systemd[1]: Started Nix Daemon.4361worker1 # [ 30.123318] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4362worker1 # [ 30.151975] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 20 ms (attempt 1/5)4363worker1 # [ 30.174716] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 2/5)4364worker1 # [ 30.203433] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 9 ms (attempt 3/5)4365worker1 # [ 30.214880] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 284 ms (attempt 4/5)4366worker1 # [ 30.505568] nix-grpc-daemon[788]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4367worker1 # [ 30.509739] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4368worker1 # [ 30.527397] nix-grpc-daemon[788]: ts=2026-09-21T12:33:07Z level=debug event=path_info_miss path=lgjx0fgzgnqvb086gw02890w1401r9xx-nixos-system-client-test error="error: path '/nix/store/lgjx0fgzgnqvb086gw02890w1401r9xx-nixos-system-client-test' is required, but there is no substituter that can build it"4369worker1 # [ 30.540336] nix-grpc-daemon[788]: ts=2026-09-21T12:33:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:360024370worker1 # [ 30.547050] nix-grpc-daemon[788]: ts=2026-09-21T12:33:07Z level=debug event=path_info_miss path=lgjx0fgzgnqvb086gw02890w1401r9xx-nixos-system-client-test error="error: path '/nix/store/lgjx0fgzgnqvb086gw02890w1401r9xx-nixos-system-client-test' is required, but there is no substituter that can build it"4371client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4372client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds)4373client: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4374client: (finished: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4375worker1 # [ 30.961213] nix-grpc-daemon[788]: ts=2026-09-21T12:33:07Z level=info event=oidc_jwks provider=mock keys=14376worker1 # [ 30.965301] nix-grpc-daemon[788]: ts=2026-09-21T12:33:07Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:360024377worker1 # [ 30.982153] nix-grpc-daemon[788]: ts=2026-09-21T12:33:07Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:360024378worker1 # [ 30.991699] nix-grpc-daemon[788]: ts=2026-09-21T12:33:07Z level=debug event=path_info_miss path=lgjx0fgzgnqvb086gw02890w1401r9xx-nixos-system-client-test error="error: path '/nix/store/lgjx0fgzgnqvb086gw02890w1401r9xx-nixos-system-client-test' is required, but there is no substituter that can build it"4379worker1 # [ 31.002664] nix-grpc-daemon[788]: ts=2026-09-21T12:33:07Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:360024380worker1 # [ 31.012319] nix-grpc-daemon[788]: ts=2026-09-21T12:33:07Z level=debug event=path_info_miss path=lgjx0fgzgnqvb086gw02890w1401r9xx-nixos-system-client-test error="error: path '/nix/store/lgjx0fgzgnqvb086gw02890w1401r9xx-nixos-system-client-test' is required, but there is no substituter that can build it"4381client: must succeed: rm -r /var/lib/nix-grpc-store4382client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds)4383worker1 # [ 31.294127] nix-grpc-daemon[788]: ts=2026-09-21T12:33:08Z level=info event=denied method=QueryMissing cn=- role=none4384worker1 # [ 31.551475] nix-grpc-daemon[788]: ts=2026-09-21T12:33:08Z level=info event=denied method=QueryMissing cn=stranger role=none4385(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.37 seconds)4386subtest: DAG build is scheduled across workers and lands in the cache4387client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24388worker1 # [ 31.809921] nix-grpc-daemon[788]: ts=2026-09-21T12:33:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:360024389client # don't know how to build these paths:4390client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4391worker2 # [ 31.840943] nix-grpc-daemon[786]: ts=2026-09-21T12:33:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419864392worker2 # [ 31.881427] nix-grpc-daemon[786]: ts=2026-09-21T12:33:08Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04393worker2 # [ 31.886128] nix-grpc-daemon[786]: ts=2026-09-21T12:33:08Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14394worker1 # [ 31.929927] nix-grpc-daemon[788]: ts=2026-09-21T12:33:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024395worker1 # [ 31.954488] nix-grpc-daemon[788]: ts=2026-09-21T12:33:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:360024396client # copying 1 paths to 'lb:50051'...4397worker1 # [ 31.970381] nix-grpc-daemon[788]: ts=2026-09-21T12:33:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:360024398client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4399worker1 # [ 31.999403] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4400worker1 # [ 32.031137] nix-grpc-daemon[788]: ts=2026-09-21T12:33:08Z level=info event=niks3_push_started pid=8644401worker1 # [ 32.439867] nix-grpc-daemon[864]: time=2026-09-21T12:33:09.242Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""4402worker1 # [ 32.691720] nix-daemon[852]: accepted connection from pid 877, user nix-grpc-daemon (trusted)4403worker1 # [ 32.706307] nix-daemon[852]: reaped child process 880, status = succeeded4404worker1 # [ 32.934818] nix-daemon[852]: accepted connection from pid 882, user nix-grpc-daemon (trusted)4405worker1 # [ 32.958905] nix-daemon[852]: reaped child process 885, status = succeeded4406lb # [ 32.961449] niks3-server[1023]: 2026/09/21 12:33:09 INFO Received uploads request method=POST path=/api/pending_closures4407worker1 # [ 33.004612] nix-grpc-daemon[864]: time=2026-09-21T12:33:09.812Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4408worker1 # [ 33.007340] nix-grpc-daemon[864]: time=2026-09-21T12:33:09.815Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4409lb # [ 33.036479] niks3-server[1023]: 2026/09/21 12:33:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4410lb # [ 33.048328] niks3-server[1023]: 2026/09/21 12:33:09 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4411lb # [ 33.062122] niks3-server[1023]: 2026/09/21 12:33:09 INFO Signed narinfos id=1 count=14412worker1 # [ 33.079237] nix-grpc-daemon[864]: time=2026-09-21T12:33:09.887Z level=INFO msg="Uploading 1 narinfos"4413lb # [ 33.071520] niks3-server[1023]: 2026/09/21 12:33:09 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4414lb # [ 33.075913] niks3-server[1023]: 2026/09/21 12:33:09 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4415lb # [ 33.084861] niks3-server[1023]: 2026/09/21 12:33:09 INFO Completed upload id=14416worker1 # [ 33.100668] nix-grpc-daemon[864]: time=2026-09-21T12:33:09.908Z level=INFO msg="Upload complete. (635ms)"4417lb # [ 33.091381] niks3-server[1023]: 2026/09/21 12:33:09 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4418worker1 # [ 33.107238] nix-grpc-daemon[788]: ts=2026-09-21T12:33:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=1 paths=1 nar_bytes_in=5044419worker1 # [ 33.113685] nix-grpc-daemon[788]: ts=2026-09-21T12:33:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024420worker1 # [ 33.123545] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4421worker1 # [ 33.131135] nix-daemon[852]: reaped child process 862, status = succeeded4422client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4423worker1 # [ 33.138358] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4424client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4425client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4426client # farm-a> LOG-farm-a-t14427worker1 # [ 34.350461] nix-daemon[852]: reaped child process 890, status = succeeded4428worker1 # [ 34.552915] nix-daemon[852]: accepted connection from pid 906, user nix-grpc-daemon (trusted)4429worker1 # [ 34.564595] nix-daemon[852]: reaped child process 909, status = succeeded4430lb # [ 34.573445] niks3-server[1023]: 2026/09/21 12:33:11 INFO Received uploads request method=POST path=/api/pending_closures4431worker1 # [ 34.601734] nix-grpc-daemon[864]: time=2026-09-21T12:33:11.409Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4432worker1 # [ 34.603562] nix-grpc-daemon[864]: time=2026-09-21T12:33:11.411Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4433lb # [ 34.624490] niks3-server[1023]: 2026/09/21 12:33:11 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4434lb # [ 34.637332] niks3-server[1023]: 2026/09/21 12:33:11 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4435lb # [ 34.640767] niks3-server[1023]: 2026/09/21 12:33:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4436worker1 # [ 34.661626] nix-grpc-daemon[864]: time=2026-09-21T12:33:11.469Z level=INFO msg="Uploading 1 narinfos"4437lb # [ 34.645827] niks3-server[1023]: 2026/09/21 12:33:11 INFO Signed narinfos id=2 count=14438lb # [ 34.651493] niks3-server[1023]: 2026/09/21 12:33:11 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4439lb # [ 34.659425] niks3-server[1023]: 2026/09/21 12:33:11 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4440lb # [ 34.665794] niks3-server[1023]: 2026/09/21 12:33:11 INFO Completed upload id=24441lb # [ 34.667626] niks3-server[1023]: 2026/09/21 12:33:11 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4442worker1 # [ 34.681493] nix-grpc-daemon[864]: time=2026-09-21T12:33:11.489Z level=INFO msg="Upload complete. (335ms)"4443worker1 # [ 34.688601] nix-grpc-daemon[788]: ts=2026-09-21T12:33:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14444worker1 # [ 34.701962] nix-daemon[852]: reaped child process 888, status = succeeded4445worker2 # [ 34.727536] nix-grpc-daemon[786]: ts=2026-09-21T12:33:11Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04446worker2 # [ 34.731539] nix-grpc-daemon[786]: ts=2026-09-21T12:33:11Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14447worker1 # [ 34.756376] nix-grpc-daemon[788]: ts=2026-09-21T12:33:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024448worker1 # [ 34.770329] nix-grpc-daemon[788]: ts=2026-09-21T12:33:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:360024449worker1 # [ 34.785656] nix-grpc-daemon[864]: time=2026-09-21T12:33:11.593Z level=INFO msg="All 1 paths already cached"4450client # copying 1 paths to 'lb:50051'...4451worker1 # [ 34.794543] nix-grpc-daemon[788]: ts=2026-09-21T12:33:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:360024452client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4453worker1 # [ 34.815185] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4454worker1 # [ 35.056823] nix-daemon[852]: accepted connection from pid 913, user nix-grpc-daemon (trusted)4455worker1 # [ 35.070591] nix-daemon[852]: reaped child process 916, status = succeeded4456worker1 # [ 35.301099] nix-daemon[852]: accepted connection from pid 918, user nix-grpc-daemon (trusted)4457worker1 # [ 35.313486] nix-daemon[852]: reaped child process 921, status = succeeded4458lb # [ 35.321858] niks3-server[1023]: 2026/09/21 12:33:11 INFO Received uploads request method=POST path=/api/pending_closures4459worker1 # [ 35.350176] nix-grpc-daemon[864]: time=2026-09-21T12:33:12.158Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4460worker1 # [ 35.351819] nix-grpc-daemon[864]: time=2026-09-21T12:33:12.159Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4461lb # [ 35.356536] niks3-server[1023]: 2026/09/21 12:33:12 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4462lb # [ 35.368323] niks3-server[1023]: 2026/09/21 12:33:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4463worker1 # [ 35.390403] nix-grpc-daemon[864]: time=2026-09-21T12:33:12.198Z level=INFO msg="Uploading 1 narinfos"4464lb # [ 35.374986] niks3-server[1023]: 2026/09/21 12:33:12 INFO Signed narinfos id=3 count=14465lb # [ 35.379325] niks3-server[1023]: 2026/09/21 12:33:12 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4466lb # [ 35.389450] niks3-server[1023]: 2026/09/21 12:33:12 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4467lb # [ 35.396342] niks3-server[1023]: 2026/09/21 12:33:12 INFO Completed upload id=34468worker1 # [ 35.411938] nix-grpc-daemon[864]: time=2026-09-21T12:33:12.219Z level=INFO msg="Upload complete. (580ms)"4469lb # [ 35.399507] niks3-server[1023]: 2026/09/21 12:33:12 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4470worker1 # [ 35.416341] nix-grpc-daemon[788]: ts=2026-09-21T12:33:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=0 paths=1 nar_bytes_in=6244471worker1 # [ 35.422283] nix-grpc-daemon[788]: ts=2026-09-21T12:33:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024472worker1 # [ 35.431712] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4473worker1 # [ 35.437924] nix-daemon[852]: reaped child process 911, status = succeeded4474worker1 # [ 35.442786] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4475client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4476client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4477client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4478client # farm-b> LOG-farm-b-t14479worker1 # [ 36.873340] nix-daemon[852]: reaped child process 925, status = succeeded4480worker1 # [ 37.082563] nix-daemon[852]: accepted connection from pid 942, user nix-grpc-daemon (trusted)4481worker1 # [ 37.095443] nix-daemon[852]: reaped child process 945, status = succeeded4482lb # [ 37.101090] niks3-server[1023]: 2026/09/21 12:33:13 INFO Received uploads request method=POST path=/api/pending_closures4483worker1 # [ 37.122098] nix-grpc-daemon[864]: time=2026-09-21T12:33:13.929Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4484worker1 # [ 37.123600] nix-grpc-daemon[864]: time=2026-09-21T12:33:13.931Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4485lb # [ 37.120715] niks3-server[1023]: 2026/09/21 12:33:13 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4486lb # [ 37.125489] niks3-server[1023]: 2026/09/21 12:33:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4487lb # [ 37.127876] niks3-server[1023]: 2026/09/21 12:33:13 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4488lb # [ 37.130784] niks3-server[1023]: 2026/09/21 12:33:13 INFO Signed narinfos id=4 count=14489worker1 # [ 37.147791] nix-grpc-daemon[864]: time=2026-09-21T12:33:13.955Z level=INFO msg="Uploading 1 narinfos"4490lb # [ 37.133212] niks3-server[1023]: 2026/09/21 12:33:13 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4491lb # [ 37.137924] niks3-server[1023]: 2026/09/21 12:33:13 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4492lb # [ 37.143804] niks3-server[1023]: 2026/09/21 12:33:13 INFO Completed upload id=44493worker1 # [ 37.159293] nix-grpc-daemon[864]: time=2026-09-21T12:33:13.967Z level=INFO msg="Upload complete. (307ms)"4494lb # [ 37.145091] niks3-server[1023]: 2026/09/21 12:33:13 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4495worker1 # [ 37.161653] nix-grpc-daemon[788]: ts=2026-09-21T12:33:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14496worker1 # [ 37.168092] nix-daemon[852]: reaped child process 923, status = succeeded4497worker2 # [ 37.187837] nix-grpc-daemon[786]: ts=2026-09-21T12:33:14Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04498worker2 # [ 37.192286] nix-grpc-daemon[786]: ts=2026-09-21T12:33:14Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14499worker1 # [ 37.220084] nix-grpc-daemon[788]: ts=2026-09-21T12:33:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024500worker1 # [ 37.232586] nix-grpc-daemon[788]: ts=2026-09-21T12:33:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:360024501worker1 # [ 37.239237] nix-grpc-daemon[864]: time=2026-09-21T12:33:14.047Z level=INFO msg="All 2 paths already cached"4502client # copying 1 paths to 'lb:50051'...4503worker1 # [ 37.244167] nix-grpc-daemon[788]: ts=2026-09-21T12:33:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:360024504client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4505worker1 # [ 37.261189] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4506worker1 # [ 37.463475] nix-daemon[852]: accepted connection from pid 949, user nix-grpc-daemon (trusted)4507worker1 # [ 37.475517] nix-daemon[852]: reaped child process 952, status = succeeded4508worker1 # [ 37.693293] nix-daemon[852]: accepted connection from pid 954, user nix-grpc-daemon (trusted)4509worker1 # [ 37.705057] nix-daemon[852]: reaped child process 957, status = succeeded4510lb # [ 37.711986] niks3-server[1023]: 2026/09/21 12:33:14 INFO Received uploads request method=POST path=/api/pending_closures4511worker1 # [ 37.732621] nix-grpc-daemon[864]: time=2026-09-21T12:33:14.540Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4512worker1 # [ 37.734042] nix-grpc-daemon[864]: time=2026-09-21T12:33:14.540Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4513lb # [ 37.727211] niks3-server[1023]: 2026/09/21 12:33:14 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4514lb # [ 37.728941] niks3-server[1023]: 2026/09/21 12:33:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4515lb # [ 37.731473] niks3-server[1023]: 2026/09/21 12:33:14 INFO Signed narinfos id=5 count=14516worker1 # [ 37.746817] nix-grpc-daemon[864]: time=2026-09-21T12:33:14.554Z level=INFO msg="Uploading 1 narinfos"4517lb # [ 37.734616] niks3-server[1023]: 2026/09/21 12:33:14 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4518lb # [ 37.738325] niks3-server[1023]: 2026/09/21 12:33:14 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4519lb # [ 37.742423] niks3-server[1023]: 2026/09/21 12:33:14 INFO Completed upload id=54520worker1 # [ 37.759115] nix-grpc-daemon[864]: time=2026-09-21T12:33:14.567Z level=INFO msg="Upload complete. (491ms)"4521lb # [ 37.744398] niks3-server[1023]: 2026/09/21 12:33:14 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4522worker1 # [ 37.760900] nix-grpc-daemon[788]: ts=2026-09-21T12:33:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=0 paths=1 nar_bytes_in=7524523worker1 # [ 37.764628] nix-grpc-daemon[788]: ts=2026-09-21T12:33:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024524worker1 # [ 37.770894] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4525worker1 # [ 37.773211] nix-daemon[852]: reaped child process 947, status = succeeded4526worker1 # [ 37.778912] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4527client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4528client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4529client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4530lb # [ 39.253744] envoy[931]: conn peer=[2001:db8:1::1]:46602 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=382 ms=19984531lb # [ 39.255556] envoy[931]: conn peer=[2001:db8:1::1]:46614 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=804532lb # [ 39.256958] envoy[931]: conn peer=[2001:db8:1::1]:46618 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=144533lb # [ 39.258331] envoy[931]: conn peer=[2001:db8:1::1]:46624 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=84534client # farm-top> LOG-farm-top-t14535worker1 # [ 39.392761] nix-daemon[852]: reaped child process 961, status = succeeded4536worker1 # [ 39.588820] nix-daemon[852]: accepted connection from pid 978, user nix-grpc-daemon (trusted)4537worker1 # [ 39.599626] nix-daemon[852]: reaped child process 981, status = succeeded4538lb # [ 39.606335] niks3-server[1023]: 2026/09/21 12:33:16 INFO Received uploads request method=POST path=/api/pending_closures4539worker1 # [ 39.629636] nix-grpc-daemon[864]: time=2026-09-21T12:33:16.437Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4540worker1 # [ 39.632060] nix-grpc-daemon[864]: time=2026-09-21T12:33:16.439Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4541lb # [ 39.629224] niks3-server[1023]: 2026/09/21 12:33:16 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4542lb # [ 39.632242] niks3-server[1023]: 2026/09/21 12:33:16 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4543lb # [ 39.636632] niks3-server[1023]: 2026/09/21 12:33:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4544lb # [ 39.638782] niks3-server[1023]: 2026/09/21 12:33:16 INFO Signed narinfos id=6 count=14545worker1 # [ 39.654287] nix-grpc-daemon[864]: time=2026-09-21T12:33:16.462Z level=INFO msg="Uploading 1 narinfos"4546lb # [ 39.641332] niks3-server[1023]: 2026/09/21 12:33:16 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4547lb # [ 39.646986] niks3-server[1023]: 2026/09/21 12:33:16 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4548lb # [ 39.650918] niks3-server[1023]: 2026/09/21 12:33:16 INFO Completed upload id=64549worker1 # [ 39.666484] nix-grpc-daemon[864]: time=2026-09-21T12:33:16.474Z level=INFO msg="Upload complete. (280ms)"4550lb # [ 39.652922] niks3-server[1023]: 2026/09/21 12:33:16 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4551worker1 # [ 39.669127] nix-grpc-daemon[788]: ts=2026-09-21T12:33:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14552worker1 # [ 39.675591] nix-daemon[852]: reaped child process 959, status = succeeded4553client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 8.14 seconds)4554client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4555client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.23 seconds)4556client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4557client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.03 seconds)4558client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4559client # copying 3 paths...4560client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4561client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4562client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4563client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.30 seconds)4564(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 8.80 seconds)4565subtest: repeat is answered Cached by the scheduler without building4566worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4567worker1 # finding garbage collector roots...4568worker1 # deleting specified paths...4569worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4570worker1 # deleting unused links...4571worker1 # note: hard linking is currently saving 0.0 KiB4572worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.32 seconds)4573worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4574worker2 # finding garbage collector roots...4575worker2 # deleting specified paths...4576worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4577worker2 # deleting unused links...4578worker2 # note: hard linking is currently saving 0.0 KiB4579worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.34 seconds)4580client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24581worker1 # [ 41.275707] nix-grpc-daemon[788]: ts=2026-09-21T12:33:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:360024582client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4583client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4584worker1 # [ 41.290347] nix-grpc-daemon[788]: ts=2026-09-21T12:33:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:360024585worker1 # [ 41.299629] nix-grpc-daemon[864]: time=2026-09-21T12:33:18.107Z level=INFO msg="All 2 paths already cached"4586worker2 # [ 41.329908] nix-grpc-daemon[786]: ts=2026-09-21T12:33:18Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14587client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.31 seconds)4588client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4589client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.20 seconds)4590lb: must succeed: curl -sf localhost:9901/clusters4591lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4592lb: must succeed: curl -sf localhost:9901/clusters4593lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4594worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4595worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4596(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.45 seconds)4597subtest: inputs already in the cache are not sent to the scheduler4598lb: must succeed: curl -sf localhost:9901/clusters4599lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4600lb: must succeed: curl -sf localhost:9901/clusters4601lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4602worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4603worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4604client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24605client # don't know how to build these paths:4606client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4607worker1 # [ 42.257229] nix-grpc-daemon[788]: ts=2026-09-21T12:33:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:360024608worker2 # [ 42.276445] nix-grpc-daemon[786]: ts=2026-09-21T12:33:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419864609worker2 # [ 42.295412] nix-grpc-daemon[786]: ts=2026-09-21T12:33:19Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04610worker2 # [ 42.297268] nix-grpc-daemon[786]: ts=2026-09-21T12:33:19Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14611worker1 # [ 42.325184] nix-grpc-daemon[788]: ts=2026-09-21T12:33:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024612worker1 # [ 42.337502] nix-grpc-daemon[788]: ts=2026-09-21T12:33:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:360024613worker1 # [ 42.351249] nix-grpc-daemon[864]: time=2026-09-21T12:33:19.158Z level=INFO msg="All 2 paths already cached"4614client # copying 1 paths to 'lb:50051'...4615client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4616worker1 # [ 42.360081] nix-grpc-daemon[788]: ts=2026-09-21T12:33:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:360024617worker1 # [ 42.371354] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4618worker1 # [ 42.600581] nix-daemon[852]: accepted connection from pid 998, user nix-grpc-daemon (trusted)4619worker1 # [ 42.614432] nix-daemon[852]: reaped child process 1002, status = succeeded4620worker1 # [ 42.835430] nix-daemon[852]: accepted connection from pid 1004, user nix-grpc-daemon (trusted)4621worker1 # [ 42.847527] nix-daemon[852]: reaped child process 1007, status = succeeded4622lb # [ 42.856397] niks3-server[1023]: 2026/09/21 12:33:19 INFO Received uploads request method=POST path=/api/pending_closures4623worker1 # [ 42.880555] nix-grpc-daemon[864]: time=2026-09-21T12:33:19.688Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4624worker1 # [ 42.881935] nix-grpc-daemon[864]: time=2026-09-21T12:33:19.688Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4625lb # [ 42.883837] niks3-server[1023]: 2026/09/21 12:33:19 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4626lb # [ 42.892200] niks3-server[1023]: 2026/09/21 12:33:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4627lb # [ 42.897233] niks3-server[1023]: 2026/09/21 12:33:19 INFO Signed narinfos id=7 count=14628worker1 # [ 42.915769] nix-grpc-daemon[864]: time=2026-09-21T12:33:19.723Z level=INFO msg="Uploading 1 narinfos"4629lb # [ 42.907992] niks3-server[1023]: 2026/09/21 12:33:19 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4630lb # [ 42.914966] niks3-server[1023]: 2026/09/21 12:33:19 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4631worker1 # [ 42.936449] nix-grpc-daemon[864]: time=2026-09-21T12:33:19.744Z level=INFO msg="Upload complete. (553ms)"4632lb # [ 42.921039] niks3-server[1023]: 2026/09/21 12:33:19 INFO Completed upload id=74633worker1 # [ 42.939055] nix-grpc-daemon[788]: ts=2026-09-21T12:33:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=0 paths=1 nar_bytes_in=7524634lb # [ 42.926365] niks3-server[1023]: 2026/09/21 12:33:19 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4635worker1 # [ 42.944877] nix-grpc-daemon[788]: ts=2026-09-21T12:33:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024636worker1 # [ 42.955181] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4637worker1 # [ 42.957643] nix-daemon[852]: reaped child process 996, status = succeeded4638worker1 # [ 42.963203] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4639client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4640client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4641client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4642client # farm-top> LOG-farm-top-t1b4643worker1 # [ 44.377753] nix-daemon[852]: reaped child process 1011, status = succeeded4644worker1 # [ 44.585414] nix-daemon[852]: accepted connection from pid 1027, user nix-grpc-daemon (trusted)4645worker1 # [ 44.597402] nix-daemon[852]: reaped child process 1030, status = succeeded4646lb # [ 44.606327] niks3-server[1023]: 2026/09/21 12:33:21 INFO Received uploads request method=POST path=/api/pending_closures4647worker1 # [ 44.630301] nix-grpc-daemon[864]: time=2026-09-21T12:33:21.438Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4648worker1 # [ 44.632205] nix-grpc-daemon[864]: time=2026-09-21T12:33:21.439Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4649lb # [ 44.647354] niks3-server[1023]: 2026/09/21 12:33:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4650worker1 # [ 44.668675] nix-grpc-daemon[864]: time=2026-09-21T12:33:21.476Z level=INFO msg="Uploading 1 narinfos"4651lb # [ 44.653130] niks3-server[1023]: 2026/09/21 12:33:21 INFO Signed narinfos id=8 count=14652lb # [ 44.666783] niks3-server[1023]: 2026/09/21 12:33:21 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4653lb # [ 44.670217] niks3-server[1023]: 2026/09/21 12:33:21 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4654lb # [ 44.676542] niks3-server[1023]: 2026/09/21 12:33:21 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4655lb # [ 44.684255] niks3-server[1023]: 2026/09/21 12:33:21 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4656lb # [ 44.689817] niks3-server[1023]: 2026/09/21 12:33:21 INFO Completed upload id=84657lb # [ 44.691482] niks3-server[1023]: 2026/09/21 12:33:21 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4658worker1 # [ 44.706662] nix-grpc-daemon[864]: time=2026-09-21T12:33:21.514Z level=INFO msg="Upload complete. (336ms)"4659worker1 # [ 44.713210] nix-grpc-daemon[788]: ts=2026-09-21T12:33:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14660worker1 # [ 44.725237] nix-daemon[852]: reaped child process 1009, status = succeeded4661client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.75 seconds)4662client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4663client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.23 seconds)4664lb: must succeed: curl -sf localhost:9901/clusters4665lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4666lb: must succeed: curl -sf localhost:9901/clusters4667lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4668worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4669worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4670(finished: subtest: inputs already in the cache are not sent to the scheduler, in 3.36 seconds)4671subtest: two clients wanting the same drv build it once4672client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4673client # [ 45.239812] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4674client # Running as unit: dedup1.service; invocation ID: db2982484a28425d9279669655cf333c4675client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.08 seconds)4676client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4677worker1 # [ 45.502363] nix-grpc-daemon[788]: ts=2026-09-21T12:33:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:360024678client # [ 45.492751] nix[892]: don't know how to build these paths:4679client # [ 45.493638] nix[892]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4680worker2 # [ 45.533461] nix-grpc-daemon[786]: ts=2026-09-21T12:33:22Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04681worker2 # [ 45.539315] nix-grpc-daemon[786]: ts=2026-09-21T12:33:22Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=14682worker2 # [ 45.544142] nix-grpc-daemon[786]: ts=2026-09-21T12:33:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419864683worker2 # [ 45.567233] nix-grpc-daemon[786]: ts=2026-09-21T12:33:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419864684client # [ 45.545980] nix[892]: copying 1 paths to 'lb:50051'...4685client # [ 45.546775] nix[892]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4686worker2 # [ 45.578285] nix-grpc-daemon[786]: ts=2026-09-21T12:33:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419864687worker2 # [ 45.590787] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4688worker2 # [ 45.611383] nix-grpc-daemon[786]: ts=2026-09-21T12:33:22Z level=info event=niks3_push_started pid=9244689worker2 # [ 45.952164] nix-grpc-daemon[924]: time=2026-09-21T12:33:22.931Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""4690worker2 # [ 46.195620] nix-daemon[869]: accepted connection from pid 937, user nix-grpc-daemon (trusted)4691worker2 # [ 46.209792] nix-daemon[869]: reaped child process 940, status = succeeded4692client # [ 46.322586] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4693client # Running as unit: dedup2.service; invocation ID: 5397d441ca3f4ec78d48c6206164e1d24694client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.09 seconds)4695worker2 # [ 46.416629] nix-daemon[869]: accepted connection from pid 942, user nix-grpc-daemon (trusted)4696worker2 # [ 46.428907] nix-daemon[869]: reaped child process 945, status = succeeded4697lb # [ 46.421147] niks3-server[1023]: 2026/09/21 12:33:23 INFO Received uploads request method=POST path=/api/pending_closures4698worker2 # [ 46.461767] nix-grpc-daemon[924]: time=2026-09-21T12:33:23.445Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4699worker2 # [ 46.464620] nix-grpc-daemon[924]: time=2026-09-21T12:33:23.448Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4700lb # [ 46.459864] niks3-server[1023]: 2026/09/21 12:33:23 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4701lb # [ 46.477420] niks3-server[1023]: 2026/09/21 12:33:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4702lb # [ 46.483278] niks3-server[1023]: 2026/09/21 12:33:23 INFO Signed narinfos id=9 count=14703worker2 # [ 46.520552] nix-grpc-daemon[924]: time=2026-09-21T12:33:23.504Z level=INFO msg="Uploading 1 narinfos"4704lb # [ 46.492843] niks3-server[1023]: 2026/09/21 12:33:23 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4705lb # [ 46.501920] niks3-server[1023]: 2026/09/21 12:33:23 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4706lb # [ 46.509627] niks3-server[1023]: 2026/09/21 12:33:23 INFO Completed upload id=94707worker2 # [ 46.540900] nix-grpc-daemon[924]: time=2026-09-21T12:33:23.524Z level=INFO msg="Upload complete. (569ms)"4708lb # [ 46.511478] niks3-server[1023]: 2026/09/21 12:33:23 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4709worker2 # [ 46.542596] nix-grpc-daemon[786]: ts=2026-09-21T12:33:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=0 paths=1 nar_bytes_in=4324710worker2 # [ 46.549331] nix-grpc-daemon[786]: ts=2026-09-21T12:33:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419864711worker2 # [ 46.553737] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4712worker2 # [ 46.556036] nix-daemon[869]: reaped child process 922, status = succeeded4713client # [ 46.529571] nix[892]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4714worker2 # [ 46.562228] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4715worker2 # [ 46.643270] nix-grpc-daemon[786]: ts=2026-09-21T12:33:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:419864716client # [ 46.623626] nix[908]: this derivation will be built:4717worker1 # [ 46.641584] nix-grpc-daemon[788]: ts=2026-09-21T12:33:23Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:360024718worker1 # [ 46.657761] nix-grpc-daemon[788]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' from 'http://lb:5751'...4719client # [ 46.664214] nix[908]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4720worker2 # [ 46.704420] nix-grpc-daemon[786]: ts=2026-09-21T12:33:23Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04721worker2 # [ 46.710633] nix-grpc-daemon[786]: ts=2026-09-21T12:33:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419864722client # [ 46.684356] nix[908]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4723client: must succeed: sleep 24724lb # [ 49.254428] envoy[931]: conn peer=[2001:db8:1::1]:46640 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9580 tx=4090 ms=78954725lb # [ 49.258380] envoy[931]: conn peer=[2001:db8:1::1]:44406 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=464726lb # [ 49.261831] envoy[931]: conn peer=[2001:db8:1::1]:44422 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4169 tx=1807 ms=24684727worker1 # [ 49.674805] nix-grpc-daemon[788]: E0000 00:00:1789994006.481353 829 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4728worker1 # [ 49.705153] nix-grpc-daemon[788]: E0000 00:00:1789994006.512264 829 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4729worker1 # [ 49.710343] nix-grpc-daemon[788]: E0000 00:00:1789994006.512477 829 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4730worker1 # [ 49.735061] nix-grpc-daemon[788]: E0000 00:00:1789994006.542956 829 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4731client: (finished: must succeed: sleep 2, in 2.06 seconds)4732client: waiting for success: ! systemctl is-active dedup1 dedup24733worker2 # [ 51.179824] nix-daemon[869]: reaped child process 953, status = succeeded4734worker2 # [ 51.371847] nix-daemon[869]: accepted connection from pid 985, user nix-grpc-daemon (trusted)4735worker2 # [ 51.382670] nix-daemon[869]: reaped child process 988, status = succeeded4736lb # [ 51.371753] niks3-server[1023]: 2026/09/21 12:33:28 INFO Received uploads request method=POST path=/api/pending_closures4737worker2 # [ 51.411755] nix-grpc-daemon[924]: time=2026-09-21T12:33:28.395Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4738worker2 # [ 51.413432] nix-grpc-daemon[924]: time=2026-09-21T12:33:28.397Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4739lb # [ 51.411174] niks3-server[1023]: 2026/09/21 12:33:28 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4740lb # [ 51.427272] niks3-server[1023]: 2026/09/21 12:33:28 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4741lb # [ 51.436250] niks3-server[1023]: 2026/09/21 12:33:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4742lb # [ 51.442129] niks3-server[1023]: 2026/09/21 12:33:28 INFO Signed narinfos id=10 count=14743worker2 # [ 51.476662] nix-grpc-daemon[924]: time=2026-09-21T12:33:28.460Z level=INFO msg="Uploading 1 narinfos"4744lb # [ 51.445497] niks3-server[1023]: 2026/09/21 12:33:28 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4745lb # [ 51.456199] niks3-server[1023]: 2026/09/21 12:33:28 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4746lb # [ 51.459870] niks3-server[1023]: 2026/09/21 12:33:28 INFO Completed upload id=104747worker2 # [ 51.491237] nix-grpc-daemon[924]: time=2026-09-21T12:33:28.475Z level=INFO msg="Upload complete. (312ms)"4748lb # [ 51.461694] niks3-server[1023]: 2026/09/21 12:33:28 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo4749worker2 # [ 51.494727] nix-grpc-daemon[786]: ts=2026-09-21T12:33:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14750worker2 # [ 51.498978] nix-grpc-daemon[786]: ts=2026-09-21T12:33:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14751worker2 # [ 51.504421] nix-daemon[869]: reaped child process 951, status = succeeded4752client # [ 51.506659] systemd[1]: dedup1.service: Deactivated successfully.4753client # [ 51.512589] systemd[1]: dedup1.service: Consumed 121ms CPU time over 6.270s wall clock time, 41.4M memory peak, 7K incoming IP traffic, 8.4K outgoing IP traffic.4754client # [ 51.552228] systemd[1]: dedup2.service: Deactivated successfully.4755client # [ 51.553675] systemd[1]: dedup2.service: Consumed 139ms CPU time over 5.231s wall clock time, 35.3M memory peak, 6.4K incoming IP traffic, 5.9K outgoing IP traffic.4756client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.12 seconds)4757client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4758client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.05 seconds)4759worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4760worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)4761worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4762worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.03 seconds)4763worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4764worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4765worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4766worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4767(finished: subtest: two clients wanting the same drv build it once, in 7.18 seconds)4768subtest: build hook: nix-daemon with builders = grpc://lb4769client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log4770client # [ 52.597666] systemd[1]: Started Nix Daemon.4771client # [ 52.826037] nix-daemon[950]: accepted connection from pid 947, user root (trusted)4772worker1 # [ 53.093882] nix-grpc-daemon[788]: ts=2026-09-21T12:33:29Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:360024773worker2 # [ 53.119503] nix-grpc-daemon[786]: ts=2026-09-21T12:33:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419864774worker2 # [ 53.131523] nix-grpc-daemon[786]: ts=2026-09-21T12:33:30Z level=debug event=want client=6 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=04775worker2 # [ 53.135782] nix-grpc-daemon[786]: ts=2026-09-21T12:33:30Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=14776worker1 # [ 53.160131] nix-grpc-daemon[788]: ts=2026-09-21T12:33:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024777worker1 # [ 53.173245] nix-grpc-daemon[788]: ts=2026-09-21T12:33:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:360024778worker1 # [ 53.185127] nix-grpc-daemon[788]: ts=2026-09-21T12:33:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:360024779worker1 # [ 53.196772] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4780worker1 # [ 53.428598] nix-daemon[852]: accepted connection from pid 1060, user nix-grpc-daemon (trusted)4781worker1 # [ 53.441510] nix-daemon[852]: reaped child process 1063, status = succeeded4782worker1 # [ 53.654171] nix-daemon[852]: accepted connection from pid 1065, user nix-grpc-daemon (trusted)4783worker1 # [ 53.665680] nix-daemon[852]: reaped child process 1068, status = succeeded4784lb # [ 53.673613] niks3-server[1023]: 2026/09/21 12:33:30 INFO Received uploads request method=POST path=/api/pending_closures4785worker1 # [ 53.697892] nix-grpc-daemon[864]: time=2026-09-21T12:33:30.505Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4786worker1 # [ 53.699260] nix-grpc-daemon[864]: time=2026-09-21T12:33:30.505Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"4787lb # [ 53.703221] niks3-server[1023]: 2026/09/21 12:33:30 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst4788lb # [ 53.716253] niks3-server[1023]: 2026/09/21 12:33:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4789worker1 # [ 53.738548] nix-grpc-daemon[864]: time=2026-09-21T12:33:30.546Z level=INFO msg="Uploading 1 narinfos"4790lb # [ 53.723078] niks3-server[1023]: 2026/09/21 12:33:30 INFO Signed narinfos id=11 count=14791lb # [ 53.726842] niks3-server[1023]: 2026/09/21 12:33:30 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls4792lb # [ 53.739800] niks3-server[1023]: 2026/09/21 12:33:30 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4793worker1 # [ 53.760826] nix-grpc-daemon[864]: time=2026-09-21T12:33:30.568Z level=INFO msg="Upload complete. (552ms)"4794lb # [ 53.745701] niks3-server[1023]: 2026/09/21 12:33:30 INFO Completed upload id=114795lb # [ 53.749539] niks3-server[1023]: 2026/09/21 12:33:30 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo4796worker1 # [ 53.763836] nix-grpc-daemon[788]: ts=2026-09-21T12:33:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=0 paths=1 nar_bytes_in=5124797worker1 # [ 53.770720] nix-grpc-daemon[788]: ts=2026-09-21T12:33:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024798worker1 # [ 53.779471] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4799worker1 # [ 53.786353] nix-daemon[852]: reaped child process 1058, status = succeeded4800worker1 # [ 53.791367] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4801worker1 # [ 54.817450] nix-daemon[852]: reaped child process 1072, status = succeeded4802worker1 # [ 55.020754] nix-daemon[852]: accepted connection from pid 1088, user nix-grpc-daemon (trusted)4803worker1 # [ 55.032374] nix-daemon[852]: reaped child process 1091, status = succeeded4804lb # [ 55.038267] niks3-server[1023]: 2026/09/21 12:33:31 INFO Received uploads request method=POST path=/api/pending_closures4805worker1 # [ 55.062618] nix-grpc-daemon[864]: time=2026-09-21T12:33:31.870Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4806worker1 # [ 55.064260] nix-grpc-daemon[864]: time=2026-09-21T12:33:31.872Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"4807lb # [ 55.077058] niks3-server[1023]: 2026/09/21 12:33:31 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv4808lb # [ 55.082761] niks3-server[1023]: 2026/09/21 12:33:31 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4809lb # [ 55.096596] niks3-server[1023]: 2026/09/21 12:33:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4810lb # [ 55.102059] niks3-server[1023]: 2026/09/21 12:33:31 INFO Signed narinfos id=12 count=14811worker1 # [ 55.121224] nix-grpc-daemon[864]: time=2026-09-21T12:33:31.928Z level=INFO msg="Uploading 1 narinfos"4812lb # [ 55.107073] niks3-server[1023]: 2026/09/21 12:33:31 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls4813lb # [ 55.116598] niks3-server[1023]: 2026/09/21 12:33:31 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4814lb # [ 55.119258] niks3-server[1023]: 2026/09/21 12:33:31 INFO Completed upload id=124815lb # [ 55.121046] niks3-server[1023]: 2026/09/21 12:33:31 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo4816worker1 # [ 55.136254] nix-grpc-daemon[864]: time=2026-09-21T12:33:31.944Z level=INFO msg="Upload complete. (337ms)"4817worker1 # [ 55.141471] nix-grpc-daemon[788]: ts=2026-09-21T12:33:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=6 outputs=14818worker1 # [ 55.154467] nix-daemon[852]: reaped child process 1070, status = succeeded4819worker2 # [ 55.203720] nix-grpc-daemon[786]: ts=2026-09-21T12:33:32Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:419864820worker2 # [ 55.218489] nix-grpc-daemon[786]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...4821worker2 # [ 55.254173] nix-grpc-daemon[786]: ts=2026-09-21T12:33:32Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:419864822worker2 # [ 55.260153] nix-grpc-daemon[786]: ts=2026-09-21T12:33:32Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=0 paths=1 nar_bytes_out=1284823worker1 # [ 55.602169] nix-grpc-daemon[788]: ts=2026-09-21T12:33:32Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:360024824worker2 # [ 55.621651] nix-grpc-daemon[786]: ts=2026-09-21T12:33:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419864825worker2 # [ 55.631757] nix-grpc-daemon[924]: time=2026-09-21T12:33:32.615Z level=INFO msg="All 1 paths already cached"4826worker2 # [ 55.645689] nix-grpc-daemon[786]: ts=2026-09-21T12:33:32Z level=debug event=want client=7 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=04827worker1 # [ 55.632379] nix-grpc-daemon[788]: ts=2026-09-21T12:33:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024828worker2 # [ 55.649910] nix-grpc-daemon[786]: ts=2026-09-21T12:33:32Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=14829worker1 # [ 55.644394] nix-grpc-daemon[788]: ts=2026-09-21T12:33:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:360024830worker1 # [ 55.657698] nix-grpc-daemon[864]: time=2026-09-21T12:33:32.465Z level=INFO msg="All 1 paths already cached"4831worker1 # [ 55.664573] nix-grpc-daemon[788]: ts=2026-09-21T12:33:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:360024832worker1 # [ 55.672950] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4833worker1 # [ 55.899112] nix-daemon[852]: accepted connection from pid 1095, user nix-grpc-daemon (trusted)4834worker1 # [ 55.911882] nix-daemon[852]: reaped child process 1098, status = succeeded4835worker1 # [ 56.122336] nix-daemon[852]: accepted connection from pid 1100, user nix-grpc-daemon (trusted)4836worker1 # [ 56.133751] nix-daemon[852]: reaped child process 1103, status = succeeded4837lb # [ 56.139281] niks3-server[1023]: 2026/09/21 12:33:32 INFO Received uploads request method=POST path=/api/pending_closures4838worker1 # [ 56.164376] nix-grpc-daemon[864]: time=2026-09-21T12:33:32.971Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4839worker1 # [ 56.165755] nix-grpc-daemon[864]: time=2026-09-21T12:33:32.971Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"4840lb # [ 56.164662] niks3-server[1023]: 2026/09/21 12:33:32 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst4841lb # [ 56.177060] niks3-server[1023]: 2026/09/21 12:33:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4842lb # [ 56.182698] niks3-server[1023]: 2026/09/21 12:33:32 INFO Signed narinfos id=13 count=14843worker1 # [ 56.201319] nix-grpc-daemon[864]: time=2026-09-21T12:33:33.009Z level=INFO msg="Uploading 1 narinfos"4844lb # [ 56.189766] niks3-server[1023]: 2026/09/21 12:33:32 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls4845lb # [ 56.199642] niks3-server[1023]: 2026/09/21 12:33:32 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4846lb # [ 56.206082] niks3-server[1023]: 2026/09/21 12:33:32 INFO Completed upload id=134847worker1 # [ 56.221606] nix-grpc-daemon[864]: time=2026-09-21T12:33:33.029Z level=INFO msg="Upload complete. (534ms)"4848lb # [ 56.208498] niks3-server[1023]: 2026/09/21 12:33:32 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo4849worker1 # [ 56.225813] nix-grpc-daemon[788]: ts=2026-09-21T12:33:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=0 paths=1 nar_bytes_in=6244850worker1 # [ 56.231228] nix-grpc-daemon[788]: ts=2026-09-21T12:33:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024851worker1 # [ 56.239875] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4852worker1 # [ 56.244536] nix-daemon[852]: reaped child process 1093, status = succeeded4853worker1 # [ 56.249784] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4854worker1 # [ 57.675802] nix-daemon[852]: reaped child process 1107, status = succeeded4855worker1 # [ 57.874715] nix-daemon[852]: accepted connection from pid 1123, user nix-grpc-daemon (trusted)4856worker1 # [ 57.886417] nix-daemon[852]: reaped child process 1126, status = succeeded4857lb # [ 57.891912] niks3-server[1023]: 2026/09/21 12:33:34 INFO Received uploads request method=POST path=/api/pending_closures4858worker1 # [ 57.916356] nix-grpc-daemon[864]: time=2026-09-21T12:33:34.723Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4859worker1 # [ 57.917721] nix-grpc-daemon[864]: time=2026-09-21T12:33:34.723Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"4860lb # [ 57.917346] niks3-server[1023]: 2026/09/21 12:33:34 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv4861lb # [ 57.930075] niks3-server[1023]: 2026/09/21 12:33:34 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst4862lb # [ 57.942667] niks3-server[1023]: 2026/09/21 12:33:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4863worker1 # [ 57.964534] nix-grpc-daemon[864]: time=2026-09-21T12:33:34.772Z level=INFO msg="Uploading 1 narinfos"4864lb # [ 57.948357] niks3-server[1023]: 2026/09/21 12:33:34 INFO Signed narinfos id=14 count=14865lb # [ 57.953525] niks3-server[1023]: 2026/09/21 12:33:34 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls4866lb # [ 57.963388] niks3-server[1023]: 2026/09/21 12:33:34 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4867lb # [ 57.969599] niks3-server[1023]: 2026/09/21 12:33:34 INFO Completed upload id=144868worker1 # [ 57.985320] nix-grpc-daemon[864]: time=2026-09-21T12:33:34.793Z level=INFO msg="Upload complete. (327ms)"4869lb # [ 57.971311] niks3-server[1023]: 2026/09/21 12:33:34 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo4870worker1 # [ 57.989769] nix-grpc-daemon[788]: ts=2026-09-21T12:33:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=7 outputs=14871worker1 # [ 58.000802] nix-daemon[852]: reaped child process 1105, status = succeeded4872worker2 # [ 58.052408] nix-grpc-daemon[786]: ts=2026-09-21T12:33:35Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:419864873worker2 # [ 58.059710] nix-grpc-daemon[786]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...4874worker1 # [ 58.070386] nix-grpc-daemon[788]: ts=2026-09-21T12:33:34Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:360024875worker1 # [ 58.075976] nix-grpc-daemon[788]: ts=2026-09-21T12:33:34Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=0 paths=1 nar_bytes_out=1764876worker2 # [ 58.408148] nix-grpc-daemon[786]: ts=2026-09-21T12:33:35Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:419864877worker2 # [ 58.414150] nix-grpc-daemon[786]: ts=2026-09-21T12:33:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419864878worker2 # [ 58.419682] nix-grpc-daemon[924]: time=2026-09-21T12:33:35.401Z level=INFO msg="All 2 paths already cached"4879worker2 # [ 58.428608] nix-grpc-daemon[786]: ts=2026-09-21T12:33:35Z level=debug event=want client=8 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=04880worker2 # [ 58.432847] nix-grpc-daemon[786]: ts=2026-09-21T12:33:35Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=14881worker1 # [ 58.419412] nix-grpc-daemon[788]: ts=2026-09-21T12:33:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024882worker1 # [ 58.431163] nix-grpc-daemon[788]: ts=2026-09-21T12:33:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:360024883worker1 # [ 58.443737] nix-grpc-daemon[864]: time=2026-09-21T12:33:35.251Z level=INFO msg="All 2 paths already cached"4884worker1 # [ 58.450833] nix-grpc-daemon[788]: ts=2026-09-21T12:33:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:360024885worker1 # [ 58.461163] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4886worker1 # [ 58.681945] nix-daemon[852]: accepted connection from pid 1130, user nix-grpc-daemon (trusted)4887worker1 # [ 58.693804] nix-daemon[852]: reaped child process 1133, status = succeeded4888worker1 # [ 58.903503] nix-daemon[852]: accepted connection from pid 1135, user nix-grpc-daemon (trusted)4889worker1 # [ 58.915219] nix-daemon[852]: reaped child process 1138, status = succeeded4890lb # [ 58.919260] niks3-server[1023]: 2026/09/21 12:33:35 INFO Received uploads request method=POST path=/api/pending_closures4891worker1 # [ 58.943684] nix-grpc-daemon[864]: time=2026-09-21T12:33:35.751Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4892worker1 # [ 58.945163] nix-grpc-daemon[864]: time=2026-09-21T12:33:35.751Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"4893lb # [ 58.946951] niks3-server[1023]: 2026/09/21 12:33:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4894lb # [ 58.952086] niks3-server[1023]: 2026/09/21 12:33:35 INFO Signed narinfos id=15 count=14895worker1 # [ 58.970526] nix-grpc-daemon[864]: time=2026-09-21T12:33:35.778Z level=INFO msg="Uploading 1 narinfos"4896lb # [ 58.961556] niks3-server[1023]: 2026/09/21 12:33:35 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst4897lb # [ 58.969236] niks3-server[1023]: 2026/09/21 12:33:35 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls4898lb # [ 58.978786] niks3-server[1023]: 2026/09/21 12:33:35 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4899lb # [ 58.984661] niks3-server[1023]: 2026/09/21 12:33:35 INFO Completed upload id=154900worker1 # [ 59.002514] nix-grpc-daemon[864]: time=2026-09-21T12:33:35.810Z level=INFO msg="Upload complete. (528ms)"4901lb # [ 58.987592] niks3-server[1023]: 2026/09/21 12:33:35 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo4902worker1 # [ 59.007261] nix-grpc-daemon[788]: ts=2026-09-21T12:33:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=0 paths=1 nar_bytes_in=7524903worker1 # [ 59.011906] nix-grpc-daemon[788]: ts=2026-09-21T12:33:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360024904worker1 # [ 59.021109] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4905worker1 # [ 59.025210] nix-daemon[852]: reaped child process 1128, status = succeeded4906worker1 # [ 59.030611] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4907lb # [ 59.255781] envoy[931]: conn peer=[2001:db8:1::1]:60134 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2998 tx=1451 ms=59894908lb # [ 59.260104] envoy[931]: conn peer=[2001:db8:1::1]:60150 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1576 tx=1487 ms=49214909lb # [ 59.263628] envoy[931]: conn peer=[2001:db8:1::1]:50120 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=594910lb # [ 59.267423] envoy[931]: conn peer=[2001:db8:1::1]:50118 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3425 tx=1845 ms=22054911lb # [ 59.270930] envoy[931]: conn peer=[2001:db8:1::1]:50132 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3893 tx=2063 ms=25254912lb # [ 59.274330] envoy[931]: conn peer=[2001:db8:1::1]:50146 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=594913worker1 # [ 60.444587] nix-daemon[852]: reaped child process 1142, status = succeeded4914worker1 # [ 60.636851] nix-daemon[852]: accepted connection from pid 1158, user nix-grpc-daemon (trusted)4915worker1 # [ 60.648556] nix-daemon[852]: reaped child process 1161, status = succeeded4916lb # [ 60.654572] niks3-server[1023]: 2026/09/21 12:33:37 INFO Received uploads request method=POST path=/api/pending_closures4917worker1 # [ 60.678556] nix-grpc-daemon[864]: time=2026-09-21T12:33:37.486Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4918worker1 # [ 60.680349] nix-grpc-daemon[864]: time=2026-09-21T12:33:37.488Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"4919lb # [ 60.698522] niks3-server[1023]: 2026/09/21 12:33:37 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst4920lb # [ 60.711255] niks3-server[1023]: 2026/09/21 12:33:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4921lb # [ 60.716253] niks3-server[1023]: 2026/09/21 12:33:37 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv4922lb # [ 60.722856] niks3-server[1023]: 2026/09/21 12:33:37 INFO Signed narinfos id=16 count=14923worker1 # [ 60.740986] nix-grpc-daemon[864]: time=2026-09-21T12:33:37.548Z level=INFO msg="Uploading 1 narinfos"4924lb # [ 60.728259] niks3-server[1023]: 2026/09/21 12:33:37 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls4925lb # [ 60.738082] niks3-server[1023]: 2026/09/21 12:33:37 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4926lb # [ 60.741500] niks3-server[1023]: 2026/09/21 12:33:37 INFO Completed upload id=164927lb # [ 60.743275] niks3-server[1023]: 2026/09/21 12:33:37 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo4928worker1 # [ 60.758465] nix-grpc-daemon[864]: time=2026-09-21T12:33:37.566Z level=INFO msg="Upload complete. (320ms)"4929worker1 # [ 60.763887] nix-grpc-daemon[788]: ts=2026-09-21T12:33:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=8 outputs=14930worker2 # [ 60.784846] nix-grpc-daemon[786]: ts=2026-09-21T12:33:37Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:419864931worker1 # [ 60.777128] nix-daemon[852]: reaped child process 1140, status = succeeded4932worker2 # [ 60.793279] nix-grpc-daemon[786]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...4933worker2 # [ 60.817241] nix-grpc-daemon[786]: ts=2026-09-21T12:33:37Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:419864934worker2 # [ 60.822121] nix-grpc-daemon[786]: ts=2026-09-21T12:33:37Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=0 paths=1 nar_bytes_out=2324935client # [ 60.929866] nix-daemon[950]: reaped child process 952, status = succeeded4936client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 8.59 seconds)4937client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top4938client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.04 seconds)4939client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4940client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4941client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4942client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4943(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 8.70 seconds)4944subtest: an input only one worker has still reaches the builder4945client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only4946client # this derivation will be built:4947client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv4948client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...4949client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.51 seconds)4950client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure4951client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.24 seconds)4952worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4953worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.27 seconds)4954worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only4955worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.03 seconds)4956client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&24957client # [ 62.343258] nix-daemon[950]: accepted connection from pid 1033, user root (trusted)4958client # this derivation will be built:4959client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv4960worker1 # [ 62.584591] nix-grpc-daemon[788]: ts=2026-09-21T12:33:39Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:360024961client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...4962worker2 # [ 62.603288] nix-grpc-daemon[786]: ts=2026-09-21T12:33:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419864963client # copying 1 paths...4964client # copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' to 'grpc://lb:50051'...4965worker2 # [ 62.614622] nix-grpc-daemon[786]: ts=2026-09-21T12:33:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419864966worker2 # [ 62.626131] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4967worker2 # [ 62.836126] nix-daemon[869]: accepted connection from pid 1013, user nix-grpc-daemon (trusted)4968worker2 # [ 62.848724] nix-daemon[869]: reaped child process 1016, status = succeeded4969lb # [ 62.838411] niks3-server[1023]: 2026/09/21 12:33:39 INFO Received uploads request method=POST path=/api/pending_closures4970worker2 # [ 62.878574] nix-grpc-daemon[924]: time=2026-09-21T12:33:39.862Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4971worker2 # [ 62.880434] nix-grpc-daemon[924]: time=2026-09-21T12:33:39.863Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"4972lb # [ 62.864667] niks3-server[1023]: 2026/09/21 12:33:39 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4973lb # [ 62.877565] niks3-server[1023]: 2026/09/21 12:33:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4974lb # [ 62.883327] niks3-server[1023]: 2026/09/21 12:33:39 INFO Signed narinfos id=17 count=14975worker2 # [ 62.917685] nix-grpc-daemon[924]: time=2026-09-21T12:33:39.901Z level=INFO msg="Uploading 1 narinfos"4976lb # [ 62.888159] niks3-server[1023]: 2026/09/21 12:33:39 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls4977lb # [ 62.900085] niks3-server[1023]: 2026/09/21 12:33:39 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4978worker2 # [ 62.937911] nix-grpc-daemon[924]: time=2026-09-21T12:33:39.921Z level=INFO msg="Upload complete. (301ms)"4979lb # [ 62.906674] niks3-server[1023]: 2026/09/21 12:33:39 INFO Completed upload id=174980worker2 # [ 62.940112] nix-grpc-daemon[786]: ts=2026-09-21T12:33:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=0 paths=1 nar_bytes_in=1204981lb # [ 62.909864] niks3-server[1023]: 2026/09/21 12:33:39 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo4982worker2 # [ 62.952783] nix-daemon[869]: reaped child process 1011, status = succeeded4983worker2 # [ 62.956804] nix-grpc-daemon[786]: ts=2026-09-21T12:33:39Z level=debug event=want client=9 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=04984worker2 # [ 62.958854] nix-grpc-daemon[786]: ts=2026-09-21T12:33:39Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14985worker2 # [ 63.000894] nix-grpc-daemon[786]: ts=2026-09-21T12:33:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419864986worker2 # [ 63.013493] nix-grpc-daemon[786]: ts=2026-09-21T12:33:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419864987client # copying 2 paths to 'lb:50051'...4988client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...4989worker2 # [ 63.029134] nix-grpc-daemon[786]: ts=2026-09-21T12:33:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419864990client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...4991worker2 # [ 63.043248] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4992worker2 # [ 63.252642] nix-daemon[869]: accepted connection from pid 1022, user nix-grpc-daemon (trusted)4993worker2 # [ 63.265650] nix-daemon[869]: reaped child process 1025, status = succeeded4994worker2 # [ 63.459303] nix-daemon[869]: accepted connection from pid 1027, user nix-grpc-daemon (trusted)4995worker2 # [ 63.470908] nix-daemon[869]: reaped child process 1030, status = succeeded4996lb # [ 63.458753] niks3-server[1023]: 2026/09/21 12:33:40 INFO Received uploads request method=POST path=/api/pending_closures4997lb # [ 63.467823] niks3-server[1023]: 2026/09/21 12:33:40 INFO Received uploads request method=POST path=/api/pending_closures4998worker2 # [ 63.506796] nix-grpc-daemon[924]: time=2026-09-21T12:33:40.490Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4999worker2 # [ 63.508953] nix-grpc-daemon[924]: time=2026-09-21T12:33:40.492Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5000worker2 # [ 63.510561] nix-grpc-daemon[924]: time=2026-09-21T12:33:40.492Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5001lb # [ 63.506653] niks3-server[1023]: 2026/09/21 12:33:40 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5002lb # [ 63.517332] niks3-server[1023]: 2026/09/21 12:33:40 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5003lb # [ 63.522918] niks3-server[1023]: 2026/09/21 12:33:40 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5004lb # [ 63.531698] niks3-server[1023]: 2026/09/21 12:33:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5005lb # [ 63.536173] niks3-server[1023]: 2026/09/21 12:33:40 INFO Signed narinfos id=18 count=15006lb # [ 63.537597] niks3-server[1023]: 2026/09/21 12:33:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5007lb # [ 63.539837] niks3-server[1023]: 2026/09/21 12:33:40 INFO Signed narinfos id=19 count=25008lb # [ 63.541857] niks3-server[1023]: 2026/09/21 12:33:40 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5009worker2 # [ 63.572136] nix-grpc-daemon[924]: time=2026-09-21T12:33:40.555Z level=INFO msg="Uploading 3 narinfos"5010lb # [ 63.551539] niks3-server[1023]: 2026/09/21 12:33:40 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5011lb # [ 63.558030] niks3-server[1023]: 2026/09/21 12:33:40 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5012lb # [ 63.560974] niks3-server[1023]: 2026/09/21 12:33:40 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5013lb # [ 63.563690] niks3-server[1023]: 2026/09/21 12:33:40 INFO Completed upload id=195014lb # [ 63.565824] niks3-server[1023]: 2026/09/21 12:33:40 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5015lb # [ 63.567822] niks3-server[1023]: 2026/09/21 12:33:40 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5016lb # [ 63.570086] niks3-server[1023]: 2026/09/21 12:33:40 INFO Completed upload id=185017worker2 # [ 63.602638] nix-grpc-daemon[924]: time=2026-09-21T12:33:40.586Z level=INFO msg="Upload complete. (544ms)"5018worker2 # [ 63.607622] nix-grpc-daemon[786]: ts=2026-09-21T12:33:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=0 paths=2 nar_bytes_in=9765019worker2 # [ 63.614350] nix-grpc-daemon[786]: ts=2026-09-21T12:33:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419865020worker2 # [ 63.628691] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5021worker2 # [ 63.636228] nix-daemon[869]: reaped child process 1020, status = succeeded5022client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5023worker2 # [ 63.642689] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5024client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5025client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5026worker2 # [ 64.656287] nix-daemon[869]: reaped child process 1034, status = succeeded5027worker2 # [ 64.839422] nix-daemon[869]: accepted connection from pid 1050, user nix-grpc-daemon (trusted)5028worker2 # [ 64.851535] nix-daemon[869]: reaped child process 1053, status = succeeded5029lb # [ 64.840767] niks3-server[1023]: 2026/09/21 12:33:41 INFO Received uploads request method=POST path=/api/pending_closures5030worker2 # [ 64.880434] nix-grpc-daemon[924]: time=2026-09-21T12:33:41.864Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5031lb # [ 64.870261] niks3-server[1023]: 2026/09/21 12:33:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5032lb # [ 64.876503] niks3-server[1023]: 2026/09/21 12:33:41 INFO Signed narinfos id=20 count=15033worker2 # [ 64.910692] nix-grpc-daemon[924]: time=2026-09-21T12:33:41.894Z level=INFO msg="Uploading 1 narinfos"5034lb # [ 64.883737] niks3-server[1023]: 2026/09/21 12:33:41 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5035lb # [ 64.894056] niks3-server[1023]: 2026/09/21 12:33:41 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5036lb # [ 64.902293] niks3-server[1023]: 2026/09/21 12:33:41 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5037lb # [ 64.907569] niks3-server[1023]: 2026/09/21 12:33:41 INFO Completed upload id=205038worker2 # [ 64.941292] nix-grpc-daemon[924]: time=2026-09-21T12:33:41.925Z level=INFO msg="Upload complete. (291ms)"5039lb # [ 64.910686] niks3-server[1023]: 2026/09/21 12:33:41 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5040worker2 # [ 64.945706] nix-grpc-daemon[786]: ts=2026-09-21T12:33:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=9 outputs=15041client # copying 1 paths...5042worker2 # [ 64.959336] nix-daemon[869]: reaped child process 1032, status = succeeded5043worker1 # [ 64.940182] nix-grpc-daemon[788]: ts=2026-09-21T12:33:41Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:360025044worker1 # [ 64.954424] nix-grpc-daemon[788]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5045client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5046worker2 # [ 65.004197] nix-grpc-daemon[786]: ts=2026-09-21T12:33:41Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:419865047worker2 # [ 65.009208] nix-grpc-daemon[786]: ts=2026-09-21T12:33:41Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=0 paths=1 nar_bytes_out=1205048client # [ 65.113655] nix-daemon[950]: reaped child process 1036, status = succeeded5049client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.03 seconds)5050client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25051client # [ 65.342378] nix-daemon[950]: accepted connection from pid 1055, user root (trusted)5052client # this derivation will be built:5053client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5054worker1 # [ 65.589938] nix-grpc-daemon[788]: ts=2026-09-21T12:33:42Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:360025055client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5056worker2 # [ 65.609523] nix-grpc-daemon[786]: ts=2026-09-21T12:33:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419865057worker2 # [ 65.619389] nix-grpc-daemon[924]: time=2026-09-21T12:33:42.603Z level=INFO msg="All 1 paths already cached"5058worker2 # [ 65.634189] nix-grpc-daemon[786]: ts=2026-09-21T12:33:42Z level=debug event=want client=10 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05059worker2 # [ 65.638391] nix-grpc-daemon[786]: ts=2026-09-21T12:33:42Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15060worker1 # [ 65.629863] nix-grpc-daemon[788]: ts=2026-09-21T12:33:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360025061worker1 # [ 65.641817] nix-grpc-daemon[788]: ts=2026-09-21T12:33:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:360025062client # copying 1 paths to 'lb:50051'...5063client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5064worker1 # [ 65.659860] nix-grpc-daemon[788]: ts=2026-09-21T12:33:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:360025065worker1 # [ 65.671385] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)5066worker1 # [ 65.688571] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 66 ms (attempt 1/5)5067worker1 # [ 65.760531] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 113 ms (attempt 2/5)5068worker1 # [ 65.879693] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 130 ms (attempt 3/5)5069worker1 # [ 66.016380] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 460 ms (attempt 4/5)5070worker1 # [ 66.482608] nix-grpc-daemon[788]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5071worker1 # [ 66.486165] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5072worker1 # [ 66.491609] nix-grpc-daemon[788]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5073worker1 # [ 66.730588] nix-daemon[852]: accepted connection from pid 1176, user nix-grpc-daemon (trusted)5074worker1 # [ 66.743827] nix-daemon[852]: reaped child process 1179, status = succeeded5075worker1 # [ 66.955139] nix-daemon[852]: accepted connection from pid 1181, user nix-grpc-daemon (trusted)5076worker1 # [ 66.966571] nix-daemon[852]: reaped child process 1184, status = succeeded5077lb # [ 66.974815] niks3-server[1023]: 2026/09/21 12:33:43 INFO Received uploads request method=POST path=/api/pending_closures5078worker1 # [ 66.999623] nix-grpc-daemon[864]: time=2026-09-21T12:33:43.807Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5079worker1 # [ 67.001459] nix-grpc-daemon[864]: time=2026-09-21T12:33:43.809Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5080lb # [ 67.003624] niks3-server[1023]: 2026/09/21 12:33:43 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5081lb # [ 67.015136] niks3-server[1023]: 2026/09/21 12:33:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5082lb # [ 67.022268] niks3-server[1023]: 2026/09/21 12:33:43 INFO Signed narinfos id=21 count=15083worker1 # [ 67.041422] nix-grpc-daemon[864]: time=2026-09-21T12:33:43.849Z level=INFO msg="Uploading 1 narinfos"5084lb # [ 67.030326] niks3-server[1023]: 2026/09/21 12:33:43 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5085lb # [ 67.039044] niks3-server[1023]: 2026/09/21 12:33:43 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5086lb # [ 67.045370] niks3-server[1023]: 2026/09/21 12:33:43 INFO Completed upload id=215087worker1 # [ 67.060897] nix-grpc-daemon[864]: time=2026-09-21T12:33:43.868Z level=INFO msg="Upload complete. (548ms)"5088lb # [ 67.049420] niks3-server[1023]: 2026/09/21 12:33:43 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5089worker1 # [ 67.065461] nix-grpc-daemon[788]: ts=2026-09-21T12:33:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=1 paths=1 nar_bytes_in=5605090worker1 # [ 67.071253] nix-grpc-daemon[788]: ts=2026-09-21T12:33:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:360025091worker1 # [ 67.079868] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)5092worker1 # [ 67.088118] nix-daemon[852]: reaped child process 1169, status = succeeded5093worker1 # [ 67.091985] nix-daemon[852]: accepted connection from pid 788, user nix-grpc-daemon (trusted)5094client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5095client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5096client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5097worker1 # [ 68.316412] nix-daemon[852]: reaped child process 1188, status = succeeded5098worker1 # [ 68.522434] nix-daemon[852]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5099worker1 # [ 68.533996] nix-daemon[852]: reaped child process 1207, status = succeeded5100lb # [ 68.541416] niks3-server[1023]: 2026/09/21 12:33:45 INFO Received uploads request method=POST path=/api/pending_closures5101worker1 # [ 68.565180] nix-grpc-daemon[864]: time=2026-09-21T12:33:45.372Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5102lb # [ 68.573605] niks3-server[1023]: 2026/09/21 12:33:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5103lb # [ 68.579256] niks3-server[1023]: 2026/09/21 12:33:45 INFO Signed narinfos id=22 count=15104worker1 # [ 68.597941] nix-grpc-daemon[864]: time=2026-09-21T12:33:45.405Z level=INFO msg="Uploading 1 narinfos"5105lb # [ 68.587772] niks3-server[1023]: 2026/09/21 12:33:45 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5106lb # [ 68.592476] niks3-server[1023]: 2026/09/21 12:33:45 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5107lb # [ 68.604525] niks3-server[1023]: 2026/09/21 12:33:45 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5108lb # [ 68.611798] niks3-server[1023]: 2026/09/21 12:33:45 INFO Completed upload id=225109worker1 # [ 68.627407] nix-grpc-daemon[864]: time=2026-09-21T12:33:45.435Z level=INFO msg="Upload complete. (328ms)"5110lb # [ 68.614252] niks3-server[1023]: 2026/09/21 12:33:45 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5111worker1 # [ 68.631517] nix-grpc-daemon[788]: ts=2026-09-21T12:33:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=10 outputs=15112client # copying 1 paths...5113worker1 # [ 68.642829] nix-grpc-daemon[788]: ts=2026-09-21T12:33:45Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:360025114client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5115worker1 # [ 68.647338] nix-daemon[852]: reaped child process 1186, status = succeeded5116worker2 # [ 68.678205] nix-grpc-daemon[786]: ts=2026-09-21T12:33:45Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:419865117worker2 # [ 68.689650] nix-grpc-daemon[786]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5118worker2 # [ 68.703823] nix-grpc-daemon[786]: ts=2026-09-21T12:33:45Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=0 paths=1 nar_bytes_out=1205119client # [ 68.775687] nix-daemon[950]: reaped child process 1058, status = succeeded5120client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.66 seconds)5121client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25122client # [ 69.005768] nix-daemon[950]: accepted connection from pid 1077, user root (trusted)5123client # this derivation will be built:5124client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5125worker2 # [ 69.276239] nix-grpc-daemon[786]: ts=2026-09-21T12:33:46Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:419865126client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5127worker2 # [ 69.283444] nix-grpc-daemon[786]: ts=2026-09-21T12:33:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419865128lb # [ 69.255223] envoy[931]: conn peer=[2001:db8:1::1]:50150 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4341 tx=2300 ms=24665129lb # [ 69.259693] envoy[931]: conn peer=[2001:db8:1::1]:50160 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=555130lb # [ 69.263226] envoy[931]: conn peer=[2001:db8:1::1]:36684 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=595131lb # [ 69.267812] envoy[931]: conn peer=[2001:db8:1::1]:50162 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4253 tx=1956 ms=24635132lb # [ 69.271329] envoy[931]: conn peer=[2001:db8:1::1]:36694 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=329 ms=805133lb # [ 69.274724] envoy[931]: conn peer=[2001:db8:1::1]:36690 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3754 tx=1900 ms=31515134worker2 # [ 69.312601] nix-grpc-daemon[924]: time=2026-09-21T12:33:46.295Z level=INFO msg="All 1 paths already cached"5135worker2 # [ 69.327196] nix-grpc-daemon[786]: ts=2026-09-21T12:33:46Z level=debug event=want client=11 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05136worker2 # [ 69.332735] nix-grpc-daemon[786]: ts=2026-09-21T12:33:46Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15137worker2 # [ 69.338548] nix-grpc-daemon[786]: ts=2026-09-21T12:33:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419865138worker2 # [ 69.350511] nix-grpc-daemon[786]: ts=2026-09-21T12:33:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419865139worker2 # [ 69.362578] nix-grpc-daemon[924]: time=2026-09-21T12:33:46.346Z level=INFO msg="All 1 paths already cached"5140client # copying 1 paths to 'lb:50051'...5141client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5142worker2 # [ 69.370715] nix-grpc-daemon[786]: ts=2026-09-21T12:33:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419865143worker2 # [ 69.379858] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5144worker2 # [ 69.589326] nix-daemon[869]: accepted connection from pid 1062, user nix-grpc-daemon (trusted)5145worker2 # [ 69.601795] nix-daemon[869]: reaped child process 1065, status = succeeded5146worker2 # [ 69.797925] nix-daemon[869]: accepted connection from pid 1067, user nix-grpc-daemon (trusted)5147worker2 # [ 69.809883] nix-daemon[869]: reaped child process 1070, status = succeeded5148lb # [ 69.799353] niks3-server[1023]: 2026/09/21 12:33:46 INFO Received uploads request method=POST path=/api/pending_closures5149worker2 # [ 69.838482] nix-grpc-daemon[924]: time=2026-09-21T12:33:46.822Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5150worker2 # [ 69.839821] nix-grpc-daemon[924]: time=2026-09-21T12:33:46.822Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5151lb # [ 69.823365] niks3-server[1023]: 2026/09/21 12:33:46 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5152lb # [ 69.834626] niks3-server[1023]: 2026/09/21 12:33:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5153worker2 # [ 69.871112] nix-grpc-daemon[924]: time=2026-09-21T12:33:46.854Z level=INFO msg="Uploading 1 narinfos"5154lb # [ 69.840292] niks3-server[1023]: 2026/09/21 12:33:46 INFO Signed narinfos id=23 count=15155lb # [ 69.845607] niks3-server[1023]: 2026/09/21 12:33:46 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5156lb # [ 69.857880] niks3-server[1023]: 2026/09/21 12:33:46 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5157worker2 # [ 69.894660] nix-grpc-daemon[924]: time=2026-09-21T12:33:46.878Z level=INFO msg="Upload complete. (501ms)"5158lb # [ 69.863599] niks3-server[1023]: 2026/09/21 12:33:46 INFO Completed upload id=235159lb # [ 69.866819] niks3-server[1023]: 2026/09/21 12:33:46 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5160worker2 # [ 69.897904] nix-grpc-daemon[786]: ts=2026-09-21T12:33:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=0 paths=1 nar_bytes_in=5605161worker2 # [ 69.903469] nix-grpc-daemon[786]: ts=2026-09-21T12:33:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419865162worker2 # [ 69.911270] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5163worker2 # [ 69.915808] nix-daemon[869]: reaped child process 1060, status = succeeded5164client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5165worker2 # [ 69.921943] nix-daemon[869]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5166client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5167client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5168worker2 # [ 71.145638] nix-daemon[869]: reaped child process 1074, status = succeeded5169worker2 # [ 71.347635] nix-daemon[869]: accepted connection from pid 1090, user nix-grpc-daemon (trusted)5170worker2 # [ 71.359639] nix-daemon[869]: reaped child process 1093, status = succeeded5171lb # [ 71.347893] niks3-server[1023]: 2026/09/21 12:33:48 INFO Received uploads request method=POST path=/api/pending_closures5172worker2 # [ 71.390586] nix-grpc-daemon[924]: time=2026-09-21T12:33:48.374Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5173lb # [ 71.384563] niks3-server[1023]: 2026/09/21 12:33:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5174lb # [ 71.389785] niks3-server[1023]: 2026/09/21 12:33:48 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5175lb # [ 71.394433] niks3-server[1023]: 2026/09/21 12:33:48 INFO Signed narinfos id=24 count=15176worker2 # [ 71.429050] nix-grpc-daemon[924]: time=2026-09-21T12:33:48.412Z level=INFO msg="Uploading 1 narinfos"5177lb # [ 71.400746] niks3-server[1023]: 2026/09/21 12:33:48 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5178lb # [ 71.411780] niks3-server[1023]: 2026/09/21 12:33:48 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5179lb # [ 71.418214] niks3-server[1023]: 2026/09/21 12:33:48 INFO Completed upload id=245180worker2 # [ 71.451840] nix-grpc-daemon[924]: time=2026-09-21T12:33:48.435Z level=INFO msg="Upload complete. (321ms)"5181lb # [ 71.421409] niks3-server[1023]: 2026/09/21 12:33:48 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5182client # copying 1 paths...5183worker2 # [ 71.458627] nix-grpc-daemon[786]: ts=2026-09-21T12:33:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41986 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=11 outputs=15184worker1 # [ 71.449320] nix-grpc-daemon[788]: ts=2026-09-21T12:33:48Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:360025185worker2 # [ 71.468788] nix-daemon[869]: reaped child process 1072, status = succeeded5186worker1 # [ 71.459350] nix-grpc-daemon[788]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5187client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5188worker1 # [ 71.490959] nix-grpc-daemon[788]: ts=2026-09-21T12:33:48Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:360025189worker1 # [ 71.495744] nix-grpc-daemon[788]: ts=2026-09-21T12:33:48Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36002 duration_s=0 paths=1 nar_bytes_out=1205190client # [ 71.613787] nix-daemon[950]: reaped child process 1080, status = succeeded5191client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.84 seconds)5192(finished: subtest: an input only one worker has still reaches the builder, in 10.57 seconds)5193subtest: upload whose reference the worker lacks is completed from the cache5194client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5195client # this derivation will be built:5196client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5197client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5198client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.49 seconds)5199client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5200client # this derivation will be built:5201client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5202client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5203client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.49 seconds)5204client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5205client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.22 seconds)5206worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5207worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.26 seconds)5208worker2 # [ 73.187263] systemd[1]: Stopping Nix worker-protocol over gRPC...5209worker2 # [ 73.738718] nix-grpc-daemon[786]: ts=2026-09-21T12:33:50Z level=info event=signal_exit5210worker2 # [ 73.991990] nix-grpc-daemon[786]: ts=2026-09-21T12:33:50Z level=info event=scheduler_yield5211worker2 # [ 73.996328] nix-grpc-daemon[786]: ts=2026-09-21T12:33:50Z level=info event=scheduler_restarting addr=lb:500515212lb # [ 73.966237] niks3-server[1023]: 2026/09/21 12:33:50 INFO lead: released remote=192.168.1.4:570065213worker2 # [ 74.003324] nix-grpc-daemon[786]: ts=2026-09-21T12:33:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:419705214worker1 # [ 73.990563] nix-grpc-daemon[788]: ts=2026-09-21T12:33:50Z level=info event=scheduler_restarting addr=lb:500515215worker2 # [ 74.008374] nix-grpc-daemon[786]: ts=2026-09-21T12:33:50Z level=info event=worker_gone addr=192.168.1.4:500515216worker2 # [ 74.011662] nix-grpc-daemon[786]: ts=2026-09-21T12:33:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:419705217worker2 # [ 74.014874] nix-grpc-daemon[786]: ts=2026-09-21T12:33:50Z level=info event=worker_gone addr=192.168.1.3:500515218worker2 # [ 74.197416] nix-grpc-daemon[786]: ts=2026-09-21T12:33:51Z level=info event=scheduler_connected addr=lb:500515219worker2 # [ 74.201363] nix-grpc-daemon[786]: ts=2026-09-21T12:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5220worker1 # [ 74.190330] nix-grpc-daemon[788]: ts=2026-09-21T12:33:50Z level=info event=scheduler_connected addr=lb:500515221worker1 # [ 74.194222] nix-grpc-daemon[788]: ts=2026-09-21T12:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5222worker2 # [ 74.402368] nix-grpc-daemon[786]: ts=2026-09-21T12:33:51Z level=info event=scheduler_connected addr=lb:500515223worker2 # [ 74.406399] nix-grpc-daemon[786]: ts=2026-09-21T12:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5224worker1 # [ 74.395134] nix-grpc-daemon[788]: ts=2026-09-21T12:33:51Z level=info event=scheduler_connected addr=lb:500515225worker1 # [ 74.399148] nix-grpc-daemon[788]: ts=2026-09-21T12:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5226worker2 # [ 74.607269] nix-grpc-daemon[786]: ts=2026-09-21T12:33:51Z level=info event=scheduler_connected addr=lb:500515227worker2 # [ 74.611131] nix-grpc-daemon[786]: ts=2026-09-21T12:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5228worker1 # [ 74.599876] nix-grpc-daemon[788]: ts=2026-09-21T12:33:51Z level=info event=scheduler_connected addr=lb:500515229worker1 # [ 74.603818] nix-grpc-daemon[788]: ts=2026-09-21T12:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5230worker2 # [ 74.813825] nix-grpc-daemon[786]: ts=2026-09-21T12:33:51Z level=info event=scheduler_connected addr=lb:500515231worker2 # [ 74.816859] nix-grpc-daemon[786]: ts=2026-09-21T12:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5232worker1 # [ 74.804655] nix-grpc-daemon[788]: ts=2026-09-21T12:33:51Z level=info event=scheduler_connected addr=lb:500515233worker1 # [ 74.809617] nix-grpc-daemon[788]: ts=2026-09-21T12:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5234worker2 # [ 74.996554] nix-grpc-daemon[786]: ts=2026-09-21T12:33:51Z level=info event=scheduler_restarting5235worker1 # [ 75.011554] nix-grpc-daemon[788]: ts=2026-09-21T12:33:51Z level=info event=scheduler_connected addr=lb:500515236worker2 # [ 75.029164] nix-grpc-daemon[786]: ts=2026-09-21T12:33:52Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5237worker1 # [ 75.015625] nix-grpc-daemon[788]: ts=2026-09-21T12:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5238worker2 # [ 75.040419] nix-daemon[869]: reaped child process 871, status = succeeded5239worker1 # [ 75.216680] nix-grpc-daemon[788]: ts=2026-09-21T12:33:52Z level=info event=scheduler_connected addr=lb:500515240worker1 # [ 75.220655] nix-grpc-daemon[788]: ts=2026-09-21T12:33:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5241worker1 # [ 75.421593] nix-grpc-daemon[788]: ts=2026-09-21T12:33:52Z level=info event=scheduler_connected addr=lb:500515242worker1 # [ 75.426285] nix-grpc-daemon[788]: ts=2026-09-21T12:33:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5243worker1 # [ 75.626472] nix-grpc-daemon[788]: ts=2026-09-21T12:33:52Z level=info event=scheduler_connected addr=lb:500515244worker1 # [ 75.630295] nix-grpc-daemon[788]: ts=2026-09-21T12:33:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5245worker1 # [ 75.830705] nix-grpc-daemon[788]: ts=2026-09-21T12:33:52Z level=info event=scheduler_connected addr=lb:500515246worker1 # [ 75.834533] nix-grpc-daemon[788]: ts=2026-09-21T12:33:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5247worker2 # [ 75.888321] nix-grpc-daemon[786]: E0000 00:00:1789994032.871286 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5248worker2 # [ 75.893277] nix-grpc-daemon[786]: E0000 00:00:1789994032.871420 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5249worker1 # [ 76.036213] nix-grpc-daemon[788]: ts=2026-09-21T12:33:52Z level=info event=scheduler_connected addr=lb:500515250worker1 # [ 76.039989] nix-grpc-daemon[788]: ts=2026-09-21T12:33:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5251worker1 # [ 76.241097] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=info event=scheduler_connected addr=lb:500515252worker1 # [ 76.245226] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5253worker2 # [ 76.454961] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5254worker2 # [ 76.460485] systemd[1]: Stopped Nix worker-protocol over gRPC.5255worker1 # [ 76.446119] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=info event=scheduler_connected addr=lb:500515256worker2 # [ 76.462695] systemd[1]: nix-grpc-daemon.service: Consumed 1.505s CPU time over 57.021s wall clock time, 53.2M memory peak, 57.5K incoming IP traffic, 159.1K outgoing IP traffic.5257worker1 # [ 76.450111] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5258worker2 # [ 76.473707] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5259worker2 # [ 76.477114] systemd[1]: Closed Nix worker-protocol over gRPC.5260worker2 # [ 76.479514] systemd[1]: nix-grpc-daemon.socket: Consumed 288us CPU time over 1min 6.999s wall clock time, 256K memory peak, 73.3K incoming IP traffic, 61.2K outgoing IP traffic.5261lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5262lb: must succeed: curl -sf localhost:9901/clusters5263worker1 # [ 76.483438] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=info event=scheduler_take_over5264lb # [ 76.468844] niks3-server[1023]: 2026/09/21 12:33:53 INFO lead: acquired remote=192.168.1.3:339645265lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5266lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.09 seconds)5267client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25268worker1 # [ 76.650995] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=info event=scheduler_connected addr=lb:500515269worker1 # [ 76.654946] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5270worker1 # [ 76.821261] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:360025271worker1 # [ 76.828574] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:360025272client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5273worker1 # [ 76.855469] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=info event=scheduler_connected addr=lb:500515274client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.30 seconds)5275worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5276worker1 # [ 76.867316] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:435445277worker1 # [ 76.872163] nix-grpc-daemon[788]: ts=2026-09-21T12:33:53Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05278worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.03 seconds)5279worker2 # [ 76.969957] systemd[1]: Listening on Nix worker-protocol over gRPC.5280worker1 # [ 77.024249] systemd[1]: Stopping Nix worker-protocol over gRPC...5281worker2 # [ 77.438960] systemd[1]: Starting scheduler-token.service...5282worker2 # [ 77.545101] systemd[1]: scheduler-token.service: Deactivated successfully.5283worker2 # [ 77.546104] systemd[1]: Finished scheduler-token.service.5284worker2 # [ 77.547182] systemd[1]: scheduler-token.service: Consumed 36ms CPU time over 105ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5285worker2 # [ 77.552879] systemd[1]: Starting Nix worker-protocol over gRPC...5286worker1 # [ 77.660384] nix-grpc-daemon[788]: ts=2026-09-21T12:33:54Z level=info event=signal_exit5287worker2 # [ 77.746297] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:54Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5288worker2 # [ 77.748466] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:54Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5289worker2 # [ 77.752398] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:54Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15290worker2 # [ 77.760065] nix-grpc-daemon[1117]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5291worker2 # [ 77.762098] nix-grpc-daemon[1117]: E0000 00:00:1789994034.743574 1117 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5292worker2 # [ 77.767059] nix-grpc-daemon[1117]: E0000 00:00:1789994034.748815 1117 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5293worker2 # [ 77.770954] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:54Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5294worker2 # [ 77.772565] systemd[1]: Started Nix worker-protocol over gRPC.5295worker2 # [ 77.782072] nix-grpc-daemon[1117]: E0000 00:00:1789994034.765437 1159 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5296worker2 # [ 77.794062] nix-grpc-daemon[1117]: E0000 00:00:1789994034.777335 1159 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5297worker2 # [ 77.797173] nix-grpc-daemon[1117]: E0000 00:00:1789994034.777412 1159 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5298worker2 # [ 77.807360] nix-grpc-daemon[1117]: E0000 00:00:1789994034.791253 1159 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5299worker2 # [ 77.816812] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:54Z level=info event=scheduler_connected addr=lb:500515300worker1 # [ 77.803675] nix-grpc-daemon[788]: ts=2026-09-21T12:33:54Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:435445301worker1 # [ 77.808208] nix-grpc-daemon[788]: ts=2026-09-21T12:33:54Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05302worker1 # [ 77.916834] nix-grpc-daemon[788]: ts=2026-09-21T12:33:54Z level=info event=scheduler_yield5303lb # [ 77.904318] niks3-server[1023]: 2026/09/21 12:33:54 INFO lead: released remote=192.168.1.3:339645304worker2 # [ 77.939918] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:54Z level=info event=scheduler_restarting addr=lb:500515305worker1 # [ 77.926634] nix-grpc-daemon[788]: ts=2026-09-21T12:33:54Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:435445306worker1 # [ 77.931651] nix-grpc-daemon[788]: ts=2026-09-21T12:33:54Z level=info event=worker_gone addr=192.168.1.4:500515307worker1 # [ 77.934677] nix-grpc-daemon[788]: ts=2026-09-21T12:33:54Z level=info event=scheduler_restarting addr=lb:500515308worker1 # [ 77.937568] nix-grpc-daemon[788]: ts=2026-09-21T12:33:54Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:435445309worker1 # [ 77.940761] nix-grpc-daemon[788]: ts=2026-09-21T12:33:54Z level=info event=worker_gone addr=192.168.1.3:500515310worker2 # [ 78.141424] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:55Z level=info event=scheduler_connected addr=lb:500515311worker2 # [ 78.145237] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5312worker1 # [ 78.131360] nix-grpc-daemon[788]: ts=2026-09-21T12:33:54Z level=info event=scheduler_connected addr=lb:500515313worker1 # [ 78.134991] nix-grpc-daemon[788]: ts=2026-09-21T12:33:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5314worker2 # [ 78.346484] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:55Z level=info event=scheduler_connected addr=lb:500515315worker2 # [ 78.350416] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5316worker1 # [ 78.335635] nix-grpc-daemon[788]: ts=2026-09-21T12:33:55Z level=info event=scheduler_connected addr=lb:500515317worker1 # [ 78.339460] nix-grpc-daemon[788]: ts=2026-09-21T12:33:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5318worker2 # [ 78.551766] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:55Z level=info event=scheduler_connected addr=lb:500515319worker1 # [ 78.540316] nix-grpc-daemon[788]: ts=2026-09-21T12:33:55Z level=info event=scheduler_connected addr=lb:500515320worker2 # [ 78.555645] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5321worker1 # [ 78.544140] nix-grpc-daemon[788]: ts=2026-09-21T12:33:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5322worker2 # [ 78.757665] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:55Z level=info event=scheduler_connected addr=lb:500515323worker2 # [ 78.761443] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5324worker1 # [ 78.746288] nix-grpc-daemon[788]: ts=2026-09-21T12:33:55Z level=info event=scheduler_connected addr=lb:500515325worker1 # [ 78.749416] nix-grpc-daemon[788]: ts=2026-09-21T12:33:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5326worker1 # [ 78.921267] nix-grpc-daemon[788]: ts=2026-09-21T12:33:55Z level=info event=scheduler_restarting5327worker2 # [ 78.962684] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:55Z level=info event=scheduler_connected addr=lb:500515328worker2 # [ 78.966708] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5329worker1 # [ 78.960629] nix-daemon[852]: reaped child process 854, status = succeeded5330worker1 # [ 78.963590] nix-grpc-daemon[788]: ts=2026-09-21T12:33:55Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5331worker2 # [ 79.167885] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:56Z level=info event=scheduler_connected addr=lb:500515332worker2 # [ 79.171802] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5333lb # [ 79.256293] envoy[931]: conn peer=[2001:db8:1::1]:36716 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=585334lb # [ 79.260225] envoy[931]: conn peer=[2001:db8:1::1]:36710 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3742 tx=2015 ms=22905335lb # [ 79.263695] envoy[931]: conn peer=[2001:db8:1::4]:44238 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=34024 tx=1856 ms=552195336lb # [ 79.267388] envoy[931]: conn peer=[2001:db8:1::1]:44782 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=545 ms=215337lb # [ 79.270748] envoy[931]: conn peer=192.168.1.3:39106 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=11326 tx=2427 ms=583255338worker2 # [ 79.372909] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:56Z level=info event=scheduler_connected addr=lb:500515339worker2 # [ 79.376875] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5340worker2 # [ 79.578163] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:56Z level=info event=scheduler_connected addr=lb:500515341worker2 # [ 79.582115] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5342worker2 # [ 79.786982] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:56Z level=info event=scheduler_connected addr=lb:500515343worker2 # [ 79.790176] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5344worker1 # [ 79.822659] nix-grpc-daemon[788]: E0000 00:00:1789994036.629798 833 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5345worker1 # [ 79.827538] nix-grpc-daemon[788]: E0000 00:00:1789994036.629938 833 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5346worker2 # [ 79.985982] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:56Z level=info event=scheduler_connected addr=lb:500515347worker2 # [ 79.989923] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5348worker2 # [ 80.191214] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:57Z level=info event=scheduler_connected addr=lb:500515349worker2 # [ 80.195225] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5350worker2 # [ 80.396419] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:57Z level=info event=scheduler_connected addr=lb:500515351worker2 # [ 80.400251] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5352worker1 # [ 80.393543] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5353worker1 # [ 80.397182] systemd[1]: Stopped Nix worker-protocol over gRPC.5354worker1 # [ 80.402070] systemd[1]: nix-grpc-daemon.service: Consumed 2.677s CPU time over 1min 1.132s wall clock time, 62.4M memory peak, 81.4K incoming IP traffic, 151.9K outgoing IP traffic.5355worker1 # [ 80.413739] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5356worker1 # [ 80.417181] systemd[1]: Closed Nix worker-protocol over gRPC.5357worker1 # [ 80.419632] systemd[1]: nix-grpc-daemon.socket: Consumed 296us CPU time over 1min 10.672s wall clock time, 256K memory peak, 84.9K incoming IP traffic, 70.3K outgoing IP traffic.5358lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5359lb: must succeed: curl -sf localhost:9901/clusters5360lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5361lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.09 seconds)5362client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25363worker2 # [ 80.601392] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:57Z level=info event=scheduler_connected addr=lb:500515364worker2 # [ 80.605280] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5365worker2 # [ 80.794737] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:368865366worker2 # [ 80.807319] nix-daemon[869]: accepted connection from pid 1117, user nix-grpc-daemon (trusted)5367worker2 # [ 80.810086] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:57Z level=info event=scheduler_connected addr=lb:500515368worker2 # [ 80.813493] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5369worker2 # [ 80.831543] nix-grpc-daemon[1117]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 22 ms (attempt 1/5)5370worker2 # [ 80.859378] nix-grpc-daemon[1117]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 32 ms (attempt 2/5)5371worker2 # [ 80.897314] nix-grpc-daemon[1117]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 362 ms (attempt 3/5)5372worker2 # [ 81.014847] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:57Z level=info event=scheduler_connected addr=lb:500515373worker2 # [ 81.018745] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5374worker2 # [ 81.219889] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:58Z level=info event=scheduler_connected addr=lb:500515375worker2 # [ 81.223792] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5376worker2 # [ 81.265161] nix-grpc-daemon[1117]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 632 ms (attempt 4/5)5377worker2 # [ 81.424843] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:58Z level=info event=scheduler_connected addr=lb:500515378worker2 # [ 81.428768] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5379worker2 # [ 81.629932] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:58Z level=info event=scheduler_connected addr=lb:500515380worker2 # [ 81.633837] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5381worker2 # [ 81.835426] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:58Z level=info event=scheduler_connected addr=lb:500515382worker2 # [ 81.839186] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5383worker2 # [ 81.903555] nix-grpc-daemon[1117]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5384worker2 # [ 81.907502] nix-grpc-daemon[1117]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5385client # copying 2 paths...5386client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5387worker2 # [ 81.930928] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:368865388client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5389worker2 # [ 81.944991] nix-daemon[869]: accepted connection from pid 1117, user nix-grpc-daemon (trusted)5390worker2 # [ 81.969128] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:58Z level=info event=niks3_push_started pid=11775391worker2 # [ 82.039760] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=scheduler_connected addr=lb:500515392worker2 # [ 82.041744] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5393worker2 # [ 82.215382] nix-grpc-daemon[1177]: time=2026-09-21T12:33:59.198Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5394worker2 # [ 82.242162] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=scheduler_connected addr=lb:500515395worker2 # [ 82.243915] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5396worker2 # [ 82.424483] nix-daemon[869]: accepted connection from pid 1190, user nix-grpc-daemon (trusted)5397worker2 # [ 82.435474] nix-daemon[869]: reaped child process 1193, status = succeeded5398worker2 # [ 82.444466] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=scheduler_connected addr=lb:500515399worker2 # [ 82.446186] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5400lb # [ 82.429715] niks3-server[1023]: 2026/09/21 12:33:59 INFO Received uploads request method=POST path=/api/pending_closures5401lb # [ 82.438955] niks3-server[1023]: 2026/09/21 12:33:59 INFO Received uploads request method=POST path=/api/pending_closures5402worker2 # [ 82.478042] nix-grpc-daemon[1177]: time=2026-09-21T12:33:59.461Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5403worker2 # [ 82.481379] nix-grpc-daemon[1177]: time=2026-09-21T12:33:59.463Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5404worker2 # [ 82.482994] nix-grpc-daemon[1177]: time=2026-09-21T12:33:59.463Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5405lb # [ 82.484504] niks3-server[1023]: 2026/09/21 12:33:59 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5406lb # [ 82.503822] niks3-server[1023]: 2026/09/21 12:33:59 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5407lb # [ 82.509198] niks3-server[1023]: 2026/09/21 12:33:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5408lb # [ 82.512058] niks3-server[1023]: 2026/09/21 12:33:59 INFO Signed narinfos id=25 count=15409lb # [ 82.514356] niks3-server[1023]: 2026/09/21 12:33:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5410lb # [ 82.517355] niks3-server[1023]: 2026/09/21 12:33:59 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5411lb # [ 82.522588] niks3-server[1023]: 2026/09/21 12:33:59 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5412lb # [ 82.526735] niks3-server[1023]: 2026/09/21 12:33:59 INFO Signed narinfos id=26 count=25413worker2 # [ 82.559155] nix-grpc-daemon[1177]: time=2026-09-21T12:33:59.542Z level=INFO msg="Uploading 3 narinfos"5414lb # [ 82.540889] niks3-server[1023]: 2026/09/21 12:33:59 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5415lb # [ 82.545974] niks3-server[1023]: 2026/09/21 12:33:59 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5416lb # [ 82.547638] niks3-server[1023]: 2026/09/21 12:33:59 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5417lb # [ 82.558911] niks3-server[1023]: 2026/09/21 12:33:59 INFO Completed upload id=265418lb # [ 82.560362] niks3-server[1023]: 2026/09/21 12:33:59 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5419lb # [ 82.564656] niks3-server[1023]: 2026/09/21 12:33:59 INFO Completed upload id=255420worker2 # [ 82.596262] nix-grpc-daemon[1177]: time=2026-09-21T12:33:59.580Z level=INFO msg="Upload complete. (376ms)"5421lb # [ 82.567096] niks3-server[1023]: 2026/09/21 12:33:59 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5422worker2 # [ 82.601330] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36886 duration_s=0 paths=2 nar_bytes_in=3125423worker2 # [ 82.614194] nix-daemon[869]: reaped child process 1175, status = succeeded5424client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 2.11 seconds)5425worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5426worker2 # [ 82.649100] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=scheduler_connected addr=lb:500515427worker2 # [ 82.653326] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5428worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.04 seconds)5429worker1 # [ 82.736756] systemd[1]: Listening on Nix worker-protocol over gRPC.5430worker1 # [ 82.744717] systemd[1]: Starting Nix worker-protocol over gRPC...5431lb # [ 82.748785] niks3-server[1023]: 2026/09/21 12:33:59 INFO lead: acquired remote=192.168.1.4:540865432worker2 # [ 82.780640] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=scheduler_take_over5433worker2 # [ 82.854622] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=scheduler_connected addr=lb:500515434worker2 # [ 82.868157] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=oidc_jwks provider=mock keys=15435worker2 # [ 82.871132] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:482525436worker2 # [ 82.875419] nix-grpc-daemon[1117]: ts=2026-09-21T12:33:59Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05437worker1 # [ 82.943718] nix-grpc-daemon[1231]: ts=2026-09-21T12:33:59Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5438worker1 # [ 82.946138] nix-grpc-daemon[1231]: ts=2026-09-21T12:33:59Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5439worker1 # [ 82.950456] nix-grpc-daemon[1231]: ts=2026-09-21T12:33:59Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15440worker1 # [ 82.958063] nix-grpc-daemon[1231]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5441worker1 # [ 82.959365] nix-grpc-daemon[1231]: E0000 00:00:1789994039.765491 1231 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5442worker1 # [ 82.964320] nix-grpc-daemon[1231]: E0000 00:00:1789994039.770480 1231 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5443worker1 # [ 82.968123] nix-grpc-daemon[1231]: ts=2026-09-21T12:33:59Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5444worker1 # [ 82.971048] systemd[1]: Started Nix worker-protocol over gRPC.5445worker1 # [ 82.978091] nix-grpc-daemon[1231]: E0000 00:00:1789994039.784547 1270 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5446lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5447lb: must succeed: curl -sf localhost:9901/clusters5448worker1 # [ 82.989443] nix-grpc-daemon[1231]: E0000 00:00:1789994039.797393 1270 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5449worker1 # [ 82.994054] nix-grpc-daemon[1231]: E0000 00:00:1789994039.799725 1270 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5450worker1 # [ 83.003668] nix-grpc-daemon[1231]: E0000 00:00:1789994039.811672 1270 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5451worker1 # [ 83.011659] nix-grpc-daemon[1231]: ts=2026-09-21T12:33:59Z level=info event=scheduler_connected addr=lb:500515452worker2 # [ 83.028664] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:00Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:482525453worker2 # [ 83.031113] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:00Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05454lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5455lb: must succeed: curl -sf localhost:9901/clusters5456lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5457lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.15 seconds)5458(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 12.48 seconds)5459subtest: scheduler down: the other node takes the lock and keeps it5460lb: must succeed: curl -sf localhost:9901/clusters5461lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5462lb: must succeed: curl -sf localhost:9901/clusters5463lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5464lb: must succeed: curl -sf localhost:9901/clusters5465lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5466lb: must succeed: curl -sf localhost:9901/clusters5467lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5468worker2 # [ 84.423198] systemd[1]: Stopping Nix worker-protocol over gRPC...5469worker2 # [ 84.768934] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:01Z level=info event=signal_exit5470worker2 # [ 84.985438] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:01Z level=info event=scheduler_yield5471worker1 # [ 84.972956] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:01Z level=info event=scheduler_restarting addr=lb:500515472lb # [ 84.957767] niks3-server[1023]: 2026/09/21 12:34:01 INFO lead: released remote=192.168.1.4:540865473worker2 # [ 84.993835] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:01Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:482525474worker2 # [ 84.998400] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:01Z level=info event=worker_gone addr=192.168.1.3:500515475worker2 # [ 85.001379] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:01Z level=info event=scheduler_restarting addr=lb:500515476worker2 # [ 85.004383] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:01Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:482525477worker2 # [ 85.007587] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:01Z level=info event=worker_gone addr=192.168.1.4:500515478worker1 # [ 85.173461] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:01Z level=info event=scheduler_connected addr=lb:500515479worker1 # [ 85.177674] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5480worker2 # [ 85.195290] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:02Z level=info event=scheduler_connected addr=lb:500515481worker2 # [ 85.199140] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5482worker1 # [ 85.378709] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:02Z level=info event=scheduler_connected addr=lb:500515483worker1 # [ 85.382710] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5484worker2 # [ 85.400309] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:02Z level=info event=scheduler_connected addr=lb:500515485worker2 # [ 85.404195] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5486worker1 # [ 85.583493] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:02Z level=info event=scheduler_connected addr=lb:500515487worker1 # [ 85.587362] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5488worker2 # [ 85.605446] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:02Z level=info event=scheduler_connected addr=lb:500515489worker2 # [ 85.609283] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5490worker1 # [ 85.788402] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:02Z level=info event=scheduler_connected addr=lb:500515491worker1 # [ 85.792386] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5492worker2 # [ 85.811322] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:02Z level=info event=scheduler_connected addr=lb:500515493worker2 # [ 85.814512] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5494worker2 # [ 85.990210] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:02Z level=info event=scheduler_restarting5495worker1 # [ 85.993862] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:02Z level=info event=scheduler_connected addr=lb:500515496worker1 # [ 85.997608] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5497worker2 # [ 86.024703] nix-daemon[869]: reaped child process 1169, status = succeeded5498worker2 # [ 86.027173] nix-grpc-daemon[1117]: ts=2026-09-21T12:34:03Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5499worker1 # [ 86.198662] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:03Z level=info event=scheduler_connected addr=lb:500515500worker1 # [ 86.202586] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5501worker1 # [ 86.403330] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:03Z level=info event=scheduler_connected addr=lb:500515502worker1 # [ 86.407157] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5503worker1 # [ 86.607890] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:03Z level=info event=scheduler_connected addr=lb:500515504worker1 # [ 86.611872] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5505worker2 # [ 86.800348] nix-grpc-daemon[1117]: E0000 00:00:1789994043.783950 1162 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5506worker2 # [ 86.805314] nix-grpc-daemon[1117]: E0000 00:00:1789994043.784199 1162 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5507worker1 # [ 86.812670] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:03Z level=info event=scheduler_connected addr=lb:500515508worker1 # [ 86.816680] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5509worker1 # [ 87.018262] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:03Z level=info event=scheduler_connected addr=lb:500515510worker1 # [ 87.022094] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5511worker1 # [ 87.223170] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:04Z level=info event=scheduler_connected addr=lb:500515512worker1 # [ 87.227285] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5513worker1 # [ 87.428135] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:04Z level=info event=scheduler_connected addr=lb:500515514worker1 # [ 87.432139] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5515worker1 # [ 87.632794] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:04Z level=info event=scheduler_connected addr=lb:500515516worker1 # [ 87.636725] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5517worker2 # [ 87.802376] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5518worker2 # [ 87.806970] systemd[1]: Stopped Nix worker-protocol over gRPC.5519worker2 # [ 87.810567] systemd[1]: nix-grpc-daemon.service: Consumed 407ms CPU time over 10.248s wall clock time, 33M memory peak, 24K incoming IP traffic, 51.8K outgoing IP traffic.5520worker2 # [ 87.819957] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5521worker2 # [ 87.823501] systemd[1]: Closed Nix worker-protocol over gRPC.5522worker2 # [ 87.826284] systemd[1]: nix-grpc-daemon.socket: Consumed 316us CPU time over 10.857s wall clock time, 256K memory peak, 20.5K incoming IP traffic, 15.2K outgoing IP traffic.5523worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5524worker1 # [ 87.837511] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:04Z level=info event=scheduler_connected addr=lb:500515525worker1 # [ 87.842232] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5526worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.09 seconds)5527worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5528worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5529worker1 # [ 87.977404] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:04Z level=info event=scheduler_take_over5530lb # [ 87.962684] niks3-server[1023]: 2026/09/21 12:34:04 INFO lead: acquired remote=192.168.1.3:339125531worker1 # [ 88.042548] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:04Z level=info event=scheduler_connected addr=lb:500515532worker1 # [ 88.045638] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5533worker1 # [ 88.246593] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:05Z level=info event=scheduler_connected addr=lb:500515534worker1 # [ 88.250573] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5535worker1 # [ 88.451372] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:05Z level=info event=scheduler_connected addr=lb:500515536worker1 # [ 88.458174] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:05Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:599425537worker1 # [ 88.461743] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:05Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05538worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5539worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5540client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&25541lb # [ 89.257184] envoy[931]: conn peer=[2001:db8:1::1]:44784 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1005 tx=282 ms=18215542lb # [ 89.262589] envoy[931]: conn peer=[2001:db8:1::4]:50198 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=24334 tx=1134 ms=82035543worker1 # [ 89.320542] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:599505544worker1 # [ 89.331378] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5545client # don't know how to build these paths:5546client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5547worker1 # [ 89.345923] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599505548worker1 # [ 89.357630] nix-grpc-daemon[1231]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 25 ms (attempt 1/5)5549worker1 # [ 89.388246] nix-grpc-daemon[1231]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 166 ms (attempt 2/5)5550worker1 # [ 89.560095] nix-grpc-daemon[1231]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 14 ms (attempt 3/5)5551worker1 # [ 89.579965] nix-grpc-daemon[1231]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 313 ms (attempt 4/5)5552worker1 # [ 89.898943] nix-grpc-daemon[1231]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5553worker1 # [ 89.903054] nix-grpc-daemon[1231]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5554worker1 # [ 89.931617] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:06Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=05555worker1 # [ 89.936551] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:06Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=15556worker1 # [ 89.981842] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599505557worker1 # [ 89.995182] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599505558client # copying 1 paths to 'lb:50051'...5559client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...5560worker1 # [ 90.007810] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:599505561worker1 # [ 90.016397] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5562worker1 # [ 90.040401] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:06Z level=info event=niks3_push_started pid=13055563worker1 # [ 90.282868] nix-grpc-daemon[1305]: time=2026-09-21T12:34:07.090Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""5564worker1 # [ 90.495686] nix-daemon[852]: accepted connection from pid 1318, user nix-grpc-daemon (trusted)5565worker1 # [ 90.506402] nix-daemon[852]: reaped child process 1321, status = succeeded5566worker1 # [ 90.728175] nix-daemon[852]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5567worker1 # [ 90.739831] nix-daemon[852]: reaped child process 1326, status = succeeded5568lb # [ 90.748410] niks3-server[1023]: 2026/09/21 12:34:07 INFO Received uploads request method=POST path=/api/pending_closures5569worker1 # [ 90.775070] nix-grpc-daemon[1305]: time=2026-09-21T12:34:07.582Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5570worker1 # [ 90.776440] nix-grpc-daemon[1305]: time=2026-09-21T12:34:07.582Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"5571lb # [ 90.791985] niks3-server[1023]: 2026/09/21 12:34:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5572lb # [ 90.803402] niks3-server[1023]: 2026/09/21 12:34:07 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst5573worker1 # [ 90.825031] nix-grpc-daemon[1305]: time=2026-09-21T12:34:07.632Z level=INFO msg="Uploading 1 narinfos"5574lb # [ 90.811511] niks3-server[1023]: 2026/09/21 12:34:07 INFO Signed narinfos id=27 count=15575lb # [ 90.828346] niks3-server[1023]: 2026/09/21 12:34:07 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5576lb # [ 90.833151] niks3-server[1023]: 2026/09/21 12:34:07 INFO Completed upload id=275577lb # [ 90.835495] niks3-server[1023]: 2026/09/21 12:34:07 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo5578worker1 # [ 90.850300] nix-grpc-daemon[1305]: time=2026-09-21T12:34:07.658Z level=INFO msg="Upload complete. (563ms)"5579lb # [ 90.840488] niks3-server[1023]: 2026/09/21 12:34:07 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls5580worker1 # [ 90.856416] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59950 duration_s=0 paths=1 nar_bytes_in=5205581worker1 # [ 90.862960] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599505582worker1 # [ 90.874110] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5583worker1 # [ 90.880275] nix-daemon[852]: reaped child process 1303, status = succeeded5584worker1 # [ 90.886141] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5585client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...5586client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5587client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5588client # farm-a> LOG-farm-a-on-standby5589worker1 # [ 91.697531] nix-daemon[852]: reaped child process 1330, status = succeeded5590worker1 # [ 91.891445] nix-daemon[852]: accepted connection from pid 1346, user nix-grpc-daemon (trusted)5591worker1 # [ 91.903490] nix-daemon[852]: reaped child process 1349, status = succeeded5592lb # [ 91.913683] niks3-server[1023]: 2026/09/21 12:34:08 INFO Received uploads request method=POST path=/api/pending_closures5593worker1 # [ 91.939332] nix-grpc-daemon[1305]: time=2026-09-21T12:34:08.747Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5594worker1 # [ 91.940870] nix-grpc-daemon[1305]: time=2026-09-21T12:34:08.748Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"5595lb # [ 91.955549] niks3-server[1023]: 2026/09/21 12:34:08 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5596lb # [ 91.965863] niks3-server[1023]: 2026/09/21 12:34:08 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5597lb # [ 91.976322] niks3-server[1023]: 2026/09/21 12:34:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5598lb # [ 91.982346] niks3-server[1023]: 2026/09/21 12:34:08 INFO Signed narinfos id=28 count=15599worker1 # [ 91.998272] nix-grpc-daemon[1305]: time=2026-09-21T12:34:08.805Z level=INFO msg="Uploading 1 narinfos"5600lb # [ 91.986164] niks3-server[1023]: 2026/09/21 12:34:08 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls5601lb # [ 91.993857] niks3-server[1023]: 2026/09/21 12:34:08 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5602lb # [ 91.998106] niks3-server[1023]: 2026/09/21 12:34:08 INFO Completed upload id=285603lb # [ 92.000234] niks3-server[1023]: 2026/09/21 12:34:08 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo5604worker1 # [ 92.014916] nix-grpc-daemon[1305]: time=2026-09-21T12:34:08.822Z level=INFO msg="Upload complete. (322ms)"5605worker1 # [ 92.022707] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59950 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=15606worker1 # [ 92.041392] nix-daemon[852]: reaped child process 1328, status = succeeded5607worker1 # [ 92.047347] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:08Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=05608worker1 # [ 92.049399] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:08Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=15609worker1 # [ 92.093681] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599505610worker1 # [ 92.106544] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599505611worker1 # [ 92.120961] nix-grpc-daemon[1305]: time=2026-09-21T12:34:08.928Z level=INFO msg="All 1 paths already cached"5612client # copying 1 paths to 'lb:50051'...5613client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...5614worker1 # [ 92.129525] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:599505615worker1 # [ 92.137989] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5616worker1 # [ 92.357253] nix-daemon[852]: accepted connection from pid 1355, user nix-grpc-daemon (trusted)5617worker1 # [ 92.369538] nix-daemon[852]: reaped child process 1358, status = succeeded5618worker1 # [ 92.585529] nix-daemon[852]: accepted connection from pid 1360, user nix-grpc-daemon (trusted)5619worker1 # [ 92.597886] nix-daemon[852]: reaped child process 1363, status = succeeded5620lb # [ 92.603534] niks3-server[1023]: 2026/09/21 12:34:09 INFO Received uploads request method=POST path=/api/pending_closures5621worker1 # [ 92.628330] nix-grpc-daemon[1305]: time=2026-09-21T12:34:09.436Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5622worker1 # [ 92.630131] nix-grpc-daemon[1305]: time=2026-09-21T12:34:09.437Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"5623lb # [ 92.636742] niks3-server[1023]: 2026/09/21 12:34:09 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst5624lb # [ 92.647065] niks3-server[1023]: 2026/09/21 12:34:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5625worker1 # [ 92.668821] nix-grpc-daemon[1305]: time=2026-09-21T12:34:09.476Z level=INFO msg="Uploading 1 narinfos"5626lb # [ 92.653221] niks3-server[1023]: 2026/09/21 12:34:09 INFO Signed narinfos id=29 count=15627lb # [ 92.660181] niks3-server[1023]: 2026/09/21 12:34:09 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls5628lb # [ 92.670172] niks3-server[1023]: 2026/09/21 12:34:09 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5629lb # [ 92.677366] niks3-server[1023]: 2026/09/21 12:34:09 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo5630lb # [ 92.683163] niks3-server[1023]: 2026/09/21 12:34:09 INFO Completed upload id=295631worker1 # [ 92.698732] nix-grpc-daemon[1305]: time=2026-09-21T12:34:09.506Z level=INFO msg="Upload complete. (547ms)"5632worker1 # [ 92.704165] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59950 duration_s=0 paths=1 nar_bytes_in=6405633worker1 # [ 92.710270] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599505634worker1 # [ 92.719935] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5635worker1 # [ 92.732127] nix-daemon[852]: reaped child process 1353, status = succeeded5636worker1 # [ 92.735668] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5637client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...5638client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5639client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5640client # farm-b> LOG-farm-b-on-standby5641worker1 # [ 93.948440] nix-daemon[852]: reaped child process 1367, status = succeeded5642worker1 # [ 94.138494] nix-daemon[852]: accepted connection from pid 1383, user nix-grpc-daemon (trusted)5643worker1 # [ 94.150432] nix-daemon[852]: reaped child process 1386, status = succeeded5644lb # [ 94.159983] niks3-server[1023]: 2026/09/21 12:34:10 INFO Received uploads request method=POST path=/api/pending_closures5645worker1 # [ 94.182296] nix-grpc-daemon[1305]: time=2026-09-21T12:34:10.990Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5646worker1 # [ 94.183828] nix-grpc-daemon[1305]: time=2026-09-21T12:34:10.991Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"5647lb # [ 94.190675] niks3-server[1023]: 2026/09/21 12:34:10 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst5648lb # [ 94.208786] niks3-server[1023]: 2026/09/21 12:34:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5649lb # [ 94.216776] niks3-server[1023]: 2026/09/21 12:34:10 INFO Signed narinfos id=30 count=15650worker1 # [ 94.235089] nix-grpc-daemon[1305]: time=2026-09-21T12:34:11.042Z level=INFO msg="Uploading 1 narinfos"5651lb # [ 94.219735] niks3-server[1023]: 2026/09/21 12:34:10 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls5652lb # [ 94.225873] niks3-server[1023]: 2026/09/21 12:34:10 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5653lb # [ 94.236372] niks3-server[1023]: 2026/09/21 12:34:10 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5654lb # [ 94.241877] niks3-server[1023]: 2026/09/21 12:34:10 INFO Completed upload id=305655lb # [ 94.243092] niks3-server[1023]: 2026/09/21 12:34:10 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo5656worker1 # [ 94.257593] nix-grpc-daemon[1305]: time=2026-09-21T12:34:11.065Z level=INFO msg="Upload complete. (315ms)"5657worker1 # [ 94.264305] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59950 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=15658worker1 # [ 94.276527] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:11Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=05659worker1 # [ 94.279941] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:11Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=15660worker1 # [ 94.288592] nix-daemon[852]: reaped child process 1365, status = succeeded5661worker1 # [ 94.320307] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599505662worker1 # [ 94.333595] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599505663worker1 # [ 94.346754] nix-grpc-daemon[1305]: time=2026-09-21T12:34:11.154Z level=INFO msg="All 2 paths already cached"5664client # copying 1 paths to 'lb:50051'...5665client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...5666worker1 # [ 94.355755] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:599505667worker1 # [ 94.366166] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5668worker1 # [ 94.579809] nix-daemon[852]: accepted connection from pid 1392, user nix-grpc-daemon (trusted)5669worker1 # [ 94.591946] nix-daemon[852]: reaped child process 1395, status = succeeded5670worker1 # [ 94.812211] nix-daemon[852]: accepted connection from pid 1397, user nix-grpc-daemon (trusted)5671worker1 # [ 94.823851] nix-daemon[852]: reaped child process 1400, status = succeeded5672lb # [ 94.831057] niks3-server[1023]: 2026/09/21 12:34:11 INFO Received uploads request method=POST path=/api/pending_closures5673worker1 # [ 94.858174] nix-grpc-daemon[1305]: time=2026-09-21T12:34:11.666Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5674worker1 # [ 94.860099] nix-grpc-daemon[1305]: time=2026-09-21T12:34:11.667Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"5675lb # [ 94.863647] niks3-server[1023]: 2026/09/21 12:34:11 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst5676lb # [ 94.874177] niks3-server[1023]: 2026/09/21 12:34:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5677worker1 # [ 94.896449] nix-grpc-daemon[1305]: time=2026-09-21T12:34:11.704Z level=INFO msg="Uploading 1 narinfos"5678lb # [ 94.880988] niks3-server[1023]: 2026/09/21 12:34:11 INFO Signed narinfos id=31 count=15679lb # [ 94.886598] niks3-server[1023]: 2026/09/21 12:34:11 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls5680lb # [ 94.897197] niks3-server[1023]: 2026/09/21 12:34:11 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5681lb # [ 94.904222] niks3-server[1023]: 2026/09/21 12:34:11 INFO Completed upload id=315682worker1 # [ 94.922130] nix-grpc-daemon[1305]: time=2026-09-21T12:34:11.730Z level=INFO msg="Upload complete. (543ms)"5683lb # [ 94.907965] niks3-server[1023]: 2026/09/21 12:34:11 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo5684worker1 # [ 94.925863] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59950 duration_s=0 paths=1 nar_bytes_in=7685685worker1 # [ 94.931280] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599505686worker1 # [ 94.939886] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5687worker1 # [ 94.948261] nix-daemon[852]: reaped child process 1390, status = succeeded5688worker1 # [ 94.953514] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5689client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...5690client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5691client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5692client # farm-top> LOG-farm-top-on-standby5693worker1 # [ 96.579708] nix-daemon[852]: reaped child process 1404, status = succeeded5694worker1 # [ 96.781909] nix-daemon[852]: accepted connection from pid 1420, user nix-grpc-daemon (trusted)5695worker1 # [ 96.792801] nix-daemon[852]: reaped child process 1423, status = succeeded5696lb # [ 96.800339] niks3-server[1023]: 2026/09/21 12:34:13 INFO Received uploads request method=POST path=/api/pending_closures5697worker1 # [ 96.827302] nix-grpc-daemon[1305]: time=2026-09-21T12:34:13.635Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5698worker1 # [ 96.829086] nix-grpc-daemon[1305]: time=2026-09-21T12:34:13.636Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"5699lb # [ 96.843805] niks3-server[1023]: 2026/09/21 12:34:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5700lb # [ 96.849296] niks3-server[1023]: 2026/09/21 12:34:13 INFO Signed narinfos id=32 count=15701worker1 # [ 96.869449] nix-grpc-daemon[1305]: time=2026-09-21T12:34:13.677Z level=INFO msg="Uploading 1 narinfos"5702lb # [ 96.856223] niks3-server[1023]: 2026/09/21 12:34:13 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5703lb # [ 96.861422] niks3-server[1023]: 2026/09/21 12:34:13 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst5704lb # [ 96.866310] niks3-server[1023]: 2026/09/21 12:34:13 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls5705lb # [ 96.870907] niks3-server[1023]: 2026/09/21 12:34:13 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5706lb # [ 96.874818] niks3-server[1023]: 2026/09/21 12:34:13 INFO Completed upload id=325707lb # [ 96.876913] niks3-server[1023]: 2026/09/21 12:34:13 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo5708worker1 # [ 96.891616] nix-grpc-daemon[1305]: time=2026-09-21T12:34:13.699Z level=INFO msg="Upload complete. (329ms)"5709worker1 # [ 96.898292] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59950 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=15710worker1 # [ 96.908659] nix-daemon[852]: reaped child process 1402, status = succeeded5711client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 7.90 seconds)5712client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath5713client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.23 seconds)5714worker2 # [ 97.262544] systemd[1]: Listening on Nix worker-protocol over gRPC.5715worker2 # [ 97.269118] systemd[1]: Starting scheduler-token.service...5716worker2 # [ 97.353624] systemd[1]: scheduler-token.service: Deactivated successfully.5717worker2 # [ 97.354747] systemd[1]: Finished scheduler-token.service.5718worker2 # [ 97.356152] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 84ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5719worker2 # [ 97.361944] systemd[1]: Starting Nix worker-protocol over gRPC...5720worker2 # [ 97.564949] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:14Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5721worker2 # [ 97.567386] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:14Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5722worker2 # [ 97.571489] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:14Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15723worker2 # [ 97.579063] nix-grpc-daemon[1217]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5724worker2 # [ 97.580374] nix-grpc-daemon[1217]: E0000 00:00:1789994054.562920 1217 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5725worker2 # [ 97.585975] nix-grpc-daemon[1217]: E0000 00:00:1789994054.568084 1217 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5726worker2 # [ 97.589776] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:14Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5727worker2 # [ 97.593112] systemd[1]: Started Nix worker-protocol over gRPC.5728lb: must succeed: curl -sf localhost:9901/clusters5729worker2 # [ 97.600084] nix-grpc-daemon[1217]: E0000 00:00:1789994054.582984 1258 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5730worker2 # [ 97.611056] nix-grpc-daemon[1217]: E0000 00:00:1789994054.594951 1258 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5731worker2 # [ 97.615056] nix-grpc-daemon[1217]: E0000 00:00:1789994054.597289 1258 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5732worker2 # [ 97.625668] nix-grpc-daemon[1217]: E0000 00:00:1789994054.609042 1258 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5733worker2 # [ 97.634913] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:14Z level=info event=scheduler_connected addr=lb:500515734worker1 # [ 97.627185] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:14Z level=info event=oidc_jwks provider=mock keys=15735worker1 # [ 97.630890] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:14Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:599425736worker1 # [ 97.634785] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:14Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05737lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5738worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5739worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5740lb: must succeed: curl -sf localhost:9901/clusters5741lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5742lb: must succeed: curl -sf localhost:9901/clusters5743lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5744worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5745worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5746(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 13.84 seconds)5747subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5748client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs5749client # [ 97.999803] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.5750client # Running as unit: rs.service; invocation ID: cf52ea8bdc714017a75550069b5ddd645751client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.07 seconds)5752client # [ 98.238710] nix[1181]: don't know how to build these paths:5753client # [ 98.240174] nix[1181]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5754worker1 # [ 98.253877] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:15Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:599505755worker1 # [ 98.265583] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:15Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=05756worker1 # [ 98.270576] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:15Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=15757worker1 # [ 98.310075] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599505758worker1 # [ 98.323317] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:599505759client # [ 98.318272] nix[1181]: copying 1 paths to 'lb:50051'...5760worker1 # [ 98.335083] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:599505761client # [ 98.320828] nix[1181]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...5762worker1 # [ 98.345858] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5763worker1 # [ 98.579965] nix-daemon[852]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)5764worker1 # [ 98.591725] nix-daemon[852]: reaped child process 1439, status = succeeded5765worker1 # [ 98.816421] nix-daemon[852]: accepted connection from pid 1441, user nix-grpc-daemon (trusted)5766worker1 # [ 98.828144] nix-daemon[852]: reaped child process 1444, status = succeeded5767lb # [ 98.835124] niks3-server[1023]: 2026/09/21 12:34:15 INFO Received uploads request method=POST path=/api/pending_closures5768worker1 # [ 98.856626] nix-grpc-daemon[1305]: time=2026-09-21T12:34:15.664Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5769worker1 # [ 98.858117] nix-grpc-daemon[1305]: time=2026-09-21T12:34:15.664Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"5770lb # [ 98.851126] niks3-server[1023]: 2026/09/21 12:34:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5771lb # [ 98.854041] niks3-server[1023]: 2026/09/21 12:34:15 INFO Signed narinfos id=33 count=15772worker1 # [ 98.871028] nix-grpc-daemon[1305]: time=2026-09-21T12:34:15.678Z level=INFO msg="Uploading 1 narinfos"5773lb # [ 98.856587] niks3-server[1023]: 2026/09/21 12:34:15 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst5774lb # [ 98.859228] niks3-server[1023]: 2026/09/21 12:34:15 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls5775lb # [ 98.864159] niks3-server[1023]: 2026/09/21 12:34:15 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5776lb # [ 98.868278] niks3-server[1023]: 2026/09/21 12:34:15 INFO Completed upload id=335777worker1 # [ 98.883831] nix-grpc-daemon[1305]: time=2026-09-21T12:34:15.691Z level=INFO msg="Upload complete. (524ms)"5778lb # [ 98.870971] niks3-server[1023]: 2026/09/21 12:34:15 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo5779worker1 # [ 98.885682] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59950 duration_s=0 paths=1 nar_bytes_in=4165780worker1 # [ 98.888669] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:599505781worker1 # [ 98.895636] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5782worker1 # [ 98.897866] nix-daemon[852]: reaped child process 1434, status = succeeded5783worker1 # [ 98.903258] nix-daemon[852]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5784client # [ 98.887074] nix[1181]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...5785lb # [ 99.256645] envoy[931]: conn peer=[2001:db8:1::1]:41464 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9650 tx=4160 ms=75935786lb: must succeed: curl -sf localhost:9901/clusters5787lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5788lb: must succeed: curl -sf localhost:9901/clusters5789lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5790lb: must succeed: curl -sf localhost:9901/clusters5791lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5792lb: must succeed: curl -sf localhost:9901/clusters5793lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5794worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5795worker2 # [ 101.022978] systemd-journald[287]: Received client request to rotate journal, rotating.5796worker2 # [ 101.044091] systemd-journald[287]: Vacuuming done, freed 0B of archived journals from /var/log/journal/a86066e9a36340529e58e0b6afb6d341.5797worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.08 seconds)5798worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5799worker1 # [ 100.950225] systemd[1]: Stopping Nix worker-protocol over gRPC...5800worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)5801client: must succeed: sleep 5.05802worker1 # [ 100.969439] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:17Z level=info event=draining5803client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5804worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5805worker1 # [ 106.323389] nix-daemon[852]: reaped child process 1448, status = succeeded5806worker1 # [ 106.499802] nix-daemon[852]: accepted connection from pid 1489, user nix-grpc-daemon (trusted)5807worker1 # [ 106.511520] nix-daemon[852]: reaped child process 1492, status = succeeded5808lb # [ 106.516773] niks3-server[1023]: 2026/09/21 12:34:23 INFO Received uploads request method=POST path=/api/pending_closures5809worker1 # [ 106.545404] nix-grpc-daemon[1305]: time=2026-09-21T12:34:23.353Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5810worker1 # [ 106.547827] nix-grpc-daemon[1305]: time=2026-09-21T12:34:23.355Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"5811lb # [ 106.559788] niks3-server[1023]: 2026/09/21 12:34:23 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5812lb # [ 106.567351] niks3-server[1023]: 2026/09/21 12:34:23 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5813lb # [ 106.576283] niks3-server[1023]: 2026/09/21 12:34:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5814lb # [ 106.580838] niks3-server[1023]: 2026/09/21 12:34:23 INFO Signed narinfos id=34 count=15815worker1 # [ 106.599069] nix-grpc-daemon[1305]: time=2026-09-21T12:34:23.406Z level=INFO msg="Uploading 1 narinfos"5816lb # [ 106.586490] niks3-server[1023]: 2026/09/21 12:34:23 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls5817lb # [ 106.591057] niks3-server[1023]: 2026/09/21 12:34:23 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5818lb # [ 106.595354] niks3-server[1023]: 2026/09/21 12:34:23 INFO Completed upload id=345819worker1 # [ 106.610983] nix-grpc-daemon[1305]: time=2026-09-21T12:34:23.418Z level=INFO msg="Upload complete. (294ms)"5820lb # [ 106.597287] niks3-server[1023]: 2026/09/21 12:34:23 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo5821worker1 # [ 106.619296] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59950 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=15822worker1 # [ 106.629288] nix-daemon[852]: reaped child process 1446, status = succeeded5823client # [ 106.642730] systemd[1]: rs.service: Deactivated successfully.5824client # [ 106.646749] systemd[1]: rs.service: Consumed 123ms CPU time over 8.645s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.5825worker1 # [ 106.971728] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:23Z level=info event=signal_exit5826worker1 # [ 107.204209] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=scheduler_yield5827lb # [ 107.191970] niks3-server[1023]: 2026/09/21 12:34:23 INFO lead: released remote=192.168.1.3:339125828worker2 # [ 107.223900] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:24Z level=info event=scheduler_restarting addr=lb:500515829worker1 # [ 107.211653] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:599425830worker1 # [ 107.215891] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=worker_gone addr=192.168.1.4:500515831worker1 # [ 107.219885] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=scheduler_restarting addr=lb:500515832worker1 # [ 107.222855] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:599425833worker1 # [ 107.226137] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=worker_gone addr=192.168.1.3:500515834worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 1.11 seconds)5835client: waiting for success: ! systemctl is-active rs5836worker2 # [ 107.424362] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515837worker2 # [ 107.428283] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5838worker1 # [ 107.416759] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515839worker1 # [ 107.421526] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5840client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)5841client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5842client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.04 seconds)5843client: must fail: journalctl -u rs | grep -q 'reconnecting'5844client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5845lb: must succeed: curl -sf localhost:9901/clusters5846worker2 # [ 107.598609] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:24Z level=info event=scheduler_take_over5847lb # [ 107.568122] niks3-server[1023]: 2026/09/21 12:34:24 INFO lead: acquired remote=192.168.1.4:442445848worker2 # [ 107.629324] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515849lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5850worker2 # [ 107.633563] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5851worker1 # [ 107.623954] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515852worker1 # [ 107.627325] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5853worker2 # [ 107.835119] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515854worker2 # [ 107.838908] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5855worker1 # [ 107.827945] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515856worker1 # [ 107.831933] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5857worker2 # [ 108.040262] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=scheduler_connected addr=lb:500515858worker2 # [ 108.044236] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5859worker1 # [ 108.032845] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515860worker1 # [ 108.036844] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5861worker1 # [ 108.208661] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:25Z level=info event=scheduler_restarting5862worker2 # [ 108.246467] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=scheduler_connected addr=lb:500515863worker1 # [ 108.233207] nix-daemon[852]: reaped child process 1295, status = succeeded5864worker2 # [ 108.250230] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5865worker1 # [ 108.249289] nix-grpc-daemon[1231]: ts=2026-09-21T12:34:25Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5866worker1 # [ 108.431766] nix-grpc-daemon[1231]: E0000 00:00:1789994065.239589 1276 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5867worker2 # [ 108.451324] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=scheduler_connected addr=lb:500515868worker1 # [ 108.436616] nix-grpc-daemon[1231]: E0000 00:00:1789994065.239726 1276 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5869worker2 # [ 108.455290] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5870lb: must succeed: curl -sf localhost:9901/clusters5871worker2 # [ 108.656300] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=scheduler_connected addr=lb:500515872worker2 # [ 108.660243] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5873lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5874worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5875worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5876worker2 # [ 108.861165] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=scheduler_connected addr=lb:500515877worker2 # [ 108.874152] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=oidc_jwks provider=mock keys=15878worker2 # [ 108.878650] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:417825879worker2 # [ 108.882440] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:25Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05880lb # [ 109.258195] envoy[931]: conn peer=[2001:db8:1::1]:39282 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2973 tx=1427 ms=83745881lb # [ 109.261970] envoy[931]: conn peer=[2001:db8:1::3]:47854 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=5568 tx=1298 ms=252305882lb: must succeed: curl -sf localhost:9901/clusters5883worker1 # [ 109.797568] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5884worker1 # [ 109.802519] systemd[1]: Stopped Nix worker-protocol over gRPC.5885worker1 # [ 109.804542] systemd[1]: nix-grpc-daemon.service: Consumed 1.301s CPU time over 27.053s wall clock time, 35.7M memory peak, 49.1K incoming IP traffic, 96.7K outgoing IP traffic.5886worker1 # [ 109.820151] systemd[1]: Starting Nix worker-protocol over gRPC...5887lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5888worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5889worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5890worker1 # [ 110.037642] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:26Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5891worker1 # [ 110.040177] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:26Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5892worker1 # [ 110.044267] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:26Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15893worker1 # [ 110.052102] nix-grpc-daemon[1499]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5894worker1 # [ 110.054941] nix-grpc-daemon[1499]: E0000 00:00:1789994066.859709 1499 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5895worker1 # [ 110.058734] nix-grpc-daemon[1499]: E0000 00:00:1789994066.862822 1499 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5896worker1 # [ 110.062310] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:26Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5897worker1 # [ 110.064370] systemd[1]: Started Nix worker-protocol over gRPC.5898worker1 # [ 110.071069] nix-grpc-daemon[1499]: E0000 00:00:1789994066.878516 1541 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5899worker1 # [ 110.082474] nix-grpc-daemon[1499]: E0000 00:00:1789994066.890457 1541 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5900worker1 # [ 110.087053] nix-grpc-daemon[1499]: E0000 00:00:1789994066.892804 1541 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5901worker1 # [ 110.096813] nix-grpc-daemon[1499]: E0000 00:00:1789994066.904818 1541 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5902worker1 # [ 110.106397] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:26Z level=info event=scheduler_connected addr=lb:500515903worker2 # [ 110.125369] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:27Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:417825904worker2 # [ 110.129550] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:27Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05905lb: must succeed: curl -sf localhost:9901/clusters5906lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5907worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5908worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5909(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 13.15 seconds)5910subtest: scheduler restart mid-build: build finishes, no double build5911client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid5912client # [ 111.148338] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.5913client # Running as unit: mid.service; invocation ID: 81c493b4ab7149e68913dd0ba9e203995914client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)5915worker1 # [ 111.414744] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:398685916worker1 # [ 111.426247] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)5917client # [ 111.422906] nix[1217]: don't know how to build these paths:5918client # [ 111.424120] nix[1217]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv5919worker2 # [ 111.465807] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:28Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=05920worker2 # [ 111.469824] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:28Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=15921worker2 # [ 111.518083] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417965922worker2 # [ 111.528864] nix-daemon[869]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5923worker2 # [ 111.547328] nix-grpc-daemon[1217]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 63 ms (attempt 1/5)5924worker2 # [ 111.616223] nix-grpc-daemon[1217]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 76 ms (attempt 2/5)5925worker2 # [ 111.698220] nix-grpc-daemon[1217]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 84 ms (attempt 3/5)5926worker2 # [ 111.788320] nix-grpc-daemon[1217]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 486 ms (attempt 4/5)5927worker2 # [ 112.280486] nix-grpc-daemon[1217]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5928worker2 # [ 112.284540] nix-grpc-daemon[1217]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5929worker2 # [ 112.303056] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417965930client # [ 112.282344] nix[1217]: copying 1 paths to 'lb:50051'...5931client # [ 112.286154] nix[1217]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...5932worker2 # [ 112.317749] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417965933worker2 # [ 112.326352] nix-daemon[869]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5934worker2 # [ 112.348073] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:29Z level=info event=niks3_push_started pid=13365935worker2 # [ 112.579882] nix-grpc-daemon[1336]: time=2026-09-21T12:34:29.563Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5936worker2 # [ 112.780498] nix-daemon[869]: accepted connection from pid 1353, user nix-grpc-daemon (trusted)5937worker2 # [ 112.791277] nix-daemon[869]: reaped child process 1356, status = succeeded5938worker2 # [ 113.009933] nix-daemon[869]: accepted connection from pid 1358, user nix-grpc-daemon (trusted)5939worker2 # [ 113.021504] nix-daemon[869]: reaped child process 1361, status = succeeded5940lb # [ 113.012985] niks3-server[1023]: 2026/09/21 12:34:29 INFO Received uploads request method=POST path=/api/pending_closures5941worker2 # [ 113.054077] nix-grpc-daemon[1336]: time=2026-09-21T12:34:30.037Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5942worker2 # [ 113.055685] nix-grpc-daemon[1336]: time=2026-09-21T12:34:30.037Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"5943lb # [ 113.045382] niks3-server[1023]: 2026/09/21 12:34:29 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst5944lb # [ 113.055329] niks3-server[1023]: 2026/09/21 12:34:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5945lb # [ 113.059990] niks3-server[1023]: 2026/09/21 12:34:29 INFO Signed narinfos id=35 count=15946worker2 # [ 113.094156] nix-grpc-daemon[1336]: time=2026-09-21T12:34:30.077Z level=INFO msg="Uploading 1 narinfos"5947lb # [ 113.071163] niks3-server[1023]: 2026/09/21 12:34:29 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls5948lb # [ 113.081911] niks3-server[1023]: 2026/09/21 12:34:29 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5949lb # [ 113.087521] niks3-server[1023]: 2026/09/21 12:34:29 INFO Completed upload id=355950worker2 # [ 113.121312] nix-grpc-daemon[1336]: time=2026-09-21T12:34:30.105Z level=INFO msg="Upload complete. (537ms)"5951lb # [ 113.091136] niks3-server[1023]: 2026/09/21 12:34:29 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo5952worker2 # [ 113.124933] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41796 duration_s=0 paths=1 nar_bytes_in=4245953worker2 # [ 113.131422] nix-grpc-daemon[1217]: ts=2026-09-21T12:34:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417965954worker2 # [ 113.139505] nix-daemon[869]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5955worker2 # [ 113.144859] nix-daemon[869]: reaped child process 1334, status = succeeded5956worker2 # [ 113.150057] nix-daemon[869]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5957client # [ 113.118898] nix[1217]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...5958lb: must succeed: curl -sf localhost:9901/clusters5959lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5960lb: must succeed: curl -sf localhost:9901/clusters5961lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5962worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5963worker2 # [ 115.108929] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1217 (nix-grpc-daemon) on client request.5964worker2 # [ 115.111697] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5965lb # [ 115.085557] niks3-server[1023]: 2026/09/21 12:34:31 INFO lead: released remote=192.168.1.4:442445966client # [ 115.094547] nix[1217]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5967worker1 # [ 115.108623] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:31Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5968worker2 # [ 115.131309] systemd[1]: Starting scheduler-token.service...5969client # [ 115.098833] nix[1217]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5970worker2 # [ 115.143564] nix-daemon[869]: reaped child process 1363, status = succeeded5971worker2 # [ 115.145355] nix-daemon[869]: reaped child process 1328, status = succeeded5972worker2 # [ 115.152086] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5973worker2 # [ 115.153667] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5974worker2 # [ 115.155390] systemd[1]: nix-grpc-daemon.service: Consumed 464ms CPU time over 17.787s wall clock time, 33.5M memory peak, 16.8K incoming IP traffic, 36.5K outgoing IP traffic.5975worker2 # [ 115.162247] nix-daemon[1365]: unexpected Nix daemon error: error: interrupted by the user5976worker2 # [ 115.167415] nix-daemon[869]: reaped child process 1365, status = failed with exit code 15977worker2 # [ 115.227737] systemd[1]: scheduler-token.service: Deactivated successfully.5978worker2 # [ 115.230332] systemd[1]: Finished scheduler-token.service.5979worker2 # [ 115.231259] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 98ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5980worker2 # [ 115.233419] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.5981worker2 # [ 115.238440] systemd[1]: Starting scheduler-token.service...5982worker2 # [ 115.323597] systemd[1]: scheduler-token.service: Deactivated successfully.5983worker2 # [ 115.324797] systemd[1]: Finished scheduler-token.service.5984worker1 # [ 115.309298] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:32Z level=info event=scheduler_connected addr=lb:500515985worker2 # [ 115.327140] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 85ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5986worker2 # [ 115.332236] systemd[1]: Starting Nix worker-protocol over gRPC...5987worker2 # [ 115.539415] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5988worker2 # [ 115.541858] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5989worker2 # [ 115.545614] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15990worker2 # [ 115.554067] nix-grpc-daemon[1408]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5991worker2 # [ 115.555682] nix-grpc-daemon[1408]: E0000 00:00:1789994072.537212 1408 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5992worker2 # [ 115.560064] nix-grpc-daemon[1408]: E0000 00:00:1789994072.542402 1408 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5993worker2 # [ 115.562182] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5994worker2 # [ 115.567296] systemd[1]: Started Nix worker-protocol over gRPC.5995worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.51 seconds)5996lb # [ 115.541852] niks3-server[1023]: 2026/09/21 12:34:32 INFO lead: acquired remote=192.168.1.4:589205997client: must succeed: sleep 5.05998worker2 # [ 115.575781] nix-grpc-daemon[1408]: E0000 00:00:1789994072.559737 1449 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5999worker2 # [ 115.579242] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=info event=scheduler_take_over6000worker1 # [ 115.561492] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:32Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6001worker2 # [ 115.589190] nix-grpc-daemon[1408]: E0000 00:00:1789994072.573095 1449 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6002worker2 # [ 115.593127] nix-grpc-daemon[1408]: E0000 00:00:1789994072.575257 1449 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6003worker2 # [ 115.602981] nix-grpc-daemon[1408]: E0000 00:00:1789994072.586879 1449 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6004worker2 # [ 115.612222] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=info event=scheduler_connected addr=lb:500516005worker2 # [ 115.617063] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=info event=oidc_jwks provider=mock keys=16006worker2 # [ 115.618511] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:454966007worker2 # [ 115.620137] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06008worker2 # [ 115.631516] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06009worker2 # [ 115.633555] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=16010worker2 # [ 115.682182] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:455006011worker2 # [ 115.687573] nix-daemon[869]: accepted connection from pid 1408, user nix-grpc-daemon (trusted)6012worker2 # [ 115.693441] nix-daemon[869]: accepted connection from pid 1408, user nix-grpc-daemon (trusted)6013worker2 # [ 115.700115] nix-daemon[869]: accepted connection from pid 1408, user nix-grpc-daemon (trusted)6014client # [ 115.668626] nix[1217]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6015worker1 # [ 115.762568] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:32Z level=info event=scheduler_connected addr=lb:500516016worker2 # [ 115.780555] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:454966017worker2 # [ 115.784575] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:32Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06018lb # [ 119.258315] envoy[931]: conn peer=[2001:db8:1::4]:44820 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=7821 tx=619 ms=174946019client: (finished: must succeed: sleep 5.0, in 5.06 seconds)6020client: waiting for success: ! systemctl is-active mid6021worker2 # [ 121.117685] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:38Z level=info event=niks3_push_started pid=14916022worker2 # [ 121.146964] nix-daemon[869]: reaped child process 1466, status = succeeded6023worker2 # [ 121.367451] nix-grpc-daemon[1491]: time=2026-09-21T12:34:38.350Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6024worker2 # [ 121.562837] nix-daemon[869]: accepted connection from pid 1504, user nix-grpc-daemon (trusted)6025worker2 # [ 121.575194] nix-daemon[869]: reaped child process 1507, status = succeeded6026lb # [ 121.565360] niks3-server[1023]: 2026/09/21 12:34:38 INFO Received uploads request method=POST path=/api/pending_closures6027worker2 # [ 121.605743] nix-grpc-daemon[1491]: time=2026-09-21T12:34:38.589Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6028worker2 # [ 121.607268] nix-grpc-daemon[1491]: time=2026-09-21T12:34:38.591Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6029lb # [ 121.604671] niks3-server[1023]: 2026/09/21 12:34:38 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6030lb # [ 121.618709] niks3-server[1023]: 2026/09/21 12:34:38 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6031lb # [ 121.629987] niks3-server[1023]: 2026/09/21 12:34:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign6032worker2 # [ 121.665655] nix-grpc-daemon[1491]: time=2026-09-21T12:34:38.649Z level=INFO msg="Uploading 1 narinfos"6033lb # [ 121.633923] niks3-server[1023]: 2026/09/21 12:34:38 INFO Signed narinfos id=36 count=16034lb # [ 121.639051] niks3-server[1023]: 2026/09/21 12:34:38 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6035lb # [ 121.649207] niks3-server[1023]: 2026/09/21 12:34:38 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete6036lb # [ 121.652675] niks3-server[1023]: 2026/09/21 12:34:38 INFO Completed upload id=366037worker2 # [ 121.684090] nix-grpc-daemon[1491]: time=2026-09-21T12:34:38.667Z level=INFO msg="Upload complete. (312ms)"6038worker2 # [ 121.688953] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45500 duration_s=6 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16039lb # [ 121.665833] niks3-server[1023]: 2026/09/21 12:34:38 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6040worker2 # [ 121.702272] nix-daemon[869]: reaped child process 1464, status = succeeded6041client # [ 121.704660] systemd[1]: mid.service: Deactivated successfully.6042client # [ 121.708187] systemd[1]: mid.service: Consumed 147ms CPU time over 10.558s wall clock time, 41.4M memory peak, 8.5K incoming IP traffic, 10.4K outgoing IP traffic.6043client: (finished: waiting for success: ! systemctl is-active mid, in 1.13 seconds)6044client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6045client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds)6046(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.97 seconds)6047subtest: interrupting the client stops the build on the worker6048client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6049client # [ 122.096943] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6050client # Running as unit: intr.service; invocation ID: 51367e7ea8fd4eacbb790fa1572f24226051client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.04 seconds)6052client # [ 122.333360] nix[1253]: don't know how to build these paths:6053client # [ 122.334193] nix[1253]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6054worker1 # [ 122.345982] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:398686055worker2 # [ 122.377480] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:39Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06056worker2 # [ 122.381868] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:39Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.3:50051 clients=16057worker1 # [ 122.405155] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686058worker1 # [ 122.418167] nix-grpc-daemon[1499]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 51 ms (attempt 1/5)6059worker1 # [ 122.474980] nix-grpc-daemon[1499]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 149 ms (attempt 2/5)6060worker1 # [ 122.630155] nix-grpc-daemon[1499]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 44 ms (attempt 3/5)6061worker1 # [ 122.680237] nix-grpc-daemon[1499]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 0 ms (attempt 4/5)6062worker1 # [ 122.686070] nix-grpc-daemon[1499]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6063worker1 # [ 122.689073] nix-grpc-daemon[1499]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6064worker1 # [ 122.708148] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686065client # [ 122.703173] nix[1253]: copying 1 paths to 'lb:50051'...6066worker1 # [ 122.719926] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686067client # [ 122.706270] nix[1253]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6068worker1 # [ 122.733472] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6069worker1 # [ 122.754089] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:39Z level=info event=niks3_push_started pid=15916070worker1 # [ 122.999482] nix-grpc-daemon[1591]: time=2026-09-21T12:34:39.807Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6071worker1 # [ 123.211715] nix-daemon[852]: accepted connection from pid 1604, user nix-grpc-daemon (trusted)6072worker1 # [ 123.222106] nix-daemon[852]: reaped child process 1607, status = succeeded6073worker1 # [ 123.442387] nix-daemon[852]: accepted connection from pid 1609, user nix-grpc-daemon (trusted)6074worker1 # [ 123.454055] nix-daemon[852]: reaped child process 1616, status = succeeded6075lb # [ 123.462262] niks3-server[1023]: 2026/09/21 12:34:40 INFO Received uploads request method=POST path=/api/pending_closures6076worker1 # [ 123.490337] nix-grpc-daemon[1591]: time=2026-09-21T12:34:40.298Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6077worker1 # [ 123.493348] nix-grpc-daemon[1591]: time=2026-09-21T12:34:40.299Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6078lb # [ 123.504323] niks3-server[1023]: 2026/09/21 12:34:40 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6079lb # [ 123.511199] niks3-server[1023]: 2026/09/21 12:34:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6080worker1 # [ 123.533231] nix-grpc-daemon[1591]: time=2026-09-21T12:34:40.340Z level=INFO msg="Uploading 1 narinfos"6081lb # [ 123.517509] niks3-server[1023]: 2026/09/21 12:34:40 INFO Signed narinfos id=37 count=16082lb # [ 123.527961] niks3-server[1023]: 2026/09/21 12:34:40 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6083lb # [ 123.534951] niks3-server[1023]: 2026/09/21 12:34:40 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6084lb # [ 123.539833] niks3-server[1023]: 2026/09/21 12:34:40 INFO Completed upload id=376085worker1 # [ 123.555551] nix-grpc-daemon[1591]: time=2026-09-21T12:34:40.363Z level=INFO msg="Upload complete. (552ms)"6086lb # [ 123.542667] niks3-server[1023]: 2026/09/21 12:34:40 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6087worker1 # [ 123.559697] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=4246088worker1 # [ 123.565481] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686089worker1 # [ 123.575059] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6090worker1 # [ 123.579115] nix-daemon[852]: reaped child process 1589, status = succeeded6091worker1 # [ 123.584249] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6092client # [ 123.568808] nix[1253]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6093client: must succeed: systemctl kill -s INT intr6094client # [ 125.719111] systemd[1]: intr.service: Sent signal SIGINT to main process 1253 (nix) on client request.6095worker1 # [ 125.742632] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:42Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6096client # [ 125.725793] nix[1253]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6097client: (finished: must succeed: systemctl kill -s INT intr, in 0.06 seconds)6098client # [ 125.733658] nix[1253]: error: interrupted by the user6099client # [ 125.765619] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6100client # [ 125.767094] systemd[1]: intr.service: Failed with result 'exit-code'.6101client # [ 125.769099] systemd[1]: intr.service: Consumed 119ms CPU time over 3.671s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.3K outgoing IP traffic.6102worker1 # [ 125.807157] nix-daemon[852]: reaped child process 1618, status = succeeded6103worker1 # [ 125.813244] nix-daemon[1620]: unexpected Nix daemon error: error: interrupted by the user6104worker1 # [ 125.818120] nix-daemon[852]: reaped child process 1620, status = failed with exit code 16105(finished: subtest: interrupting the client stops the build on the worker, in 3.81 seconds)6106subtest: a deploy mid-build drains: the build finishes, the new generation takes over6107client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6108client # [ 125.912465] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6109client # Running as unit: sw.service; invocation ID: ea3de3763a1043b0af5833840167f1cb6110client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.04 seconds)6111client # [ 126.162982] nix[1274]: don't know how to build these paths:6112client # [ 126.163869] nix[1274]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6113worker1 # [ 126.176882] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:42Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:398686114worker2 # [ 126.204918] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:43Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06115worker2 # [ 126.209218] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:43Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.4:50051 clients=16116worker2 # [ 126.246853] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:455006117worker2 # [ 126.261240] nix-grpc-daemon[1408]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 83 ms (attempt 1/5)6118worker2 # [ 126.350157] nix-grpc-daemon[1408]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 76 ms (attempt 2/5)6119worker2 # [ 126.431997] nix-grpc-daemon[1408]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 88 ms (attempt 3/5)6120worker2 # [ 126.526102] nix-grpc-daemon[1408]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 408 ms (attempt 4/5)6121worker2 # [ 126.940439] nix-grpc-daemon[1408]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6122worker2 # [ 126.943463] nix-grpc-daemon[1408]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6123worker2 # [ 126.963326] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:455006124client # [ 126.943187] nix[1274]: copying 1 paths to 'lb:50051'...6125worker2 # [ 126.975806] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:455006126client # [ 126.946823] nix[1274]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6127worker2 # [ 126.990395] nix-daemon[869]: accepted connection from pid 1408, user nix-grpc-daemon (trusted)6128worker2 # [ 127.201182] nix-daemon[869]: accepted connection from pid 1543, user nix-grpc-daemon (trusted)6129worker2 # [ 127.216481] nix-daemon[869]: reaped child process 1549, status = succeeded6130worker2 # [ 127.401401] nix-daemon[869]: accepted connection from pid 1552, user nix-grpc-daemon (trusted)6131worker2 # [ 127.413174] nix-daemon[869]: reaped child process 1555, status = succeeded6132lb # [ 127.402210] niks3-server[1023]: 2026/09/21 12:34:44 INFO Received uploads request method=POST path=/api/pending_closures6133worker2 # [ 127.444098] nix-grpc-daemon[1491]: time=2026-09-21T12:34:44.427Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6134worker2 # [ 127.445514] nix-grpc-daemon[1491]: time=2026-09-21T12:34:44.427Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6135lb # [ 127.430646] niks3-server[1023]: 2026/09/21 12:34:44 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6136lb # [ 127.442724] niks3-server[1023]: 2026/09/21 12:34:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6137worker2 # [ 127.479033] nix-grpc-daemon[1491]: time=2026-09-21T12:34:44.462Z level=INFO msg="Uploading 1 narinfos"6138lb # [ 127.448099] niks3-server[1023]: 2026/09/21 12:34:44 INFO Signed narinfos id=38 count=16139lb # [ 127.453435] niks3-server[1023]: 2026/09/21 12:34:44 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6140lb # [ 127.462972] niks3-server[1023]: 2026/09/21 12:34:44 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6141lb # [ 127.469439] niks3-server[1023]: 2026/09/21 12:34:44 INFO Completed upload id=386142worker2 # [ 127.500624] nix-grpc-daemon[1491]: time=2026-09-21T12:34:44.484Z level=INFO msg="Upload complete. (498ms)"6143lb # [ 127.470683] niks3-server[1023]: 2026/09/21 12:34:44 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6144worker2 # [ 127.504635] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45500 duration_s=0 paths=1 nar_bytes_in=4166145worker2 # [ 127.510279] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:455006146worker2 # [ 127.519075] nix-daemon[869]: accepted connection from pid 1408, user nix-grpc-daemon (trusted)6147worker2 # [ 127.523453] nix-daemon[869]: reaped child process 1541, status = succeeded6148worker2 # [ 127.528415] nix-daemon[869]: accepted connection from pid 1408, user nix-grpc-daemon (trusted)6149client # [ 127.496827] nix[1274]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6150worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&26151worker2 # Checking switch inhibitors... done6152lb # [ 129.259437] envoy[931]: conn peer=[2001:db8:1::1]:37004 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3921 tx=1831 ms=102826153lb # [ 129.263276] envoy[931]: conn peer=[2001:db8:1::1]:49494 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3006 tx=849 ms=34126154worker2 # [ 129.566570] nixos[1587]: switching to system configuration /nix/store/fmnm271kx1lhx269nfzafhdfg9ra810l-nixos-system-worker2-test6155worker2 # activating the configuration...6156worker2 # [ 129.570704] systemd[1]: Stopped target Remote File Systems.6157worker2 # [ 129.571976] systemd[1]: Stopped target Local File Systems.6158worker2 # setting up /etc...6159worker2 # [ 130.359869] systemd[1]: Reload requested from client PID 1587 ('.switch-to-conf') (unit backdoor.service)...6160worker2 # [ 130.361437] systemd[1]: Reloading...6161worker2 # [ 131.248275] systemd[1]: Reloading finished in 883 ms.6162worker2 # restarting sysinit-reactivation.target6163worker2 # [ 131.308263] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....6164worker2 # [ 131.461371] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..6165worker2 # [ 131.463668] systemd[1]: Reached target Reactivate sysinit units.6166worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service6167worker2 # [ 131.469335] systemd[1]: Reloading D-Bus System Message Bus...6168worker2 # [ 131.470626] dbus-broker-launch[533]: Caught SIGHUP, trigger reload.6169worker2 # [ 131.475611] systemd[1]: Reloading Nix worker-protocol over gRPC...6170worker2 # [ 131.514986] dbus-broker-launch[533]: Looking up NSS user entry for 'systemd-timesync'...6171worker2 # [ 131.516639] dbus-broker-launch[533]: NSS returned no entry for 'systemd-timesync'6172worker2 # [ 131.518171] dbus-broker-launch[533]: Invalid user-name in /nix/store/s93qka0z548yjbxqnmj4c4b32apww6jv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"6173worker2 # [ 131.525538] systemd[1]: Reloaded Nix worker-protocol over gRPC.6174worker2 # [ 131.553540] systemd[1]: Reloaded D-Bus System Message Bus.6175worker2 # [ 131.556980] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os6176worker2 # [ 131.558502] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki6177worker2 # [ 131.572157] systemd[1]: Reached target Local File Systems.6178worker2 # [ 131.574485] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:48Z level=info event=draining6179worker2 # [ 131.576138] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore6180worker2 # [ 131.579353] systemd[1]: Starting Create SUID/SGID Wrappers...6181worker2 # [ 131.581139] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.6182worker2 # [ 131.582845] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc676183worker2 # [ 131.585596] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore6184worker2 # [ 131.587415] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6185worker2 # [ 131.590641] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6186worker2 # [ 131.591787] systemd[1]: Reached target Remote File Systems.6187worker2 # [ 132.127740] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.6188worker2 # [ 132.130554] systemd[1]: Finished Create SUID/SGID Wrappers.6189worker2 # [ 132.134382] systemd[1]: Starting linger-users.service...6190worker2 # [ 132.186344] systemd[1]: linger-users.service: Deactivated successfully.6191worker2 # [ 132.187484] systemd[1]: Finished linger-users.service.6192worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service6193worker2 # [ 132.444395] nixos[1587]: finished switching to system configuration /nix/store/fmnm271kx1lhx269nfzafhdfg9ra810l-nixos-system-worker2-test6194worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 3.92 seconds)6195worker2: must succeed: pgrep -f 'read -t [9]0 x'6196worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.06 seconds)6197client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26198worker1 # [ 132.774231] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:398686199client # don't know how to build these paths:6200client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6201worker1 # [ 132.782967] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686202worker2 # [ 132.821446] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:49Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06203worker2 # [ 132.823910] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:49Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.3:50051 clients=16204worker1 # [ 132.862954] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686205worker1 # [ 132.876582] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686206client # copying 1 paths to 'lb:50051'...6207worker1 # [ 132.889627] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686208client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6209worker1 # [ 132.911509] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6210worker1 # [ 133.142376] nix-daemon[852]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6211worker1 # [ 133.154705] nix-daemon[852]: reaped child process 1670, status = succeeded6212worker1 # [ 133.373212] nix-daemon[852]: accepted connection from pid 1672, user nix-grpc-daemon (trusted)6213worker1 # [ 133.385089] nix-daemon[852]: reaped child process 1675, status = succeeded6214lb # [ 133.393848] niks3-server[1023]: 2026/09/21 12:34:50 INFO Received uploads request method=POST path=/api/pending_closures6215worker1 # [ 133.417847] nix-grpc-daemon[1591]: time=2026-09-21T12:34:50.225Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6216worker1 # [ 133.419568] nix-grpc-daemon[1591]: time=2026-09-21T12:34:50.227Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6217lb # [ 133.426736] niks3-server[1023]: 2026/09/21 12:34:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6218lb # [ 133.431351] niks3-server[1023]: 2026/09/21 12:34:50 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6219worker1 # [ 133.454579] nix-grpc-daemon[1591]: time=2026-09-21T12:34:50.262Z level=INFO msg="Uploading 1 narinfos"6220lb # [ 133.439745] niks3-server[1023]: 2026/09/21 12:34:50 INFO Signed narinfos id=39 count=16221lb # [ 133.446870] niks3-server[1023]: 2026/09/21 12:34:50 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6222lb # [ 133.457566] niks3-server[1023]: 2026/09/21 12:34:50 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6223lb # [ 133.465224] niks3-server[1023]: 2026/09/21 12:34:50 INFO Completed upload id=396224worker1 # [ 133.481065] nix-grpc-daemon[1591]: time=2026-09-21T12:34:50.289Z level=INFO msg="Upload complete. (557ms)"6225lb # [ 133.468835] niks3-server[1023]: 2026/09/21 12:34:50 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6226worker1 # [ 133.484743] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=5286227worker1 # [ 133.490207] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686228worker1 # [ 133.498727] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6229worker1 # [ 133.503522] nix-daemon[852]: reaped child process 1665, status = succeeded6230worker1 # [ 133.508585] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6231client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6232client # farm-a> node-a: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6233client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6234client # farm-a> LOG-farm-a-during-drain6235worker1 # [ 135.134503] nix-daemon[852]: reaped child process 1679, status = succeeded6236worker1 # [ 135.345563] nix-daemon[852]: accepted connection from pid 1695, user nix-grpc-daemon (trusted)6237worker1 # [ 135.357318] nix-daemon[852]: reaped child process 1698, status = succeeded6238lb # [ 135.366384] niks3-server[1023]: 2026/09/21 12:34:52 INFO Received uploads request method=POST path=/api/pending_closures6239worker1 # [ 135.392322] nix-grpc-daemon[1591]: time=2026-09-21T12:34:52.198Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6240worker1 # [ 135.393784] nix-grpc-daemon[1591]: time=2026-09-21T12:34:52.198Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6241lb # [ 135.402137] niks3-server[1023]: 2026/09/21 12:34:52 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6242lb # [ 135.414885] niks3-server[1023]: 2026/09/21 12:34:52 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6243lb # [ 135.426103] niks3-server[1023]: 2026/09/21 12:34:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6244lb # [ 135.430214] niks3-server[1023]: 2026/09/21 12:34:52 INFO Signed narinfos id=40 count=16245worker1 # [ 135.448595] nix-grpc-daemon[1591]: time=2026-09-21T12:34:52.256Z level=INFO msg="Uploading 1 narinfos"6246lb # [ 135.435727] niks3-server[1023]: 2026/09/21 12:34:52 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6247lb # [ 135.445131] niks3-server[1023]: 2026/09/21 12:34:52 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6248lb # [ 135.448988] niks3-server[1023]: 2026/09/21 12:34:52 INFO Completed upload id=406249lb # [ 135.451160] niks3-server[1023]: 2026/09/21 12:34:52 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6250worker1 # [ 135.465789] nix-grpc-daemon[1591]: time=2026-09-21T12:34:52.273Z level=INFO msg="Upload complete. (348ms)"6251worker1 # [ 135.470771] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16252worker2 # [ 135.491763] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:52Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06253worker2 # [ 135.493834] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:52Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.3:50051 clients=16254worker1 # [ 135.485283] nix-daemon[852]: reaped child process 1677, status = succeeded6255worker1 # [ 135.522954] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686256worker1 # [ 135.535657] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686257worker1 # [ 135.549432] nix-grpc-daemon[1591]: time=2026-09-21T12:34:52.357Z level=INFO msg="All 1 paths already cached"6258client # copying 1 paths to 'lb:50051'...6259worker1 # [ 135.557940] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686260client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6261worker1 # [ 135.575521] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6262worker1 # [ 135.801921] nix-daemon[852]: accepted connection from pid 1702, user nix-grpc-daemon (trusted)6263worker1 # [ 135.813665] nix-daemon[852]: reaped child process 1705, status = succeeded6264worker1 # [ 136.038463] nix-daemon[852]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6265worker1 # [ 136.049882] nix-daemon[852]: reaped child process 1710, status = succeeded6266lb # [ 136.058592] niks3-server[1023]: 2026/09/21 12:34:52 INFO Received uploads request method=POST path=/api/pending_closures6267worker1 # [ 136.082256] nix-grpc-daemon[1591]: time=2026-09-21T12:34:52.890Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6268worker1 # [ 136.084256] nix-grpc-daemon[1591]: time=2026-09-21T12:34:52.891Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6269lb # [ 136.088423] niks3-server[1023]: 2026/09/21 12:34:52 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6270lb # [ 136.097234] niks3-server[1023]: 2026/09/21 12:34:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6271lb # [ 136.101865] niks3-server[1023]: 2026/09/21 12:34:52 INFO Signed narinfos id=41 count=16272worker1 # [ 136.120392] nix-grpc-daemon[1591]: time=2026-09-21T12:34:52.928Z level=INFO msg="Uploading 1 narinfos"6273lb # [ 136.111153] niks3-server[1023]: 2026/09/21 12:34:52 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6274lb # [ 136.119858] niks3-server[1023]: 2026/09/21 12:34:52 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6275worker1 # [ 136.140192] nix-grpc-daemon[1591]: time=2026-09-21T12:34:52.948Z level=INFO msg="Upload complete. (550ms)"6276lb # [ 136.124720] niks3-server[1023]: 2026/09/21 12:34:52 INFO Completed upload id=416277worker1 # [ 136.142946] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=6406278lb # [ 136.130179] niks3-server[1023]: 2026/09/21 12:34:52 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6279worker1 # [ 136.149690] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686280worker1 # [ 136.158396] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6281worker1 # [ 136.160710] nix-daemon[852]: reaped child process 1700, status = succeeded6282worker1 # [ 136.166287] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6283client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6284client # farm-b> node-a: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6285client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6286client # farm-b> LOG-farm-b-during-drain6287worker1 # [ 137.190710] nix-daemon[852]: reaped child process 1714, status = succeeded6288worker1 # [ 137.399292] nix-daemon[852]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6289worker1 # [ 137.411082] nix-daemon[852]: reaped child process 1733, status = succeeded6290lb # [ 137.419843] niks3-server[1023]: 2026/09/21 12:34:54 INFO Received uploads request method=POST path=/api/pending_closures6291worker1 # [ 137.446372] nix-grpc-daemon[1591]: time=2026-09-21T12:34:54.254Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6292worker1 # [ 137.448687] nix-grpc-daemon[1591]: time=2026-09-21T12:34:54.256Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6293lb # [ 137.459277] niks3-server[1023]: 2026/09/21 12:34:54 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6294lb # [ 137.465810] niks3-server[1023]: 2026/09/21 12:34:54 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6295lb # [ 137.478749] niks3-server[1023]: 2026/09/21 12:34:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6296lb # [ 137.483606] niks3-server[1023]: 2026/09/21 12:34:54 INFO Signed narinfos id=42 count=16297worker1 # [ 137.501574] nix-grpc-daemon[1591]: time=2026-09-21T12:34:54.309Z level=INFO msg="Uploading 1 narinfos"6298lb # [ 137.490059] niks3-server[1023]: 2026/09/21 12:34:54 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6299lb # [ 137.499304] niks3-server[1023]: 2026/09/21 12:34:54 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6300lb # [ 137.504245] niks3-server[1023]: 2026/09/21 12:34:54 INFO Completed upload id=426301worker1 # [ 137.519993] nix-grpc-daemon[1591]: time=2026-09-21T12:34:54.328Z level=INFO msg="Upload complete. (346ms)"6302lb # [ 137.506915] niks3-server[1023]: 2026/09/21 12:34:54 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6303worker1 # [ 137.525634] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16304worker2 # [ 137.545826] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:54Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06305worker2 # [ 137.547510] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:54Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.3:50051 clients=16306worker1 # [ 137.540606] nix-daemon[852]: reaped child process 1712, status = succeeded6307worker1 # [ 137.574541] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686308worker1 # [ 137.587602] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686309worker1 # [ 137.601595] nix-grpc-daemon[1591]: time=2026-09-21T12:34:54.409Z level=INFO msg="All 2 paths already cached"6310client # copying 1 paths to 'lb:50051'...6311worker1 # [ 137.610872] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686312client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6313worker1 # [ 137.636171] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6314worker1 # [ 137.860637] nix-daemon[852]: accepted connection from pid 1737, user nix-grpc-daemon (trusted)6315worker1 # [ 137.873669] nix-daemon[852]: reaped child process 1740, status = succeeded6316worker1 # [ 138.094242] nix-daemon[852]: accepted connection from pid 1742, user nix-grpc-daemon (trusted)6317worker1 # [ 138.105685] nix-daemon[852]: reaped child process 1745, status = succeeded6318lb # [ 138.113969] niks3-server[1023]: 2026/09/21 12:34:54 INFO Received uploads request method=POST path=/api/pending_closures6319worker1 # [ 138.138335] nix-grpc-daemon[1591]: time=2026-09-21T12:34:54.946Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6320worker1 # [ 138.140188] nix-grpc-daemon[1591]: time=2026-09-21T12:34:54.947Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6321lb # [ 138.146361] niks3-server[1023]: 2026/09/21 12:34:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6322worker1 # [ 138.167468] nix-grpc-daemon[1591]: time=2026-09-21T12:34:54.975Z level=INFO msg="Uploading 1 narinfos"6323lb # [ 138.153358] niks3-server[1023]: 2026/09/21 12:34:54 INFO Signed narinfos id=43 count=16324lb # [ 138.158995] niks3-server[1023]: 2026/09/21 12:34:54 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6325lb # [ 138.164916] niks3-server[1023]: 2026/09/21 12:34:54 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6326lb # [ 138.176764] niks3-server[1023]: 2026/09/21 12:34:54 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6327lb # [ 138.184540] niks3-server[1023]: 2026/09/21 12:34:54 INFO Completed upload id=436328worker1 # [ 138.199850] nix-grpc-daemon[1591]: time=2026-09-21T12:34:55.007Z level=INFO msg="Upload complete. (550ms)"6329lb # [ 138.187709] niks3-server[1023]: 2026/09/21 12:34:54 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6330worker1 # [ 138.203327] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=7686331worker1 # [ 138.209677] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686332worker1 # [ 138.219830] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6333worker1 # [ 138.222753] nix-daemon[852]: reaped child process 1735, status = succeeded6334worker1 # [ 138.228192] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6335client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6336client # farm-top> node-a: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6337client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6338client # farm-top> LOG-farm-top-during-drain6339worker1 # [ 139.241478] nix-daemon[852]: reaped child process 1749, status = succeeded6340worker1 # [ 139.440904] nix-daemon[852]: accepted connection from pid 1765, user nix-grpc-daemon (trusted)6341worker1 # [ 139.452632] nix-daemon[852]: reaped child process 1768, status = succeeded6342lb # [ 139.462276] niks3-server[1023]: 2026/09/21 12:34:56 INFO Received uploads request method=POST path=/api/pending_closures6343worker1 # [ 139.488558] nix-grpc-daemon[1591]: time=2026-09-21T12:34:56.296Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6344worker1 # [ 139.490098] nix-grpc-daemon[1591]: time=2026-09-21T12:34:56.298Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6345lb # [ 139.500878] niks3-server[1023]: 2026/09/21 12:34:56 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6346lb # [ 139.512397] niks3-server[1023]: 2026/09/21 12:34:56 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6347lb # [ 139.520821] niks3-server[1023]: 2026/09/21 12:34:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6348worker1 # [ 139.541198] nix-grpc-daemon[1591]: time=2026-09-21T12:34:56.349Z level=INFO msg="Uploading 1 narinfos"6349lb # [ 139.525907] niks3-server[1023]: 2026/09/21 12:34:56 INFO Signed narinfos id=44 count=16350lb # [ 139.531546] niks3-server[1023]: 2026/09/21 12:34:56 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6351lb # [ 139.539626] niks3-server[1023]: 2026/09/21 12:34:56 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6352lb # [ 139.545782] niks3-server[1023]: 2026/09/21 12:34:56 INFO Completed upload id=446353worker1 # [ 139.561461] nix-grpc-daemon[1591]: time=2026-09-21T12:34:56.369Z level=INFO msg="Upload complete. (326ms)"6354lb # [ 139.548931] niks3-server[1023]: 2026/09/21 12:34:56 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6355worker1 # [ 139.568309] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16356worker1 # [ 139.580589] nix-daemon[852]: reaped child process 1747, status = succeeded6357client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 7.12 seconds)6358client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6359client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.24 seconds)6360worker2: must succeed: pkill -f 'read -t [9]0 x'6361worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds)6362client: waiting for success: ! systemctl is-active sw6363worker2 # [ 139.952742] nix-daemon[869]: reaped child process 1559, status = succeeded6364worker2 # [ 140.147493] nix-daemon[869]: accepted connection from pid 1733, user nix-grpc-daemon (trusted)6365worker2 # [ 140.159344] nix-daemon[869]: reaped child process 1736, status = succeeded6366lb # [ 140.150138] niks3-server[1023]: 2026/09/21 12:34:56 INFO Received uploads request method=POST path=/api/pending_closures6367worker2 # [ 140.189453] nix-grpc-daemon[1491]: time=2026-09-21T12:34:57.173Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6368worker2 # [ 140.190968] nix-grpc-daemon[1491]: time=2026-09-21T12:34:57.174Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6369lb # [ 140.188139] niks3-server[1023]: 2026/09/21 12:34:56 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6370lb # [ 140.197375] niks3-server[1023]: 2026/09/21 12:34:56 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6371lb # [ 140.205816] niks3-server[1023]: 2026/09/21 12:34:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6372lb # [ 140.212586] niks3-server[1023]: 2026/09/21 12:34:56 INFO Signed narinfos id=45 count=16373worker2 # [ 140.246895] nix-grpc-daemon[1491]: time=2026-09-21T12:34:57.230Z level=INFO msg="Uploading 1 narinfos"6374lb # [ 140.215349] niks3-server[1023]: 2026/09/21 12:34:56 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6375lb # [ 140.226574] niks3-server[1023]: 2026/09/21 12:34:56 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6376lb # [ 140.231534] niks3-server[1023]: 2026/09/21 12:34:56 INFO Completed upload id=456377worker2 # [ 140.262952] nix-grpc-daemon[1491]: time=2026-09-21T12:34:57.246Z level=INFO msg="Upload complete. (316ms)"6378lb # [ 140.234286] niks3-server[1023]: 2026/09/21 12:34:56 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6379worker2 # [ 140.270669] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45500 duration_s=12 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16380worker2 # [ 140.283403] nix-daemon[869]: reaped child process 1557, status = succeeded6381client # [ 140.279907] systemd[1]: sw.service: Deactivated successfully.6382client # [ 140.283862] systemd[1]: sw.service: Consumed 128ms CPU time over 14.369s wall clock time, 41.4M memory peak, 7.3K incoming IP traffic, 8.3K outgoing IP traffic.6383worker2 # [ 140.575613] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:57Z level=info event=signal_exit6384worker2 # [ 140.795435] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:57Z level=info event=scheduler_yield6385lb # [ 140.766896] niks3-server[1023]: 2026/09/21 12:34:57 INFO lead: released remote=192.168.1.4:589206386worker1 # [ 140.783560] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:57Z level=info event=scheduler_restarting addr=lb:500516387worker2 # [ 140.802719] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:57Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:454966388worker2 # [ 140.807648] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:57Z level=info event=worker_gone addr=192.168.1.3:500516389worker2 # [ 140.810730] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:57Z level=info event=scheduler_restarting addr=lb:500516390worker2 # [ 140.813711] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:57Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:454966391worker2 # [ 140.816944] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:57Z level=info event=worker_gone addr=192.168.1.4:500516392worker1 # [ 140.983850] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:57Z level=info event=scheduler_connected addr=lb:500516393worker1 # [ 140.987814] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6394worker2 # [ 141.008442] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:57Z level=info event=scheduler_connected addr=lb:500516395worker2 # [ 141.012379] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6396client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)6397client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6398client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds)6399worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6400worker2 # [ 141.156422] systemd[1]: Starting scheduler-token.service...6401worker1 # [ 141.188824] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:57Z level=info event=scheduler_connected addr=lb:500516402worker1 # [ 141.192960] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6403worker2 # [ 141.212631] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:58Z level=info event=scheduler_connected addr=lb:500516404worker2 # [ 141.214527] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6405worker2 # [ 141.241407] systemd[1]: scheduler-token.service: Deactivated successfully.6406worker2 # [ 141.243070] systemd[1]: Finished scheduler-token.service.6407worker2 # [ 141.244414] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 86ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6408worker1 # [ 141.393857] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:58Z level=info event=scheduler_connected addr=lb:500516409worker1 # [ 141.398148] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6410worker2 # [ 141.417361] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:58Z level=info event=scheduler_connected addr=lb:500516411worker2 # [ 141.420554] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6412worker1 # [ 141.598885] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:58Z level=info event=scheduler_connected addr=lb:500516413worker2 # [ 141.618387] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:58Z level=info event=scheduler_connected addr=lb:500516414worker1 # [ 141.602918] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6415worker2 # [ 141.622232] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6416worker2 # [ 141.800303] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:58Z level=info event=scheduler_restarting6417worker1 # [ 141.804654] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:58Z level=info event=scheduler_connected addr=lb:500516418worker1 # [ 141.808500] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6419worker2 # [ 141.834663] nix-daemon[869]: reaped child process 1462, status = succeeded6420worker2 # [ 141.837128] nix-grpc-daemon[1408]: ts=2026-09-21T12:34:58Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6421worker2 # [ 141.894494] nix-grpc-daemon[1408]: E0000 00:00:1789994098.878231 1453 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true6422worker2 # [ 141.899662] nix-grpc-daemon[1408]: E0000 00:00:1789994098.883091 1453 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6423worker1 # [ 142.009482] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:58Z level=info event=scheduler_connected addr=lb:500516424worker1 # [ 142.013417] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6425worker1 # [ 142.214314] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:59Z level=info event=scheduler_connected addr=lb:500516426worker1 # [ 142.218312] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6427worker1 # [ 142.419253] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:59Z level=info event=scheduler_connected addr=lb:500516428worker1 # [ 142.423305] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6429worker1 # [ 142.623973] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:59Z level=info event=scheduler_connected addr=lb:500516430worker1 # [ 142.628154] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6431worker1 # [ 142.829726] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:59Z level=info event=scheduler_connected addr=lb:500516432worker1 # [ 142.833574] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6433worker1 # [ 143.034486] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:59Z level=info event=scheduler_connected addr=lb:500516434worker1 # [ 143.038411] nix-grpc-daemon[1499]: ts=2026-09-21T12:34:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6435worker1 # [ 143.239313] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:00Z level=info event=scheduler_connected addr=lb:500516436worker1 # [ 143.243232] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6437worker1 # [ 143.443832] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:00Z level=info event=scheduler_connected addr=lb:500516438worker1 # [ 143.447766] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6439worker2 # [ 143.611955] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6440worker2 # [ 143.615414] systemd[1]: nix-grpc-daemon.service: Consumed 675ms CPU time over 28.460s wall clock time, 33.1M memory peak, 25.3K incoming IP traffic, 36K outgoing IP traffic.6441worker2 # [ 143.632285] systemd[1]: Starting Nix worker-protocol over gRPC...6442worker1 # [ 143.648852] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:00Z level=info event=scheduler_connected addr=lb:500516443worker1 # [ 143.652868] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6444worker2 # [ 143.842849] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:00Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6445worker2 # [ 143.844580] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:00Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6446worker2 # [ 143.848479] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:00Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16447worker2 # [ 143.856062] nix-grpc-daemon[1753]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6448worker2 # [ 143.858898] nix-grpc-daemon[1753]: E0000 00:00:1789994100.839689 1753 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6449worker2 # [ 143.864302] nix-grpc-daemon[1753]: E0000 00:00:1789994100.842684 1753 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6450worker2 # [ 143.867141] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:00Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6451worker2 # [ 143.870315] systemd[1]: Started Nix worker-protocol over gRPC.6452worker1 # [ 143.854457] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:00Z level=info event=scheduler_connected addr=lb:500516453worker2 # [ 143.873606] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:00Z level=info event=scheduler_take_over6454lb # [ 143.842774] niks3-server[1023]: 2026/09/21 12:35:00 INFO lead: acquired remote=192.168.1.4:590026455worker1 # [ 143.857950] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6456worker2 # [ 143.877430] nix-grpc-daemon[1753]: E0000 00:00:1789994100.861308 1795 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6457worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.81 seconds)6458lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6459lb: must succeed: curl -sf localhost:9901/clusters6460worker2 # [ 143.891070] nix-grpc-daemon[1753]: E0000 00:00:1789994100.874154 1795 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6461worker2 # [ 143.893148] nix-grpc-daemon[1753]: E0000 00:00:1789994100.874240 1795 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6462worker2 # [ 143.904233] nix-grpc-daemon[1753]: E0000 00:00:1789994100.888135 1795 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6463worker2 # [ 143.916900] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:00Z level=info event=scheduler_connected addr=lb:500516464worker2 # [ 143.918412] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6465lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6466worker1 # [ 144.058836] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:00Z level=info event=scheduler_connected addr=lb:500516467worker1 # [ 144.062826] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6468worker2 # [ 144.119971] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=scheduler_connected addr=lb:500516469worker2 # [ 144.123884] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6470worker1 # [ 144.263722] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:01Z level=info event=scheduler_connected addr=lb:500516471worker1 # [ 144.267615] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6472worker2 # [ 144.324947] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=scheduler_connected addr=lb:500516473worker2 # [ 144.328925] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6474worker1 # [ 144.468708] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:01Z level=info event=scheduler_connected addr=lb:500516475worker1 # [ 144.472659] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6476worker2 # [ 144.530172] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=scheduler_connected addr=lb:500516477worker2 # [ 144.534099] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6478worker1 # [ 144.673635] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:01Z level=info event=scheduler_connected addr=lb:500516479worker2 # [ 144.695206] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:411966480worker2 # [ 144.698898] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06481worker2 # [ 144.735055] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=scheduler_connected addr=lb:500516482worker2 # [ 144.744135] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=oidc_jwks provider=mock keys=16483worker2 # [ 144.748067] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:411966484worker2 # [ 144.751817] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:01Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06485lb: must succeed: curl -sf localhost:9901/clusters6486lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6487lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.17 seconds)6488lb: must succeed: curl -sf localhost:9901/clusters6489lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6490worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6491worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6492(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 19.29 seconds)6493subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6494client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16495worker2 # [ 145.459817] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:412026496worker2 # [ 145.471354] nix-daemon[869]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)6497worker2 # [ 145.485279] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:412026498worker2 # [ 145.497295] nix-grpc-daemon[1753]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 99 ms (attempt 1/5)6499worker2 # [ 145.602824] nix-grpc-daemon[1753]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 64 ms (attempt 2/5)6500worker2 # [ 145.672709] nix-grpc-daemon[1753]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 157 ms (attempt 3/5)6501worker2 # [ 145.835448] nix-grpc-daemon[1753]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 394 ms (attempt 4/5)6502worker2 # [ 146.235724] nix-grpc-daemon[1753]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6503worker2 # [ 146.239553] nix-grpc-daemon[1753]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6504worker2 # [ 146.286679] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:03Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06505worker2 # [ 146.291573] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:03Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16506worker2 # [ 146.295909] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:03Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06507worker2 # [ 146.299605] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:03Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06508worker1 # [ 146.321289] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686509worker1 # [ 146.333815] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686510worker1 # [ 146.345553] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686511worker1 # [ 146.354888] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6512worker1 # [ 146.582172] nix-daemon[852]: accepted connection from pid 1772, user nix-grpc-daemon (trusted)6513worker1 # [ 146.594202] nix-daemon[852]: reaped child process 1775, status = succeeded6514worker1 # [ 146.812427] nix-daemon[852]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6515worker1 # [ 146.824249] nix-daemon[852]: reaped child process 1780, status = succeeded6516lb # [ 146.834369] niks3-server[1023]: 2026/09/21 12:35:03 INFO Received uploads request method=POST path=/api/pending_closures6517worker1 # [ 146.864978] nix-grpc-daemon[1591]: time=2026-09-21T12:35:03.672Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6518worker1 # [ 146.866423] nix-grpc-daemon[1591]: time=2026-09-21T12:35:03.672Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6519lb # [ 146.876158] niks3-server[1023]: 2026/09/21 12:35:03 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6520lb # [ 146.881250] niks3-server[1023]: 2026/09/21 12:35:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6521lb # [ 146.888361] niks3-server[1023]: 2026/09/21 12:35:03 INFO Signed narinfos id=46 count=16522worker1 # [ 146.906942] nix-grpc-daemon[1591]: time=2026-09-21T12:35:03.714Z level=INFO msg="Uploading 1 narinfos"6523lb # [ 146.894134] niks3-server[1023]: 2026/09/21 12:35:03 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6524lb # [ 146.903834] niks3-server[1023]: 2026/09/21 12:35:03 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6525lb # [ 146.909506] niks3-server[1023]: 2026/09/21 12:35:03 INFO Completed upload id=466526lb # [ 146.911124] niks3-server[1023]: 2026/09/21 12:35:03 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6527worker1 # [ 146.926339] nix-grpc-daemon[1591]: time=2026-09-21T12:35:03.734Z level=INFO msg="Upload complete. (557ms)"6528worker1 # [ 146.930509] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=5046529worker1 # [ 146.936436] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686530worker1 # [ 146.946088] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6531worker1 # [ 146.957186] nix-daemon[852]: reaped child process 1770, status = succeeded6532worker1 # [ 146.960838] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6533worker1 # [ 147.985428] nix-daemon[852]: reaped child process 1784, status = succeeded6534worker1 # [ 148.197498] nix-daemon[852]: accepted connection from pid 1800, user nix-grpc-daemon (trusted)6535worker1 # [ 148.208173] nix-daemon[852]: reaped child process 1803, status = succeeded6536lb # [ 148.217314] niks3-server[1023]: 2026/09/21 12:35:04 INFO Received uploads request method=POST path=/api/pending_closures6537worker1 # [ 148.242102] nix-grpc-daemon[1591]: time=2026-09-21T12:35:05.049Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6538worker1 # [ 148.244401] nix-grpc-daemon[1591]: time=2026-09-21T12:35:05.051Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6539lb # [ 148.262641] niks3-server[1023]: 2026/09/21 12:35:04 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6540lb # [ 148.270062] niks3-server[1023]: 2026/09/21 12:35:04 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6541lb # [ 148.278259] niks3-server[1023]: 2026/09/21 12:35:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6542worker1 # [ 148.298698] nix-grpc-daemon[1591]: time=2026-09-21T12:35:05.106Z level=INFO msg="Uploading 1 narinfos"6543lb # [ 148.282874] niks3-server[1023]: 2026/09/21 12:35:04 INFO Signed narinfos id=47 count=16544lb # [ 148.289958] niks3-server[1023]: 2026/09/21 12:35:04 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6545lb # [ 148.299168] niks3-server[1023]: 2026/09/21 12:35:04 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6546lb # [ 148.304490] niks3-server[1023]: 2026/09/21 12:35:04 INFO Completed upload id=476547worker1 # [ 148.320251] nix-grpc-daemon[1591]: time=2026-09-21T12:35:05.128Z level=INFO msg="Upload complete. (351ms)"6548lb # [ 148.306857] niks3-server[1023]: 2026/09/21 12:35:04 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6549worker1 # [ 148.325879] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=16550worker2 # [ 148.345959] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:05Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16551worker2 # [ 148.353749] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:05Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06552worker1 # [ 148.340409] nix-daemon[852]: reaped child process 1782, status = succeeded6553worker1 # [ 148.379345] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686554worker1 # [ 148.392235] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686555worker1 # [ 148.403771] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686556worker1 # [ 148.411890] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6557worker1 # [ 148.630351] nix-daemon[852]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)6558worker1 # [ 148.642255] nix-daemon[852]: reaped child process 1810, status = succeeded6559worker1 # [ 148.863131] nix-daemon[852]: accepted connection from pid 1812, user nix-grpc-daemon (trusted)6560worker1 # [ 148.874768] nix-daemon[852]: reaped child process 1815, status = succeeded6561lb # [ 148.878551] niks3-server[1023]: 2026/09/21 12:35:05 INFO Received uploads request method=POST path=/api/pending_closures6562worker1 # [ 148.904576] nix-grpc-daemon[1591]: time=2026-09-21T12:35:05.712Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6563worker1 # [ 148.906511] nix-grpc-daemon[1591]: time=2026-09-21T12:35:05.714Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6564lb # [ 148.910168] niks3-server[1023]: 2026/09/21 12:35:05 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6565lb # [ 148.921163] niks3-server[1023]: 2026/09/21 12:35:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6566worker1 # [ 148.942982] nix-grpc-daemon[1591]: time=2026-09-21T12:35:05.750Z level=INFO msg="Uploading 1 narinfos"6567lb # [ 148.927424] niks3-server[1023]: 2026/09/21 12:35:05 INFO Signed narinfos id=48 count=16568lb # [ 148.931718] niks3-server[1023]: 2026/09/21 12:35:05 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6569lb # [ 148.943981] niks3-server[1023]: 2026/09/21 12:35:05 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6570lb # [ 148.949918] niks3-server[1023]: 2026/09/21 12:35:05 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6571lb # [ 148.955654] niks3-server[1023]: 2026/09/21 12:35:05 INFO Completed upload id=486572worker1 # [ 148.971351] nix-grpc-daemon[1591]: time=2026-09-21T12:35:05.779Z level=INFO msg="Upload complete. (545ms)"6573worker1 # [ 148.976188] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=5046574worker1 # [ 148.981837] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686575worker1 # [ 148.995641] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6576worker1 # [ 148.999948] nix-daemon[852]: reaped child process 1805, status = succeeded6577worker1 # [ 149.008350] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6578lb # [ 149.260686] envoy[931]: conn peer=[2001:db8:1::1]:38538 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9628 tx=4068 ms=68096579lb # [ 149.264500] envoy[931]: conn peer=[2001:db8:1::1]:38526 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2936 tx=1454 ms=140876580lb # [ 149.267995] envoy[931]: conn peer=[2001:db8:1::4]:37764 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=4876 tx=539 ms=262166581worker1 # [ 150.222269] nix-daemon[852]: reaped child process 1819, status = succeeded6582worker1 # [ 150.415547] nix-daemon[852]: accepted connection from pid 1835, user nix-grpc-daemon (trusted)6583worker1 # [ 150.427328] nix-daemon[852]: reaped child process 1838, status = succeeded6584lb # [ 150.435930] niks3-server[1023]: 2026/09/21 12:35:07 INFO Received uploads request method=POST path=/api/pending_closures6585worker1 # [ 150.460689] nix-grpc-daemon[1591]: time=2026-09-21T12:35:07.268Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6586worker1 # [ 150.462233] nix-grpc-daemon[1591]: time=2026-09-21T12:35:07.270Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6587lb # [ 150.477557] niks3-server[1023]: 2026/09/21 12:35:07 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6588lb # [ 150.485314] niks3-server[1023]: 2026/09/21 12:35:07 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6589lb # [ 150.496112] niks3-server[1023]: 2026/09/21 12:35:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6590worker1 # [ 150.519036] nix-grpc-daemon[1591]: time=2026-09-21T12:35:07.326Z level=INFO msg="Uploading 1 narinfos"6591lb # [ 150.503265] niks3-server[1023]: 2026/09/21 12:35:07 INFO Signed narinfos id=49 count=16592lb # [ 150.506844] niks3-server[1023]: 2026/09/21 12:35:07 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6593lb # [ 150.516473] niks3-server[1023]: 2026/09/21 12:35:07 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6594lb # [ 150.521391] niks3-server[1023]: 2026/09/21 12:35:07 INFO Completed upload id=496595worker1 # [ 150.536516] nix-grpc-daemon[1591]: time=2026-09-21T12:35:07.344Z level=INFO msg="Upload complete. (321ms)"6596lb # [ 150.524674] niks3-server[1023]: 2026/09/21 12:35:07 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6597worker2 # [ 150.560384] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:07Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16598worker1 # [ 150.541529] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=16599worker1 # [ 150.553194] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686600worker2 # [ 150.569996] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:07Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06601worker1 # [ 150.561491] nix-daemon[852]: reaped child process 1817, status = succeeded6602worker1 # [ 150.567275] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686603worker1 # [ 150.576811] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686604worker1 # [ 150.585132] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6605worker1 # [ 150.789598] nix-daemon[852]: accepted connection from pid 1842, user nix-grpc-daemon (trusted)6606worker1 # [ 150.801419] nix-daemon[852]: reaped child process 1845, status = succeeded6607worker1 # [ 151.014665] nix-daemon[852]: accepted connection from pid 1847, user nix-grpc-daemon (trusted)6608worker1 # [ 151.026154] nix-daemon[852]: reaped child process 1850, status = succeeded6609lb # [ 151.031130] niks3-server[1023]: 2026/09/21 12:35:07 INFO Received uploads request method=POST path=/api/pending_closures6610worker1 # [ 151.055028] nix-grpc-daemon[1591]: time=2026-09-21T12:35:07.863Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6611worker1 # [ 151.056801] nix-grpc-daemon[1591]: time=2026-09-21T12:35:07.864Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6612lb # [ 151.061322] niks3-server[1023]: 2026/09/21 12:35:07 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6613lb # [ 151.069611] niks3-server[1023]: 2026/09/21 12:35:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6614lb # [ 151.073806] niks3-server[1023]: 2026/09/21 12:35:07 INFO Signed narinfos id=50 count=16615worker1 # [ 151.092352] nix-grpc-daemon[1591]: time=2026-09-21T12:35:07.900Z level=INFO msg="Uploading 1 narinfos"6616lb # [ 151.083369] niks3-server[1023]: 2026/09/21 12:35:07 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6617lb # [ 151.092658] niks3-server[1023]: 2026/09/21 12:35:07 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6618lb # [ 151.098316] niks3-server[1023]: 2026/09/21 12:35:07 INFO Completed upload id=506619worker1 # [ 151.113842] nix-grpc-daemon[1591]: time=2026-09-21T12:35:07.921Z level=INFO msg="Upload complete. (522ms)"6620lb # [ 151.101261] niks3-server[1023]: 2026/09/21 12:35:07 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6621worker1 # [ 151.117422] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=5046622worker1 # [ 151.123201] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686623worker1 # [ 151.133130] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6624worker1 # [ 151.135764] nix-daemon[852]: reaped child process 1840, status = succeeded6625worker1 # [ 151.140916] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6626worker1 # [ 152.565995] nix-daemon[852]: reaped child process 1854, status = succeeded6627worker1 # [ 152.778909] nix-daemon[852]: accepted connection from pid 1870, user nix-grpc-daemon (trusted)6628worker1 # [ 152.790554] nix-daemon[852]: reaped child process 1873, status = succeeded6629lb # [ 152.801541] niks3-server[1023]: 2026/09/21 12:35:09 INFO Received uploads request method=POST path=/api/pending_closures6630worker1 # [ 152.825965] nix-grpc-daemon[1591]: time=2026-09-21T12:35:09.633Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6631worker1 # [ 152.827652] nix-grpc-daemon[1591]: time=2026-09-21T12:35:09.635Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"6632lb # [ 152.833588] niks3-server[1023]: 2026/09/21 12:35:09 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv6633lb # [ 152.846584] niks3-server[1023]: 2026/09/21 12:35:09 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6634lb # [ 152.858411] niks3-server[1023]: 2026/09/21 12:35:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6635lb # [ 152.864743] niks3-server[1023]: 2026/09/21 12:35:09 INFO Signed narinfos id=51 count=16636worker1 # [ 152.882668] nix-grpc-daemon[1591]: time=2026-09-21T12:35:09.690Z level=INFO msg="Uploading 1 narinfos"6637lb # [ 152.868841] niks3-server[1023]: 2026/09/21 12:35:09 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls6638lb # [ 152.879316] niks3-server[1023]: 2026/09/21 12:35:09 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6639lb # [ 152.885144] niks3-server[1023]: 2026/09/21 12:35:09 INFO Completed upload id=516640worker1 # [ 152.900855] nix-grpc-daemon[1591]: time=2026-09-21T12:35:09.708Z level=INFO msg="Upload complete. (351ms)"6641lb # [ 152.886953] niks3-server[1023]: 2026/09/21 12:35:09 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo6642worker1 # [ 152.906343] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=16643worker2 # [ 152.922991] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:09Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=16644worker2 # [ 152.933919] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:09Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=06645worker1 # [ 152.920861] nix-daemon[852]: reaped child process 1852, status = succeeded6646worker1 # [ 152.953466] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686647worker1 # [ 152.966556] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686648worker1 # [ 152.980739] nix-grpc-daemon[1591]: time=2026-09-21T12:35:09.788Z level=INFO msg="All 1 paths already cached"6649worker1 # [ 152.988119] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686650worker1 # [ 152.996353] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6651worker1 # [ 153.219544] nix-daemon[852]: accepted connection from pid 1877, user nix-grpc-daemon (trusted)6652worker1 # [ 153.231464] nix-daemon[852]: reaped child process 1880, status = succeeded6653worker1 # [ 153.456616] nix-daemon[852]: accepted connection from pid 1882, user nix-grpc-daemon (trusted)6654worker1 # [ 153.468557] nix-daemon[852]: reaped child process 1885, status = succeeded6655lb # [ 153.477394] niks3-server[1023]: 2026/09/21 12:35:10 INFO Received uploads request method=POST path=/api/pending_closures6656worker1 # [ 153.501511] nix-grpc-daemon[1591]: time=2026-09-21T12:35:10.309Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6657worker1 # [ 153.502864] nix-grpc-daemon[1591]: time=2026-09-21T12:35:10.309Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"6658lb # [ 153.503071] niks3-server[1023]: 2026/09/21 12:35:10 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst6659lb # [ 153.515407] niks3-server[1023]: 2026/09/21 12:35:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6660lb # [ 153.522297] niks3-server[1023]: 2026/09/21 12:35:10 INFO Signed narinfos id=52 count=16661worker1 # [ 153.540824] nix-grpc-daemon[1591]: time=2026-09-21T12:35:10.348Z level=INFO msg="Uploading 1 narinfos"6662lb # [ 153.525386] niks3-server[1023]: 2026/09/21 12:35:10 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls6663lb # [ 153.538618] niks3-server[1023]: 2026/09/21 12:35:10 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6664lb # [ 153.544606] niks3-server[1023]: 2026/09/21 12:35:10 INFO Completed upload id=526665worker1 # [ 153.560296] nix-grpc-daemon[1591]: time=2026-09-21T12:35:10.368Z level=INFO msg="Upload complete. (550ms)"6666lb # [ 153.546864] niks3-server[1023]: 2026/09/21 12:35:10 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo6667worker1 # [ 153.563846] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=6246668worker1 # [ 153.568902] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686669worker1 # [ 153.581146] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6670worker1 # [ 153.583262] nix-daemon[852]: reaped child process 1875, status = succeeded6671worker1 # [ 153.588511] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6672worker1 # [ 154.603315] nix-daemon[852]: reaped child process 1889, status = succeeded6673worker1 # [ 154.805690] nix-daemon[852]: accepted connection from pid 1905, user nix-grpc-daemon (trusted)6674worker1 # [ 154.818197] nix-daemon[852]: reaped child process 1908, status = succeeded6675lb # [ 154.825349] niks3-server[1023]: 2026/09/21 12:35:11 INFO Received uploads request method=POST path=/api/pending_closures6676worker1 # [ 154.853638] nix-grpc-daemon[1591]: time=2026-09-21T12:35:11.661Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6677worker1 # [ 154.855264] nix-grpc-daemon[1591]: time=2026-09-21T12:35:11.663Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"6678lb # [ 154.874304] niks3-server[1023]: 2026/09/21 12:35:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6679worker1 # [ 154.894254] nix-grpc-daemon[1591]: time=2026-09-21T12:35:11.702Z level=INFO msg="Uploading 1 narinfos"6680lb # [ 154.879708] niks3-server[1023]: 2026/09/21 12:35:11 INFO Signed narinfos id=53 count=16681lb # [ 154.892368] niks3-server[1023]: 2026/09/21 12:35:11 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv6682lb # [ 154.898174] niks3-server[1023]: 2026/09/21 12:35:11 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst6683lb # [ 154.909361] niks3-server[1023]: 2026/09/21 12:35:11 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6684lb # [ 154.916976] niks3-server[1023]: 2026/09/21 12:35:11 INFO Completed upload id=536685worker1 # [ 154.932356] nix-grpc-daemon[1591]: time=2026-09-21T12:35:11.740Z level=INFO msg="Upload complete. (337ms)"6686lb # [ 154.919224] niks3-server[1023]: 2026/09/21 12:35:11 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls6687lb # [ 154.923138] niks3-server[1023]: 2026/09/21 12:35:11 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo6688worker2 # [ 154.954860] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:11Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=16689worker1 # [ 154.937972] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=16690worker1 # [ 154.946442] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686691worker2 # [ 154.964644] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:11Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=06692worker1 # [ 154.958588] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686693worker1 # [ 154.962184] nix-daemon[852]: reaped child process 1887, status = succeeded6694worker1 # [ 154.966270] nix-grpc-daemon[1591]: time=2026-09-21T12:35:11.774Z level=INFO msg="All 1 paths already cached"6695worker1 # [ 154.970105] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686696worker1 # [ 154.974916] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6697worker1 # [ 155.188350] nix-daemon[852]: accepted connection from pid 1912, user nix-grpc-daemon (trusted)6698worker1 # [ 155.202379] nix-daemon[852]: reaped child process 1915, status = succeeded6699worker1 # [ 155.423646] nix-daemon[852]: accepted connection from pid 1917, user nix-grpc-daemon (trusted)6700worker1 # [ 155.437419] nix-daemon[852]: reaped child process 1920, status = succeeded6701lb # [ 155.441282] niks3-server[1023]: 2026/09/21 12:35:12 INFO Received uploads request method=POST path=/api/pending_closures6702worker1 # [ 155.465870] nix-grpc-daemon[1591]: time=2026-09-21T12:35:12.273Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6703worker1 # [ 155.467670] nix-grpc-daemon[1591]: time=2026-09-21T12:35:12.275Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"6704lb # [ 155.471212] niks3-server[1023]: 2026/09/21 12:35:12 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst6705lb # [ 155.483698] niks3-server[1023]: 2026/09/21 12:35:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6706lb # [ 155.489236] niks3-server[1023]: 2026/09/21 12:35:12 INFO Signed narinfos id=54 count=16707worker1 # [ 155.507819] nix-grpc-daemon[1591]: time=2026-09-21T12:35:12.315Z level=INFO msg="Uploading 1 narinfos"6708lb # [ 155.496384] niks3-server[1023]: 2026/09/21 12:35:12 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls6709lb # [ 155.506169] niks3-server[1023]: 2026/09/21 12:35:12 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6710lb # [ 155.514757] niks3-server[1023]: 2026/09/21 12:35:12 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo6711lb # [ 155.518707] niks3-server[1023]: 2026/09/21 12:35:12 INFO Completed upload id=546712worker1 # [ 155.534349] nix-grpc-daemon[1591]: time=2026-09-21T12:35:12.342Z level=INFO msg="Upload complete. (552ms)"6713worker1 # [ 155.539223] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=6246714worker1 # [ 155.545574] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686715worker1 # [ 155.556493] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6716worker1 # [ 155.567059] nix-daemon[852]: reaped child process 1910, status = succeeded6717worker1 # [ 155.572557] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6718worker1 # [ 156.398836] nix-daemon[852]: reaped child process 1924, status = succeeded6719worker1 # [ 156.612532] nix-daemon[852]: accepted connection from pid 1940, user nix-grpc-daemon (trusted)6720worker1 # [ 156.626606] nix-daemon[852]: reaped child process 1943, status = succeeded6721lb # [ 156.632876] niks3-server[1023]: 2026/09/21 12:35:13 INFO Received uploads request method=POST path=/api/pending_closures6722worker1 # [ 156.657078] nix-grpc-daemon[1591]: time=2026-09-21T12:35:13.464Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6723worker1 # [ 156.659510] nix-grpc-daemon[1591]: time=2026-09-21T12:35:13.466Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"6724lb # [ 156.664261] niks3-server[1023]: 2026/09/21 12:35:13 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv6725lb # [ 156.682150] niks3-server[1023]: 2026/09/21 12:35:13 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst6726lb # [ 156.691205] niks3-server[1023]: 2026/09/21 12:35:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6727worker1 # [ 156.710730] nix-grpc-daemon[1591]: time=2026-09-21T12:35:13.518Z level=INFO msg="Uploading 1 narinfos"6728lb # [ 156.695520] niks3-server[1023]: 2026/09/21 12:35:13 INFO Signed narinfos id=55 count=16729lb # [ 156.699870] niks3-server[1023]: 2026/09/21 12:35:13 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls6730lb # [ 156.709842] niks3-server[1023]: 2026/09/21 12:35:13 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6731lb # [ 156.713214] niks3-server[1023]: 2026/09/21 12:35:13 INFO Completed upload id=556732worker1 # [ 156.730172] nix-grpc-daemon[1591]: time=2026-09-21T12:35:13.538Z level=INFO msg="Upload complete. (351ms)"6733lb # [ 156.716373] niks3-server[1023]: 2026/09/21 12:35:13 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo6734worker1 # [ 156.738390] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=16735worker2 # [ 156.756193] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:13Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=16736worker1 # [ 156.744971] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686737worker2 # [ 156.762225] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:13Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=06738worker1 # [ 156.757540] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686739worker1 # [ 156.765247] nix-daemon[852]: reaped child process 1922, status = succeeded6740worker1 # [ 156.767879] nix-grpc-daemon[1591]: time=2026-09-21T12:35:13.575Z level=INFO msg="All 1 paths already cached"6741worker1 # [ 156.772504] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686742worker1 # [ 156.778075] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6743worker1 # [ 156.992636] nix-daemon[852]: accepted connection from pid 1947, user nix-grpc-daemon (trusted)6744worker1 # [ 157.007042] nix-daemon[852]: reaped child process 1950, status = succeeded6745worker1 # [ 157.233104] nix-daemon[852]: accepted connection from pid 1952, user nix-grpc-daemon (trusted)6746worker1 # [ 157.245496] nix-daemon[852]: reaped child process 1955, status = succeeded6747lb # [ 157.251250] niks3-server[1023]: 2026/09/21 12:35:13 INFO Received uploads request method=POST path=/api/pending_closures6748worker1 # [ 157.274708] nix-grpc-daemon[1591]: time=2026-09-21T12:35:14.082Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6749worker1 # [ 157.276307] nix-grpc-daemon[1591]: time=2026-09-21T12:35:14.084Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"6750lb # [ 157.282408] niks3-server[1023]: 2026/09/21 12:35:13 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst6751lb # [ 157.292418] niks3-server[1023]: 2026/09/21 12:35:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6752worker1 # [ 157.313157] nix-grpc-daemon[1591]: time=2026-09-21T12:35:14.121Z level=INFO msg="Uploading 1 narinfos"6753lb # [ 157.298597] niks3-server[1023]: 2026/09/21 12:35:13 INFO Signed narinfos id=56 count=16754lb # [ 157.303901] niks3-server[1023]: 2026/09/21 12:35:13 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls6755lb # [ 157.315804] niks3-server[1023]: 2026/09/21 12:35:13 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6756lb # [ 157.323412] niks3-server[1023]: 2026/09/21 12:35:13 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo6757lb # [ 157.328771] niks3-server[1023]: 2026/09/21 12:35:14 INFO Completed upload id=566758worker1 # [ 157.344471] nix-grpc-daemon[1591]: time=2026-09-21T12:35:14.152Z level=INFO msg="Upload complete. (561ms)"6759worker1 # [ 157.348262] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=6246760worker1 # [ 157.353230] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686761worker1 # [ 157.362841] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6762worker1 # [ 157.373156] nix-daemon[852]: reaped child process 1945, status = succeeded6763worker1 # [ 157.376933] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6764worker1 # [ 158.591803] nix-daemon[852]: reaped child process 1959, status = succeeded6765worker1 # [ 158.802950] nix-daemon[852]: accepted connection from pid 1975, user nix-grpc-daemon (trusted)6766worker1 # [ 158.815428] nix-daemon[852]: reaped child process 1978, status = succeeded6767lb # [ 158.823835] niks3-server[1023]: 2026/09/21 12:35:15 INFO Received uploads request method=POST path=/api/pending_closures6768worker1 # [ 158.850663] nix-grpc-daemon[1591]: time=2026-09-21T12:35:15.658Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6769worker1 # [ 158.852268] nix-grpc-daemon[1591]: time=2026-09-21T12:35:15.660Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"6770lb # [ 158.866390] niks3-server[1023]: 2026/09/21 12:35:15 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv6771lb # [ 158.872110] niks3-server[1023]: 2026/09/21 12:35:15 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst6772lb # [ 158.884874] niks3-server[1023]: 2026/09/21 12:35:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6773worker1 # [ 158.904307] nix-grpc-daemon[1591]: time=2026-09-21T12:35:15.712Z level=INFO msg="Uploading 1 narinfos"6774lb # [ 158.889133] niks3-server[1023]: 2026/09/21 12:35:15 INFO Signed narinfos id=57 count=16775lb # [ 158.894476] niks3-server[1023]: 2026/09/21 12:35:15 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls6776lb # [ 158.902145] niks3-server[1023]: 2026/09/21 12:35:15 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6777lb # [ 158.906233] niks3-server[1023]: 2026/09/21 12:35:15 INFO Completed upload id=576778lb # [ 158.908157] niks3-server[1023]: 2026/09/21 12:35:15 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo6779worker1 # [ 158.923152] nix-grpc-daemon[1591]: time=2026-09-21T12:35:15.731Z level=INFO msg="Upload complete. (338ms)"6780worker2 # [ 158.945388] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:15Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=16781worker1 # [ 158.931753] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=16782worker1 # [ 158.936622] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686783worker2 # [ 158.955468] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:15Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=06784worker1 # [ 158.949571] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686785worker1 # [ 158.953191] nix-daemon[852]: reaped child process 1957, status = succeeded6786worker1 # [ 158.957781] nix-grpc-daemon[1591]: time=2026-09-21T12:35:15.765Z level=INFO msg="All 2 paths already cached"6787worker1 # [ 158.962210] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686788worker1 # [ 158.966720] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6789worker1 # [ 159.181678] nix-daemon[852]: accepted connection from pid 1982, user nix-grpc-daemon (trusted)6790worker1 # [ 159.195991] nix-daemon[852]: reaped child process 1985, status = succeeded6791worker1 # [ 159.428523] nix-daemon[852]: accepted connection from pid 1987, user nix-grpc-daemon (trusted)6792worker1 # [ 159.442429] nix-daemon[852]: reaped child process 1990, status = succeeded6793lb # [ 159.450495] niks3-server[1023]: 2026/09/21 12:35:16 INFO Received uploads request method=POST path=/api/pending_closures6794worker1 # [ 159.475529] nix-grpc-daemon[1591]: time=2026-09-21T12:35:16.283Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6795worker1 # [ 159.476936] nix-grpc-daemon[1591]: time=2026-09-21T12:35:16.283Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"6796lb # [ 159.477943] niks3-server[1023]: 2026/09/21 12:35:16 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst6797lb # [ 159.490240] niks3-server[1023]: 2026/09/21 12:35:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6798lb # [ 159.497271] niks3-server[1023]: 2026/09/21 12:35:16 INFO Signed narinfos id=58 count=16799worker1 # [ 159.516152] nix-grpc-daemon[1591]: time=2026-09-21T12:35:16.323Z level=INFO msg="Uploading 1 narinfos"6800lb # [ 159.502786] niks3-server[1023]: 2026/09/21 12:35:16 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls6801lb # [ 159.513805] niks3-server[1023]: 2026/09/21 12:35:16 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6802worker1 # [ 159.536159] nix-grpc-daemon[1591]: time=2026-09-21T12:35:16.344Z level=INFO msg="Upload complete. (562ms)"6803lb # [ 159.520623] niks3-server[1023]: 2026/09/21 12:35:16 INFO Completed upload id=586804lb # [ 159.524369] niks3-server[1023]: 2026/09/21 12:35:16 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo6805worker1 # [ 159.539080] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=7526806worker1 # [ 159.544643] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686807worker1 # [ 159.552807] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6808worker1 # [ 159.561939] nix-daemon[852]: reaped child process 1980, status = succeeded6809worker1 # [ 159.567814] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6810worker1 # [ 160.595953] nix-daemon[852]: reaped child process 1994, status = succeeded6811worker1 # [ 160.817120] nix-daemon[852]: accepted connection from pid 2010, user nix-grpc-daemon (trusted)6812worker1 # [ 160.829735] nix-daemon[852]: reaped child process 2013, status = succeeded6813lb # [ 160.838558] niks3-server[1023]: 2026/09/21 12:35:17 INFO Received uploads request method=POST path=/api/pending_closures6814worker1 # [ 160.863542] nix-grpc-daemon[1591]: time=2026-09-21T12:35:17.671Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6815worker1 # [ 160.865194] nix-grpc-daemon[1591]: time=2026-09-21T12:35:17.673Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"6816lb # [ 160.874652] niks3-server[1023]: 2026/09/21 12:35:17 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv6817lb # [ 160.888857] niks3-server[1023]: 2026/09/21 12:35:17 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst6818lb # [ 160.898108] niks3-server[1023]: 2026/09/21 12:35:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6819lb # [ 160.902078] niks3-server[1023]: 2026/09/21 12:35:17 INFO Signed narinfos id=59 count=16820worker1 # [ 160.917725] nix-grpc-daemon[1591]: time=2026-09-21T12:35:17.725Z level=INFO msg="Uploading 1 narinfos"6821lb # [ 160.907540] niks3-server[1023]: 2026/09/21 12:35:17 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls6822lb # [ 160.916198] niks3-server[1023]: 2026/09/21 12:35:17 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6823lb # [ 160.919497] niks3-server[1023]: 2026/09/21 12:35:17 INFO Completed upload id=596824lb # [ 160.921753] niks3-server[1023]: 2026/09/21 12:35:17 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo6825worker1 # [ 160.936391] nix-grpc-daemon[1591]: time=2026-09-21T12:35:17.744Z level=INFO msg="Upload complete. (360ms)"6826worker2 # [ 160.958636] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:17Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=16827worker1 # [ 160.941845] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=16828worker1 # [ 160.956840] nix-daemon[852]: reaped child process 1992, status = succeeded6829worker1 # [ 160.987132] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686830worker1 # [ 161.000674] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686831worker1 # [ 161.014542] nix-grpc-daemon[1591]: time=2026-09-21T12:35:17.822Z level=INFO msg="All 2 paths already cached"6832worker1 # [ 161.022069] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686833worker1 # [ 161.032174] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6834worker1 # [ 161.267736] nix-daemon[852]: accepted connection from pid 2017, user nix-grpc-daemon (trusted)6835worker1 # [ 161.281891] nix-daemon[852]: reaped child process 2020, status = succeeded6836worker1 # [ 161.515234] nix-daemon[852]: accepted connection from pid 2022, user nix-grpc-daemon (trusted)6837worker1 # [ 161.529114] nix-daemon[852]: reaped child process 2025, status = succeeded6838lb # [ 161.537831] niks3-server[1023]: 2026/09/21 12:35:18 INFO Received uploads request method=POST path=/api/pending_closures6839worker1 # [ 161.561447] nix-grpc-daemon[1591]: time=2026-09-21T12:35:18.369Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6840worker1 # [ 161.563042] nix-grpc-daemon[1591]: time=2026-09-21T12:35:18.371Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"6841lb # [ 161.567673] niks3-server[1023]: 2026/09/21 12:35:18 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst6842lb # [ 161.579301] niks3-server[1023]: 2026/09/21 12:35:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6843lb # [ 161.585756] niks3-server[1023]: 2026/09/21 12:35:18 INFO Signed narinfos id=60 count=16844worker1 # [ 161.604248] nix-grpc-daemon[1591]: time=2026-09-21T12:35:18.412Z level=INFO msg="Uploading 1 narinfos"6845lb # [ 161.589223] niks3-server[1023]: 2026/09/21 12:35:18 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls6846lb # [ 161.603173] niks3-server[1023]: 2026/09/21 12:35:18 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6847worker1 # [ 161.625375] nix-grpc-daemon[1591]: time=2026-09-21T12:35:18.433Z level=INFO msg="Upload complete. (580ms)"6848lb # [ 161.609888] niks3-server[1023]: 2026/09/21 12:35:18 INFO Completed upload id=606849lb # [ 161.613684] niks3-server[1023]: 2026/09/21 12:35:18 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo6850worker1 # [ 161.628700] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=7526851worker1 # [ 161.633461] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686852worker1 # [ 161.644088] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6853worker1 # [ 161.649730] nix-daemon[852]: reaped child process 2015, status = succeeded6854worker1 # [ 161.655236] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6855worker1 # [ 163.273633] nix-daemon[852]: reaped child process 2029, status = succeeded6856worker1 # [ 163.486868] nix-daemon[852]: accepted connection from pid 2045, user nix-grpc-daemon (trusted)6857worker1 # [ 163.501398] nix-daemon[852]: reaped child process 2048, status = succeeded6858lb # [ 163.510865] niks3-server[1023]: 2026/09/21 12:35:20 INFO Received uploads request method=POST path=/api/pending_closures6859worker1 # [ 163.535387] nix-grpc-daemon[1591]: time=2026-09-21T12:35:20.343Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6860worker1 # [ 163.539293] nix-grpc-daemon[1591]: time=2026-09-21T12:35:20.345Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"6861lb # [ 163.551256] niks3-server[1023]: 2026/09/21 12:35:20 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst6862lb # [ 163.557174] niks3-server[1023]: 2026/09/21 12:35:20 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv6863lb # [ 163.566652] niks3-server[1023]: 2026/09/21 12:35:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6864worker1 # [ 163.587292] nix-grpc-daemon[1591]: time=2026-09-21T12:35:20.395Z level=INFO msg="Uploading 1 narinfos"6865lb # [ 163.572156] niks3-server[1023]: 2026/09/21 12:35:20 INFO Signed narinfos id=61 count=16866lb # [ 163.577762] niks3-server[1023]: 2026/09/21 12:35:20 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls6867lb # [ 163.587654] niks3-server[1023]: 2026/09/21 12:35:20 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6868lb # [ 163.592778] niks3-server[1023]: 2026/09/21 12:35:20 INFO Completed upload id=616869worker1 # [ 163.608475] nix-grpc-daemon[1591]: time=2026-09-21T12:35:20.416Z level=INFO msg="Upload complete. (344ms)"6870lb # [ 163.595211] niks3-server[1023]: 2026/09/21 12:35:20 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo6871worker1 # [ 163.613463] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=16872worker2 # [ 163.630858] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:20Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=16873worker1 # [ 163.628727] nix-daemon[852]: reaped child process 2027, status = succeeded6874worker1 # [ 163.657798] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686875worker1 # [ 163.670905] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398686876worker1 # [ 163.683872] nix-grpc-daemon[1591]: time=2026-09-21T12:35:20.491Z level=INFO msg="All 2 paths already cached"6877worker1 # [ 163.691188] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398686878worker1 # [ 163.701174] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6879worker1 # [ 163.934261] nix-daemon[852]: accepted connection from pid 2052, user nix-grpc-daemon (trusted)6880worker1 # [ 163.948544] nix-daemon[852]: reaped child process 2055, status = succeeded6881worker1 # [ 164.183444] nix-daemon[852]: accepted connection from pid 2057, user nix-grpc-daemon (trusted)6882worker1 # [ 164.197332] nix-daemon[852]: reaped child process 2060, status = succeeded6883lb # [ 164.205683] niks3-server[1023]: 2026/09/21 12:35:20 INFO Received uploads request method=POST path=/api/pending_closures6884worker1 # [ 164.229428] nix-grpc-daemon[1591]: time=2026-09-21T12:35:21.037Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6885worker1 # [ 164.231029] nix-grpc-daemon[1591]: time=2026-09-21T12:35:21.039Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"6886lb # [ 164.234634] niks3-server[1023]: 2026/09/21 12:35:20 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst6887lb # [ 164.245984] niks3-server[1023]: 2026/09/21 12:35:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6888lb # [ 164.250645] niks3-server[1023]: 2026/09/21 12:35:20 INFO Signed narinfos id=62 count=16889worker1 # [ 164.269192] nix-grpc-daemon[1591]: time=2026-09-21T12:35:21.077Z level=INFO msg="Uploading 1 narinfos"6890lb # [ 164.259329] niks3-server[1023]: 2026/09/21 12:35:20 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls6891lb # [ 164.269486] niks3-server[1023]: 2026/09/21 12:35:20 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6892lb # [ 164.276075] niks3-server[1023]: 2026/09/21 12:35:20 INFO Completed upload id=626893worker1 # [ 164.291484] nix-grpc-daemon[1591]: time=2026-09-21T12:35:21.099Z level=INFO msg="Upload complete. (577ms)"6894lb # [ 164.279281] niks3-server[1023]: 2026/09/21 12:35:20 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo6895worker1 # [ 164.296068] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=0 paths=1 nar_bytes_in=7526896worker1 # [ 164.301433] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398686897worker1 # [ 164.310955] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6898worker1 # [ 164.316369] nix-daemon[852]: reaped child process 2050, status = succeeded6899worker1 # [ 164.321971] nix-daemon[852]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)6900worker1 # [ 165.336559] nix-daemon[852]: reaped child process 2064, status = succeeded6901worker1 # [ 165.540967] nix-daemon[852]: accepted connection from pid 2080, user nix-grpc-daemon (trusted)6902worker1 # [ 165.555223] nix-daemon[852]: reaped child process 2083, status = succeeded6903lb # [ 165.564861] niks3-server[1023]: 2026/09/21 12:35:22 INFO Received uploads request method=POST path=/api/pending_closures6904worker1 # [ 165.589806] nix-grpc-daemon[1591]: time=2026-09-21T12:35:22.397Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6905worker1 # [ 165.591375] nix-grpc-daemon[1591]: time=2026-09-21T12:35:22.399Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"6906lb # [ 165.611893] niks3-server[1023]: 2026/09/21 12:35:22 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst6907lb # [ 165.617797] niks3-server[1023]: 2026/09/21 12:35:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6908worker1 # [ 165.638339] nix-grpc-daemon[1591]: time=2026-09-21T12:35:22.446Z level=INFO msg="Uploading 1 narinfos"6909lb # [ 165.622175] niks3-server[1023]: 2026/09/21 12:35:22 INFO Signed narinfos id=63 count=16910lb # [ 165.626086] niks3-server[1023]: 2026/09/21 12:35:22 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv6911lb # [ 165.632547] niks3-server[1023]: 2026/09/21 12:35:22 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls6912lb # [ 165.640859] niks3-server[1023]: 2026/09/21 12:35:22 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6913lb # [ 165.645053] niks3-server[1023]: 2026/09/21 12:35:22 INFO Completed upload id=636914lb # [ 165.647604] niks3-server[1023]: 2026/09/21 12:35:22 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo6915worker1 # [ 165.662181] nix-grpc-daemon[1591]: time=2026-09-21T12:35:22.470Z level=INFO msg="Upload complete. (333ms)"6916worker1 # [ 165.668130] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39868 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=16917worker1 # [ 165.679490] nix-daemon[852]: reaped child process 2062, status = succeeded6918client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 20.54 seconds)6919worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6920worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6921client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'6922client # [ 165.841924] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6923client # Running as unit: gpu.service; invocation ID: a125b3b70f9e48ee97ec21ef4e4cbe5b6924client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.07 seconds)6925lb: must succeed: curl -sf localhost:9901/clusters6926lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6927lb: must succeed: curl -sf localhost:9901/clusters6928lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6929worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6930worker2 # [ 166.098203] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:412026931client # [ 166.072370] nix[1354]: don't know how to build these paths:6932client # [ 166.073617] nix[1354]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv6933worker2 # [ 166.106130] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:412026934worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)6935worker2 # [ 166.126026] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:23Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=06936client # [ 166.097271] nix[1354]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker6937lb: must succeed: curl -sf localhost:9901/clusters6938lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6939lb: must succeed: curl -sf localhost:9901/clusters6940lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6941worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6942worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6943client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6944lb # [ 169.261699] envoy[931]: conn peer=[2001:db8:1::1]:33800 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27171 tx=11443 ms=202466945client # [ 196.099730] nix[1354]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}6946client # [ 196.109159] nix[1354]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed6947client # [ 196.143814] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6948client # [ 196.145224] systemd[1]: gpu.service: Failed with result 'exit-code'.6949client # [ 196.147291] systemd[1]: gpu.service: Consumed 147ms CPU time over 30.305s wall clock time, 35.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.6950client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.89 seconds)6951client: must succeed: journalctl -u gpu -o cat6952client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)6953lb: must succeed: curl -sf localhost:9901/clusters6954lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6955lb: must succeed: curl -sf localhost:9901/clusters6956lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6957worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6958worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6959(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 52.29 seconds)6960subtest: low disk drains a worker and builds go to the other6961worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6962worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.98 seconds)6963worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6964worker1 # [ 199.095599] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:55Z level=info event=unhealthy reason=min_free6965lb # [ 199.263772] envoy[931]: conn peer=[2001:db8:1::1]:60078 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300566966worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.14 seconds)6967client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&26968worker1 # [ 199.832663] nix-grpc-daemon[1499]: ts=2026-09-21T12:35:56Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:398686969client # don't know how to build these paths:6970client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv6971worker2 # [ 199.860624] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:412026972worker2 # [ 199.881088] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:56Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=06973worker2 # [ 199.886481] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:56Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=16974worker2 # [ 199.924912] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:412026975worker2 # [ 199.937674] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:412026976client # copying 1 paths to 'lb:50051'...6977client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...6978worker2 # [ 199.950353] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:412026979worker2 # [ 199.961909] nix-daemon[869]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)6980worker2 # [ 199.980853] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:56Z level=info event=niks3_push_started pid=18436981worker2 # [ 200.213607] nix-grpc-daemon[1843]: time=2026-09-21T12:35:57.196Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6982worker2 # [ 200.412899] nix-daemon[869]: accepted connection from pid 1856, user nix-grpc-daemon (trusted)6983worker2 # [ 200.423673] nix-daemon[869]: reaped child process 1859, status = succeeded6984worker2 # [ 200.626686] nix-daemon[869]: accepted connection from pid 1861, user nix-grpc-daemon (trusted)6985worker2 # [ 200.638418] nix-daemon[869]: reaped child process 1864, status = succeeded6986lb # [ 200.629785] niks3-server[1023]: 2026/09/21 12:35:57 INFO Received uploads request method=POST path=/api/pending_closures6987worker2 # [ 200.670646] nix-grpc-daemon[1843]: time=2026-09-21T12:35:57.654Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6988worker2 # [ 200.672325] nix-grpc-daemon[1843]: time=2026-09-21T12:35:57.656Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"6989lb # [ 200.674202] niks3-server[1023]: 2026/09/21 12:35:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6990lb # [ 200.683292] niks3-server[1023]: 2026/09/21 12:35:57 INFO Signed narinfos id=64 count=16991worker2 # [ 200.718429] nix-grpc-daemon[1843]: time=2026-09-21T12:35:57.702Z level=INFO msg="Uploading 1 narinfos"6992lb # [ 200.688197] niks3-server[1023]: 2026/09/21 12:35:57 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls6993lb # [ 200.695591] niks3-server[1023]: 2026/09/21 12:35:57 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst6994lb # [ 200.706628] niks3-server[1023]: 2026/09/21 12:35:57 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6995lb # [ 200.711670] niks3-server[1023]: 2026/09/21 12:35:57 INFO Completed upload id=646996worker2 # [ 200.744148] nix-grpc-daemon[1843]: time=2026-09-21T12:35:57.726Z level=INFO msg="Upload complete. (525ms)"6997lb # [ 200.715796] niks3-server[1023]: 2026/09/21 12:35:57 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo6998worker2 # [ 200.746581] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41202 duration_s=0 paths=1 nar_bytes_in=5126999worker2 # [ 200.754783] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:412027000worker2 # [ 200.764510] nix-daemon[869]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)7001worker2 # [ 200.767585] nix-daemon[869]: reaped child process 1841, status = succeeded7002client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7003worker2 # [ 200.774118] nix-daemon[869]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)7004client # farm-a> worker2-next: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7005client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7006client # farm-a> LOG-farm-a-drain7007worker2 # [ 201.998103] nix-daemon[869]: reaped child process 1868, status = succeeded7008worker2 # [ 202.200195] nix-daemon[869]: accepted connection from pid 1884, user nix-grpc-daemon (trusted)7009worker2 # [ 202.212385] nix-daemon[869]: reaped child process 1887, status = succeeded7010lb # [ 202.201409] niks3-server[1023]: 2026/09/21 12:35:58 INFO Received uploads request method=POST path=/api/pending_closures7011worker2 # [ 202.243530] nix-grpc-daemon[1843]: time=2026-09-21T12:35:59.227Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7012worker2 # [ 202.245078] nix-grpc-daemon[1843]: time=2026-09-21T12:35:59.228Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7013lb # [ 202.243704] niks3-server[1023]: 2026/09/21 12:35:58 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7014lb # [ 202.258495] niks3-server[1023]: 2026/09/21 12:35:58 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7015lb # [ 202.265449] niks3-server[1023]: 2026/09/21 12:35:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7016lb # [ 202.269979] niks3-server[1023]: 2026/09/21 12:35:58 INFO Signed narinfos id=65 count=17017worker2 # [ 202.301685] nix-grpc-daemon[1843]: time=2026-09-21T12:35:59.284Z level=INFO msg="Uploading 1 narinfos"7018lb # [ 202.275424] niks3-server[1023]: 2026/09/21 12:35:58 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7019lb # [ 202.279492] niks3-server[1023]: 2026/09/21 12:35:58 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7020lb # [ 202.283325] niks3-server[1023]: 2026/09/21 12:35:58 INFO Completed upload id=657021worker2 # [ 202.314783] nix-grpc-daemon[1843]: time=2026-09-21T12:35:59.298Z level=INFO msg="Upload complete. (333ms)"7022lb # [ 202.285209] niks3-server[1023]: 2026/09/21 12:35:58 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7023worker2 # [ 202.322735] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41202 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=17024worker2 # [ 202.332563] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:59Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07025worker2 # [ 202.336510] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:59Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17026worker2 # [ 202.344788] nix-daemon[869]: reaped child process 1866, status = succeeded7027worker2 # [ 202.377936] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:412027028worker2 # [ 202.390234] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:412027029worker2 # [ 202.403126] nix-grpc-daemon[1843]: time=2026-09-21T12:35:59.386Z level=INFO msg="All 1 paths already cached"7030client # copying 1 paths to 'lb:50051'...7031client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7032worker2 # [ 202.411298] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:412027033worker2 # [ 202.419970] nix-daemon[869]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)7034worker2 # [ 202.622144] nix-daemon[869]: accepted connection from pid 1893, user nix-grpc-daemon (trusted)7035worker2 # [ 202.634455] nix-daemon[869]: reaped child process 1896, status = succeeded7036worker2 # [ 202.834718] nix-daemon[869]: accepted connection from pid 1898, user nix-grpc-daemon (trusted)7037worker2 # [ 202.847317] nix-daemon[869]: reaped child process 1901, status = succeeded7038lb # [ 202.836921] niks3-server[1023]: 2026/09/21 12:35:59 INFO Received uploads request method=POST path=/api/pending_closures7039worker2 # [ 202.876259] nix-grpc-daemon[1843]: time=2026-09-21T12:35:59.860Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7040worker2 # [ 202.878289] nix-grpc-daemon[1843]: time=2026-09-21T12:35:59.861Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7041lb # [ 202.864752] niks3-server[1023]: 2026/09/21 12:35:59 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7042lb # [ 202.876684] niks3-server[1023]: 2026/09/21 12:35:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7043worker2 # [ 202.912638] nix-grpc-daemon[1843]: time=2026-09-21T12:35:59.896Z level=INFO msg="Uploading 1 narinfos"7044lb # [ 202.881687] niks3-server[1023]: 2026/09/21 12:35:59 INFO Signed narinfos id=66 count=17045lb # [ 202.887883] niks3-server[1023]: 2026/09/21 12:35:59 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7046lb # [ 202.899135] niks3-server[1023]: 2026/09/21 12:35:59 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7047lb # [ 202.904170] niks3-server[1023]: 2026/09/21 12:35:59 INFO Completed upload id=667048worker2 # [ 202.935475] nix-grpc-daemon[1843]: time=2026-09-21T12:35:59.919Z level=INFO msg="Upload complete. (502ms)"7049lb # [ 202.907050] niks3-server[1023]: 2026/09/21 12:35:59 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7050worker2 # [ 202.940123] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41202 duration_s=0 paths=1 nar_bytes_in=6247051worker2 # [ 202.944857] nix-grpc-daemon[1753]: ts=2026-09-21T12:35:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:412027052worker2 # [ 202.955403] nix-daemon[869]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)7053worker2 # [ 202.957910] nix-daemon[869]: reaped child process 1891, status = succeeded7054client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7055worker2 # [ 202.964166] nix-daemon[869]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)7056client # farm-b> worker2-next: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7057client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7058client # farm-b> LOG-farm-b-drain7059worker2 # [ 204.376364] nix-daemon[869]: reaped child process 1905, status = succeeded7060worker2 # [ 204.557397] nix-daemon[869]: accepted connection from pid 1921, user nix-grpc-daemon (trusted)7061worker2 # [ 204.569629] nix-daemon[869]: reaped child process 1924, status = succeeded7062lb # [ 204.559416] niks3-server[1023]: 2026/09/21 12:36:01 INFO Received uploads request method=POST path=/api/pending_closures7063worker2 # [ 204.599822] nix-grpc-daemon[1843]: time=2026-09-21T12:36:01.583Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7064worker2 # [ 204.602056] nix-grpc-daemon[1843]: time=2026-09-21T12:36:01.585Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7065lb # [ 204.590609] niks3-server[1023]: 2026/09/21 12:36:01 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7066lb # [ 204.605223] niks3-server[1023]: 2026/09/21 12:36:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7067lb # [ 204.610505] niks3-server[1023]: 2026/09/21 12:36:01 INFO Signed narinfos id=67 count=17068worker2 # [ 204.644936] nix-grpc-daemon[1843]: time=2026-09-21T12:36:01.628Z level=INFO msg="Uploading 1 narinfos"7069lb # [ 204.618772] niks3-server[1023]: 2026/09/21 12:36:01 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7070lb # [ 204.624735] niks3-server[1023]: 2026/09/21 12:36:01 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7071lb # [ 204.632400] niks3-server[1023]: 2026/09/21 12:36:01 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7072lb # [ 204.637734] niks3-server[1023]: 2026/09/21 12:36:01 INFO Completed upload id=677073worker2 # [ 204.669285] nix-grpc-daemon[1843]: time=2026-09-21T12:36:01.653Z level=INFO msg="Upload complete. (298ms)"7074lb # [ 204.641094] niks3-server[1023]: 2026/09/21 12:36:01 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7075worker2 # [ 204.676494] nix-grpc-daemon[1753]: ts=2026-09-21T12:36:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41202 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=17076worker2 # [ 204.687872] nix-grpc-daemon[1753]: ts=2026-09-21T12:36:01Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07077worker2 # [ 204.692227] nix-grpc-daemon[1753]: ts=2026-09-21T12:36:01Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17078worker2 # [ 204.699858] nix-daemon[869]: reaped child process 1903, status = succeeded7079worker2 # [ 204.731503] nix-grpc-daemon[1753]: ts=2026-09-21T12:36:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:412027080worker2 # [ 204.744132] nix-grpc-daemon[1753]: ts=2026-09-21T12:36:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:412027081worker2 # [ 204.761922] nix-grpc-daemon[1843]: time=2026-09-21T12:36:01.745Z level=INFO msg="All 2 paths already cached"7082client # copying 1 paths to 'lb:50051'...7083client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7084worker2 # [ 204.770529] nix-grpc-daemon[1753]: ts=2026-09-21T12:36:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:412027085worker2 # [ 204.779278] nix-daemon[869]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)7086worker2 # [ 204.982322] nix-daemon[869]: accepted connection from pid 1930, user nix-grpc-daemon (trusted)7087worker2 # [ 204.994874] nix-daemon[869]: reaped child process 1933, status = succeeded7088worker2 # [ 205.185765] nix-daemon[869]: accepted connection from pid 1935, user nix-grpc-daemon (trusted)7089worker2 # [ 205.197623] nix-daemon[869]: reaped child process 1938, status = succeeded7090lb # [ 205.185908] niks3-server[1023]: 2026/09/21 12:36:01 INFO Received uploads request method=POST path=/api/pending_closures7091worker2 # [ 205.225683] nix-grpc-daemon[1843]: time=2026-09-21T12:36:02.209Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7092worker2 # [ 205.227080] nix-grpc-daemon[1843]: time=2026-09-21T12:36:02.209Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7093lb # [ 205.210471] niks3-server[1023]: 2026/09/21 12:36:01 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7094lb # [ 205.223062] niks3-server[1023]: 2026/09/21 12:36:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7095worker2 # [ 205.258599] nix-grpc-daemon[1843]: time=2026-09-21T12:36:02.242Z level=INFO msg="Uploading 1 narinfos"7096lb # [ 205.227260] niks3-server[1023]: 2026/09/21 12:36:01 INFO Signed narinfos id=68 count=17097lb # [ 205.234267] niks3-server[1023]: 2026/09/21 12:36:01 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7098lb # [ 205.243902] niks3-server[1023]: 2026/09/21 12:36:01 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7099lb # [ 205.250355] niks3-server[1023]: 2026/09/21 12:36:01 INFO Completed upload id=687100worker2 # [ 205.281811] nix-grpc-daemon[1843]: time=2026-09-21T12:36:02.265Z level=INFO msg="Upload complete. (488ms)"7101worker2 # [ 205.284162] nix-grpc-daemon[1753]: ts=2026-09-21T12:36:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41202 duration_s=0 paths=1 nar_bytes_in=7527102lb # [ 205.253896] niks3-server[1023]: 2026/09/21 12:36:01 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7103worker2 # [ 205.288959] nix-grpc-daemon[1753]: ts=2026-09-21T12:36:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:412027104worker2 # [ 205.294677] nix-daemon[869]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)7105worker2 # [ 205.296927] nix-daemon[869]: reaped child process 1928, status = succeeded7106worker2 # [ 205.302416] nix-daemon[869]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)7107client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7108client # farm-top> worker2-next: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7109client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7110client # farm-top> LOG-farm-top-drain7111worker2 # [ 206.115119] nix-daemon[869]: reaped child process 1942, status = succeeded7112worker2 # [ 206.289500] nix-daemon[869]: accepted connection from pid 1958, user nix-grpc-daemon (trusted)7113worker2 # [ 206.301783] nix-daemon[869]: reaped child process 1961, status = succeeded7114lb # [ 206.290882] niks3-server[1023]: 2026/09/21 12:36:02 INFO Received uploads request method=POST path=/api/pending_closures7115worker2 # [ 206.330951] nix-grpc-daemon[1843]: time=2026-09-21T12:36:03.314Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7116worker2 # [ 206.332605] nix-grpc-daemon[1843]: time=2026-09-21T12:36:03.316Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7117lb # [ 206.322442] niks3-server[1023]: 2026/09/21 12:36:02 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7118lb # [ 206.335071] niks3-server[1023]: 2026/09/21 12:36:03 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7119lb # [ 206.346559] niks3-server[1023]: 2026/09/21 12:36:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7120worker2 # [ 206.382976] nix-grpc-daemon[1843]: time=2026-09-21T12:36:03.366Z level=INFO msg="Uploading 1 narinfos"7121lb # [ 206.351539] niks3-server[1023]: 2026/09/21 12:36:03 INFO Signed narinfos id=69 count=17122lb # [ 206.357922] niks3-server[1023]: 2026/09/21 12:36:03 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7123lb # [ 206.367224] niks3-server[1023]: 2026/09/21 12:36:03 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7124lb # [ 206.373271] niks3-server[1023]: 2026/09/21 12:36:03 INFO Completed upload id=697125worker2 # [ 206.404617] nix-grpc-daemon[1843]: time=2026-09-21T12:36:03.388Z level=INFO msg="Upload complete. (296ms)"7126lb # [ 206.375082] niks3-server[1023]: 2026/09/21 12:36:03 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7127worker2 # [ 206.410860] nix-grpc-daemon[1753]: ts=2026-09-21T12:36:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41202 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=17128worker2 # [ 206.425088] nix-daemon[869]: reaped child process 1940, status = succeeded7129client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 6.86 seconds)7130client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7131client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.23 seconds)7132worker1: must succeed: rm /nix/.rw-store/fill7133worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.14 seconds)7134worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7135worker1 # [ 207.099954] nix-grpc-daemon[1499]: ts=2026-09-21T12:36:03Z level=info event=healthy reason=min_free7136worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.14 seconds)7137(finished: subtest: low disk drains a worker and builds go to the other, in 10.59 seconds)7138(finished: run the VM test script, in 208.99 seconds)7139test script finished in 209.27s7140cleanup7141kill QemuMachine (pid 49)7142client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7143kill QemuMachine (pid 50)7144lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7145kill QemuMachine (pid 51)7146worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7147kill QemuMachine (pid 52)7148worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7149(finished: cleanup, in 0.99 seconds)7150additionally exposed symbols:7151 client, lb, worker1, worker2,7152 vlan1,7153 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh