vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #497
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker2: QEMU running (pid 52)13worker2 # Disk image does not exist, creating the virtualisation disk image...14worker2 # Formatting '/build/vm-state-worker2/tmp.WcUnTSO9dj', fmt=raw size=107374182415worker2 # mke2fs 1.47.4 (6-Mar-2025)16worker2 # Discarding device blocks: 0/262144 done17worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes18worker2 # Filesystem UUID: de790940-3e3b-4282-a820-89c8def9c95919worker2 # Superblock backups stored on blocks:20worker2 # 32768, 98304, 163840, 22937621worker2 # 22worker2 # Allocating group tables: 0/8 done23worker2 # Writing inode tables: 0/8 done24worker2 # Creating journal (8192 blocks): done25worker2 # Writing superblocks and filesystem accounting information: 0/8 done26worker2 # 27worker2 # Virtualisation disk image created.28worker2 # Starting virtiofs daemons...29worker2 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)30worker2 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether31worker2 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)32worker2 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether33worker2 # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...34lb # Disk image does not exist, creating the virtualisation disk image...35worker2 # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...36lb # Formatting '/build/vm-state-lb/tmp.49aCVaTM3Z', fmt=raw size=107374182437worker2 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)38lb # mke2fs 1.47.4 (6-Mar-2025)39worker2 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether40lb # Discarding device blocks: 0/262144 done41worker2 # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...42lb # Creating filesystem with 262144 4k blocks and 65536 inodes43worker2 # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests44lb # Filesystem UUID: 2889e468-078f-4b61-be2f-08828f93377345lb # Superblock backups stored on blocks:46worker1: QEMU running (pid 51)47worker1 # Disk image does not exist, creating the virtualisation disk image...48lb # 32768, 98304, 163840, 22937649worker2 # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests50lb: QEMU running (pid 50)51worker1 # Formatting '/build/vm-state-worker1/tmp.53KL03rafn', fmt=raw size=107374182452lb # 53worker1 # mke2fs 1.47.4 (6-Mar-2025)54lb # Allocating group tables: 0/8 done55worker1 # Discarding device blocks: 0/262144 done56lb # Writing inode tables: 0/8 done57worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes58lb # Creating journal (8192 blocks): done59worker1 # Filesystem UUID: c08cad62-9032-4186-bfc3-8fdaf7b86aaa60lb # Writing superblocks and filesystem accounting information: 0/8 done61worker1 # Superblock backups stored on blocks:62lb # 63client # Disk image does not exist, creating the virtualisation disk image...64worker1 # 32768, 98304, 163840, 22937665lb # Virtualisation disk image created.66client # Formatting '/build/vm-state-client/tmp.4B85MuuL2O', fmt=raw size=107374182467worker1 # 68lb # Starting virtiofs daemons...69client # mke2fs 1.47.4 (6-Mar-2025)70worker1 # Allocating group tables: 0/8 done71lb # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)72client # Discarding device blocks: 0/262144 done73worker1 # Writing inode tables: 0/8 done74lb # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether75client # Creating filesystem with 262144 4k blocks and 65536 inodes76worker1 # Creating journal (8192 blocks): done77lb # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...78client # Filesystem UUID: 2e6968cc-8085-486f-a88e-8c11acb3693979worker1 # Writing superblocks and filesystem accounting information: 0/8 done80lb # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)81client # Superblock backups stored on blocks:82worker1 # 83lb # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether84client # 32768, 98304, 163840, 22937685worker1 # Virtualisation disk image created.86lb # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...87client # 88worker1 # Starting virtiofs daemons...89lb # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)90client # Allocating group tables: 0/8 done91worker1 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)92lb # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether93client # Writing inode tables: 0/8 done94worker1 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether95lb # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...96client # Creating journal (8192 blocks): done97worker1 # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...98lb # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests99client # Writing superblocks and filesystem accounting information: 0/8 done100worker1 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)101lb # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests102client # 103worker1 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether104lb # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests105client # Virtualisation disk image created.106worker1 # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...107client: QEMU running (pid 49)108worker2 # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests109client # Starting virtiofs daemons...110worker1 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)111client # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)112worker1 # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether113client # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether114worker1 # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...115client # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...116worker1 # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests117client # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)118worker1 # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests119client # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether120worker1 # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests121client # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...122(finished: start all VMs, in 0.23 seconds)123client # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)124client: waiting for unit network-addresses-eth1.service125client # [2026-09-22T11:32:18Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether126client: waiting for the VM to finish booting127client # [2026-09-22T11:32:18Z INFO virtiofsd] Waiting for vhost-user socket connection...128client # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests129client # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests130client # [2026-09-22T11:32:18Z INFO virtiofsd] Client connected, servicing requests131worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)132worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)133client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)134lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)135worker2 # 136worker2 # 137worker1 # 138worker1 # 139worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00140worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00141worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...142worker2 # 143client # 144worker2 # 145client # 146worker2 # 147worker2 # 148worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...149worker1 # 150worker1 # 151worker1 # 152worker1 # 153client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00154worker2 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00155worker1 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00156client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...157client # 158client # 159client # 160client # 161worker2 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...162worker2 # 163worker2 # 164client # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00165worker1 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...166worker1 # 167worker1 # 168client # Press Ctrl-B to configure iPXE (PCI 00:05.0)...169client # 170client # 171lb # 172lb # 173lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00174lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...175lb # 176lb # 177lb # 178lb # 179lb # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00180lb # Press Ctrl-B to configure iPXE (PCI 00:05.0)...181lb # 182lb # 183worker1 # Booting from ROM...184worker2 # Booting from ROM...185client # Booting from ROM...186lb # Booting from ROM...187worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026188worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/9l48jkcr6cdiqp8y6v9sxvagdw5w59l4-nixos-system-worker1-test/init regInfo=/nix/store/jjgcx7snicxga06ahrah0z8a6l3fgmhf-closure-info/registration console=ttyS0,115200n8 console=tty0189worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026190worker1 # [ 0.000000] BIOS-provided physical RAM map:191worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable192worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved193worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved194worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable195worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/2r960cj2h4zwi2h592n0agq168cha874-nixos-system-worker2-test/init regInfo=/nix/store/7g0xk07bpn0vi477049b6nn9r1qix82k-closure-info/registration console=ttyS0,115200n8 console=tty0196worker2 # [ 0.000000] BIOS-provided physical RAM map:197client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026198worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved199worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable200worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved201worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved202worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved203worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved204worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved205worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable206worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved207client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/6zdlf97nxsya8j2z50jj0v2glbqyi1nw-nixos-system-client-test/init regInfo=/nix/store/iayv2xqplqy5pxgnqa549zwg9kilmzp9-closure-info/registration console=ttyS0,115200n8 console=tty0208worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved209client # [ 0.000000] BIOS-provided physical RAM map:210worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved211worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved212client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable213worker1 # [ 0.000000] NX (Execute Disable) protection: active214worker1 # [ 0.000000] APIC: Static calls initialized215worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved216client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved217worker1 # [ 0.000000] SMBIOS 2.8 present.218worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved219client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved220worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014221client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable222worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved223worker1 # [ 0.000000] DMI: Memory slots populated: 1/1224worker1 # [ 0.000000] Hypervisor detected: KVM225worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved226client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved227worker2 # [ 0.000000] NX (Execute Disable) protection: active228worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000229client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved230worker2 # [ 0.000000] APIC: Static calls initialized231worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00232worker2 # [ 0.000000] SMBIOS 2.8 present.233client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved234worker1 # [ 0.000000] kvm-clock: using sched offset of 476237589 cycles235client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved236worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014237worker2 # [ 0.000000] DMI: Memory slots populated: 1/1238worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns239client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved240worker2 # [ 0.000000] Hypervisor detected: KVM241worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor242client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved243worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000244worker1 # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000245client # [ 0.000000] NX (Execute Disable) protection: active246worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00247client # [ 0.000000] APIC: Static calls initialized248worker1 # [ 0.000840] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs249client # [ 0.000000] SMBIOS 2.8 present.250worker2 # [ 0.000001] kvm-clock: using sched offset of 473636102 cycles251worker1 # [ 0.000842] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT252client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014253worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns254worker1 # [ 0.002727] found SMP MP-table at [mem 0x000f5450-0x000f545f]255client # [ 0.000000] DMI: Memory slots populated: 1/1256worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor257worker1 # [ 0.002739] Using GB pages for direct mapping258client # [ 0.000000] Hypervisor detected: KVM259worker2 # [ 0.000813] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000260worker1 # [ 0.002866] RAMDISK: [mem 0x5e370000-0x5ffcffff]261client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000262worker1 # [ 0.002875] ACPI: Early table checksum verification disabled263client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00264worker2 # [ 0.000840] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs265worker1 # [ 0.002877] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )266client # [ 0.000000] kvm-clock: using sched offset of 462979805 cycles267worker2 # [ 0.000843] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT268worker1 # [ 0.002881] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)269worker2 # [ 0.002729] found SMP MP-table at [mem 0x000f5450-0x000f545f]270client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns271worker2 # [ 0.002740] Using GB pages for direct mapping272client # [ 0.000006] tsc: Detected 2400.012 MHz processor273worker1 # [ 0.002886] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)274worker2 # [ 0.002876] RAMDISK: [mem 0x5e36f000-0x5ffcffff]275client # [ 0.000816] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000276worker2 # [ 0.002884] ACPI: Early table checksum verification disabled277worker1 # [ 0.002892] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)278worker2 # [ 0.002887] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )279client # [ 0.000849] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs280worker1 # [ 0.002894] ACPI: FACS 0x000000005FFE0000 000040281client # [ 0.000852] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT282worker2 # [ 0.002891] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)283worker1 # [ 0.002895] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)284client # [ 0.002741] found SMP MP-table at [mem 0x000f5450-0x000f545f]285client # [ 0.002753] Using GB pages for direct mapping286worker2 # [ 0.002895] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)287worker1 # [ 0.002897] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)288client # [ 0.002873] RAMDISK: [mem 0x5e370000-0x5ffcffff]289worker2 # [ 0.002902] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)290worker1 # [ 0.002898] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)291client # [ 0.002882] ACPI: Early table checksum verification disabled292lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026293worker2 # [ 0.002904] ACPI: FACS 0x000000005FFE0000 000040294client # [ 0.002885] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )295worker1 # [ 0.002900] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)296worker2 # [ 0.002906] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)297client # [ 0.002890] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)298worker1 # [ 0.002901] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]299worker2 # [ 0.002907] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)300worker1 # [ 0.002902] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]301client # [ 0.002895] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)302worker2 # [ 0.002909] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)303worker1 # [ 0.002903] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]304client # [ 0.002902] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)305lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qfr7fghmwm537hb3f4f5is3yb8phm6jx-nixos-system-lb-test/init regInfo=/nix/store/s1480xaqc00gh7hz31b4z2fwjpnxi3sa-closure-info/registration console=ttyS0,115200n8 console=tty0306worker2 # [ 0.002910] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)307client # [ 0.002904] ACPI: FACS 0x000000005FFE0000 000040308lb # [ 0.000000] BIOS-provided physical RAM map:309worker1 # [ 0.002903] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]310worker2 # [ 0.002911] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]311worker1 # [ 0.002904] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]312lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable313client # [ 0.002905] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)314worker2 # [ 0.002912] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]315worker1 # [ 0.002904] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]316lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved317client # [ 0.002907] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)318worker2 # [ 0.002913] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]319worker1 # [ 0.002905] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]320lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved321client # [ 0.002908] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)322worker1 # [ 0.003129] No NUMA configuration found323worker2 # [ 0.002913] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]324lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable325worker1 # [ 0.003129] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]326client # [ 0.002910] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)327worker2 # [ 0.002914] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]328lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved329worker1 # [ 0.003132] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]330worker1 # [ 0.005410] Zone ranges:331client # [ 0.002911] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]332worker2 # [ 0.002914] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]333lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved334client # [ 0.002912] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]335worker1 # [ 0.005411] DMA [mem 0x0000000000001000-0x0000000000ffffff]336worker2 # [ 0.002915] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]337lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved338worker2 # [ 0.003141] No NUMA configuration found339worker1 # [ 0.005412] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]340client # [ 0.002913] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]341worker1 # [ 0.005413] Normal empty342lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved343worker1 # [ 0.005414] Device empty344worker2 # [ 0.003142] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]345client # [ 0.002913] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]346lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved347worker1 # [ 0.005414] Movable zone start for each node348worker2 # [ 0.003144] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]349worker1 # [ 0.005415] Early memory node ranges350worker2 # [ 0.005365] Zone ranges:351client # [ 0.002914] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]352lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved353worker1 # [ 0.005415] node 0: [mem 0x0000000000001000-0x000000000009efff]354worker2 # [ 0.005366] DMA [mem 0x0000000000001000-0x0000000000ffffff]355client # [ 0.002914] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]356lb # [ 0.000000] NX (Execute Disable) protection: active357lb # [ 0.000000] APIC: Static calls initialized358worker2 # [ 0.005368] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]359worker1 # [ 0.005416] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]360client # [ 0.002915] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]361worker2 # [ 0.005369] Normal empty362lb # [ 0.000000] SMBIOS 2.8 present.363client # [ 0.003143] No NUMA configuration found364worker2 # [ 0.005370] Device empty365worker1 # [ 0.005418] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]366worker2 # [ 0.005370] Movable zone start for each node367worker1 # [ 0.005437] On node 0, zone DMA: 1 pages in unavailable ranges368lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014369client # [ 0.003144] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]370worker2 # [ 0.005371] Early memory node ranges371lb # [ 0.000000] DMI: Memory slots populated: 1/1372worker1 # [ 0.005707] On node 0, zone DMA: 97 pages in unavailable ranges373client # [ 0.003147] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]374worker2 # [ 0.005372] node 0: [mem 0x0000000000001000-0x000000000009efff]375client # [ 0.005357] Zone ranges:376lb # [ 0.000000] Hypervisor detected: KVM377worker1 # [ 0.032774] On node 0, zone DMA32: 40 pages in unavailable ranges378worker2 # [ 0.005373] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]379lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000380client # [ 0.005359] DMA [mem 0x0000000000001000-0x0000000000ffffff]381worker1 # [ 0.033227] ACPI: PM-Timer IO Port: 0x608382lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00383worker2 # [ 0.005374] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]384client # [ 0.005360] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]385worker1 # [ 0.033239] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])386client # [ 0.005361] Normal empty387lb # [ 0.000000] kvm-clock: using sched offset of 492142319 cycles388worker2 # [ 0.005393] On node 0, zone DMA: 1 pages in unavailable ranges389client # [ 0.005362] Device empty390worker1 # [ 0.033265] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23391client # [ 0.005363] Movable zone start for each node392worker2 # [ 0.005652] On node 0, zone DMA: 97 pages in unavailable ranges393lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns394client # [ 0.005364] Early memory node ranges395worker1 # [ 0.033268] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)396worker2 # [ 0.031950] On node 0, zone DMA32: 40 pages in unavailable ranges397lb # [ 0.000005] tsc: Detected 2400.012 MHz processor398client # [ 0.005364] node 0: [mem 0x0000000000001000-0x000000000009efff]399worker1 # [ 0.033269] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)400worker2 # [ 0.032416] ACPI: PM-Timer IO Port: 0x608401lb # [ 0.000814] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000402client # [ 0.005365] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]403worker2 # [ 0.032427] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])404worker1 # [ 0.033271] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)405lb # [ 0.000842] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs406client # [ 0.005366] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]407worker2 # [ 0.032457] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23408worker1 # [ 0.033272] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)409client # [ 0.005386] On node 0, zone DMA: 1 pages in unavailable ranges410lb # [ 0.000844] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT411worker2 # [ 0.032460] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)412worker1 # [ 0.033272] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)413client # [ 0.005650] On node 0, zone DMA: 97 pages in unavailable ranges414lb # [ 0.002734] found SMP MP-table at [mem 0x000f5450-0x000f545f]415worker2 # [ 0.032461] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)416worker1 # [ 0.033275] ACPI: Using ACPI (MADT) for SMP configuration information417lb # [ 0.002745] Using GB pages for direct mapping418client # [ 0.031943] On node 0, zone DMA32: 40 pages in unavailable ranges419worker1 # [ 0.033276] ACPI: HPET id: 0x8086a201 base: 0xfed00000420worker2 # [ 0.032463] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)421lb # [ 0.002848] RAMDISK: [mem 0x5e36e000-0x5ffcffff]422client # [ 0.032414] ACPI: PM-Timer IO Port: 0x608423worker1 # [ 0.033279] TSC deadline timer available424client # [ 0.032426] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])425lb # [ 0.002857] ACPI: Early table checksum verification disabled426worker2 # [ 0.032464] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)427worker1 # [ 0.033283] CPU topo: Max. logical packages: 1428lb # [ 0.002860] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )429worker1 # [ 0.033284] CPU topo: Max. logical dies: 1430client # [ 0.032456] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23431worker2 # [ 0.032464] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)432worker1 # [ 0.033284] CPU topo: Max. dies per package: 1433worker2 # [ 0.032467] ACPI: Using ACPI (MADT) for SMP configuration information434client # [ 0.032459] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)435lb # [ 0.002864] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)436worker1 # [ 0.033288] CPU topo: Max. threads per core: 1437worker2 # [ 0.032468] ACPI: HPET id: 0x8086a201 base: 0xfed00000438client # [ 0.032461] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)439worker1 # [ 0.033288] CPU topo: Num. cores per package: 1440lb # [ 0.002867] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)441worker2 # [ 0.032471] TSC deadline timer available442worker1 # [ 0.033289] CPU topo: Num. threads per package: 1443client # [ 0.032462] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)444worker2 # [ 0.032475] CPU topo: Max. logical packages: 1445lb # [ 0.002875] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)446worker2 # [ 0.032476] CPU topo: Max. logical dies: 1447worker1 # [ 0.033289] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs448client # [ 0.032463] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)449lb # [ 0.002877] ACPI: FACS 0x000000005FFE0000 000040450worker2 # [ 0.032476] CPU topo: Max. dies per package: 1451worker1 # [ 0.033305] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()452client # [ 0.032464] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)453worker2 # [ 0.032480] CPU topo: Max. threads per core: 1454lb # [ 0.002878] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)455worker1 # [ 0.033333] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]456worker2 # [ 0.032480] CPU topo: Num. cores per package: 1457client # [ 0.032466] ACPI: Using ACPI (MADT) for SMP configuration information458worker2 # [ 0.032481] CPU topo: Num. threads per package: 1459client # [ 0.032467] ACPI: HPET id: 0x8086a201 base: 0xfed00000460lb # [ 0.002880] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)461worker1 # [ 0.033335] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]462client # [ 0.032472] TSC deadline timer available463worker2 # [ 0.032481] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs464worker1 # [ 0.033336] [mem 0x60000000-0xafffffff] available for PCI devices465client # [ 0.032476] CPU topo: Max. logical packages: 1466lb # [ 0.002881] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)467worker2 # [ 0.032498] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()468worker1 # [ 0.033338] Booting paravirtualized kernel on KVM469client # [ 0.032476] CPU topo: Max. logical dies: 1470client # [ 0.032477] CPU topo: Max. dies per package: 1471lb # [ 0.002883] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)472worker2 # [ 0.032530] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]473client # [ 0.032480] CPU topo: Max. threads per core: 1474worker1 # [ 0.033340] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns475lb # [ 0.002884] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]476client # [ 0.032481] CPU topo: Num. cores per package: 1477worker2 # [ 0.032532] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]478worker1 # [ 0.037821] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1479client # [ 0.032481] CPU topo: Num. threads per package: 1480lb # [ 0.002885] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]481worker2 # [ 0.032533] [mem 0x60000000-0xafffffff] available for PCI devices482worker1 # [ 0.040305] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152483client # [ 0.032481] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs484worker2 # [ 0.032535] Booting paravirtualized kernel on KVM485lb # [ 0.002885] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]486worker1 # [ 0.040349] kvm-guest: PV spinlocks disabled, single CPU487client # [ 0.032502] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()488lb # [ 0.002886] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]489worker2 # [ 0.032538] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns490client # [ 0.032539] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]491lb # [ 0.002886] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]492worker2 # [ 0.037022] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1493client # [ 0.032541] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]494lb # [ 0.002887] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]495worker2 # [ 0.039157] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152496client # [ 0.032542] [mem 0x60000000-0xafffffff] available for PCI devices497worker2 # [ 0.039203] kvm-guest: PV spinlocks disabled, single CPU498lb # [ 0.002887] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]499client # [ 0.032544] Booting paravirtualized kernel on KVM500lb # [ 0.003127] No NUMA configuration found501worker1 # [ 0.040350] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/9l48jkcr6cdiqp8y6v9sxvagdw5w59l4-nixos-system-worker1-test/init regInfo=/nix/store/jjgcx7snicxga06ahrah0z8a6l3fgmhf-closure-info/registration console=ttyS0,115200n8 console=tty0502lb # [ 0.003127] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]503client # [ 0.032547] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns504lb # [ 0.003130] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]505lb # [ 0.005374] Zone ranges:506client # [ 0.037024] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1507worker1 # [ 0.040442] Unknown kernel command line parameters "regInfo=/nix/store/jjgcx7snicxga06ahrah0z8a6l3fgmhf-closure-info/registration", will be passed to user space.508worker1 # [ 0.040454] random: crng init done509lb # [ 0.005375] DMA [mem 0x0000000000001000-0x0000000000ffffff]510client # [ 0.039191] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152511worker2 # [ 0.039205] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/2r960cj2h4zwi2h592n0agq168cha874-nixos-system-worker2-test/init regInfo=/nix/store/7g0xk07bpn0vi477049b6nn9r1qix82k-closure-info/registration console=ttyS0,115200n8 console=tty0512lb # [ 0.005376] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]513client # [ 0.039238] kvm-guest: PV spinlocks disabled, single CPU514worker1 # [ 0.040455] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes515lb # [ 0.005377] Normal empty516lb # [ 0.005378] Device empty517worker1 # [ 0.042624] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)518lb # [ 0.005379] Movable zone start for each node519worker2 # [ 0.039300] Unknown kernel command line parameters "regInfo=/nix/store/7g0xk07bpn0vi477049b6nn9r1qix82k-closure-info/registration", will be passed to user space.520lb # [ 0.005380] Early memory node ranges521worker1 # [ 0.043706] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)522worker2 # [ 0.039320] random: crng init done523worker1 # [ 0.043743] Fallback order for Node 0: 0524lb # [ 0.005380] node 0: [mem 0x0000000000001000-0x000000000009efff]525worker2 # [ 0.039320] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes526lb # [ 0.005381] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]527worker1 # [ 0.043746] Built 1 zonelists, mobility grouping on. Total pages: 393078528worker1 # [ 0.043747] Policy zone: DMA32529client # [ 0.039240] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/6zdlf97nxsya8j2z50jj0v2glbqyi1nw-nixos-system-client-test/init regInfo=/nix/store/iayv2xqplqy5pxgnqa549zwg9kilmzp9-closure-info/registration console=ttyS0,115200n8 console=tty0530worker2 # [ 0.041443] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)531lb # [ 0.005382] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]532worker1 # [ 0.046258] mem auto-init: stack:all(zero), heap alloc:on, heap free:off533lb # [ 0.005403] On node 0, zone DMA: 1 pages in unavailable ranges534worker2 # [ 0.042676] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)535worker1 # [ 0.049341] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1536worker2 # [ 0.042708] Fallback order for Node 0: 0537lb # [ 0.005662] On node 0, zone DMA: 97 pages in unavailable ranges538client # [ 0.039345] Unknown kernel command line parameters "regInfo=/nix/store/iayv2xqplqy5pxgnqa549zwg9kilmzp9-closure-info/registration", will be passed to user space.539worker1 # [ 0.053021] allocated 3145728 bytes of page_ext540client # [ 0.039357] random: crng init done541worker2 # [ 0.042711] Built 1 zonelists, mobility grouping on. Total pages: 393078542lb # [ 0.031940] On node 0, zone DMA32: 40 pages in unavailable ranges543worker1 # [ 0.062727] ftrace: allocating 48787 entries in 192 pages544worker2 # [ 0.042712] Policy zone: DMA32545lb # [ 0.032404] ACPI: PM-Timer IO Port: 0x608546client # [ 0.039358] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes547worker1 # [ 0.062728] ftrace: allocated 192 pages with 2 groups548worker2 # [ 0.045189] mem auto-init: stack:all(zero), heap alloc:on, heap free:off549worker1 # [ 0.063562] Dynamic Preempt: lazy550lb # [ 0.032416] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])551client # [ 0.041477] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)552worker1 # [ 0.063735] rcu: Preemptible hierarchical RCU implementation.553worker2 # [ 0.048428] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1554lb # [ 0.032445] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23555worker1 # [ 0.063736] rcu: RCU event tracing is enabled.556worker2 # [ 0.051734] allocated 3145728 bytes of page_ext557client # [ 0.042723] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)558lb # [ 0.032448] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)559worker2 # [ 0.061446] ftrace: allocating 48787 entries in 192 pages560client # [ 0.042755] Fallback order for Node 0: 0561worker1 # [ 0.063737] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.562lb # [ 0.032450] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)563worker2 # [ 0.061448] ftrace: allocated 192 pages with 2 groups564client # [ 0.042758] Built 1 zonelists, mobility grouping on. Total pages: 393078565worker1 # [ 0.063738] Trampoline variant of Tasks RCU enabled.566worker2 # [ 0.062285] Dynamic Preempt: lazy567client # [ 0.042760] Policy zone: DMA32568worker1 # [ 0.063739] Rude variant of Tasks RCU enabled.569lb # [ 0.032451] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)570worker2 # [ 0.062419] rcu: Preemptible hierarchical RCU implementation.571client # [ 0.045263] mem auto-init: stack:all(zero), heap alloc:on, heap free:off572worker1 # [ 0.063739] Tracing variant of Tasks RCU enabled.573lb # [ 0.032452] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)574worker2 # [ 0.062419] rcu: RCU event tracing is enabled.575client # [ 0.048529] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1576worker1 # [ 0.063739] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.577lb # [ 0.032452] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)578worker2 # [ 0.062420] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.579client # [ 0.051832] allocated 3145728 bytes of page_ext580worker2 # [ 0.062422] Trampoline variant of Tasks RCU enabled.581lb # [ 0.032455] ACPI: Using ACPI (MADT) for SMP configuration information582worker1 # [ 0.063740] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1583client # [ 0.061651] ftrace: allocating 48787 entries in 192 pages584worker2 # [ 0.062422] Rude variant of Tasks RCU enabled.585lb # [ 0.032456] ACPI: HPET id: 0x8086a201 base: 0xfed00000586client # [ 0.061652] ftrace: allocated 192 pages with 2 groups587worker1 # [ 0.063762] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.588worker2 # [ 0.062422] Tracing variant of Tasks RCU enabled.589lb # [ 0.032459] TSC deadline timer available590client # [ 0.062498] Dynamic Preempt: lazy591lb # [ 0.032463] CPU topo: Max. logical packages: 1592client # [ 0.062636] rcu: Preemptible hierarchical RCU implementation.593worker2 # [ 0.062423] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.594worker1 # [ 0.063763] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.595lb # [ 0.032464] CPU topo: Max. logical dies: 1596client # [ 0.062637] rcu: RCU event tracing is enabled.597lb # [ 0.032464] CPU topo: Max. dies per package: 1598worker2 # [ 0.062424] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1599client # [ 0.062637] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.600worker1 # [ 0.063764] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.601lb # [ 0.032467] CPU topo: Max. threads per core: 1602worker2 # [ 0.062443] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.603client # [ 0.062639] Trampoline variant of Tasks RCU enabled.604lb # [ 0.032468] CPU topo: Num. cores per package: 1605worker1 # [ 0.068180] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16606lb # [ 0.032468] CPU topo: Num. threads per package: 1607client # [ 0.062640] Rude variant of Tasks RCU enabled.608worker1 # [ 0.068448] rcu: srcu_init: Setting srcu_struct sizes based on contention.609worker2 # [ 0.062445] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.610client # [ 0.062640] Tracing variant of Tasks RCU enabled.611lb # [ 0.032469] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs612client # [ 0.062641] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.613worker2 # [ 0.062446] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.614lb # [ 0.032487] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()615worker1 # [ 0.068455] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns616client # [ 0.062641] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1617worker2 # [ 0.067039] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16618lb # [ 0.032521] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]619worker1 # [ 0.068565] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)620worker2 # [ 0.067357] rcu: srcu_init: Setting srcu_struct sizes based on contention.621worker1 # [ 0.072085] Console: colour VGA+ 80x25622client # [ 0.062656] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.623lb # [ 0.032523] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]624worker1 # [ 0.072088] printk: legacy console [tty0] enabled625lb # [ 0.032524] [mem 0x60000000-0xafffffff] available for PCI devices626worker2 # [ 0.067365] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns627client # [ 0.062658] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.628worker1 # [ 0.101591] printk: legacy console [ttyS0] enabled629lb # [ 0.032526] Booting paravirtualized kernel on KVM630worker1 # [ 0.198634] ACPI: Core revision 20250807631worker2 # [ 0.067470] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)632client # [ 0.062659] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.633worker2 # [ 0.070998] Console: colour VGA+ 80x25634lb # [ 0.032528] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns635client # [ 0.067022] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16636worker2 # [ 0.071001] printk: legacy console [tty0] enabled637worker1 # [ 0.199543] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns638lb # [ 0.037004] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1639client # [ 0.067308] rcu: srcu_init: Setting srcu_struct sizes based on contention.640worker2 # [ 0.100571] printk: legacy console [ttyS0] enabled641lb # [ 0.039157] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152642worker1 # [ 0.201085] APIC: Switch to symmetric I/O mode setup643worker2 # [ 0.196090] ACPI: Core revision 20250807644client # [ 0.067316] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns645lb # [ 0.039203] kvm-guest: PV spinlocks disabled, single CPU646worker1 # [ 0.202032] x2apic enabled647client # [ 0.067421] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)648worker2 # [ 0.196983] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns649worker1 # [ 0.202776] APIC: Switched APIC routing to: physical x2apic650client # [ 0.071190] Console: colour VGA+ 80x25651client # [ 0.071194] printk: legacy console [tty0] enabled652worker2 # [ 0.198484] APIC: Switch to symmetric I/O mode setup653client # [ 0.100741] printk: legacy console [ttyS0] enabled654worker2 # [ 0.199446] x2apic enabled655worker1 # [ 0.204581] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1656client # [ 0.195831] ACPI: Core revision 20250807657worker2 # [ 0.200212] APIC: Switched APIC routing to: physical x2apic658lb # [ 0.039204] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qfr7fghmwm537hb3f4f5is3yb8phm6jx-nixos-system-lb-test/init regInfo=/nix/store/s1480xaqc00gh7hz31b4z2fwjpnxi3sa-closure-info/registration console=ttyS0,115200n8 console=tty0659worker1 # [ 0.205572] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns660client # [ 0.196742] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns661lb # [ 0.039299] Unknown kernel command line parameters "regInfo=/nix/store/s1480xaqc00gh7hz31b4z2fwjpnxi3sa-closure-info/registration", will be passed to user space.662worker2 # [ 0.202045] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1663client # [ 0.198288] APIC: Switch to symmetric I/O mode setup664lb # [ 0.039311] random: crng init done665worker1 # [ 0.207255] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)666client # [ 0.199262] x2apic enabled667lb # [ 0.039311] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes668worker1 # [ 0.208338] x86/cpu: User Mode Instruction Prevention (UMIP) activated669worker2 # [ 0.203010] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns670client # [ 0.200021] APIC: Switched APIC routing to: physical x2apic671lb # [ 0.041408] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)672worker1 # [ 0.209376] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127673worker2 # [ 0.204636] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)674lb # [ 0.042797] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)675lb # [ 0.042828] Fallback order for Node 0: 0676worker1 # [ 0.210253] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0677client # [ 0.201812] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1678worker2 # [ 0.205723] x86/cpu: User Mode Instruction Prevention (UMIP) activated679lb # [ 0.042831] Built 1 zonelists, mobility grouping on. Total pages: 393078680lb # [ 0.042832] Policy zone: DMA32681worker2 # [ 0.206757] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127682worker1 # [ 0.211257] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto683lb # [ 0.045523] mem auto-init: stack:all(zero), heap alloc:on, heap free:off684client # [ 0.202770] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns685worker2 # [ 0.207633] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0686lb # [ 0.048735] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1687worker1 # [ 0.213252] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl688client # [ 0.204391] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)689lb # [ 0.052162] allocated 3145728 bytes of page_ext690lb # [ 0.061784] ftrace: allocating 48787 entries in 192 pages691worker1 # [ 0.214253] Transient Scheduler Attacks: Vulnerable: No microcode692worker2 # [ 0.208637] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto693client # [ 0.205477] x86/cpu: User Mode Instruction Prevention (UMIP) activated694lb # [ 0.061786] ftrace: allocated 192 pages with 2 groups695lb # [ 0.062693] Dynamic Preempt: lazy696worker1 # [ 0.215252] Spectre V2 : Mitigation: Enhanced / Automatic IBRS697client # [ 0.206505] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127698lb # [ 0.062867] rcu: Preemptible hierarchical RCU implementation.699worker2 # [ 0.209633] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl700lb # [ 0.062868] rcu: RCU event tracing is enabled.701worker1 # [ 0.216253] Speculative Return Stack Overflow: Mitigation: Safe RET702client # [ 0.207388] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0703worker2 # [ 0.210633] Transient Scheduler Attacks: Vulnerable: No microcode704lb # [ 0.062869] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.705lb # [ 0.062870] Trampoline variant of Tasks RCU enabled.706worker2 # [ 0.211633] Spectre V2 : Mitigation: Enhanced / Automatic IBRS707lb # [ 0.062871] Rude variant of Tasks RCU enabled.708client # [ 0.208392] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto709worker2 # [ 0.212633] Speculative Return Stack Overflow: Mitigation: Safe RET710lb # [ 0.062871] Tracing variant of Tasks RCU enabled.711client # [ 0.209389] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl712lb # [ 0.062872] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.713worker1 # [ 0.217252] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization714lb # [ 0.062872] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1715client # [ 0.210388] Transient Scheduler Attacks: Vulnerable: No microcode716worker1 # [ 0.219260] Spectre V2 : Enabling IBPB for BPF717client # [ 0.211388] Spectre V2 : Mitigation: Enhanced / Automatic IBRS718lb # [ 0.062887] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.719worker1 # [ 0.220254] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier720client # [ 0.212388] Speculative Return Stack Overflow: Mitigation: Safe RET721worker2 # [ 0.213633] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization722lb # [ 0.062888] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.723worker1 # [ 0.221253] active return thunk: srso_alias_return_thunk724worker2 # [ 0.215641] Spectre V2 : Enabling IBPB for BPF725lb # [ 0.062889] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.726lb # [ 0.067465] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16727worker1 # [ 0.222273] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'728worker2 # [ 0.216634] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier729lb # [ 0.067740] rcu: srcu_init: Setting srcu_struct sizes based on contention.730worker2 # [ 0.217633] active return thunk: srso_alias_return_thunk731worker1 # [ 0.223252] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'732client # [ 0.213388] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization733lb # [ 0.067746] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns734worker1 # [ 0.225001] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'735worker2 # [ 0.218655] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'736client # [ 0.215395] Spectre V2 : Enabling IBPB for BPF737worker1 # [ 0.225986] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'738lb # [ 0.067848] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)739worker2 # [ 0.219633] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'740lb # [ 0.071375] Console: colour VGA+ 80x25741client # [ 0.216091] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier742worker1 # [ 0.226997] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'743lb # [ 0.071378] printk: legacy console [tty0] enabled744worker2 # [ 0.220633] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'745lb # [ 0.100840] printk: legacy console [ttyS0] enabled746client # [ 0.217388] active return thunk: srso_alias_return_thunk747worker1 # [ 0.228000] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'748worker2 # [ 0.221633] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'749lb # [ 0.197274] ACPI: Core revision 20250807750client # [ 0.218410] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'751worker2 # [ 0.222633] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'752worker1 # [ 0.229038] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'753client # [ 0.219388] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'754lb # [ 0.198207] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns755worker2 # [ 0.223633] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'756worker1 # [ 0.230253] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256757client # [ 0.220388] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'758lb # [ 0.199681] APIC: Switch to symmetric I/O mode setup759worker1 # [ 0.231252] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64760worker2 # [ 0.224633] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'761lb # [ 0.200651] x2apic enabled762client # [ 0.221388] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'763worker1 # [ 0.232252] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512764worker2 # [ 0.225633] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256765lb # [ 0.201392] APIC: Switched APIC routing to: physical x2apic766client # [ 0.222388] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'767worker1 # [ 0.233252] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024768worker2 # [ 0.226633] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64769client # [ 0.223388] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'770worker1 # [ 0.234175] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8771worker2 # [ 0.227633] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512772lb # [ 0.203202] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1773client # [ 0.224388] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'774worker2 # [ 0.228633] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024775worker1 # [ 0.234983] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.776client # [ 0.225388] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256777worker2 # [ 0.229633] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8778lb # [ 0.204176] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns779client # [ 0.226388] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64780lb # [ 0.205837] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)781worker2 # [ 0.230633] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.782client # [ 0.227388] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512783lb # [ 0.206921] x86/cpu: User Mode Instruction Prevention (UMIP) activated784client # [ 0.228388] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024785client # [ 0.229388] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8786lb # [ 0.207973] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127787lb # [ 0.209563] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0788client # [ 0.230388] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.789lb # [ 0.210640] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto790lb # [ 0.211834] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl791lb # [ 0.212834] Transient Scheduler Attacks: Vulnerable: No microcode792lb # [ 0.213833] Spectre V2 : Mitigation: Enhanced / Automatic IBRS793lb # [ 0.214834] Speculative Return Stack Overflow: Mitigation: Safe RET794lb # [ 0.215833] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization795lb # [ 0.217842] Spectre V2 : Enabling IBPB for BPF796lb # [ 0.218603] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier797lb # [ 0.219834] active return thunk: srso_alias_return_thunk798lb # [ 0.220854] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'799lb # [ 0.221833] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'800lb # [ 0.223643] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'801lb # [ 0.224644] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'802lb # [ 0.225654] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'803lb # [ 0.226648] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'804lb # [ 0.227833] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'805lb # [ 0.228833] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256806lb # [ 0.229833] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64807lb # [ 0.230833] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512808lb # [ 0.231833] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024809lb # [ 0.232833] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8810lb # [ 0.233833] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.811worker1 # [ 0.270024] Freeing SMP alternatives memory: 44K812worker1 # [ 0.270829] pid_max: default: 32768 minimum: 301813worker2 # [ 0.265766] Freeing SMP alternatives memory: 44K814worker1 # [ 0.271344] LSM: initializing lsm=capability,landlock,yama,bpf,ima815worker2 # [ 0.266538] pid_max: default: 32768 minimum: 301816worker1 # [ 0.272350] landlock: Up and running.817worker1 # [ 0.273253] Yama: becoming mindful.818worker2 # [ 0.267257] LSM: initializing lsm=capability,landlock,yama,bpf,ima819worker1 # [ 0.274341] LSM support for eBPF active820worker2 # [ 0.267755] landlock: Up and running.821worker2 # [ 0.269058] Yama: becoming mindful.822client # [ 0.265196] Freeing SMP alternatives memory: 44K823worker1 # [ 0.275106] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)824worker2 # [ 0.269838] LSM support for eBPF active825client # [ 0.265933] pid_max: default: 32768 minimum: 301826worker1 # [ 0.276292] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)827worker2 # [ 0.270664] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)828client # [ 0.266495] LSM: initializing lsm=capability,landlock,yama,bpf,ima829client # [ 0.267494] landlock: Up and running.830worker2 # [ 0.271673] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)831client # [ 0.268389] Yama: becoming mindful.832worker1 # [ 0.278712] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)833client # [ 0.269395] LSM support for eBPF active834client # [ 0.270163] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)835worker2 # [ 0.274145] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)836worker1 # [ 0.279822] Performance Events: Fam17h+ core perfctr, AMD PMU driver.837worker1 # [ 0.280258] ... version: 2838client # [ 0.271416] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)839worker1 # [ 0.280967] ... bit width: 48840worker2 # [ 0.275140] Performance Events: Fam17h+ core perfctr, AMD PMU driver.841worker1 # [ 0.281255] ... generic counters: 6842worker2 # [ 0.275638] ... version: 2843worker2 # [ 0.276354] ... bit width: 48844worker1 # [ 0.281978] ... generic bitmap: 000000000000003f845client # [ 0.273629] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)846worker2 # [ 0.276635] ... generic counters: 6847worker1 # [ 0.282261] ... fixed-purpose counters: 0848worker2 # [ 0.277306] ... generic bitmap: 000000000000003f849worker1 # [ 0.282965] ... fixed-purpose bitmap: 0000000000000000850client # [ 0.274971] Performance Events: Fam17h+ core perfctr, AMD PMU driver.851worker2 # [ 0.277683] ... fixed-purpose counters: 0852worker1 # [ 0.283255] ... value mask: 0000ffffffffffff853client # [ 0.275398] ... version: 2854worker2 # [ 0.278392] ... fixed-purpose bitmap: 0000000000000000855worker1 # [ 0.284148] ... max period: 00007fffffffffff856client # [ 0.276102] ... bit width: 48857worker2 # [ 0.278636] ... value mask: 0000ffffffffffff858client # [ 0.276391] ... generic counters: 6859worker1 # [ 0.284928] ... global_ctrl mask: 000000000000003f860worker2 # [ 0.279511] ... max period: 00007fffffffffff861worker1 # [ 0.285351] signal: max sigframe size: 3376862client # [ 0.277094] ... generic bitmap: 000000000000003f863worker2 # [ 0.280275] ... global_ctrl mask: 000000000000003f864client # [ 0.277391] ... fixed-purpose counters: 0865worker1 # [ 0.286166] rcu: Hierarchical SRCU implementation.866worker2 # [ 0.280750] signal: max sigframe size: 3376867client # [ 0.278050] ... fixed-purpose bitmap: 0000000000000000868worker1 # [ 0.286868] rcu: Max phase no-delay instances is 400.869worker2 # [ 0.281536] rcu: Hierarchical SRCU implementation.870client # [ 0.278391] ... value mask: 0000ffffffffffff871worker2 # [ 0.282224] rcu: Max phase no-delay instances is 400.872client # [ 0.279255] ... max period: 00007fffffffffff873client # [ 0.280064] ... global_ctrl mask: 000000000000003f874client # [ 0.280497] signal: max sigframe size: 3376875client # [ 0.281283] rcu: Hierarchical SRCU implementation.876client # [ 0.281987] rcu: Max phase no-delay instances is 400.877worker1 # [ 0.291790] smp: Bringing up secondary CPUs ...878worker1 # [ 0.292269] smp: Brought up 1 node, 1 CPU879worker2 # [ 0.287232] smp: Bringing up secondary CPUs ...880worker1 # [ 0.292968] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)881worker2 # [ 0.287649] smp: Brought up 1 node, 1 CPU882worker2 # [ 0.288363] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)883worker1 # [ 0.293426] Memory: 1454592K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109312K reserved, 0K cma-reserved)884worker1 # [ 0.294472] devtmpfs: initialized885client # [ 0.287050] smp: Bringing up secondary CPUs ...886worker2 # [ 0.288864] Memory: 1454596K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109312K reserved, 0K cma-reserved)887client # [ 0.287406] smp: Brought up 1 node, 1 CPU888worker1 # [ 0.295197] x86/mm: Memory block size: 128MB889worker2 # [ 0.289929] devtmpfs: initialized890client # [ 0.288077] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)891lb # [ 0.268703] Freeing SMP alternatives memory: 44K892worker2 # [ 0.290731] x86/mm: Memory block size: 128MB893lb # [ 0.269410] pid_max: default: 32768 minimum: 301894worker1 # [ 0.296853] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)895lb # [ 0.269927] LSM: initializing lsm=capability,landlock,yama,bpf,ima896client # [ 0.288593] Memory: 1454632K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)897worker1 # [ 0.297279] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).898lb # [ 0.270937] landlock: Up and running.899worker2 # [ 0.292510] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)900client # [ 0.289680] devtmpfs: initialized901lb # [ 0.271834] Yama: becoming mindful.902worker1 # [ 0.298342] pinctrl core: initialized pinctrl subsystem903client # [ 0.290492] x86/mm: Memory block size: 128MB904worker2 # [ 0.293535] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).905lb # [ 0.272910] LSM support for eBPF active906worker1 # [ 0.299441] PM: RTC time: 11:32:18, date: 2026-09-22907worker2 # [ 0.294724] pinctrl core: initialized pinctrl subsystem908lb # [ 0.273652] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)909worker2 # [ 0.295839] PM: RTC time: 11:32:18, date: 2026-09-22910client # [ 0.292238] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)911lb # [ 0.274878] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)912client # [ 0.293262] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).913client # [ 0.294472] pinctrl core: initialized pinctrl subsystem914worker1 # [ 0.303383] NET: Registered PF_NETLINK/PF_ROUTE protocol family915lb # [ 0.277200] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)916client # [ 0.295601] PM: RTC time: 11:32:18, date: 2026-09-22917worker2 # [ 0.299397] NET: Registered PF_NETLINK/PF_ROUTE protocol family918worker1 # [ 0.304755] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations919lb # [ 0.278391] Performance Events: Fam17h+ core perfctr, AMD PMU driver.920worker2 # [ 0.300130] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations921worker1 # [ 0.305277] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations922lb # [ 0.278842] ... version: 2923lb # [ 0.279526] ... bit width: 48924worker2 # [ 0.300660] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations925lb # [ 0.279836] ... generic counters: 6926worker1 # [ 0.306547] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations927client # [ 0.299075] NET: Registered PF_NETLINK/PF_ROUTE protocol family928lb # [ 0.280518] ... generic bitmap: 000000000000003f929worker1 # [ 0.307265] audit: initializing netlink subsys (disabled)930worker2 # [ 0.301926] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations931lb # [ 0.280836] ... fixed-purpose counters: 0932client # [ 0.299879] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations933worker2 # [ 0.302647] audit: initializing netlink subsys (disabled)934worker1 # [ 0.308414] thermal_sys: Registered thermal governor 'fair_share'935lb # [ 0.281533] ... fixed-purpose bitmap: 0000000000000000936client # [ 0.300418] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations937worker1 # [ 0.308416] thermal_sys: Registered thermal governor 'bang_bang'938lb # [ 0.281835] ... value mask: 0000ffffffffffff939worker2 # [ 0.303638] audit: type=2000 audit(1790076739.377:1): state=initialized audit_enabled=0 res=1940lb # [ 0.282673] ... max period: 00007fffffffffff941worker1 # [ 0.309256] thermal_sys: Registered thermal governor 'step_wise'942client # [ 0.301666] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations943worker2 # [ 0.304773] thermal_sys: Registered thermal governor 'fair_share'944lb # [ 0.283482] ... global_ctrl mask: 000000000000003f945worker1 # [ 0.310179] thermal_sys: Registered thermal governor 'user_space'946client # [ 0.302401] audit: initializing netlink subsys (disabled)947worker2 # [ 0.304775] thermal_sys: Registered thermal governor 'bang_bang'948lb # [ 0.283931] signal: max sigframe size: 3376949worker1 # [ 0.310980] audit: type=2000 audit(1790076739.375:1): state=initialized audit_enabled=0 res=1950worker2 # [ 0.305636] thermal_sys: Registered thermal governor 'step_wise'951lb # [ 0.284704] rcu: Hierarchical SRCU implementation.952client # [ 0.303637] thermal_sys: Registered thermal governor 'fair_share'953worker1 # [ 0.312258] thermal_sys: Registered thermal governor 'power_allocator'954lb # [ 0.285396] rcu: Max phase no-delay instances is 400.955worker2 # [ 0.306561] thermal_sys: Registered thermal governor 'user_space'956client # [ 0.303640] thermal_sys: Registered thermal governor 'bang_bang'957worker1 # [ 0.312273] cpuidle: using governor menu958client # [ 0.304392] thermal_sys: Registered thermal governor 'step_wise'959worker2 # [ 0.307342] thermal_sys: Registered thermal governor 'power_allocator'960worker2 # [ 0.307649] cpuidle: using governor menu961client # [ 0.305288] thermal_sys: Registered thermal governor 'user_space'962worker1 # [ 0.314285] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5963worker2 # [ 0.309844] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5964client # [ 0.306116] audit: type=2000 audit(1790076739.382:1): state=initialized audit_enabled=0 res=1965worker1 # [ 0.315489] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]966client # [ 0.307393] thermal_sys: Registered thermal governor 'power_allocator'967worker1 # [ 0.316258] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry968worker2 # [ 0.310923] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]969client # [ 0.307408] cpuidle: using governor menu970lb # [ 0.290380] smp: Bringing up secondary CPUs ...971worker1 # [ 0.317266] PCI: Using configuration type 1 for base access972worker2 # [ 0.311638] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry973lb # [ 0.290850] smp: Brought up 1 node, 1 CPU974client # [ 0.309618] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5975worker2 # [ 0.312647] PCI: Using configuration type 1 for base access976lb # [ 0.291524] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)977worker1 # [ 0.318355] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.978client # [ 0.310670] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]979worker2 # [ 0.313732] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.980client # [ 0.311394] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry981lb # [ 0.292019] Memory: 1454588K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109316K reserved, 0K cma-reserved)982client # [ 0.312402] PCI: Using configuration type 1 for base access983lb # [ 0.293136] devtmpfs: initialized984lb # [ 0.293942] x86/mm: Memory block size: 128MB985client # [ 0.313467] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.986lb # [ 0.295758] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)987worker1 # [ 0.325399] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages988lb # [ 0.296767] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).989worker1 # [ 0.326256] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page990worker2 # [ 0.320650] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages991lb # [ 0.297912] pinctrl core: initialized pinctrl subsystem992worker2 # [ 0.321636] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page993lb # [ 0.299026] PM: RTC time: 11:32:18, date: 2026-09-22994client # [ 0.318476] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages995client # [ 0.319391] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page996lb # [ 0.302536] NET: Registered PF_NETLINK/PF_ROUTE protocol family997worker1 # [ 0.331256] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages998worker2 # [ 0.326637] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages999worker1 # [ 0.332256] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1000lb # [ 0.303318] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations1001worker2 # [ 0.327636] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1002client # [ 0.324391] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1003lb # [ 0.303860] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations1004client # [ 0.325390] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1005lb # [ 0.305109] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations1006lb # [ 0.305846] audit: initializing netlink subsys (disabled)1007lb # [ 0.307063] thermal_sys: Registered thermal governor 'fair_share'1008lb # [ 0.307065] thermal_sys: Registered thermal governor 'bang_bang'1009lb # [ 0.307841] audit: type=2000 audit(1790076739.409:1): state=initialized audit_enabled=0 res=11010lb # [ 0.309838] thermal_sys: Registered thermal governor 'step_wise'1011lb # [ 0.309839] thermal_sys: Registered thermal governor 'user_space'1012worker1 # [ 0.342678] ACPI: Added _OSI(Module Device)1013worker1 # [ 0.343257] ACPI: Added _OSI(Processor Device)1014lb # [ 0.310835] thermal_sys: Registered thermal governor 'power_allocator'1015lb # [ 0.311847] cpuidle: using governor menu1016worker1 # [ 0.343989] ACPI: Added _OSI(Processor Aggregator Device)1017worker2 # [ 0.338904] ACPI: Added _OSI(Module Device)1018client # [ 0.335760] ACPI: Added _OSI(Module Device)1019worker2 # [ 0.339637] ACPI: Added _OSI(Processor Device)1020client # [ 0.336392] ACPI: Added _OSI(Processor Device)1021lb # [ 0.314050] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.51022worker2 # [ 0.340345] ACPI: Added _OSI(Processor Aggregator Device)1023client # [ 0.337118] ACPI: Added _OSI(Processor Aggregator Device)1024lb # [ 0.315055] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]1025lb # [ 0.315839] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry1026lb # [ 0.316848] PCI: Using configuration type 1 for base access1027lb # [ 0.317980] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.1028worker1 # [ 0.350835] ACPI: 1 ACPI AML tables successfully acquired and loaded1029worker2 # [ 0.345974] ACPI: 1 ACPI AML tables successfully acquired and loaded1030client # [ 0.345454] ACPI: 1 ACPI AML tables successfully acquired and loaded1031worker1 # [ 0.354663] ACPI: Interpreter enabled1032lb # [ 0.322950] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1033worker2 # [ 0.350037] ACPI: Interpreter enabled1034worker1 # [ 0.355271] ACPI: PM: (supports S0 S3 S4 S5)1035worker2 # [ 0.350650] ACPI: PM: (supports S0 S3 S4 S5)1036worker1 # [ 0.355970] ACPI: Using IOAPIC for interrupt routing1037lb # [ 0.323837] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1038worker2 # [ 0.351335] ACPI: Using IOAPIC for interrupt routing1039client # [ 0.349790] ACPI: Interpreter enabled1040client # [ 0.350405] ACPI: PM: (supports S0 S3 S4 S5)1041worker1 # [ 0.358318] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1042client # [ 0.351110] ACPI: Using IOAPIC for interrupt routing1043worker2 # [ 0.353702] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1044worker1 # [ 0.359256] PCI: Using E820 reservations for host bridge windows1045worker2 # [ 0.354637] PCI: Using E820 reservations for host bridge windows1046lb # [ 0.328837] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1047lb # [ 0.329836] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1048client # [ 0.353439] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1049worker1 # [ 0.362349] ACPI: Enabled 2 GPEs in block 00 to 3F1050worker2 # [ 0.357724] ACPI: Enabled 2 GPEs in block 00 to 3F1051client # [ 0.354391] PCI: Using E820 reservations for host bridge windows1052client # [ 0.357462] ACPI: Enabled 2 GPEs in block 00 to 3F1053worker1 # [ 0.369312] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1054worker2 # [ 0.365130] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1055worker1 # [ 0.370260] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1056lb # [ 0.340181] ACPI: Added _OSI(Module Device)1057worker2 # [ 0.365641] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1058lb # [ 0.340837] ACPI: Added _OSI(Processor Device)1059worker1 # [ 0.371332] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1060lb # [ 0.341591] ACPI: Added _OSI(Processor Aggregator Device)1061worker2 # [ 0.366713] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1062worker1 # [ 0.372378] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1063client # [ 0.364954] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1064worker2 # [ 0.367757] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1065worker1 # [ 0.373594] PCI host bridge to bus 0000:001066client # [ 0.365403] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1067worker2 # [ 0.368996] PCI host bridge to bus 0000:001068worker1 # [ 0.374259] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1069client # [ 0.366469] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1070worker2 # [ 0.369641] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1071worker1 # [ 0.375255] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1072client # [ 0.367513] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1073worker2 # [ 0.370636] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1074worker1 # [ 0.376255] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1075client # [ 0.368775] PCI host bridge to bus 0000:001076worker2 # [ 0.371636] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1077worker1 # [ 0.377256] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1078client # [ 0.369395] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1079worker2 # [ 0.372635] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1080worker1 # [ 0.378255] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1081client # [ 0.370391] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1082worker2 # [ 0.373635] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1083worker1 # [ 0.379255] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1084client # [ 0.371391] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1085lb # [ 0.349899] ACPI: 1 ACPI AML tables successfully acquired and loaded1086worker2 # [ 0.374635] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1087worker1 # [ 0.380255] pci_bus 0000:00: root bus resource [bus 00-ff]1088client # [ 0.372391] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1089worker2 # [ 0.375649] pci_bus 0000:00: root bus resource [bus 00-ff]1090worker1 # [ 0.381276] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1091client # [ 0.373391] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1092worker2 # [ 0.376675] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1093client # [ 0.374391] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1094worker1 # [ 0.382706] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1095lb # [ 0.354240] ACPI: Interpreter enabled1096client # [ 0.375391] pci_bus 0000:00: root bus resource [bus 00-ff]1097worker2 # [ 0.378100] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1098lb # [ 0.354851] ACPI: PM: (supports S0 S3 S4 S5)1099lb # [ 0.355596] ACPI: Using IOAPIC for interrupt routing1100client # [ 0.376362] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1101client # [ 0.377792] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1102lb # [ 0.357879] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1103lb # [ 0.358837] PCI: Using E820 reservations for host bridge windows1104worker1 # [ 0.385319] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1105worker2 # [ 0.380789] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1106lb # [ 0.361962] ACPI: Enabled 2 GPEs in block 00 to 3F1107worker1 # [ 0.386267] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1108worker2 # [ 0.381648] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1109worker2 # [ 0.382576] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1110worker1 # [ 0.387234] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1111client # [ 0.380481] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1112worker2 # [ 0.383568] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1113worker1 # [ 0.388609] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1114client # [ 0.381403] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1115client # [ 0.382349] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1116worker1 # [ 0.389974] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1117worker2 # [ 0.385410] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1118client # [ 0.383625] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1119lb # [ 0.369488] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1120worker1 # [ 0.392275] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1121worker2 # [ 0.388673] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1122client # [ 0.385119] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1123worker1 # [ 0.393122] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1124lb # [ 0.369843] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1125worker2 # [ 0.389482] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1126worker1 # [ 0.393968] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1127lb # [ 0.370915] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1128worker2 # [ 0.390330] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1129worker1 # [ 0.394261] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1130worker2 # [ 0.390649] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1131lb # [ 0.371967] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1132client # [ 0.388425] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1133lb # [ 0.373180] PCI host bridge to bus 0000:001134client # [ 0.389360] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1135worker1 # [ 0.396027] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1136worker2 # [ 0.392329] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1137lb # [ 0.373840] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1138client # [ 0.390112] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1139lb # [ 0.374836] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1140client # [ 0.390396] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1141worker1 # [ 0.398274] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1142lb # [ 0.375836] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1143worker2 # [ 0.394673] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1144worker1 # [ 0.399187] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1145client # [ 0.392318] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1146lb # [ 0.376836] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1147worker2 # [ 0.395491] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1148worker1 # [ 0.399980] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1149worker2 # [ 0.396325] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1150lb # [ 0.377836] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1151worker1 # [ 0.400803] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1152lb # [ 0.378836] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1153client # [ 0.394423] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1154worker2 # [ 0.397204] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1155lb # [ 0.379836] pci_bus 0000:00: root bus resource [bus 00-ff]1156client # [ 0.395257] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1157client # [ 0.396124] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1158lb # [ 0.380813] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1159client # [ 0.396976] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1160lb # [ 0.382290] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1161worker1 # [ 0.403263] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1162worker2 # [ 0.399831] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1163worker1 # [ 0.404261] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1164worker2 # [ 0.400642] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1165worker1 # [ 0.405276] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1166worker2 # [ 0.401596] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1167worker1 # [ 0.406806] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1168client # [ 0.399399] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1169worker2 # [ 0.403029] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1170client # [ 0.400995] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1171worker1 # [ 0.408274] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1172client # [ 0.401412] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1173worker1 # [ 0.409120] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1174worker2 # [ 0.404652] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1175client # [ 0.402961] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1176worker2 # [ 0.405470] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1177worker1 # [ 0.410027] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1178lb # [ 0.384972] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1179worker1 # [ 0.411052] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1180worker2 # [ 0.406359] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1181lb # [ 0.385971] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1182worker2 # [ 0.406641] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1183client # [ 0.404421] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1184lb # [ 0.386860] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1185worker1 # [ 0.413035] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1186client # [ 0.405309] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1187worker2 # [ 0.408176] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1188client # [ 0.406130] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1189lb # [ 0.388203] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1190worker1 # [ 0.415089] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1191client # [ 0.406397] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1192worker1 # [ 0.415999] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1193worker2 # [ 0.409678] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1194lb # [ 0.389571] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1195worker2 # [ 0.410627] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1196worker1 # [ 0.416813] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1197client # [ 0.408704] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1198worker2 # [ 0.412065] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1199lb # [ 0.392874] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1200client # [ 0.410428] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1201worker1 # [ 0.418269] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1202lb # [ 0.393738] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1203client # [ 0.411396] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1204worker1 # [ 0.419203] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1205worker2 # [ 0.413678] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1206lb # [ 0.394564] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1207lb # [ 0.394841] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1208worker2 # [ 0.414621] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1209client # [ 0.412972] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1210worker1 # [ 0.421063] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1211lb # [ 0.396853] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1212worker2 # [ 0.416254] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1213client # [ 0.414427] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1214worker1 # [ 0.423203] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1215worker1 # [ 0.424003] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1216worker2 # [ 0.417655] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1217client # [ 0.415395] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1218lb # [ 0.398871] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1219worker2 # [ 0.418612] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1220lb # [ 0.399709] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1221worker1 # [ 0.424813] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1222client # [ 0.416927] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1223lb # [ 0.400549] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1224worker2 # [ 0.419985] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1225worker1 # [ 0.426277] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1226client # [ 0.418397] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1227lb # [ 0.401409] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1228worker1 # [ 0.427277] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1229worker2 # [ 0.421649] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1230client # [ 0.419344] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1231worker2 # [ 0.422600] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1232client # [ 0.420750] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1233worker1 # [ 0.429292] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1234worker2 # [ 0.424203] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1235lb # [ 0.403844] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1236client # [ 0.422429] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1237worker1 # [ 0.431263] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1238lb # [ 0.405485] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1239worker1 # [ 0.432089] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1240client # [ 0.423365] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1241worker2 # [ 0.425655] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1242lb # [ 0.405858] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1243worker1 # [ 0.432994] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1244worker2 # [ 0.426457] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1245client # [ 0.424953] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1246lb # [ 0.407380] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1247worker2 # [ 0.427382] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1248worker1 # [ 0.433804] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1249worker2 # [ 0.428224] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1250client # [ 0.426429] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1251lb # [ 0.408844] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1252client # [ 0.427251] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1253lb # [ 0.409677] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1254worker1 # [ 0.435275] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1255client # [ 0.428114] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1256worker1 # [ 0.436262] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1257lb # [ 0.410628] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1258worker2 # [ 0.429866] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1259worker1 # [ 0.437175] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1260lb # [ 0.411685] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1261client # [ 0.428960] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1262worker2 # [ 0.430642] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1263worker2 # [ 0.431640] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1264worker1 # [ 0.438599] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1265lb # [ 0.413391] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1266worker1 # [ 0.439793] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1267worker2 # [ 0.433194] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1268client # [ 0.430419] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1269lb # [ 0.414849] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1270worker1 # [ 0.440458] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1271client # [ 0.432072] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1272worker2 # [ 0.434356] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1273lb # [ 0.415811] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1274client # [ 0.432412] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1275worker1 # [ 0.441791] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1276worker2 # [ 0.434832] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1277worker1 # [ 0.442469] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1278lb # [ 0.417310] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1279client # [ 0.434001] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1280worker2 # [ 0.436250] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1281client # [ 0.435124] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1282worker1 # [ 0.444283] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1283lb # [ 0.418849] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1284worker2 # [ 0.436837] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1285client # [ 0.435621] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1286worker2 # [ 0.438307] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1287lb # [ 0.419776] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1288client # [ 0.437054] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1289lb # [ 0.421397] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1290client # [ 0.437589] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1291lb # [ 0.422849] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1292client # [ 0.439059] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1293lb # [ 0.423858] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1294lb # [ 0.425441] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1295lb # [ 0.426881] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1296worker1 # [ 0.445347] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1297lb # [ 0.427858] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1298worker1 # [ 0.447780] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1299worker2 # [ 0.438900] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1300lb # [ 0.429407] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1301worker1 # [ 0.448529] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1302worker2 # [ 0.441327] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1303worker1 # [ 0.449554] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1304lb # [ 0.430880] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1305lb # [ 0.431786] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1306worker2 # [ 0.441991] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1307worker1 # [ 0.450502] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1308client # [ 0.439689] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1309lb # [ 0.432532] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1310worker2 # [ 0.442922] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1311worker1 # [ 0.452276] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1312client # [ 0.442071] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1313lb # [ 0.433373] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1314worker1 # [ 0.453128] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1315worker2 # [ 0.443881] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1316client # [ 0.442627] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1317worker1 # [ 0.454261] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1318worker2 # [ 0.445652] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1319client # [ 0.443700] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1320lb # [ 0.435778] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1321worker1 # [ 0.455795] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1322worker2 # [ 0.446546] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1323lb # [ 0.436482] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1324client # [ 0.444638] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1325worker2 # [ 0.447695] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1326lb # [ 0.436858] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1327worker2 # [ 0.449236] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1328client # [ 0.446391] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1329worker1 # [ 0.459233] ACPI: PCI: Interrupt link LNKA configured for IRQ 101330lb # [ 0.438391] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1331client # [ 0.447249] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1332worker1 # [ 0.460052] ACPI: PCI: Interrupt link LNKB configured for IRQ 101333lb # [ 0.439485] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1334client # [ 0.448397] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1335worker1 # [ 0.461123] ACPI: PCI: Interrupt link LNKC configured for IRQ 111336lb # [ 0.440046] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1337worker1 # [ 0.462094] ACPI: PCI: Interrupt link LNKD configured for IRQ 111338client # [ 0.450045] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1339worker2 # [ 0.453146] ACPI: PCI: Interrupt link LNKA configured for IRQ 101340worker1 # [ 0.463052] ACPI: PCI: Interrupt link LNKE configured for IRQ 101341worker2 # [ 0.453741] ACPI: PCI: Interrupt link LNKB configured for IRQ 101342lb # [ 0.441514] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1343worker1 # [ 0.464086] ACPI: PCI: Interrupt link LNKF configured for IRQ 101344worker2 # [ 0.454733] ACPI: PCI: Interrupt link LNKC configured for IRQ 111345worker1 # [ 0.465094] ACPI: PCI: Interrupt link LNKG configured for IRQ 111346lb # [ 0.442033] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1347worker2 # [ 0.455735] ACPI: PCI: Interrupt link LNKD configured for IRQ 111348worker1 # [ 0.466072] ACPI: PCI: Interrupt link LNKH configured for IRQ 111349client # [ 0.453935] ACPI: PCI: Interrupt link LNKA configured for IRQ 101350worker2 # [ 0.456733] ACPI: PCI: Interrupt link LNKE configured for IRQ 101351lb # [ 0.443503] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1352worker1 # [ 0.467016] ACPI: PCI: Interrupt link GSIA configured for IRQ 161353client # [ 0.454494] ACPI: PCI: Interrupt link LNKB configured for IRQ 101354worker2 # [ 0.457732] ACPI: PCI: Interrupt link LNKF configured for IRQ 101355worker1 # [ 0.467271] ACPI: PCI: Interrupt link GSIB configured for IRQ 171356client # [ 0.455490] ACPI: PCI: Interrupt link LNKC configured for IRQ 111357worker2 # [ 0.458734] ACPI: PCI: Interrupt link LNKG configured for IRQ 111358worker1 # [ 0.468226] ACPI: PCI: Interrupt link GSIC configured for IRQ 181359client # [ 0.456487] ACPI: PCI: Interrupt link LNKD configured for IRQ 111360worker2 # [ 0.459734] ACPI: PCI: Interrupt link LNKH configured for IRQ 111361worker1 # [ 0.468987] ACPI: PCI: Interrupt link GSID configured for IRQ 191362client # [ 0.457487] ACPI: PCI: Interrupt link LNKE configured for IRQ 101363worker1 # [ 0.469269] ACPI: PCI: Interrupt link GSIE configured for IRQ 201364worker2 # [ 0.460683] ACPI: PCI: Interrupt link GSIA configured for IRQ 161365client # [ 0.458489] ACPI: PCI: Interrupt link LNKF configured for IRQ 101366worker1 # [ 0.470267] ACPI: PCI: Interrupt link GSIF configured for IRQ 211367worker2 # [ 0.461652] ACPI: PCI: Interrupt link GSIB configured for IRQ 171368client # [ 0.459490] ACPI: PCI: Interrupt link LNKG configured for IRQ 111369worker1 # [ 0.471205] ACPI: PCI: Interrupt link GSIG configured for IRQ 221370worker2 # [ 0.462649] ACPI: PCI: Interrupt link GSIC configured for IRQ 181371client # [ 0.460491] ACPI: PCI: Interrupt link LNKH configured for IRQ 111372worker1 # [ 0.471978] ACPI: PCI: Interrupt link GSIH configured for IRQ 231373worker2 # [ 0.463649] ACPI: PCI: Interrupt link GSID configured for IRQ 191374client # [ 0.461428] ACPI: PCI: Interrupt link GSIA configured for IRQ 161375worker2 # [ 0.464623] ACPI: PCI: Interrupt link GSIE configured for IRQ 201376worker1 # [ 0.473261] iommu: Default domain type: Translated1377client # [ 0.462366] ACPI: PCI: Interrupt link GSIB configured for IRQ 171378worker2 # [ 0.465340] ACPI: PCI: Interrupt link GSIF configured for IRQ 211379worker1 # [ 0.474029] iommu: DMA domain TLB invalidation policy: lazy mode1380client # [ 0.463229] ACPI: PCI: Interrupt link GSIC configured for IRQ 181381worker2 # [ 0.465647] ACPI: PCI: Interrupt link GSIG configured for IRQ 221382worker1 # [ 0.474538] ACPI: bus type USB registered1383client # [ 0.464146] ACPI: PCI: Interrupt link GSID configured for IRQ 191384worker2 # [ 0.466594] ACPI: PCI: Interrupt link GSIH configured for IRQ 231385worker1 # [ 0.475245] usbcore: registered new interface driver usbfs1386client # [ 0.464404] ACPI: PCI: Interrupt link GSIE configured for IRQ 201387worker1 # [ 0.475929] usbcore: registered new interface driver hub1388client # [ 0.465354] ACPI: PCI: Interrupt link GSIF configured for IRQ 211389lb # [ 0.444090] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1390worker2 # [ 0.468444] iommu: Default domain type: Translated1391worker1 # [ 0.476264] usbcore: registered new device driver usb1392client # [ 0.466100] ACPI: PCI: Interrupt link GSIG configured for IRQ 221393worker2 # [ 0.469228] iommu: DMA domain TLB invalidation policy: lazy mode1394lb # [ 0.446481] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1395client # [ 0.466406] ACPI: PCI: Interrupt link GSIH configured for IRQ 231396worker1 # [ 0.477890] NetLabel: Initializing1397worker2 # [ 0.469898] ACPI: bus type USB registered1398worker1 # [ 0.478256] NetLabel: domain hash size = 1281399worker2 # [ 0.470656] usbcore: registered new interface driver usbfs1400lb # [ 0.447070] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1401worker1 # [ 0.479036] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1402client # [ 0.468475] iommu: Default domain type: Translated1403worker2 # [ 0.471547] usbcore: registered new interface driver hub1404worker1 # [ 0.479295] NetLabel: unlabeled traffic allowed by default1405worker2 # [ 0.472305] usbcore: registered new device driver usb1406client # [ 0.469254] iommu: DMA domain TLB invalidation policy: lazy mode1407lb # [ 0.448136] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1408worker1 # [ 0.480172] PCI: Using ACPI for IRQ routing1409client # [ 0.470363] ACPI: bus type USB registered1410worker2 # [ 0.473476] NetLabel: Initializing1411client # [ 0.470937] usbcore: registered new interface driver usbfs1412lb # [ 0.449108] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1413worker2 # [ 0.474037] NetLabel: domain hash size = 1281414client # [ 0.471405] usbcore: registered new interface driver hub1415worker2 # [ 0.474635] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1416client # [ 0.472221] usbcore: registered new device driver usb1417lb # [ 0.450813] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1418worker2 # [ 0.475657] NetLabel: unlabeled traffic allowed by default1419lb # [ 0.451564] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1420worker2 # [ 0.476578] PCI: Using ACPI for IRQ routing1421client # [ 0.473854] NetLabel: Initializing1422client # [ 0.474391] NetLabel: domain hash size = 1281423lb # [ 0.452150] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1424client # [ 0.475081] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1425client # [ 0.475429] NetLabel: unlabeled traffic allowed by default1426lb # [ 0.453489] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1427client # [ 0.476302] PCI: Using ACPI for IRQ routing1428lb # [ 0.457483] ACPI: PCI: Interrupt link LNKA configured for IRQ 101429lb # [ 0.457951] ACPI: PCI: Interrupt link LNKB configured for IRQ 101430lb # [ 0.458966] ACPI: PCI: Interrupt link LNKC configured for IRQ 111431lb # [ 0.459933] ACPI: PCI: Interrupt link LNKD configured for IRQ 111432lb # [ 0.460933] ACPI: PCI: Interrupt link LNKE configured for IRQ 101433lb # [ 0.461936] ACPI: PCI: Interrupt link LNKF configured for IRQ 101434lb # [ 0.462957] ACPI: PCI: Interrupt link LNKG configured for IRQ 111435lb # [ 0.463932] ACPI: PCI: Interrupt link LNKH configured for IRQ 111436lb # [ 0.464874] ACPI: PCI: Interrupt link GSIA configured for IRQ 161437lb # [ 0.465853] ACPI: PCI: Interrupt link GSIB configured for IRQ 171438lb # [ 0.466856] ACPI: PCI: Interrupt link GSIC configured for IRQ 181439lb # [ 0.467852] ACPI: PCI: Interrupt link GSID configured for IRQ 191440lb # [ 0.468849] ACPI: PCI: Interrupt link GSIE configured for IRQ 201441lb # [ 0.469851] ACPI: PCI: Interrupt link GSIF configured for IRQ 211442lb # [ 0.470847] ACPI: PCI: Interrupt link GSIG configured for IRQ 221443lb # [ 0.471847] ACPI: PCI: Interrupt link GSIH configured for IRQ 231444lb # [ 0.473896] iommu: Default domain type: Translated1445lb # [ 0.474726] iommu: DMA domain TLB invalidation policy: lazy mode1446lb # [ 0.475913] ACPI: bus type USB registered1447lb # [ 0.476690] usbcore: registered new interface driver usbfs1448lb # [ 0.477558] usbcore: registered new interface driver hub1449lb # [ 0.477845] usbcore: registered new device driver usb1450lb # [ 0.479712] NetLabel: Initializing1451lb # [ 0.480284] NetLabel: domain hash size = 1281452lb # [ 0.480836] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1453lb # [ 0.481854] NetLabel: unlabeled traffic allowed by default1454lb # [ 0.482824] PCI: Using ACPI for IRQ routing1455worker1 # [ 0.523745] pci 0000:00:01.0: vgaarb: setting as boot VGA device1456worker1 # [ 0.524251] pci 0000:00:01.0: vgaarb: bridge control possible1457worker1 # [ 0.524251] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1458worker1 # [ 0.524259] vgaarb: loaded1459worker1 # [ 0.525003] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01460worker2 # [ 0.520989] pci 0000:00:01.0: vgaarb: setting as boot VGA device1461worker1 # [ 0.525257] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1462worker2 # [ 0.521632] pci 0000:00:01.0: vgaarb: bridge control possible1463worker2 # [ 0.521632] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1464client # [ 0.519251] pci 0000:00:01.0: vgaarb: setting as boot VGA device1465worker2 # [ 0.521641] vgaarb: loaded1466client # [ 0.519387] pci 0000:00:01.0: vgaarb: bridge control possible1467worker2 # [ 0.522390] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01468worker1 # [ 0.528340] clocksource: Switched to clocksource kvm-clock1469client # [ 0.519387] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1470worker2 # [ 0.522636] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1471client # [ 0.519395] vgaarb: loaded1472client # [ 0.520107] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01473client # [ 0.520391] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1474worker1 # [ 0.531806] VFS: Disk quotas dquot_6.6.01475worker1 # [ 0.532499] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1476worker1 # [ 0.533882] pnp: PnP ACPI init1477worker2 # [ 0.526726] clocksource: Switched to clocksource kvm-clock1478worker1 # [ 0.534683] ACPI: IRQ 4 override to edge(!), high(!)1479client # [ 0.523473] clocksource: Switched to clocksource kvm-clock1480worker1 # [ 0.535614] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1481worker2 # [ 0.530288] VFS: Disk quotas dquot_6.6.01482worker1 # [ 0.537201] pnp: PnP ACPI: found 5 devices1483client # [ 0.526930] VFS: Disk quotas dquot_6.6.01484worker2 # [ 0.530952] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1485worker2 # [ 0.532331] pnp: PnP ACPI init1486client # [ 0.527602] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1487client # [ 0.528883] pnp: PnP ACPI init1488worker2 # [ 0.533060] ACPI: IRQ 4 override to edge(!), high(!)1489client # [ 0.529650] ACPI: IRQ 4 override to edge(!), high(!)1490worker2 # [ 0.534022] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1491worker2 # [ 0.535507] pnp: PnP ACPI: found 5 devices1492client # [ 0.530579] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1493client # [ 0.532059] pnp: PnP ACPI: found 5 devices1494worker1 # [ 0.544790] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1495worker1 # [ 0.546316] clocksource: Switched to clocksource acpi_pm1496worker1 # [ 0.547320] NET: Registered PF_INET protocol family1497worker1 # [ 0.548518] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1498worker2 # [ 0.543085] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1499worker2 # [ 0.544505] clocksource: Switched to clocksource acpi_pm1500client # [ 0.539655] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1501worker2 # [ 0.545476] NET: Registered PF_INET protocol family1502client # [ 0.541032] clocksource: Switched to clocksource acpi_pm1503client # [ 0.541971] NET: Registered PF_INET protocol family1504worker2 # [ 0.546626] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1505client # [ 0.543146] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1506lb # [ 0.526511] pci 0000:00:01.0: vgaarb: setting as boot VGA device1507lb # [ 0.526832] pci 0000:00:01.0: vgaarb: bridge control possible1508lb # [ 0.526832] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1509lb # [ 0.526840] vgaarb: loaded1510lb # [ 0.527632] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01511lb # [ 0.528502] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1512worker1 # [ 0.565270] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1513worker1 # [ 0.566861] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1514worker1 # [ 0.568208] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1515lb # [ 0.531920] clocksource: Switched to clocksource kvm-clock1516worker1 # [ 0.569589] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1517worker2 # [ 0.563412] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1518worker1 # [ 0.570856] TCP: Hash tables configured (established 16384 bind 16384)1519client # [ 0.559797] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1520worker2 # [ 0.564914] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1521worker1 # [ 0.571974] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1522lb # [ 0.535505] VFS: Disk quotas dquot_6.6.01523client # [ 0.561282] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1524worker1 # [ 0.573214] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1525worker2 # [ 0.566179] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1526lb # [ 0.536463] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1527client # [ 0.562519] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1528worker1 # [ 0.574253] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1529worker2 # [ 0.567504] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1530lb # [ 0.537784] pnp: PnP ACPI init1531worker1 # [ 0.575396] NET: Registered PF_UNIX/PF_LOCAL protocol family1532client # [ 0.563801] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1533worker2 # [ 0.568706] TCP: Hash tables configured (established 16384 bind 16384)1534lb # [ 0.538596] ACPI: IRQ 4 override to edge(!), high(!)1535worker1 # [ 0.576325] NET: Registered PF_XDP protocol family1536client # [ 0.564991] TCP: Hash tables configured (established 16384 bind 16384)1537worker2 # [ 0.569790] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1538worker1 # [ 0.577144] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1539lb # [ 0.539543] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1540client # [ 0.566103] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1541worker2 # [ 0.571000] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1542worker1 # [ 0.578108] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1543lb # [ 0.541037] pnp: PnP ACPI: found 5 devices1544client # [ 0.567301] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1545worker1 # [ 0.579058] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1546worker2 # [ 0.572052] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1547client # [ 0.568346] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1548worker2 # [ 0.573222] NET: Registered PF_UNIX/PF_LOCAL protocol family1549worker1 # [ 0.580134] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1550client # [ 0.569495] NET: Registered PF_UNIX/PF_LOCAL protocol family1551worker2 # [ 0.574149] NET: Registered PF_XDP protocol family1552worker1 # [ 0.581229] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1553client # [ 0.570450] NET: Registered PF_XDP protocol family1554worker2 # [ 0.574983] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1555worker1 # [ 0.582333] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1556client # [ 0.571269] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1557worker2 # [ 0.576012] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1558client # [ 0.572236] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1559worker1 # [ 0.584013] ACPI: \_SB_.GSIA: Enabled at IRQ 161560worker2 # [ 0.577025] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1561client # [ 0.573204] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1562worker2 # [ 0.578094] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1563client # [ 0.574256] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1564worker2 # [ 0.579141] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1565worker1 # [ 0.586492] ACPI: \_SB_.GSIB: Enabled at IRQ 171566client # [ 0.575305] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1567lb # [ 0.549023] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1568worker2 # [ 0.580170] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1569lb # [ 0.550481] clocksource: Switched to clocksource acpi_pm1570client # [ 0.576344] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1571worker1 # [ 0.588380] ACPI: \_SB_.GSIC: Enabled at IRQ 181572worker2 # [ 0.581808] ACPI: \_SB_.GSIA: Enabled at IRQ 161573lb # [ 0.551477] NET: Registered PF_INET protocol family1574client # [ 0.577968] ACPI: \_SB_.GSIA: Enabled at IRQ 161575worker1 # [ 0.590298] ACPI: \_SB_.GSID: Enabled at IRQ 191576lb # [ 0.552779] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1577worker2 # [ 0.584021] ACPI: \_SB_.GSIB: Enabled at IRQ 171578client # [ 0.580207] ACPI: \_SB_.GSIB: Enabled at IRQ 171579worker1 # [ 0.591938] PCI: CLS 0 bytes, default 641580worker2 # [ 0.586015] ACPI: \_SB_.GSIC: Enabled at IRQ 181581worker1 # [ 0.592760] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1582client # [ 0.582481] ACPI: \_SB_.GSIC: Enabled at IRQ 181583worker1 # [ 0.594504] Trying to unpack rootfs image as initramfs...1584worker2 # [ 0.587976] ACPI: \_SB_.GSID: Enabled at IRQ 191585client # [ 0.584468] ACPI: \_SB_.GSID: Enabled at IRQ 191586worker2 # [ 0.589649] PCI: CLS 0 bytes, default 641587client # [ 0.586248] PCI: CLS 0 bytes, default 641588worker2 # [ 0.590587] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1589client # [ 0.587197] Trying to unpack rootfs image as initramfs...1590worker2 # [ 0.592258] Trying to unpack rootfs image as initramfs...1591client # [ 0.593263] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1592lb # [ 0.569516] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1593lb # [ 0.571029] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1594lb # [ 0.572342] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1595lb # [ 0.573773] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1596lb # [ 0.574995] TCP: Hash tables configured (established 16384 bind 16384)1597lb # [ 0.576108] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1598lb # [ 0.577335] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1599lb # [ 0.578407] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1600lb # [ 0.579605] NET: Registered PF_UNIX/PF_LOCAL protocol family1601lb # [ 0.580542] NET: Registered PF_XDP protocol family1602lb # [ 0.581380] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1603lb # [ 0.582365] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1604lb # [ 0.583369] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1605lb # [ 0.584436] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1606lb # [ 0.585528] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1607lb # [ 0.586603] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1608lb # [ 0.588333] ACPI: \_SB_.GSIA: Enabled at IRQ 161609lb # [ 0.590718] ACPI: \_SB_.GSIB: Enabled at IRQ 171610lb # [ 0.593119] ACPI: \_SB_.GSIC: Enabled at IRQ 181611lb # [ 0.595161] ACPI: \_SB_.GSID: Enabled at IRQ 191612lb # [ 0.596839] PCI: CLS 0 bytes, default 641613lb # [ 0.597797] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1614lb # [ 0.599506] Trying to unpack rootfs image as initramfs...1615worker1 # [ 0.637239] Initialise system trusted keyrings1616worker1 # [ 0.639875] workingset: timestamp_bits=40 max_order=19 bucket_order=01617worker2 # [ 0.634451] Initialise system trusted keyrings1618worker2 # [ 0.637343] workingset: timestamp_bits=40 max_order=19 bucket_order=01619client # [ 0.634921] Initialise system trusted keyrings1620client # [ 0.638154] workingset: timestamp_bits=40 max_order=19 bucket_order=01621worker1 # [ 0.662159] Key type asymmetric registered1622worker1 # [ 0.664832] Asymmetric key parser 'x509' registered1623worker2 # [ 0.658537] Key type asymmetric registered1624worker1 # [ 0.665684] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1625worker2 # [ 0.659232] Asymmetric key parser 'x509' registered1626worker2 # [ 0.660066] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1627client # [ 0.657369] Key type asymmetric registered1628worker1 # [ 0.668960] io scheduler mq-deadline registered1629worker1 # [ 0.669732] io scheduler kyber registered1630client # [ 0.660104] Asymmetric key parser 'x509' registered1631worker2 # [ 0.665400] io scheduler mq-deadline registered1632worker1 # [ 0.672390] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1633client # [ 0.660972] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1634worker2 # [ 0.666110] io scheduler kyber registered1635worker1 # [ 0.673633] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1636client # [ 0.664205] io scheduler mq-deadline registered1637worker2 # [ 0.668833] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1638client # [ 0.664989] io scheduler kyber registered1639worker2 # [ 0.670058] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1640worker1 # [ 0.679222] Linux agpgart interface v0.1031641worker1 # [ 0.679999] ACPI: bus type drm_connector registered1642lb # [ 0.643246] Initialise system trusted keyrings1643client # [ 0.669114] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1644client # [ 0.670375] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1645worker2 # [ 0.675440] Linux agpgart interface v0.1031646worker2 # [ 0.676151] ACPI: bus type drm_connector registered1647worker1 # [ 0.683236] usbcore: registered new interface driver usbserial_generic1648lb # [ 0.646155] workingset: timestamp_bits=40 max_order=19 bucket_order=01649worker1 # [ 0.684307] usbserial: USB Serial support registered for generic1650worker2 # [ 0.678420] usbcore: registered new interface driver usbserial_generic1651worker2 # [ 0.679473] usbserial: USB Serial support registered for generic1652client # [ 0.675922] Linux agpgart interface v0.1031653client # [ 0.676707] ACPI: bus type drm_connector registered1654worker1 # [ 0.687837] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1655worker1 # [ 0.687837] Please enable it if your BIOS has the CPPC option.1656worker2 # [ 0.682303] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1657worker1 # [ 0.690077] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1658worker2 # [ 0.682303] Please enable it if your BIOS has the CPPC option.1659worker2 # [ 0.684468] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1660client # [ 0.681227] usbcore: registered new interface driver usbserial_generic1661worker1 # [ 0.692963] drop_monitor: Initializing network drop monitor service1662client # [ 0.682334] usbserial: USB Serial support registered for generic1663worker1 # [ 0.694148] NET: Registered PF_INET6 protocol family1664worker2 # [ 0.689465] drop_monitor: Initializing network drop monitor service1665worker1 # [ 0.697320] Segment Routing with IPv61666client # [ 0.685104] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1667worker2 # [ 0.690605] NET: Registered PF_INET6 protocol family1668client # [ 0.685104] Please enable it if your BIOS has the CPPC option.1669client # [ 0.687310] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1670worker2 # [ 0.692720] Segment Routing with IPv61671worker1 # [ 0.699887] In-situ OAM (IOAM) with IPv61672worker1 # [ 0.700936] IPI shorthand broadcast: enabled1673client # [ 0.690306] drop_monitor: Initializing network drop monitor service1674worker2 # [ 0.695311] In-situ OAM (IOAM) with IPv61675client # [ 0.691526] NET: Registered PF_INET6 protocol family1676worker2 # [ 0.696393] IPI shorthand broadcast: enabled1677lb # [ 0.669214] Key type asymmetric registered1678lb # [ 0.669931] Asymmetric key parser 'x509' registered1679client # [ 0.696570] Segment Routing with IPv61680client # [ 0.697264] In-situ OAM (IOAM) with IPv61681lb # [ 0.670791] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1682worker1 # [ 0.708298] sched_clock: Marking stable (573013842, 134796566)->(778202339, -70391931)1683client # [ 0.700398] IPI shorthand broadcast: enabled1684lb # [ 0.674261] io scheduler mq-deadline registered1685worker2 # [ 0.704694] sched_clock: Marking stable (571014599, 133267684)->(776485177, -72202894)1686lb # [ 0.675271] io scheduler kyber registered1687worker1 # [ 0.715055] registered taskstats version 11688worker2 # [ 0.708403] registered taskstats version 11689worker1 # [ 0.716047] Loading compiled-in X.509 certificates1690worker2 # [ 0.709321] Loading compiled-in X.509 certificates1691lb # [ 0.679296] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1692lb # [ 0.680642] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1693client # [ 0.708747] sched_clock: Marking stable (575014565, 133071445)->(780693846, -72607836)1694lb # [ 0.685867] Linux agpgart interface v0.1031695client # [ 0.712176] registered taskstats version 11696lb # [ 0.686676] ACPI: bus type drm_connector registered1697client # [ 0.713165] Loading compiled-in X.509 certificates1698lb # [ 0.689851] usbcore: registered new interface driver usbserial_generic1699lb # [ 0.690975] usbserial: USB Serial support registered for generic1700worker1 # [ 0.732380] Demotion targets for Node 0: null1701lb # [ 0.695120] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1702lb # [ 0.695120] Please enable it if your BIOS has the CPPC option.1703worker2 # [ 0.728296] Demotion targets for Node 0: null1704lb # [ 0.697423] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1705worker1 # [ 0.735859] Key type .fscrypt registered1706worker2 # [ 0.729129] Key type .fscrypt registered1707worker1 # [ 0.736522] Key type fscrypt-provisioning registered1708worker1 # [ 0.737482] ima: No TPM chip found, activating TPM-bypass!1709lb # [ 0.700267] drop_monitor: Initializing network drop monitor service1710lb # [ 0.701496] NET: Registered PF_INET6 protocol family1711worker2 # [ 0.732288] Key type fscrypt-provisioning registered1712worker1 # [ 0.739827] ima: Allocated hash algorithm: sha11713worker2 # [ 0.733172] ima: No TPM chip found, activating TPM-bypass!1714worker1 # [ 0.740605] ima: No architecture policies found1715worker2 # [ 0.734033] ima: Allocated hash algorithm: sha11716worker1 # [ 0.742823] PM: Magic number: 10:316:5301717worker2 # [ 0.736315] ima: No architecture policies found1718client # [ 0.732104] Demotion targets for Node 0: null1719lb # [ 0.706560] Segment Routing with IPv61720worker2 # [ 0.737224] PM: Magic number: 10:316:5301721client # [ 0.732977] Key type .fscrypt registered1722worker1 # [ 0.744424] RAS: Correctable Errors collector initialized.1723lb # [ 0.707248] In-situ OAM (IOAM) with IPv61724client # [ 0.736091] Key type fscrypt-provisioning registered1725lb # [ 0.710394] IPI shorthand broadcast: enabled1726worker2 # [ 0.740903] RAS: Correctable Errors collector initialized.1727client # [ 0.736998] ima: No TPM chip found, activating TPM-bypass!1728client # [ 0.737922] ima: Allocated hash algorithm: sha11729client # [ 0.740118] ima: No architecture policies found1730client # [ 0.741119] PM: Magic number: 10:316:5301731worker1 # [ 0.753435] clk: Disabling unused clocks1732client # [ 0.742800] RAS: Correctable Errors collector initialized.1733worker2 # [ 0.749607] clk: Disabling unused clocks1734lb # [ 0.718771] sched_clock: Marking stable (584015367, 134086659)->(792552057, -74450031)1735worker1 # [ 0.756829] PM: genpd: Disabling unused power domains1736worker2 # [ 0.752306] PM: genpd: Disabling unused power domains1737lb # [ 0.722292] registered taskstats version 11738lb # [ 0.723265] Loading compiled-in X.509 certificates1739client # [ 0.752797] clk: Disabling unused clocks1740client # [ 0.755106] PM: genpd: Disabling unused power domains1741lb # [ 0.739726] Demotion targets for Node 0: null1742lb # [ 0.743149] Key type .fscrypt registered1743lb # [ 0.743837] Key type fscrypt-provisioning registered1744lb # [ 0.744827] ima: No TPM chip found, activating TPM-bypass!1745lb # [ 0.747113] ima: Allocated hash algorithm: sha11746lb # [ 0.747921] ima: No architecture policies found1747lb # [ 0.752112] PM: Magic number: 10:316:5301748lb # [ 0.753767] RAS: Correctable Errors collector initialized.1749lb # [ 0.762819] clk: Disabling unused clocks1750lb # [ 0.765115] PM: genpd: Disabling unused power domains1751worker2 # [ 0.919825] Freeing initrd memory: 29060K1752worker1 # [ 0.928001] Freeing initrd memory: 29056K1753client # [ 0.918266] Freeing initrd memory: 29056K1754worker2 # [ 0.923002] Freeing unused decrypted memory: 2028K1755worker1 # [ 0.931219] Freeing unused decrypted memory: 2028K1756worker2 # [ 0.925595] Freeing unused kernel image (initmem) memory: 3652K1757client # [ 0.921477] Freeing unused decrypted memory: 2028K1758worker2 # [ 0.926664] Write protecting the kernel read-only data: 32768k1759worker1 # [ 0.933903] Freeing unused kernel image (initmem) memory: 3652K1760worker1 # [ 0.934992] Write protecting the kernel read-only data: 32768k1761client # [ 0.924035] Freeing unused kernel image (initmem) memory: 3652K1762worker2 # [ 0.928478] Freeing unused kernel image (text/rodata gap) memory: 1184K1763client # [ 0.925108] Write protecting the kernel read-only data: 32768k1764worker1 # [ 0.936861] Freeing unused kernel image (text/rodata gap) memory: 1184K1765worker2 # [ 0.929931] Freeing unused kernel image (rodata/data gap) memory: 720K1766worker1 # [ 0.938339] Freeing unused kernel image (rodata/data gap) memory: 720K1767client # [ 0.926936] Freeing unused kernel image (text/rodata gap) memory: 1184K1768client # [ 0.928473] Freeing unused kernel image (rodata/data gap) memory: 720K1769lb # [ 0.934138] Freeing initrd memory: 29064K1770lb # [ 0.937317] Freeing unused decrypted memory: 2028K1771lb # [ 0.939921] Freeing unused kernel image (initmem) memory: 3652K1772lb # [ 0.941054] Write protecting the kernel read-only data: 32768k1773lb # [ 0.942978] Freeing unused kernel image (text/rodata gap) memory: 1184K1774lb # [ 0.944529] Freeing unused kernel image (rodata/data gap) memory: 720K1775worker2 # [ 0.980545] x86/mm: Checked W+X mappings: passed, no W+X pages found.1776worker2 # [ 0.981588] Run /init as init process1777worker1 # [ 0.989162] x86/mm: Checked W+X mappings: passed, no W+X pages found.1778worker1 # [ 0.990238] Run /init as init process1779client # [ 0.979478] x86/mm: Checked W+X mappings: passed, no W+X pages found.1780client # [ 0.980828] Run /init as init process1781worker2 # [ 0.992669] systemd[1]: Inserted module 'autofs4'1782worker1 # [ 1.000884] systemd[1]: Inserted module 'autofs4'1783client # [ 0.991898] systemd[1]: Inserted module 'autofs4'1784worker2 # [ 1.012478] fuse: init (API version 7.45)1785worker1 # [ 1.020714] fuse: init (API version 7.45)1786client # [ 1.011950] fuse: init (API version 7.45)1787worker2 # [ 1.019834] ACPI: \_SB_.GSIG: Enabled at IRQ 221788worker1 # [ 1.028123] ACPI: \_SB_.GSIG: Enabled at IRQ 221789worker2 # [ 1.022253] ACPI: \_SB_.GSIH: Enabled at IRQ 231790worker1 # [ 1.030730] ACPI: \_SB_.GSIH: Enabled at IRQ 231791client # [ 1.019350] ACPI: \_SB_.GSIG: Enabled at IRQ 221792worker2 # [ 1.025633] ACPI: \_SB_.GSIE: Enabled at IRQ 201793client # [ 1.021732] ACPI: \_SB_.GSIH: Enabled at IRQ 231794lb # [ 0.995526] x86/mm: Checked W+X mappings: passed, no W+X pages found.1795lb # [ 0.996633] Run /init as init process1796worker1 # [ 1.034785] ACPI: \_SB_.GSIE: Enabled at IRQ 201797worker2 # [ 1.027956] ACPI: \_SB_.GSIF: Enabled at IRQ 211798client # [ 1.025142] ACPI: \_SB_.GSIE: Enabled at IRQ 201799worker1 # [ 1.037106] ACPI: \_SB_.GSIF: Enabled at IRQ 211800client # [ 1.027532] ACPI: \_SB_.GSIF: Enabled at IRQ 211801worker2 # [ 1.033781] virtiofs virtio5: discovered new tag: nix-store1802worker2 # [ 1.035388] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1803worker1 # [ 1.042494] virtiofs virtio5: discovered new tag: nix-store1804worker1 # [ 1.044239] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1805lb # [ 1.007509] systemd[1]: Inserted module 'autofs4'1806client # [ 1.033518] virtiofs virtio5: discovered new tag: nix-store1807client # [ 1.035219] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1808worker2 # [ 1.042872] virtiofs virtio6: discovered new tag: shared1809worker1 # [ 1.051347] virtiofs virtio6: discovered new tag: shared1810worker2 # [ 1.044443] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1811worker1 # [ 1.053018] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1812worker2 # [ 1.047834] virtiofs virtio7: discovered new tag: xchg1813client # [ 1.043818] virtiofs virtio6: discovered new tag: shared1814worker2 # [ 1.049321] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1815worker1 # [ 1.056626] virtiofs virtio7: discovered new tag: xchg1816client # [ 1.045376] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1817worker1 # [ 1.058113] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1818client # [ 1.049036] virtiofs virtio7: discovered new tag: xchg1819client # [ 1.050504] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1820lb # [ 1.027280] fuse: init (API version 7.45)1821lb # [ 1.034638] ACPI: \_SB_.GSIG: Enabled at IRQ 221822lb # [ 1.037192] ACPI: \_SB_.GSIH: Enabled at IRQ 231823lb # [ 1.040177] ACPI: \_SB_.GSIE: Enabled at IRQ 201824worker2 # [ 1.071414] systemd[1]: Successfully made /usr/ read-only.1825lb # [ 1.042170] ACPI: \_SB_.GSIF: Enabled at IRQ 211826worker1 # [ 1.081997] systemd[1]: Successfully made /usr/ read-only.1827lb # [ 1.047296] virtiofs virtio5: discovered new tag: nix-store1828client # [ 1.074153] systemd[1]: Successfully made /usr/ read-only.1829lb # [ 1.048913] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1830lb # [ 1.055298] virtiofs virtio6: discovered new tag: shared1831lb # [ 1.056816] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1832lb # [ 1.060205] virtiofs virtio7: discovered new tag: xchg1833lb # [ 1.061732] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1834lb # [ 1.080880] systemd[1]: Successfully made /usr/ read-only.1835worker2 # [ 1.406976] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1836worker2 # [ 1.412065] systemd[1]: Detected virtualization kvm.1837worker2 # [ 1.412923] systemd[1]: Detected architecture x86-64.1838worker2 # [ 1.413794] systemd[1]: Running in initrd.1839worker1 # [ 1.417698] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1840worker2 # [ 1.414969] systemd[1]: Initializing machine ID from random generator.1841worker1 # [ 1.422756] systemd[1]: Detected virtualization kvm.1842worker2 # [ 1.416160] systemd[1]: Hostname set to <worker2>.1843worker1 # [ 1.423597] systemd[1]: Detected architecture x86-64.1844worker1 # [ 1.424472] systemd[1]: Running in initrd.1845worker1 # [ 1.425648] systemd[1]: Initializing machine ID from random generator.1846worker1 # [ 1.426939] systemd[1]: Hostname set to <worker1>.1847client # [ 1.410922] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1848client # [ 1.423142] systemd[1]: Detected virtualization kvm.1849client # [ 1.425276] systemd[1]: Detected architecture x86-64.1850client # [ 1.427367] systemd[1]: Running in initrd.1851client # [ 1.429815] systemd[1]: Initializing machine ID from random generator.1852client # [ 1.432627] systemd[1]: Hostname set to <client>.1853lb # [ 1.416479] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1854lb # [ 1.421421] systemd[1]: Detected virtualization kvm.1855lb # [ 1.422269] systemd[1]: Detected architecture x86-64.1856lb # [ 1.423073] systemd[1]: Running in initrd.1857lb # [ 1.424128] systemd[1]: Initializing machine ID from random generator.1858lb # [ 1.425341] systemd[1]: Hostname set to <lb>.1859worker1 # [ 1.622674] systemd[1]: bpf-restrict-fs: LSM BPF program attached1860worker2 # [ 1.625778] systemd[1]: bpf-restrict-fs: LSM BPF program attached1861client # [ 1.643899] systemd[1]: bpf-restrict-fs: LSM BPF program attached1862lb # [ 1.620579] systemd[1]: bpf-restrict-fs: LSM BPF program attached1863worker1 # [ 1.662519] systemd[1]: Queued start job for default target Initrd Default Target.1864worker1 # [ 1.667246] systemd[1]: Created slice Slice /system/modprobe.1865worker1 # [ 1.668483] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1866worker1 # [ 1.669881] systemd[1]: Expecting device /dev/disk/by-label/nixos...1867worker1 # [ 1.670961] systemd[1]: Reached target Path Units.1868worker1 # [ 1.671873] systemd[1]: Reached target Slice Units.1869worker1 # [ 1.672693] systemd[1]: Reached target Swaps.1870worker1 # [ 1.673461] systemd[1]: Reached target Timer Units.1871worker1 # [ 1.674433] systemd[1]: Listening on D-Bus System Message Bus Socket.1872worker1 # [ 1.675622] systemd[1]: Listening on Journal Socket (/dev/log).1873worker1 # [ 1.676726] systemd[1]: Listening on Journal Sockets.1874worker1 # [ 1.677689] systemd[1]: Listening on udev Control Socket.1875worker1 # [ 1.678655] systemd[1]: Listening on udev Kernel Socket.1876worker1 # [ 1.679642] systemd[1]: Reached target Socket Units.1877worker1 # [ 1.681554] systemd[1]: Starting Create List of Static Device Nodes...1878worker1 # [ 1.685883] systemd[1]: Starting Load Kernel Module configfs...1879worker2 # [ 1.680067] systemd[1]: Queued start job for default target Initrd Default Target.1880worker2 # [ 1.684674] systemd[1]: Created slice Slice /system/modprobe.1881worker2 # [ 1.685902] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1882worker2 # [ 1.687379] systemd[1]: Expecting device /dev/disk/by-label/nixos...1883client # [ 1.683285] systemd[1]: Queued start job for default target Initrd Default Target.1884worker2 # [ 1.688508] systemd[1]: Reached target Path Units.1885lb # [ 1.658354] systemd[1]: Queued start job for default target Initrd Default Target.1886worker2 # [ 1.689420] systemd[1]: Reached target Slice Units.1887worker2 # [ 1.690258] systemd[1]: Reached target Swaps.1888worker2 # [ 1.691057] systemd[1]: Reached target Timer Units.1889client # [ 1.687435] systemd[1]: Created slice Slice /system/modprobe.1890worker2 # [ 1.692081] systemd[1]: Listening on D-Bus System Message Bus Socket.1891lb # [ 1.662409] systemd[1]: Created slice Slice /system/modprobe.1892worker2 # [ 1.693308] systemd[1]: Listening on Journal Socket (/dev/log).1893client # [ 1.688606] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1894worker2 # [ 1.694403] systemd[1]: Listening on Journal Sockets.1895lb # [ 1.663647] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1896client # [ 1.690011] systemd[1]: Expecting device /dev/disk/by-label/nixos...1897worker2 # [ 1.695409] systemd[1]: Listening on udev Control Socket.1898client # [ 1.691125] systemd[1]: Reached target Path Units.1899lb # [ 1.665061] systemd[1]: Expecting device /dev/disk/by-label/nixos...1900worker2 # [ 1.696419] systemd[1]: Listening on udev Kernel Socket.1901client # [ 1.691994] systemd[1]: Reached target Slice Units.1902lb # [ 1.666135] systemd[1]: Reached target Path Units.1903client # [ 1.692874] systemd[1]: Reached target Swaps.1904worker2 # [ 1.697338] systemd[1]: Reached target Socket Units.1905lb # [ 1.666942] systemd[1]: Reached target Slice Units.1906client # [ 1.693626] systemd[1]: Reached target Timer Units.1907lb # [ 1.667822] systemd[1]: Reached target Swaps.1908lb # [ 1.668640] systemd[1]: Reached target Timer Units.1909worker2 # [ 1.699143] systemd[1]: Starting Create List of Static Device Nodes...1910client # [ 1.694697] systemd[1]: Listening on D-Bus System Message Bus Socket.1911worker1 # [ 1.695325] systemd[1]: Starting Journal Service...1912lb # [ 1.669600] systemd[1]: Listening on D-Bus System Message Bus Socket.1913client # [ 1.695900] systemd[1]: Listening on Journal Socket (/dev/log).1914lb # [ 1.670794] systemd[1]: Listening on Journal Socket (/dev/log).1915client # [ 1.697048] systemd[1]: Listening on Journal Sockets.1916lb # [ 1.671981] systemd[1]: Listening on Journal Sockets.1917client # [ 1.698058] systemd[1]: Listening on udev Control Socket.1918worker2 # [ 1.703354] systemd[1]: Starting Load Kernel Module configfs...1919lb # [ 1.672987] systemd[1]: Listening on udev Control Socket.1920client # [ 1.699109] systemd[1]: Listening on udev Kernel Socket.1921client # [ 1.700100] systemd[1]: Reached target Socket Units.1922lb # [ 1.674001] systemd[1]: Listening on udev Kernel Socket.1923lb # [ 1.675034] systemd[1]: Reached target Socket Units.1924client # [ 1.701988] systemd[1]: Starting Create List of Static Device Nodes...1925lb # [ 1.676851] systemd[1]: Starting Create List of Static Device Nodes...1926client # [ 1.706023] systemd[1]: Starting Load Kernel Module configfs...1927lb # [ 1.682183] systemd[1]: Starting Load Kernel Module configfs...1928worker1 # [ 1.721990] systemd[1]: Starting Load Kernel Modules...1929worker1 # [ 1.725915] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1930worker2 # [ 1.714356] systemd[1]: Starting Journal Service...1931worker1 # [ 1.732993] systemd[1]: Starting Coldplug All udev Devices...1932client # [ 1.715748] systemd[1]: Starting Journal Service...1933lb # [ 1.690696] systemd[1]: Starting Journal Service...1934worker1 # [ 1.744689] systemd-journald[66]: Collecting audit messages is disabled.1935worker2 # [ 1.737933] systemd[1]: Starting Load Kernel Modules...1936worker1 # [ 1.747953] systemd[1]: Finished Create List of Static Device Nodes.1937worker2 # [ 1.742420] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1938client # [ 1.742275] systemd[1]: Starting Load Kernel Modules...1939lb # [ 1.716294] systemd[1]: Starting Load Kernel Modules...1940worker1 # [ 1.755497] systemd[1]: modprobe@configfs.service: Deactivated successfully.1941lb # [ 1.720208] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1942client # [ 1.746203] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1943worker2 # [ 1.751539] systemd[1]: Starting Coldplug All udev Devices...1944worker1 # [ 1.761259] systemd[1]: Finished Load Kernel Module configfs.1945client # [ 1.753154] systemd[1]: Starting Coldplug All udev Devices...1946lb # [ 1.728285] systemd[1]: Starting Coldplug All udev Devices...1947worker1 # [ 1.767262] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1948worker2 # [ 1.762854] systemd-journald[66]: Collecting audit messages is disabled.1949worker2 # [ 1.764560] systemd[1]: Finished Create List of Static Device Nodes.1950lb # [ 1.739407] systemd-journald[66]: Collecting audit messages is disabled.1951client # [ 1.765656] systemd-journald[66]: Collecting audit messages is disabled.1952lb # [ 1.741189] systemd[1]: Finished Create List of Static Device Nodes.1953worker1 # [ 1.778885] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1954worker2 # [ 1.772435] systemd[1]: modprobe@configfs.service: Deactivated successfully.1955client # [ 1.769436] systemd[1]: Finished Create List of Static Device Nodes.1956worker2 # [ 1.776685] systemd[1]: Finished Load Kernel Module configfs.1957client # [ 1.773278] systemd[1]: modprobe@configfs.service: Deactivated successfully.1958lb # [ 1.747510] systemd[1]: modprobe@configfs.service: Deactivated successfully.1959worker2 # [ 1.781699] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1960client # [ 1.780484] systemd[1]: Finished Load Kernel Module configfs.1961lb # [ 1.754511] systemd[1]: Finished Load Kernel Module configfs.1962worker1 # [ 1.791232] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1963client # [ 1.786528] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1964lb # [ 1.760515] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1965worker1 # [ 1.799848] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1966worker2 # [ 1.794353] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1967lb # [ 1.771164] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1968client # [ 1.798185] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1969worker2 # [ 1.806415] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1970lb # [ 1.780641] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1971client # [ 1.809525] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1972worker2 # [ 1.814564] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1973worker1 # [ 1.822440] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1974lb # [ 1.789134] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1975client # [ 1.819122] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1976worker1 # [ 1.831019] systemd[1]: Starting Create Static Device Nodes in /dev...1977worker1 # [ 1.842880] systemd[1]: Finished Load Kernel Modules.1978worker2 # [ 1.835486] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1979lb # [ 1.812676] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1980worker2 # [ 1.844571] systemd[1]: Starting Create Static Device Nodes in /dev...1981worker1 # [ 1.853542] systemd[1]: Starting Apply Kernel Variables...1982client # [ 1.842711] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1983lb # [ 1.821314] systemd[1]: Starting Create Static Device Nodes in /dev...1984client # [ 1.852316] systemd[1]: Starting Create Static Device Nodes in /dev...1985worker2 # [ 1.858362] systemd[1]: Finished Load Kernel Modules.1986lb # [ 1.832162] systemd[1]: Finished Load Kernel Modules.1987worker1 # [ 1.869994] systemd[1]: Started Journal Service.1988worker1 # [ 1.737057] systemd-modules-load[68]: Inserted module 'dm_mod'1989worker1 # [ 1.738380] systemd-modules-load[68]: Inserted module 'virtio_balloon'1990worker1 # [ 1.740256] systemd-modules-load[68]: Inserted module 'virtio_gpu'1991worker2 # [ 1.868386] systemd[1]: Starting Apply Kernel Variables...1992client # [ 1.864207] systemd[1]: Finished Load Kernel Modules.1993lb # [ 1.841333] systemd[1]: Starting Apply Kernel Variables...1994worker1 # [ 1.747125] systemd[1]: Finished Create Static Device Nodes in /dev.1995client # [ 1.873174] systemd[1]: Starting Apply Kernel Variables...1996worker1 # [ 1.752912] systemd[1]: Reached target Preparation for Local File Systems.1997worker1 # [ 1.757080] systemd[1]: Reached target Local File Systems.1998worker1 # [ 1.759069] systemd[1]: Starting Create System Files and Directories...1999worker2 # [ 1.886875] systemd[1]: Finished Create Static Device Nodes in /dev.2000worker2 # [ 1.755148] systemd-modules-load[67]: Inserted module 'dm_mod'
loading 2,669 hidden lines…
4670worker2 # [ 37.163695] nix-daemon[867]: reaped child process 981, status = succeeded4671worker2 # [ 37.240584] nix-daemon[867]: accepted connection from pid 997, user nix-grpc-daemon (trusted)4672worker2 # [ 37.254422] nix-daemon[867]: reaped child process 1000, status = succeeded4673lb # [ 37.229428] niks3-server[949]: 2026/09/22 11:32:55 INFO Received uploads request method=POST path=/api/pending_closures4674worker2 # [ 37.271154] nix-grpc-daemon[882]: time=2026-09-22T11:32:56.437Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4675worker2 # [ 37.272927] nix-grpc-daemon[882]: time=2026-09-22T11:32:56.438Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4676lb # [ 37.271703] niks3-server[949]: 2026/09/22 11:32:55 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4677lb # [ 37.282287] niks3-server[949]: 2026/09/22 11:32:55 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4678lb # [ 37.291417] niks3-server[949]: 2026/09/22 11:32:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4679lb # [ 37.297142] niks3-server[949]: 2026/09/22 11:32:55 INFO Signed narinfos id=6 count=14680worker2 # [ 37.332451] nix-grpc-daemon[882]: time=2026-09-22T11:32:56.498Z level=INFO msg="Uploading 1 narinfos"4681lb # [ 37.308378] niks3-server[949]: 2026/09/22 11:32:55 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4682lb # [ 37.313046] niks3-server[949]: 2026/09/22 11:32:55 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4683lb # [ 37.317421] niks3-server[949]: 2026/09/22 11:32:55 INFO Completed upload id=64684worker2 # [ 37.349157] nix-grpc-daemon[882]: time=2026-09-22T11:32:56.515Z level=INFO msg="Upload complete. (196ms)"4685lb # [ 37.320060] niks3-server[949]: 2026/09/22 11:32:55 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4686worker2 # [ 37.356453] nix-grpc-daemon[783]: ts=2026-09-22T11:32:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14687worker2 # [ 37.372123] nix-daemon[867]: reaped child process 979, status = succeeded4688client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&2, in 4.96 seconds)4689client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath4690client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath, in 0.11 seconds)4691client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4692client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.01 seconds)4693client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4694client # copying 3 paths...4695client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4696client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4697client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4698client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.13 seconds)4699(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 5.26 seconds)4700subtest: repeat is answered Cached by the scheduler without building4701worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4702worker1 # finding garbage collector roots...4703worker1 # deleting specified paths...4704worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4705worker1 # deleting unused links...4706worker1 # note: hard linking is currently saving 0.0 KiB4707worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.18 seconds)4708worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4709worker2 # finding garbage collector roots...4710worker2 # deleting specified paths...4711worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4712worker2 # deleting unused links...4713worker2 # note: hard linking is currently saving 0.0 KiB4714worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.13 seconds)4715client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&24716client # don't know how to build these paths:4717worker1 # [ 38.189786] nix-grpc-daemon[780]: ts=2026-09-22T11:32:56Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:462024718client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4719worker1 # [ 38.192810] nix-grpc-daemon[780]: ts=2026-09-22T11:32:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462024720worker2 # [ 38.215513] nix-grpc-daemon[783]: ts=2026-09-22T11:32:57Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14721client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&2, in 0.23 seconds)4722client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath4723client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath, in 0.10 seconds)4724lb: must succeed: curl -sf localhost:9901/clusters4725lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4726lb: must succeed: curl -sf localhost:9901/clusters4727lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)4728worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4729worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4730(finished: subtest: repeat is answered Cached by the scheduler without building, in 0.80 seconds)4731subtest: inputs already in the cache are not sent to the scheduler4732lb: must succeed: curl -sf localhost:9901/clusters4733lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4734lb: must succeed: curl -sf localhost:9901/clusters4735lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)4736worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4737worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.04 seconds)4738client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b >&24739worker2 # [ 38.724896] nix-grpc-daemon[783]: ts=2026-09-22T11:32:57Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558324740client # don't know how to build these paths:4741client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4742worker2 # [ 38.733535] nix-grpc-daemon[783]: ts=2026-09-22T11:32:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558324743worker2 # [ 38.743293] nix-grpc-daemon[882]: time=2026-09-22T11:32:57.908Z level=INFO msg="All 2 paths already cached"4744worker2 # [ 38.755591] nix-grpc-daemon[783]: ts=2026-09-22T11:32:57Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04745worker2 # [ 38.761947] nix-grpc-daemon[783]: ts=2026-09-22T11:32:57Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.4:50051 clients=14746worker2 # [ 38.766925] nix-grpc-daemon[783]: ts=2026-09-22T11:32:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558324747worker2 # [ 38.777923] nix-grpc-daemon[783]: ts=2026-09-22T11:32:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558324748worker2 # [ 38.791057] nix-grpc-daemon[882]: time=2026-09-22T11:32:57.957Z level=INFO msg="All 2 paths already cached"4749client # copying 1 paths to 'lb:50051'...4750client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4751worker2 # [ 38.799783] nix-grpc-daemon[783]: ts=2026-09-22T11:32:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558324752worker2 # [ 38.811497] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4753worker2 # [ 38.906649] nix-daemon[867]: accepted connection from pid 1030, user nix-grpc-daemon (trusted)4754worker2 # [ 38.920631] nix-daemon[867]: reaped child process 1033, status = succeeded4755worker2 # [ 39.003976] nix-daemon[867]: accepted connection from pid 1036, user nix-grpc-daemon (trusted)4756worker2 # [ 39.018894] nix-daemon[867]: reaped child process 1039, status = succeeded4757lb # [ 38.995721] niks3-server[949]: 2026/09/22 11:32:57 INFO Received uploads request method=POST path=/api/pending_closures4758worker2 # [ 39.034313] nix-grpc-daemon[882]: time=2026-09-22T11:32:58.200Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4759worker2 # [ 39.036336] nix-grpc-daemon[882]: time=2026-09-22T11:32:58.200Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4760lb # [ 39.024707] niks3-server[949]: 2026/09/22 11:32:57 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4761lb # [ 39.044099] niks3-server[949]: 2026/09/22 11:32:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4762lb # [ 39.050377] niks3-server[949]: 2026/09/22 11:32:57 INFO Signed narinfos id=7 count=14763worker2 # [ 39.084726] nix-grpc-daemon[882]: time=2026-09-22T11:32:58.250Z level=INFO msg="Uploading 1 narinfos"4764lb # [ 39.056467] niks3-server[949]: 2026/09/22 11:32:57 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4765lb # [ 39.068241] niks3-server[949]: 2026/09/22 11:32:57 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4766lb # [ 39.072159] niks3-server[949]: 2026/09/22 11:32:57 INFO Completed upload id=74767worker2 # [ 39.103440] nix-grpc-daemon[882]: time=2026-09-22T11:32:58.269Z level=INFO msg="Upload complete. (282ms)"4768lb # [ 39.075668] niks3-server[949]: 2026/09/22 11:32:57 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4769worker2 # [ 39.107892] nix-grpc-daemon[783]: ts=2026-09-22T11:32:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 paths=1 nar_bytes_in=7524770worker2 # [ 39.112505] nix-grpc-daemon[783]: ts=2026-09-22T11:32:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558324771worker2 # [ 39.123103] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4772worker2 # [ 39.133165] nix-daemon[867]: reaped child process 1028, status = succeeded4773worker2 # [ 39.135929] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4774client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4775client # farm-top> worker2: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4776client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4777client # farm-top> LOG-farm-top-t1b4778worker2 # [ 39.757728] nix-daemon[867]: reaped child process 1043, status = succeeded4779worker2 # [ 39.843865] nix-daemon[867]: accepted connection from pid 1059, user nix-grpc-daemon (trusted)4780worker2 # [ 39.856368] nix-daemon[867]: reaped child process 1062, status = succeeded4781lb # [ 39.833597] niks3-server[949]: 2026/09/22 11:32:58 INFO Received uploads request method=POST path=/api/pending_closures4782worker2 # [ 39.874438] nix-grpc-daemon[882]: time=2026-09-22T11:32:59.040Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4783worker2 # [ 39.876168] nix-grpc-daemon[882]: time=2026-09-22T11:32:59.042Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4784lb # [ 39.880345] niks3-server[949]: 2026/09/22 11:32:58 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4785lb # [ 39.890577] niks3-server[949]: 2026/09/22 11:32:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4786lb # [ 39.897259] niks3-server[949]: 2026/09/22 11:32:58 INFO Signed narinfos id=8 count=14787worker2 # [ 39.932109] nix-grpc-daemon[882]: time=2026-09-22T11:32:59.098Z level=INFO msg="Uploading 1 narinfos"4788lb # [ 39.901519] niks3-server[949]: 2026/09/22 11:32:58 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4789lb # [ 39.905085] niks3-server[949]: 2026/09/22 11:32:58 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4790lb # [ 39.914379] niks3-server[949]: 2026/09/22 11:32:58 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4791lb # [ 39.919127] niks3-server[949]: 2026/09/22 11:32:58 INFO Completed upload id=84792lb # [ 39.921640] niks3-server[949]: 2026/09/22 11:32:58 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4793worker2 # [ 39.952197] nix-grpc-daemon[882]: time=2026-09-22T11:32:59.118Z level=INFO msg="Upload complete. (208ms)"4794worker2 # [ 39.958628] nix-grpc-daemon[783]: ts=2026-09-22T11:32:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14795worker2 # [ 39.971168] nix-daemon[867]: reaped child process 1041, status = succeeded4796client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.41 seconds)4797client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b outPath4798client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.11 seconds)4799lb: must succeed: curl -sf localhost:9901/clusters4800lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4801lb: must succeed: curl -sf localhost:9901/clusters4802lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4803worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4804worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.02 seconds)4805(finished: subtest: inputs already in the cache are not sent to the scheduler, in 1.71 seconds)4806subtest: outputs read back through any worker carry the cache signature4807client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig >&24808worker1 # [ 40.352079] nix-grpc-daemon[780]: ts=2026-09-22T11:32:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:462024809client # don't know how to build these paths:4810client # /nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv4811worker2 # [ 40.352799] nix-grpc-daemon[783]: ts=2026-09-22T11:32:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558324812worker2 # [ 40.364308] nix-grpc-daemon[783]: ts=2026-09-22T11:32:59Z level=debug event=want client=4 drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv cached=04813worker2 # [ 40.366275] nix-grpc-daemon[783]: ts=2026-09-22T11:32:59Z level=debug event=assigned drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv worker=192.168.1.3:50051 clients=14814worker1 # [ 40.373668] nix-grpc-daemon[780]: ts=2026-09-22T11:32:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462024815worker1 # [ 40.381320] nix-grpc-daemon[780]: ts=2026-09-22T11:32:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462024816client # copying 1 paths to 'lb:50051'...4817client # copying path '/nix/store/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv' to 'lb:50051'...4818worker1 # [ 40.388514] nix-grpc-daemon[780]: ts=2026-09-22T11:32:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462024819worker1 # [ 40.395510] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4820worker1 # [ 40.406883] nix-grpc-daemon[780]: ts=2026-09-22T11:32:58Z level=info event=niks3_push_started pid=8724821worker1 # [ 40.532524] nix-grpc-daemon[872]: time=2026-09-22T11:32:58.896Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""4822worker1 # [ 40.623972] nix-daemon[844]: accepted connection from pid 884, user nix-grpc-daemon (trusted)4823worker1 # [ 40.637677] nix-daemon[844]: reaped child process 888, status = succeeded4824worker1 # [ 40.712328] nix-daemon[844]: accepted connection from pid 890, user nix-grpc-daemon (trusted)4825worker1 # [ 40.727214] nix-daemon[844]: reaped child process 893, status = succeeded4826lb # [ 40.691827] niks3-server[949]: 2026/09/22 11:32:59 INFO Received uploads request method=POST path=/api/pending_closures4827worker1 # [ 40.737859] nix-grpc-daemon[872]: time=2026-09-22T11:32:59.102Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4828worker1 # [ 40.740803] nix-grpc-daemon[872]: time=2026-09-22T11:32:59.103Z level=INFO msg="Uploading lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv (504B)"4829lb # [ 40.723658] niks3-server[949]: 2026/09/22 11:32:59 INFO Registered completed upload object_key=nar/0bddczcnpndyzwmafx9yqfqrkrp1jg9rr5hsixz6rpy99kg84g1g.nar.zst4830lb # [ 40.735650] niks3-server[949]: 2026/09/22 11:32:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4831worker1 # [ 40.782902] nix-grpc-daemon[872]: time=2026-09-22T11:32:59.147Z level=INFO msg="Uploading 1 narinfos"4832lb # [ 40.746960] niks3-server[949]: 2026/09/22 11:32:59 INFO Signed narinfos id=9 count=14833lb # [ 40.756124] niks3-server[949]: 2026/09/22 11:32:59 INFO Registered completed upload object_key=lqi1rvcqjj8q9l279yzgrgaghp394zs5.ls4834lb # [ 40.773947] niks3-server[949]: 2026/09/22 11:32:59 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4835lb # [ 40.778510] niks3-server[949]: 2026/09/22 11:32:59 INFO Completed upload id=94836lb # [ 40.781649] niks3-server[949]: 2026/09/22 11:32:59 INFO Registered completed upload object_key=lqi1rvcqjj8q9l279yzgrgaghp394zs5.narinfo4837worker1 # [ 40.817550] nix-grpc-daemon[872]: time=2026-09-22T11:32:59.181Z level=INFO msg="Upload complete. (280ms)"4838worker1 # [ 40.822821] nix-grpc-daemon[780]: ts=2026-09-22T11:32:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 paths=1 nar_bytes_in=5044839worker1 # [ 40.833223] nix-grpc-daemon[780]: ts=2026-09-22T11:32:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462024840worker1 # [ 40.844109] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4841worker1 # [ 40.848931] nix-daemon[844]: reaped child process 870, status = succeeded4842worker1 # [ 40.856896] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4843client # building '/nix/store/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv' on lb:50051...4844client # farm-a> node-a: building lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv4845client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4846client # farm-a> LOG-farm-a-sig4847worker1 # [ 42.096662] nix-daemon[844]: reaped child process 897, status = succeeded4848worker1 # [ 42.178566] nix-daemon[844]: accepted connection from pid 913, user nix-grpc-daemon (trusted)4849worker1 # [ 42.190852] nix-daemon[844]: reaped child process 916, status = succeeded4850lb # [ 42.162801] niks3-server[949]: 2026/09/22 11:33:00 INFO Received uploads request method=POST path=/api/pending_closures4851worker1 # [ 42.208990] nix-grpc-daemon[872]: time=2026-09-22T11:33:00.573Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4852worker1 # [ 42.210697] nix-grpc-daemon[872]: time=2026-09-22T11:33:00.575Z level=INFO msg="Uploading qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a (128B)"4853lb # [ 42.205860] niks3-server[949]: 2026/09/22 11:33:00 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst4854lb # [ 42.210835] niks3-server[949]: 2026/09/22 11:33:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4855lb # [ 42.214069] niks3-server[949]: 2026/09/22 11:33:00 INFO Signed narinfos id=10 count=14856worker1 # [ 42.252631] nix-grpc-daemon[872]: time=2026-09-22T11:33:00.616Z level=INFO msg="Uploading 1 narinfos"4857lb # [ 42.219064] niks3-server[949]: 2026/09/22 11:33:00 INFO Registered completed upload object_key=log/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv4858lb # [ 42.223123] niks3-server[949]: 2026/09/22 11:33:00 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.ls4859lb # [ 42.225384] niks3-server[949]: 2026/09/22 11:33:00 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4860lb # [ 42.230971] niks3-server[949]: 2026/09/22 11:33:00 INFO Completed upload id=104861worker1 # [ 42.267695] nix-grpc-daemon[872]: time=2026-09-22T11:33:00.632Z level=INFO msg="Upload complete. (202ms)"4862lb # [ 42.235076] niks3-server[949]: 2026/09/22 11:33:00 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.narinfo4863worker1 # [ 42.270258] nix-grpc-daemon[780]: ts=2026-09-22T11:33:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=1 drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv assign_id=5 outputs=14864worker2 # [ 42.271858] nix-grpc-daemon[783]: ts=2026-09-22T11:33:01Z level=debug event=want client=4 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv cached=04865worker1 # [ 42.280212] nix-daemon[844]: reaped child process 895, status = succeeded4866worker2 # [ 42.273879] nix-grpc-daemon[783]: ts=2026-09-22T11:33:01Z level=debug event=assigned drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv worker=192.168.1.3:50051 clients=14867worker1 # [ 42.323326] nix-grpc-daemon[780]: ts=2026-09-22T11:33:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462024868worker1 # [ 42.334128] nix-grpc-daemon[780]: ts=2026-09-22T11:33:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462024869worker1 # [ 42.344411] nix-grpc-daemon[872]: time=2026-09-22T11:33:00.708Z level=INFO msg="All 1 paths already cached"4870client # copying 1 paths to 'lb:50051'...4871client # copying path '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' to 'lb:50051'...4872worker1 # [ 42.352945] nix-grpc-daemon[780]: ts=2026-09-22T11:33:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462024873worker1 # [ 42.362368] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4874worker1 # [ 42.461242] nix-daemon[844]: accepted connection from pid 920, user nix-grpc-daemon (trusted)4875worker1 # [ 42.475079] nix-daemon[844]: reaped child process 923, status = succeeded4876worker1 # [ 42.557763] nix-daemon[844]: accepted connection from pid 925, user nix-grpc-daemon (trusted)4877worker1 # [ 42.570088] nix-daemon[844]: reaped child process 928, status = succeeded4878lb # [ 42.540917] niks3-server[949]: 2026/09/22 11:33:00 INFO Received uploads request method=POST path=/api/pending_closures4879worker1 # [ 42.582504] nix-grpc-daemon[872]: time=2026-09-22T11:33:00.946Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4880worker1 # [ 42.583899] nix-grpc-daemon[872]: time=2026-09-22T11:33:00.946Z level=INFO msg="Uploading 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv (624B)"4881lb # [ 42.555152] niks3-server[949]: 2026/09/22 11:33:00 INFO Registered completed upload object_key=nar/1m1mvw8bmb91glp26qj699lxqv98q2iz8wm4ik3miz0xj6vi3mw6.nar.zst4882lb # [ 42.558977] niks3-server[949]: 2026/09/22 11:33:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4883lb # [ 42.561341] niks3-server[949]: 2026/09/22 11:33:00 INFO Signed narinfos id=11 count=14884worker1 # [ 42.598409] nix-grpc-daemon[872]: time=2026-09-22T11:33:00.962Z level=INFO msg="Uploading 1 narinfos"4885lb # [ 42.563775] niks3-server[949]: 2026/09/22 11:33:00 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.ls4886lb # [ 42.568771] niks3-server[949]: 2026/09/22 11:33:00 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4887lb # [ 42.572631] niks3-server[949]: 2026/09/22 11:33:00 INFO Completed upload id=114888worker1 # [ 42.609446] nix-grpc-daemon[872]: time=2026-09-22T11:33:00.973Z level=INFO msg="Upload complete. (235ms)"4889lb # [ 42.574391] niks3-server[949]: 2026/09/22 11:33:00 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.narinfo4890worker1 # [ 42.611599] nix-grpc-daemon[780]: ts=2026-09-22T11:33:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 paths=1 nar_bytes_in=6244891worker1 # [ 42.616391] nix-grpc-daemon[780]: ts=2026-09-22T11:33:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462024892worker1 # [ 42.620874] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4893worker1 # [ 42.622965] nix-daemon[844]: reaped child process 918, status = succeeded4894worker1 # [ 42.627929] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4895client # building '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' on lb:50051...4896client # farm-b> node-a: building 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv4897client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4898client # farm-b> LOG-farm-b-sig4899worker1 # [ 43.456445] nix-daemon[844]: reaped child process 932, status = succeeded4900worker1 # [ 43.531557] nix-daemon[844]: accepted connection from pid 948, user nix-grpc-daemon (trusted)4901worker1 # [ 43.542671] nix-daemon[844]: reaped child process 951, status = succeeded4902lb # [ 43.514899] niks3-server[949]: 2026/09/22 11:33:01 INFO Received uploads request method=POST path=/api/pending_closures4903worker1 # [ 43.561188] nix-grpc-daemon[872]: time=2026-09-22T11:33:01.925Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4904worker1 # [ 43.562655] nix-grpc-daemon[872]: time=2026-09-22T11:33:01.927Z level=INFO msg="Uploading 2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b (176B)"4905lb # [ 43.554187] niks3-server[949]: 2026/09/22 11:33:01 INFO Registered completed upload object_key=nar/0y3xlf15z3z540fgrc9sgnn2rl0aj3w6x9mjqd50hhqgmaxbiivd.nar.zst4906lb # [ 43.564853] niks3-server[949]: 2026/09/22 11:33:01 INFO Registered completed upload object_key=log/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv4907lb # [ 43.571139] niks3-server[949]: 2026/09/22 11:33:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4908lb # [ 43.576930] niks3-server[949]: 2026/09/22 11:33:01 INFO Signed narinfos id=12 count=14909worker1 # [ 43.617194] nix-grpc-daemon[872]: time=2026-09-22T11:33:01.981Z level=INFO msg="Uploading 1 narinfos"4910lb # [ 43.590806] niks3-server[949]: 2026/09/22 11:33:01 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4911lb # [ 43.597171] niks3-server[949]: 2026/09/22 11:33:01 INFO Completed upload id=124912worker1 # [ 43.634496] nix-grpc-daemon[872]: time=2026-09-22T11:33:01.998Z level=INFO msg="Upload complete. (200ms)"4913lb # [ 43.599786] niks3-server[949]: 2026/09/22 11:33:01 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.ls4914lb # [ 43.602107] niks3-server[949]: 2026/09/22 11:33:01 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.narinfo4915worker2 # [ 43.641049] nix-grpc-daemon[783]: ts=2026-09-22T11:33:02Z level=debug event=want client=4 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv cached=04916worker1 # [ 43.641137] nix-grpc-daemon[780]: ts=2026-09-22T11:33:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=1 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv assign_id=6 outputs=14917worker2 # [ 43.642948] nix-grpc-daemon[783]: ts=2026-09-22T11:33:02Z level=debug event=assigned drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv worker=192.168.1.3:50051 clients=14918worker1 # [ 43.656970] nix-daemon[844]: reaped child process 930, status = succeeded4919worker1 # [ 43.690884] nix-grpc-daemon[780]: ts=2026-09-22T11:33:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462024920worker1 # [ 43.703458] nix-grpc-daemon[780]: ts=2026-09-22T11:33:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462024921worker1 # [ 43.717487] nix-grpc-daemon[872]: time=2026-09-22T11:33:02.081Z level=INFO msg="All 2 paths already cached"4922client # copying 1 paths to 'lb:50051'...4923client # copying path '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' to 'lb:50051'...4924worker1 # [ 43.726651] nix-grpc-daemon[780]: ts=2026-09-22T11:33:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462024925worker1 # [ 43.735318] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4926worker1 # [ 43.834511] nix-daemon[844]: accepted connection from pid 955, user nix-grpc-daemon (trusted)4927worker1 # [ 43.847707] nix-daemon[844]: reaped child process 958, status = succeeded4928worker1 # [ 43.932553] nix-daemon[844]: accepted connection from pid 960, user nix-grpc-daemon (trusted)4929worker1 # [ 43.944327] nix-daemon[844]: reaped child process 963, status = succeeded4930lb # [ 43.916389] niks3-server[949]: 2026/09/22 11:33:02 INFO Received uploads request method=POST path=/api/pending_closures4931worker1 # [ 43.962781] nix-grpc-daemon[872]: time=2026-09-22T11:33:02.327Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4932worker1 # [ 43.964182] nix-grpc-daemon[872]: time=2026-09-22T11:33:02.327Z level=INFO msg="Uploading cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv (752B)"4933lb # [ 43.945468] niks3-server[949]: 2026/09/22 11:33:02 INFO Registered completed upload object_key=nar/0w21abkvlzg18nqkl2x4zxs7d0376kbld1n0fkr68vcpqpq1g5zg.nar.zst4934lb # [ 43.957146] niks3-server[949]: 2026/09/22 11:33:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4935lb # [ 43.961712] niks3-server[949]: 2026/09/22 11:33:02 INFO Signed narinfos id=13 count=14936worker1 # [ 44.001132] nix-grpc-daemon[872]: time=2026-09-22T11:33:02.365Z level=INFO msg="Uploading 1 narinfos"4937lb # [ 43.973188] niks3-server[949]: 2026/09/22 11:33:02 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4938lb # [ 43.979326] niks3-server[949]: 2026/09/22 11:33:02 INFO Completed upload id=134939worker1 # [ 44.016181] nix-grpc-daemon[872]: time=2026-09-22T11:33:02.380Z level=INFO msg="Upload complete. (267ms)"4940worker1 # [ 44.018165] nix-grpc-daemon[780]: ts=2026-09-22T11:33:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 paths=1 nar_bytes_in=7524941lb # [ 43.984479] niks3-server[949]: 2026/09/22 11:33:02 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.ls4942worker1 # [ 44.021443] nix-grpc-daemon[780]: ts=2026-09-22T11:33:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462024943lb # [ 43.987873] niks3-server[949]: 2026/09/22 11:33:02 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.narinfo4944worker1 # [ 44.029063] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4945worker1 # [ 44.030991] nix-daemon[844]: reaped child process 953, status = succeeded4946worker1 # [ 44.036169] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4947client # building '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' on lb:50051...4948client # farm-top> node-a: building cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv4949client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4950client # farm-top> LOG-farm-top-sig4951worker1 # [ 45.275979] nix-daemon[844]: reaped child process 967, status = succeeded4952worker1 # [ 45.356613] nix-daemon[844]: accepted connection from pid 983, user nix-grpc-daemon (trusted)4953worker1 # [ 45.368951] nix-daemon[844]: reaped child process 986, status = succeeded4954lb # [ 45.340394] niks3-server[949]: 2026/09/22 11:33:03 INFO Received uploads request method=POST path=/api/pending_closures4955worker1 # [ 45.385791] nix-grpc-daemon[872]: time=2026-09-22T11:33:03.750Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4956worker1 # [ 45.387306] nix-grpc-daemon[872]: time=2026-09-22T11:33:03.751Z level=INFO msg="Uploading dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top (232B)"4957lb # [ 45.379724] niks3-server[949]: 2026/09/22 11:33:03 INFO Registered completed upload object_key=log/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv4958lb # [ 45.390520] niks3-server[949]: 2026/09/22 11:33:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4959lb # [ 45.397735] niks3-server[949]: 2026/09/22 11:33:03 INFO Signed narinfos id=14 count=14960worker1 # [ 45.437461] nix-grpc-daemon[872]: time=2026-09-22T11:33:03.801Z level=INFO msg="Uploading 1 narinfos"4961lb # [ 45.405230] niks3-server[949]: 2026/09/22 11:33:03 INFO Registered completed upload object_key=nar/0h4zvhqjxsvdqli2l6i5ajh5sh40hz57gzchdirnvbisq2aks0k8.nar.zst4962lb # [ 45.410196] niks3-server[949]: 2026/09/22 11:33:03 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.ls4963lb # [ 45.418476] niks3-server[949]: 2026/09/22 11:33:03 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4964lb # [ 45.423355] niks3-server[949]: 2026/09/22 11:33:03 INFO Completed upload id=144965worker1 # [ 45.460493] nix-grpc-daemon[872]: time=2026-09-22T11:33:03.824Z level=INFO msg="Upload complete. (215ms)"4966lb # [ 45.427463] niks3-server[949]: 2026/09/22 11:33:03 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.narinfo4967worker1 # [ 45.466504] nix-grpc-daemon[780]: ts=2026-09-22T11:33:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=1 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv assign_id=7 outputs=14968worker1 # [ 45.481350] nix-daemon[844]: reaped child process 965, status = succeeded4969client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig >&2, in 5.28 seconds)4970client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath4971client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath, in 0.10 seconds)4972client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top4973worker1 # [ 45.688501] nix-grpc-daemon[780]: ts=2026-09-22T11:33:04Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:510724974worker1 # [ 45.730063] nix-grpc-daemon[780]: ts=2026-09-22T11:33:04Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:510724975client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.21 seconds)4976client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top4977worker2 # [ 45.902982] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:468924978client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):4979client # /nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a4980client # /nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b4981client # /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top4982worker2 # [ 45.947161] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:468924983worker2 # [ 45.950251] nix-grpc-daemon[783]: copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'http://lb:5751'...4984worker2 # [ 45.959559] nix-grpc-daemon[783]: copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'http://lb:5751'...4985worker2 # [ 45.967165] nix-grpc-daemon[783]: copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'http://lb:5751'...4986client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.20 seconds)4987client: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top4988client # finding garbage collector roots...4989client # removing stale temporary roots file "/nix/var/nix/temproots/837"4990client # removing stale temporary roots file "/nix/var/nix/temproots/800"4991client # removing stale temporary roots file "/nix/var/nix/temproots/879"4992client # removing stale temporary roots file "/nix/var/nix/temproots/858"4993client # removing stale temporary roots file "/nix/var/nix/temproots/850"4994client # removing stale temporary roots file "/nix/var/nix/temproots/816"4995client # removing stale temporary roots file "/nix/var/nix/temproots/871"4996client # removing stale temporary roots file "/nix/var/nix/temproots/895"4997client # deleting specified paths...4998client # deleting '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top'4999client # deleting unused links...5000client # note: hard linking is currently saving 0.0 KiB5001worker2 # [ 46.236852] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558325002worker1 # [ 46.246399] nix-grpc-daemon[780]: ts=2026-09-22T11:33:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462025003worker1 # [ 46.255596] nix-grpc-daemon[872]: time=2026-09-22T11:33:04.620Z level=INFO msg="All 1 paths already cached"5004worker1 # [ 46.262541] nix-grpc-daemon[780]: ts=2026-09-22T11:33:04Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:462025005worker2 # [ 46.264243] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558325006client # copying 3 paths...5007worker2 # [ 46.271456] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558325008client # copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'grpc://lb:50051'...5009worker1 # [ 46.290455] nix-grpc-daemon[780]: ts=2026-09-22T11:33:04Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:462025010worker2 # [ 46.291627] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558325011worker1 # [ 46.295825] nix-grpc-daemon[780]: ts=2026-09-22T11:33:04Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 paths=1 nar_bytes_out=2325012worker2 # [ 46.294166] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 paths=1 nar_bytes_out=1765013worker2 # [ 46.298256] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558325014worker2 # [ 46.300221] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 paths=1 nar_bytes_out=1285015client # copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'grpc://lb:50051'...5016client # copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'grpc://lb:50051'...5017client: (finished: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.39 seconds)5018(finished: subtest: outputs read back through any worker carry the cache signature, in 6.17 seconds)5019subtest: two clients wanting the same drv build it once5020client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup5021client # [ 46.401292] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.5022client # Running as unit: dedup1.service; invocation ID: 88862faeaa5942f8a1aae7e39693c8715023client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 0.03 seconds)5024client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup5025client # [ 46.541096] nix[948]: don't know how to build these paths:5026client # [ 46.542022] nix[948]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5027worker2 # [ 46.543688] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558325028worker2 # [ 46.554898] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05029worker2 # [ 46.560331] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=15030worker2 # [ 46.599282] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325031worker2 # [ 46.611337] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558325032client # [ 46.617204] nix[948]: copying 1 paths to 'lb:50051'...5033worker2 # [ 46.622777] nix-grpc-daemon[783]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558325034client # [ 46.620109] nix[948]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...5035worker2 # [ 46.636256] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5036worker2 # [ 46.734457] nix-daemon[867]: accepted connection from pid 1083, user nix-grpc-daemon (trusted)5037worker2 # [ 46.749754] nix-daemon[867]: reaped child process 1086, status = succeeded5038worker2 # [ 46.826635] nix-daemon[867]: accepted connection from pid 1088, user nix-grpc-daemon (trusted)5039worker2 # [ 46.840762] nix-daemon[867]: reaped child process 1091, status = succeeded5040lb # [ 46.814950] niks3-server[949]: 2026/09/22 11:33:05 INFO Received uploads request method=POST path=/api/pending_closures5041worker2 # [ 46.855285] nix-grpc-daemon[882]: time=2026-09-22T11:33:06.021Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5042worker2 # [ 46.856666] nix-grpc-daemon[882]: time=2026-09-22T11:33:06.021Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"5043lb # [ 46.843213] niks3-server[949]: 2026/09/22 11:33:05 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst5044lb # [ 46.855607] niks3-server[949]: 2026/09/22 11:33:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5045worker2 # [ 46.895185] nix-grpc-daemon[882]: time=2026-09-22T11:33:06.061Z level=INFO msg="Uploading 1 narinfos"5046lb # [ 46.863638] niks3-server[949]: 2026/09/22 11:33:05 INFO Signed narinfos id=15 count=15047lb # [ 46.867919] niks3-server[949]: 2026/09/22 11:33:05 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls5048lb # [ 46.879554] niks3-server[949]: 2026/09/22 11:33:05 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5049lb # [ 46.886395] niks3-server[949]: 2026/09/22 11:33:05 INFO Completed upload id=155050worker2 # [ 46.917948] nix-grpc-daemon[882]: time=2026-09-22T11:33:06.084Z level=INFO msg="Upload complete. (270ms)"5051lb # [ 46.891108] niks3-server[949]: 2026/09/22 11:33:05 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo5052worker2 # [ 46.922482] nix-grpc-daemon[783]: ts=2026-09-22T11:33:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 paths=1 nar_bytes_in=4325053worker2 # [ 46.927340] nix-grpc-daemon[783]: ts=2026-09-22T11:33:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325054worker2 # [ 46.936823] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5055worker2 # [ 46.944199] nix-daemon[867]: reaped child process 1081, status = succeeded5056worker2 # [ 46.947726] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5057client # [ 46.942431] nix[948]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5058lb # [ 47.037819] envoy[918]: conn peer=[2001:db8:1::1]:47444 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9425 tx=4383 ms=48135059lb # [ 47.041708] envoy[918]: conn peer=[2001:db8:1::1]:42436 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=455060lb # [ 47.045199] envoy[918]: conn peer=[2001:db8:1::1]:42446 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4139 tx=1901 ms=12525061lb # [ 47.048693] envoy[918]: conn peer=[2001:db8:1::1]:42458 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9494 tx=4331 ms=51245062lb # [ 47.052217] envoy[918]: conn peer=[2001:db8:1::1]:42542 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2889 ms=1315063lb # [ 47.056505] envoy[918]: conn peer=[2001:db8:1::1]:42554 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=410 ms=825064lb # [ 47.059935] envoy[918]: conn peer=[2001:db8:1::1]:42580 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=330 ms=685065lb # [ 47.063474] envoy[918]: conn peer=[2001:db8:1::1]:42570 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=376 ms=755066client # [ 47.455935] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.5067client # Running as unit: dedup2.service; invocation ID: afbdb6aae3f4466397d23e8984e001fd5068client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 1.06 seconds)5069worker1 # [ 47.625756] nix-grpc-daemon[780]: ts=2026-09-22T11:33:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:462025070client # [ 47.621804] nix[964]: don't know how to build these paths:5071client # [ 47.622675] nix[964]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5072worker2 # [ 47.632267] nix-grpc-daemon[783]: ts=2026-09-22T11:33:06Z level=debug event=want client=6 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05073client # [ 47.634059] nix[964]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5074worker2 # [ 47.638076] nix-grpc-daemon[783]: ts=2026-09-22T11:33:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325075client: must succeed: sleep 25076client: (finished: must succeed: sleep 2, in 2.03 seconds)5077client: waiting for success: ! systemctl is-active dedup1 dedup25078worker2 # [ 50.985736] nix-daemon[867]: reaped child process 1095, status = succeeded5079worker2 # [ 51.072103] nix-daemon[867]: accepted connection from pid 1127, user nix-grpc-daemon (trusted)5080worker2 # [ 51.085946] nix-daemon[867]: reaped child process 1130, status = succeeded5081lb # [ 51.061471] niks3-server[949]: 2026/09/22 11:33:09 INFO Received uploads request method=POST path=/api/pending_closures5082worker2 # [ 51.353353] nix-grpc-daemon[882]: time=2026-09-22T11:33:10.519Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5083worker2 # [ 51.356802] nix-grpc-daemon[882]: time=2026-09-22T11:33:10.521Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"5084lb # [ 51.343851] niks3-server[949]: 2026/09/22 11:33:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5085lb # [ 51.346903] niks3-server[949]: 2026/09/22 11:33:09 INFO Signed narinfos id=16 count=15086worker2 # [ 51.378327] nix-grpc-daemon[882]: time=2026-09-22T11:33:10.544Z level=INFO msg="Uploading 1 narinfos"5087lb # [ 51.352373] niks3-server[949]: 2026/09/22 11:33:09 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5088lb # [ 51.380331] niks3-server[949]: 2026/09/22 11:33:09 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5089lb # [ 51.393928] niks3-server[949]: 2026/09/22 11:33:09 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst5090lb # [ 51.433135] niks3-server[949]: 2026/09/22 11:33:09 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls5091worker2 # [ 51.481355] nix-grpc-daemon[882]: time=2026-09-22T11:33:10.647Z level=INFO msg="Upload complete. (522ms)"5092lb # [ 51.449809] niks3-server[949]: 2026/09/22 11:33:09 INFO Completed upload id=165093worker2 # [ 51.484692] nix-grpc-daemon[783]: ts=2026-09-22T11:33:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15094worker2 # [ 51.488138] nix-grpc-daemon[783]: ts=2026-09-22T11:33:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15095worker2 # [ 51.495611] nix-daemon[867]: reaped child process 1093, status = succeeded5096lb # [ 51.477232] niks3-server[949]: 2026/09/22 11:33:09 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo5097client # [ 51.515493] systemd[1]: dedup2.service: Deactivated successfully.5098client # [ 51.519696] systemd[1]: dedup2.service: Consumed 142ms CPU time over 4.062s wall clock time, 35.4M memory peak, 6K incoming IP traffic, 5.6K outgoing IP traffic.5099client # [ 51.554673] systemd[1]: dedup1.service: Deactivated successfully.5100client # [ 51.557664] systemd[1]: dedup1.service: Consumed 127ms CPU time over 5.155s wall clock time, 39.5M memory peak, 7.3K incoming IP traffic, 8.5K outgoing IP traffic.5101client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.09 seconds)5102client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }5103client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)5104worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5105worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.02 seconds)5106worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5107worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5108worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5109worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)5110worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5111worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5112(finished: subtest: two clients wanting the same drv build it once, in 5.65 seconds)5113subtest: --timeout and --max-silent-time reach the builder5114worker2 # [ 52.171650] nix-grpc-daemon[783]: ts=2026-09-22T11:33:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558325115worker2 # [ 52.179355] nix-grpc-daemon[783]: ts=2026-09-22T11:33:11Z level=debug event=want client=7 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv cached=05116worker2 # [ 52.181431] nix-grpc-daemon[783]: ts=2026-09-22T11:33:11Z level=debug event=assigned drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv worker=192.168.1.3:50051 clients=15117worker1 # [ 52.188920] nix-grpc-daemon[780]: ts=2026-09-22T11:33:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462025118worker1 # [ 52.195243] nix-grpc-daemon[780]: ts=2026-09-22T11:33:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462025119worker1 # [ 52.200294] nix-grpc-daemon[780]: ts=2026-09-22T11:33:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462025120worker1 # [ 52.205419] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5121worker1 # [ 52.287742] nix-daemon[844]: accepted connection from pid 1011, user nix-grpc-daemon (trusted)5122worker1 # [ 52.301044] nix-daemon[844]: reaped child process 1014, status = succeeded5123worker1 # [ 52.379955] nix-daemon[844]: accepted connection from pid 1016, user nix-grpc-daemon (trusted)5124worker1 # [ 52.394122] nix-daemon[844]: reaped child process 1019, status = succeeded5125lb # [ 52.360071] niks3-server[949]: 2026/09/22 11:33:10 INFO Received uploads request method=POST path=/api/pending_closures5126worker1 # [ 52.401938] nix-grpc-daemon[872]: time=2026-09-22T11:33:10.766Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5127worker1 # [ 52.403238] nix-grpc-daemon[872]: time=2026-09-22T11:33:10.766Z level=INFO msg="Uploading qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv (424B)"5128lb # [ 52.372582] niks3-server[949]: 2026/09/22 11:33:10 INFO Registered completed upload object_key=nar/1s4sciw269ksj1s7ymwz5cy4dw9n7bxv3k8wqw69h694spdh4ha2.nar.zst5129lb # [ 52.381637] niks3-server[949]: 2026/09/22 11:33:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5130lb # [ 52.386828] niks3-server[949]: 2026/09/22 11:33:10 INFO Signed narinfos id=17 count=15131worker1 # [ 52.428056] nix-grpc-daemon[872]: time=2026-09-22T11:33:10.792Z level=INFO msg="Uploading 1 narinfos"5132lb # [ 52.393158] niks3-server[949]: 2026/09/22 11:33:10 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.ls5133lb # [ 52.399921] niks3-server[949]: 2026/09/22 11:33:10 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5134lb # [ 52.404356] niks3-server[949]: 2026/09/22 11:33:10 INFO Completed upload id=175135worker1 # [ 52.441335] nix-grpc-daemon[872]: time=2026-09-22T11:33:10.805Z level=INFO msg="Upload complete. (231ms)"5136lb # [ 52.406362] niks3-server[949]: 2026/09/22 11:33:10 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.narinfo5137worker1 # [ 52.443489] nix-grpc-daemon[780]: ts=2026-09-22T11:33:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 paths=1 nar_bytes_in=4245138worker1 # [ 52.447540] nix-grpc-daemon[780]: ts=2026-09-22T11:33:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462025139worker1 # [ 52.453179] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5140worker1 # [ 52.455422] nix-daemon[844]: reaped child process 1009, status = succeeded5141worker1 # [ 52.460310] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5142lb # [ 57.036875] envoy[918]: conn peer=[2001:db8:1::1]:42602 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1427 tx=1197 ms=38835143lb # [ 57.039478] envoy[918]: conn peer=[2001:db8:1::1]:42588 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3015 tx=1556 ms=50025144worker1 # [ 58.472177] nix-grpc-daemon[780]: ts=2026-09-22T11:33:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=6 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv assign_id=9 outputs=05145worker1 # [ 58.483250] nix-daemon[844]: reaped child process 1021, status = succeeded5146worker1 # [ 58.484425] nix-daemon[844]: reaped child process 1023, status = succeeded5147worker2 # [ 58.671384] nix-grpc-daemon[783]: ts=2026-09-22T11:33:17Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558325148worker2 # [ 58.684913] nix-grpc-daemon[783]: ts=2026-09-22T11:33:17Z level=debug event=want client=8 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv cached=05149worker2 # [ 58.691617] nix-grpc-daemon[783]: ts=2026-09-22T11:33:17Z level=debug event=assigned drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv worker=192.168.1.4:50051 clients=15150worker2 # [ 58.696460] nix-grpc-daemon[783]: ts=2026-09-22T11:33:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325151worker2 # [ 58.704518] nix-grpc-daemon[783]: ts=2026-09-22T11:33:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558325152worker2 # [ 58.716142] nix-grpc-daemon[783]: ts=2026-09-22T11:33:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558325153worker2 # [ 58.726736] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5154worker2 # [ 58.829182] nix-daemon[867]: accepted connection from pid 1147, user nix-grpc-daemon (trusted)5155worker2 # [ 58.843995] nix-daemon[867]: reaped child process 1150, status = succeeded5156worker2 # [ 58.927952] nix-daemon[867]: accepted connection from pid 1152, user nix-grpc-daemon (trusted)5157worker2 # [ 58.940970] nix-daemon[867]: reaped child process 1155, status = succeeded5158lb # [ 58.916648] niks3-server[949]: 2026/09/22 11:33:17 INFO Received uploads request method=POST path=/api/pending_closures5159worker2 # [ 58.953261] nix-grpc-daemon[882]: time=2026-09-22T11:33:18.119Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5160worker2 # [ 58.954742] nix-grpc-daemon[882]: time=2026-09-22T11:33:18.121Z level=INFO msg="Uploading fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv (424B)"5161lb # [ 58.940913] niks3-server[949]: 2026/09/22 11:33:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5162lb # [ 58.943720] niks3-server[949]: 2026/09/22 11:33:17 INFO Registered completed upload object_key=nar/013y5sy2r6v0qdwv0dzpgvdfy08bds4mfh2qpvbgjazzrg2v71hp.nar.zst5163lb # [ 58.947065] niks3-server[949]: 2026/09/22 11:33:17 INFO Signed narinfos id=18 count=15164worker2 # [ 58.978688] nix-grpc-daemon[882]: time=2026-09-22T11:33:18.144Z level=INFO msg="Uploading 1 narinfos"5165lb # [ 58.952386] niks3-server[949]: 2026/09/22 11:33:17 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.ls5166lb # [ 58.958679] niks3-server[949]: 2026/09/22 11:33:17 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5167lb # [ 58.962664] niks3-server[949]: 2026/09/22 11:33:17 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.narinfo5168lb # [ 58.965777] niks3-server[949]: 2026/09/22 11:33:17 INFO Completed upload id=185169worker2 # [ 58.997267] nix-grpc-daemon[882]: time=2026-09-22T11:33:18.163Z level=INFO msg="Upload complete. (260ms)"5170worker2 # [ 58.999310] nix-grpc-daemon[783]: ts=2026-09-22T11:33:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 paths=1 nar_bytes_in=4245171worker2 # [ 59.002688] nix-grpc-daemon[783]: ts=2026-09-22T11:33:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325172worker2 # [ 59.009732] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5173worker2 # [ 59.012097] nix-daemon[867]: reaped child process 1145, status = succeeded5174worker2 # [ 59.017092] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5175worker2 # [ 65.432579] nix-grpc-daemon[783]: ts=2026-09-22T11:33:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=6 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv assign_id=10 outputs=05176worker2 # [ 65.457868] nix-daemon[867]: reaped child process 1157, status = succeeded5177worker2 # [ 65.459933] nix-daemon[867]: reaped child process 1159, status = succeeded5178(finished: subtest: --timeout and --max-silent-time reach the builder, in 13.53 seconds)5179subtest: the first client leaving does not fail the second5180client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share5181client # [ 65.576912] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.5182client # Running as unit: share1.service; invocation ID: 4c70e241e3c84141a160726b72688e2d5183client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 0.03 seconds)5184client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share5185client # [ 65.704171] nix[1037]: don't know how to build these paths:5186client # [ 65.705072] nix[1037]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5187worker1 # [ 65.713622] nix-grpc-daemon[780]: ts=2026-09-22T11:33:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:462025188worker2 # [ 65.716297] nix-grpc-daemon[783]: ts=2026-09-22T11:33:24Z level=debug event=want client=9 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05189worker1 # [ 65.724170] nix-grpc-daemon[780]: ts=2026-09-22T11:33:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462025190worker2 # [ 65.720975] nix-grpc-daemon[783]: ts=2026-09-22T11:33:24Z level=debug event=assigned drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv worker=192.168.1.3:50051 clients=15191worker1 # [ 65.736375] nix-grpc-daemon[780]: ts=2026-09-22T11:33:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462025192client # [ 65.736708] nix[1037]: copying 1 paths to 'lb:50051'...5193client # [ 65.737556] nix[1037]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' to 'lb:50051'...5194worker1 # [ 65.747363] nix-grpc-daemon[780]: ts=2026-09-22T11:33:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462025195worker1 # [ 65.757932] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5196worker1 # [ 65.855485] nix-daemon[844]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)5197worker1 # [ 65.867445] nix-daemon[844]: reaped child process 1049, status = succeeded5198worker1 # [ 65.947729] nix-daemon[844]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)5199worker1 # [ 65.960511] nix-daemon[844]: reaped child process 1054, status = succeeded5200lb # [ 65.930819] niks3-server[949]: 2026/09/22 11:33:24 INFO Received uploads request method=POST path=/api/pending_closures5201worker1 # [ 65.977745] nix-grpc-daemon[872]: time=2026-09-22T11:33:24.341Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5202worker1 # [ 65.979162] nix-grpc-daemon[872]: time=2026-09-22T11:33:24.341Z level=INFO msg="Uploading j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv (432B)"5203lb # [ 65.965861] niks3-server[949]: 2026/09/22 11:33:24 INFO Registered completed upload object_key=nar/0s0b0gfhn36p0aylk1qa64cgw01bi7a8ca93iv4zg2q7wqcjcqhs.nar.zst5204lb # [ 65.977824] niks3-server[949]: 2026/09/22 11:33:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5205lb # [ 65.983736] niks3-server[949]: 2026/09/22 11:33:24 INFO Signed narinfos id=19 count=15206worker1 # [ 66.023631] nix-grpc-daemon[872]: time=2026-09-22T11:33:24.388Z level=INFO msg="Uploading 1 narinfos"5207lb # [ 65.991878] niks3-server[949]: 2026/09/22 11:33:24 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.ls5208lb # [ 66.004274] niks3-server[949]: 2026/09/22 11:33:24 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5209lb # [ 66.007816] niks3-server[949]: 2026/09/22 11:33:24 INFO Completed upload id=195210worker1 # [ 66.045975] nix-grpc-daemon[872]: time=2026-09-22T11:33:24.410Z level=INFO msg="Upload complete. (276ms)"5211lb # [ 66.010159] niks3-server[949]: 2026/09/22 11:33:24 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.narinfo5212worker1 # [ 66.048062] nix-grpc-daemon[780]: ts=2026-09-22T11:33:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 paths=1 nar_bytes_in=4325213worker1 # [ 66.051431] nix-grpc-daemon[780]: ts=2026-09-22T11:33:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462025214worker1 # [ 66.059708] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5215worker1 # [ 66.061573] nix-daemon[844]: reaped child process 1044, status = succeeded5216client # [ 66.056033] nix[1037]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5217worker1 # [ 66.066156] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5218client # [ 66.630087] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.5219client # Running as unit: share2.service; invocation ID: ac83e25dadf24afb92d2abedf0efdad55220client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 1.07 seconds)5221client: must succeed: sleep 2; systemctl kill -s INT share15222worker2 # [ 66.783529] nix-grpc-daemon[783]: ts=2026-09-22T11:33:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558325223client # [ 66.784924] nix[1053]: don't know how to build these paths:5224client # [ 66.785864] nix[1053]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5225worker2 # [ 66.795687] nix-grpc-daemon[783]: ts=2026-09-22T11:33:25Z level=debug event=want client=10 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05226worker1 # [ 66.806223] nix-grpc-daemon[780]: ts=2026-09-22T11:33:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462025227client # [ 66.799189] nix[1053]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5228lb # [ 67.038259] envoy[918]: conn peer=[2001:db8:1::1]:42612 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=933 ms=63505229lb # [ 67.042079] envoy[918]: conn peer=[2001:db8:1::1]:36670 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2952 tx=936 ms=67735230client # [ 68.777971] systemd[1]: share1.service: Sent signal SIGINT to main process 1037 (nix) on client request.5231client # [ 68.780827] nix[1037]: shutting downj2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5232client # [ 68.784136] nix[1037]: error: interrupted by the user5233client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.04 seconds)5234client: waiting for success: ! systemctl is-active share15235client # [ 68.811814] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE5236client # [ 68.813438] systemd[1]: share1.service: Failed with result 'exit-code'.5237client # [ 68.815401] systemd[1]: share1.service: Consumed 102ms CPU time over 3.236s wall clock time, 43.5M memory peak, 6K incoming IP traffic, 8K outgoing IP traffic.5238client: (finished: waiting for success: ! systemctl is-active share1, in 1.06 seconds)5239client: waiting for success: ! systemctl is-active share25240worker1 # [ 70.102695] nix-daemon[844]: reaped child process 1058, status = succeeded5241worker1 # [ 70.171894] nix-daemon[844]: accepted connection from pid 1089, user nix-grpc-daemon (trusted)5242worker1 # [ 70.184348] nix-daemon[844]: reaped child process 1092, status = succeeded5243lb # [ 70.154580] niks3-server[949]: 2026/09/22 11:33:28 INFO Received uploads request method=POST path=/api/pending_closures5244worker1 # [ 70.202434] nix-grpc-daemon[872]: time=2026-09-22T11:33:28.566Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5245worker1 # [ 70.204537] nix-grpc-daemon[872]: time=2026-09-22T11:33:28.568Z level=INFO msg="Uploading b9lg6znd0rinl36gy6ixq1dcj6b8bhhw-slow-share (120B)"5246lb # [ 70.198148] niks3-server[949]: 2026/09/22 11:33:28 INFO Registered completed upload object_key=log/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5247lb # [ 70.203431] niks3-server[949]: 2026/09/22 11:33:28 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst5248lb # [ 70.217100] niks3-server[949]: 2026/09/22 11:33:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5249lb # [ 70.224735] niks3-server[949]: 2026/09/22 11:33:28 INFO Signed narinfos id=20 count=15250worker1 # [ 70.261224] nix-grpc-daemon[872]: time=2026-09-22T11:33:28.625Z level=INFO msg="Uploading 1 narinfos"5251lb # [ 70.229432] niks3-server[949]: 2026/09/22 11:33:28 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.ls5252lb # [ 70.240435] niks3-server[949]: 2026/09/22 11:33:28 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5253lb # [ 70.245321] niks3-server[949]: 2026/09/22 11:33:28 INFO Completed upload id=205254lb # [ 70.247616] niks3-server[949]: 2026/09/22 11:33:28 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.narinfo5255worker1 # [ 70.283774] nix-grpc-daemon[872]: time=2026-09-22T11:33:28.648Z level=INFO msg="Upload complete. (208ms)"5256worker1 # [ 70.291149] nix-grpc-daemon[780]: ts=2026-09-22T11:33:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=3 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15257worker1 # [ 70.296822] nix-grpc-daemon[780]: ts=2026-09-22T11:33:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=4 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15258worker1 # [ 70.309859] nix-daemon[844]: reaped child process 1056, status = succeeded5259client # [ 70.353955] systemd[1]: share2.service: Deactivated successfully.5260client # [ 70.357778] systemd[1]: share2.service: Consumed 122ms CPU time over 3.727s wall clock time, 35.4M memory peak, 5.9K incoming IP traffic, 5.8K outgoing IP traffic.5261client: (finished: waiting for success: ! systemctl is-active share2, in 1.08 seconds)5262client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }5263client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.03 seconds)5264(finished: subtest: the first client leaving does not fail the second, in 5.53 seconds)5265subtest: build hook: nix-daemon with builders = grpc://lb5266client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log5267client # [ 71.177517] systemd[1]: Started Nix Daemon.5268client # [ 71.268764] nix-daemon[1103]: accepted connection from pid 1100, user root (trusted)5269worker1 # [ 71.401810] nix-grpc-daemon[780]: ts=2026-09-22T11:33:29Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:462025270worker2 # [ 71.400319] nix-grpc-daemon[783]: ts=2026-09-22T11:33:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558325271worker2 # [ 71.416152] nix-grpc-daemon[783]: ts=2026-09-22T11:33:30Z level=debug event=want client=11 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05272worker2 # [ 71.421634] nix-grpc-daemon[783]: ts=2026-09-22T11:33:30Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.4:50051 clients=15273worker2 # [ 71.428077] nix-grpc-daemon[783]: ts=2026-09-22T11:33:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325274worker2 # [ 71.435925] nix-grpc-daemon[783]: ts=2026-09-22T11:33:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558325275worker2 # [ 71.448421] nix-grpc-daemon[783]: ts=2026-09-22T11:33:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558325276worker2 # [ 71.458848] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5277worker2 # [ 71.553949] nix-daemon[867]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)5278worker2 # [ 71.567585] nix-daemon[867]: reaped child process 1203, status = succeeded5279worker2 # [ 71.649097] nix-daemon[867]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5280worker2 # [ 71.660860] nix-daemon[867]: reaped child process 1208, status = succeeded5281lb # [ 71.636819] niks3-server[949]: 2026/09/22 11:33:29 INFO Received uploads request method=POST path=/api/pending_closures5282worker2 # [ 71.677517] nix-grpc-daemon[882]: time=2026-09-22T11:33:30.843Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5283worker2 # [ 71.678972] nix-grpc-daemon[882]: time=2026-09-22T11:33:30.845Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5284lb # [ 71.666584] niks3-server[949]: 2026/09/22 11:33:30 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5285lb # [ 71.679949] niks3-server[949]: 2026/09/22 11:33:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5286lb # [ 71.686282] niks3-server[949]: 2026/09/22 11:33:30 INFO Signed narinfos id=21 count=15287worker2 # [ 71.720447] nix-grpc-daemon[882]: time=2026-09-22T11:33:30.886Z level=INFO msg="Uploading 1 narinfos"5288lb # [ 71.697092] niks3-server[949]: 2026/09/22 11:33:30 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5289lb # [ 71.702759] niks3-server[949]: 2026/09/22 11:33:30 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5290lb # [ 71.707719] niks3-server[949]: 2026/09/22 11:33:30 INFO Completed upload id=215291worker2 # [ 71.740343] nix-grpc-daemon[882]: time=2026-09-22T11:33:30.906Z level=INFO msg="Upload complete. (270ms)"5292lb # [ 71.710910] niks3-server[949]: 2026/09/22 11:33:30 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo5293worker2 # [ 71.743251] nix-grpc-daemon[783]: ts=2026-09-22T11:33:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 paths=1 nar_bytes_in=5125294worker2 # [ 71.748472] nix-grpc-daemon[783]: ts=2026-09-22T11:33:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325295worker2 # [ 71.759149] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5296worker2 # [ 71.764156] nix-daemon[867]: reaped child process 1198, status = succeeded5297worker2 # [ 71.767694] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5298worker2 # [ 72.984344] nix-daemon[867]: reaped child process 1212, status = succeeded5299worker2 # [ 73.058159] nix-daemon[867]: accepted connection from pid 1228, user nix-grpc-daemon (trusted)5300worker2 # [ 73.071713] nix-daemon[867]: reaped child process 1231, status = succeeded5301lb # [ 73.049955] niks3-server[949]: 2026/09/22 11:33:31 INFO Received uploads request method=POST path=/api/pending_closures5302worker2 # [ 73.087725] nix-grpc-daemon[882]: time=2026-09-22T11:33:32.253Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5303worker2 # [ 73.089274] nix-grpc-daemon[882]: time=2026-09-22T11:33:32.255Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5304lb # [ 73.080362] niks3-server[949]: 2026/09/22 11:33:31 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5305lb # [ 73.098994] niks3-server[949]: 2026/09/22 11:33:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5306lb # [ 73.104394] niks3-server[949]: 2026/09/22 11:33:31 INFO Signed narinfos id=22 count=15307worker2 # [ 73.139142] nix-grpc-daemon[882]: time=2026-09-22T11:33:32.305Z level=INFO msg="Uploading 1 narinfos"5308lb # [ 73.112646] niks3-server[949]: 2026/09/22 11:33:31 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5309lb # [ 73.122935] niks3-server[949]: 2026/09/22 11:33:31 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5310lb # [ 73.129118] niks3-server[949]: 2026/09/22 11:33:31 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5311lb # [ 73.134226] niks3-server[949]: 2026/09/22 11:33:31 INFO Completed upload id=225312lb # [ 73.136424] niks3-server[949]: 2026/09/22 11:33:31 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5313worker2 # [ 73.165934] nix-grpc-daemon[882]: time=2026-09-22T11:33:32.332Z level=INFO msg="Upload complete. (190ms)"5314worker2 # [ 73.173964] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=12 outputs=15315worker2 # [ 73.182153] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558325316worker2 # [ 73.193327] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558325317worker2 # [ 73.196571] nix-daemon[867]: reaped child process 1210, status = succeeded5318worker2 # [ 73.199271] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 paths=1 nar_bytes_out=1285319worker2 # [ 73.383588] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:558325320worker2 # [ 73.390175] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558325321worker2 # [ 73.397390] nix-grpc-daemon[882]: time=2026-09-22T11:33:32.562Z level=INFO msg="All 1 paths already cached"5322worker2 # [ 73.406439] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=debug event=want client=12 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05323worker2 # [ 73.412970] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.4:50051 clients=15324worker2 # [ 73.417897] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325325worker2 # [ 73.427544] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558325326worker2 # [ 73.437900] nix-grpc-daemon[882]: time=2026-09-22T11:33:32.604Z level=INFO msg="All 1 paths already cached"5327worker2 # [ 73.444427] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558325328worker2 # [ 73.454143] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5329worker2 # [ 73.546800] nix-daemon[867]: accepted connection from pid 1237, user nix-grpc-daemon (trusted)5330worker2 # [ 73.560679] nix-daemon[867]: reaped child process 1240, status = succeeded5331worker2 # [ 73.643297] nix-daemon[867]: accepted connection from pid 1242, user nix-grpc-daemon (trusted)5332worker2 # [ 73.657963] nix-daemon[867]: reaped child process 1245, status = succeeded5333lb # [ 73.631714] niks3-server[949]: 2026/09/22 11:33:31 INFO Received uploads request method=POST path=/api/pending_closures5334worker2 # [ 73.667827] nix-grpc-daemon[882]: time=2026-09-22T11:33:32.833Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5335worker2 # [ 73.669192] nix-grpc-daemon[882]: time=2026-09-22T11:33:32.833Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5336lb # [ 73.646545] niks3-server[949]: 2026/09/22 11:33:31 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5337lb # [ 73.649914] niks3-server[949]: 2026/09/22 11:33:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5338lb # [ 73.652246] niks3-server[949]: 2026/09/22 11:33:32 INFO Signed narinfos id=23 count=15339worker2 # [ 73.683619] nix-grpc-daemon[882]: time=2026-09-22T11:33:32.849Z level=INFO msg="Uploading 1 narinfos"5340lb # [ 73.655199] niks3-server[949]: 2026/09/22 11:33:32 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5341lb # [ 73.659894] niks3-server[949]: 2026/09/22 11:33:32 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5342lb # [ 73.664652] niks3-server[949]: 2026/09/22 11:33:32 INFO Completed upload id=235343worker2 # [ 73.696126] nix-grpc-daemon[882]: time=2026-09-22T11:33:32.862Z level=INFO msg="Upload complete. (231ms)"5344lb # [ 73.665855] niks3-server[949]: 2026/09/22 11:33:32 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5345worker2 # [ 73.698278] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 paths=1 nar_bytes_in=6245346worker2 # [ 73.701666] nix-grpc-daemon[783]: ts=2026-09-22T11:33:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325347worker2 # [ 73.707503] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5348worker2 # [ 73.713149] nix-daemon[867]: reaped child process 1235, status = succeeded5349worker2 # [ 73.718085] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5350worker2 # [ 74.738270] nix-daemon[867]: reaped child process 1249, status = succeeded5351worker2 # [ 74.809870] nix-daemon[867]: accepted connection from pid 1265, user nix-grpc-daemon (trusted)5352worker2 # [ 74.824522] nix-daemon[867]: reaped child process 1268, status = succeeded5353lb # [ 74.799251] niks3-server[949]: 2026/09/22 11:33:33 INFO Received uploads request method=POST path=/api/pending_closures5354worker2 # [ 74.836751] nix-grpc-daemon[882]: time=2026-09-22T11:33:34.002Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5355worker2 # [ 74.840059] nix-grpc-daemon[882]: time=2026-09-22T11:33:34.004Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5356lb # [ 74.824175] niks3-server[949]: 2026/09/22 11:33:33 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5357lb # [ 74.829272] niks3-server[949]: 2026/09/22 11:33:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5358lb # [ 74.832462] niks3-server[949]: 2026/09/22 11:33:33 INFO Signed narinfos id=24 count=15359worker2 # [ 74.863807] nix-grpc-daemon[882]: time=2026-09-22T11:33:34.029Z level=INFO msg="Uploading 1 narinfos"5360lb # [ 74.836506] niks3-server[949]: 2026/09/22 11:33:33 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5361lb # [ 74.839213] niks3-server[949]: 2026/09/22 11:33:33 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5362lb # [ 74.842865] niks3-server[949]: 2026/09/22 11:33:33 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5363lb # [ 74.845551] niks3-server[949]: 2026/09/22 11:33:33 INFO Completed upload id=245364worker2 # [ 74.877047] nix-grpc-daemon[882]: time=2026-09-22T11:33:34.043Z level=INFO msg="Upload complete. (152ms)"5365lb # [ 74.848520] niks3-server[949]: 2026/09/22 11:33:33 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5366worker2 # [ 74.881061] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=13 outputs=15367worker2 # [ 74.889294] nix-daemon[867]: reaped child process 1247, status = succeeded5368worker1 # [ 74.935659] nix-grpc-daemon[780]: ts=2026-09-22T11:33:33Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:462025369worker1 # [ 74.949678] nix-grpc-daemon[780]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5370worker1 # [ 74.960870] nix-grpc-daemon[780]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...5371worker2 # [ 74.974134] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558325372worker2 # [ 74.979523] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 paths=1 nar_bytes_out=1765373worker2 # [ 75.127149] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:558325374worker2 # [ 75.134534] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558325375worker2 # [ 75.142775] nix-grpc-daemon[882]: time=2026-09-22T11:33:34.308Z level=INFO msg="All 2 paths already cached"5376worker2 # [ 75.157154] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=debug event=want client=13 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05377worker2 # [ 75.162967] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.4:50051 clients=15378worker2 # [ 75.167935] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325379worker2 # [ 75.177379] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558325380worker2 # [ 75.187796] nix-grpc-daemon[882]: time=2026-09-22T11:33:34.353Z level=INFO msg="All 2 paths already cached"5381worker2 # [ 75.194923] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558325382worker2 # [ 75.203335] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5383worker2 # [ 75.301974] nix-daemon[867]: accepted connection from pid 1274, user nix-grpc-daemon (trusted)5384worker2 # [ 75.317861] nix-daemon[867]: reaped child process 1277, status = succeeded5385worker2 # [ 75.396934] nix-daemon[867]: accepted connection from pid 1279, user nix-grpc-daemon (trusted)5386worker2 # [ 75.412231] nix-daemon[867]: reaped child process 1282, status = succeeded5387lb # [ 75.385449] niks3-server[949]: 2026/09/22 11:33:33 INFO Received uploads request method=POST path=/api/pending_closures5388worker2 # [ 75.421151] nix-grpc-daemon[882]: time=2026-09-22T11:33:34.587Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5389worker2 # [ 75.422680] nix-grpc-daemon[882]: time=2026-09-22T11:33:34.588Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5390lb # [ 75.403433] niks3-server[949]: 2026/09/22 11:33:33 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5391lb # [ 75.406168] niks3-server[949]: 2026/09/22 11:33:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5392lb # [ 75.408432] niks3-server[949]: 2026/09/22 11:33:33 INFO Signed narinfos id=25 count=15393worker2 # [ 75.441119] nix-grpc-daemon[882]: time=2026-09-22T11:33:34.607Z level=INFO msg="Uploading 1 narinfos"5394lb # [ 75.412322] niks3-server[949]: 2026/09/22 11:33:33 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5395lb # [ 75.418294] niks3-server[949]: 2026/09/22 11:33:33 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5396lb # [ 75.420624] niks3-server[949]: 2026/09/22 11:33:33 INFO Completed upload id=255397worker2 # [ 75.453368] nix-grpc-daemon[882]: time=2026-09-22T11:33:34.619Z level=INFO msg="Upload complete. (237ms)"5398lb # [ 75.423539] niks3-server[949]: 2026/09/22 11:33:33 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5399worker2 # [ 75.455569] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 paths=1 nar_bytes_in=7525400worker2 # [ 75.459313] nix-grpc-daemon[783]: ts=2026-09-22T11:33:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325401worker2 # [ 75.466054] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5402worker2 # [ 75.468339] nix-daemon[867]: reaped child process 1272, status = succeeded5403worker2 # [ 75.473630] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5404worker2 # [ 76.719652] nix-daemon[867]: reaped child process 1286, status = succeeded5405worker2 # [ 76.794870] nix-daemon[867]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5406worker2 # [ 76.809596] nix-daemon[867]: reaped child process 1305, status = succeeded5407lb # [ 76.784643] niks3-server[949]: 2026/09/22 11:33:35 INFO Received uploads request method=POST path=/api/pending_closures5408worker2 # [ 76.821244] nix-grpc-daemon[882]: time=2026-09-22T11:33:35.987Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5409worker2 # [ 76.822819] nix-grpc-daemon[882]: time=2026-09-22T11:33:35.989Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5410lb # [ 76.819339] niks3-server[949]: 2026/09/22 11:33:35 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5411lb # [ 76.831621] niks3-server[949]: 2026/09/22 11:33:35 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5412lb # [ 76.840860] niks3-server[949]: 2026/09/22 11:33:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5413lb # [ 76.845771] niks3-server[949]: 2026/09/22 11:33:35 INFO Signed narinfos id=26 count=15414worker2 # [ 76.879982] nix-grpc-daemon[882]: time=2026-09-22T11:33:36.045Z level=INFO msg="Uploading 1 narinfos"5415lb # [ 76.851236] niks3-server[949]: 2026/09/22 11:33:35 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5416lb # [ 76.861063] niks3-server[949]: 2026/09/22 11:33:35 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5417lb # [ 76.864792] niks3-server[949]: 2026/09/22 11:33:35 INFO Completed upload id=265418worker2 # [ 76.896600] nix-grpc-daemon[882]: time=2026-09-22T11:33:36.062Z level=INFO msg="Upload complete. (215ms)"5419lb # [ 76.869809] niks3-server[949]: 2026/09/22 11:33:35 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5420worker2 # [ 76.904568] nix-grpc-daemon[783]: ts=2026-09-22T11:33:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=14 outputs=15421worker2 # [ 76.914594] nix-grpc-daemon[783]: ts=2026-09-22T11:33:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558325422worker2 # [ 76.924732] nix-daemon[867]: reaped child process 1284, status = succeeded5423worker2 # [ 76.928685] nix-grpc-daemon[783]: ts=2026-09-22T11:33:36Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558325424worker2 # [ 76.932894] nix-grpc-daemon[783]: ts=2026-09-22T11:33:36Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=0 paths=1 nar_bytes_out=2325425client # [ 77.039527] nix-daemon[1103]: reaped child process 1105, status = succeeded5426client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log, in 5.97 seconds)5427client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5428lb # [ 77.039579] envoy[918]: conn peer=[2001:db8:1::1]:37922 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2965 tx=770 ms=30895429lb # [ 77.043277] envoy[918]: conn peer=[2001:db8:1::1]:37924 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1440 tx=1197 ms=35565430client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.02 seconds)5431client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5432lb # [ 77.046824] envoy[918]: conn peer=[2001:db8:1::1]:37936 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3447 tx=1941 ms=18585433lb # [ 77.050485] envoy[918]: conn peer=[2001:db8:1::1]:58672 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=565434lb # [ 77.053926] envoy[918]: conn peer=[2001:db8:1::1]:58688 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=461 tx=378 ms=115435lb # [ 77.057363] envoy[918]: conn peer=[2001:db8:1::1]:58684 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3876 tx=2140 ms=16035436lb # [ 77.060862] envoy[918]: conn peer=[2001:db8:1::1]:58700 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4365 tx=2389 ms=18565437lb # [ 77.065101] envoy[918]: conn peer=[2001:db8:1::1]:58714 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=545438client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.02 seconds)5439client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5440client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.02 seconds)5441(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 6.03 seconds)5442subtest: an input only one worker has still reaches the builder5443client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only5444client # this derivation will be built:5445client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5446client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5447client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only, in 0.24 seconds)5448client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5449client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.09 seconds)5450worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5451worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.13 seconds)5452worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5453worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)5454client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&25455client # [ 77.707687] nix-daemon[1103]: accepted connection from pid 1186, user root (trusted)5456client # this derivation will be built:5457client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5458worker2 # [ 77.832799] nix-grpc-daemon[783]: ts=2026-09-22T11:33:36Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:558325459client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5460worker1 # [ 77.845161] nix-grpc-daemon[780]: ts=2026-09-22T11:33:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462025461worker1 # [ 77.953282] nix-daemon[844]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5462worker1 # [ 77.964694] nix-daemon[844]: reaped child process 1104, status = succeeded5463lb # [ 77.935886] niks3-server[949]: 2026/09/22 11:33:36 INFO Received uploads request method=POST path=/api/pending_closures5464worker1 # [ 77.982148] nix-grpc-daemon[872]: time=2026-09-22T11:33:36.346Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5465worker1 # [ 77.983483] nix-grpc-daemon[872]: time=2026-09-22T11:33:36.346Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5466lb # [ 77.966590] niks3-server[949]: 2026/09/22 11:33:36 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5467lb # [ 77.975741] niks3-server[949]: 2026/09/22 11:33:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5468worker1 # [ 78.019574] nix-grpc-daemon[872]: time=2026-09-22T11:33:36.383Z level=INFO msg="Uploading 1 narinfos"5469lb # [ 77.982672] niks3-server[949]: 2026/09/22 11:33:36 INFO Signed narinfos id=27 count=15470lb # [ 77.988653] niks3-server[949]: 2026/09/22 11:33:36 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5471lb # [ 78.003121] niks3-server[949]: 2026/09/22 11:33:36 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5472lb # [ 78.008381] niks3-server[949]: 2026/09/22 11:33:36 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5473lb # [ 78.010943] niks3-server[949]: 2026/09/22 11:33:36 INFO Completed upload id=275474worker1 # [ 78.048105] nix-grpc-daemon[872]: time=2026-09-22T11:33:36.412Z level=INFO msg="Upload complete. (198ms)"5475worker2 # [ 78.061859] nix-grpc-daemon[783]: ts=2026-09-22T11:33:37Z level=debug event=want client=14 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05476worker2 # [ 78.066911] nix-grpc-daemon[783]: ts=2026-09-22T11:33:37Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=15477worker1 # [ 78.074804] nix-grpc-daemon[780]: ts=2026-09-22T11:33:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462025478worker1 # [ 78.085370] nix-grpc-daemon[780]: ts=2026-09-22T11:33:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462025479client # copying 2 paths to 'lb:50051'...5480client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5481worker1 # [ 78.098756] nix-grpc-daemon[780]: ts=2026-09-22T11:33:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462025482client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5483worker1 # [ 78.110136] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5484worker1 # [ 78.206746] nix-daemon[844]: accepted connection from pid 1108, user nix-grpc-daemon (trusted)5485worker1 # [ 78.220358] nix-daemon[844]: reaped child process 1111, status = succeeded5486worker1 # [ 78.302106] nix-daemon[844]: accepted connection from pid 1113, user nix-grpc-daemon (trusted)5487worker1 # [ 78.316809] nix-daemon[844]: reaped child process 1116, status = succeeded5488lb # [ 78.286698] niks3-server[949]: 2026/09/22 11:33:36 INFO Received uploads request method=POST path=/api/pending_closures5489lb # [ 78.294693] niks3-server[949]: 2026/09/22 11:33:36 INFO Received uploads request method=POST path=/api/pending_closures5490worker1 # [ 78.340648] nix-grpc-daemon[872]: time=2026-09-22T11:33:36.705Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5491worker1 # [ 78.342046] nix-grpc-daemon[872]: time=2026-09-22T11:33:36.706Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5492worker1 # [ 78.344423] nix-grpc-daemon[872]: time=2026-09-22T11:33:36.708Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5493lb # [ 78.328766] niks3-server[949]: 2026/09/22 11:33:36 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5494lb # [ 78.351477] niks3-server[949]: 2026/09/22 11:33:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5495lb # [ 78.353993] niks3-server[949]: 2026/09/22 11:33:36 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5496lb # [ 78.357052] niks3-server[949]: 2026/09/22 11:33:36 INFO Signed narinfos id=29 count=25497lb # [ 78.358704] niks3-server[949]: 2026/09/22 11:33:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5498lb # [ 78.360631] niks3-server[949]: 2026/09/22 11:33:36 INFO Signed narinfos id=28 count=15499worker1 # [ 78.398421] nix-grpc-daemon[872]: time=2026-09-22T11:33:36.762Z level=INFO msg="Uploading 3 narinfos"5500lb # [ 78.362968] niks3-server[949]: 2026/09/22 11:33:36 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5501lb # [ 78.375156] niks3-server[949]: 2026/09/22 11:33:36 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5502lb # [ 78.378175] niks3-server[949]: 2026/09/22 11:33:36 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5503lb # [ 78.383420] niks3-server[949]: 2026/09/22 11:33:36 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5504lb # [ 78.386488] niks3-server[949]: 2026/09/22 11:33:36 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5505lb # [ 78.390141] niks3-server[949]: 2026/09/22 11:33:36 INFO Completed upload id=285506lb # [ 78.391806] niks3-server[949]: 2026/09/22 11:33:36 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5507lb # [ 78.394273] niks3-server[949]: 2026/09/22 11:33:36 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5508lb # [ 78.396709] niks3-server[949]: 2026/09/22 11:33:36 INFO Completed upload id=295509worker1 # [ 78.435075] nix-grpc-daemon[872]: time=2026-09-22T11:33:36.799Z level=INFO msg="Upload complete. (311ms)"5510worker1 # [ 78.439941] nix-grpc-daemon[780]: ts=2026-09-22T11:33:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 paths=2 nar_bytes_in=9765511worker1 # [ 78.445553] nix-grpc-daemon[780]: ts=2026-09-22T11:33:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462025512worker1 # [ 78.457124] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5513worker1 # [ 78.468288] nix-daemon[844]: reaped child process 1106, status = succeeded5514worker1 # [ 78.472577] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5515client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5516client # uses-input-w1onlya> node-a: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5517client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5518worker1 # [ 79.508942] nix-daemon[844]: reaped child process 1120, status = succeeded5519worker1 # [ 79.585942] nix-daemon[844]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5520worker1 # [ 79.598729] nix-daemon[844]: reaped child process 1139, status = succeeded5521lb # [ 79.569455] niks3-server[949]: 2026/09/22 11:33:37 INFO Received uploads request method=POST path=/api/pending_closures5522worker1 # [ 79.614716] nix-grpc-daemon[872]: time=2026-09-22T11:33:37.979Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5523lb # [ 79.599103] niks3-server[949]: 2026/09/22 11:33:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5524lb # [ 79.607970] niks3-server[949]: 2026/09/22 11:33:37 INFO Signed narinfos id=30 count=15525worker1 # [ 79.648433] nix-grpc-daemon[872]: time=2026-09-22T11:33:38.012Z level=INFO msg="Uploading 1 narinfos"5526lb # [ 79.614926] niks3-server[949]: 2026/09/22 11:33:37 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5527lb # [ 79.621121] niks3-server[949]: 2026/09/22 11:33:37 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5528lb # [ 79.632972] niks3-server[949]: 2026/09/22 11:33:37 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5529worker1 # [ 79.678377] nix-grpc-daemon[872]: time=2026-09-22T11:33:38.042Z level=INFO msg="Upload complete. (197ms)"5530lb # [ 79.641598] niks3-server[949]: 2026/09/22 11:33:37 INFO Completed upload id=305531worker1 # [ 79.680876] nix-grpc-daemon[780]: ts=2026-09-22T11:33:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=15 outputs=15532lb # [ 79.644814] niks3-server[949]: 2026/09/22 11:33:37 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5533worker1 # [ 79.688077] nix-daemon[844]: reaped child process 1118, status = succeeded5534client # copying 1 paths...5535worker1 # [ 79.728853] nix-grpc-daemon[780]: ts=2026-09-22T11:33:38Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:462025536client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5537worker1 # [ 79.749801] nix-grpc-daemon[780]: ts=2026-09-22T11:33:38Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:462025538worker1 # [ 79.754793] nix-grpc-daemon[780]: ts=2026-09-22T11:33:38Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 paths=1 nar_bytes_out=1205539client # [ 79.846905] nix-daemon[1103]: reaped child process 1189, status = succeeded5540client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.26 seconds)5541client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&25542client # [ 79.946197] nix-daemon[1103]: accepted connection from pid 1208, user root (trusted)5543client # this derivation will be built:5544client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5545worker1 # [ 80.080678] nix-grpc-daemon[780]: ts=2026-09-22T11:33:38Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:462025546client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5547worker2 # [ 80.079811] nix-grpc-daemon[783]: ts=2026-09-22T11:33:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558325548worker2 # [ 80.101206] nix-grpc-daemon[783]: ts=2026-09-22T11:33:39Z level=debug event=want client=15 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05549worker2 # [ 80.106722] nix-grpc-daemon[783]: ts=2026-09-22T11:33:39Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=15550worker2 # [ 80.112519] nix-grpc-daemon[783]: ts=2026-09-22T11:33:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325551worker2 # [ 80.123416] nix-grpc-daemon[783]: ts=2026-09-22T11:33:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558325552client # copying 1 paths to 'lb:50051'...5553client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5554worker2 # [ 80.139086] nix-grpc-daemon[783]: ts=2026-09-22T11:33:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558325555worker2 # [ 80.147726] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5556worker2 # [ 80.164914] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 44 ms (attempt 1/5)5557worker2 # [ 80.215477] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 174 ms (attempt 2/5)5558worker2 # [ 80.391609] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 136 ms (attempt 3/5)5559worker2 # [ 80.529993] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 763 ms (attempt 4/5)5560worker2 # [ 81.295203] nix-grpc-daemon[783]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5561worker2 # [ 81.296788] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5562worker2 # [ 81.298971] nix-grpc-daemon[783]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5563worker2 # [ 81.392204] nix-daemon[867]: accepted connection from pid 1321, user nix-grpc-daemon (trusted)5564worker2 # [ 81.408483] nix-daemon[867]: reaped child process 1324, status = succeeded5565worker2 # [ 81.489866] nix-daemon[867]: accepted connection from pid 1326, user nix-grpc-daemon (trusted)5566worker2 # [ 81.506610] nix-daemon[867]: reaped child process 1329, status = succeeded5567lb # [ 81.482615] niks3-server[949]: 2026/09/22 11:33:39 INFO Received uploads request method=POST path=/api/pending_closures5568worker2 # [ 81.522908] nix-grpc-daemon[882]: time=2026-09-22T11:33:40.688Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5569worker2 # [ 81.524861] nix-grpc-daemon[882]: time=2026-09-22T11:33:40.690Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5570lb # [ 81.513981] niks3-server[949]: 2026/09/22 11:33:39 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5571lb # [ 81.525790] niks3-server[949]: 2026/09/22 11:33:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5572worker2 # [ 81.562286] nix-grpc-daemon[882]: time=2026-09-22T11:33:40.728Z level=INFO msg="Uploading 1 narinfos"5573lb # [ 81.530616] niks3-server[949]: 2026/09/22 11:33:39 INFO Signed narinfos id=31 count=15574lb # [ 81.538290] niks3-server[949]: 2026/09/22 11:33:39 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5575lb # [ 81.548294] niks3-server[949]: 2026/09/22 11:33:39 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5576worker2 # [ 81.587325] nix-grpc-daemon[882]: time=2026-09-22T11:33:40.753Z level=INFO msg="Upload complete. (275ms)"5577lb # [ 81.555844] niks3-server[949]: 2026/09/22 11:33:39 INFO Completed upload id=315578worker2 # [ 81.589907] nix-grpc-daemon[783]: ts=2026-09-22T11:33:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=1 paths=1 nar_bytes_in=5605579lb # [ 81.560874] niks3-server[949]: 2026/09/22 11:33:39 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5580worker2 # [ 81.594847] nix-grpc-daemon[783]: ts=2026-09-22T11:33:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558325581worker2 # [ 81.604124] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5582worker2 # [ 81.614829] nix-daemon[867]: reaped child process 1314, status = succeeded5583worker2 # [ 81.616558] nix-grpc-daemon[783]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5584worker2 # [ 81.624404] nix-daemon[867]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5585client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5586client # uses-input-w1onlyb> worker2: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5587client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5588worker2 # [ 82.647222] nix-daemon[867]: reaped child process 1334, status = succeeded5589worker2 # [ 82.727370] nix-daemon[867]: accepted connection from pid 1350, user nix-grpc-daemon (trusted)5590worker2 # [ 82.741087] nix-daemon[867]: reaped child process 1353, status = succeeded5591lb # [ 82.718401] niks3-server[949]: 2026/09/22 11:33:41 INFO Received uploads request method=POST path=/api/pending_closures5592worker2 # [ 82.758144] nix-grpc-daemon[882]: time=2026-09-22T11:33:41.924Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5593lb # [ 82.757688] niks3-server[949]: 2026/09/22 11:33:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5594worker2 # [ 82.794696] nix-grpc-daemon[882]: time=2026-09-22T11:33:41.960Z level=INFO msg="Uploading 1 narinfos"5595lb # [ 82.763263] niks3-server[949]: 2026/09/22 11:33:41 INFO Signed narinfos id=32 count=15596lb # [ 82.770842] niks3-server[949]: 2026/09/22 11:33:41 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5597lb # [ 82.780360] niks3-server[949]: 2026/09/22 11:33:41 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5598lb # [ 82.790800] niks3-server[949]: 2026/09/22 11:33:41 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5599lb # [ 82.799123] niks3-server[949]: 2026/09/22 11:33:41 INFO Completed upload id=325600worker2 # [ 82.830781] nix-grpc-daemon[882]: time=2026-09-22T11:33:41.996Z level=INFO msg="Upload complete. (197ms)"5601lb # [ 82.801304] niks3-server[949]: 2026/09/22 11:33:41 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5602client # copying 1 paths...5603worker2 # [ 82.839452] nix-grpc-daemon[783]: ts=2026-09-22T11:33:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55832 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=16 outputs=15604worker1 # [ 82.851221] nix-grpc-daemon[780]: ts=2026-09-22T11:33:41Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:462025605worker1 # [ 82.856794] nix-grpc-daemon[780]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5606worker2 # [ 82.853553] nix-daemon[867]: reaped child process 1331, status = succeeded5607client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5608worker1 # [ 82.882347] nix-grpc-daemon[780]: ts=2026-09-22T11:33:41Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:462025609worker1 # [ 82.887382] nix-grpc-daemon[780]: ts=2026-09-22T11:33:41Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 paths=1 nar_bytes_out=1205610client # [ 82.987268] nix-daemon[1103]: reaped child process 1211, status = succeeded5611client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.14 seconds)5612client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&25613client # [ 83.088873] nix-daemon[1103]: accepted connection from pid 1230, user root (trusted)5614client # this derivation will be built:5615client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5616worker1 # [ 83.215144] nix-grpc-daemon[780]: ts=2026-09-22T11:33:41Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:462025617client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5618worker1 # [ 83.218288] nix-grpc-daemon[780]: ts=2026-09-22T11:33:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462025619worker1 # [ 83.222629] nix-grpc-daemon[872]: time=2026-09-22T11:33:41.586Z level=INFO msg="All 1 paths already cached"5620worker2 # [ 83.231258] nix-grpc-daemon[783]: ts=2026-09-22T11:33:42Z level=debug event=want client=16 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05621worker2 # [ 83.235570] nix-grpc-daemon[783]: ts=2026-09-22T11:33:42Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=15622worker1 # [ 83.248596] nix-grpc-daemon[780]: ts=2026-09-22T11:33:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462025623worker1 # [ 83.256927] nix-grpc-daemon[780]: ts=2026-09-22T11:33:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462025624worker1 # [ 83.265964] nix-grpc-daemon[872]: time=2026-09-22T11:33:41.630Z level=INFO msg="All 1 paths already cached"5625client # copying 1 paths to 'lb:50051'...5626client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5627worker1 # [ 83.273473] nix-grpc-daemon[780]: ts=2026-09-22T11:33:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462025628worker1 # [ 83.284328] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5629worker1 # [ 83.382429] nix-daemon[844]: accepted connection from pid 1144, user nix-grpc-daemon (trusted)5630worker1 # [ 83.395221] nix-daemon[844]: reaped child process 1147, status = succeeded5631worker1 # [ 83.478036] nix-daemon[844]: accepted connection from pid 1149, user nix-grpc-daemon (trusted)5632worker1 # [ 83.490340] nix-daemon[844]: reaped child process 1152, status = succeeded5633lb # [ 83.461176] niks3-server[949]: 2026/09/22 11:33:41 INFO Received uploads request method=POST path=/api/pending_closures5634worker1 # [ 83.509691] nix-grpc-daemon[872]: time=2026-09-22T11:33:41.874Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5635worker1 # [ 83.511235] nix-grpc-daemon[872]: time=2026-09-22T11:33:41.875Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5636lb # [ 83.497130] niks3-server[949]: 2026/09/22 11:33:41 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5637lb # [ 83.509129] niks3-server[949]: 2026/09/22 11:33:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5638lb # [ 83.513499] niks3-server[949]: 2026/09/22 11:33:41 INFO Signed narinfos id=33 count=15639worker1 # [ 83.554603] nix-grpc-daemon[872]: time=2026-09-22T11:33:41.919Z level=INFO msg="Uploading 1 narinfos"5640lb # [ 83.521973] niks3-server[949]: 2026/09/22 11:33:41 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5641lb # [ 83.527805] niks3-server[949]: 2026/09/22 11:33:41 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5642lb # [ 83.531372] niks3-server[949]: 2026/09/22 11:33:41 INFO Completed upload id=335643worker1 # [ 83.568527] nix-grpc-daemon[872]: time=2026-09-22T11:33:41.932Z level=INFO msg="Upload complete. (272ms)"5644lb # [ 83.533347] niks3-server[949]: 2026/09/22 11:33:41 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5645worker1 # [ 83.572602] nix-grpc-daemon[780]: ts=2026-09-22T11:33:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 paths=1 nar_bytes_in=5605646worker1 # [ 83.577380] nix-grpc-daemon[780]: ts=2026-09-22T11:33:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462025647worker1 # [ 83.588277] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5648worker1 # [ 83.591911] nix-daemon[844]: reaped child process 1142, status = succeeded5649worker1 # [ 83.596505] nix-daemon[844]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5650client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5651client # uses-input-w1onlyc> node-a: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5652client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5653worker1 # [ 84.229562] nix-daemon[844]: reaped child process 1156, status = succeeded5654worker1 # [ 84.300481] nix-daemon[844]: accepted connection from pid 1172, user nix-grpc-daemon (trusted)5655worker1 # [ 84.312547] nix-daemon[844]: reaped child process 1175, status = succeeded5656lb # [ 84.283831] niks3-server[949]: 2026/09/22 11:33:42 INFO Received uploads request method=POST path=/api/pending_closures5657worker1 # [ 84.329359] nix-grpc-daemon[872]: time=2026-09-22T11:33:42.693Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5658lb # [ 84.313254] niks3-server[949]: 2026/09/22 11:33:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5659worker1 # [ 84.355692] nix-grpc-daemon[872]: time=2026-09-22T11:33:42.720Z level=INFO msg="Uploading 1 narinfos"5660lb # [ 84.318500] niks3-server[949]: 2026/09/22 11:33:42 INFO Signed narinfos id=34 count=15661lb # [ 84.329410] niks3-server[949]: 2026/09/22 11:33:42 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5662lb # [ 84.339977] niks3-server[949]: 2026/09/22 11:33:42 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5663lb # [ 84.347118] niks3-server[949]: 2026/09/22 11:33:42 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5664lb # [ 84.353665] niks3-server[949]: 2026/09/22 11:33:42 INFO Completed upload id=345665worker1 # [ 84.390816] nix-grpc-daemon[872]: time=2026-09-22T11:33:42.755Z level=INFO msg="Upload complete. (188ms)"5666lb # [ 84.358093] niks3-server[949]: 2026/09/22 11:33:42 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5667worker1 # [ 84.396921] nix-grpc-daemon[780]: ts=2026-09-22T11:33:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=17 outputs=15668client # copying 1 paths...5669worker1 # [ 84.410193] nix-daemon[844]: reaped child process 1154, status = succeeded5670worker2 # [ 84.404993] nix-grpc-daemon[783]: ts=2026-09-22T11:33:43Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558325671worker2 # [ 84.415536] nix-grpc-daemon[783]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5672client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5673worker1 # [ 84.452822] nix-grpc-daemon[780]: ts=2026-09-22T11:33:42Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:462025674worker1 # [ 84.458488] nix-grpc-daemon[780]: ts=2026-09-22T11:33:42Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:46202 duration_s=0 paths=1 nar_bytes_out=1205675client # [ 84.515837] nix-daemon[1103]: reaped child process 1233, status = succeeded5676client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&2, in 1.53 seconds)5677(finished: subtest: an input only one worker has still reaches the builder, in 7.41 seconds)5678subtest: upload whose reference the worker lacks is completed from the cache5679client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache5680client # this derivation will be built:5681client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5682client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5683client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache, in 0.25 seconds)5684client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache5685client # this derivation will be built:5686client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5687client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5688client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache, in 0.25 seconds)5689client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5690client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.10 seconds)5691worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5692worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.10 seconds)5693worker2 # [ 85.276686] systemd[1]: Stopping Nix worker-protocol over gRPC...5694worker2 # [ 85.305083] nix-grpc-daemon[783]: ts=2026-09-22T11:33:44Z level=info event=signal_exit5695lb # [ 85.344620] niks3-server[949]: 2026/09/22 11:33:43 INFO lead: released remote=192.168.1.4:380225696worker1 # [ 86.380822] nix-grpc-daemon[780]: ts=2026-09-22T11:33:44Z level=info event=scheduler_restarting addr=lb:500515697worker2 # [ 86.381255] nix-grpc-daemon[783]: ts=2026-09-22T11:33:45Z level=info event=scheduler_restarting5698worker2 # [ 86.384879] nix-grpc-daemon[783]: ts=2026-09-22T11:33:45Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:558285699worker2 # [ 86.389094] nix-grpc-daemon[783]: ts=2026-09-22T11:33:45Z level=info event=worker_gone addr=192.168.1.3:500515700worker2 # [ 86.392612] nix-grpc-daemon[783]: ts=2026-09-22T11:33:45Z level=info event=scheduler_restarting addr=lb:500515701worker2 # [ 86.394249] nix-grpc-daemon[783]: ts=2026-09-22T11:33:45Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:558285702worker2 # [ 86.395670] nix-grpc-daemon[783]: ts=2026-09-22T11:33:45Z level=info event=worker_gone addr=192.168.1.4:500515703worker2 # [ 86.407233] nix-daemon[867]: reaped child process 869, status = succeeded5704worker2 # [ 86.422896] nix-grpc-daemon[783]: E0000 00:00:1790076825.589097 828 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true5705worker2 # [ 86.424832] nix-grpc-daemon[783]: E0000 00:00:1790076825.589174 828 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 10, started true, shutdown true5706worker1 # [ 86.581563] nix-grpc-daemon[780]: ts=2026-09-22T11:33:44Z level=info event=scheduler_connected addr=lb:500515707worker1 # [ 86.585633] nix-grpc-daemon[780]: ts=2026-09-22T11:33:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5708worker2 # [ 86.594524] nix-grpc-daemon[783]: ts=2026-09-22T11:33:45Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5709worker1 # [ 86.785960] nix-grpc-daemon[780]: ts=2026-09-22T11:33:45Z level=info event=scheduler_connected addr=lb:500515710worker1 # [ 86.787767] nix-grpc-daemon[780]: ts=2026-09-22T11:33:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5711worker1 # [ 86.988300] nix-grpc-daemon[780]: ts=2026-09-22T11:33:45Z level=info event=scheduler_connected addr=lb:500515712worker1 # [ 86.990360] nix-grpc-daemon[780]: ts=2026-09-22T11:33:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5713lb # [ 87.040674] envoy[918]: conn peer=[2001:db8:1::1]:58728 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3861 tx=2065 ms=19695714lb # [ 87.044415] envoy[918]: conn peer=[2001:db8:1::1]:58738 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=575715lb # [ 87.047884] envoy[918]: conn peer=[2001:db8:1::1]:53222 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=565716lb # [ 87.051441] envoy[918]: conn peer=[2001:db8:1::1]:58740 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3727 tx=2148 ms=28605717lb # [ 87.055154] envoy[918]: conn peer=[2001:db8:1::1]:53230 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3712 tx=2080 ms=12535718lb # [ 87.058666] envoy[918]: conn peer=[2001:db8:1::1]:53232 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=421 tx=328 ms=185719lb # [ 87.062119] envoy[918]: conn peer=[2001:db8:1::4]:60902 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=47646 tx=2927 ms=692555720worker1 # [ 87.190994] nix-grpc-daemon[780]: ts=2026-09-22T11:33:45Z level=info event=scheduler_connected addr=lb:500515721worker1 # [ 87.192806] nix-grpc-daemon[780]: ts=2026-09-22T11:33:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5722worker1 # [ 87.372540] nix-grpc-daemon[780]: ts=2026-09-22T11:33:45Z level=info event=scheduler_take_over5723lb # [ 87.335951] niks3-server[949]: 2026/09/22 11:33:45 INFO lead: acquired remote=192.168.1.3:425845724worker1 # [ 87.393598] nix-grpc-daemon[780]: ts=2026-09-22T11:33:45Z level=info event=scheduler_connected addr=lb:500515725worker1 # [ 87.398654] nix-grpc-daemon[780]: ts=2026-09-22T11:33:45Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:453945726worker1 # [ 87.400145] nix-grpc-daemon[780]: ts=2026-09-22T11:33:45Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05727worker2 # [ 88.093689] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5728worker2 # [ 88.097573] systemd[1]: Stopped Nix worker-protocol over gRPC.5729worker2 # [ 88.101189] systemd[1]: nix-grpc-daemon.service: Consumed 3.001s CPU time over 1min 10.938s wall clock time, 70.4M memory peak, 106.5K incoming IP traffic, 268K outgoing IP traffic.5730worker2 # [ 88.112894] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5731worker2 # [ 88.117093] systemd[1]: Closed Nix worker-protocol over gRPC.5732worker2 # [ 88.119882] systemd[1]: nix-grpc-daemon.socket: Consumed 355us CPU time over 1min 20.516s wall clock time, 256K memory peak, 116.3K incoming IP traffic, 98.3K outgoing IP traffic.5733lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5734lb: must succeed: curl -sf localhost:9901/clusters5735lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5736lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.03 seconds)5737client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25738worker1 # [ 88.303814] nix-grpc-daemon[780]: ts=2026-09-22T11:33:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:462025739client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5740worker1 # [ 88.306165] nix-grpc-daemon[780]: ts=2026-09-22T11:33:46Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:462025741client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.16 seconds)5742worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5743worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.02 seconds)5744worker2 # [ 88.379781] systemd[1]: Listening on Nix worker-protocol over gRPC.5745worker1 # [ 88.411265] systemd[1]: Stopping Nix worker-protocol over gRPC...5746worker2 # [ 89.088644] systemd[1]: Starting scheduler-token.service...5747worker2 # [ 89.153106] systemd[1]: scheduler-token.service: Deactivated successfully.5748worker2 # [ 89.156131] systemd[1]: Finished scheduler-token.service.5749worker2 # [ 89.159453] systemd[1]: scheduler-token.service: Consumed 36ms CPU time over 64ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5750worker2 # [ 89.167739] systemd[1]: Starting Nix worker-protocol over gRPC...5751worker2 # [ 89.246861] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:48Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5752worker2 # [ 89.249227] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:48Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5753worker1 # [ 89.258128] nix-grpc-daemon[780]: ts=2026-09-22T11:33:47Z level=info event=signal_exit5754worker2 # [ 89.252646] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:48Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15755worker2 # [ 89.257092] nix-grpc-daemon[1380]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5756worker2 # [ 89.259039] nix-grpc-daemon[1380]: E0000 00:00:1790076828.423193 1380 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5757worker2 # [ 89.266068] nix-grpc-daemon[1380]: E0000 00:00:1790076828.429234 1380 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5758worker2 # [ 89.268520] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:48Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5759worker2 # [ 89.273027] systemd[1]: Started Nix worker-protocol over gRPC.5760worker2 # [ 89.277831] nix-grpc-daemon[1380]: E0000 00:00:1790076828.443402 1422 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5761worker2 # [ 89.291546] nix-grpc-daemon[1380]: E0000 00:00:1790076828.457774 1422 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5762worker2 # [ 89.296128] nix-grpc-daemon[1380]: E0000 00:00:1790076828.460381 1422 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5763worker2 # [ 89.306468] nix-grpc-daemon[1380]: E0000 00:00:1790076828.472712 1422 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5764worker2 # [ 89.314796] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:48Z level=info event=scheduler_connected addr=lb:500515765worker1 # [ 89.321847] nix-grpc-daemon[780]: ts=2026-09-22T11:33:47Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:453945766worker1 # [ 89.323776] nix-grpc-daemon[780]: ts=2026-09-22T11:33:47Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05767lb # [ 89.341925] niks3-server[949]: 2026/09/22 11:33:47 INFO lead: released remote=192.168.1.3:425845768worker2 # [ 90.374231] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:49Z level=info event=scheduler_restarting addr=lb:500515769worker1 # [ 90.376795] nix-grpc-daemon[780]: ts=2026-09-22T11:33:48Z level=info event=scheduler_restarting5770worker1 # [ 90.386554] nix-grpc-daemon[780]: ts=2026-09-22T11:33:48Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:453945771worker1 # [ 90.390881] nix-grpc-daemon[780]: ts=2026-09-22T11:33:48Z level=info event=worker_gone addr=192.168.1.4:500515772worker1 # [ 90.394304] nix-grpc-daemon[780]: ts=2026-09-22T11:33:48Z level=info event=scheduler_restarting addr=lb:500515773worker1 # [ 90.397594] nix-grpc-daemon[780]: ts=2026-09-22T11:33:48Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:453945774worker1 # [ 90.401288] nix-grpc-daemon[780]: ts=2026-09-22T11:33:48Z level=info event=worker_gone addr=192.168.1.3:500515775worker1 # [ 90.404713] nix-grpc-daemon[780]: E0000 00:00:1790076828.755248 825 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5776worker1 # [ 90.422550] nix-daemon[844]: reaped child process 846, status = succeeded5777worker2 # [ 90.574431] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:49Z level=info event=scheduler_connected addr=lb:500515778worker2 # [ 90.578937] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5779worker1 # [ 90.595399] nix-grpc-daemon[780]: ts=2026-09-22T11:33:48Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5780worker2 # [ 90.780218] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:49Z level=info event=scheduler_connected addr=lb:500515781worker2 # [ 90.784615] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5782worker2 # [ 90.985975] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:50Z level=info event=scheduler_connected addr=lb:500515783worker2 # [ 90.990543] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5784worker2 # [ 91.192059] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:50Z level=info event=scheduler_connected addr=lb:500515785worker2 # [ 91.196610] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5786worker1 # [ 91.386537] nix-grpc-daemon[780]: E0000 00:00:1790076829.750338 825 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true5787worker2 # [ 91.399247] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:50Z level=info event=scheduler_connected addr=lb:500515788worker2 # [ 91.402917] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5789worker2 # [ 91.600466] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:50Z level=info event=scheduler_connected addr=lb:500515790worker2 # [ 91.605353] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5791worker2 # [ 91.806332] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:50Z level=info event=scheduler_connected addr=lb:500515792worker2 # [ 91.810713] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5793worker2 # [ 92.012119] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:51Z level=info event=scheduler_connected addr=lb:500515794worker2 # [ 92.016702] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5795worker1 # [ 92.054898] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5796worker1 # [ 92.056492] systemd[1]: Stopped Nix worker-protocol over gRPC.5797worker1 # [ 92.058101] systemd[1]: nix-grpc-daemon.service: Consumed 2.227s CPU time over 1min 14.984s wall clock time, 61.1M memory peak, 81.7K incoming IP traffic, 148.5K outgoing IP traffic.5798worker1 # [ 92.062881] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5799worker1 # [ 92.064629] systemd[1]: Closed Nix worker-protocol over gRPC.5800worker1 # [ 92.066084] systemd[1]: nix-grpc-daemon.socket: Consumed 295us CPU time over 1min 24.253s wall clock time, 256K memory peak, 91.8K incoming IP traffic, 80.7K outgoing IP traffic.5801lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5802lb: must succeed: curl -sf localhost:9901/clusters5803lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5804lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.03 seconds)5805client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25806worker2 # [ 92.211874] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:530545807worker2 # [ 92.219312] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:51Z level=info event=scheduler_connected addr=lb:500515808worker2 # [ 92.223191] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5809worker2 # [ 92.230788] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)5810worker2 # [ 92.249858] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 35 ms (attempt 1/5)5811worker2 # [ 92.293185] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 1 ms (attempt 2/5)5812worker2 # [ 92.296528] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 387 ms (attempt 3/5)5813worker2 # [ 92.421396] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:51Z level=info event=scheduler_connected addr=lb:500515814worker2 # [ 92.425162] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5815worker2 # [ 92.622070] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:51Z level=info event=scheduler_connected addr=lb:500515816worker2 # [ 92.623877] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5817worker2 # [ 92.687476] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 594 ms (attempt 4/5)5818worker2 # [ 92.824625] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:51Z level=info event=scheduler_connected addr=lb:500515819worker2 # [ 92.826522] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5820worker2 # [ 93.027340] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:52Z level=info event=scheduler_connected addr=lb:500515821worker2 # [ 93.029233] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5822worker2 # [ 93.229938] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:52Z level=info event=scheduler_connected addr=lb:500515823worker2 # [ 93.233468] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5824worker2 # [ 93.284326] nix-grpc-daemon[1380]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5825worker2 # [ 93.287925] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5826client # copying 2 paths...5827client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5828worker2 # [ 93.308645] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:530545829client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5830worker2 # [ 93.322505] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)5831worker2 # [ 93.343675] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:52Z level=info event=niks3_push_started pid=14405832worker2 # [ 93.436524] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:52Z level=info event=scheduler_connected addr=lb:500515833worker2 # [ 93.440116] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5834worker2 # [ 93.447352] nix-grpc-daemon[1440]: time=2026-09-22T11:33:52.613Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""5835worker2 # [ 93.530404] nix-daemon[867]: accepted connection from pid 1453, user nix-grpc-daemon (trusted)5836worker2 # [ 93.545618] nix-daemon[867]: reaped child process 1456, status = succeeded5837lb # [ 93.520807] niks3-server[949]: 2026/09/22 11:33:51 INFO Received uploads request method=POST path=/api/pending_closures5838lb # [ 93.526649] niks3-server[949]: 2026/09/22 11:33:51 INFO Received uploads request method=POST path=/api/pending_closures5839worker2 # [ 93.561377] nix-grpc-daemon[1440]: time=2026-09-22T11:33:52.727Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5840worker2 # [ 93.562853] nix-grpc-daemon[1440]: time=2026-09-22T11:33:52.729Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5841worker2 # [ 93.565324] nix-grpc-daemon[1440]: time=2026-09-22T11:33:52.730Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5842lb # [ 93.553472] niks3-server[949]: 2026/09/22 11:33:51 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5843lb # [ 93.558127] niks3-server[949]: 2026/09/22 11:33:51 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5844lb # [ 93.564678] niks3-server[949]: 2026/09/22 11:33:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5845lb # [ 93.567564] niks3-server[949]: 2026/09/22 11:33:51 INFO Signed narinfos id=35 count=15846lb # [ 93.570183] niks3-server[949]: 2026/09/22 11:33:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5847lb # [ 93.572594] niks3-server[949]: 2026/09/22 11:33:51 INFO Signed narinfos id=36 count=25848worker2 # [ 93.604805] nix-grpc-daemon[1440]: time=2026-09-22T11:33:52.770Z level=INFO msg="Uploading 3 narinfos"5849lb # [ 93.577150] niks3-server[949]: 2026/09/22 11:33:51 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5850lb # [ 93.579512] niks3-server[949]: 2026/09/22 11:33:51 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5851lb # [ 93.592190] niks3-server[949]: 2026/09/22 11:33:51 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5852lb # [ 93.599713] niks3-server[949]: 2026/09/22 11:33:51 INFO Completed upload id=355853lb # [ 93.603630] niks3-server[949]: 2026/09/22 11:33:51 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5854lb # [ 93.605166] niks3-server[949]: 2026/09/22 11:33:51 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5855worker2 # [ 93.639827] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:52Z level=info event=scheduler_connected addr=lb:500515856worker2 # [ 93.641620] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5857worker2 # [ 93.643415] nix-grpc-daemon[1440]: time=2026-09-22T11:33:52.809Z level=INFO msg="Upload complete. (192ms)"5858lb # [ 93.612808] niks3-server[949]: 2026/09/22 11:33:51 INFO Completed upload id=365859worker2 # [ 93.645753] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53054 duration_s=0 paths=2 nar_bytes_in=3125860worker2 # [ 93.652749] nix-daemon[867]: reaped child process 1438, status = succeeded5861lb # [ 93.622855] niks3-server[949]: 2026/09/22 11:33:51 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5862lb # [ 93.625442] niks3-server[949]: 2026/09/22 11:33:51 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5863client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.58 seconds)5864worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5865worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.01 seconds)5866worker1 # [ 93.730849] systemd[1]: Listening on Nix worker-protocol over gRPC.5867worker1 # [ 93.736313] systemd[1]: Starting Nix worker-protocol over gRPC...5868worker1 # [ 93.830979] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5869worker1 # [ 93.835596] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5870worker1 # [ 93.837590] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15871worker1 # [ 93.840923] nix-grpc-daemon[1198]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5872worker1 # [ 93.842198] nix-grpc-daemon[1198]: E0000 00:00:1790076832.202998 1198 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5873worker2 # [ 93.842459] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:53Z level=info event=scheduler_connected addr=lb:500515874worker2 # [ 93.844617] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5875worker1 # [ 93.849967] nix-grpc-daemon[1198]: E0000 00:00:1790076832.210143 1198 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5876worker1 # [ 93.854110] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5877worker1 # [ 93.856331] systemd[1]: Started Nix worker-protocol over gRPC.5878worker1 # [ 93.857831] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=scheduler_take_over5879lb # [ 93.819723] niks3-server[949]: 2026/09/22 11:33:52 INFO lead: acquired remote=192.168.1.3:567785880worker1 # [ 93.862286] nix-grpc-daemon[1198]: E0000 00:00:1790076832.221096 1240 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5881worker1 # [ 93.873072] nix-grpc-daemon[1198]: E0000 00:00:1790076832.236743 1240 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5882worker1 # [ 93.875187] nix-grpc-daemon[1198]: E0000 00:00:1790076832.236892 1240 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5883lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5884lb: must succeed: curl -sf localhost:9901/clusters5885worker1 # [ 93.887921] nix-grpc-daemon[1198]: E0000 00:00:1790076832.252376 1240 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5886worker1 # [ 93.902942] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=scheduler_connected addr=lb:500515887worker1 # [ 93.904812] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5888lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5889worker2 # [ 94.045642] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:53Z level=info event=scheduler_connected addr=lb:500515890worker2 # [ 94.048075] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5891worker1 # [ 94.106292] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=scheduler_connected addr=lb:500515892worker1 # [ 94.113106] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:519265893worker1 # [ 94.116684] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05894worker2 # [ 94.249553] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:53Z level=info event=scheduler_connected addr=lb:500515895worker1 # [ 94.264189] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=oidc_jwks provider=mock keys=15896worker1 # [ 94.268434] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:519265897worker1 # [ 94.272365] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:52Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05898lb: must succeed: curl -sf localhost:9901/clusters5899lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5900lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.10 seconds)5901(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.45 seconds)5902subtest: a niks3 restart does not cost the leader its role5903lb: must succeed: curl -sf localhost:9901/clusters5904lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5905lb: must succeed: curl -sf localhost:9901/clusters5906lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5907lb # [ 95.023077] systemd[1]: Stopping niks3 server...5908lb # [ 95.025450] niks3-server[949]: 2026/09/22 11:33:53 INFO Shutdown signal received, draining in-flight requests timeout=10s5909lb # [ 95.028832] niks3-server[949]: 2026/09/22 11:33:53 INFO lead: released remote=192.168.1.3:567785910worker2 # [ 95.061380] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:54Z level=info event=scheduler_lock_lost status=2005911worker1 # [ 95.067288] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:53Z level=info event=scheduler_lock_lost status=2005912lb # [ 95.043975] systemd[1]: niks3.service: Deactivated successfully.5913lb # [ 95.046105] systemd[1]: Stopped niks3 server.5914lb # [ 95.046799] systemd[1]: niks3.service: Consumed 672ms CPU time over 1min 17.941s wall clock time, 28.6M memory peak, 94.1K incoming IP traffic, 136.2K outgoing IP traffic.5915lb # [ 95.054141] systemd[1]: Starting niks3 server...5916lb # [ 95.105438] niks3-server[1092]: 2026/09/22 11:33:53 goose: no migrations to run. current version: 202609200000005917lb # [ 95.111727] niks3-server[1092]: 2026/09/22 11:33:53 OK 1_commit_pending_closure.sql (4.39ms)5918lb # [ 95.115447] niks3-server[1092]: 2026/09/22 11:33:53 OK 2_object_stats_trigger.sql (2.05ms)5919lb # [ 95.116610] niks3-server[1092]: 2026/09/22 11:33:53 goose: up to current file version: 25920lb # [ 95.118140] niks3-server[1092]: 2026/09/22 11:33:53 INFO Loaded signing key name=farm-test-1 path=/nix/store/hk0nhypl37jvkn3gr36v3y7qx593p39i-key5921lb # [ 95.129922] niks3-server[1092]: 2026/09/22 11:33:53 INFO Read proxy enabled — serving cache objects from S35922lb # [ 95.131209] niks3-server[1092]: 2026/09/22 11:33:53 INFO Using socket-activated listener address=0.0.0.0:57515923lb # [ 95.133367] niks3-server[1092]: 2026/09/22 11:33:53 INFO systemd watchdog enabled interval=15s5924lb # [ 95.134672] systemd[1]: Started niks3 server.5925lb # [ 95.136813] niks3-server[1092]: 2026/09/22 11:33:53 INFO Starting HTTP server address=0.0.0.0:57515926lb: waiting for unit niks3.service5927lb: (finished: waiting for unit niks3.service, in 0.02 seconds)5928lb: must succeed: curl -sf localhost:9901/clusters5929lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5930worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5931worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5932lb: must succeed: curl -sf localhost:9901/clusters5933lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5934lb: must succeed: curl -sf localhost:9901/clusters5935lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5936worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5937worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.05 seconds)5938client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&25939worker2 # [ 95.481440] nix-grpc-daemon[1380]: ts=2026-09-22T11:33:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:530545940client # don't know how to build these paths:5941client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5942worker1 # [ 95.501485] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:519385943worker1 # [ 95.512752] nix-daemon[844]: accepted connection from pid 1198, user nix-grpc-daemon (trusted)5944worker1 # [ 95.529264] nix-grpc-daemon[1198]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 12 ms (attempt 1/5)5945worker1 # [ 95.548329] nix-grpc-daemon[1198]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 47 ms (attempt 2/5)5946worker1 # [ 95.603305] nix-grpc-daemon[1198]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 356 ms (attempt 3/5)5947worker1 # [ 95.961554] nix-grpc-daemon[1198]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 665 ms (attempt 4/5)5948lb # [ 96.037460] niks3-server[1092]: 2026/09/22 11:33:54 INFO lead: acquired remote=192.168.1.3:567865949worker1 # [ 96.629967] nix-grpc-daemon[1198]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5950worker1 # [ 96.633924] nix-grpc-daemon[1198]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5951worker1 # [ 96.668574] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:55Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=05952worker1 # [ 96.670901] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:55Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.3:50051 clients=15953worker1 # [ 96.712942] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519385954worker1 # [ 96.722150] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:519385955client # copying 1 paths to 'lb:50051'...5956client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...5957worker1 # [ 96.729577] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:519385958worker1 # [ 96.735962] nix-daemon[844]: accepted connection from pid 1198, user nix-grpc-daemon (trusted)5959worker1 # [ 96.746221] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:55Z level=info event=niks3_push_started pid=12705960worker1 # [ 96.844791] nix-grpc-daemon[1270]: time=2026-09-22T11:33:55.208Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""5961worker1 # [ 96.936858] nix-daemon[844]: accepted connection from pid 1283, user nix-grpc-daemon (trusted)5962worker1 # [ 96.952080] nix-daemon[844]: reaped child process 1286, status = succeeded5963worker1 # [ 97.041540] nix-daemon[844]: accepted connection from pid 1288, user nix-grpc-daemon (trusted)5964worker1 # [ 97.057079] nix-daemon[844]: reaped child process 1291, status = succeeded5965lb # [ 97.026813] niks3-server[1092]: 2026/09/22 11:33:55 INFO Received uploads request method=POST path=/api/pending_closures5966worker1 # [ 97.077079] nix-grpc-daemon[1270]: time=2026-09-22T11:33:55.440Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5967worker1 # [ 97.078481] nix-grpc-daemon[1270]: time=2026-09-22T11:33:55.440Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"5968lb # [ 97.040449] envoy[918]: conn peer=[2001:db8:1::1]:53244 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=519 ms=95969lb # [ 97.044941] envoy[918]: conn peer=[2001:db8:1::3]:43530 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=13892 tx=2867 ms=732985970lb # [ 97.050452] envoy[918]: conn peer=[2001:db8:1::1]:53254 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1002 tx=282 ms=14445971lb # [ 97.079892] niks3-server[1092]: 2026/09/22 11:33:55 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst5972lb # [ 97.086621] niks3-server[1092]: 2026/09/22 11:33:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5973lb # [ 97.101308] niks3-server[1092]: 2026/09/22 11:33:55 INFO Signed narinfos id=37 count=15974worker1 # [ 97.138648] nix-grpc-daemon[1270]: time=2026-09-22T11:33:55.502Z level=INFO msg="Uploading 1 narinfos"5975lb # [ 97.108056] niks3-server[1092]: 2026/09/22 11:33:55 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5976lb # [ 97.117403] niks3-server[1092]: 2026/09/22 11:33:55 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls5977lb # [ 97.125735] niks3-server[1092]: 2026/09/22 11:33:55 INFO Completed upload id=375978worker1 # [ 97.164308] nix-grpc-daemon[1270]: time=2026-09-22T11:33:55.528Z level=INFO msg="Upload complete. (315ms)"5979lb # [ 97.129950] niks3-server[1092]: 2026/09/22 11:33:55 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo5980worker1 # [ 97.166936] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51938 duration_s=0 paths=1 nar_bytes_in=5365981worker1 # [ 97.171654] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519385982worker1 # [ 97.178956] nix-daemon[844]: accepted connection from pid 1198, user nix-grpc-daemon (trusted)5983worker1 # [ 97.181523] nix-daemon[844]: reaped child process 1268, status = succeeded5984worker1 # [ 97.187099] nix-daemon[844]: accepted connection from pid 1198, user nix-grpc-daemon (trusted)5985client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...5986client # farm-a> node-a: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5987client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5988client # farm-a> LOG-farm-a-after-niks3-restart5989worker1 # [ 97.803577] nix-daemon[844]: reaped child process 1295, status = succeeded5990worker1 # [ 97.884643] nix-daemon[844]: accepted connection from pid 1312, user nix-grpc-daemon (trusted)5991worker1 # [ 97.899969] nix-daemon[844]: reaped child process 1315, status = succeeded5992lb # [ 97.869287] niks3-server[1092]: 2026/09/22 11:33:56 INFO Received uploads request method=POST path=/api/pending_closures5993worker1 # [ 97.918056] nix-grpc-daemon[1270]: time=2026-09-22T11:33:56.282Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5994worker1 # [ 97.920051] nix-grpc-daemon[1270]: time=2026-09-22T11:33:56.284Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"5995lb # [ 97.908473] niks3-server[1092]: 2026/09/22 11:33:56 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5996lb # [ 97.922697] niks3-server[1092]: 2026/09/22 11:33:56 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5997lb # [ 97.935511] niks3-server[1092]: 2026/09/22 11:33:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5998lb # [ 97.941447] niks3-server[1092]: 2026/09/22 11:33:56 INFO Signed narinfos id=38 count=15999worker1 # [ 97.978512] nix-grpc-daemon[1270]: time=2026-09-22T11:33:56.342Z level=INFO msg="Uploading 1 narinfos"6000lb # [ 97.947652] niks3-server[1092]: 2026/09/22 11:33:56 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls6001lb # [ 97.954241] niks3-server[1092]: 2026/09/22 11:33:56 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6002lb # [ 97.957142] niks3-server[1092]: 2026/09/22 11:33:56 INFO Completed upload id=386003worker1 # [ 97.995507] nix-grpc-daemon[1270]: time=2026-09-22T11:33:56.359Z level=INFO msg="Upload complete. (201ms)"6004lb # [ 97.960284] niks3-server[1092]: 2026/09/22 11:33:56 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo6005worker1 # [ 98.001759] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51938 duration_s=0 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=16006worker1 # [ 98.016896] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:56Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=06007worker1 # [ 98.022092] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:56Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.3:50051 clients=16008worker1 # [ 98.031229] nix-daemon[844]: reaped child process 1293, status = succeeded6009worker1 # [ 98.060797] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519386010worker1 # [ 98.073828] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:519386011worker1 # [ 98.088393] nix-grpc-daemon[1270]: time=2026-09-22T11:33:56.452Z level=INFO msg="All 1 paths already cached"6012client # copying 1 paths to 'lb:50051'...6013client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...6014worker1 # [ 98.098191] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:519386015worker1 # [ 98.108175] nix-daemon[844]: accepted connection from pid 1198, user nix-grpc-daemon (trusted)6016worker1 # [ 98.208100] nix-daemon[844]: accepted connection from pid 1321, user nix-grpc-daemon (trusted)6017worker1 # [ 98.222933] nix-daemon[844]: reaped child process 1324, status = succeeded6018worker1 # [ 98.310969] nix-daemon[844]: accepted connection from pid 1326, user nix-grpc-daemon (trusted)6019worker1 # [ 98.326045] nix-daemon[844]: reaped child process 1329, status = succeeded6020lb # [ 98.296294] niks3-server[1092]: 2026/09/22 11:33:56 INFO Received uploads request method=POST path=/api/pending_closures6021worker1 # [ 98.343881] nix-grpc-daemon[1270]: time=2026-09-22T11:33:56.707Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6022worker1 # [ 98.345283] nix-grpc-daemon[1270]: time=2026-09-22T11:33:56.708Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"6023lb # [ 98.324572] niks3-server[1092]: 2026/09/22 11:33:56 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst6024lb # [ 98.337718] niks3-server[1092]: 2026/09/22 11:33:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6025lb # [ 98.345956] niks3-server[1092]: 2026/09/22 11:33:56 INFO Signed narinfos id=39 count=16026worker1 # [ 98.385905] nix-grpc-daemon[1270]: time=2026-09-22T11:33:56.750Z level=INFO msg="Uploading 1 narinfos"6027lb # [ 98.351333] niks3-server[1092]: 2026/09/22 11:33:56 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls6028lb # [ 98.363896] niks3-server[1092]: 2026/09/22 11:33:56 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6029lb # [ 98.372664] niks3-server[1092]: 2026/09/22 11:33:56 INFO Completed upload id=396030worker1 # [ 98.409761] nix-grpc-daemon[1270]: time=2026-09-22T11:33:56.774Z level=INFO msg="Upload complete. (291ms)"6031lb # [ 98.375579] niks3-server[1092]: 2026/09/22 11:33:56 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo6032worker1 # [ 98.414532] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51938 duration_s=0 paths=1 nar_bytes_in=6566033worker1 # [ 98.420546] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519386034worker1 # [ 98.432111] nix-daemon[844]: accepted connection from pid 1198, user nix-grpc-daemon (trusted)6035worker1 # [ 98.437934] nix-daemon[844]: reaped child process 1319, status = succeeded6036worker1 # [ 98.442864] nix-daemon[844]: accepted connection from pid 1198, user nix-grpc-daemon (trusted)6037client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...6038client # farm-b> node-a: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6039client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6040client # farm-b> LOG-farm-b-after-niks3-restart6041worker1 # [ 99.878896] nix-daemon[844]: reaped child process 1333, status = succeeded6042worker1 # [ 99.970608] nix-daemon[844]: accepted connection from pid 1349, user nix-grpc-daemon (trusted)6043worker1 # [ 99.986036] nix-daemon[844]: reaped child process 1352, status = succeeded6044lb # [ 99.956406] niks3-server[1092]: 2026/09/22 11:33:58 INFO Received uploads request method=POST path=/api/pending_closures6045worker1 # [ 100.003065] nix-grpc-daemon[1270]: time=2026-09-22T11:33:58.367Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6046worker1 # [ 100.004994] nix-grpc-daemon[1270]: time=2026-09-22T11:33:58.369Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"6047lb # [ 100.001556] niks3-server[1092]: 2026/09/22 11:33:58 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6048lb # [ 100.014827] niks3-server[1092]: 2026/09/22 11:33:58 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst6049lb # [ 100.023513] niks3-server[1092]: 2026/09/22 11:33:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6050lb # [ 100.027299] niks3-server[1092]: 2026/09/22 11:33:58 INFO Signed narinfos id=40 count=16051worker1 # [ 100.067196] nix-grpc-daemon[1270]: time=2026-09-22T11:33:58.431Z level=INFO msg="Uploading 1 narinfos"6052lb # [ 100.033733] niks3-server[1092]: 2026/09/22 11:33:58 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls6053lb # [ 100.038237] niks3-server[1092]: 2026/09/22 11:33:58 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6054lb # [ 100.043281] niks3-server[1092]: 2026/09/22 11:33:58 INFO Completed upload id=406055worker1 # [ 100.080450] nix-grpc-daemon[1270]: time=2026-09-22T11:33:58.444Z level=INFO msg="Upload complete. (227ms)"6056lb # [ 100.045773] niks3-server[1092]: 2026/09/22 11:33:58 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo6057worker1 # [ 100.086997] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51938 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=16058worker1 # [ 100.102556] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:58Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=06059worker1 # [ 100.105865] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:58Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.3:50051 clients=16060worker1 # [ 100.117560] nix-daemon[844]: reaped child process 1331, status = succeeded6061worker1 # [ 100.145730] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519386062worker1 # [ 100.158760] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:519386063worker1 # [ 100.172398] nix-grpc-daemon[1270]: time=2026-09-22T11:33:58.536Z level=INFO msg="All 2 paths already cached"6064client # copying 1 paths to 'lb:50051'...6065client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...6066worker1 # [ 100.180559] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:519386067worker1 # [ 100.188720] nix-daemon[844]: accepted connection from pid 1198, user nix-grpc-daemon (trusted)6068worker1 # [ 100.292850] nix-daemon[844]: accepted connection from pid 1358, user nix-grpc-daemon (trusted)6069worker1 # [ 100.308388] nix-daemon[844]: reaped child process 1361, status = succeeded6070worker1 # [ 100.395199] nix-daemon[844]: accepted connection from pid 1363, user nix-grpc-daemon (trusted)6071worker1 # [ 100.409977] nix-daemon[844]: reaped child process 1366, status = succeeded6072lb # [ 100.379945] niks3-server[1092]: 2026/09/22 11:33:58 INFO Received uploads request method=POST path=/api/pending_closures6073worker1 # [ 100.426805] nix-grpc-daemon[1270]: time=2026-09-22T11:33:58.791Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6074worker1 # [ 100.428844] nix-grpc-daemon[1270]: time=2026-09-22T11:33:58.792Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"6075lb # [ 100.411661] niks3-server[1092]: 2026/09/22 11:33:58 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst6076lb # [ 100.419193] niks3-server[1092]: 2026/09/22 11:33:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6077worker1 # [ 100.464439] nix-grpc-daemon[1270]: time=2026-09-22T11:33:58.828Z level=INFO msg="Uploading 1 narinfos"6078lb # [ 100.427323] niks3-server[1092]: 2026/09/22 11:33:58 INFO Signed narinfos id=41 count=16079lb # [ 100.433894] niks3-server[1092]: 2026/09/22 11:33:58 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls6080lb # [ 100.444775] niks3-server[1092]: 2026/09/22 11:33:58 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6081lb # [ 100.449191] niks3-server[1092]: 2026/09/22 11:33:58 INFO Completed upload id=416082worker1 # [ 100.486124] nix-grpc-daemon[1270]: time=2026-09-22T11:33:58.850Z level=INFO msg="Upload complete. (284ms)"6083lb # [ 100.451404] niks3-server[1092]: 2026/09/22 11:33:58 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo6084worker1 # [ 100.489416] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51938 duration_s=0 paths=1 nar_bytes_in=7846085worker1 # [ 100.494346] nix-grpc-daemon[1198]: ts=2026-09-22T11:33:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519386086worker1 # [ 100.503581] nix-daemon[844]: accepted connection from pid 1198, user nix-grpc-daemon (trusted)6087worker1 # [ 100.510209] nix-daemon[844]: reaped child process 1356, status = succeeded6088client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...6089worker1 # [ 100.516342] nix-daemon[844]: accepted connection from pid 1198, user nix-grpc-daemon (trusted)6090client # farm-top> node-a: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6091client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6092client # farm-top> LOG-farm-top-after-niks3-restart6093worker1 # [ 101.532953] nix-daemon[844]: reaped child process 1370, status = succeeded6094worker1 # [ 101.614300] nix-daemon[844]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)6095worker1 # [ 101.629769] nix-daemon[844]: reaped child process 1389, status = succeeded6096lb # [ 101.601257] niks3-server[1092]: 2026/09/22 11:33:59 INFO Received uploads request method=POST path=/api/pending_closures6097worker1 # [ 101.647442] nix-grpc-daemon[1270]: time=2026-09-22T11:34:00.011Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6098worker1 # [ 101.648992] nix-grpc-daemon[1270]: time=2026-09-22T11:34:00.013Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"6099lb # [ 101.638265] niks3-server[1092]: 2026/09/22 11:33:59 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6100lb # [ 101.646853] niks3-server[1092]: 2026/09/22 11:33:59 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst6101lb # [ 101.661568] niks3-server[1092]: 2026/09/22 11:34:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6102lb # [ 101.668452] niks3-server[1092]: 2026/09/22 11:34:00 INFO Signed narinfos id=42 count=16103worker1 # [ 101.708173] nix-grpc-daemon[1270]: time=2026-09-22T11:34:00.072Z level=INFO msg="Uploading 1 narinfos"6104lb # [ 101.671424] niks3-server[1092]: 2026/09/22 11:34:00 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls6105lb # [ 101.682206] niks3-server[1092]: 2026/09/22 11:34:00 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6106lb # [ 101.685880] niks3-server[1092]: 2026/09/22 11:34:00 INFO Completed upload id=426107worker1 # [ 101.722971] nix-grpc-daemon[1270]: time=2026-09-22T11:34:00.087Z level=INFO msg="Upload complete. (200ms)"6108lb # [ 101.687717] niks3-server[1092]: 2026/09/22 11:34:00 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo6109worker1 # [ 101.731942] nix-grpc-daemon[1198]: ts=2026-09-22T11:34:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51938 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=16110worker1 # [ 101.746932] nix-daemon[844]: reaped child process 1368, status = succeeded6111client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&2, in 6.40 seconds)6112client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath6113client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath, in 0.12 seconds)6114(finished: subtest: a niks3 restart does not cost the leader its role, in 6.91 seconds)6115subtest: scheduler down: the other node takes the lock and keeps it6116lb: must succeed: curl -sf localhost:9901/clusters6117lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6118lb: must succeed: curl -sf localhost:9901/clusters6119lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6120lb: must succeed: curl -sf localhost:9901/clusters6121lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6122lb: must succeed: curl -sf localhost:9901/clusters6123lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6124worker1 # [ 102.024622] systemd[1]: Stopping Nix worker-protocol over gRPC...6125worker1 # [ 102.852071] nix-grpc-daemon[1198]: ts=2026-09-22T11:34:01Z level=info event=signal_exit6126lb # [ 103.041872] niks3-server[1092]: 2026/09/22 11:34:01 INFO lead: released remote=192.168.1.3:567866127worker1 # [ 104.077208] nix-grpc-daemon[1198]: ts=2026-09-22T11:34:02Z level=info event=scheduler_restarting6128worker2 # [ 104.074469] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:03Z level=info event=scheduler_restarting addr=lb:500516129worker1 # [ 104.080626] nix-grpc-daemon[1198]: ts=2026-09-22T11:34:02Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:519266130worker1 # [ 104.083781] nix-grpc-daemon[1198]: ts=2026-09-22T11:34:02Z level=info event=worker_gone addr=192.168.1.4:500516131worker1 # [ 104.085323] nix-grpc-daemon[1198]: ts=2026-09-22T11:34:02Z level=info event=scheduler_restarting addr=lb:500516132worker1 # [ 104.086644] nix-grpc-daemon[1198]: ts=2026-09-22T11:34:02Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:519266133worker1 # [ 104.088143] nix-grpc-daemon[1198]: ts=2026-09-22T11:34:02Z level=info event=worker_gone addr=192.168.1.3:500516134worker1 # [ 104.101437] nix-grpc-daemon[1198]: E0000 00:00:1790076842.465812 1243 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 10, started true, shutdown true6135worker1 # [ 104.107220] nix-grpc-daemon[1198]: E0000 00:00:1790076842.471166 1243 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6136worker1 # [ 104.119371] nix-daemon[844]: reaped child process 1258, status = succeeded6137worker2 # [ 104.274513] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:03Z level=info event=scheduler_connected addr=lb:500516138worker2 # [ 104.278779] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6139worker1 # [ 104.294922] nix-grpc-daemon[1198]: ts=2026-09-22T11:34:02Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6140worker2 # [ 104.480069] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:03Z level=info event=scheduler_connected addr=lb:500516141worker2 # [ 104.484137] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6142worker2 # [ 104.685313] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:03Z level=info event=scheduler_connected addr=lb:500516143worker2 # [ 104.689352] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6144worker2 # [ 104.890445] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:04Z level=info event=scheduler_connected addr=lb:500516145worker2 # [ 104.894515] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6146worker2 # [ 105.096499] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:04Z level=info event=scheduler_connected addr=lb:500516147worker2 # [ 105.100242] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6148worker1 # [ 105.291471] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6149worker1 # [ 105.293335] systemd[1]: Stopped Nix worker-protocol over gRPC.6150worker1 # [ 105.295333] systemd[1]: nix-grpc-daemon.service: Consumed 1.033s CPU time over 11.557s wall clock time, 38.8M memory peak, 35.2K incoming IP traffic, 58.4K outgoing IP traffic.6151worker1 # [ 105.299876] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.6152worker1 # [ 105.302181] systemd[1]: Closed Nix worker-protocol over gRPC.6153worker1 # [ 105.305835] systemd[1]: nix-grpc-daemon.socket: Consumed 563us CPU time over 11.578s wall clock time, 256K memory peak, 33.4K incoming IP traffic, 24.3K outgoing IP traffic.6154worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over6155worker2 # [ 105.301183] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:04Z level=info event=scheduler_connected addr=lb:500516156worker2 # [ 105.305168] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6157worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.05 seconds)6158worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6159worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)6160worker2 # [ 105.506304] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:04Z level=info event=scheduler_connected addr=lb:500516161worker2 # [ 105.510315] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6162worker2 # [ 105.710927] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:04Z level=info event=scheduler_connected addr=lb:500516163worker2 # [ 105.712714] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6164worker2 # [ 105.914161] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=info event=scheduler_connected addr=lb:500516165worker2 # [ 105.918192] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6166worker2 # [ 106.070483] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=info event=scheduler_take_over6167lb # [ 106.038973] niks3-server[1092]: 2026/09/22 11:34:04 INFO lead: acquired remote=192.168.1.4:433846168worker2 # [ 106.120372] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=info event=scheduler_connected addr=lb:500516169worker2 # [ 106.123466] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6170worker2 # [ 106.320858] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=info event=scheduler_connected addr=lb:500516171worker2 # [ 106.335132] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=info event=oidc_jwks provider=mock keys=16172worker2 # [ 106.338858] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:397486173worker2 # [ 106.342678] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06174worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6175worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6176client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&26177worker2 # [ 106.575275] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:530546178client # don't know how to build these paths:6179client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6180worker2 # [ 106.584127] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:530546181worker2 # [ 106.609919] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=06182worker2 # [ 106.615516] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.4:50051 clients=16183worker2 # [ 106.654659] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:530546184worker2 # [ 106.669084] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:530546185client # copying 1 paths to 'lb:50051'...6186client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...6187worker2 # [ 106.682057] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:530546188worker2 # [ 106.693208] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6189worker2 # [ 106.803115] nix-daemon[867]: accepted connection from pid 1479, user nix-grpc-daemon (trusted)6190worker2 # [ 106.817831] nix-daemon[867]: reaped child process 1482, status = succeeded6191worker2 # [ 106.906522] nix-daemon[867]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6192worker2 # [ 106.920700] nix-daemon[867]: reaped child process 1487, status = succeeded6193lb # [ 106.896039] niks3-server[1092]: 2026/09/22 11:34:05 INFO Received uploads request method=POST path=/api/pending_closures6194worker2 # [ 106.933068] nix-grpc-daemon[1440]: time=2026-09-22T11:34:06.098Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6195worker2 # [ 106.934432] nix-grpc-daemon[1440]: time=2026-09-22T11:34:06.098Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"6196lb # [ 106.909298] niks3-server[1092]: 2026/09/22 11:34:05 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst6197lb # [ 106.912829] niks3-server[1092]: 2026/09/22 11:34:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6198worker2 # [ 106.944473] nix-grpc-daemon[1440]: time=2026-09-22T11:34:06.110Z level=INFO msg="Uploading 1 narinfos"6199lb # [ 106.914452] niks3-server[1092]: 2026/09/22 11:34:05 INFO Signed narinfos id=43 count=16200lb # [ 106.922558] niks3-server[1092]: 2026/09/22 11:34:05 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6201worker2 # [ 106.958575] nix-grpc-daemon[1440]: time=2026-09-22T11:34:06.124Z level=INFO msg="Upload complete. (254ms)"6202lb # [ 106.928245] niks3-server[1092]: 2026/09/22 11:34:05 INFO Completed upload id=436203worker2 # [ 106.960841] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53054 duration_s=0 paths=1 nar_bytes_in=5206204lb # [ 106.930778] niks3-server[1092]: 2026/09/22 11:34:05 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls6205worker2 # [ 106.965975] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:530546206worker2 # [ 106.972873] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6207worker2 # [ 106.975200] nix-daemon[867]: reaped child process 1477, status = succeeded6208client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...6209client # farm-a> worker2: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6210worker2 # [ 106.980659] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6211lb # [ 107.040790] envoy[918]: conn peer=[2001:db8:1::1]:44050 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9666 tx=4426 ms=62596212lb # [ 107.045302] envoy[918]: conn peer=[2001:db8:1::3]:47754 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1328 tx=513 ms=103906213client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6214client # farm-a> LOG-farm-a-on-standby6215worker2 # [ 108.197249] nix-daemon[867]: reaped child process 1491, status = succeeded6216worker2 # [ 108.277336] nix-daemon[867]: accepted connection from pid 1507, user nix-grpc-daemon (trusted)6217worker2 # [ 108.290946] nix-daemon[867]: reaped child process 1510, status = succeeded6218lb # [ 108.269584] niks3-server[1092]: 2026/09/22 11:34:06 INFO Received uploads request method=POST path=/api/pending_closures6219worker2 # [ 108.311231] nix-grpc-daemon[1440]: time=2026-09-22T11:34:07.477Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6220worker2 # [ 108.312862] nix-grpc-daemon[1440]: time=2026-09-22T11:34:07.479Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"6221lb # [ 108.313270] niks3-server[1092]: 2026/09/22 11:34:06 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6222lb # [ 108.327614] niks3-server[1092]: 2026/09/22 11:34:06 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst6223lb # [ 108.332636] niks3-server[1092]: 2026/09/22 11:34:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6224lb # [ 108.338102] niks3-server[1092]: 2026/09/22 11:34:06 INFO Signed narinfos id=44 count=16225worker2 # [ 108.369360] nix-grpc-daemon[1440]: time=2026-09-22T11:34:07.535Z level=INFO msg="Uploading 1 narinfos"6226lb # [ 108.347056] niks3-server[1092]: 2026/09/22 11:34:06 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls6227lb # [ 108.350896] niks3-server[1092]: 2026/09/22 11:34:06 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6228lb # [ 108.356221] niks3-server[1092]: 2026/09/22 11:34:06 INFO Completed upload id=446229worker2 # [ 108.388099] nix-grpc-daemon[1440]: time=2026-09-22T11:34:07.554Z level=INFO msg="Upload complete. (199ms)"6230lb # [ 108.358972] niks3-server[1092]: 2026/09/22 11:34:06 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo6231worker2 # [ 108.395650] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53054 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=16232worker2 # [ 108.407944] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:07Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=06233worker2 # [ 108.411984] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:07Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.4:50051 clients=16234worker2 # [ 108.422416] nix-daemon[867]: reaped child process 1489, status = succeeded6235worker2 # [ 108.452405] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:530546236worker2 # [ 108.464528] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:530546237worker2 # [ 108.479093] nix-grpc-daemon[1440]: time=2026-09-22T11:34:07.645Z level=INFO msg="All 1 paths already cached"6238client # copying 1 paths to 'lb:50051'...6239client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...6240worker2 # [ 108.487742] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:530546241worker2 # [ 108.497321] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6242worker2 # [ 108.599759] nix-daemon[867]: accepted connection from pid 1516, user nix-grpc-daemon (trusted)6243worker2 # [ 108.613755] nix-daemon[867]: reaped child process 1519, status = succeeded6244worker2 # [ 108.703386] nix-daemon[867]: accepted connection from pid 1521, user nix-grpc-daemon (trusted)6245worker2 # [ 108.717629] nix-daemon[867]: reaped child process 1524, status = succeeded6246lb # [ 108.692066] niks3-server[1092]: 2026/09/22 11:34:07 INFO Received uploads request method=POST path=/api/pending_closures6247worker2 # [ 108.732024] nix-grpc-daemon[1440]: time=2026-09-22T11:34:07.898Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6248worker2 # [ 108.733513] nix-grpc-daemon[1440]: time=2026-09-22T11:34:07.899Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"6249lb # [ 108.725257] niks3-server[1092]: 2026/09/22 11:34:07 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst6250lb # [ 108.734104] niks3-server[1092]: 2026/09/22 11:34:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6251lb # [ 108.742458] niks3-server[1092]: 2026/09/22 11:34:07 INFO Signed narinfos id=45 count=16252worker2 # [ 108.777937] nix-grpc-daemon[1440]: time=2026-09-22T11:34:07.944Z level=INFO msg="Uploading 1 narinfos"6253lb # [ 108.750807] niks3-server[1092]: 2026/09/22 11:34:07 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls6254lb # [ 108.761977] niks3-server[1092]: 2026/09/22 11:34:07 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6255lb # [ 108.768260] niks3-server[1092]: 2026/09/22 11:34:07 INFO Completed upload id=456256worker2 # [ 108.799825] nix-grpc-daemon[1440]: time=2026-09-22T11:34:07.966Z level=INFO msg="Upload complete. (290ms)"6257lb # [ 108.772657] niks3-server[1092]: 2026/09/22 11:34:07 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo6258worker2 # [ 108.804485] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53054 duration_s=0 paths=1 nar_bytes_in=6406259worker2 # [ 108.810764] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:530546260worker2 # [ 108.820116] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6261worker2 # [ 108.828603] nix-daemon[867]: reaped child process 1514, status = succeeded6262worker2 # [ 108.833738] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6263client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...6264client # farm-b> worker2: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6265client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6266client # farm-b> LOG-farm-b-on-standby6267worker2 # [ 109.878779] nix-daemon[867]: reaped child process 1528, status = succeeded6268worker2 # [ 109.956890] nix-daemon[867]: accepted connection from pid 1544, user nix-grpc-daemon (trusted)6269worker2 # [ 109.969645] nix-daemon[867]: reaped child process 1547, status = succeeded6270lb # [ 109.944704] niks3-server[1092]: 2026/09/22 11:34:08 INFO Received uploads request method=POST path=/api/pending_closures6271worker2 # [ 109.984513] nix-grpc-daemon[1440]: time=2026-09-22T11:34:09.150Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6272worker2 # [ 109.986763] nix-grpc-daemon[1440]: time=2026-09-22T11:34:09.152Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"6273lb # [ 109.981202] niks3-server[1092]: 2026/09/22 11:34:08 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6274lb # [ 109.991182] niks3-server[1092]: 2026/09/22 11:34:08 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst6275lb # [ 109.999845] niks3-server[1092]: 2026/09/22 11:34:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6276lb # [ 110.006206] niks3-server[1092]: 2026/09/22 11:34:08 INFO Signed narinfos id=46 count=16277worker2 # [ 110.040284] nix-grpc-daemon[1440]: time=2026-09-22T11:34:09.206Z level=INFO msg="Uploading 1 narinfos"6278lb # [ 110.016744] niks3-server[1092]: 2026/09/22 11:34:08 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls6279lb # [ 110.021331] niks3-server[1092]: 2026/09/22 11:34:08 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6280lb # [ 110.027984] niks3-server[1092]: 2026/09/22 11:34:08 INFO Completed upload id=466281worker2 # [ 110.059683] nix-grpc-daemon[1440]: time=2026-09-22T11:34:09.225Z level=INFO msg="Upload complete. (218ms)"6282lb # [ 110.030096] niks3-server[1092]: 2026/09/22 11:34:08 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo6283worker2 # [ 110.066640] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53054 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=16284worker2 # [ 110.078524] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:09Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=06285worker2 # [ 110.082182] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:09Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.4:50051 clients=16286worker2 # [ 110.092821] nix-daemon[867]: reaped child process 1526, status = succeeded6287worker2 # [ 110.122738] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:530546288worker2 # [ 110.134683] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:530546289worker2 # [ 110.148520] nix-grpc-daemon[1440]: time=2026-09-22T11:34:09.314Z level=INFO msg="All 2 paths already cached"6290client # copying 1 paths to 'lb:50051'...6291client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...6292worker2 # [ 110.157452] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:530546293worker2 # [ 110.165733] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6294worker2 # [ 110.269245] nix-daemon[867]: accepted connection from pid 1553, user nix-grpc-daemon (trusted)6295worker2 # [ 110.283190] nix-daemon[867]: reaped child process 1556, status = succeeded6296worker2 # [ 110.373350] nix-daemon[867]: accepted connection from pid 1558, user nix-grpc-daemon (trusted)6297worker2 # [ 110.387060] nix-daemon[867]: reaped child process 1561, status = succeeded6298lb # [ 110.362739] niks3-server[1092]: 2026/09/22 11:34:08 INFO Received uploads request method=POST path=/api/pending_closures6299worker2 # [ 110.402243] nix-grpc-daemon[1440]: time=2026-09-22T11:34:09.568Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6300worker2 # [ 110.404119] nix-grpc-daemon[1440]: time=2026-09-22T11:34:09.569Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6301lb # [ 110.392457] niks3-server[1092]: 2026/09/22 11:34:08 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6302lb # [ 110.398621] niks3-server[1092]: 2026/09/22 11:34:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6303lb # [ 110.403504] niks3-server[1092]: 2026/09/22 11:34:08 INFO Signed narinfos id=47 count=16304worker2 # [ 110.438137] nix-grpc-daemon[1440]: time=2026-09-22T11:34:09.604Z level=INFO msg="Uploading 1 narinfos"6305lb # [ 110.411094] niks3-server[1092]: 2026/09/22 11:34:08 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6306lb # [ 110.425363] niks3-server[1092]: 2026/09/22 11:34:08 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6307worker2 # [ 110.462183] nix-grpc-daemon[1440]: time=2026-09-22T11:34:09.628Z level=INFO msg="Upload complete. (283ms)"6308lb # [ 110.431146] niks3-server[1092]: 2026/09/22 11:34:08 INFO Completed upload id=476309worker2 # [ 110.464433] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53054 duration_s=0 paths=1 nar_bytes_in=7686310worker2 # [ 110.467610] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:530546311lb # [ 110.436240] niks3-server[1092]: 2026/09/22 11:34:08 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6312worker2 # [ 110.475476] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6313worker2 # [ 110.477780] nix-daemon[867]: reaped child process 1551, status = succeeded6314client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6315client # farm-top> worker2: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6316worker2 # [ 110.483217] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6317client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6318client # farm-top> LOG-farm-top-on-standby6319worker2 # [ 111.499696] nix-daemon[867]: reaped child process 1565, status = succeeded6320worker2 # [ 111.580471] nix-daemon[867]: accepted connection from pid 1581, user nix-grpc-daemon (trusted)6321worker2 # [ 111.594637] nix-daemon[867]: reaped child process 1584, status = succeeded6322lb # [ 111.570316] niks3-server[1092]: 2026/09/22 11:34:09 INFO Received uploads request method=POST path=/api/pending_closures6323worker2 # [ 111.612414] nix-grpc-daemon[1440]: time=2026-09-22T11:34:10.778Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6324worker2 # [ 111.613899] nix-grpc-daemon[1440]: time=2026-09-22T11:34:10.780Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6325lb # [ 111.612460] niks3-server[1092]: 2026/09/22 11:34:09 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6326lb # [ 111.618685] niks3-server[1092]: 2026/09/22 11:34:09 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6327lb # [ 111.626388] niks3-server[1092]: 2026/09/22 11:34:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6328worker2 # [ 111.663638] nix-grpc-daemon[1440]: time=2026-09-22T11:34:10.829Z level=INFO msg="Uploading 1 narinfos"6329lb # [ 111.633142] niks3-server[1092]: 2026/09/22 11:34:09 INFO Signed narinfos id=48 count=16330lb # [ 111.643214] niks3-server[1092]: 2026/09/22 11:34:09 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6331lb # [ 111.649544] niks3-server[1092]: 2026/09/22 11:34:10 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6332lb # [ 111.656350] niks3-server[1092]: 2026/09/22 11:34:10 INFO Completed upload id=486333worker2 # [ 111.687978] nix-grpc-daemon[1440]: time=2026-09-22T11:34:10.854Z level=INFO msg="Upload complete. (197ms)"6334lb # [ 111.659255] niks3-server[1092]: 2026/09/22 11:34:10 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6335worker2 # [ 111.696385] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53054 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16336worker2 # [ 111.710071] nix-daemon[867]: reaped child process 1563, status = succeeded6337client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&2, in 5.30 seconds)6338client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath6339client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath, in 0.12 seconds)6340worker1 # [ 111.878295] systemd[1]: Listening on Nix worker-protocol over gRPC.6341worker1 # [ 111.880821] systemd[1]: Starting Nix worker-protocol over gRPC...6342worker1 # [ 111.965378] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:10Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6343worker1 # [ 111.967505] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:10Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6344worker1 # [ 111.971190] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:10Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16345worker1 # [ 111.975056] nix-grpc-daemon[1407]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6346worker1 # [ 111.976380] nix-grpc-daemon[1407]: E0000 00:00:1790076850.339409 1407 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6347worker1 # [ 111.982552] nix-grpc-daemon[1407]: E0000 00:00:1790076850.344660 1407 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6348worker1 # [ 111.986915] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:10Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6349worker1 # [ 111.988442] systemd[1]: Started Nix worker-protocol over gRPC.6350worker1 # [ 111.995066] nix-grpc-daemon[1407]: E0000 00:00:1790076850.356637 1449 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6351worker1 # [ 112.007084] nix-grpc-daemon[1407]: E0000 00:00:1790076850.370315 1449 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6352worker1 # [ 112.010167] nix-grpc-daemon[1407]: E0000 00:00:1790076850.370730 1449 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6353lb: must succeed: curl -sf localhost:9901/clusters6354worker1 # [ 112.020675] nix-grpc-daemon[1407]: E0000 00:00:1790076850.385195 1449 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6355worker1 # [ 112.032745] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:10Z level=info event=scheduler_connected addr=lb:500516356worker2 # [ 112.029555] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:11Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:397486357worker2 # [ 112.033315] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:11Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06358lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6359worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6360worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6361lb: must succeed: curl -sf localhost:9901/clusters6362lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6363lb: must succeed: curl -sf localhost:9901/clusters6364lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6365worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6366worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)6367(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 10.30 seconds)6368subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6369client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs6370client # [ 112.226754] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs.6371client # Running as unit: rs.service; invocation ID: e7e05586ac714dcd9e7acfec0c7b90ba6372client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs, in 0.06 seconds)6373client # [ 112.370856] nix[1357]: don't know how to build these paths:6374client # [ 112.371859] nix[1357]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6375worker2 # [ 112.373826] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:530546376worker2 # [ 112.384145] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:11Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06377worker2 # [ 112.389459] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:11Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.4:50051 clients=16378worker2 # [ 112.428595] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:530546379worker2 # [ 112.440914] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:530546380client # [ 112.447182] nix[1357]: copying 1 paths to 'lb:50051'...6381worker2 # [ 112.451920] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:530546382client # [ 112.449100] nix[1357]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6383worker2 # [ 112.463129] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6384worker2 # [ 112.567271] nix-daemon[867]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6385worker2 # [ 112.582357] nix-daemon[867]: reaped child process 1600, status = succeeded6386worker2 # [ 112.668616] nix-daemon[867]: accepted connection from pid 1602, user nix-grpc-daemon (trusted)6387worker2 # [ 112.681995] nix-daemon[867]: reaped child process 1605, status = succeeded6388lb # [ 112.658518] niks3-server[1092]: 2026/09/22 11:34:11 INFO Received uploads request method=POST path=/api/pending_closures6389worker2 # [ 112.698957] nix-grpc-daemon[1440]: time=2026-09-22T11:34:11.864Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6390worker2 # [ 112.700369] nix-grpc-daemon[1440]: time=2026-09-22T11:34:11.864Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6391lb # [ 112.689323] niks3-server[1092]: 2026/09/22 11:34:11 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6392lb # [ 112.702960] niks3-server[1092]: 2026/09/22 11:34:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6393worker2 # [ 112.739617] nix-grpc-daemon[1440]: time=2026-09-22T11:34:11.905Z level=INFO msg="Uploading 1 narinfos"6394lb # [ 112.708151] niks3-server[1092]: 2026/09/22 11:34:11 INFO Signed narinfos id=49 count=16395lb # [ 112.718118] niks3-server[1092]: 2026/09/22 11:34:11 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6396lb # [ 112.729118] niks3-server[1092]: 2026/09/22 11:34:11 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6397lb # [ 112.735284] niks3-server[1092]: 2026/09/22 11:34:11 INFO Completed upload id=496398lb # [ 112.737036] niks3-server[1092]: 2026/09/22 11:34:11 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6399worker2 # [ 112.768197] nix-grpc-daemon[1440]: time=2026-09-22T11:34:11.934Z level=INFO msg="Upload complete. (293ms)"6400worker2 # [ 112.772601] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53054 duration_s=0 paths=1 nar_bytes_in=4166401worker2 # [ 112.777852] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:530546402worker2 # [ 112.789105] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6403worker2 # [ 112.794135] nix-daemon[867]: reaped child process 1595, status = succeeded6404worker2 # [ 112.798705] nix-daemon[867]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6405client # [ 112.794244] nix[1357]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6406lb: must succeed: curl -sf localhost:9901/clusters6407lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6408lb: must succeed: curl -sf localhost:9901/clusters6409lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6410lb: must succeed: curl -sf localhost:9901/clusters6411lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6412lb: must succeed: curl -sf localhost:9901/clusters6413lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6414worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6415worker1 # [ 114.863279] systemd-journald[284]: Received client request to rotate journal, rotating.6416worker1 # [ 114.879978] systemd-journald[284]: Vacuuming done, freed 0B of archived journals from /var/log/journal/3601fef59206499596e6cc623bbc8868.6417worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.06 seconds)6418worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service6419worker2 # [ 114.785331] systemd[1]: Stopping Nix worker-protocol over gRPC...6420worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.03 seconds)6421client: must succeed: sleep 5.06422worker2 # [ 115.275918] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:14Z level=info event=draining6423lb # [ 117.040295] envoy[918]: conn peer=[2001:db8:1::1]:59990 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9603 tx=4395 ms=51326424client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6425worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6426worker2 # [ 120.019512] nix-daemon[867]: reaped child process 1609, status = succeeded6427worker2 # [ 120.098080] nix-daemon[867]: accepted connection from pid 1650, user nix-grpc-daemon (trusted)6428worker2 # [ 120.109827] nix-daemon[867]: reaped child process 1653, status = succeeded6429lb # [ 120.087341] niks3-server[1092]: 2026/09/22 11:34:18 INFO Received uploads request method=POST path=/api/pending_closures6430worker2 # [ 120.127606] nix-grpc-daemon[1440]: time=2026-09-22T11:34:19.293Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6431worker2 # [ 120.129082] nix-grpc-daemon[1440]: time=2026-09-22T11:34:19.295Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6432lb # [ 120.121205] niks3-server[1092]: 2026/09/22 11:34:18 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6433lb # [ 120.138136] niks3-server[1092]: 2026/09/22 11:34:18 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6434lb # [ 120.149511] niks3-server[1092]: 2026/09/22 11:34:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6435worker2 # [ 120.185914] nix-grpc-daemon[1440]: time=2026-09-22T11:34:19.352Z level=INFO msg="Uploading 1 narinfos"6436lb # [ 120.154834] niks3-server[1092]: 2026/09/22 11:34:18 INFO Signed narinfos id=50 count=16437lb # [ 120.159853] niks3-server[1092]: 2026/09/22 11:34:18 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6438lb # [ 120.168602] niks3-server[1092]: 2026/09/22 11:34:18 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6439lb # [ 120.172190] niks3-server[1092]: 2026/09/22 11:34:18 INFO Completed upload id=506440worker2 # [ 120.203787] nix-grpc-daemon[1440]: time=2026-09-22T11:34:19.370Z level=INFO msg="Upload complete. (192ms)"6441lb # [ 120.175054] niks3-server[1092]: 2026/09/22 11:34:18 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6442worker2 # [ 120.211426] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53054 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16443worker2 # [ 120.224541] nix-daemon[867]: reaped child process 1607, status = succeeded6444client # [ 120.234954] systemd[1]: rs.service: Deactivated successfully.6445client # [ 120.238687] systemd[1]: rs.service: Consumed 131ms CPU time over 8.011s wall clock time, 43.4M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.6446worker2 # [ 120.277496] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:19Z level=info event=signal_exit6447lb # [ 121.040226] niks3-server[1092]: 2026/09/22 11:34:19 INFO lead: released remote=192.168.1.4:433846448lb # [ 121.957986] niks3-server[1092]: 2026/09/22 11:34:20 INFO lead: acquired remote=192.168.1.3:362906449worker1 # [ 121.995137] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:20Z level=info event=scheduler_take_over6450worker1 # [ 122.077851] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:20Z level=info event=scheduler_restarting addr=lb:500516451worker2 # [ 122.074732] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:21Z level=info event=scheduler_restarting6452worker2 # [ 122.079967] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:21Z level=info event=scheduler_restarting addr=lb:500516453worker2 # [ 122.083806] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:397486454worker2 # [ 122.087798] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:21Z level=info event=worker_gone addr=192.168.1.3:500516455worker2 # [ 122.090987] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:397486456worker2 # [ 122.094493] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:21Z level=info event=worker_gone addr=192.168.1.4:500516457worker2 # [ 122.110451] nix-daemon[867]: reaped child process 1432, status = succeeded6458worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.13 seconds)6459client: waiting for success: ! systemctl is-active rs6460client: (finished: waiting for success: ! systemctl is-active rs, in 0.04 seconds)6461client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6462client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)6463client: must fail: journalctl -u rs | grep -q 'reconnecting'6464client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.01 seconds)6465lb: must succeed: curl -sf localhost:9901/clusters6466lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6467worker1 # [ 122.279305] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:20Z level=info event=scheduler_connected addr=lb:500516468worker1 # [ 122.283369] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6469worker2 # [ 122.286849] nix-grpc-daemon[1380]: ts=2026-09-22T11:34:21Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6470worker2 # [ 122.344911] nix-grpc-daemon[1380]: E0000 00:00:1790076861.510633 1425 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true6471worker2 # [ 122.349947] nix-grpc-daemon[1380]: E0000 00:00:1790076861.510932 1425 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6472worker1 # [ 122.484077] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:20Z level=info event=scheduler_connected addr=lb:500516473worker1 # [ 122.491173] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:20Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:605666474worker1 # [ 122.494879] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:20Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06475lb: must succeed: curl -sf localhost:9901/clusters6476lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6477worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6478worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6479worker2 # [ 124.135964] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6480worker2 # [ 124.139091] systemd[1]: Stopped Nix worker-protocol over gRPC.6481worker2 # [ 124.142448] systemd[1]: nix-grpc-daemon.service: Consumed 1.482s CPU time over 34.967s wall clock time, 39.5M memory peak, 60.3K incoming IP traffic, 145.2K outgoing IP traffic.6482worker2 # [ 124.157151] systemd[1]: Starting scheduler-token.service...6483worker2 # [ 124.202456] systemd[1]: scheduler-token.service: Deactivated successfully.6484worker2 # [ 124.204238] systemd[1]: Finished scheduler-token.service.6485worker2 # [ 124.206100] systemd[1]: scheduler-token.service: Consumed 24ms CPU time over 47ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6486worker2 # [ 124.210086] systemd[1]: Starting Nix worker-protocol over gRPC...6487worker2 # [ 124.286785] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:23Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6488worker2 # [ 124.288888] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:23Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6489worker2 # [ 124.292629] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:23Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16490worker2 # [ 124.297063] nix-grpc-daemon[1663]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6491worker2 # [ 124.298413] nix-grpc-daemon[1663]: E0000 00:00:1790076863.462652 1663 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6492worker2 # [ 124.303764] nix-grpc-daemon[1663]: E0000 00:00:1790076863.467843 1663 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6493worker2 # [ 124.306300] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:23Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6494worker2 # [ 124.307831] systemd[1]: Started Nix worker-protocol over gRPC.6495worker2 # [ 124.312063] nix-grpc-daemon[1663]: E0000 00:00:1790076863.478102 1705 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6496worker2 # [ 124.323984] nix-grpc-daemon[1663]: E0000 00:00:1790076863.490266 1705 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6497worker2 # [ 124.327205] nix-grpc-daemon[1663]: E0000 00:00:1790076863.490488 1705 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6498worker2 # [ 124.338247] nix-grpc-daemon[1663]: E0000 00:00:1790076863.504489 1705 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6499worker2 # [ 124.347412] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:23Z level=info event=scheduler_connected addr=lb:500516500lb: must succeed: curl -sf localhost:9901/clusters6501worker1 # [ 124.363109] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:22Z level=info event=oidc_jwks provider=mock keys=16502worker1 # [ 124.366204] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:22Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:605666503worker1 # [ 124.370149] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:22Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06504lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6505worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6506worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)6507(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.24 seconds)6508subtest: scheduler restart mid-build: build finishes, no double build6509client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid6510client # [ 124.466564] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid.6511client # Running as unit: mid.service; invocation ID: 59f3f172c8dc44c2a3100061f9dfa07f6512client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid, in 0.06 seconds)6513worker1 # [ 124.619253] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:450266514worker1 # [ 124.630687] nix-daemon[844]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6515client # [ 124.633076] nix[1393]: don't know how to build these paths:6516client # [ 124.634213] nix[1393]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6517worker1 # [ 124.648920] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:23Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06518worker1 # [ 124.653526] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:23Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16519worker1 # [ 124.697691] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450266520worker1 # [ 124.708257] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 22 ms (attempt 1/5)6521worker1 # [ 124.737177] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 129 ms (attempt 2/5)6522worker1 # [ 124.868360] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 317 ms (attempt 3/5)6523worker1 # [ 125.187221] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 659 ms (attempt 4/5)6524worker1 # [ 125.855317] nix-grpc-daemon[1407]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6525worker1 # [ 125.858502] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6526worker1 # [ 125.875332] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:450266527client # [ 125.876309] nix[1393]: copying 1 paths to 'lb:50051'...6528client # [ 125.879737] nix[1393]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6529worker1 # [ 125.889784] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:450266530worker1 # [ 125.900296] nix-daemon[844]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6531worker1 # [ 125.919774] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:24Z level=info event=niks3_push_started pid=15336532worker1 # [ 126.017724] nix-grpc-daemon[1533]: time=2026-09-22T11:34:24.381Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6533worker1 # [ 126.101890] nix-daemon[844]: accepted connection from pid 1546, user nix-grpc-daemon (trusted)6534worker1 # [ 126.113583] nix-daemon[844]: reaped child process 1549, status = succeeded6535worker1 # [ 126.191295] nix-daemon[844]: accepted connection from pid 1551, user nix-grpc-daemon (trusted)6536worker1 # [ 126.204305] nix-daemon[844]: reaped child process 1554, status = succeeded6537lb # [ 126.173837] niks3-server[1092]: 2026/09/22 11:34:24 INFO Received uploads request method=POST path=/api/pending_closures6538worker1 # [ 126.217718] nix-grpc-daemon[1533]: time=2026-09-22T11:34:24.582Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6539worker1 # [ 126.220771] nix-grpc-daemon[1533]: time=2026-09-22T11:34:24.583Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6540lb # [ 126.212642] niks3-server[1092]: 2026/09/22 11:34:24 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6541lb # [ 126.224659] niks3-server[1092]: 2026/09/22 11:34:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6542lb # [ 126.232492] niks3-server[1092]: 2026/09/22 11:34:24 INFO Signed narinfos id=51 count=16543worker1 # [ 126.272601] nix-grpc-daemon[1533]: time=2026-09-22T11:34:24.636Z level=INFO msg="Uploading 1 narinfos"6544lb # [ 126.243834] niks3-server[1092]: 2026/09/22 11:34:24 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6545lb # [ 126.252885] niks3-server[1092]: 2026/09/22 11:34:24 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6546lb # [ 126.256535] niks3-server[1092]: 2026/09/22 11:34:24 INFO Completed upload id=516547worker1 # [ 126.293615] nix-grpc-daemon[1533]: time=2026-09-22T11:34:24.658Z level=INFO msg="Upload complete. (272ms)"6548lb # [ 126.259950] niks3-server[1092]: 2026/09/22 11:34:24 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6549worker1 # [ 126.297543] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45026 duration_s=0 paths=1 nar_bytes_in=4246550worker1 # [ 126.303953] nix-grpc-daemon[1407]: ts=2026-09-22T11:34:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450266551worker1 # [ 126.313124] nix-daemon[844]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6552worker1 # [ 126.323130] nix-daemon[844]: reaped child process 1531, status = succeeded6553worker1 # [ 126.326734] nix-daemon[844]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6554client # [ 126.316866] nix[1393]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6555lb # [ 127.041281] envoy[918]: conn peer=[2001:db8:1::1]:60004 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2960 tx=1517 ms=78466556lb # [ 127.042817] envoy[918]: conn peer=[2001:db8:1::4]:33332 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=26829 tx=1588 ms=329646557lb: must succeed: curl -sf localhost:9901/clusters6558lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6559lb: must succeed: curl -sf localhost:9901/clusters6560lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6561worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6562worker1 # [ 128.020262] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1407 (nix-grpc-daemon) on client request.6563worker1 # [ 128.022320] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6564lb # [ 127.993121] niks3-server[1092]: 2026/09/22 11:34:26 INFO lead: released remote=192.168.1.3:362906565worker1 # [ 128.035760] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6566worker1 # [ 128.037282] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6567client # [ 128.024220] nix[1393]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6568worker2 # [ 128.028349] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:27Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6569worker1 # [ 128.039132] systemd[1]: nix-grpc-daemon.service: Consumed 416ms CPU time over 16.152s wall clock time, 35.6M memory peak, 14.5K incoming IP traffic, 26.6K outgoing IP traffic.6570client # [ 128.029147] nix[1393]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6571worker1 # [ 128.049351] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6572worker1 # [ 128.050915] systemd[1]: Starting Nix worker-protocol over gRPC...6573worker1 # [ 128.074379] nix-daemon[844]: reaped child process 1556, status = succeeded6574worker1 # [ 128.081729] nix-daemon[844]: reaped child process 1519, status = succeeded6575worker1 # [ 128.090784] nix-daemon[1558]: unexpected Nix daemon error: error: interrupted by the user6576worker1 # [ 128.099206] nix-daemon[844]: reaped child process 1558, status = failed with exit code 16577worker1 # [ 128.141512] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6578worker1 # [ 128.143630] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6579worker1 # [ 128.146820] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16580worker1 # [ 128.151395] nix-grpc-daemon[1594]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6581worker1 # [ 128.154189] nix-grpc-daemon[1594]: E0000 00:00:1790076866.515927 1594 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6582worker1 # [ 128.160796] nix-grpc-daemon[1594]: E0000 00:00:1790076866.521092 1594 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6583worker1 # [ 128.163852] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6584worker1 # [ 128.166370] systemd[1]: Started Nix worker-protocol over gRPC.6585lb # [ 128.131434] niks3-server[1092]: 2026/09/22 11:34:26 INFO lead: acquired remote=192.168.1.3:551066586worker1 # [ 128.171174] nix-grpc-daemon[1594]: E0000 00:00:1790076866.531792 1636 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6587worker1 # [ 128.174106] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=info event=scheduler_take_over6588worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.19 seconds)6589client: must succeed: sleep 5.06590worker1 # [ 128.182068] nix-grpc-daemon[1594]: E0000 00:00:1790076866.546512 1636 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6591worker1 # [ 128.185159] nix-grpc-daemon[1594]: E0000 00:00:1790076866.548771 1636 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6592worker1 # [ 128.196150] nix-grpc-daemon[1594]: E0000 00:00:1790076866.560643 1636 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6593worker1 # [ 128.205372] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=info event=scheduler_connected addr=lb:500516594worker1 # [ 128.211874] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:450346595worker1 # [ 128.213560] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06596worker2 # [ 128.229186] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:27Z level=info event=scheduler_connected addr=lb:500516597worker1 # [ 128.239459] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=info event=oidc_jwks provider=mock keys=16598worker1 # [ 128.240817] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:450346599worker1 # [ 128.242461] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06600worker1 # [ 128.546562] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06601worker1 # [ 128.552369] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16602worker1 # [ 128.593140] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450466603worker1 # [ 128.601685] nix-daemon[844]: accepted connection from pid 1594, user nix-grpc-daemon (trusted)6604worker1 # [ 128.617200] nix-daemon[844]: accepted connection from pid 1594, user nix-grpc-daemon (trusted)6605client # [ 128.620300] nix[1393]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6606worker1 # [ 128.631672] nix-daemon[844]: accepted connection from pid 1594, user nix-grpc-daemon (trusted)6607client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6608client: waiting for success: ! systemctl is-active mid6609worker1 # [ 133.449523] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:31Z level=info event=niks3_push_started pid=16786610worker1 # [ 133.481613] nix-daemon[844]: reaped child process 1653, status = succeeded6611worker1 # [ 133.557631] nix-grpc-daemon[1678]: time=2026-09-22T11:34:31.921Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6612worker1 # [ 133.640203] nix-daemon[844]: accepted connection from pid 1691, user nix-grpc-daemon (trusted)6613worker1 # [ 133.652975] nix-daemon[844]: reaped child process 1694, status = succeeded6614lb # [ 133.625712] niks3-server[1092]: 2026/09/22 11:34:31 INFO Received uploads request method=POST path=/api/pending_closures6615worker1 # [ 133.668796] nix-grpc-daemon[1678]: time=2026-09-22T11:34:32.033Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6616worker1 # [ 133.670464] nix-grpc-daemon[1678]: time=2026-09-22T11:34:32.034Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6617lb # [ 133.668676] niks3-server[1092]: 2026/09/22 11:34:32 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6618worker1 # [ 133.726072] nix-grpc-daemon[1678]: time=2026-09-22T11:34:32.090Z level=INFO msg="Uploading 1 narinfos"6619lb # [ 133.688447] niks3-server[1092]: 2026/09/22 11:34:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6620lb # [ 133.692731] niks3-server[1092]: 2026/09/22 11:34:32 INFO Signed narinfos id=52 count=16621lb # [ 133.701279] niks3-server[1092]: 2026/09/22 11:34:32 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6622lb # [ 133.704512] niks3-server[1092]: 2026/09/22 11:34:32 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6623lb # [ 133.707770] niks3-server[1092]: 2026/09/22 11:34:32 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6624lb # [ 133.713461] niks3-server[1092]: 2026/09/22 11:34:32 INFO Completed upload id=526625worker1 # [ 133.750550] nix-grpc-daemon[1678]: time=2026-09-22T11:34:32.114Z level=INFO msg="Upload complete. (189ms)"6626lb # [ 133.716905] niks3-server[1092]: 2026/09/22 11:34:32 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6627worker1 # [ 133.758912] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45046 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16628worker1 # [ 133.770869] nix-daemon[844]: reaped child process 1651, status = succeeded6629client # [ 133.772935] systemd[1]: mid.service: Deactivated successfully.6630client # [ 133.774754] systemd[1]: mid.service: Consumed 141ms CPU time over 9.308s wall clock time, 41.4M memory peak, 8.5K incoming IP traffic, 10.2K outgoing IP traffic.6631client: (finished: waiting for success: ! systemctl is-active mid, in 1.04 seconds)6632client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6633client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.02 seconds)6634(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.01 seconds)6635subtest: interrupting the client stops the build on the worker6636client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr6637client # [ 134.452025] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr.6638client # Running as unit: intr.service; invocation ID: b82dfccd2934450abb860fd5a734fb566639client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr, in 0.03 seconds)6640worker1 # [ 134.581620] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:450466641client # [ 134.578089] nix[1429]: don't know how to build these paths:6642client # [ 134.579065] nix[1429]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6643worker1 # [ 134.599893] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:32Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06644worker1 # [ 134.603850] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:32Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16645worker2 # [ 134.648858] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449806646worker2 # [ 134.660281] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6647worker2 # [ 134.676607] nix-grpc-daemon[1663]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 72 ms (attempt 1/5)6648worker2 # [ 134.756453] nix-grpc-daemon[1663]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 165 ms (attempt 2/5)6649worker2 # [ 134.929637] nix-grpc-daemon[1663]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 225 ms (attempt 3/5)6650worker2 # [ 135.159276] nix-grpc-daemon[1663]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 362 ms (attempt 4/5)6651worker2 # [ 135.529264] nix-grpc-daemon[1663]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6652worker2 # [ 135.532215] nix-grpc-daemon[1663]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6653worker2 # [ 135.546529] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:449806654client # [ 135.551168] nix[1429]: copying 1 paths to 'lb:50051'...6655client # [ 135.555142] nix[1429]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6656worker2 # [ 135.559424] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:449806657worker2 # [ 135.567634] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6658worker2 # [ 135.589718] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:34Z level=info event=niks3_push_started pid=17556659worker2 # [ 135.696378] nix-grpc-daemon[1755]: time=2026-09-22T11:34:34.862Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""6660worker2 # [ 135.779580] nix-daemon[867]: accepted connection from pid 1771, user nix-grpc-daemon (trusted)6661worker2 # [ 135.793388] nix-daemon[867]: reaped child process 1774, status = succeeded6662worker2 # [ 135.867124] nix-daemon[867]: accepted connection from pid 1776, user nix-grpc-daemon (trusted)6663worker2 # [ 135.881236] nix-daemon[867]: reaped child process 1779, status = succeeded6664lb # [ 135.854904] niks3-server[1092]: 2026/09/22 11:34:34 INFO Received uploads request method=POST path=/api/pending_closures6665worker2 # [ 135.897067] nix-grpc-daemon[1755]: time=2026-09-22T11:34:35.062Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6666worker2 # [ 135.898416] nix-grpc-daemon[1755]: time=2026-09-22T11:34:35.062Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6667lb # [ 135.879844] niks3-server[1092]: 2026/09/22 11:34:34 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6668lb # [ 135.883925] niks3-server[1092]: 2026/09/22 11:34:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6669lb # [ 135.886609] niks3-server[1092]: 2026/09/22 11:34:34 INFO Signed narinfos id=53 count=16670worker2 # [ 135.918377] nix-grpc-daemon[1755]: time=2026-09-22T11:34:35.084Z level=INFO msg="Uploading 1 narinfos"6671lb # [ 135.894125] niks3-server[1092]: 2026/09/22 11:34:34 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6672lb # [ 135.897357] niks3-server[1092]: 2026/09/22 11:34:34 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6673lb # [ 135.900972] niks3-server[1092]: 2026/09/22 11:34:34 INFO Completed upload id=536674worker2 # [ 135.932539] nix-grpc-daemon[1755]: time=2026-09-22T11:34:35.098Z level=INFO msg="Upload complete. (233ms)"6675lb # [ 135.902810] niks3-server[1092]: 2026/09/22 11:34:34 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6676worker2 # [ 135.934944] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=0 paths=1 nar_bytes_in=4246677worker2 # [ 135.938711] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449806678worker2 # [ 135.943459] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6679worker2 # [ 135.948470] nix-daemon[867]: reaped child process 1753, status = succeeded6680worker2 # [ 135.952912] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6681client # [ 135.949225] nix[1429]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6682client: must succeed: systemctl kill -s INT intr6683client # [ 136.808605] systemd[1]: intr.service: Sent signal SIGINT to main process 1429 (nix) on client request.6684worker2 # [ 136.818768] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:35Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6685client # [ 136.814448] nix[1429]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6686client: (finished: must succeed: systemctl kill -s INT intr, in 0.04 seconds)6687client # [ 136.822372] nix[1429]: error: interrupted by the user6688client # [ 136.848510] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6689client # [ 136.851325] systemd[1]: intr.service: Failed with result 'exit-code'.6690client # [ 136.854724] systemd[1]: intr.service: Consumed 120ms CPU time over 2.399s wall clock time, 43.4M memory peak, 6.4K incoming IP traffic, 8.2K outgoing IP traffic.6691worker2 # [ 136.971814] nix-daemon[867]: reaped child process 1781, status = succeeded6692worker2 # [ 136.974260] nix-daemon[1783]: unexpected Nix daemon error: error: interrupted by the user6693worker2 # [ 136.978281] nix-daemon[867]: reaped child process 1783, status = failed with exit code 16694lb # [ 137.041560] envoy[918]: conn peer=[2001:db8:1::3]:51860 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=993 tx=399 ms=160076695lb # [ 137.045487] envoy[918]: conn peer=[2001:db8:1::1]:59138 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3878 tx=1946 ms=91556696lb # [ 137.048956] envoy[918]: conn peer=[2001:db8:1::1]:59018 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2967 tx=828 ms=22586697(finished: subtest: interrupting the client stops the build on the worker, in 3.49 seconds)6698subtest: a deploy mid-build drains: the build finishes, the new generation takes over6699client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw6700client # [ 137.945408] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw.6701client # Running as unit: sw.service; invocation ID: d2880e256a894b859dbdd600906a68506702client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw, in 0.03 seconds)6703worker1 # [ 138.073778] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:36Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:450466704client # [ 138.070040] nix[1449]: don't know how to build these paths:6705client # [ 138.071027] nix[1449]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6706worker1 # [ 138.087248] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:36Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06707worker1 # [ 138.092563] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:36Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16708worker1 # [ 138.131124] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450466709worker1 # [ 138.141641] nix-grpc-daemon[1594]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 75 ms (attempt 1/5)6710worker1 # [ 138.224066] nix-grpc-daemon[1594]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 119 ms (attempt 2/5)6711worker1 # [ 138.344127] nix-grpc-daemon[1594]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 241 ms (attempt 3/5)6712worker1 # [ 138.588635] nix-grpc-daemon[1594]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 594 ms (attempt 4/5)6713worker1 # [ 139.185822] nix-grpc-daemon[1594]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6714worker1 # [ 139.189808] nix-grpc-daemon[1594]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6715worker1 # [ 139.206396] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:450466716client # [ 139.207411] nix[1449]: copying 1 paths to 'lb:50051'...6717client # [ 139.208254] nix[1449]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6718worker1 # [ 139.217558] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:450466719worker1 # [ 139.228244] nix-daemon[844]: accepted connection from pid 1594, user nix-grpc-daemon (trusted)6720worker1 # [ 139.327423] nix-daemon[844]: accepted connection from pid 1734, user nix-grpc-daemon (trusted)6721worker1 # [ 139.340310] nix-daemon[844]: reaped child process 1737, status = succeeded6722worker1 # [ 139.420637] nix-daemon[844]: accepted connection from pid 1739, user nix-grpc-daemon (trusted)6723worker1 # [ 139.435599] nix-daemon[844]: reaped child process 1742, status = succeeded6724lb # [ 139.398756] niks3-server[1092]: 2026/09/22 11:34:37 INFO Received uploads request method=POST path=/api/pending_closures6725worker1 # [ 139.444721] nix-grpc-daemon[1678]: time=2026-09-22T11:34:37.808Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6726worker1 # [ 139.446176] nix-grpc-daemon[1678]: time=2026-09-22T11:34:37.808Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6727lb # [ 139.425555] niks3-server[1092]: 2026/09/22 11:34:37 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6728lb # [ 139.437873] niks3-server[1092]: 2026/09/22 11:34:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6729worker1 # [ 139.479636] nix-grpc-daemon[1678]: time=2026-09-22T11:34:37.843Z level=INFO msg="Uploading 1 narinfos"6730lb # [ 139.443132] niks3-server[1092]: 2026/09/22 11:34:37 INFO Signed narinfos id=54 count=16731lb # [ 139.452136] niks3-server[1092]: 2026/09/22 11:34:37 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6732lb # [ 139.460240] niks3-server[1092]: 2026/09/22 11:34:37 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6733worker1 # [ 139.504350] nix-grpc-daemon[1678]: time=2026-09-22T11:34:37.868Z level=INFO msg="Upload complete. (265ms)"6734lb # [ 139.467487] niks3-server[1092]: 2026/09/22 11:34:37 INFO Completed upload id=546735worker1 # [ 139.506360] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45046 duration_s=0 paths=1 nar_bytes_in=4166736lb # [ 139.472620] niks3-server[1092]: 2026/09/22 11:34:37 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6737worker1 # [ 139.511657] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:450466738worker1 # [ 139.519807] nix-daemon[844]: accepted connection from pid 1594, user nix-grpc-daemon (trusted)6739worker1 # [ 139.523375] nix-daemon[844]: reaped child process 1732, status = succeeded6740worker1 # [ 139.527722] nix-daemon[844]: accepted connection from pid 1594, user nix-grpc-daemon (trusted)6741client # [ 139.517885] nix[1449]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6742worker1: must succeed: systemctl reload nix-grpc-daemon.service6743worker1 # [ 141.338932] systemd[1]: Reloading Nix worker-protocol over gRPC...6744worker1 # [ 141.364254] systemd[1]: Reloaded Nix worker-protocol over gRPC.6745worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.06 seconds)6746worker1: must succeed: pgrep -f 'read -t [9]0 x'6747worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.03 seconds)6748client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&26749worker2 # [ 141.545383] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:449806750client # don't know how to build these paths:6751client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6752worker2 # [ 141.550241] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:449806753worker1 # [ 141.573341] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:39Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06754worker1 # [ 141.575206] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:39Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16755worker2 # [ 141.621873] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449806756worker2 # [ 141.630662] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:449806757client # copying 1 paths to 'lb:50051'...6758worker2 # [ 141.639368] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:449806759client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6760worker2 # [ 141.659942] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6761worker2 # [ 141.739803] nix-daemon[867]: accepted connection from pid 1834, user nix-grpc-daemon (trusted)6762worker2 # [ 141.754469] nix-daemon[867]: reaped child process 1837, status = succeeded6763worker2 # [ 141.828287] nix-daemon[867]: accepted connection from pid 1839, user nix-grpc-daemon (trusted)6764worker2 # [ 141.843124] nix-daemon[867]: reaped child process 1842, status = succeeded6765lb # [ 141.815954] niks3-server[1092]: 2026/09/22 11:34:40 INFO Received uploads request method=POST path=/api/pending_closures6766worker2 # [ 141.856161] nix-grpc-daemon[1755]: time=2026-09-22T11:34:41.021Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6767worker2 # [ 141.857515] nix-grpc-daemon[1755]: time=2026-09-22T11:34:41.021Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6768lb # [ 141.851839] niks3-server[1092]: 2026/09/22 11:34:40 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6769lb # [ 141.869175] niks3-server[1092]: 2026/09/22 11:34:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6770worker2 # [ 141.905855] nix-grpc-daemon[1755]: time=2026-09-22T11:34:41.071Z level=INFO msg="Uploading 1 narinfos"6771lb # [ 141.874491] niks3-server[1092]: 2026/09/22 11:34:40 INFO Signed narinfos id=55 count=16772lb # [ 141.881222] niks3-server[1092]: 2026/09/22 11:34:40 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6773lb # [ 141.891634] niks3-server[1092]: 2026/09/22 11:34:40 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6774lb # [ 141.895879] niks3-server[1092]: 2026/09/22 11:34:40 INFO Completed upload id=556775worker2 # [ 141.927152] nix-grpc-daemon[1755]: time=2026-09-22T11:34:41.093Z level=INFO msg="Upload complete. (262ms)"6776lb # [ 141.897254] niks3-server[1092]: 2026/09/22 11:34:40 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6777worker2 # [ 141.929392] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=0 paths=1 nar_bytes_in=5286778worker2 # [ 141.933401] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449806779worker2 # [ 141.939273] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6780worker2 # [ 141.945156] nix-daemon[867]: reaped child process 1832, status = succeeded6781worker2 # [ 141.947821] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6782client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6783client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6784worker1 # [ 142.164831] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:40Z level=info event=draining6785client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6786client # farm-a> LOG-farm-a-during-drain6787worker2 # [ 142.963875] nix-daemon[867]: reaped child process 1846, status = succeeded6788worker2 # [ 143.034555] nix-daemon[867]: accepted connection from pid 1862, user nix-grpc-daemon (trusted)6789worker2 # [ 143.047082] nix-daemon[867]: reaped child process 1865, status = succeeded6790lb # [ 143.021497] niks3-server[1092]: 2026/09/22 11:34:41 INFO Received uploads request method=POST path=/api/pending_closures6791worker2 # [ 143.063409] nix-grpc-daemon[1755]: time=2026-09-22T11:34:42.229Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6792worker2 # [ 143.064874] nix-grpc-daemon[1755]: time=2026-09-22T11:34:42.231Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6793lb # [ 143.053936] niks3-server[1092]: 2026/09/22 11:34:41 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6794lb # [ 143.072177] niks3-server[1092]: 2026/09/22 11:34:41 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6795lb # [ 143.075933] niks3-server[1092]: 2026/09/22 11:34:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6796lb # [ 143.078814] niks3-server[1092]: 2026/09/22 11:34:41 INFO Signed narinfos id=56 count=16797worker2 # [ 143.111094] nix-grpc-daemon[1755]: time=2026-09-22T11:34:42.276Z level=INFO msg="Uploading 1 narinfos"6798lb # [ 143.084939] niks3-server[1092]: 2026/09/22 11:34:41 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6799lb # [ 143.089153] niks3-server[1092]: 2026/09/22 11:34:41 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6800lb # [ 143.092623] niks3-server[1092]: 2026/09/22 11:34:41 INFO Completed upload id=566801worker2 # [ 143.124143] nix-grpc-daemon[1755]: time=2026-09-22T11:34:42.290Z level=INFO msg="Upload complete. (170ms)"6802lb # [ 143.096556] niks3-server[1092]: 2026/09/22 11:34:41 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6803worker2 # [ 143.127546] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16804worker1 # [ 143.138815] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:41Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06805worker1 # [ 143.140610] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:41Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16806worker2 # [ 143.139366] nix-daemon[867]: reaped child process 1844, status = succeeded6807worker2 # [ 143.176932] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449806808worker2 # [ 143.187392] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:449806809worker2 # [ 143.197700] nix-grpc-daemon[1755]: time=2026-09-22T11:34:42.363Z level=INFO msg="All 1 paths already cached"6810client # copying 1 paths to 'lb:50051'...6811client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6812worker2 # [ 143.206157] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:449806813worker2 # [ 143.217075] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6814worker2 # [ 143.306798] nix-daemon[867]: accepted connection from pid 1869, user nix-grpc-daemon (trusted)6815worker2 # [ 143.321978] nix-daemon[867]: reaped child process 1872, status = succeeded6816worker2 # [ 143.402136] nix-daemon[867]: accepted connection from pid 1874, user nix-grpc-daemon (trusted)6817worker2 # [ 143.415641] nix-daemon[867]: reaped child process 1877, status = succeeded6818lb # [ 143.390700] niks3-server[1092]: 2026/09/22 11:34:41 INFO Received uploads request method=POST path=/api/pending_closures6819worker2 # [ 143.430367] nix-grpc-daemon[1755]: time=2026-09-22T11:34:42.596Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6820worker2 # [ 143.432048] nix-grpc-daemon[1755]: time=2026-09-22T11:34:42.598Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6821lb # [ 143.428194] niks3-server[1092]: 2026/09/22 11:34:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6822worker2 # [ 143.464831] nix-grpc-daemon[1755]: time=2026-09-22T11:34:42.630Z level=INFO msg="Uploading 1 narinfos"6823lb # [ 143.433812] niks3-server[1092]: 2026/09/22 11:34:41 INFO Signed narinfos id=57 count=16824lb # [ 143.437376] niks3-server[1092]: 2026/09/22 11:34:41 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6825lb # [ 143.442193] niks3-server[1092]: 2026/09/22 11:34:41 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6826lb # [ 143.453649] niks3-server[1092]: 2026/09/22 11:34:41 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6827lb # [ 143.459477] niks3-server[1092]: 2026/09/22 11:34:41 INFO Completed upload id=576828worker2 # [ 143.492721] nix-grpc-daemon[1755]: time=2026-09-22T11:34:42.658Z level=INFO msg="Upload complete. (266ms)"6829lb # [ 143.462674] niks3-server[1092]: 2026/09/22 11:34:41 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6830worker2 # [ 143.496433] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=0 paths=1 nar_bytes_in=6406831worker2 # [ 143.501288] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449806832worker2 # [ 143.511436] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6833worker2 # [ 143.516137] nix-daemon[867]: reaped child process 1867, status = succeeded6834worker2 # [ 143.521321] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6835client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6836client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6837client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6838client # farm-b> LOG-farm-b-during-drain6839worker2 # [ 144.763116] nix-daemon[867]: reaped child process 1881, status = succeeded6840worker2 # [ 144.839097] nix-daemon[867]: accepted connection from pid 1897, user nix-grpc-daemon (trusted)6841worker2 # [ 144.851069] nix-daemon[867]: reaped child process 1900, status = succeeded6842lb # [ 144.827659] niks3-server[1092]: 2026/09/22 11:34:43 INFO Received uploads request method=POST path=/api/pending_closures6843worker2 # [ 144.867624] nix-grpc-daemon[1755]: time=2026-09-22T11:34:44.033Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6844worker2 # [ 144.869105] nix-grpc-daemon[1755]: time=2026-09-22T11:34:44.035Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6845lb # [ 144.859852] niks3-server[1092]: 2026/09/22 11:34:43 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6846lb # [ 144.877434] niks3-server[1092]: 2026/09/22 11:34:43 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6847lb # [ 144.886804] niks3-server[1092]: 2026/09/22 11:34:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6848lb # [ 144.891439] niks3-server[1092]: 2026/09/22 11:34:43 INFO Signed narinfos id=58 count=16849worker2 # [ 144.923101] nix-grpc-daemon[1755]: time=2026-09-22T11:34:44.089Z level=INFO msg="Uploading 1 narinfos"6850lb # [ 144.899727] niks3-server[1092]: 2026/09/22 11:34:43 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6851lb # [ 144.908075] niks3-server[1092]: 2026/09/22 11:34:43 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6852lb # [ 144.912122] niks3-server[1092]: 2026/09/22 11:34:43 INFO Completed upload id=586853worker2 # [ 144.943776] nix-grpc-daemon[1755]: time=2026-09-22T11:34:44.109Z level=INFO msg="Upload complete. (214ms)"6854lb # [ 144.914227] niks3-server[1092]: 2026/09/22 11:34:43 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6855worker2 # [ 144.949827] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16856worker1 # [ 144.959521] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:43Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06857worker1 # [ 144.962238] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:43Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16858worker2 # [ 144.965704] nix-daemon[867]: reaped child process 1879, status = succeeded6859worker2 # [ 144.997909] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449806860worker2 # [ 145.010948] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:449806861worker2 # [ 145.023694] nix-grpc-daemon[1755]: time=2026-09-22T11:34:44.189Z level=INFO msg="All 2 paths already cached"6862client # copying 1 paths to 'lb:50051'...6863client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6864worker2 # [ 145.033127] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:449806865worker2 # [ 145.045530] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6866worker2 # [ 145.140661] nix-daemon[867]: accepted connection from pid 1905, user nix-grpc-daemon (trusted)6867worker2 # [ 145.155508] nix-daemon[867]: reaped child process 1908, status = succeeded6868worker2 # [ 145.229662] nix-daemon[867]: accepted connection from pid 1910, user nix-grpc-daemon (trusted)6869worker2 # [ 145.243876] nix-daemon[867]: reaped child process 1913, status = succeeded6870lb # [ 145.216730] niks3-server[1092]: 2026/09/22 11:34:43 INFO Received uploads request method=POST path=/api/pending_closures6871worker2 # [ 145.252298] nix-grpc-daemon[1755]: time=2026-09-22T11:34:44.418Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6872worker2 # [ 145.254093] nix-grpc-daemon[1755]: time=2026-09-22T11:34:44.420Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6873lb # [ 145.242768] niks3-server[1092]: 2026/09/22 11:34:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6874lb # [ 145.248129] niks3-server[1092]: 2026/09/22 11:34:43 INFO Signed narinfos id=59 count=16875worker2 # [ 145.283056] nix-grpc-daemon[1755]: time=2026-09-22T11:34:44.449Z level=INFO msg="Uploading 1 narinfos"6876lb # [ 145.257822] niks3-server[1092]: 2026/09/22 11:34:43 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6877lb # [ 145.267729] niks3-server[1092]: 2026/09/22 11:34:43 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6878worker2 # [ 145.306195] nix-grpc-daemon[1755]: time=2026-09-22T11:34:44.472Z level=INFO msg="Upload complete. (246ms)"6879lb # [ 145.274764] niks3-server[1092]: 2026/09/22 11:34:43 INFO Completed upload id=596880worker2 # [ 145.308282] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=0 paths=1 nar_bytes_in=7686881worker2 # [ 145.312338] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449806882lb # [ 145.281107] niks3-server[1092]: 2026/09/22 11:34:43 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6883lb # [ 145.287136] niks3-server[1092]: 2026/09/22 11:34:43 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6884worker2 # [ 145.319453] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6885worker2 # [ 145.321851] nix-daemon[867]: reaped child process 1903, status = succeeded6886worker2 # [ 145.327130] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)6887client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6888client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6889client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6890client # farm-top> LOG-farm-top-during-drain6891worker2 # [ 146.797905] nix-daemon[867]: reaped child process 1917, status = succeeded6892worker2 # [ 146.839854] nix-daemon[867]: accepted connection from pid 1933, user nix-grpc-daemon (trusted)6893lb # [ 146.822760] niks3-server[1092]: 2026/09/22 11:34:45 INFO Received uploads request method=POST path=/api/pending_closures6894worker2 # [ 146.854702] nix-daemon[867]: reaped child process 1936, status = succeeded6895worker2 # [ 146.858822] nix-grpc-daemon[1755]: time=2026-09-22T11:34:46.024Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6896worker2 # [ 146.860291] nix-grpc-daemon[1755]: time=2026-09-22T11:34:46.026Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6897lb # [ 146.842694] niks3-server[1092]: 2026/09/22 11:34:45 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6898lb # [ 146.847408] niks3-server[1092]: 2026/09/22 11:34:45 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6899lb # [ 146.849902] niks3-server[1092]: 2026/09/22 11:34:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6900lb # [ 146.851683] niks3-server[1092]: 2026/09/22 11:34:45 INFO Signed narinfos id=60 count=16901worker2 # [ 146.884460] nix-grpc-daemon[1755]: time=2026-09-22T11:34:46.050Z level=INFO msg="Uploading 1 narinfos"6902lb # [ 146.858572] niks3-server[1092]: 2026/09/22 11:34:45 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6903lb # [ 146.861304] niks3-server[1092]: 2026/09/22 11:34:45 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6904lb # [ 146.864837] niks3-server[1092]: 2026/09/22 11:34:45 INFO Completed upload id=606905worker2 # [ 146.896298] nix-grpc-daemon[1755]: time=2026-09-22T11:34:46.062Z level=INFO msg="Upload complete. (160ms)"6906lb # [ 146.868175] niks3-server[1092]: 2026/09/22 11:34:45 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6907worker2 # [ 146.899035] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16908worker2 # [ 146.906209] nix-daemon[867]: reaped child process 1915, status = succeeded6909client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&2, in 5.54 seconds)6910client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath6911client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath, in 0.12 seconds)6912worker1: must succeed: pkill -f 'read -t [9]0 x'6913lb # [ 147.041267] envoy[918]: conn peer=[2001:db8:1::1]:59026 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9598 tx=4384 ms=53686914worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.04 seconds)6915client: waiting for success: ! systemctl is-active sw6916worker1 # [ 147.148509] nix-daemon[844]: reaped child process 1746, status = succeeded6917worker1 # [ 147.221922] nix-daemon[844]: accepted connection from pid 1793, user nix-grpc-daemon (trusted)6918worker1 # [ 147.234520] nix-daemon[844]: reaped child process 1796, status = succeeded6919lb # [ 147.204310] niks3-server[1092]: 2026/09/22 11:34:45 INFO Received uploads request method=POST path=/api/pending_closures6920worker1 # [ 147.249773] nix-grpc-daemon[1678]: time=2026-09-22T11:34:45.614Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6921worker1 # [ 147.251437] nix-grpc-daemon[1678]: time=2026-09-22T11:34:45.615Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6922lb # [ 147.243547] niks3-server[1092]: 2026/09/22 11:34:45 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6923lb # [ 147.254097] niks3-server[1092]: 2026/09/22 11:34:45 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6924lb # [ 147.258470] niks3-server[1092]: 2026/09/22 11:34:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6925lb # [ 147.263920] niks3-server[1092]: 2026/09/22 11:34:45 INFO Signed narinfos id=61 count=16926worker1 # [ 147.304234] nix-grpc-daemon[1678]: time=2026-09-22T11:34:45.668Z level=INFO msg="Uploading 1 narinfos"6927lb # [ 147.278771] niks3-server[1092]: 2026/09/22 11:34:45 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6928lb # [ 147.283329] niks3-server[1092]: 2026/09/22 11:34:45 INFO Completed upload id=616929worker1 # [ 147.320076] nix-grpc-daemon[1678]: time=2026-09-22T11:34:45.684Z level=INFO msg="Upload complete. (179ms)"6930lb # [ 147.285703] niks3-server[1092]: 2026/09/22 11:34:45 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6931worker1 # [ 147.323194] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45046 duration_s=7 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16932lb # [ 147.289265] niks3-server[1092]: 2026/09/22 11:34:45 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6933worker1 # [ 147.330897] nix-daemon[844]: reaped child process 1744, status = succeeded6934client # [ 147.349226] systemd[1]: sw.service: Deactivated successfully.6935client # [ 147.353220] systemd[1]: sw.service: Consumed 123ms CPU time over 9.407s wall clock time, 39.4M memory peak, 7.5K incoming IP traffic, 8.5K outgoing IP traffic.6936client: (finished: waiting for success: ! systemctl is-active sw, in 1.05 seconds)6937client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6938client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.02 seconds)6939worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6940worker1 # [ 148.166859] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:46Z level=info event=signal_exit6941lb # [ 148.134511] niks3-server[1092]: 2026/09/22 11:34:46 INFO lead: released remote=192.168.1.3:551066942worker2 # [ 149.166869] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:48Z level=info event=scheduler_restarting addr=lb:500516943worker1 # [ 149.177307] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:47Z level=info event=scheduler_restarting6944worker1 # [ 149.180844] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:47Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:450346945worker1 # [ 149.185233] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:47Z level=info event=worker_gone addr=192.168.1.4:500516946worker1 # [ 149.188616] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:47Z level=info event=scheduler_restarting addr=lb:500516947worker1 # [ 149.192424] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:47Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:450346948worker1 # [ 149.196125] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:47Z level=info event=worker_gone addr=192.168.1.3:500516949worker1 # [ 149.212248] nix-daemon[844]: reaped child process 1649, status = succeeded6950worker2 # [ 149.313362] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:48Z level=info event=scheduler_take_over6951lb # [ 149.282392] niks3-server[1092]: 2026/09/22 11:34:47 INFO lead: acquired remote=192.168.1.4:513926952worker2 # [ 149.367305] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:48Z level=info event=scheduler_connected addr=lb:500516953worker2 # [ 149.369313] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6954worker1 # [ 149.375565] nix-grpc-daemon[1594]: E0000 00:00:1790076887.739985 1640 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true6955worker1 # [ 149.377797] nix-grpc-daemon[1594]: E0000 00:00:1790076887.740068 1640 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6956worker1 # [ 149.385145] nix-grpc-daemon[1594]: ts=2026-09-22T11:34:47Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6957worker2 # [ 149.570228] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:48Z level=info event=scheduler_connected addr=lb:500516958worker2 # [ 149.572128] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6959worker2 # [ 149.772886] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:48Z level=info event=scheduler_connected addr=lb:500516960worker2 # [ 149.774772] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6961worker2 # [ 149.975663] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:49Z level=info event=scheduler_connected addr=lb:500516962worker2 # [ 149.977555] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6963worker2 # [ 150.178801] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:49Z level=info event=scheduler_connected addr=lb:500516964worker2 # [ 150.180443] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6965worker1 # [ 150.189246] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6966worker1 # [ 150.193154] systemd[1]: nix-grpc-daemon.service: Consumed 607ms CPU time over 22.155s wall clock time, 36.3M memory peak, 24.6K incoming IP traffic, 31.7K outgoing IP traffic.6967worker1 # [ 150.207868] systemd[1]: Starting Nix worker-protocol over gRPC...6968worker1 # [ 150.293459] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:48Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6969worker1 # [ 150.295993] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:48Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6970worker1 # [ 150.300343] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:48Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16971worker1 # [ 150.304127] nix-grpc-daemon[1810]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6972worker1 # [ 150.307695] nix-grpc-daemon[1810]: E0000 00:00:1790076888.664735 1810 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6973worker1 # [ 150.311690] nix-grpc-daemon[1810]: E0000 00:00:1790076888.668477 1810 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6974worker1 # [ 150.315169] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:48Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6975worker1 # [ 150.319127] systemd[1]: Started Nix worker-protocol over gRPC.6976worker1 # [ 150.319939] nix-grpc-daemon[1810]: E0000 00:00:1790076888.682148 1851 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6977worker1 # [ 150.331577] nix-grpc-daemon[1810]: E0000 00:00:1790076888.696104 1851 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6978worker1 # [ 150.336147] nix-grpc-daemon[1810]: E0000 00:00:1790076888.698268 1851 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6979worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.18 seconds)6980lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6981lb: must succeed: curl -sf localhost:9901/clusters6982worker1 # [ 150.346799] nix-grpc-daemon[1810]: E0000 00:00:1790076888.711285 1851 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6983worker1 # [ 150.356721] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:48Z level=info event=scheduler_connected addr=lb:500516984worker2 # [ 150.362443] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:49Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:476086985worker2 # [ 150.364108] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:49Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06986worker2 # [ 150.380796] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:49Z level=info event=scheduler_connected addr=lb:500516987worker2 # [ 150.388068] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:49Z level=info event=oidc_jwks provider=mock keys=16988worker2 # [ 150.389587] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:49Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:476086989worker2 # [ 150.391218] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:49Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06990lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6991lb: must succeed: curl -sf localhost:9901/clusters6992lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6993lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.11 seconds)6994lb: must succeed: curl -sf localhost:9901/clusters6995lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6996worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6997worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)6998(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 13.57 seconds)6999subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does7000client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&17001worker1 # [ 151.639825] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:482647002worker1 # [ 151.651331] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7003worker2 # [ 151.661820] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:449807004worker2 # [ 151.701456] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:50Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=07005worker2 # [ 151.703490] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:50Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=17006worker2 # [ 151.705369] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:50Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=07007worker2 # [ 151.706881] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:50Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=07008worker1 # [ 151.751764] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647009worker1 # [ 151.764723] nix-grpc-daemon[1810]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 59 ms (attempt 1/5)7010worker1 # [ 151.831085] nix-grpc-daemon[1810]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 69 ms (attempt 2/5)7011worker1 # [ 151.902236] nix-grpc-daemon[1810]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 154 ms (attempt 3/5)7012worker1 # [ 152.057307] nix-grpc-daemon[1810]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 51 ms (attempt 4/5)7013worker1 # [ 152.112364] nix-grpc-daemon[1810]: disabling binary cache 'https://cache.nixos.org' for 60 seconds7014worker1 # [ 152.114033] nix-grpc-daemon[1810]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org7015worker1 # [ 152.125455] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482647016worker1 # [ 152.132825] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482647017worker1 # [ 152.137843] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7018worker1 # [ 152.148365] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:50Z level=info event=niks3_push_started pid=18707019worker1 # [ 152.231945] nix-grpc-daemon[1870]: time=2026-09-22T11:34:50.596Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""7020worker1 # [ 152.310316] nix-daemon[844]: accepted connection from pid 1883, user nix-grpc-daemon (trusted)7021worker1 # [ 152.327114] nix-daemon[844]: reaped child process 1886, status = succeeded7022worker1 # [ 152.396286] nix-daemon[844]: accepted connection from pid 1888, user nix-grpc-daemon (trusted)7023worker1 # [ 152.409643] nix-daemon[844]: reaped child process 1891, status = succeeded7024lb # [ 152.377696] niks3-server[1092]: 2026/09/22 11:34:50 INFO Received uploads request method=POST path=/api/pending_closures7025worker1 # [ 152.424721] nix-grpc-daemon[1870]: time=2026-09-22T11:34:50.789Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7026worker1 # [ 152.427900] nix-grpc-daemon[1870]: time=2026-09-22T11:34:50.790Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"7027lb # [ 152.427164] niks3-server[1092]: 2026/09/22 11:34:50 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst7028lb # [ 152.432663] niks3-server[1092]: 2026/09/22 11:34:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign7029lb # [ 152.438061] niks3-server[1092]: 2026/09/22 11:34:50 INFO Signed narinfos id=62 count=17030worker1 # [ 152.477527] nix-grpc-daemon[1870]: time=2026-09-22T11:34:50.841Z level=INFO msg="Uploading 1 narinfos"7031lb # [ 152.453333] niks3-server[1092]: 2026/09/22 11:34:50 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls7032lb # [ 152.454827] niks3-server[1092]: 2026/09/22 11:34:50 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete7033worker1 # [ 152.495403] nix-grpc-daemon[1870]: time=2026-09-22T11:34:50.859Z level=INFO msg="Upload complete. (260ms)"7034lb # [ 152.459296] niks3-server[1092]: 2026/09/22 11:34:50 INFO Completed upload id=627035worker1 # [ 152.497627] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=0 paths=1 nar_bytes_in=5047036worker1 # [ 152.501814] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647037worker1 # [ 152.507703] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7038worker1 # [ 152.509960] nix-daemon[844]: reaped child process 1868, status = succeeded7039worker1 # [ 152.514941] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7040worker1 # [ 153.529571] nix-daemon[844]: reaped child process 1895, status = succeeded7041worker1 # [ 153.602720] nix-daemon[844]: accepted connection from pid 1911, user nix-grpc-daemon (trusted)7042worker1 # [ 153.615086] nix-daemon[844]: reaped child process 1914, status = succeeded7043lb # [ 153.585951] niks3-server[1092]: 2026/09/22 11:34:51 INFO Received uploads request method=POST path=/api/pending_closures7044worker1 # [ 153.635965] nix-grpc-daemon[1870]: time=2026-09-22T11:34:52.000Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7045worker1 # [ 153.637662] nix-grpc-daemon[1870]: time=2026-09-22T11:34:52.001Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"7046lb # [ 153.632615] niks3-server[1092]: 2026/09/22 11:34:51 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv7047lb # [ 153.639459] niks3-server[1092]: 2026/09/22 11:34:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign7048worker1 # [ 153.683740] nix-grpc-daemon[1870]: time=2026-09-22T11:34:52.047Z level=INFO msg="Uploading 1 narinfos"7049lb # [ 153.646694] niks3-server[1092]: 2026/09/22 11:34:51 INFO Signed narinfos id=63 count=17050lb # [ 153.650089] niks3-server[1092]: 2026/09/22 11:34:52 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst7051lb # [ 153.656321] niks3-server[1092]: 2026/09/22 11:34:52 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete7052lb # [ 153.659437] niks3-server[1092]: 2026/09/22 11:34:52 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls7053lb # [ 153.665211] niks3-server[1092]: 2026/09/22 11:34:52 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo7054lb # [ 153.674164] niks3-server[1092]: 2026/09/22 11:34:52 INFO Completed upload id=637055worker1 # [ 153.712650] nix-grpc-daemon[1870]: time=2026-09-22T11:34:52.077Z level=INFO msg="Upload complete. (191ms)"7056worker1 # [ 153.717945] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=17057worker2 # [ 153.718804] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:52Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=17058worker1 # [ 153.733211] nix-daemon[844]: reaped child process 1893, status = succeeded7059worker2 # [ 153.726736] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:52Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=07060worker1 # [ 153.774508] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647061worker1 # [ 153.786684] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482647062worker1 # [ 153.798655] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482647063worker1 # [ 153.806694] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7064worker1 # [ 153.899452] nix-daemon[844]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)7065worker1 # [ 153.911636] nix-daemon[844]: reaped child process 1921, status = succeeded7066worker1 # [ 153.985491] nix-daemon[844]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)7067worker1 # [ 153.998041] nix-daemon[844]: reaped child process 1926, status = succeeded7068lb # [ 153.966428] niks3-server[1092]: 2026/09/22 11:34:52 INFO Received uploads request method=POST path=/api/pending_closures7069worker1 # [ 154.013743] nix-grpc-daemon[1870]: time=2026-09-22T11:34:52.378Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7070worker1 # [ 154.015729] nix-grpc-daemon[1870]: time=2026-09-22T11:34:52.379Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"7071lb # [ 153.998145] niks3-server[1092]: 2026/09/22 11:34:52 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst7072lb # [ 154.010424] niks3-server[1092]: 2026/09/22 11:34:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7073worker1 # [ 154.052897] nix-grpc-daemon[1870]: time=2026-09-22T11:34:52.417Z level=INFO msg="Uploading 1 narinfos"7074lb # [ 154.015639] niks3-server[1092]: 2026/09/22 11:34:52 INFO Signed narinfos id=64 count=17075lb # [ 154.029978] niks3-server[1092]: 2026/09/22 11:34:52 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7076lb # [ 154.037626] niks3-server[1092]: 2026/09/22 11:34:52 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls7077lb # [ 154.042097] niks3-server[1092]: 2026/09/22 11:34:52 INFO Completed upload id=647078worker1 # [ 154.079092] nix-grpc-daemon[1870]: time=2026-09-22T11:34:52.443Z level=INFO msg="Upload complete. (259ms)"7079worker1 # [ 154.081174] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=0 paths=1 nar_bytes_in=5047080lb # [ 154.045258] niks3-server[1092]: 2026/09/22 11:34:52 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo7081worker1 # [ 154.087401] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647082worker1 # [ 154.091506] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7083worker1 # [ 154.093653] nix-daemon[844]: reaped child process 1916, status = succeeded7084worker1 # [ 154.098652] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7085worker1 # [ 154.964036] nix-daemon[844]: reaped child process 1930, status = succeeded7086worker1 # [ 155.004184] nix-daemon[844]: accepted connection from pid 1946, user nix-grpc-daemon (trusted)7087worker1 # [ 155.016456] nix-daemon[844]: reaped child process 1949, status = succeeded7088lb # [ 154.985808] niks3-server[1092]: 2026/09/22 11:34:53 INFO Received uploads request method=POST path=/api/pending_closures7089worker1 # [ 155.031044] nix-grpc-daemon[1870]: time=2026-09-22T11:34:53.395Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7090worker1 # [ 155.032728] nix-grpc-daemon[1870]: time=2026-09-22T11:34:53.397Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"7091lb # [ 155.024696] niks3-server[1092]: 2026/09/22 11:34:53 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst7092lb # [ 155.031164] niks3-server[1092]: 2026/09/22 11:34:53 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv7093lb # [ 155.041795] niks3-server[1092]: 2026/09/22 11:34:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7094lb # [ 155.047849] niks3-server[1092]: 2026/09/22 11:34:53 INFO Signed narinfos id=65 count=17095worker1 # [ 155.087771] nix-grpc-daemon[1870]: time=2026-09-22T11:34:53.452Z level=INFO msg="Uploading 1 narinfos"7096lb # [ 155.056786] niks3-server[1092]: 2026/09/22 11:34:53 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls7097lb # [ 155.065130] niks3-server[1092]: 2026/09/22 11:34:53 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7098lb # [ 155.072155] niks3-server[1092]: 2026/09/22 11:34:53 INFO Completed upload id=657099lb # [ 155.073730] niks3-server[1092]: 2026/09/22 11:34:53 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo7100worker1 # [ 155.109369] nix-grpc-daemon[1870]: time=2026-09-22T11:34:53.473Z level=INFO msg="Upload complete. (203ms)"7101worker1 # [ 155.115661] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=17102worker2 # [ 155.111708] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:54Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=17103worker1 # [ 155.124373] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647104worker2 # [ 155.120391] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:54Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=07105worker1 # [ 155.135843] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482647106worker1 # [ 155.143705] nix-daemon[844]: reaped child process 1928, status = succeeded7107worker1 # [ 155.146885] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482647108worker1 # [ 155.152739] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7109worker1 # [ 155.233275] nix-daemon[844]: accepted connection from pid 1953, user nix-grpc-daemon (trusted)7110worker1 # [ 155.245644] nix-daemon[844]: reaped child process 1956, status = succeeded7111worker1 # [ 155.320108] nix-daemon[844]: accepted connection from pid 1958, user nix-grpc-daemon (trusted)7112worker1 # [ 155.333240] nix-daemon[844]: reaped child process 1961, status = succeeded7113lb # [ 155.300482] niks3-server[1092]: 2026/09/22 11:34:53 INFO Received uploads request method=POST path=/api/pending_closures7114worker1 # [ 155.345072] nix-grpc-daemon[1870]: time=2026-09-22T11:34:53.709Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7115worker1 # [ 155.346505] nix-grpc-daemon[1870]: time=2026-09-22T11:34:53.709Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"7116lb # [ 155.325457] niks3-server[1092]: 2026/09/22 11:34:53 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst7117lb # [ 155.338818] niks3-server[1092]: 2026/09/22 11:34:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7118worker1 # [ 155.380743] nix-grpc-daemon[1870]: time=2026-09-22T11:34:53.745Z level=INFO msg="Uploading 1 narinfos"7119lb # [ 155.343801] niks3-server[1092]: 2026/09/22 11:34:53 INFO Signed narinfos id=66 count=17120lb # [ 155.349919] niks3-server[1092]: 2026/09/22 11:34:53 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls7121lb # [ 155.364761] niks3-server[1092]: 2026/09/22 11:34:53 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7122worker1 # [ 155.406911] nix-grpc-daemon[1870]: time=2026-09-22T11:34:53.771Z level=INFO msg="Upload complete. (246ms)"7123lb # [ 155.370580] niks3-server[1092]: 2026/09/22 11:34:53 INFO Completed upload id=667124worker1 # [ 155.409165] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=0 paths=1 nar_bytes_in=5047125lb # [ 155.374154] niks3-server[1092]: 2026/09/22 11:34:53 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo7126worker1 # [ 155.413335] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647127worker1 # [ 155.417508] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7128worker1 # [ 155.422224] nix-daemon[844]: reaped child process 1951, status = succeeded7129worker1 # [ 155.427526] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7130worker1 # [ 156.243132] nix-daemon[844]: reaped child process 1965, status = succeeded7131worker1 # [ 156.310478] nix-daemon[844]: accepted connection from pid 1981, user nix-grpc-daemon (trusted)7132worker1 # [ 156.321681] nix-daemon[844]: reaped child process 1984, status = succeeded7133lb # [ 156.291459] niks3-server[1092]: 2026/09/22 11:34:54 INFO Received uploads request method=POST path=/api/pending_closures7134worker1 # [ 156.336586] nix-grpc-daemon[1870]: time=2026-09-22T11:34:54.700Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7135worker1 # [ 156.338199] nix-grpc-daemon[1870]: time=2026-09-22T11:34:54.702Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"7136lb # [ 156.322939] niks3-server[1092]: 2026/09/22 11:34:54 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst7137lb # [ 156.337170] niks3-server[1092]: 2026/09/22 11:34:54 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv7138worker1 # [ 156.384963] nix-grpc-daemon[1870]: time=2026-09-22T11:34:54.749Z level=INFO msg="Uploading 1 narinfos"7139lb # [ 156.348042] niks3-server[1092]: 2026/09/22 11:34:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7140lb # [ 156.349539] niks3-server[1092]: 2026/09/22 11:34:54 INFO Signed narinfos id=67 count=17141lb # [ 156.353641] niks3-server[1092]: 2026/09/22 11:34:54 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls7142lb # [ 156.357467] niks3-server[1092]: 2026/09/22 11:34:54 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7143lb # [ 156.361244] niks3-server[1092]: 2026/09/22 11:34:54 INFO Completed upload id=677144worker1 # [ 156.397951] nix-grpc-daemon[1870]: time=2026-09-22T11:34:54.762Z level=INFO msg="Upload complete. (165ms)"7145lb # [ 156.362657] niks3-server[1092]: 2026/09/22 11:34:54 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo7146worker1 # [ 156.400499] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=0 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=17147worker1 # [ 156.407856] nix-daemon[844]: reaped child process 1963, status = succeeded7148worker2 # [ 156.397897] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:55Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=17149worker2 # [ 156.407822] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:55Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=07150worker1 # [ 156.453790] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647151worker1 # [ 156.460139] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482647152worker1 # [ 156.466168] nix-grpc-daemon[1870]: time=2026-09-22T11:34:54.830Z level=INFO msg="All 1 paths already cached"7153worker1 # [ 156.470037] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482647154worker1 # [ 156.473714] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7155worker1 # [ 156.551350] nix-daemon[844]: accepted connection from pid 1988, user nix-grpc-daemon (trusted)7156worker1 # [ 156.563902] nix-daemon[844]: reaped child process 1991, status = succeeded7157worker1 # [ 156.637109] nix-daemon[844]: accepted connection from pid 1993, user nix-grpc-daemon (trusted)7158worker1 # [ 156.650630] nix-daemon[844]: reaped child process 1996, status = succeeded7159lb # [ 156.617356] niks3-server[1092]: 2026/09/22 11:34:54 INFO Received uploads request method=POST path=/api/pending_closures7160worker1 # [ 156.658855] nix-grpc-daemon[1870]: time=2026-09-22T11:34:55.023Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7161worker1 # [ 156.660256] nix-grpc-daemon[1870]: time=2026-09-22T11:34:55.023Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"7162lb # [ 156.629174] niks3-server[1092]: 2026/09/22 11:34:54 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst7163lb # [ 156.634334] niks3-server[1092]: 2026/09/22 11:34:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7164lb # [ 156.641465] niks3-server[1092]: 2026/09/22 11:34:54 INFO Signed narinfos id=68 count=17165worker1 # [ 156.680930] nix-grpc-daemon[1870]: time=2026-09-22T11:34:55.045Z level=INFO msg="Uploading 1 narinfos"7166lb # [ 156.645334] niks3-server[1092]: 2026/09/22 11:34:54 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls7167lb # [ 156.656317] niks3-server[1092]: 2026/09/22 11:34:55 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7168lb # [ 156.660802] niks3-server[1092]: 2026/09/22 11:34:55 INFO Completed upload id=687169worker1 # [ 156.697751] nix-grpc-daemon[1870]: time=2026-09-22T11:34:55.062Z level=INFO msg="Upload complete. (218ms)"7170lb # [ 156.662933] niks3-server[1092]: 2026/09/22 11:34:55 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo7171worker1 # [ 156.699924] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=0 paths=1 nar_bytes_in=6247172worker1 # [ 156.705386] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647173worker1 # [ 156.709967] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7174worker1 # [ 156.712254] nix-daemon[844]: reaped child process 1986, status = succeeded7175worker1 # [ 156.717225] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7176lb # [ 157.042552] envoy[918]: conn peer=[2001:db8:1::1]:59020 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3018 tx=1558 ms=92687177lb # [ 157.046245] envoy[918]: conn peer=[2001:db8:1::3]:56854 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=958 tx=344 ms=211807178worker1 # [ 158.186321] nix-daemon[844]: reaped child process 2000, status = succeeded7179worker1 # [ 158.230228] nix-daemon[844]: accepted connection from pid 2016, user nix-grpc-daemon (trusted)7180worker1 # [ 158.244453] nix-daemon[844]: reaped child process 2019, status = succeeded7181lb # [ 158.212990] niks3-server[1092]: 2026/09/22 11:34:56 INFO Received uploads request method=POST path=/api/pending_closures7182worker1 # [ 158.254416] nix-grpc-daemon[1870]: time=2026-09-22T11:34:56.618Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7183worker1 # [ 158.255881] nix-grpc-daemon[1870]: time=2026-09-22T11:34:56.620Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"7184lb # [ 158.235578] niks3-server[1092]: 2026/09/22 11:34:56 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst7185lb # [ 158.238355] niks3-server[1092]: 2026/09/22 11:34:56 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv7186lb # [ 158.244439] niks3-server[1092]: 2026/09/22 11:34:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7187lb # [ 158.248681] niks3-server[1092]: 2026/09/22 11:34:56 INFO Signed narinfos id=69 count=17188worker1 # [ 158.289237] nix-grpc-daemon[1870]: time=2026-09-22T11:34:56.653Z level=INFO msg="Uploading 1 narinfos"7189lb # [ 158.255767] niks3-server[1092]: 2026/09/22 11:34:56 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls7190lb # [ 158.259818] niks3-server[1092]: 2026/09/22 11:34:56 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7191lb # [ 158.263411] niks3-server[1092]: 2026/09/22 11:34:56 INFO Completed upload id=697192worker1 # [ 158.300402] nix-grpc-daemon[1870]: time=2026-09-22T11:34:56.664Z level=INFO msg="Upload complete. (175ms)"7193lb # [ 158.265250] niks3-server[1092]: 2026/09/22 11:34:56 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo7194worker1 # [ 158.303033] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=17195worker1 # [ 158.310839] nix-daemon[844]: reaped child process 1998, status = succeeded7196worker2 # [ 158.300831] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:57Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=17197worker1 # [ 158.312719] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647198worker2 # [ 158.310292] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:57Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=07199worker1 # [ 158.318949] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482647200worker1 # [ 158.324505] nix-grpc-daemon[1870]: time=2026-09-22T11:34:56.689Z level=INFO msg="All 1 paths already cached"7201worker1 # [ 158.328415] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482647202worker1 # [ 158.333236] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7203worker1 # [ 158.416171] nix-daemon[844]: accepted connection from pid 2023, user nix-grpc-daemon (trusted)7204worker1 # [ 158.430366] nix-daemon[844]: reaped child process 2026, status = succeeded7205worker1 # [ 158.509877] nix-daemon[844]: accepted connection from pid 2028, user nix-grpc-daemon (trusted)7206worker1 # [ 158.523659] nix-daemon[844]: reaped child process 2031, status = succeeded7207lb # [ 158.491560] niks3-server[1092]: 2026/09/22 11:34:56 INFO Received uploads request method=POST path=/api/pending_closures7208worker1 # [ 158.536604] nix-grpc-daemon[1870]: time=2026-09-22T11:34:56.900Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7209worker1 # [ 158.538031] nix-grpc-daemon[1870]: time=2026-09-22T11:34:56.900Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"7210lb # [ 158.514706] niks3-server[1092]: 2026/09/22 11:34:56 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst7211lb # [ 158.531124] niks3-server[1092]: 2026/09/22 11:34:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7212worker1 # [ 158.573401] nix-grpc-daemon[1870]: time=2026-09-22T11:34:56.937Z level=INFO msg="Uploading 1 narinfos"7213lb # [ 158.536277] niks3-server[1092]: 2026/09/22 11:34:56 INFO Signed narinfos id=70 count=17214lb # [ 158.542096] niks3-server[1092]: 2026/09/22 11:34:56 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls7215lb # [ 158.552771] niks3-server[1092]: 2026/09/22 11:34:56 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7216worker1 # [ 158.597568] nix-grpc-daemon[1870]: time=2026-09-22T11:34:56.962Z level=INFO msg="Upload complete. (259ms)"7217lb # [ 158.560721] niks3-server[1092]: 2026/09/22 11:34:56 INFO Completed upload id=707218worker1 # [ 158.599609] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=0 paths=1 nar_bytes_in=6247219lb # [ 158.564053] niks3-server[1092]: 2026/09/22 11:34:56 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo7220worker1 # [ 158.605695] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647221worker1 # [ 158.614907] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7222worker1 # [ 158.616284] nix-daemon[844]: reaped child process 2021, status = succeeded7223worker1 # [ 158.621493] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7224worker1 # [ 159.836369] nix-daemon[844]: reaped child process 2035, status = succeeded7225worker1 # [ 159.909766] nix-daemon[844]: accepted connection from pid 2051, user nix-grpc-daemon (trusted)7226worker1 # [ 159.921931] nix-daemon[844]: reaped child process 2054, status = succeeded7227lb # [ 159.892604] niks3-server[1092]: 2026/09/22 11:34:58 INFO Received uploads request method=POST path=/api/pending_closures7228worker1 # [ 159.938478] nix-grpc-daemon[1870]: time=2026-09-22T11:34:58.302Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7229worker1 # [ 159.940142] nix-grpc-daemon[1870]: time=2026-09-22T11:34:58.304Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"7230lb # [ 159.933116] niks3-server[1092]: 2026/09/22 11:34:58 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst7231lb # [ 159.941911] niks3-server[1092]: 2026/09/22 11:34:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7232worker1 # [ 159.984083] nix-grpc-daemon[1870]: time=2026-09-22T11:34:58.348Z level=INFO msg="Uploading 1 narinfos"7233lb # [ 159.947466] niks3-server[1092]: 2026/09/22 11:34:58 INFO Signed narinfos id=71 count=17234lb # [ 159.953180] niks3-server[1092]: 2026/09/22 11:34:58 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv7235lb # [ 159.961259] niks3-server[1092]: 2026/09/22 11:34:58 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls7236lb # [ 159.963169] niks3-server[1092]: 2026/09/22 11:34:58 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7237lb # [ 159.966841] niks3-server[1092]: 2026/09/22 11:34:58 INFO Completed upload id=717238worker1 # [ 160.003551] nix-grpc-daemon[1870]: time=2026-09-22T11:34:58.368Z level=INFO msg="Upload complete. (174ms)"7239lb # [ 159.970051] niks3-server[1092]: 2026/09/22 11:34:58 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo7240worker1 # [ 160.006125] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=17241worker1 # [ 160.013289] nix-daemon[844]: reaped child process 2033, status = succeeded7242worker2 # [ 160.002820] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:59Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=17243worker2 # [ 160.012507] nix-grpc-daemon[1663]: ts=2026-09-22T11:34:59Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=07244worker1 # [ 160.057655] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647245worker1 # [ 160.065236] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482647246worker1 # [ 160.073573] nix-grpc-daemon[1870]: time=2026-09-22T11:34:58.438Z level=INFO msg="All 1 paths already cached"7247worker1 # [ 160.077448] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482647248worker1 # [ 160.081158] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7249worker1 # [ 160.167515] nix-daemon[844]: accepted connection from pid 2058, user nix-grpc-daemon (trusted)7250worker1 # [ 160.182315] nix-daemon[844]: reaped child process 2061, status = succeeded7251worker1 # [ 160.260916] nix-daemon[844]: accepted connection from pid 2063, user nix-grpc-daemon (trusted)7252worker1 # [ 160.273223] nix-daemon[844]: reaped child process 2066, status = succeeded7253lb # [ 160.242855] niks3-server[1092]: 2026/09/22 11:34:58 INFO Received uploads request method=POST path=/api/pending_closures7254worker1 # [ 160.289066] nix-grpc-daemon[1870]: time=2026-09-22T11:34:58.653Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7255worker1 # [ 160.290493] nix-grpc-daemon[1870]: time=2026-09-22T11:34:58.653Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"7256lb # [ 160.270799] niks3-server[1092]: 2026/09/22 11:34:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7257lb # [ 160.276319] niks3-server[1092]: 2026/09/22 11:34:58 INFO Signed narinfos id=72 count=17258worker1 # [ 160.319300] nix-grpc-daemon[1870]: time=2026-09-22T11:34:58.683Z level=INFO msg="Uploading 1 narinfos"7259lb # [ 160.285335] niks3-server[1092]: 2026/09/22 11:34:58 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst7260lb # [ 160.289220] niks3-server[1092]: 2026/09/22 11:34:58 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls7261lb # [ 160.299260] niks3-server[1092]: 2026/09/22 11:34:58 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7262lb # [ 160.302206] niks3-server[1092]: 2026/09/22 11:34:58 INFO Completed upload id=727263worker1 # [ 160.338985] nix-grpc-daemon[1870]: time=2026-09-22T11:34:58.703Z level=INFO msg="Upload complete. (251ms)"7264worker1 # [ 160.340967] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=0 paths=1 nar_bytes_in=6247265lb # [ 160.305642] niks3-server[1092]: 2026/09/22 11:34:58 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo7266worker1 # [ 160.344785] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647267worker1 # [ 160.351077] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7268worker1 # [ 160.353352] nix-daemon[844]: reaped child process 2056, status = succeeded7269worker1 # [ 160.358324] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7270worker1 # [ 161.376854] nix-daemon[844]: reaped child process 2070, status = succeeded7271worker1 # [ 161.440891] nix-daemon[844]: accepted connection from pid 2086, user nix-grpc-daemon (trusted)7272worker1 # [ 161.454523] nix-daemon[844]: reaped child process 2089, status = succeeded7273lb # [ 161.421663] niks3-server[1092]: 2026/09/22 11:34:59 INFO Received uploads request method=POST path=/api/pending_closures7274worker1 # [ 161.463135] nix-grpc-daemon[1870]: time=2026-09-22T11:34:59.827Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7275worker1 # [ 161.464743] nix-grpc-daemon[1870]: time=2026-09-22T11:34:59.829Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"7276lb # [ 161.443091] niks3-server[1092]: 2026/09/22 11:34:59 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv7277lb # [ 161.447996] niks3-server[1092]: 2026/09/22 11:34:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7278lb # [ 161.450750] niks3-server[1092]: 2026/09/22 11:34:59 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst7279lb # [ 161.453566] niks3-server[1092]: 2026/09/22 11:34:59 INFO Signed narinfos id=73 count=17280worker1 # [ 161.490660] nix-grpc-daemon[1870]: time=2026-09-22T11:34:59.855Z level=INFO msg="Uploading 1 narinfos"7281lb # [ 161.456642] niks3-server[1092]: 2026/09/22 11:34:59 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls7282lb # [ 161.462662] niks3-server[1092]: 2026/09/22 11:34:59 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7283lb # [ 161.465551] niks3-server[1092]: 2026/09/22 11:34:59 INFO Completed upload id=737284worker1 # [ 161.502443] nix-grpc-daemon[1870]: time=2026-09-22T11:34:59.866Z level=INFO msg="Upload complete. (137ms)"7285lb # [ 161.468234] niks3-server[1092]: 2026/09/22 11:34:59 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo7286worker1 # [ 161.504732] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=17287worker1 # [ 161.509415] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647288worker2 # [ 161.502601] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:00Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=17289worker1 # [ 161.515096] nix-daemon[844]: reaped child process 2068, status = succeeded7290worker2 # [ 161.512240] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:00Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=07291worker1 # [ 161.552667] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482647292worker1 # [ 161.561634] nix-grpc-daemon[1870]: time=2026-09-22T11:34:59.926Z level=INFO msg="All 2 paths already cached"7293worker1 # [ 161.568175] nix-grpc-daemon[1810]: ts=2026-09-22T11:34:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482647294worker1 # [ 161.574688] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7295worker1 # [ 161.660610] nix-daemon[844]: accepted connection from pid 2093, user nix-grpc-daemon (trusted)7296worker1 # [ 161.673632] nix-daemon[844]: reaped child process 2096, status = succeeded7297worker1 # [ 161.752970] nix-daemon[844]: accepted connection from pid 2098, user nix-grpc-daemon (trusted)7298worker1 # [ 161.765201] nix-daemon[844]: reaped child process 2101, status = succeeded7299lb # [ 161.735034] niks3-server[1092]: 2026/09/22 11:35:00 INFO Received uploads request method=POST path=/api/pending_closures7300worker1 # [ 161.776214] nix-grpc-daemon[1870]: time=2026-09-22T11:35:00.140Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7301worker1 # [ 161.777551] nix-grpc-daemon[1870]: time=2026-09-22T11:35:00.140Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7302lb # [ 161.747618] niks3-server[1092]: 2026/09/22 11:35:00 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7303lb # [ 161.750747] niks3-server[1092]: 2026/09/22 11:35:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7304lb # [ 161.752561] niks3-server[1092]: 2026/09/22 11:35:00 INFO Signed narinfos id=74 count=17305worker1 # [ 161.789745] nix-grpc-daemon[1870]: time=2026-09-22T11:35:00.154Z level=INFO msg="Uploading 1 narinfos"7306lb # [ 161.758673] niks3-server[1092]: 2026/09/22 11:35:00 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7307lb # [ 161.761994] niks3-server[1092]: 2026/09/22 11:35:00 INFO Completed upload id=747308worker1 # [ 161.798903] nix-grpc-daemon[1870]: time=2026-09-22T11:35:00.163Z level=INFO msg="Upload complete. (214ms)"7309lb # [ 161.763594] niks3-server[1092]: 2026/09/22 11:35:00 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7310worker1 # [ 161.800818] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=0 paths=1 nar_bytes_in=7527311lb # [ 161.766939] niks3-server[1092]: 2026/09/22 11:35:00 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7312worker1 # [ 161.806385] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647313worker1 # [ 161.810561] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7314worker1 # [ 161.812717] nix-daemon[844]: reaped child process 2091, status = succeeded7315worker1 # [ 161.817857] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7316worker1 # [ 163.032376] nix-daemon[844]: reaped child process 2105, status = succeeded7317worker1 # [ 163.105086] nix-daemon[844]: accepted connection from pid 2121, user nix-grpc-daemon (trusted)7318worker1 # [ 163.118688] nix-daemon[844]: reaped child process 2124, status = succeeded7319lb # [ 163.086077] niks3-server[1092]: 2026/09/22 11:35:01 INFO Received uploads request method=POST path=/api/pending_closures7320worker1 # [ 163.132619] nix-grpc-daemon[1870]: time=2026-09-22T11:35:01.496Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7321worker1 # [ 163.135050] nix-grpc-daemon[1870]: time=2026-09-22T11:35:01.498Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7322lb # [ 163.127499] niks3-server[1092]: 2026/09/22 11:35:01 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7323lb # [ 163.138127] niks3-server[1092]: 2026/09/22 11:35:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7324worker1 # [ 163.182535] nix-grpc-daemon[1870]: time=2026-09-22T11:35:01.546Z level=INFO msg="Uploading 1 narinfos"7325lb # [ 163.145700] niks3-server[1092]: 2026/09/22 11:35:01 INFO Signed narinfos id=75 count=17326lb # [ 163.153582] niks3-server[1092]: 2026/09/22 11:35:01 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7327lb # [ 163.158224] niks3-server[1092]: 2026/09/22 11:35:01 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7328lb # [ 163.170330] niks3-server[1092]: 2026/09/22 11:35:01 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7329lb # [ 163.174133] niks3-server[1092]: 2026/09/22 11:35:01 INFO Completed upload id=757330worker1 # [ 163.212630] nix-grpc-daemon[1870]: time=2026-09-22T11:35:01.577Z level=INFO msg="Upload complete. (188ms)"7331lb # [ 163.176578] niks3-server[1092]: 2026/09/22 11:35:01 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7332worker1 # [ 163.215721] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=17333worker1 # [ 163.219518] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647334worker2 # [ 163.212260] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:02Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17335worker1 # [ 163.226284] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482647336worker1 # [ 163.228152] nix-daemon[844]: reaped child process 2103, status = succeeded7337worker1 # [ 163.232732] nix-grpc-daemon[1870]: time=2026-09-22T11:35:01.597Z level=INFO msg="All 2 paths already cached"7338worker1 # [ 163.236929] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482647339worker1 # [ 163.240475] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7340worker1 # [ 163.326129] nix-daemon[844]: accepted connection from pid 2128, user nix-grpc-daemon (trusted)7341worker1 # [ 163.340985] nix-daemon[844]: reaped child process 2131, status = succeeded7342worker1 # [ 163.419167] nix-daemon[844]: accepted connection from pid 2133, user nix-grpc-daemon (trusted)7343worker1 # [ 163.432871] nix-daemon[844]: reaped child process 2136, status = succeeded7344lb # [ 163.400450] niks3-server[1092]: 2026/09/22 11:35:01 INFO Received uploads request method=POST path=/api/pending_closures7345worker1 # [ 163.445526] nix-grpc-daemon[1870]: time=2026-09-22T11:35:01.810Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7346worker1 # [ 163.446999] nix-grpc-daemon[1870]: time=2026-09-22T11:35:01.811Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7347lb # [ 163.426424] niks3-server[1092]: 2026/09/22 11:35:01 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7348lb # [ 163.440203] niks3-server[1092]: 2026/09/22 11:35:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign7349worker1 # [ 163.482446] nix-grpc-daemon[1870]: time=2026-09-22T11:35:01.846Z level=INFO msg="Uploading 1 narinfos"7350lb # [ 163.445129] niks3-server[1092]: 2026/09/22 11:35:01 INFO Signed narinfos id=76 count=17351lb # [ 163.456520] niks3-server[1092]: 2026/09/22 11:35:01 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7352lb # [ 163.465245] niks3-server[1092]: 2026/09/22 11:35:01 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete7353worker1 # [ 163.507467] nix-grpc-daemon[1870]: time=2026-09-22T11:35:01.871Z level=INFO msg="Upload complete. (261ms)"7354lb # [ 163.470624] niks3-server[1092]: 2026/09/22 11:35:01 INFO Completed upload id=767355worker1 # [ 163.509466] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=0 paths=1 nar_bytes_in=7527356lb # [ 163.477074] niks3-server[1092]: 2026/09/22 11:35:01 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7357worker1 # [ 163.513934] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647358worker1 # [ 163.519734] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7359worker1 # [ 163.522048] nix-daemon[844]: reaped child process 2126, status = succeeded7360worker1 # [ 163.527246] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7361worker1 # [ 164.565665] nix-daemon[844]: reaped child process 2140, status = succeeded7362worker1 # [ 164.645970] nix-daemon[844]: accepted connection from pid 2156, user nix-grpc-daemon (trusted)7363worker1 # [ 164.658563] nix-daemon[844]: reaped child process 2159, status = succeeded7364lb # [ 164.629690] niks3-server[1092]: 2026/09/22 11:35:02 INFO Received uploads request method=POST path=/api/pending_closures7365worker1 # [ 164.677071] nix-grpc-daemon[1870]: time=2026-09-22T11:35:03.041Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7366worker1 # [ 164.678545] nix-grpc-daemon[1870]: time=2026-09-22T11:35:03.043Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7367lb # [ 164.668483] niks3-server[1092]: 2026/09/22 11:35:03 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7368lb # [ 164.673920] niks3-server[1092]: 2026/09/22 11:35:03 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7369lb # [ 164.685682] niks3-server[1092]: 2026/09/22 11:35:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign7370lb # [ 164.687650] niks3-server[1092]: 2026/09/22 11:35:03 INFO Signed narinfos id=77 count=17371worker1 # [ 164.724765] nix-grpc-daemon[1870]: time=2026-09-22T11:35:03.088Z level=INFO msg="Uploading 1 narinfos"7372lb # [ 164.691859] niks3-server[1092]: 2026/09/22 11:35:03 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7373lb # [ 164.696699] niks3-server[1092]: 2026/09/22 11:35:03 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete7374lb # [ 164.699932] niks3-server[1092]: 2026/09/22 11:35:03 INFO Completed upload id=777375lb # [ 164.701745] niks3-server[1092]: 2026/09/22 11:35:03 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7376worker1 # [ 164.738708] nix-grpc-daemon[1870]: time=2026-09-22T11:35:03.103Z level=INFO msg="Upload complete. (203ms)"7377worker2 # [ 164.739626] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:03Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17378worker1 # [ 164.743971] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=17379worker1 # [ 164.758150] nix-daemon[844]: reaped child process 2138, status = succeeded7380worker1 # [ 164.788071] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647381worker1 # [ 164.799968] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482647382worker1 # [ 164.812457] nix-grpc-daemon[1870]: time=2026-09-22T11:35:03.176Z level=INFO msg="All 2 paths already cached"7383worker1 # [ 164.820214] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482647384worker1 # [ 164.829157] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7385worker1 # [ 164.920878] nix-daemon[844]: accepted connection from pid 2163, user nix-grpc-daemon (trusted)7386worker1 # [ 164.933778] nix-daemon[844]: reaped child process 2166, status = succeeded7387worker1 # [ 165.012905] nix-daemon[844]: accepted connection from pid 2168, user nix-grpc-daemon (trusted)7388worker1 # [ 165.026782] nix-daemon[844]: reaped child process 2171, status = succeeded7389lb # [ 164.993985] niks3-server[1092]: 2026/09/22 11:35:03 INFO Received uploads request method=POST path=/api/pending_closures7390worker1 # [ 165.039133] nix-grpc-daemon[1870]: time=2026-09-22T11:35:03.403Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7391worker1 # [ 165.040513] nix-grpc-daemon[1870]: time=2026-09-22T11:35:03.403Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7392lb # [ 165.018903] niks3-server[1092]: 2026/09/22 11:35:03 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7393lb # [ 165.030988] niks3-server[1092]: 2026/09/22 11:35:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign7394worker1 # [ 165.073187] nix-grpc-daemon[1870]: time=2026-09-22T11:35:03.437Z level=INFO msg="Uploading 1 narinfos"7395lb # [ 165.036331] niks3-server[1092]: 2026/09/22 11:35:03 INFO Signed narinfos id=78 count=17396lb # [ 165.041726] niks3-server[1092]: 2026/09/22 11:35:03 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7397lb # [ 165.055600] niks3-server[1092]: 2026/09/22 11:35:03 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete7398worker1 # [ 165.098891] nix-grpc-daemon[1870]: time=2026-09-22T11:35:03.463Z level=INFO msg="Upload complete. (258ms)"7399lb # [ 165.062263] niks3-server[1092]: 2026/09/22 11:35:03 INFO Completed upload id=787400worker1 # [ 165.101426] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=0 paths=1 nar_bytes_in=7527401lb # [ 165.066290] niks3-server[1092]: 2026/09/22 11:35:03 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7402worker1 # [ 165.106070] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482647403worker1 # [ 165.115686] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7404worker1 # [ 165.117836] nix-daemon[844]: reaped child process 2161, status = succeeded7405worker1 # [ 165.123200] nix-daemon[844]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)7406worker1 # [ 166.339288] nix-daemon[844]: reaped child process 2175, status = succeeded7407worker1 # [ 166.412082] nix-daemon[844]: accepted connection from pid 2191, user nix-grpc-daemon (trusted)7408worker1 # [ 166.423504] nix-daemon[844]: reaped child process 2194, status = succeeded7409lb # [ 166.392749] niks3-server[1092]: 2026/09/22 11:35:04 INFO Received uploads request method=POST path=/api/pending_closures7410worker1 # [ 166.434698] nix-grpc-daemon[1870]: time=2026-09-22T11:35:04.799Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7411worker1 # [ 166.436304] nix-grpc-daemon[1870]: time=2026-09-22T11:35:04.800Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7412lb # [ 166.413329] niks3-server[1092]: 2026/09/22 11:35:04 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7413lb # [ 166.419093] niks3-server[1092]: 2026/09/22 11:35:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign7414lb # [ 166.421627] niks3-server[1092]: 2026/09/22 11:35:04 INFO Signed narinfos id=79 count=17415worker1 # [ 166.458361] nix-grpc-daemon[1870]: time=2026-09-22T11:35:04.822Z level=INFO msg="Uploading 1 narinfos"7416lb # [ 166.425189] niks3-server[1092]: 2026/09/22 11:35:04 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7417lb # [ 166.428575] niks3-server[1092]: 2026/09/22 11:35:04 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7418lb # [ 166.433379] niks3-server[1092]: 2026/09/22 11:35:04 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete7419lb # [ 166.436099] niks3-server[1092]: 2026/09/22 11:35:04 INFO Completed upload id=797420worker1 # [ 166.472988] nix-grpc-daemon[1870]: time=2026-09-22T11:35:04.837Z level=INFO msg="Upload complete. (143ms)"7421worker1 # [ 166.475484] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48264 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=17422lb # [ 166.439828] niks3-server[1092]: 2026/09/22 11:35:04 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7423worker1 # [ 166.482349] nix-daemon[844]: reaped child process 2173, status = succeeded7424client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 15.02 seconds)7425worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7426worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)7427client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]'7428client # [ 166.575979] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7429client # Running as unit: gpu.service; invocation ID: a54b3bdb7d9b4973906db387516f0e5f7430client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.06 seconds)7431lb: must succeed: curl -sf localhost:9901/clusters7432lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7433lb: must succeed: curl -sf localhost:9901/clusters7434lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7435worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7436worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7437worker1 # [ 166.721947] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:482647438client # [ 166.716078] nix[1529]: don't know how to build these paths:7439client # [ 166.717086] nix[1529]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7440worker1 # [ 166.729305] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482647441client # [ 166.739664] nix[1529]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker7442worker2 # [ 166.743900] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:05Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07443lb # [ 167.043297] envoy[918]: conn peer=[2001:db8:1::1]:46156 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27111 tx=12327 ms=148597444lb: must succeed: curl -sf localhost:9901/clusters7445lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7446lb: must succeed: curl -sf localhost:9901/clusters7447lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7448worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7449worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7450client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7451client # [ 196.740933] nix[1529]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}7452client # [ 196.749408] nix[1529]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7453client # [ 196.781708] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7454client # [ 196.785465] systemd[1]: gpu.service: Failed with result 'exit-code'.7455client # [ 196.789421] systemd[1]: gpu.service: Consumed 135ms CPU time over 30.209s wall clock time, 35.6M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.7456client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.11 seconds)7457client: must succeed: journalctl -u gpu -o cat7458client: (finished: must succeed: journalctl -u gpu -o cat, in 0.01 seconds)7459lb: must succeed: curl -sf localhost:9901/clusters7460lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7461lb: must succeed: curl -sf localhost:9901/clusters7462lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7463worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7464worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7465(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 45.51 seconds)7466subtest: low disk drains a worker and builds go to the other7467worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7468lb # [ 197.044748] envoy[918]: conn peer=[2001:db8:1::1]:50810 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300517469worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.90 seconds)7470worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7471worker1 # [ 198.321495] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:36Z level=info event=unhealthy reason=min_free7472worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.05 seconds)7473client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&27474client # don't know how to build these paths:7475client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7476worker2 # [ 199.075990] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:449807477worker2 # [ 199.082316] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:449807478worker2 # [ 199.102162] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:38Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07479worker2 # [ 199.106988] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:38Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17480worker2 # [ 199.151166] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449807481worker2 # [ 199.163284] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:449807482client # copying 1 paths to 'lb:50051'...7483client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7484worker2 # [ 199.174901] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:449807485worker2 # [ 199.186167] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)7486worker2 # [ 199.285892] nix-daemon[867]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7487worker2 # [ 199.299768] nix-daemon[867]: reaped child process 1973, status = succeeded7488worker2 # [ 199.381800] nix-daemon[867]: accepted connection from pid 1975, user nix-grpc-daemon (trusted)7489worker2 # [ 199.395947] nix-daemon[867]: reaped child process 1978, status = succeeded7490lb # [ 199.370555] niks3-server[1092]: 2026/09/22 11:35:37 INFO Received uploads request method=POST path=/api/pending_closures7491worker2 # [ 199.410226] nix-grpc-daemon[1755]: time=2026-09-22T11:35:38.576Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7492worker2 # [ 199.412104] nix-grpc-daemon[1755]: time=2026-09-22T11:35:38.578Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7493lb # [ 199.400877] niks3-server[1092]: 2026/09/22 11:35:37 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7494lb # [ 199.410356] niks3-server[1092]: 2026/09/22 11:35:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign7495lb # [ 199.417195] niks3-server[1092]: 2026/09/22 11:35:37 INFO Signed narinfos id=80 count=17496worker2 # [ 199.453145] nix-grpc-daemon[1755]: time=2026-09-22T11:35:38.619Z level=INFO msg="Uploading 1 narinfos"7497lb # [ 199.429114] niks3-server[1092]: 2026/09/22 11:35:37 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7498lb # [ 199.441505] niks3-server[1092]: 2026/09/22 11:35:37 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete7499lb # [ 199.445408] niks3-server[1092]: 2026/09/22 11:35:37 INFO Completed upload id=807500worker2 # [ 199.476840] nix-grpc-daemon[1755]: time=2026-09-22T11:35:38.643Z level=INFO msg="Upload complete. (278ms)"7501worker2 # [ 199.480620] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=0 paths=1 nar_bytes_in=5127502lb # [ 199.452197] niks3-server[1092]: 2026/09/22 11:35:37 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7503worker2 # [ 199.485931] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449807504worker2 # [ 199.495651] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)7505worker2 # [ 199.504337] nix-daemon[867]: reaped child process 1968, status = succeeded7506worker2 # [ 199.509100] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)7507client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7508client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7509client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7510client # farm-a> LOG-farm-a-drain7511worker2 # [ 200.573419] nix-daemon[867]: reaped child process 1982, status = succeeded7512worker2 # [ 200.617347] nix-daemon[867]: accepted connection from pid 1998, user nix-grpc-daemon (trusted)7513worker2 # [ 200.630747] nix-daemon[867]: reaped child process 2001, status = succeeded7514lb # [ 200.605715] niks3-server[1092]: 2026/09/22 11:35:38 INFO Received uploads request method=POST path=/api/pending_closures7515worker2 # [ 200.646949] nix-grpc-daemon[1755]: time=2026-09-22T11:35:39.812Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7516worker2 # [ 200.648521] nix-grpc-daemon[1755]: time=2026-09-22T11:35:39.814Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7517lb # [ 200.647398] niks3-server[1092]: 2026/09/22 11:35:39 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7518lb # [ 200.657749] niks3-server[1092]: 2026/09/22 11:35:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign7519lb # [ 200.662233] niks3-server[1092]: 2026/09/22 11:35:39 INFO Signed narinfos id=81 count=17520worker2 # [ 200.697074] nix-grpc-daemon[1755]: time=2026-09-22T11:35:39.863Z level=INFO msg="Uploading 1 narinfos"7521lb # [ 200.668441] niks3-server[1092]: 2026/09/22 11:35:39 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7522lb # [ 200.674734] niks3-server[1092]: 2026/09/22 11:35:39 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7523lb # [ 200.681882] niks3-server[1092]: 2026/09/22 11:35:39 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete7524lb # [ 200.686056] niks3-server[1092]: 2026/09/22 11:35:39 INFO Completed upload id=817525worker2 # [ 200.718790] nix-grpc-daemon[1755]: time=2026-09-22T11:35:39.885Z level=INFO msg="Upload complete. (202ms)"7526lb # [ 200.687732] niks3-server[1092]: 2026/09/22 11:35:39 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7527worker2 # [ 200.722206] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=17528worker2 # [ 200.728160] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:39Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07529worker2 # [ 200.730140] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:39Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17530worker2 # [ 200.735760] nix-daemon[867]: reaped child process 1980, status = succeeded7531worker2 # [ 200.771342] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449807532worker2 # [ 200.777600] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:449807533worker2 # [ 200.783571] nix-grpc-daemon[1755]: time=2026-09-22T11:35:39.949Z level=INFO msg="All 1 paths already cached"7534client # copying 1 paths to 'lb:50051'...7535worker2 # [ 200.788861] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:449807536client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7537worker2 # [ 200.793954] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)7538worker2 # [ 200.877867] nix-daemon[867]: accepted connection from pid 2007, user nix-grpc-daemon (trusted)7539worker2 # [ 200.892437] nix-daemon[867]: reaped child process 2010, status = succeeded7540worker2 # [ 200.971956] nix-daemon[867]: accepted connection from pid 2012, user nix-grpc-daemon (trusted)7541worker2 # [ 200.986711] nix-daemon[867]: reaped child process 2015, status = succeeded7542lb # [ 200.959641] niks3-server[1092]: 2026/09/22 11:35:39 INFO Received uploads request method=POST path=/api/pending_closures7543worker2 # [ 200.996286] nix-grpc-daemon[1755]: time=2026-09-22T11:35:40.162Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7544worker2 # [ 200.997652] nix-grpc-daemon[1755]: time=2026-09-22T11:35:40.162Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7545lb # [ 200.973345] niks3-server[1092]: 2026/09/22 11:35:39 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7546lb # [ 200.979278] niks3-server[1092]: 2026/09/22 11:35:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign7547lb # [ 200.983327] niks3-server[1092]: 2026/09/22 11:35:39 INFO Signed narinfos id=82 count=17548worker2 # [ 201.017221] nix-grpc-daemon[1755]: time=2026-09-22T11:35:40.183Z level=INFO msg="Uploading 1 narinfos"7549lb # [ 200.990450] niks3-server[1092]: 2026/09/22 11:35:39 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7550lb # [ 200.994899] niks3-server[1092]: 2026/09/22 11:35:39 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete7551lb # [ 200.998840] niks3-server[1092]: 2026/09/22 11:35:39 INFO Completed upload id=827552worker2 # [ 201.030372] nix-grpc-daemon[1755]: time=2026-09-22T11:35:40.196Z level=INFO msg="Upload complete. (230ms)"7553worker2 # [ 201.032379] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=0 paths=1 nar_bytes_in=6247554lb # [ 201.002336] niks3-server[1092]: 2026/09/22 11:35:39 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7555worker2 # [ 201.036192] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449807556worker2 # [ 201.042678] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)7557worker2 # [ 201.045130] nix-daemon[867]: reaped child process 2005, status = succeeded7558worker2 # [ 201.050179] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)7559client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7560client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7561client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7562client # farm-b> LOG-farm-b-drain7563worker2 # [ 202.281982] nix-daemon[867]: reaped child process 2019, status = succeeded7564worker2 # [ 202.356751] nix-daemon[867]: accepted connection from pid 2035, user nix-grpc-daemon (trusted)7565worker2 # [ 202.371943] nix-daemon[867]: reaped child process 2038, status = succeeded7566lb # [ 202.344151] niks3-server[1092]: 2026/09/22 11:35:40 INFO Received uploads request method=POST path=/api/pending_closures7567worker2 # [ 202.380947] nix-grpc-daemon[1755]: time=2026-09-22T11:35:41.547Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7568worker2 # [ 202.384055] nix-grpc-daemon[1755]: time=2026-09-22T11:35:41.548Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7569lb # [ 202.367748] niks3-server[1092]: 2026/09/22 11:35:40 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7570lb # [ 202.373395] niks3-server[1092]: 2026/09/22 11:35:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign7571lb # [ 202.375362] niks3-server[1092]: 2026/09/22 11:35:40 INFO Signed narinfos id=83 count=17572worker2 # [ 202.407138] nix-grpc-daemon[1755]: time=2026-09-22T11:35:41.573Z level=INFO msg="Uploading 1 narinfos"7573lb # [ 202.378935] niks3-server[1092]: 2026/09/22 11:35:40 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7574lb # [ 202.384581] niks3-server[1092]: 2026/09/22 11:35:40 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete7575lb # [ 202.388219] niks3-server[1092]: 2026/09/22 11:35:40 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7576lb # [ 202.390992] niks3-server[1092]: 2026/09/22 11:35:40 INFO Completed upload id=837577worker2 # [ 202.422447] nix-grpc-daemon[1755]: time=2026-09-22T11:35:41.588Z level=INFO msg="Upload complete. (164ms)"7578lb # [ 202.393146] niks3-server[1092]: 2026/09/22 11:35:40 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7579worker2 # [ 202.425111] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=17580worker2 # [ 202.431828] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:41Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07581worker2 # [ 202.433813] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:41Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17582worker2 # [ 202.439102] nix-daemon[867]: reaped child process 2017, status = succeeded7583worker2 # [ 202.475184] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449807584worker2 # [ 202.483642] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:449807585worker2 # [ 202.492427] nix-grpc-daemon[1755]: time=2026-09-22T11:35:41.658Z level=INFO msg="All 2 paths already cached"7586client # copying 1 paths to 'lb:50051'...7587client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7588worker2 # [ 202.498501] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:449807589worker2 # [ 202.507127] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)7590worker2 # [ 202.587140] nix-daemon[867]: accepted connection from pid 2044, user nix-grpc-daemon (trusted)7591worker2 # [ 202.605829] nix-daemon[867]: reaped child process 2047, status = succeeded7592worker2 # [ 202.675438] nix-daemon[867]: accepted connection from pid 2049, user nix-grpc-daemon (trusted)7593worker2 # [ 202.689598] nix-daemon[867]: reaped child process 2052, status = succeeded7594lb # [ 202.662106] niks3-server[1092]: 2026/09/22 11:35:41 INFO Received uploads request method=POST path=/api/pending_closures7595worker2 # [ 202.697494] nix-grpc-daemon[1755]: time=2026-09-22T11:35:41.863Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7596worker2 # [ 202.699342] nix-grpc-daemon[1755]: time=2026-09-22T11:35:41.865Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7597lb # [ 202.676500] niks3-server[1092]: 2026/09/22 11:35:41 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7598lb # [ 202.680893] niks3-server[1092]: 2026/09/22 11:35:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign7599lb # [ 202.683506] niks3-server[1092]: 2026/09/22 11:35:41 INFO Signed narinfos id=84 count=17600worker2 # [ 202.716316] nix-grpc-daemon[1755]: time=2026-09-22T11:35:41.882Z level=INFO msg="Uploading 1 narinfos"7601lb # [ 202.686649] niks3-server[1092]: 2026/09/22 11:35:41 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7602lb # [ 202.691867] niks3-server[1092]: 2026/09/22 11:35:41 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete7603lb # [ 202.696284] niks3-server[1092]: 2026/09/22 11:35:41 INFO Completed upload id=847604worker2 # [ 202.727763] nix-grpc-daemon[1755]: time=2026-09-22T11:35:41.894Z level=INFO msg="Upload complete. (213ms)"7605lb # [ 202.698301] niks3-server[1092]: 2026/09/22 11:35:41 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7606worker2 # [ 202.729849] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=0 paths=1 nar_bytes_in=7527607worker2 # [ 202.734244] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:449807608worker2 # [ 202.739898] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)7609worker2 # [ 202.742270] nix-daemon[867]: reaped child process 2042, status = succeeded7610client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7611worker2 # [ 202.748085] nix-daemon[867]: accepted connection from pid 1663, user nix-grpc-daemon (trusted)7612client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7613client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7614client # farm-top> LOG-farm-top-drain7615worker2 # [ 204.186143] nix-daemon[867]: reaped child process 2056, status = succeeded7616worker2 # [ 204.269061] nix-daemon[867]: accepted connection from pid 2072, user nix-grpc-daemon (trusted)7617worker2 # [ 204.282882] nix-daemon[867]: reaped child process 2075, status = succeeded7618lb # [ 204.258595] niks3-server[1092]: 2026/09/22 11:35:42 INFO Received uploads request method=POST path=/api/pending_closures7619worker2 # [ 204.300188] nix-grpc-daemon[1755]: time=2026-09-22T11:35:43.466Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7620worker2 # [ 204.301713] nix-grpc-daemon[1755]: time=2026-09-22T11:35:43.467Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7621lb # [ 204.298147] niks3-server[1092]: 2026/09/22 11:35:42 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7622lb # [ 204.305346] niks3-server[1092]: 2026/09/22 11:35:42 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7623lb # [ 204.320861] niks3-server[1092]: 2026/09/22 11:35:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign7624lb # [ 204.325665] niks3-server[1092]: 2026/09/22 11:35:42 INFO Signed narinfos id=85 count=17625worker2 # [ 204.359698] nix-grpc-daemon[1755]: time=2026-09-22T11:35:43.525Z level=INFO msg="Uploading 1 narinfos"7626lb # [ 204.337740] niks3-server[1092]: 2026/09/22 11:35:42 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete7627lb # [ 204.339299] niks3-server[1092]: 2026/09/22 11:35:42 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7628lb # [ 204.343390] niks3-server[1092]: 2026/09/22 11:35:42 INFO Completed upload id=857629worker2 # [ 204.375184] nix-grpc-daemon[1755]: time=2026-09-22T11:35:43.541Z level=INFO msg="Upload complete. (219ms)"7630lb # [ 204.345578] niks3-server[1092]: 2026/09/22 11:35:42 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7631worker2 # [ 204.381965] nix-grpc-daemon[1663]: ts=2026-09-22T11:35:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44980 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=17632worker2 # [ 204.397272] nix-daemon[867]: reaped child process 2054, status = succeeded7633client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&2, in 5.46 seconds)7634client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath7635client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath, in 0.12 seconds)7636worker1: must succeed: rm /nix/.rw-store/fill7637worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.12 seconds)7638worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7639worker1 # [ 205.324221] nix-grpc-daemon[1810]: ts=2026-09-22T11:35:43Z level=info event=healthy reason=min_free7640worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.07 seconds)7641(finished: subtest: low disk drains a worker and builds go to the other, in 8.76 seconds)7642(finished: run the VM test script, in 206.67 seconds)7643test script finished in 206.85s7644cleanup7645kill QemuMachine (pid 49)7646client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7647client # [2026-09-22T11:35:45Z INFO virtiofsd] Client disconnected, shutting down7648client # [2026-09-22T11:35:45Z INFO virtiofsd] Client disconnected, shutting down7649client # [2026-09-22T11:35:45Z INFO virtiofsd] Client disconnected, shutting down7650kill QemuMachine (pid 50)7651lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7652lb # [2026-09-22T11:35:45Z INFO virtiofsd] Client disconnected, shutting down7653lb # [2026-09-22T11:35:45Z INFO virtiofsd] Client disconnected, shutting down7654lb # [2026-09-22T11:35:45Z INFO virtiofsd] Client disconnected, shutting down7655kill QemuMachine (pid 51)7656worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7657worker1 # [2026-09-22T11:35:45Z INFO virtiofsd] Client disconnected, shutting down7658worker1 # [2026-09-22T11:35:45Z INFO virtiofsd] Client disconnected, shutting down7659worker1 # [2026-09-22T11:35:45Z INFO virtiofsd] Client disconnected, shutting down7660kill QemuMachine (pid 52)7661worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7662worker2 # [2026-09-22T11:35:46Z INFO virtiofsd] Client disconnected, shutting down7663worker2 # [2026-09-22T11:35:46Z INFO virtiofsd] Client disconnected, shutting down7664worker2 # [2026-09-22T11:35:46Z INFO virtiofsd] Client disconnected, shutting down7665(finished: cleanup, in 1.09 seconds)7666additionally exposed symbols:7667 client, lb, worker1, worker2,7668 vlan1,7669 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh