vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #480
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker1 # Disk image does not exist, creating the virtualisation disk image...13worker1: QEMU running (pid 50)14worker1 # Formatting '/build/vm-state-worker1/tmp.XYgQzG8yZY', fmt=raw size=107374182415lb: QEMU running (pid 51)16worker1 # mke2fs 1.47.4 (6-Mar-2025)17worker2 # Disk image does not exist, creating the virtualisation disk image...18client # Disk image does not exist, creating the virtualisation disk image...19client: QEMU running (pid 49)20worker2: QEMU running (pid 52)21lb # Disk image does not exist, creating the virtualisation disk image...22worker1 # Discarding device blocks: 0/262144 done23client # Formatting '/build/vm-state-client/tmp.mgVH6d2ey5', fmt=raw size=107374182424worker2 # Formatting '/build/vm-state-worker2/tmp.89i8RE13U4', fmt=raw size=107374182425worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes26client # mke2fs 1.47.4 (6-Mar-2025)27worker2 # mke2fs 1.47.4 (6-Mar-2025)28worker1 # Filesystem UUID: 688df2ac-7044-48b9-9783-5d722ba4479029lb # Formatting '/build/vm-state-lb/tmp.jJh7ISGLai', fmt=raw size=107374182430(finished: start all VMs, in 0.24 seconds)31worker1 # Superblock backups stored on blocks:32worker1 # 32768, 98304, 163840, 22937633worker1 # 34client: waiting for unit network-addresses-eth1.service35worker2 # Discarding device blocks: 0/262144 done36client # Discarding device blocks: 0/262144 done37worker1 # Allocating group tables: 0/8 done38worker1 # Writing inode tables: 0/8 done39lb # mke2fs 1.47.4 (6-Mar-2025)40worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes41client: waiting for the VM to finish booting42lb # Discarding device blocks: 0/262144 done43worker2 # Filesystem UUID: 82997474-45fa-4f34-9f32-e5a34a68465a44client # Creating filesystem with 262144 4k blocks and 65536 inodes45worker1 # Creating journal (8192 blocks): done46lb # Creating filesystem with 262144 4k blocks and 65536 inodes47worker2 # Superblock backups stored on blocks:48client # Filesystem UUID: accde53a-de24-4fec-b7f0-5a65914d703149worker1 # Writing superblocks and filesystem accounting information: 0/8 done50lb # Filesystem UUID: 68373e2e-e441-4427-b5a7-8bf2eee61f3351worker2 # 32768, 98304, 163840, 22937652client # Superblock backups stored on blocks:53worker1 # 54lb # Superblock backups stored on blocks:55worker2 # 56client # 32768, 98304, 163840, 22937657worker1 # Virtualisation disk image created.58lb # 32768, 98304, 163840, 22937659worker2 # Allocating group tables: 0/8 done60client # 61worker1 # Starting virtiofs daemons...62lb # 63worker2 # Writing inode tables: 0/8 done64client # Allocating group tables: 0/8 done65worker1 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)66lb # Allocating group tables: 0/8 done67worker2 # Creating journal (8192 blocks): done68client # Writing inode tables: 0/8 done69worker1 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether70lb # Writing inode tables: 0/8 done71worker2 # Writing superblocks and filesystem accounting information: 0/8 done72client # Creating journal (8192 blocks): done73worker1 # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...74lb # Creating journal (8192 blocks): done75worker2 # 76client # Writing superblocks and filesystem accounting information: 0/8 done77worker1 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)78lb # Writing superblocks and filesystem accounting information: 0/8 done79worker2 # Virtualisation disk image created.80client # 81worker1 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether82lb # 83worker2 # Starting virtiofs daemons...84client # Virtualisation disk image created.85worker1 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)86lb # Virtualisation disk image created.87worker2 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)88client # Starting virtiofs daemons...89worker1 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether90lb # Starting virtiofs daemons...91worker2 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether92client # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)93worker1 # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...94lb # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)95worker2 # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...96client # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether97worker1 # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...98lb # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether99worker2 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)100client # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...101worker1 # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests102lb # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...103worker2 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether104client # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)105worker1 # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests106lb # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)107worker2 # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...108client # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether109worker1 # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests110lb # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether111worker2 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)112lb # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)113worker2 # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether114lb # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether115worker2 # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...116lb # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...117worker2 # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests118lb # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...119worker2 # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests120lb # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests121worker2 # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests122lb # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests123client # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...124lb # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests125client # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)126client # [2026-09-21T14:43:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether127client # [2026-09-21T14:43:11Z INFO virtiofsd] Waiting for vhost-user socket connection...128client # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests129client # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests130client # [2026-09-21T14:43:11Z INFO virtiofsd] Client connected, servicing requests131lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)132worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)133worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)134client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)135lb # 136lb # 137lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00138worker1 # 139worker2 # 140worker1 # 141worker2 # 142worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00143worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00144lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...145lb # 146lb # 147lb # 148lb # 149lb # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00150worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...151worker1 # 152worker1 # 153worker1 # 154worker1 # 155worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...156worker2 # 157worker2 # 158worker2 # 159worker2 # 160worker1 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00161worker2 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00162client # 163lb # Press Ctrl-B to configure iPXE (PCI 00:05.0)...164client # 165lb # 166lb # 167client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00168worker1 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...169worker1 # 170worker2 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...171worker1 # 172worker2 # 173worker2 # 174client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...175client # 176client # 177client # 178client # 179client # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00180client # Press Ctrl-B to configure iPXE (PCI 00:05.0)...181client # 182client # 183lb # Booting from ROM...184worker2 # Booting from ROM...185worker1 # Booting from ROM...186client # Booting from ROM...187lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026188worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026189worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026190lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ywx2cfw2bd45f1sp7w8ib468r0dscq27-nixos-system-lb-test/init regInfo=/nix/store/lhzhb7lk7gq3c4r6bxsdb6n35xs7cgxx-closure-info/registration console=ttyS0,115200n8 console=tty0191lb # [ 0.000000] BIOS-provided physical RAM map:192lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable193worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/34sn5q3k4s5dxk2cjrq9ddxmzik393xp-nixos-system-worker2-test/init regInfo=/nix/store/h9n6cylsxy742q82s34bdn6a6b8y5qpj-closure-info/registration console=ttyS0,115200n8 console=tty0194worker2 # [ 0.000000] BIOS-provided physical RAM map:195worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/31zfsnqjsdrlaxry041lgw659q8r9wr8-nixos-system-worker1-test/init regInfo=/nix/store/521hynx6hwn1lbh7ayndc850rf0kyhm0-closure-info/registration console=ttyS0,115200n8 console=tty0196lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved197worker1 # [ 0.000000] BIOS-provided physical RAM map:198worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable199lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved200worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable201worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved202lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable203worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved204worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved205lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved206worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved207worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable208lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved209worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable210worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved211lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved212worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved213worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved214lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved215worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved216worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved217lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved218worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved219worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved220lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved221worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved222client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026223lb # [ 0.000000] NX (Execute Disable) protection: active224worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved225lb # [ 0.000000] APIC: Static calls initialized226worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved227lb # [ 0.000000] SMBIOS 2.8 present.228worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved229worker2 # [ 0.000000] NX (Execute Disable) protection: active230worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved231worker2 # [ 0.000000] APIC: Static calls initialized232worker1 # [ 0.000000] NX (Execute Disable) protection: active233lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014234worker2 # [ 0.000000] SMBIOS 2.8 present.235worker1 # [ 0.000000] APIC: Static calls initialized236lb # [ 0.000000] DMI: Memory slots populated: 1/1237worker1 # [ 0.000000] SMBIOS 2.8 present.238lb # [ 0.000000] Hypervisor detected: KVM239worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014240client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/p6lhqb373376nh9xdab84lh6d9781g76-nixos-system-client-test/init regInfo=/nix/store/1aad4czwnalfgj7xv63axcwczg4d46d5-closure-info/registration console=ttyS0,115200n8 console=tty0241lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000242worker2 # [ 0.000000] DMI: Memory slots populated: 1/1243client # [ 0.000000] BIOS-provided physical RAM map:244worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014245lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00246worker2 # [ 0.000000] Hypervisor detected: KVM247worker1 # [ 0.000000] DMI: Memory slots populated: 1/1248client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable249worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000250worker1 # [ 0.000000] Hypervisor detected: KVM251lb # [ 0.000001] kvm-clock: using sched offset of 489856842 cycles252worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00253client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved254worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000255worker2 # [ 0.000000] kvm-clock: using sched offset of 470196928 cycles256lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns257worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00258lb # [ 0.000005] tsc: Detected 2400.012 MHz processor259client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved260worker1 # [ 0.000001] kvm-clock: using sched offset of 476851846 cycles261lb # [ 0.000815] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000262client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable263worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns264worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor265client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved266worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns267lb # [ 0.000842] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs268worker2 # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000269worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor270client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved271lb # [ 0.000845] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT272worker1 # [ 0.000810] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000273worker2 # [ 0.000839] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs274lb # [ 0.002738] found SMP MP-table at [mem 0x000f5450-0x000f545f]275client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved276lb # [ 0.002749] Using GB pages for direct mapping277worker2 # [ 0.000842] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT278worker1 # [ 0.000837] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs279client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved280lb # [ 0.002838] RAMDISK: [mem 0x5e36e000-0x5ffcffff]281worker2 # [ 0.002734] found SMP MP-table at [mem 0x000f5450-0x000f545f]282worker1 # [ 0.000840] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT283client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved284lb # [ 0.002846] ACPI: Early table checksum verification disabled285worker2 # [ 0.002745] Using GB pages for direct mapping286worker1 # [ 0.002732] found SMP MP-table at [mem 0x000f5450-0x000f545f]287worker2 # [ 0.002844] RAMDISK: [mem 0x5e36f000-0x5ffcffff]288lb # [ 0.002850] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )289worker1 # [ 0.002743] Using GB pages for direct mapping290client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved291client # [ 0.000000] NX (Execute Disable) protection: active292worker1 # [ 0.002877] RAMDISK: [mem 0x5e370000-0x5ffcffff]293worker2 # [ 0.002853] ACPI: Early table checksum verification disabled294lb # [ 0.002854] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)295client # [ 0.000000] APIC: Static calls initialized296worker2 # [ 0.002856] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )297worker1 # [ 0.002886] ACPI: Early table checksum verification disabled298client # [ 0.000000] SMBIOS 2.8 present.299worker1 # [ 0.002889] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )300lb # [ 0.002858] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)301worker2 # [ 0.002860] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)302client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014303worker1 # [ 0.002893] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)304lb # [ 0.002866] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)305client # [ 0.000000] DMI: Memory slots populated: 1/1306worker2 # [ 0.002864] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)307lb # [ 0.002868] ACPI: FACS 0x000000005FFE0000 000040308client # [ 0.000000] Hypervisor detected: KVM309worker1 # [ 0.002898] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)310worker2 # [ 0.002871] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)311client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000312lb # [ 0.002869] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)313worker2 # [ 0.002873] ACPI: FACS 0x000000005FFE0000 000040314client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00315worker1 # [ 0.002905] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)316worker1 # [ 0.002907] ACPI: FACS 0x000000005FFE0000 000040317lb # [ 0.002871] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)318client # [ 0.000000] kvm-clock: using sched offset of 482200680 cycles319worker2 # [ 0.002874] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)320lb # [ 0.002873] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)321worker1 # [ 0.002909] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)322worker2 # [ 0.002876] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)323client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns324client # [ 0.000005] tsc: Detected 2400.012 MHz processor325worker1 # [ 0.002910] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)326lb # [ 0.002874] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)327worker2 # [ 0.002877] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)328client # [ 0.000806] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000329lb # [ 0.002875] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]330worker1 # [ 0.002912] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)331worker2 # [ 0.002879] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)332client # [ 0.000833] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs333lb # [ 0.002877] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]334worker1 # [ 0.002913] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)335worker2 # [ 0.002880] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]336client # [ 0.000835] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT337lb # [ 0.002877] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]338worker1 # [ 0.002915] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]339worker2 # [ 0.002881] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]340client # [ 0.002724] found SMP MP-table at [mem 0x000f5450-0x000f545f]341lb # [ 0.002878] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]342client # [ 0.002735] Using GB pages for direct mapping343worker1 # [ 0.002916] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]344worker2 # [ 0.002882] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]345client # [ 0.002878] RAMDISK: [mem 0x5e370000-0x5ffcffff]346lb # [ 0.002878] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]347worker1 # [ 0.002916] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]348worker2 # [ 0.002882] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]349client # [ 0.002888] ACPI: Early table checksum verification disabled350lb # [ 0.002878] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]351worker1 # [ 0.002917] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]352worker2 # [ 0.002883] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]353client # [ 0.002891] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )354lb # [ 0.002879] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]355worker1 # [ 0.002917] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]356worker2 # [ 0.002883] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]357lb # [ 0.003114] No NUMA configuration found358client # [ 0.002895] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)359worker1 # [ 0.002917] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]360lb # [ 0.003115] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]361worker2 # [ 0.002884] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]362worker2 # [ 0.003111] No NUMA configuration found363client # [ 0.002899] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)364lb # [ 0.003117] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]365worker1 # [ 0.002918] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]366lb # [ 0.005537] Zone ranges:367worker1 # [ 0.003156] No NUMA configuration found368worker2 # [ 0.003112] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]369client # [ 0.002907] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)370lb # [ 0.005538] DMA [mem 0x0000000000001000-0x0000000000ffffff]371worker2 # [ 0.003115] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]372worker1 # [ 0.003156] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]373client # [ 0.002909] ACPI: FACS 0x000000005FFE0000 000040374worker2 # [ 0.005485] Zone ranges:375lb # [ 0.005540] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]376worker1 # [ 0.003159] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]377lb # [ 0.005541] Normal empty378client # [ 0.002910] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)379worker1 # [ 0.005970] Zone ranges:380worker2 # [ 0.005486] DMA [mem 0x0000000000001000-0x0000000000ffffff]381lb # [ 0.005542] Device empty382worker2 # [ 0.005487] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]383lb # [ 0.005542] Movable zone start for each node384worker1 # [ 0.005971] DMA [mem 0x0000000000001000-0x0000000000ffffff]385worker2 # [ 0.005489] Normal empty386lb # [ 0.005543] Early memory node ranges387client # [ 0.002912] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)388worker2 # [ 0.005490] Device empty389worker1 # [ 0.005973] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]390worker1 # [ 0.005974] Normal empty391lb # [ 0.005544] node 0: [mem 0x0000000000001000-0x000000000009efff]392worker2 # [ 0.005490] Movable zone start for each node393client # [ 0.002914] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)394worker1 # [ 0.005975] Device empty395worker2 # [ 0.005491] Early memory node ranges396lb # [ 0.005545] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]397worker1 # [ 0.005976] Movable zone start for each node398worker2 # [ 0.005491] node 0: [mem 0x0000000000001000-0x000000000009efff]399client # [ 0.002915] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)400worker1 # [ 0.005976] Early memory node ranges401lb # [ 0.005546] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]402worker2 # [ 0.005492] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]403worker1 # [ 0.005977] node 0: [mem 0x0000000000001000-0x000000000009efff]404client # [ 0.002916] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]405lb # [ 0.005570] On node 0, zone DMA: 1 pages in unavailable ranges406worker2 # [ 0.005493] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]407worker1 # [ 0.005978] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]408client # [ 0.002918] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]409lb # [ 0.005852] On node 0, zone DMA: 97 pages in unavailable ranges410worker2 # [ 0.005519] On node 0, zone DMA: 1 pages in unavailable ranges411worker1 # [ 0.005979] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]412client # [ 0.002918] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]413lb # [ 0.034002] On node 0, zone DMA32: 40 pages in unavailable ranges414lb # [ 0.034465] ACPI: PM-Timer IO Port: 0x608415worker2 # [ 0.005797] On node 0, zone DMA: 97 pages in unavailable ranges416worker1 # [ 0.005999] On node 0, zone DMA: 1 pages in unavailable ranges417client # [ 0.002919] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]418worker2 # [ 0.033069] On node 0, zone DMA32: 40 pages in unavailable ranges419lb # [ 0.034477] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])420worker1 # [ 0.006294] On node 0, zone DMA: 97 pages in unavailable ranges421client # [ 0.002919] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]422worker2 # [ 0.033539] ACPI: PM-Timer IO Port: 0x608423worker1 # [ 0.033433] On node 0, zone DMA32: 40 pages in unavailable ranges424lb # [ 0.034507] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23425worker2 # [ 0.033551] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])426client # [ 0.002920] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]427worker1 # [ 0.033888] ACPI: PM-Timer IO Port: 0x608428lb # [ 0.034510] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)429worker2 # [ 0.033582] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23430worker1 # [ 0.033900] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])431client # [ 0.002920] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]432lb # [ 0.034511] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)433client # [ 0.003161] No NUMA configuration found434worker2 # [ 0.033585] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)435worker1 # [ 0.033930] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23436lb # [ 0.034512] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)437worker2 # [ 0.033586] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)438worker1 # [ 0.033933] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)439client # [ 0.003163] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]440lb # [ 0.034514] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)441client # [ 0.003165] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]442worker2 # [ 0.033587] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)443worker1 # [ 0.033934] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)444client # [ 0.005591] Zone ranges:445lb # [ 0.034514] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)446worker2 # [ 0.033588] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)447worker1 # [ 0.033935] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)448client # [ 0.005593] DMA [mem 0x0000000000001000-0x0000000000ffffff]449lb # [ 0.034517] ACPI: Using ACPI (MADT) for SMP configuration information450worker1 # [ 0.033937] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)451client # [ 0.005594] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]452worker2 # [ 0.033589] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)453client # [ 0.005595] Normal empty454lb # [ 0.034518] ACPI: HPET id: 0x8086a201 base: 0xfed00000455worker2 # [ 0.033591] ACPI: Using ACPI (MADT) for SMP configuration information456lb # [ 0.034521] TSC deadline timer available457worker1 # [ 0.033937] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)458client # [ 0.005596] Device empty459client # [ 0.005597] Movable zone start for each node460lb # [ 0.034525] CPU topo: Max. logical packages: 1461worker2 # [ 0.033593] ACPI: HPET id: 0x8086a201 base: 0xfed00000462worker1 # [ 0.033940] ACPI: Using ACPI (MADT) for SMP configuration information463client # [ 0.005598] Early memory node ranges464worker2 # [ 0.033596] TSC deadline timer available465lb # [ 0.034526] CPU topo: Max. logical dies: 1466worker1 # [ 0.033941] ACPI: HPET id: 0x8086a201 base: 0xfed00000467client # [ 0.005598] node 0: [mem 0x0000000000001000-0x000000000009efff]468lb # [ 0.034526] CPU topo: Max. dies per package: 1469worker2 # [ 0.033600] CPU topo: Max. logical packages: 1470worker1 # [ 0.033944] TSC deadline timer available471worker2 # [ 0.033601] CPU topo: Max. logical dies: 1472lb # [ 0.034529] CPU topo: Max. threads per core: 1473worker1 # [ 0.033949] CPU topo: Max. logical packages: 1474client # [ 0.005599] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]475worker2 # [ 0.033601] CPU topo: Max. dies per package: 1476lb # [ 0.034530] CPU topo: Num. cores per package: 1477worker1 # [ 0.033949] CPU topo: Max. logical dies: 1478client # [ 0.005600] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]479worker2 # [ 0.033604] CPU topo: Max. threads per core: 1480lb # [ 0.034530] CPU topo: Num. threads per package: 1481worker1 # [ 0.033950] CPU topo: Max. dies per package: 1482worker2 # [ 0.033605] CPU topo: Num. cores per package: 1483client # [ 0.005621] On node 0, zone DMA: 1 pages in unavailable ranges484lb # [ 0.034531] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs485worker1 # [ 0.033953] CPU topo: Max. threads per core: 1486worker2 # [ 0.033605] CPU topo: Num. threads per package: 1487client # [ 0.005894] On node 0, zone DMA: 97 pages in unavailable ranges488worker1 # [ 0.033954] CPU topo: Num. cores per package: 1489lb # [ 0.034550] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()490worker2 # [ 0.033605] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs491worker1 # [ 0.033954] CPU topo: Num. threads per package: 1492client # [ 0.032414] On node 0, zone DMA32: 40 pages in unavailable ranges493client # [ 0.032884] ACPI: PM-Timer IO Port: 0x608494lb # [ 0.034580] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]495worker2 # [ 0.033623] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()496worker1 # [ 0.033954] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs497client # [ 0.032896] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])498lb # [ 0.034581] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]499worker1 # [ 0.033974] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()500worker2 # [ 0.033656] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]501client # [ 0.032926] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23502lb # [ 0.034583] [mem 0x60000000-0xafffffff] available for PCI devices503worker1 # [ 0.034006] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]504worker2 # [ 0.033657] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]505lb # [ 0.034584] Booting paravirtualized kernel on KVM506client # [ 0.032929] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)507worker2 # [ 0.033659] [mem 0x60000000-0xafffffff] available for PCI devices508worker1 # [ 0.034007] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]509client # [ 0.032931] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)510worker2 # [ 0.033660] Booting paravirtualized kernel on KVM511lb # [ 0.034587] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns512worker1 # [ 0.034009] [mem 0x60000000-0xafffffff] available for PCI devices513client # [ 0.032932] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)514worker1 # [ 0.034010] Booting paravirtualized kernel on KVM515lb # [ 0.039094] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1516worker2 # [ 0.033663] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns517client # [ 0.032933] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)518lb # [ 0.041757] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152519worker2 # [ 0.038159] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1520worker1 # [ 0.034013] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns521client # [ 0.032934] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)522lb # [ 0.041804] kvm-guest: PV spinlocks disabled, single CPU523worker2 # [ 0.040321] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152524worker1 # [ 0.038514] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1525client # [ 0.032936] ACPI: Using ACPI (MADT) for SMP configuration information526worker2 # [ 0.040369] kvm-guest: PV spinlocks disabled, single CPU527client # [ 0.032938] ACPI: HPET id: 0x8086a201 base: 0xfed00000528worker1 # [ 0.040669] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152529client # [ 0.032942] TSC deadline timer available530worker1 # [ 0.040716] kvm-guest: PV spinlocks disabled, single CPU531client # [ 0.032946] CPU topo: Max. logical packages: 1532client # [ 0.032947] CPU topo: Max. logical dies: 1533client # [ 0.032947] CPU topo: Max. dies per package: 1534lb # [ 0.041805] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ywx2cfw2bd45f1sp7w8ib468r0dscq27-nixos-system-lb-test/init regInfo=/nix/store/lhzhb7lk7gq3c4r6bxsdb6n35xs7cgxx-closure-info/registration console=ttyS0,115200n8 console=tty0535client # [ 0.032951] CPU topo: Max. threads per core: 1536client # [ 0.032951] CPU topo: Num. cores per package: 1537worker2 # [ 0.040370] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/34sn5q3k4s5dxk2cjrq9ddxmzik393xp-nixos-system-worker2-test/init regInfo=/nix/store/h9n6cylsxy742q82s34bdn6a6b8y5qpj-closure-info/registration console=ttyS0,115200n8 console=tty0538client # [ 0.032952] CPU topo: Num. threads per package: 1539lb # [ 0.041902] Unknown kernel command line parameters "regInfo=/nix/store/lhzhb7lk7gq3c4r6bxsdb6n35xs7cgxx-closure-info/registration", will be passed to user space.540worker1 # [ 0.040718] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/31zfsnqjsdrlaxry041lgw659q8r9wr8-nixos-system-worker1-test/init regInfo=/nix/store/521hynx6hwn1lbh7ayndc850rf0kyhm0-closure-info/registration console=ttyS0,115200n8 console=tty0541client # [ 0.032952] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs542lb # [ 0.041915] random: crng init done543worker2 # [ 0.040466] Unknown kernel command line parameters "regInfo=/nix/store/h9n6cylsxy742q82s34bdn6a6b8y5qpj-closure-info/registration", will be passed to user space.544client # [ 0.032977] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()545lb # [ 0.041915] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes546worker2 # [ 0.040478] random: crng init done547worker1 # [ 0.040814] Unknown kernel command line parameters "regInfo=/nix/store/521hynx6hwn1lbh7ayndc850rf0kyhm0-closure-info/registration", will be passed to user space.548client # [ 0.033008] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]549lb # [ 0.044154] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)550worker2 # [ 0.040479] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes551worker1 # [ 0.040826] random: crng init done552client # [ 0.033009] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]553lb # [ 0.045274] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)554worker1 # [ 0.040827] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes555worker2 # [ 0.042598] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)556lb # [ 0.045304] Fallback order for Node 0: 0557client # [ 0.033011] [mem 0x60000000-0xafffffff] available for PCI devices558worker1 # [ 0.042950] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)559worker2 # [ 0.043850] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)560client # [ 0.033012] Booting paravirtualized kernel on KVM561lb # [ 0.045308] Built 1 zonelists, mobility grouping on. Total pages: 393078562worker2 # [ 0.043882] Fallback order for Node 0: 0563lb # [ 0.045309] Policy zone: DMA32564worker1 # [ 0.044022] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)565worker2 # [ 0.043885] Built 1 zonelists, mobility grouping on. Total pages: 393078566worker1 # [ 0.044052] Fallback order for Node 0: 0567lb # [ 0.047891] mem auto-init: stack:all(zero), heap alloc:on, heap free:off568client # [ 0.033015] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns569worker2 # [ 0.043886] Policy zone: DMA32570worker1 # [ 0.044055] Built 1 zonelists, mobility grouping on. Total pages: 393078571lb # [ 0.051199] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1572client # [ 0.037499] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1573worker2 # [ 0.046368] mem auto-init: stack:all(zero), heap alloc:on, heap free:off574worker1 # [ 0.044056] Policy zone: DMA32575lb # [ 0.054698] allocated 3145728 bytes of page_ext576client # [ 0.039833] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152577worker2 # [ 0.049626] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1578worker1 # [ 0.046741] mem auto-init: stack:all(zero), heap alloc:on, heap free:off579lb # [ 0.065050] ftrace: allocating 48787 entries in 192 pages580client # [ 0.039880] kvm-guest: PV spinlocks disabled, single CPU581worker2 # [ 0.052955] allocated 3145728 bytes of page_ext582lb # [ 0.065052] ftrace: allocated 192 pages with 2 groups583worker1 # [ 0.049898] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1584worker2 # [ 0.063230] ftrace: allocating 48787 entries in 192 pages585lb # [ 0.065893] Dynamic Preempt: lazy586worker1 # [ 0.053224] allocated 3145728 bytes of page_ext587worker2 # [ 0.063232] ftrace: allocated 192 pages with 2 groups588lb # [ 0.066015] rcu: Preemptible hierarchical RCU implementation.589worker1 # [ 0.063567] ftrace: allocating 48787 entries in 192 pages590worker2 # [ 0.064067] Dynamic Preempt: lazy591lb # [ 0.066015] rcu: RCU event tracing is enabled.592worker1 # [ 0.063569] ftrace: allocated 192 pages with 2 groups593worker2 # [ 0.064230] rcu: Preemptible hierarchical RCU implementation.594worker1 # [ 0.064423] Dynamic Preempt: lazy595lb # [ 0.066016] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.596worker2 # [ 0.064231] rcu: RCU event tracing is enabled.597worker1 # [ 0.064579] rcu: Preemptible hierarchical RCU implementation.598lb # [ 0.066017] Trampoline variant of Tasks RCU enabled.599worker2 # [ 0.064232] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.600client # [ 0.039881] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/p6lhqb373376nh9xdab84lh6d9781g76-nixos-system-client-test/init regInfo=/nix/store/1aad4czwnalfgj7xv63axcwczg4d46d5-closure-info/registration console=ttyS0,115200n8 console=tty0601worker1 # [ 0.064579] rcu: RCU event tracing is enabled.602lb # [ 0.066018] Rude variant of Tasks RCU enabled.603worker2 # [ 0.064233] Trampoline variant of Tasks RCU enabled.604lb # [ 0.066018] Tracing variant of Tasks RCU enabled.605worker1 # [ 0.064580] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.606worker2 # [ 0.064234] Rude variant of Tasks RCU enabled.607worker1 # [ 0.064582] Trampoline variant of Tasks RCU enabled.608lb # [ 0.066019] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.609client # [ 0.039985] Unknown kernel command line parameters "regInfo=/nix/store/1aad4czwnalfgj7xv63axcwczg4d46d5-closure-info/registration", will be passed to user space.610worker2 # [ 0.064234] Tracing variant of Tasks RCU enabled.611worker1 # [ 0.064582] Rude variant of Tasks RCU enabled.612client # [ 0.039997] random: crng init done613lb # [ 0.066019] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1614worker1 # [ 0.064583] Tracing variant of Tasks RCU enabled.615worker2 # [ 0.064234] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.616client # [ 0.039998] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes617worker2 # [ 0.064235] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1618lb # [ 0.066036] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.619worker1 # [ 0.064583] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.620client # [ 0.042092] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)621worker1 # [ 0.064584] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1622worker2 # [ 0.064256] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.623lb # [ 0.066038] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.624client # [ 0.043131] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)625client # [ 0.043161] Fallback order for Node 0: 0626worker1 # [ 0.064606] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.627worker2 # [ 0.064258] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.628lb # [ 0.066038] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.629client # [ 0.043164] Built 1 zonelists, mobility grouping on. Total pages: 393078630lb # [ 0.070451] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16631client # [ 0.043165] Policy zone: DMA32632worker1 # [ 0.064607] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.633worker2 # [ 0.064259] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.634lb # [ 0.070728] rcu: srcu_init: Setting srcu_struct sizes based on contention.635worker2 # [ 0.068878] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16636client # [ 0.045601] mem auto-init: stack:all(zero), heap alloc:on, heap free:off637worker1 # [ 0.064608] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.638client # [ 0.048702] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1639worker2 # [ 0.069144] rcu: srcu_init: Setting srcu_struct sizes based on contention.640worker1 # [ 0.069062] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16641lb # [ 0.070734] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns642client # [ 0.052173] allocated 3145728 bytes of page_ext643worker1 # [ 0.069354] rcu: srcu_init: Setting srcu_struct sizes based on contention.644worker2 # [ 0.069150] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns645client # [ 0.062347] ftrace: allocating 48787 entries in 192 pages646lb # [ 0.070839] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)647client # [ 0.062349] ftrace: allocated 192 pages with 2 groups648lb # [ 0.074383] Console: colour VGA+ 80x25649worker1 # [ 0.069361] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns650client # [ 0.063194] Dynamic Preempt: lazy651worker2 # [ 0.069250] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)652lb # [ 0.074386] printk: legacy console [tty0] enabled653worker2 # [ 0.072818] Console: colour VGA+ 80x25654client # [ 0.063327] rcu: Preemptible hierarchical RCU implementation.655lb # [ 0.104018] printk: legacy console [ttyS0] enabled656worker1 # [ 0.069466] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)657worker2 # [ 0.072820] printk: legacy console [tty0] enabled658client # [ 0.063328] rcu: RCU event tracing is enabled.659worker1 # [ 0.073001] Console: colour VGA+ 80x25660lb # [ 0.199910] ACPI: Core revision 20250807661worker2 # [ 0.102231] printk: legacy console [ttyS0] enabled662client # [ 0.063329] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.663worker1 # [ 0.073004] printk: legacy console [tty0] enabled664worker2 # [ 0.197670] ACPI: Core revision 20250807665client # [ 0.063330] Trampoline variant of Tasks RCU enabled.666worker1 # [ 0.102473] printk: legacy console [ttyS0] enabled667client # [ 0.063331] Rude variant of Tasks RCU enabled.668lb # [ 0.200760] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns669worker1 # [ 0.198291] ACPI: Core revision 20250807670client # [ 0.063331] Tracing variant of Tasks RCU enabled.671worker2 # [ 0.198549] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns672lb # [ 0.202250] APIC: Switch to symmetric I/O mode setup673client # [ 0.063332] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.674lb # [ 0.203208] x2apic enabled675worker2 # [ 0.200051] APIC: Switch to symmetric I/O mode setup676worker1 # [ 0.199159] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns677client # [ 0.063332] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1678worker2 # [ 0.201017] x2apic enabled679lb # [ 0.203947] APIC: Switched APIC routing to: physical x2apic680worker1 # [ 0.200646] APIC: Switch to symmetric I/O mode setup681client # [ 0.063347] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.682worker1 # [ 0.201581] x2apic enabled683worker2 # [ 0.201794] APIC: Switched APIC routing to: physical x2apic684client # [ 0.063348] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.685worker1 # [ 0.202324] APIC: Switched APIC routing to: physical x2apic686lb # [ 0.205785] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1687client # [ 0.063349] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.688worker2 # [ 0.203625] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1689client # [ 0.067760] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16690lb # [ 0.206773] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns691worker1 # [ 0.204105] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1692client # [ 0.068031] rcu: srcu_init: Setting srcu_struct sizes based on contention.693worker2 # [ 0.204620] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns694lb # [ 0.208454] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)695worker1 # [ 0.205057] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns696client # [ 0.068037] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns697worker2 # [ 0.206266] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)698lb # [ 0.209541] x86/cpu: User Mode Instruction Prevention (UMIP) activated699client # [ 0.068145] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)700worker1 # [ 0.206669] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)701worker2 # [ 0.207352] x86/cpu: User Mode Instruction Prevention (UMIP) activated702client # [ 0.071637] Console: colour VGA+ 80x25703lb # [ 0.210583] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127704client # [ 0.071639] printk: legacy console [tty0] enabled705worker1 # [ 0.207755] x86/cpu: User Mode Instruction Prevention (UMIP) activated706worker2 # [ 0.208376] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127707client # [ 0.101112] printk: legacy console [ttyS0] enabled708lb # [ 0.212134] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0709worker1 # [ 0.208843] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127710client # [ 0.197437] ACPI: Core revision 20250807711worker2 # [ 0.209979] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0712worker1 # [ 0.209666] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0713lb # [ 0.212455] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto714client # [ 0.198318] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns715worker2 # [ 0.210266] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto716lb # [ 0.214451] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl717worker1 # [ 0.210670] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto718client # [ 0.199867] APIC: Switch to symmetric I/O mode setup719worker2 # [ 0.212262] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl720client # [ 0.200857] x2apic enabled721lb # [ 0.215452] Transient Scheduler Attacks: Vulnerable: No microcode722worker1 # [ 0.211666] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl723worker2 # [ 0.213263] Transient Scheduler Attacks: Vulnerable: No microcode724lb # [ 0.217164] Spectre V2 : Mitigation: Enhanced / Automatic IBRS725client # [ 0.201643] APIC: Switched APIC routing to: physical x2apic726worker1 # [ 0.212666] Transient Scheduler Attacks: Vulnerable: No microcode727worker2 # [ 0.214980] Spectre V2 : Mitigation: Enhanced / Automatic IBRS728lb # [ 0.217451] Speculative Return Stack Overflow: Mitigation: Safe RET729worker1 # [ 0.213666] Spectre V2 : Mitigation: Enhanced / Automatic IBRS730worker2 # [ 0.215262] Speculative Return Stack Overflow: Mitigation: Safe RET731client # [ 0.203716] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1732worker1 # [ 0.214666] Speculative Return Stack Overflow: Mitigation: Safe RET733client # [ 0.204692] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns734lb # [ 0.218451] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization735client # [ 0.206416] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)736lb # [ 0.220458] Spectre V2 : Enabling IBPB for BPF737worker2 # [ 0.216262] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization738worker1 # [ 0.215665] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization739client # [ 0.207501] x86/cpu: User Mode Instruction Prevention (UMIP) activated740worker2 # [ 0.218270] Spectre V2 : Enabling IBPB for BPF741worker1 # [ 0.217673] Spectre V2 : Enabling IBPB for BPF742lb # [ 0.221452] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier743client # [ 0.209566] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127744worker2 # [ 0.219263] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier745lb # [ 0.222451] active return thunk: srso_alias_return_thunk746worker1 # [ 0.218401] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier747client # [ 0.210413] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0748worker2 # [ 0.220263] active return thunk: srso_alias_return_thunk749worker1 # [ 0.219666] active return thunk: srso_alias_return_thunk750lb # [ 0.224093] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'751worker2 # [ 0.221283] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'752lb # [ 0.225328] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'753client # [ 0.211403] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto754worker1 # [ 0.220686] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'755worker2 # [ 0.222262] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'756lb # [ 0.226179] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'757worker1 # [ 0.221666] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'758client # [ 0.212412] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl759worker2 # [ 0.223982] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'760lb # [ 0.227180] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'761worker1 # [ 0.222665] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'762client # [ 0.214412] Transient Scheduler Attacks: Vulnerable: No microcode763worker2 # [ 0.224977] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'764lb # [ 0.228182] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'765worker1 # [ 0.223665] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'766client # [ 0.215412] Spectre V2 : Mitigation: Enhanced / Automatic IBRS767worker2 # [ 0.225262] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'768lb # [ 0.229175] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'769worker1 # [ 0.224666] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'770client # [ 0.216412] Speculative Return Stack Overflow: Mitigation: Safe RET771worker2 # [ 0.226262] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'772lb # [ 0.230216] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'773worker1 # [ 0.225666] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'774worker2 # [ 0.227262] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'775lb # [ 0.231451] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256776worker1 # [ 0.226666] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'777worker2 # [ 0.229262] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256778lb # [ 0.232451] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64779worker1 # [ 0.227666] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256780worker2 # [ 0.230192] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64781lb # [ 0.233451] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512782worker1 # [ 0.228666] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64783client # [ 0.217412] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization784worker2 # [ 0.230991] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512785lb # [ 0.234385] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024786client # [ 0.219426] Spectre V2 : Enabling IBPB for BPF787worker1 # [ 0.229666] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512788worker2 # [ 0.231971] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024789lb # [ 0.235159] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8790worker1 # [ 0.230665] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024791client # [ 0.220132] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier792worker2 # [ 0.232989] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8793worker1 # [ 0.231666] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8794client # [ 0.221412] active return thunk: srso_alias_return_thunk795lb # [ 0.236163] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.796worker2 # [ 0.233970] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.797client # [ 0.222427] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'798worker1 # [ 0.232666] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.799client # [ 0.223412] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'800client # [ 0.224412] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'801client # [ 0.225411] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'802client # [ 0.226412] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'803client # [ 0.227412] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'804client # [ 0.228412] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'805client # [ 0.229412] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256806client # [ 0.231160] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64807client # [ 0.232158] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512808client # [ 0.233155] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024809client # [ 0.234154] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8810client # [ 0.235157] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.811lb # [ 0.271286] Freeing SMP alternatives memory: 44K812worker2 # [ 0.268339] Freeing SMP alternatives memory: 44K813lb # [ 0.272042] pid_max: default: 32768 minimum: 301814worker2 # [ 0.269137] pid_max: default: 32768 minimum: 301815worker1 # [ 0.267944] Freeing SMP alternatives memory: 44K816lb # [ 0.272569] LSM: initializing lsm=capability,landlock,yama,bpf,ima817worker2 # [ 0.269962] LSM: initializing lsm=capability,landlock,yama,bpf,ima818worker1 # [ 0.268669] pid_max: default: 32768 minimum: 301819lb # [ 0.273560] landlock: Up and running.820worker2 # [ 0.271085] landlock: Up and running.821worker1 # [ 0.269508] LSM: initializing lsm=capability,landlock,yama,bpf,ima822lb # [ 0.274451] Yama: becoming mindful.823worker2 # [ 0.271703] Yama: becoming mindful.824worker1 # [ 0.270467] landlock: Up and running.825lb # [ 0.275595] LSM support for eBPF active826worker2 # [ 0.272474] LSM support for eBPF active827worker1 # [ 0.271124] Yama: becoming mindful.828worker1 # [ 0.271880] LSM support for eBPF active829lb # [ 0.276357] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)830worker2 # [ 0.273349] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)831worker1 # [ 0.272796] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)832lb # [ 0.277493] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)833worker2 # [ 0.274300] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)834worker1 # [ 0.273714] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)835worker2 # [ 0.276938] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)836lb # [ 0.280036] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)837worker1 # [ 0.276517] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)838worker2 # [ 0.277772] Performance Events: Fam17h+ core perfctr, AMD PMU driver.839lb # [ 0.281024] Performance Events: Fam17h+ core perfctr, AMD PMU driver.840worker2 # [ 0.278267] ... version: 2841lb # [ 0.281455] ... version: 2842worker1 # [ 0.277244] Performance Events: Fam17h+ core perfctr, AMD PMU driver.843worker2 # [ 0.278995] ... bit width: 48844lb # [ 0.282146] ... bit width: 48845worker1 # [ 0.277670] ... version: 2846worker2 # [ 0.279264] ... generic counters: 6847lb # [ 0.282453] ... generic counters: 6848worker1 # [ 0.278376] ... bit width: 48849worker2 # [ 0.280003] ... generic bitmap: 000000000000003f850client # [ 0.270253] Freeing SMP alternatives memory: 44K851lb # [ 0.283176] ... generic bitmap: 000000000000003f852worker1 # [ 0.278669] ... generic counters: 6853worker2 # [ 0.280313] ... fixed-purpose counters: 0854lb # [ 0.283459] ... fixed-purpose counters: 0855client # [ 0.270980] pid_max: default: 32768 minimum: 301856worker1 # [ 0.279353] ... generic bitmap: 000000000000003f857worker2 # [ 0.281033] ... fixed-purpose bitmap: 0000000000000000858worker1 # [ 0.279678] ... fixed-purpose counters: 0859lb # [ 0.284123] ... fixed-purpose bitmap: 0000000000000000860client # [ 0.271522] LSM: initializing lsm=capability,landlock,yama,bpf,ima861worker1 # [ 0.280356] ... fixed-purpose bitmap: 0000000000000000862worker2 # [ 0.281265] ... value mask: 0000ffffffffffff863lb # [ 0.284454] ... value mask: 0000ffffffffffff864client # [ 0.272526] landlock: Up and running.865client # [ 0.273413] Yama: becoming mindful.866worker1 # [ 0.280668] ... value mask: 0000ffffffffffff867worker2 # [ 0.282130] ... max period: 00007fffffffffff868lb # [ 0.285321] ... max period: 00007fffffffffff869client # [ 0.274480] LSM support for eBPF active870worker2 # [ 0.282928] ... global_ctrl mask: 000000000000003f871worker1 # [ 0.281528] ... max period: 00007fffffffffff872lb # [ 0.286134] ... global_ctrl mask: 000000000000003f873worker1 # [ 0.282322] ... global_ctrl mask: 000000000000003f874lb # [ 0.286568] signal: max sigframe size: 3376875worker2 # [ 0.283388] signal: max sigframe size: 3376876client # [ 0.275223] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)877worker1 # [ 0.282782] signal: max sigframe size: 3376878lb # [ 0.287324] rcu: Hierarchical SRCU implementation.879worker2 # [ 0.284170] rcu: Hierarchical SRCU implementation.880worker2 # [ 0.284827] rcu: Max phase no-delay instances is 400.881client # [ 0.276435] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)882lb # [ 0.288012] rcu: Max phase no-delay instances is 400.883worker1 # [ 0.283545] rcu: Hierarchical SRCU implementation.884worker1 # [ 0.284219] rcu: Max phase no-delay instances is 400.885client # [ 0.278765] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)886client # [ 0.279995] Performance Events: Fam17h+ core perfctr, AMD PMU driver.887client # [ 0.280421] ... version: 2888worker2 # [ 0.289810] smp: Bringing up secondary CPUs ...889lb # [ 0.293219] smp: Bringing up secondary CPUs ...890client # [ 0.281160] ... bit width: 48891worker2 # [ 0.290277] smp: Brought up 1 node, 1 CPU892worker1 # [ 0.289294] smp: Bringing up secondary CPUs ...893client # [ 0.281415] ... generic counters: 6894lb # [ 0.293468] smp: Brought up 1 node, 1 CPU895worker1 # [ 0.289682] smp: Brought up 1 node, 1 CPU896worker2 # [ 0.290936] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)897client # [ 0.282094] ... generic bitmap: 000000000000003f898lb # [ 0.294124] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)899worker1 # [ 0.290333] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)900client # [ 0.282415] ... fixed-purpose counters: 0901client # [ 0.283166] ... fixed-purpose bitmap: 0000000000000000902worker2 # [ 0.291453] Memory: 1454592K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109312K reserved, 0K cma-reserved)903lb # [ 0.294611] Memory: 1454624K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109316K reserved, 0K cma-reserved)904client # [ 0.283415] ... value mask: 0000ffffffffffff905worker2 # [ 0.292525] devtmpfs: initialized906lb # [ 0.295731] devtmpfs: initialized907worker1 # [ 0.291132] Memory: 1454600K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)908client # [ 0.284302] ... max period: 00007fffffffffff909worker2 # [ 0.293384] x86/mm: Memory block size: 128MB910lb # [ 0.296486] x86/mm: Memory block size: 128MB911worker1 # [ 0.291934] devtmpfs: initialized912client # [ 0.285048] ... global_ctrl mask: 000000000000003f913worker1 # [ 0.292799] x86/mm: Memory block size: 128MB914client # [ 0.285537] signal: max sigframe size: 3376915client # [ 0.286332] rcu: Hierarchical SRCU implementation.916worker2 # [ 0.295169] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)917lb # [ 0.298299] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)918client # [ 0.287015] rcu: Max phase no-delay instances is 400.919worker1 # [ 0.294672] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)920worker2 # [ 0.296197] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).921lb # [ 0.299358] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).922worker2 # [ 0.297341] pinctrl core: initialized pinctrl subsystem923lb # [ 0.300543] pinctrl core: initialized pinctrl subsystem924worker1 # [ 0.295696] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).925worker2 # [ 0.298439] PM: RTC time: 14:43:11, date: 2026-09-21926worker1 # [ 0.296750] pinctrl core: initialized pinctrl subsystem927lb # [ 0.301645] PM: RTC time: 14:43:11, date: 2026-09-21928worker1 # [ 0.297843] PM: RTC time: 14:43:11, date: 2026-09-21929client # [ 0.292157] smp: Bringing up secondary CPUs ...930client # [ 0.292429] smp: Brought up 1 node, 1 CPU931worker2 # [ 0.302005] NET: Registered PF_NETLINK/PF_ROUTE protocol family932lb # [ 0.305354] NET: Registered PF_NETLINK/PF_ROUTE protocol family933client # [ 0.293119] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)934worker1 # [ 0.301586] NET: Registered PF_NETLINK/PF_ROUTE protocol family935worker2 # [ 0.302792] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations936lb # [ 0.306699] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations937worker2 # [ 0.303291] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations938worker1 # [ 0.302879] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations939client # [ 0.293591] Memory: 1454632K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)940lb # [ 0.307478] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations941client # [ 0.294684] devtmpfs: initialized942worker1 # [ 0.303692] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations943worker2 # [ 0.304536] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations944lb # [ 0.308722] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations945client # [ 0.295508] x86/mm: Memory block size: 128MB946worker2 # [ 0.305274] audit: initializing netlink subsys (disabled)947lb # [ 0.309463] audit: initializing netlink subsys (disabled)948worker1 # [ 0.304939] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations949worker1 # [ 0.305679] audit: initializing netlink subsys (disabled)950worker2 # [ 0.306267] audit: type=2000 audit(1790001792.062:1): state=initialized audit_enabled=0 res=1951lb # [ 0.310673] thermal_sys: Registered thermal governor 'fair_share'952client # [ 0.297201] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)953worker1 # [ 0.306763] thermal_sys: Registered thermal governor 'fair_share'954worker2 # [ 0.307401] thermal_sys: Registered thermal governor 'fair_share'955lb # [ 0.310675] thermal_sys: Registered thermal governor 'bang_bang'956client # [ 0.298270] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).957worker1 # [ 0.306765] thermal_sys: Registered thermal governor 'bang_bang'958worker2 # [ 0.307403] thermal_sys: Registered thermal governor 'bang_bang'959lb # [ 0.311455] thermal_sys: Registered thermal governor 'step_wise'960client # [ 0.299495] pinctrl core: initialized pinctrl subsystem961worker1 # [ 0.307670] thermal_sys: Registered thermal governor 'step_wise'962lb # [ 0.312374] thermal_sys: Registered thermal governor 'user_space'963worker2 # [ 0.308266] thermal_sys: Registered thermal governor 'step_wise'964client # [ 0.300632] PM: RTC time: 14:43:11, date: 2026-09-21965worker1 # [ 0.308582] thermal_sys: Registered thermal governor 'user_space'966worker2 # [ 0.309196] thermal_sys: Registered thermal governor 'user_space'967lb # [ 0.313150] audit: type=2000 audit(1790001792.063:1): state=initialized audit_enabled=0 res=1968worker2 # [ 0.309998] thermal_sys: Registered thermal governor 'power_allocator'969worker1 # [ 0.309358] audit: type=2000 audit(1790001792.062:1): state=initialized audit_enabled=0 res=1970lb # [ 0.314456] thermal_sys: Registered thermal governor 'power_allocator'971worker2 # [ 0.310286] cpuidle: using governor menu972lb # [ 0.314469] cpuidle: using governor menu973worker1 # [ 0.310671] thermal_sys: Registered thermal governor 'power_allocator'974worker1 # [ 0.310686] cpuidle: using governor menu975worker2 # [ 0.312501] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5976client # [ 0.304195] NET: Registered PF_NETLINK/PF_ROUTE protocol family977lb # [ 0.316712] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5978worker1 # [ 0.312900] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5979worker2 # [ 0.313547] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]980client # [ 0.305652] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations981lb # [ 0.317721] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]982worker1 # [ 0.313891] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]983worker2 # [ 0.314268] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry984client # [ 0.306444] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations985lb # [ 0.318456] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry986worker2 # [ 0.315276] PCI: Using configuration type 1 for base access987worker1 # [ 0.314672] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry988lb # [ 0.319465] PCI: Using configuration type 1 for base access989client # [ 0.307689] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations990worker1 # [ 0.315680] PCI: Using configuration type 1 for base access991worker2 # [ 0.316352] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.992client # [ 0.308426] audit: initializing netlink subsys (disabled)993lb # [ 0.320593] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.994worker1 # [ 0.316775] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.995client # [ 0.309524] thermal_sys: Registered thermal governor 'fair_share'996client # [ 0.309526] thermal_sys: Registered thermal governor 'bang_bang'997client # [ 0.310416] thermal_sys: Registered thermal governor 'step_wise'998client # [ 0.311419] audit: type=2000 audit(1790001792.077:1): state=initialized audit_enabled=0 res=1999lb # [ 0.325694] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1000client # [ 0.313418] thermal_sys: Registered thermal governor 'user_space'1001lb # [ 0.326454] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1002client # [ 0.313420] thermal_sys: Registered thermal governor 'power_allocator'1003worker2 # [ 0.323279] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1004client # [ 0.314418] cpuidle: using governor menu1005worker2 # [ 0.324265] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1006worker1 # [ 0.323926] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1007client # [ 0.316642] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.51008worker1 # [ 0.324669] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1009client # [ 0.317655] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]1010lb # [ 0.331454] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1011client # [ 0.318418] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry1012lb # [ 0.332454] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1013client # [ 0.319427] PCI: Using configuration type 1 for base access1014worker2 # [ 0.329266] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1015worker2 # [ 0.330265] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1016client # [ 0.320541] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.1017worker1 # [ 0.329669] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1018worker1 # [ 0.330669] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1019client # [ 0.325687] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1020client # [ 0.326416] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1021lb # [ 0.342826] ACPI: Added _OSI(Module Device)1022lb # [ 0.343455] ACPI: Added _OSI(Processor Device)1023worker1 # [ 0.339279] ACPI: Added _OSI(Module Device)1024worker1 # [ 0.339670] ACPI: Added _OSI(Processor Device)1025lb # [ 0.344244] ACPI: Added _OSI(Processor Aggregator Device)1026worker2 # [ 0.341505] ACPI: Added _OSI(Module Device)1027worker1 # [ 0.340415] ACPI: Added _OSI(Processor Aggregator Device)1028worker2 # [ 0.342202] ACPI: Added _OSI(Processor Device)1029client # [ 0.331416] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1030worker2 # [ 0.342815] ACPI: Added _OSI(Processor Aggregator Device)1031client # [ 0.332415] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1032worker1 # [ 0.347184] ACPI: 1 ACPI AML tables successfully acquired and loaded1033worker2 # [ 0.348615] ACPI: 1 ACPI AML tables successfully acquired and loaded1034lb # [ 0.350446] ACPI: 1 ACPI AML tables successfully acquired and loaded1035worker1 # [ 0.350627] ACPI: Interpreter enabled1036client # [ 0.342825] ACPI: Added _OSI(Module Device)1037worker2 # [ 0.352669] ACPI: Interpreter enabled1038worker1 # [ 0.351169] ACPI: PM: (supports S0 S3 S4 S5)1039lb # [ 0.356863] ACPI: Interpreter enabled1040worker2 # [ 0.353281] ACPI: PM: (supports S0 S3 S4 S5)1041client # [ 0.343417] ACPI: Added _OSI(Processor Device)1042worker1 # [ 0.351670] ACPI: Using IOAPIC for interrupt routing1043lb # [ 0.357470] ACPI: PM: (supports S0 S3 S4 S5)1044worker2 # [ 0.353984] ACPI: Using IOAPIC for interrupt routing1045client # [ 0.344144] ACPI: Added _OSI(Processor Aggregator Device)1046lb # [ 0.358186] ACPI: Using IOAPIC for interrupt routing1047worker1 # [ 0.352505] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1048worker2 # [ 0.356326] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1049lb # [ 0.360507] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1050worker2 # [ 0.357266] PCI: Using E820 reservations for host bridge windows1051lb # [ 0.361454] PCI: Using E820 reservations for host bridge windows1052worker1 # [ 0.357669] PCI: Using E820 reservations for host bridge windows1053worker1 # [ 0.358803] ACPI: Enabled 2 GPEs in block 00 to 3F1054worker2 # [ 0.360380] ACPI: Enabled 2 GPEs in block 00 to 3F1055lb # [ 0.364590] ACPI: Enabled 2 GPEs in block 00 to 3F1056client # [ 0.352974] ACPI: 1 ACPI AML tables successfully acquired and loaded1057client # [ 0.356836] ACPI: Interpreter enabled1058client # [ 0.357431] ACPI: PM: (supports S0 S3 S4 S5)1059worker2 # [ 0.367804] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1060client # [ 0.358168] ACPI: Using IOAPIC for interrupt routing1061worker1 # [ 0.367513] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1062lb # [ 0.372121] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1063worker2 # [ 0.368271] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1064client # [ 0.358469] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1065lb # [ 0.372459] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1066client # [ 0.359415] PCI: Using E820 reservations for host bridge windows1067worker1 # [ 0.368866] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1068worker2 # [ 0.369343] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1069lb # [ 0.373532] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1070worker1 # [ 0.369746] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1071worker2 # [ 0.370394] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1072lb # [ 0.374577] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1073worker1 # [ 0.370801] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1074worker2 # [ 0.371661] PCI host bridge to bus 0000:001075client # [ 0.362462] ACPI: Enabled 2 GPEs in block 00 to 3F1076lb # [ 0.375817] PCI host bridge to bus 0000:001077worker1 # [ 0.372030] PCI host bridge to bus 0000:001078worker2 # [ 0.372269] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1079lb # [ 0.376459] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1080worker1 # [ 0.372673] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1081worker2 # [ 0.373265] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1082lb # [ 0.377454] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1083worker1 # [ 0.373669] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1084worker2 # [ 0.374265] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1085lb # [ 0.378453] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1086worker1 # [ 0.374668] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1087worker2 # [ 0.375265] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1088worker1 # [ 0.375668] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1089lb # [ 0.379454] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1090worker2 # [ 0.376265] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1091worker1 # [ 0.376668] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1092lb # [ 0.380453] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1093worker2 # [ 0.377265] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1094worker2 # [ 0.378265] pci_bus 0000:00: root bus resource [bus 00-ff]1095worker1 # [ 0.377674] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1096lb # [ 0.381454] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1097lb # [ 0.382454] pci_bus 0000:00: root bus resource [bus 00-ff]1098worker1 # [ 0.378668] pci_bus 0000:00: root bus resource [bus 00-ff]1099client # [ 0.371041] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1100worker2 # [ 0.379276] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1101worker1 # [ 0.379670] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1102lb # [ 0.383467] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1103client # [ 0.371422] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1104worker2 # [ 0.380728] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1105worker1 # [ 0.381122] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1106client # [ 0.372713] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1107lb # [ 0.384904] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1108client # [ 0.373539] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1109client # [ 0.374866] PCI host bridge to bus 0000:001110client # [ 0.375420] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1111client # [ 0.376415] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1112client # [ 0.377415] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1113client # [ 0.378415] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1114worker2 # [ 0.383409] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1115client # [ 0.379415] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1116worker1 # [ 0.383737] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1117worker2 # [ 0.384305] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1118lb # [ 0.387540] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1119client # [ 0.380415] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1120worker1 # [ 0.384680] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1121lb # [ 0.388536] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1122worker2 # [ 0.385215] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1123client # [ 0.381416] pci_bus 0000:00: root bus resource [bus 00-ff]1124worker1 # [ 0.385595] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1125lb # [ 0.389477] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1126worker2 # [ 0.386509] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1127client # [ 0.382354] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1128worker1 # [ 0.386966] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1129lb # [ 0.390813] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1130client # [ 0.383856] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1131worker2 # [ 0.388178] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1132worker1 # [ 0.388602] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1133lb # [ 0.392172] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1134worker2 # [ 0.390302] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1135worker1 # [ 0.391686] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1136worker2 # [ 0.391218] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1137lb # [ 0.395488] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1138worker1 # [ 0.392533] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1139worker2 # [ 0.392010] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1140lb # [ 0.396361] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1141worker1 # [ 0.393416] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1142lb # [ 0.397205] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1143worker2 # [ 0.392271] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1144worker1 # [ 0.393675] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1145lb # [ 0.397460] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1146worker2 # [ 0.393865] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1147client # [ 0.386579] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1148worker1 # [ 0.395219] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1149lb # [ 0.399037] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1150client # [ 0.387428] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1151client # [ 0.388414] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1152worker2 # [ 0.395273] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1153worker1 # [ 0.396686] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1154worker2 # [ 0.396228] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1155lb # [ 0.401313] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1156client # [ 0.389745] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1157worker1 # [ 0.397576] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1158worker2 # [ 0.397024] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1159lb # [ 0.402066] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1160worker1 # [ 0.398390] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1161lb # [ 0.402476] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1162client # [ 0.391158] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1163worker2 # [ 0.397802] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1164worker1 # [ 0.399234] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1165lb # [ 0.404038] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1166client # [ 0.393454] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1167client # [ 0.394344] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1168worker2 # [ 0.400304] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1169worker1 # [ 0.401688] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1170client # [ 0.395166] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1171worker2 # [ 0.401188] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1172lb # [ 0.406486] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1173worker1 # [ 0.402538] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1174lb # [ 0.407365] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1175client # [ 0.395421] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1176worker2 # [ 0.401998] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1177worker1 # [ 0.403383] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1178lb # [ 0.408189] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1179worker2 # [ 0.402813] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1180client # [ 0.396964] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1181worker1 # [ 0.404258] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1182lb # [ 0.409032] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1183client # [ 0.398423] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1184worker2 # [ 0.404302] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1185worker1 # [ 0.405686] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1186client # [ 0.399252] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1187worker2 # [ 0.405194] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1188lb # [ 0.410483] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1189worker1 # [ 0.406523] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1190client # [ 0.400155] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1191lb # [ 0.411320] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1192worker2 # [ 0.406000] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1193worker1 # [ 0.407370] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1194worker2 # [ 0.406272] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1195client # [ 0.401006] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1196worker1 # [ 0.407676] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1197lb # [ 0.412189] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1198lb # [ 0.412462] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1199worker2 # [ 0.407841] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1200worker1 # [ 0.409168] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1201lb # [ 0.413997] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1202client # [ 0.403456] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1203worker2 # [ 0.409307] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1204worker1 # [ 0.410693] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1205client # [ 0.404317] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1206worker2 # [ 0.410275] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1207worker1 # [ 0.411635] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1208lb # [ 0.415497] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1209client # [ 0.405179] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1210lb # [ 0.416459] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1211worker2 # [ 0.411822] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1212worker1 # [ 0.413052] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1213client # [ 0.405968] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1214lb # [ 0.418025] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1215worker1 # [ 0.414691] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1216worker2 # [ 0.413306] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1217client # [ 0.408212] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1218worker1 # [ 0.415654] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1219worker2 # [ 0.414288] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1220client # [ 0.409099] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1221lb # [ 0.419466] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1222lb # [ 0.421212] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1223worker1 # [ 0.417093] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1224worker2 # [ 0.415831] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1225client # [ 0.409437] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1226client # [ 0.410422] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1227lb # [ 0.422038] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1228worker1 # [ 0.418681] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1229worker2 # [ 0.417307] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1230worker1 # [ 0.419612] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1231worker2 # [ 0.418270] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1232client # [ 0.411966] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1233lb # [ 0.423466] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1234worker1 # [ 0.421050] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1235lb # [ 0.424459] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1236worker2 # [ 0.419807] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1237client # [ 0.413452] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1238client # [ 0.414437] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1239worker1 # [ 0.422681] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1240lb # [ 0.426032] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1241worker2 # [ 0.421310] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1242client # [ 0.415980] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1243worker1 # [ 0.423645] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1244worker2 # [ 0.422287] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1245lb # [ 0.427467] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1246worker1 # [ 0.425204] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1247lb # [ 0.428476] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1248worker2 # [ 0.423811] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1249client # [ 0.417457] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1250lb # [ 0.429991] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1251client # [ 0.418438] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1252worker1 # [ 0.426689] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1253worker2 # [ 0.425945] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1254worker1 # [ 0.427522] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1255worker2 # [ 0.426273] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1256client # [ 0.420174] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1257worker1 # [ 0.428437] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1258lb # [ 0.431489] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1259worker2 # [ 0.427257] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1260lb # [ 0.432460] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1261worker1 # [ 0.430045] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1262client # [ 0.422427] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1263lb # [ 0.433453] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1264client # [ 0.423408] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1265worker2 # [ 0.428717] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1266lb # [ 0.434998] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1267client # [ 0.424957] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1268worker1 # [ 0.431688] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1269worker2 # [ 0.430301] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1270worker1 # [ 0.433322] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1271worker2 # [ 0.431142] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1272lb # [ 0.436498] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1273client # [ 0.427142] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1274worker1 # [ 0.433690] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1275worker2 # [ 0.432006] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1276lb # [ 0.437356] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1277client # [ 0.427437] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1278lb # [ 0.438192] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1279worker1 # [ 0.435250] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1280worker2 # [ 0.432854] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1281client # [ 0.428949] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1282worker1 # [ 0.436383] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1283lb # [ 0.439025] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1284worker2 # [ 0.433951] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1285worker1 # [ 0.436896] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1286lb # [ 0.440169] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1287worker2 # [ 0.434473] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1288client # [ 0.430433] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1289client # [ 0.431239] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1290worker1 # [ 0.438347] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1291lb # [ 0.440649] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1292client # [ 0.432194] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1293worker2 # [ 0.436614] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1294worker1 # [ 0.438875] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1295lb # [ 0.442482] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1296worker2 # [ 0.437487] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1297client # [ 0.433885] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1298worker1 # [ 0.440351] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1299lb # [ 0.443555] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1300worker2 # [ 0.438962] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1301lb # [ 0.445117] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1302client # [ 0.435423] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1303client # [ 0.436279] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1304client # [ 0.437241] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1305client # [ 0.438941] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1306client # [ 0.440244] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1307client # [ 0.441376] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1308client # [ 0.443143] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1309worker1 # [ 0.440883] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1310client # [ 0.443610] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1311worker2 # [ 0.439478] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1312worker1 # [ 0.443290] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1313lb # [ 0.445667] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1314client # [ 0.445053] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1315worker2 # [ 0.441912] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1316worker1 # [ 0.443943] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1317lb # [ 0.448171] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1318worker2 # [ 0.442534] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1319worker1 # [ 0.444963] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1320lb # [ 0.448770] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1321worker2 # [ 0.443705] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1322worker1 # [ 0.445926] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1323lb # [ 0.449751] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1324worker2 # [ 0.444515] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1325worker1 # [ 0.447696] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1326worker1 # [ 0.448559] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1327lb # [ 0.450709] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1328worker2 # [ 0.446281] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1329worker2 # [ 0.447198] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1330worker1 # [ 0.449729] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1331lb # [ 0.452482] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1332lb # [ 0.453361] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1333worker2 # [ 0.448358] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1334worker1 # [ 0.451507] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1335lb # [ 0.454537] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1336worker2 # [ 0.449879] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1337lb # [ 0.456099] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1338client # [ 0.445631] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1339worker1 # [ 0.455495] ACPI: PCI: Interrupt link LNKA configured for IRQ 101340client # [ 0.448089] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1341worker1 # [ 0.456564] ACPI: PCI: Interrupt link LNKB configured for IRQ 101342worker2 # [ 0.453662] ACPI: PCI: Interrupt link LNKA configured for IRQ 101343worker1 # [ 0.457541] ACPI: PCI: Interrupt link LNKC configured for IRQ 111344client # [ 0.448707] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1345worker2 # [ 0.454385] ACPI: PCI: Interrupt link LNKB configured for IRQ 101346lb # [ 0.460147] ACPI: PCI: Interrupt link LNKA configured for IRQ 101347worker1 # [ 0.458510] ACPI: PCI: Interrupt link LNKD configured for IRQ 111348worker2 # [ 0.455377] ACPI: PCI: Interrupt link LNKC configured for IRQ 111349client # [ 0.449706] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1350lb # [ 0.460569] ACPI: PCI: Interrupt link LNKB configured for IRQ 101351worker1 # [ 0.459488] ACPI: PCI: Interrupt link LNKE configured for IRQ 101352worker2 # [ 0.456366] ACPI: PCI: Interrupt link LNKD configured for IRQ 111353worker1 # [ 0.460475] ACPI: PCI: Interrupt link LNKF configured for IRQ 101354lb # [ 0.461555] ACPI: PCI: Interrupt link LNKC configured for IRQ 111355client # [ 0.450707] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1356worker2 # [ 0.457360] ACPI: PCI: Interrupt link LNKE configured for IRQ 101357worker1 # [ 0.461481] ACPI: PCI: Interrupt link LNKG configured for IRQ 111358lb # [ 0.462550] ACPI: PCI: Interrupt link LNKD configured for IRQ 111359worker2 # [ 0.458362] ACPI: PCI: Interrupt link LNKF configured for IRQ 101360client # [ 0.452408] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1361worker1 # [ 0.462495] ACPI: PCI: Interrupt link LNKH configured for IRQ 111362lb # [ 0.463550] ACPI: PCI: Interrupt link LNKE configured for IRQ 101363worker2 # [ 0.459363] ACPI: PCI: Interrupt link LNKG configured for IRQ 111364client # [ 0.453255] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1365worker1 # [ 0.463406] ACPI: PCI: Interrupt link GSIA configured for IRQ 161366lb # [ 0.464549] ACPI: PCI: Interrupt link LNKF configured for IRQ 101367worker1 # [ 0.463680] ACPI: PCI: Interrupt link GSIB configured for IRQ 171368worker2 # [ 0.460362] ACPI: PCI: Interrupt link LNKH configured for IRQ 111369client # [ 0.454441] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1370worker2 # [ 0.461303] ACPI: PCI: Interrupt link GSIA configured for IRQ 161371worker1 # [ 0.464684] ACPI: PCI: Interrupt link GSIC configured for IRQ 181372lb # [ 0.465555] ACPI: PCI: Interrupt link LNKG configured for IRQ 111373worker2 # [ 0.462285] ACPI: PCI: Interrupt link GSIB configured for IRQ 171374worker1 # [ 0.465682] ACPI: PCI: Interrupt link GSID configured for IRQ 191375client # [ 0.456055] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1376lb # [ 0.466559] ACPI: PCI: Interrupt link LNKH configured for IRQ 111377worker2 # [ 0.463280] ACPI: PCI: Interrupt link GSIC configured for IRQ 181378worker1 # [ 0.466647] ACPI: PCI: Interrupt link GSIE configured for IRQ 201379lb # [ 0.467492] ACPI: PCI: Interrupt link GSIA configured for IRQ 161380worker1 # [ 0.467405] ACPI: PCI: Interrupt link GSIF configured for IRQ 211381worker2 # [ 0.464279] ACPI: PCI: Interrupt link GSID configured for IRQ 191382lb # [ 0.468468] ACPI: PCI: Interrupt link GSIB configured for IRQ 171383worker1 # [ 0.467685] ACPI: PCI: Interrupt link GSIG configured for IRQ 221384worker2 # [ 0.465253] ACPI: PCI: Interrupt link GSIE configured for IRQ 201385lb # [ 0.469425] ACPI: PCI: Interrupt link GSIC configured for IRQ 181386worker1 # [ 0.468636] ACPI: PCI: Interrupt link GSIH configured for IRQ 231387worker2 # [ 0.466020] ACPI: PCI: Interrupt link GSIF configured for IRQ 211388lb # [ 0.470180] ACPI: PCI: Interrupt link GSID configured for IRQ 191389client # [ 0.460079] ACPI: PCI: Interrupt link LNKA configured for IRQ 101390worker2 # [ 0.466277] ACPI: PCI: Interrupt link GSIG configured for IRQ 221391lb # [ 0.470465] ACPI: PCI: Interrupt link GSIE configured for IRQ 201392client # [ 0.460543] ACPI: PCI: Interrupt link LNKB configured for IRQ 101393worker1 # [ 0.470550] iommu: Default domain type: Translated1394worker2 # [ 0.467257] ACPI: PCI: Interrupt link GSIH configured for IRQ 231395lb # [ 0.471429] ACPI: PCI: Interrupt link GSIF configured for IRQ 211396client # [ 0.461519] ACPI: PCI: Interrupt link LNKC configured for IRQ 111397worker1 # [ 0.471250] iommu: DMA domain TLB invalidation policy: lazy mode1398lb # [ 0.472241] ACPI: PCI: Interrupt link GSIG configured for IRQ 221399worker1 # [ 0.471942] ACPI: bus type USB registered1400client # [ 0.462520] ACPI: PCI: Interrupt link LNKD configured for IRQ 111401worker2 # [ 0.469304] iommu: Default domain type: Translated1402lb # [ 0.473232] ACPI: PCI: Interrupt link GSIH configured for IRQ 231403worker1 # [ 0.472676] usbcore: registered new interface driver usbfs1404worker2 # [ 0.470102] iommu: DMA domain TLB invalidation policy: lazy mode1405client # [ 0.463515] ACPI: PCI: Interrupt link LNKE configured for IRQ 101406worker1 # [ 0.473554] usbcore: registered new interface driver hub1407worker2 # [ 0.471274] ACPI: bus type USB registered1408client # [ 0.464513] ACPI: PCI: Interrupt link LNKF configured for IRQ 101409lb # [ 0.474490] iommu: Default domain type: Translated1410worker1 # [ 0.474333] usbcore: registered new device driver usb1411worker2 # [ 0.472032] usbcore: registered new interface driver usbfs1412lb # [ 0.475285] iommu: DMA domain TLB invalidation policy: lazy mode1413client # [ 0.465516] ACPI: PCI: Interrupt link LNKG configured for IRQ 111414worker2 # [ 0.472289] usbcore: registered new interface driver hub1415lb # [ 0.476498] ACPI: bus type USB registered1416worker1 # [ 0.475523] NetLabel: Initializing1417client # [ 0.466522] ACPI: PCI: Interrupt link LNKH configured for IRQ 111418worker2 # [ 0.473183] usbcore: registered new device driver usb1419worker1 # [ 0.476085] NetLabel: domain hash size = 1281420lb # [ 0.477256] usbcore: registered new interface driver usbfs1421client # [ 0.467458] ACPI: PCI: Interrupt link GSIA configured for IRQ 161422worker1 # [ 0.476668] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1423lb # [ 0.478145] usbcore: registered new interface driver hub1424worker2 # [ 0.474798] NetLabel: Initializing1425worker1 # [ 0.477637] NetLabel: unlabeled traffic allowed by default1426client # [ 0.468431] ACPI: PCI: Interrupt link GSIB configured for IRQ 171427lb # [ 0.478473] usbcore: registered new device driver usb1428worker2 # [ 0.475266] NetLabel: domain hash size = 1281429worker1 # [ 0.478334] PCI: Using ACPI for IRQ routing1430client # [ 0.469392] ACPI: PCI: Interrupt link GSIC configured for IRQ 181431worker2 # [ 0.476007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1432lb # [ 0.480244] NetLabel: Initializing1433client # [ 0.470182] ACPI: PCI: Interrupt link GSID configured for IRQ 191434worker2 # [ 0.476303] NetLabel: unlabeled traffic allowed by default1435lb # [ 0.480460] NetLabel: domain hash size = 1281436worker2 # [ 0.477222] PCI: Using ACPI for IRQ routing1437client # [ 0.470428] ACPI: PCI: Interrupt link GSIE configured for IRQ 201438lb # [ 0.481220] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1439client # [ 0.471427] ACPI: PCI: Interrupt link GSIF configured for IRQ 211440lb # [ 0.481492] NetLabel: unlabeled traffic allowed by default1441client # [ 0.472426] ACPI: PCI: Interrupt link GSIG configured for IRQ 221442lb # [ 0.482466] PCI: Using ACPI for IRQ routing1443client # [ 0.473433] ACPI: PCI: Interrupt link GSIH configured for IRQ 231444client # [ 0.475554] iommu: Default domain type: Translated1445client # [ 0.476382] iommu: DMA domain TLB invalidation policy: lazy mode1446client # [ 0.477510] ACPI: bus type USB registered1447client # [ 0.478274] usbcore: registered new interface driver usbfs1448client # [ 0.479133] usbcore: registered new interface driver hub1449client # [ 0.479425] usbcore: registered new device driver usb1450client # [ 0.481196] NetLabel: Initializing1451client # [ 0.481421] NetLabel: domain hash size = 1281452client # [ 0.482171] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1453client # [ 0.482457] NetLabel: unlabeled traffic allowed by default1454client # [ 0.483383] PCI: Using ACPI for IRQ routing1455worker1 # [ 0.521638] pci 0000:00:01.0: vgaarb: setting as boot VGA device1456worker1 # [ 0.521664] pci 0000:00:01.0: vgaarb: bridge control possible1457worker1 # [ 0.521664] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1458worker2 # [ 0.521104] pci 0000:00:01.0: vgaarb: setting as boot VGA device1459worker1 # [ 0.521673] vgaarb: loaded1460worker2 # [ 0.521261] pci 0000:00:01.0: vgaarb: bridge control possible1461worker1 # [ 0.522441] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01462worker2 # [ 0.521261] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1463worker1 # [ 0.522669] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1464lb # [ 0.526660] pci 0000:00:01.0: vgaarb: setting as boot VGA device1465worker2 # [ 0.521270] vgaarb: loaded1466lb # [ 0.527450] pci 0000:00:01.0: vgaarb: bridge control possible1467worker2 # [ 0.522015] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01468worker2 # [ 0.522265] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1469lb # [ 0.527450] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1470lb # [ 0.527457] vgaarb: loaded1471lb # [ 0.528252] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01472worker1 # [ 0.526757] clocksource: Switched to clocksource kvm-clock1473lb # [ 0.529116] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1474worker2 # [ 0.525350] clocksource: Switched to clocksource kvm-clock1475worker1 # [ 0.530270] VFS: Disk quotas dquot_6.6.01476worker1 # [ 0.531018] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1477worker2 # [ 0.528974] VFS: Disk quotas dquot_6.6.01478lb # [ 0.533553] clocksource: Switched to clocksource kvm-clock1479worker1 # [ 0.532413] pnp: PnP ACPI init1480worker2 # [ 0.529667] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1481worker1 # [ 0.533253] ACPI: IRQ 4 override to edge(!), high(!)1482worker2 # [ 0.531043] pnp: PnP ACPI init1483worker2 # [ 0.531847] ACPI: IRQ 4 override to edge(!), high(!)1484worker1 # [ 0.534238] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1485client # [ 0.528484] pci 0000:00:01.0: vgaarb: setting as boot VGA device1486lb # [ 0.537301] VFS: Disk quotas dquot_6.6.01487worker1 # [ 0.535775] pnp: PnP ACPI: found 5 devices1488worker2 # [ 0.532790] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1489client # [ 0.529411] pci 0000:00:01.0: vgaarb: bridge control possible1490lb # [ 0.537982] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1491worker2 # [ 0.534281] pnp: PnP ACPI: found 5 devices1492client # [ 0.529411] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1493lb # [ 0.539294] pnp: PnP ACPI init1494client # [ 0.529419] vgaarb: loaded1495lb # [ 0.540085] ACPI: IRQ 4 override to edge(!), high(!)1496client # [ 0.530209] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01497lb # [ 0.541091] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1498client # [ 0.531066] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1499lb # [ 0.542661] pnp: PnP ACPI: found 5 devices1500worker1 # [ 0.543346] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1501client # [ 0.534503] clocksource: Switched to clocksource kvm-clock1502worker1 # [ 0.544811] clocksource: Switched to clocksource acpi_pm1503worker2 # [ 0.541855] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1504client # [ 0.536206] VFS: Disk quotas dquot_6.6.01505worker1 # [ 0.545792] NET: Registered PF_INET protocol family1506worker2 # [ 0.543304] clocksource: Switched to clocksource acpi_pm1507worker2 # [ 0.544260] NET: Registered PF_INET protocol family1508worker1 # [ 0.547132] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1509worker2 # [ 0.545467] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1510client # [ 0.538718] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1511client # [ 0.540049] pnp: PnP ACPI init1512lb # [ 0.550354] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1513lb # [ 0.551886] clocksource: Switched to clocksource acpi_pm1514client # [ 0.540926] ACPI: IRQ 4 override to edge(!), high(!)1515lb # [ 0.552921] NET: Registered PF_INET protocol family1516client # [ 0.541914] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1517lb # [ 0.554130] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1518client # [ 0.543498] pnp: PnP ACPI: found 5 devices1519client # [ 0.551437] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1520client # [ 0.552983] clocksource: Switched to clocksource acpi_pm1521client # [ 0.554113] NET: Registered PF_INET protocol family1522worker1 # [ 0.563870] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1523client # [ 0.555440] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1524worker2 # [ 0.562487] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1525worker1 # [ 0.565444] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1526worker1 # [ 0.566701] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1527worker2 # [ 0.563981] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1528worker1 # [ 0.568052] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1529worker2 # [ 0.565291] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1530worker1 # [ 0.569299] TCP: Hash tables configured (established 16384 bind 16384)1531worker2 # [ 0.566633] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1532lb # [ 0.570848] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1533worker1 # [ 0.570408] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1534worker2 # [ 0.567857] TCP: Hash tables configured (established 16384 bind 16384)1535lb # [ 0.572382] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1536worker1 # [ 0.571660] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1537worker2 # [ 0.568962] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1538lb # [ 0.573742] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1539worker1 # [ 0.572737] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1540worker2 # [ 0.570231] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1541lb # [ 0.575083] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1542worker1 # [ 0.573917] NET: Registered PF_UNIX/PF_LOCAL protocol family1543worker2 # [ 0.571288] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1544worker1 # [ 0.574865] NET: Registered PF_XDP protocol family1545lb # [ 0.576323] TCP: Hash tables configured (established 16384 bind 16384)1546worker2 # [ 0.572466] NET: Registered PF_UNIX/PF_LOCAL protocol family1547worker1 # [ 0.575707] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1548worker2 # [ 0.573418] NET: Registered PF_XDP protocol family1549lb # [ 0.577442] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1550worker1 # [ 0.576710] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1551worker2 # [ 0.574258] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1552lb # [ 0.578782] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1553worker1 # [ 0.577691] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1554worker2 # [ 0.575252] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1555lb # [ 0.579884] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1556worker1 # [ 0.578789] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1557worker2 # [ 0.576263] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1558lb # [ 0.581055] NET: Registered PF_UNIX/PF_LOCAL protocol family1559worker1 # [ 0.579910] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1560worker2 # [ 0.577375] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1561lb # [ 0.582014] NET: Registered PF_XDP protocol family1562worker1 # [ 0.580996] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1563lb # [ 0.582821] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1564worker2 # [ 0.578476] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1565lb # [ 0.583825] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1566worker1 # [ 0.582665] ACPI: \_SB_.GSIA: Enabled at IRQ 161567worker2 # [ 0.579575] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1568client # [ 0.572601] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1569lb # [ 0.584820] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1570worker2 # [ 0.581252] ACPI: \_SB_.GSIA: Enabled at IRQ 161571client # [ 0.573975] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1572lb # [ 0.585936] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1573worker1 # [ 0.585021] ACPI: \_SB_.GSIB: Enabled at IRQ 171574client # [ 0.575304] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1575lb # [ 0.587009] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1576worker2 # [ 0.583481] ACPI: \_SB_.GSIB: Enabled at IRQ 171577client # [ 0.576679] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1578lb # [ 0.588114] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1579worker1 # [ 0.587442] ACPI: \_SB_.GSIC: Enabled at IRQ 181580client # [ 0.577914] TCP: Hash tables configured (established 16384 bind 16384)1581lb # [ 0.589791] ACPI: \_SB_.GSIA: Enabled at IRQ 161582worker2 # [ 0.585763] ACPI: \_SB_.GSIC: Enabled at IRQ 181583client # [ 0.579068] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1584worker1 # [ 0.589630] ACPI: \_SB_.GSID: Enabled at IRQ 191585client # [ 0.580431] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1586worker2 # [ 0.587733] ACPI: \_SB_.GSID: Enabled at IRQ 191587lb # [ 0.592181] ACPI: \_SB_.GSIB: Enabled at IRQ 171588worker1 # [ 0.591408] PCI: CLS 0 bytes, default 641589client # [ 0.581521] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1590worker2 # [ 0.589434] PCI: CLS 0 bytes, default 641591worker1 # [ 0.592434] Trying to unpack rootfs image as initramfs...1592client # [ 0.582773] NET: Registered PF_UNIX/PF_LOCAL protocol family1593lb # [ 0.594851] ACPI: \_SB_.GSIC: Enabled at IRQ 181594client # [ 0.583744] NET: Registered PF_XDP protocol family1595worker2 # [ 0.590434] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1596client # [ 0.584611] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1597worker2 # [ 0.592154] Trying to unpack rootfs image as initramfs...1598lb # [ 0.596841] ACPI: \_SB_.GSID: Enabled at IRQ 191599client # [ 0.585637] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1600client # [ 0.586673] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1601lb # [ 0.598594] PCI: CLS 0 bytes, default 641602worker1 # [ 0.596652] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1603client # [ 0.587803] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1604client # [ 0.588916] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1605lb # [ 0.599592] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1606lb # [ 0.601389] Trying to unpack rootfs image as initramfs...1607client # [ 0.590029] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1608client # [ 0.591851] ACPI: \_SB_.GSIA: Enabled at IRQ 161609client # [ 0.594645] ACPI: \_SB_.GSIB: Enabled at IRQ 171610client # [ 0.596969] ACPI: \_SB_.GSIC: Enabled at IRQ 181611client # [ 0.599003] ACPI: \_SB_.GSID: Enabled at IRQ 191612client # [ 0.600796] PCI: CLS 0 bytes, default 641613client # [ 0.601868] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1614client # [ 0.603686] Trying to unpack rootfs image as initramfs...1615worker2 # [ 0.635309] Initialise system trusted keyrings1616worker1 # [ 0.638234] Initialise system trusted keyrings1617worker2 # [ 0.638190] workingset: timestamp_bits=40 max_order=19 bucket_order=01618worker1 # [ 0.641386] workingset: timestamp_bits=40 max_order=19 bucket_order=01619lb # [ 0.645672] Initialise system trusted keyrings1620lb # [ 0.648804] workingset: timestamp_bits=40 max_order=19 bucket_order=01621client # [ 0.648868] Initialise system trusted keyrings1622client # [ 0.651463] workingset: timestamp_bits=40 max_order=19 bucket_order=01623worker2 # [ 0.659409] Key type asymmetric registered1624worker1 # [ 0.662586] Key type asymmetric registered1625worker2 # [ 0.660163] Asymmetric key parser 'x509' registered1626worker1 # [ 0.663288] Asymmetric key parser 'x509' registered1627worker2 # [ 0.660991] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1628worker1 # [ 0.664151] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1629lb # [ 0.669054] Key type asymmetric registered1630worker2 # [ 0.666288] io scheduler mq-deadline registered1631worker1 # [ 0.669441] io scheduler mq-deadline registered1632worker2 # [ 0.667057] io scheduler kyber registered1633worker1 # [ 0.670215] io scheduler kyber registered1634lb # [ 0.671749] Asymmetric key parser 'x509' registered1635lb # [ 0.672623] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1636worker2 # [ 0.669769] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1637worker1 # [ 0.672957] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1638worker2 # [ 0.671063] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1639worker1 # [ 0.674260] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1640lb # [ 0.677913] io scheduler mq-deadline registered1641lb # [ 0.678767] io scheduler kyber registered1642worker2 # [ 0.676537] Linux agpgart interface v0.1031643worker1 # [ 0.679943] Linux agpgart interface v0.1031644worker2 # [ 0.677322] ACPI: bus type drm_connector registered1645worker1 # [ 0.680747] ACPI: bus type drm_connector registered1646lb # [ 0.682323] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1647lb # [ 0.683655] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1648client # [ 0.673452] Key type asymmetric registered1649worker2 # [ 0.680585] usbcore: registered new interface driver usbserial_generic1650client # [ 0.674190] Asymmetric key parser 'x509' registered1651worker1 # [ 0.683774] usbcore: registered new interface driver usbserial_generic1652worker2 # [ 0.681618] usbserial: USB Serial support registered for generic1653worker1 # [ 0.684847] usbserial: USB Serial support registered for generic1654client # [ 0.675026] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1655lb # [ 0.689020] Linux agpgart interface v0.1031656worker2 # [ 0.684153] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1657worker2 # [ 0.684153] Please enable it if your BIOS has the CPPC option.1658lb # [ 0.689825] ACPI: bus type drm_connector registered1659worker1 # [ 0.688348] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1660worker2 # [ 0.686394] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1661worker1 # [ 0.688348] Please enable it if your BIOS has the CPPC option.1662worker1 # [ 0.690664] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1663client # [ 0.681526] io scheduler mq-deadline registered1664client # [ 0.682304] io scheduler kyber registered1665lb # [ 0.693140] usbcore: registered new interface driver usbserial_generic1666lb # [ 0.694242] usbserial: USB Serial support registered for generic1667worker2 # [ 0.690280] drop_monitor: Initializing network drop monitor service1668worker2 # [ 0.691448] NET: Registered PF_INET6 protocol family1669worker1 # [ 0.694474] drop_monitor: Initializing network drop monitor service1670worker1 # [ 0.695662] NET: Registered PF_INET6 protocol family1671lb # [ 0.696746] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1672client # [ 0.686059] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1673lb # [ 0.696746] Please enable it if your BIOS has the CPPC option.1674worker2 # [ 0.694590] Segment Routing with IPv61675client # [ 0.687316] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1676lb # [ 0.698946] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1677worker1 # [ 0.698795] Segment Routing with IPv61678worker1 # [ 0.699549] In-situ OAM (IOAM) with IPv61679worker2 # [ 0.697184] In-situ OAM (IOAM) with IPv61680client # [ 0.691266] Linux agpgart interface v0.1031681worker2 # [ 0.698255] IPI shorthand broadcast: enabled1682worker1 # [ 0.701629] IPI shorthand broadcast: enabled1683client # [ 0.692023] ACPI: bus type drm_connector registered1684lb # [ 0.703890] drop_monitor: Initializing network drop monitor service1685lb # [ 0.705073] NET: Registered PF_INET6 protocol family1686client # [ 0.694854] usbcore: registered new interface driver usbserial_generic1687client # [ 0.696004] usbserial: USB Serial support registered for generic1688lb # [ 0.708209] Segment Routing with IPv61689lb # [ 0.708912] In-situ OAM (IOAM) with IPv61690lb # [ 0.711023] IPI shorthand broadcast: enabled1691worker1 # [ 0.709906] sched_clock: Marking stable (576015509, 133303823)->(785506162, -76186830)1692client # [ 0.700412] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1693worker2 # [ 0.707617] sched_clock: Marking stable (574015804, 133111265)->(786462298, -79335229)1694client # [ 0.700412] Please enable it if your BIOS has the CPPC option.1695client # [ 0.702656] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1696worker2 # [ 0.710220] registered taskstats version 11697worker2 # [ 0.711196] Loading compiled-in X.509 certificates1698worker1 # [ 0.714481] registered taskstats version 11699worker1 # [ 0.715443] Loading compiled-in X.509 certificates1700client # [ 0.705591] drop_monitor: Initializing network drop monitor service1701client # [ 0.706785] NET: Registered PF_INET6 protocol family1702lb # [ 0.719229] sched_clock: Marking stable (585014440, 133715891)->(796915275, -78184944)1703client # [ 0.710898] Segment Routing with IPv61704client # [ 0.713426] In-situ OAM (IOAM) with IPv61705lb # [ 0.724865] registered taskstats version 11706client # [ 0.714516] IPI shorthand broadcast: enabled1707lb # [ 0.725835] Loading compiled-in X.509 certificates1708client # [ 0.722626] sched_clock: Marking stable (589013112, 133374138)->(796132879, -73745629)1709worker1 # [ 0.733338] Demotion targets for Node 0: null1710worker2 # [ 0.731134] Demotion targets for Node 0: null1711worker2 # [ 0.732018] Key type .fscrypt registered1712worker1 # [ 0.736364] Key type .fscrypt registered1713client # [ 0.726667] registered taskstats version 11714worker1 # [ 0.737017] Key type fscrypt-provisioning registered1715client # [ 0.727654] Loading compiled-in X.509 certificates1716worker2 # [ 0.735140] Key type fscrypt-provisioning registered1717worker1 # [ 0.737995] ima: No TPM chip found, activating TPM-bypass!1718worker2 # [ 0.736082] ima: No TPM chip found, activating TPM-bypass!1719worker2 # [ 0.737050] ima: Allocated hash algorithm: sha11720worker1 # [ 0.740337] ima: Allocated hash algorithm: sha11721worker1 # [ 0.741124] ima: No architecture policies found1722lb # [ 0.742742] Demotion targets for Node 0: null1723worker2 # [ 0.739158] ima: No architecture policies found1724lb # [ 0.743640] Key type .fscrypt registered1725worker1 # [ 0.743332] PM: Magic number: 10:765:7381726worker2 # [ 0.741145] PM: Magic number: 10:765:7381727lb # [ 0.745741] Key type fscrypt-provisioning registered1728worker1 # [ 0.745006] RAS: Correctable Errors collector initialized.1729lb # [ 0.746700] ima: No TPM chip found, activating TPM-bypass!1730worker2 # [ 0.742807] RAS: Correctable Errors collector initialized.1731lb # [ 0.747637] ima: Allocated hash algorithm: sha11732lb # [ 0.750762] ima: No architecture policies found1733lb # [ 0.753751] PM: Magic number: 10:765:7381734lb # [ 0.755419] RAS: Correctable Errors collector initialized.1735worker2 # [ 0.751807] clk: Disabling unused clocks1736worker1 # [ 0.756036] clk: Disabling unused clocks1737client # [ 0.746403] Demotion targets for Node 0: null1738worker1 # [ 0.758345] PM: genpd: Disabling unused power domains1739worker2 # [ 0.756162] PM: genpd: Disabling unused power domains1740client # [ 0.749454] Key type .fscrypt registered1741client # [ 0.750139] Key type fscrypt-provisioning registered1742client # [ 0.751120] ima: No TPM chip found, activating TPM-bypass!1743lb # [ 0.764403] clk: Disabling unused clocks1744client # [ 0.753405] ima: Allocated hash algorithm: sha11745client # [ 0.754202] ima: No architecture policies found1746client # [ 0.756409] PM: Magic number: 10:765:7381747lb # [ 0.768746] PM: genpd: Disabling unused power domains1748client # [ 0.758154] RAS: Correctable Errors collector initialized.1749client # [ 0.769171] clk: Disabling unused clocks1750client # [ 0.771402] PM: genpd: Disabling unused power domains1751worker2 # [ 0.919882] Freeing initrd memory: 29060K1752worker1 # [ 0.922918] Freeing initrd memory: 29056K1753worker2 # [ 0.923214] Freeing unused decrypted memory: 2028K1754worker1 # [ 0.926239] Freeing unused decrypted memory: 2028K1755lb # [ 0.928865] Freeing initrd memory: 29064K1756worker2 # [ 0.925938] Freeing unused kernel image (initmem) memory: 3652K1757worker1 # [ 0.928994] Freeing unused kernel image (initmem) memory: 3652K1758worker2 # [ 0.927036] Write protecting the kernel read-only data: 32768k1759worker1 # [ 0.930105] Write protecting the kernel read-only data: 32768k1760lb # [ 0.932151] Freeing unused decrypted memory: 2028K1761worker2 # [ 0.928945] Freeing unused kernel image (text/rodata gap) memory: 1184K1762worker1 # [ 0.932040] Freeing unused kernel image (text/rodata gap) memory: 1184K1763worker2 # [ 0.930502] Freeing unused kernel image (rodata/data gap) memory: 720K1764lb # [ 0.934876] Freeing unused kernel image (initmem) memory: 3652K1765worker1 # [ 0.933590] Freeing unused kernel image (rodata/data gap) memory: 720K1766lb # [ 0.935967] Write protecting the kernel read-only data: 32768k1767lb # [ 0.937935] Freeing unused kernel image (text/rodata gap) memory: 1184K1768lb # [ 0.939467] Freeing unused kernel image (rodata/data gap) memory: 720K1769client # [ 0.934700] Freeing initrd memory: 29056K1770client # [ 0.938190] Freeing unused decrypted memory: 2028K1771client # [ 0.941133] Freeing unused kernel image (initmem) memory: 3652K1772client # [ 0.942268] Write protecting the kernel read-only data: 32768k1773client # [ 0.944288] Freeing unused kernel image (text/rodata gap) memory: 1184K1774client # [ 0.945920] Freeing unused kernel image (rodata/data gap) memory: 720K1775worker2 # [ 0.981139] x86/mm: Checked W+X mappings: passed, no W+X pages found.1776worker1 # [ 0.984523] x86/mm: Checked W+X mappings: passed, no W+X pages found.1777worker2 # [ 0.982232] Run /init as init process1778worker1 # [ 0.985621] Run /init as init process1779lb # [ 0.990384] x86/mm: Checked W+X mappings: passed, no W+X pages found.1780lb # [ 0.991531] Run /init as init process1781worker2 # [ 0.993296] systemd[1]: Inserted module 'autofs4'1782worker1 # [ 0.996587] systemd[1]: Inserted module 'autofs4'1783lb # [ 1.002361] systemd[1]: Inserted module 'autofs4'1784client # [ 0.996939] x86/mm: Checked W+X mappings: passed, no W+X pages found.1785client # [ 0.998033] Run /init as init process1786worker2 # [ 1.013220] fuse: init (API version 7.45)1787worker1 # [ 1.016484] fuse: init (API version 7.45)1788client # [ 1.009602] systemd[1]: Inserted module 'autofs4'1789lb # [ 1.022355] fuse: init (API version 7.45)1790worker2 # [ 1.020632] ACPI: \_SB_.GSIG: Enabled at IRQ 221791worker1 # [ 1.023895] ACPI: \_SB_.GSIG: Enabled at IRQ 221792worker2 # [ 1.023035] ACPI: \_SB_.GSIH: Enabled at IRQ 231793worker1 # [ 1.026352] ACPI: \_SB_.GSIH: Enabled at IRQ 231794lb # [ 1.029884] ACPI: \_SB_.GSIG: Enabled at IRQ 221795worker2 # [ 1.026094] ACPI: \_SB_.GSIE: Enabled at IRQ 201796worker1 # [ 1.029517] ACPI: \_SB_.GSIE: Enabled at IRQ 201797lb # [ 1.032385] ACPI: \_SB_.GSIH: Enabled at IRQ 231798worker2 # [ 1.028142] ACPI: \_SB_.GSIF: Enabled at IRQ 211799worker1 # [ 1.032195] ACPI: \_SB_.GSIF: Enabled at IRQ 211800lb # [ 1.035514] ACPI: \_SB_.GSIE: Enabled at IRQ 201801lb # [ 1.037571] ACPI: \_SB_.GSIF: Enabled at IRQ 211802worker2 # [ 1.033288] virtiofs virtio5: discovered new tag: nix-store1803worker2 # [ 1.034811] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1804worker1 # [ 1.038351] virtiofs virtio5: discovered new tag: nix-store1805client # [ 1.029860] fuse: init (API version 7.45)1806worker1 # [ 1.039988] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1807lb # [ 1.042865] virtiofs virtio5: discovered new tag: nix-store1808lb # [ 1.044460] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1809worker2 # [ 1.041431] virtiofs virtio6: discovered new tag: shared1810worker2 # [ 1.042942] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1811client # [ 1.037456] ACPI: \_SB_.GSIG: Enabled at IRQ 221812worker1 # [ 1.047124] virtiofs virtio6: discovered new tag: shared1813worker1 # [ 1.048784] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1814worker2 # [ 1.046387] virtiofs virtio7: discovered new tag: xchg1815lb # [ 1.051137] virtiofs virtio6: discovered new tag: shared1816client # [ 1.040061] ACPI: \_SB_.GSIH: Enabled at IRQ 231817worker2 # [ 1.047879] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1818lb # [ 1.052573] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1819worker1 # [ 1.052485] virtiofs virtio7: discovered new tag: xchg1820client # [ 1.043507] ACPI: \_SB_.GSIE: Enabled at IRQ 201821worker1 # [ 1.054035] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1822lb # [ 1.055935] virtiofs virtio7: discovered new tag: xchg1823client # [ 1.045729] ACPI: \_SB_.GSIF: Enabled at IRQ 211824lb # [ 1.057344] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1825client # [ 1.051590] virtiofs virtio5: discovered new tag: nix-store1826client # [ 1.053323] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1827client # [ 1.061320] virtiofs virtio6: discovered new tag: shared1828client # [ 1.062787] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1829worker2 # [ 1.070241] systemd[1]: Successfully made /usr/ read-only.1830worker1 # [ 1.075117] systemd[1]: Successfully made /usr/ read-only.1831client # [ 1.066947] virtiofs virtio7: discovered new tag: xchg1832client # [ 1.068506] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1833lb # [ 1.079916] systemd[1]: Successfully made /usr/ read-only.1834client # [ 1.090471] systemd[1]: Successfully made /usr/ read-only.1835worker1 # [ 1.410707] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1836worker1 # [ 1.415638] systemd[1]: Detected virtualization kvm.1837worker1 # [ 1.416475] systemd[1]: Detected architecture x86-64.1838worker2 # [ 1.405904] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1839worker1 # [ 1.417322] systemd[1]: Running in initrd.1840lb # [ 1.415596] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1841worker1 # [ 1.418356] systemd[1]: Initializing machine ID from random generator.1842lb # [ 1.420441] systemd[1]: Detected virtualization kvm.1843worker2 # [ 1.415411] systemd[1]: Detected virtualization kvm.1844worker1 # [ 1.419479] systemd[1]: Hostname set to <worker1>.1845lb # [ 1.421291] systemd[1]: Detected architecture x86-64.1846lb # [ 1.422152] systemd[1]: Running in initrd.1847worker2 # [ 1.417491] systemd[1]: Detected architecture x86-64.1848lb # [ 1.423260] systemd[1]: Initializing machine ID from random generator.1849lb # [ 1.424495] systemd[1]: Hostname set to <lb>.1850worker2 # [ 1.419572] systemd[1]: Running in initrd.1851worker2 # [ 1.422000] systemd[1]: Initializing machine ID from random generator.1852worker2 # [ 1.424856] systemd[1]: Hostname set to <worker2>.1853client # [ 1.427120] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1854client # [ 1.439120] systemd[1]: Detected virtualization kvm.1855client # [ 1.441318] systemd[1]: Detected architecture x86-64.1856client # [ 1.443494] systemd[1]: Running in initrd.1857client # [ 1.446064] systemd[1]: Initializing machine ID from random generator.1858client # [ 1.449036] systemd[1]: Hostname set to <client>.1859worker1 # [ 1.613638] systemd[1]: bpf-restrict-fs: LSM BPF program attached1860lb # [ 1.615500] systemd[1]: bpf-restrict-fs: LSM BPF program attached1861worker1 # [ 1.651072] systemd[1]: Queued start job for default target Initrd Default Target.1862lb # [ 1.654825] systemd[1]: Queued start job for default target Initrd Default Target.1863worker1 # [ 1.655614] systemd[1]: Created slice Slice /system/modprobe.1864worker2 # [ 1.651848] systemd[1]: bpf-restrict-fs: LSM BPF program attached1865worker1 # [ 1.656805] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1866lb # [ 1.659072] systemd[1]: Created slice Slice /system/modprobe.1867worker1 # [ 1.658228] systemd[1]: Expecting device /dev/disk/by-label/nixos...1868lb # [ 1.660232] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1869worker1 # [ 1.659367] systemd[1]: Reached target Path Units.1870worker1 # [ 1.660202] systemd[1]: Reached target Slice Units.1871lb # [ 1.661645] systemd[1]: Expecting device /dev/disk/by-label/nixos...1872worker1 # [ 1.661082] systemd[1]: Reached target Swaps.1873lb # [ 1.662777] systemd[1]: Reached target Path Units.1874worker1 # [ 1.661890] systemd[1]: Reached target Timer Units.1875lb # [ 1.663658] systemd[1]: Reached target Slice Units.1876lb # [ 1.664545] systemd[1]: Reached target Swaps.1877worker1 # [ 1.662869] systemd[1]: Listening on D-Bus System Message Bus Socket.1878lb # [ 1.665344] systemd[1]: Reached target Timer Units.1879worker1 # [ 1.664092] systemd[1]: Listening on Journal Socket (/dev/log).1880lb # [ 1.666346] systemd[1]: Listening on D-Bus System Message Bus Socket.1881worker1 # [ 1.665257] systemd[1]: Listening on Journal Sockets.1882lb # [ 1.667638] systemd[1]: Listening on Journal Socket (/dev/log).1883worker1 # [ 1.666260] systemd[1]: Listening on udev Control Socket.1884lb # [ 1.668780] systemd[1]: Listening on Journal Sockets.1885worker1 # [ 1.667248] systemd[1]: Listening on udev Kernel Socket.1886worker1 # [ 1.668221] systemd[1]: Reached target Socket Units.1887lb # [ 1.669772] systemd[1]: Listening on udev Control Socket.1888lb # [ 1.670819] systemd[1]: Listening on udev Kernel Socket.1889worker1 # [ 1.670036] systemd[1]: Starting Create List of Static Device Nodes...1890lb # [ 1.671803] systemd[1]: Reached target Socket Units.1891lb # [ 1.673645] systemd[1]: Starting Create List of Static Device Nodes...1892worker1 # [ 1.674390] systemd[1]: Starting Load Kernel Module configfs...1893lb # [ 1.677562] systemd[1]: Starting Load Kernel Module configfs...1894client # [ 1.677148] systemd[1]: bpf-restrict-fs: LSM BPF program attached1895worker1 # [ 1.683896] systemd[1]: Starting Journal Service...1896lb # [ 1.687923] systemd[1]: Starting Journal Service...1897worker2 # [ 1.700926] systemd[1]: Queued start job for default target Initrd Default Target.1898worker1 # [ 1.707897] systemd[1]: Starting Load Kernel Modules...1899worker2 # [ 1.705499] systemd[1]: Created slice Slice /system/modprobe.1900worker2 # [ 1.706681] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1901worker2 # [ 1.708098] systemd[1]: Expecting device /dev/disk/by-label/nixos...1902lb # [ 1.713375] systemd[1]: Starting Load Kernel Modules...1903worker2 # [ 1.709254] systemd[1]: Reached target Path Units.1904worker2 # [ 1.710143] systemd[1]: Reached target Slice Units.1905worker1 # [ 1.712444] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1906worker2 # [ 1.711007] systemd[1]: Reached target Swaps.1907worker2 # [ 1.711831] systemd[1]: Reached target Timer Units.1908worker2 # [ 1.712809] systemd[1]: Listening on D-Bus System Message Bus Socket.1909worker2 # [ 1.714049] systemd[1]: Listening on Journal Socket (/dev/log).1910lb # [ 1.717863] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1911worker2 # [ 1.715267] systemd[1]: Listening on Journal Sockets.1912worker2 # [ 1.716334] systemd[1]: Listening on udev Control Socket.1913worker1 # [ 1.719385] systemd[1]: Starting Coldplug All udev Devices...1914worker2 # [ 1.717355] systemd[1]: Listening on udev Kernel Socket.1915worker2 # [ 1.718318] systemd[1]: Reached target Socket Units.1916worker2 # [ 1.720204] systemd[1]: Starting Create List of Static Device Nodes...1917lb # [ 1.724920] systemd[1]: Starting Coldplug All udev Devices...1918worker2 # [ 1.723958] systemd[1]: Starting Load Kernel Module configfs...1919worker1 # [ 1.732671] systemd[1]: Finished Create List of Static Device Nodes.1920worker1 # [ 1.734015] systemd-journald[66]: Collecting audit messages is disabled.1921worker1 # [ 1.737938] systemd[1]: modprobe@configfs.service: Deactivated successfully.1922lb # [ 1.740082] systemd[1]: Finished Create List of Static Device Nodes.1923lb # [ 1.741489] systemd-journald[66]: Collecting audit messages is disabled.1924lb # [ 1.743439] systemd[1]: modprobe@configfs.service: Deactivated successfully.1925client # [ 1.733475] systemd[1]: Queued start job for default target Initrd Default Target.1926worker1 # [ 1.745713] systemd[1]: Finished Load Kernel Module configfs.1927client # [ 1.736809] systemd[1]: Created slice Slice /system/modprobe.1928worker2 # [ 1.734324] systemd[1]: Starting Journal Service...1929client # [ 1.738084] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1930client # [ 1.739649] systemd[1]: Expecting device /dev/disk/by-label/nixos...1931lb # [ 1.751783] systemd[1]: Finished Load Kernel Module configfs.1932client # [ 1.740843] systemd[1]: Reached target Path Units.1933client # [ 1.741755] systemd[1]: Reached target Slice Units.1934client # [ 1.742701] systemd[1]: Reached target Swaps.1935client # [ 1.743565] systemd[1]: Reached target Timer Units.1936worker1 # [ 1.752777] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1937client # [ 1.744708] systemd[1]: Listening on D-Bus System Message Bus Socket.1938client # [ 1.745985] systemd[1]: Listening on Journal Socket (/dev/log).1939client # [ 1.747178] systemd[1]: Listening on Journal Sockets.1940lb # [ 1.758200] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1941client # [ 1.748194] systemd[1]: Listening on udev Control Socket.1942client # [ 1.749404] systemd[1]: Listening on udev Kernel Socket.1943client # [ 1.750435] systemd[1]: Reached target Socket Units.1944client # [ 1.752359] systemd[1]: Starting Create List of Static Device Nodes...1945worker2 # [ 1.759785] systemd[1]: Starting Load Kernel Modules...1946worker1 # [ 1.764409] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1947client # [ 1.756239] systemd[1]: Starting Load Kernel Module configfs...1948worker2 # [ 1.764272] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1949lb # [ 1.771817] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1950worker1 # [ 1.773017] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1951worker2 # [ 1.772214] systemd[1]: Starting Coldplug All udev Devices...1952lb # [ 1.781017] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1953worker1 # [ 1.783640] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1954client # [ 1.766105] systemd[1]: Starting Journal Service...1955lb # [ 1.788862] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1956worker2 # [ 1.785462] systemd-journald[66]: Collecting audit messages is disabled.1957worker2 # [ 1.786746] systemd[1]: Finished Create List of Static Device Nodes.1958worker2 # [ 1.793713] systemd[1]: modprobe@configfs.service: Deactivated successfully.1959client # [ 1.791467] systemd[1]: Starting Load Kernel Modules...1960worker2 # [ 1.800540] systemd[1]: Finished Load Kernel Module configfs.1961worker1 # [ 1.805389] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1962client # [ 1.795503] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1963lb # [ 1.811321] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1964worker2 # [ 1.806571] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1965worker1 # [ 1.812522] systemd[1]: Starting Create Static Device Nodes in /dev...1966client # [ 1.803577] systemd[1]: Starting Coldplug All udev Devices...1967worker2 # [ 1.818205] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1968lb # [ 1.822648] systemd[1]: Starting Create Static Device Nodes in /dev...1969client # [ 1.814709] systemd-journald[66]: Collecting audit messages is disabled.1970worker2 # [ 1.821931] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1971worker1 # [ 1.827392] systemd[1]: Finished Load Kernel Modules.1972client # [ 1.817507] systemd[1]: Finished Create List of Static Device Nodes.1973lb # [ 1.832292] systemd[1]: Finished Load Kernel Modules.1974client # [ 1.823338] systemd[1]: modprobe@configfs.service: Deactivated successfully.1975worker1 # [ 1.835542] systemd[1]: Starting Apply Kernel Variables...1976lb # [ 1.841947] systemd[1]: Starting Apply Kernel Variables...1977worker2 # [ 1.837329] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1978client # [ 1.830860] systemd[1]: Finished Load Kernel Module configfs.1979client # [ 1.837301] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1980worker1 # [ 1.849483] systemd[1]: Finished Create Static Device Nodes in /dev.1981worker1 # [ 1.855551] systemd[1]: Reached target Preparation for Local File Systems.1982client # [ 1.847473] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1983worker1 # [ 1.723857] systemd-modules-load[67]: Inserted module 'dm_mod'1984lb # [ 1.862037] systemd[1]: Started Journal Service.1985worker1 # [ 1.727323] systemd-modules-load[67]: Inserted module 'virtio_balloon'1986worker1 # [ 1.728328] systemd-modules-load[67]: Inserted module 'virtio_gpu'1987worker2 # [ 1.858742] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1988worker1 # [ 1.864024] systemd[1]: Reached target Local File Systems.1989lb # [ 1.729343] systemd-modules-load[67]: Inserted module 'dm_mod'1990lb # [ 1.733305] systemd-modules-load[67]: Inserted module 'virtio_balloon'1991lb # [ 1.734431] systemd-modules-load[67]: Inserted module 'virtio_gpu'1992worker1 # [ 1.868504] systemd[1]: Starting Rule-based Manager for Device Events and Files...1993lb # [ 1.738238] systemd[1]: Finished Create Static Device Nodes in /dev.1994lb # [ 1.739464] systemd[1]: Reached target Preparation for Local File Systems.1995client # [ 1.861333] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1996worker2 # [ 1.869369] systemd[1]: Starting Create Static Device Nodes in /dev...1997lb # [ 1.741310] systemd[1]: Reached target Local File Systems.1998worker1 # [ 1.875510] systemd[1]: Started Journal Service.1999lb # [ 1.746062] systemd[1]: Starting Create System Files and Directories...2000worker1 # [ 1.747204] systemd[1]: Finished Apply Kernel Variables.
loading 2,390 hidden lines…
4391worker1 # [ 25.576333] nix-grpc-daemon[780]: ts=2026-09-21T14:43:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4392worker2 # [ 25.657442] nix-grpc-daemon[782]: ts=2026-09-21T14:43:37Z level=info event=scheduler_connected addr=lb:500514393worker2 # [ 25.661255] nix-grpc-daemon[782]: ts=2026-09-21T14:43:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4394lb: must succeed: curl -sf localhost:9901/clusters4395lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4396worker1 # [ 25.776763] nix-grpc-daemon[780]: ts=2026-09-21T14:43:37Z level=info event=scheduler_connected addr=lb:500514397worker1 # [ 25.780433] nix-grpc-daemon[780]: ts=2026-09-21T14:43:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4398worker2 # [ 25.861997] nix-grpc-daemon[782]: ts=2026-09-21T14:43:38Z level=info event=scheduler_connected addr=lb:500514399worker2 # [ 25.865921] nix-grpc-daemon[782]: ts=2026-09-21T14:43:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4400worker1 # [ 25.982498] nix-grpc-daemon[780]: ts=2026-09-21T14:43:38Z level=info event=scheduler_connected addr=lb:500514401worker1 # [ 25.985941] nix-grpc-daemon[780]: ts=2026-09-21T14:43:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4402worker2 # [ 26.067943] nix-grpc-daemon[782]: ts=2026-09-21T14:43:38Z level=info event=scheduler_connected addr=lb:500514403worker2 # [ 26.071737] nix-grpc-daemon[782]: ts=2026-09-21T14:43:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4404worker1 # [ 26.186917] nix-grpc-daemon[780]: ts=2026-09-21T14:43:38Z level=info event=scheduler_connected addr=lb:500514405worker1 # [ 26.191159] nix-grpc-daemon[780]: ts=2026-09-21T14:43:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4406worker2 # [ 26.270196] nix-grpc-daemon[782]: ts=2026-09-21T14:43:38Z level=info event=scheduler_connected addr=lb:500514407worker2 # [ 26.274291] nix-grpc-daemon[782]: ts=2026-09-21T14:43:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4408worker1 # [ 26.392101] nix-grpc-daemon[780]: ts=2026-09-21T14:43:38Z level=info event=scheduler_connected addr=lb:500514409worker1 # [ 26.396192] nix-grpc-daemon[780]: ts=2026-09-21T14:43:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4410worker2 # [ 26.475365] nix-grpc-daemon[782]: ts=2026-09-21T14:43:38Z level=info event=scheduler_connected addr=lb:500514411worker2 # [ 26.479279] nix-grpc-daemon[782]: ts=2026-09-21T14:43:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4412worker1 # [ 26.596919] nix-grpc-daemon[780]: ts=2026-09-21T14:43:38Z level=info event=scheduler_connected addr=lb:500514413worker1 # [ 26.600881] nix-grpc-daemon[780]: ts=2026-09-21T14:43:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4414worker2 # [ 26.680236] nix-grpc-daemon[782]: ts=2026-09-21T14:43:38Z level=info event=scheduler_connected addr=lb:500514415worker2 # [ 26.683960] nix-grpc-daemon[782]: ts=2026-09-21T14:43:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4416lb: must succeed: curl -sf localhost:9901/clusters4417lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4418worker1 # [ 26.801597] nix-grpc-daemon[780]: ts=2026-09-21T14:43:38Z level=info event=scheduler_connected addr=lb:500514419worker1 # [ 26.805555] nix-grpc-daemon[780]: ts=2026-09-21T14:43:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4420worker2 # [ 26.884923] nix-grpc-daemon[782]: ts=2026-09-21T14:43:39Z level=info event=scheduler_connected addr=lb:500514421worker2 # [ 26.888594] nix-grpc-daemon[782]: ts=2026-09-21T14:43:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4422worker1 # [ 27.007313] nix-grpc-daemon[780]: ts=2026-09-21T14:43:39Z level=info event=scheduler_connected addr=lb:500514423worker1 # [ 27.011137] nix-grpc-daemon[780]: ts=2026-09-21T14:43:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4424worker2 # [ 27.091379] nix-grpc-daemon[782]: ts=2026-09-21T14:43:39Z level=info event=scheduler_connected addr=lb:500514425worker2 # [ 27.094498] nix-grpc-daemon[782]: ts=2026-09-21T14:43:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4426worker1 # [ 27.212159] nix-grpc-daemon[780]: ts=2026-09-21T14:43:39Z level=info event=scheduler_connected addr=lb:500514427worker1 # [ 27.216345] nix-grpc-daemon[780]: ts=2026-09-21T14:43:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4428worker2 # [ 27.292786] nix-grpc-daemon[782]: ts=2026-09-21T14:43:39Z level=info event=scheduler_connected addr=lb:500514429worker2 # [ 27.296589] nix-grpc-daemon[782]: ts=2026-09-21T14:43:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4430worker1 # [ 27.417177] nix-grpc-daemon[780]: ts=2026-09-21T14:43:39Z level=info event=scheduler_connected addr=lb:500514431worker1 # [ 27.421187] nix-grpc-daemon[780]: ts=2026-09-21T14:43:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4432worker2 # [ 27.497560] nix-grpc-daemon[782]: ts=2026-09-21T14:43:39Z level=info event=scheduler_connected addr=lb:500514433worker2 # [ 27.501390] nix-grpc-daemon[782]: ts=2026-09-21T14:43:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4434worker1 # [ 27.622070] nix-grpc-daemon[780]: ts=2026-09-21T14:43:39Z level=info event=scheduler_connected addr=lb:500514435worker1 # [ 27.626373] nix-grpc-daemon[780]: ts=2026-09-21T14:43:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4436worker2 # [ 27.702422] nix-grpc-daemon[782]: ts=2026-09-21T14:43:39Z level=info event=scheduler_connected addr=lb:500514437worker2 # [ 27.706432] nix-grpc-daemon[782]: ts=2026-09-21T14:43:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4438lb: must succeed: curl -sf localhost:9901/clusters4439worker1 # [ 27.827142] nix-grpc-daemon[780]: ts=2026-09-21T14:43:39Z level=info event=scheduler_connected addr=lb:500514440worker1 # [ 27.831122] nix-grpc-daemon[780]: ts=2026-09-21T14:43:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4441lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4442worker2 # [ 27.906995] nix-grpc-daemon[782]: ts=2026-09-21T14:43:40Z level=info event=scheduler_connected addr=lb:500514443worker2 # [ 27.910854] nix-grpc-daemon[782]: ts=2026-09-21T14:43:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4444worker1 # [ 28.031947] nix-grpc-daemon[780]: ts=2026-09-21T14:43:40Z level=info event=scheduler_connected addr=lb:500514445worker1 # [ 28.033637] nix-grpc-daemon[780]: ts=2026-09-21T14:43:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4446worker2 # [ 28.113290] nix-grpc-daemon[782]: ts=2026-09-21T14:43:40Z level=info event=scheduler_connected addr=lb:500514447worker2 # [ 28.116422] nix-grpc-daemon[782]: ts=2026-09-21T14:43:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4448worker1 # [ 28.234293] nix-grpc-daemon[780]: ts=2026-09-21T14:43:40Z level=info event=scheduler_connected addr=lb:500514449worker1 # [ 28.238265] nix-grpc-daemon[780]: ts=2026-09-21T14:43:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4450worker2 # [ 28.314371] nix-grpc-daemon[782]: ts=2026-09-21T14:43:40Z level=info event=scheduler_connected addr=lb:500514451worker2 # [ 28.318219] nix-grpc-daemon[782]: ts=2026-09-21T14:43:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4452worker1 # [ 28.438992] nix-grpc-daemon[780]: ts=2026-09-21T14:43:40Z level=info event=scheduler_connected addr=lb:500514453worker1 # [ 28.443169] nix-grpc-daemon[780]: ts=2026-09-21T14:43:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4454worker2 # [ 28.519307] nix-grpc-daemon[782]: ts=2026-09-21T14:43:40Z level=info event=scheduler_connected addr=lb:500514455worker2 # [ 28.523216] nix-grpc-daemon[782]: ts=2026-09-21T14:43:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4456worker1 # [ 28.643896] nix-grpc-daemon[780]: ts=2026-09-21T14:43:40Z level=info event=scheduler_connected addr=lb:500514457worker1 # [ 28.647832] nix-grpc-daemon[780]: ts=2026-09-21T14:43:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4458worker2 # [ 28.724293] nix-grpc-daemon[782]: ts=2026-09-21T14:43:40Z level=info event=scheduler_connected addr=lb:500514459worker2 # [ 28.728137] nix-grpc-daemon[782]: ts=2026-09-21T14:43:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4460lb: must succeed: curl -sf localhost:9901/clusters4461worker1 # [ 28.848645] nix-grpc-daemon[780]: ts=2026-09-21T14:43:41Z level=info event=scheduler_connected addr=lb:500514462worker1 # [ 28.852414] nix-grpc-daemon[780]: ts=2026-09-21T14:43:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4463lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4464worker2 # [ 28.928958] nix-grpc-daemon[782]: ts=2026-09-21T14:43:41Z level=info event=scheduler_connected addr=lb:500514465worker2 # [ 28.932894] nix-grpc-daemon[782]: ts=2026-09-21T14:43:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4466worker1 # [ 29.054570] nix-grpc-daemon[780]: ts=2026-09-21T14:43:41Z level=info event=scheduler_connected addr=lb:500514467worker1 # [ 29.058455] nix-grpc-daemon[780]: ts=2026-09-21T14:43:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4468worker2 # [ 29.132276] nix-grpc-daemon[782]: ts=2026-09-21T14:43:41Z level=info event=scheduler_connected addr=lb:500514469worker2 # [ 29.135441] nix-grpc-daemon[782]: ts=2026-09-21T14:43:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4470worker1 # [ 29.259452] nix-grpc-daemon[780]: ts=2026-09-21T14:43:41Z level=info event=scheduler_connected addr=lb:500514471worker1 # [ 29.263295] nix-grpc-daemon[780]: ts=2026-09-21T14:43:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4472worker2 # [ 29.333641] nix-grpc-daemon[782]: ts=2026-09-21T14:43:41Z level=info event=scheduler_connected addr=lb:500514473worker2 # [ 29.337449] nix-grpc-daemon[782]: ts=2026-09-21T14:43:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4474worker1 # [ 29.464123] nix-grpc-daemon[780]: ts=2026-09-21T14:43:41Z level=info event=scheduler_connected addr=lb:500514475worker1 # [ 29.468120] nix-grpc-daemon[780]: ts=2026-09-21T14:43:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4476worker2 # [ 29.538559] nix-grpc-daemon[782]: ts=2026-09-21T14:43:41Z level=info event=scheduler_connected addr=lb:500514477worker2 # [ 29.542306] nix-grpc-daemon[782]: ts=2026-09-21T14:43:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4478worker1 # [ 29.669058] nix-grpc-daemon[780]: ts=2026-09-21T14:43:41Z level=info event=scheduler_connected addr=lb:500514479worker1 # [ 29.673418] nix-grpc-daemon[780]: ts=2026-09-21T14:43:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4480worker2 # [ 29.743350] nix-grpc-daemon[782]: ts=2026-09-21T14:43:41Z level=info event=scheduler_connected addr=lb:500514481worker2 # [ 29.747174] nix-grpc-daemon[782]: ts=2026-09-21T14:43:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4482worker1 # [ 29.873925] nix-grpc-daemon[780]: ts=2026-09-21T14:43:42Z level=info event=scheduler_connected addr=lb:500514483worker1 # [ 29.877722] nix-grpc-daemon[780]: ts=2026-09-21T14:43:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4484lb: must succeed: curl -sf localhost:9901/clusters4485lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4486worker2 # [ 29.948230] nix-grpc-daemon[782]: ts=2026-09-21T14:43:42Z level=info event=scheduler_connected addr=lb:500514487worker2 # [ 29.951997] nix-grpc-daemon[782]: ts=2026-09-21T14:43:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4488worker1 # [ 30.079354] nix-grpc-daemon[780]: ts=2026-09-21T14:43:42Z level=info event=scheduler_connected addr=lb:500514489worker1 # [ 30.083844] nix-grpc-daemon[780]: ts=2026-09-21T14:43:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4490worker2 # [ 30.153787] nix-grpc-daemon[782]: ts=2026-09-21T14:43:42Z level=info event=scheduler_connected addr=lb:500514491worker2 # [ 30.157488] nix-grpc-daemon[782]: ts=2026-09-21T14:43:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4492worker1 # [ 30.283970] nix-grpc-daemon[780]: ts=2026-09-21T14:43:42Z level=info event=scheduler_connected addr=lb:500514493worker1 # [ 30.287909] nix-grpc-daemon[780]: ts=2026-09-21T14:43:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4494worker2 # [ 30.355864] nix-grpc-daemon[782]: ts=2026-09-21T14:43:42Z level=info event=scheduler_connected addr=lb:500514495worker2 # [ 30.359741] nix-grpc-daemon[782]: ts=2026-09-21T14:43:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4496worker1 # [ 30.488709] nix-grpc-daemon[780]: ts=2026-09-21T14:43:42Z level=info event=scheduler_connected addr=lb:500514497worker1 # [ 30.492476] nix-grpc-daemon[780]: ts=2026-09-21T14:43:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4498worker2 # [ 30.561146] nix-grpc-daemon[782]: ts=2026-09-21T14:43:42Z level=info event=scheduler_connected addr=lb:500514499worker2 # [ 30.564984] nix-grpc-daemon[782]: ts=2026-09-21T14:43:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4500worker1 # [ 30.591355] nix-grpc-daemon[780]: ts=2026-09-21T14:43:42Z level=info event=scheduler_take_over4501lb # [ 30.592520] niks3-server[982]: 2026/09/21 14:43:42 INFO lead: acquired remote=192.168.1.3:487304502worker1 # [ 30.693382] nix-grpc-daemon[780]: ts=2026-09-21T14:43:42Z level=info event=scheduler_connected addr=lb:500514503worker1 # [ 30.697241] nix-grpc-daemon[780]: ts=2026-09-21T14:43:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4504worker2 # [ 30.765838] nix-grpc-daemon[782]: ts=2026-09-21T14:43:42Z level=info event=scheduler_connected addr=lb:500514505worker2 # [ 30.769651] nix-grpc-daemon[782]: ts=2026-09-21T14:43:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4506worker1 # [ 30.898208] nix-grpc-daemon[780]: ts=2026-09-21T14:43:43Z level=info event=scheduler_connected addr=lb:500514507worker1 # [ 30.910498] nix-grpc-daemon[780]: ts=2026-09-21T14:43:43Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:485064508worker1 # [ 30.914072] nix-grpc-daemon[780]: ts=2026-09-21T14:43:43Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04509lb: must succeed: curl -sf localhost:9901/clusters4510worker2 # [ 30.970426] nix-grpc-daemon[782]: ts=2026-09-21T14:43:43Z level=info event=scheduler_connected addr=lb:500514511worker1 # [ 30.985557] nix-grpc-daemon[780]: ts=2026-09-21T14:43:43Z level=info event=oidc_jwks provider=mock keys=14512lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4513worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4514worker1 # [ 30.988476] nix-grpc-daemon[780]: ts=2026-09-21T14:43:43Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:485064515worker1 # [ 30.992823] nix-grpc-daemon[780]: ts=2026-09-21T14:43:43Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04516worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4517lb: must succeed: curl -sf localhost:9901/clusters4518lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4519lb: must succeed: curl -sf localhost:9901/clusters4520lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4521lb: must succeed: curl -sf localhost:9901/clusters4522lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)4523lb: must succeed: curl -sf localhost:9901/clusters4524lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)4525worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4526worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.04 seconds)4527worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4528worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)4529lb: must succeed: curl -sf localhost:9901/clusters4530lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4531lb: must succeed: curl -sf localhost:9901/clusters4532lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)4533worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4534worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4535worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4536worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4537worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4538worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)4539worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4540worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)4541(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 9.88 seconds)4542subtest: balancer auth: client cert, bearer token, nothing, unknown CN4543worker2 # [ 31.591569] nix-grpc-daemon[782]: ts=2026-09-21T14:43:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:373024544worker2 # [ 31.600936] systemd[1]: Started Nix Daemon.4545worker2 # [ 31.708862] nix-daemon[850]: accepted connection from pid 782, user nix-grpc-daemon (trusted)4546worker2 # [ 31.724177] nix-grpc-daemon[782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 32 ms (attempt 1/5)4547worker2 # [ 31.759436] nix-grpc-daemon[782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 117 ms (attempt 2/5)4548worker2 # [ 31.878643] nix-grpc-daemon[782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 245 ms (attempt 3/5)4549worker2 # [ 32.125338] nix-grpc-daemon[782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 332 ms (attempt 4/5)4550worker2 # [ 32.464682] nix-grpc-daemon[782]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4551worker2 # [ 32.466233] nix-grpc-daemon[782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4552worker1 # [ 32.495426] nix-grpc-daemon[780]: ts=2026-09-21T14:43:44Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:485224553worker1 # [ 32.500710] systemd[1]: Started Nix Daemon.4554worker1 # [ 32.594806] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4555worker1 # [ 32.613648] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 85 ms (attempt 1/5)4556worker1 # [ 32.702601] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 161 ms (attempt 2/5)4557worker1 # [ 32.865615] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 156 ms (attempt 3/5)4558worker1 # [ 33.023816] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 302 ms (attempt 4/5)4559worker1 # [ 33.329245] nix-grpc-daemon[780]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4560worker1 # [ 33.332644] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4561worker1 # [ 33.345343] nix-grpc-daemon[780]: ts=2026-09-21T14:43:45Z level=debug event=path_info_miss path=p6lhqb373376nh9xdab84lh6d9781g76-nixos-system-client-test error="error: path '/nix/store/p6lhqb373376nh9xdab84lh6d9781g76-nixos-system-client-test' is required, but there is no substituter that can build it"4562worker1 # [ 33.360868] nix-grpc-daemon[780]: ts=2026-09-21T14:43:45Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:485224563worker1 # [ 33.370418] nix-grpc-daemon[780]: ts=2026-09-21T14:43:45Z level=debug event=path_info_miss path=p6lhqb373376nh9xdab84lh6d9781g76-nixos-system-client-test error="error: path '/nix/store/p6lhqb373376nh9xdab84lh6d9781g76-nixos-system-client-test' is required, but there is no substituter that can build it"4564client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4565client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.05 seconds)4566client: must succeed: install -D /nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4567client: (finished: must succeed: install -D /nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.02 seconds)4568worker2 # [ 33.598810] nix-grpc-daemon[782]: ts=2026-09-21T14:43:45Z level=info event=oidc_jwks provider=mock keys=14569worker2 # [ 33.603304] nix-grpc-daemon[782]: ts=2026-09-21T14:43:45Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:373024570worker2 # [ 33.615213] nix-grpc-daemon[782]: ts=2026-09-21T14:43:45Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:373024571worker2 # [ 33.625922] nix-grpc-daemon[782]: ts=2026-09-21T14:43:45Z level=debug event=path_info_miss path=p6lhqb373376nh9xdab84lh6d9781g76-nixos-system-client-test error="error: path '/nix/store/p6lhqb373376nh9xdab84lh6d9781g76-nixos-system-client-test' is required, but there is no substituter that can build it"4572worker1 # [ 33.632906] nix-grpc-daemon[780]: ts=2026-09-21T14:43:45Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:485224573worker1 # [ 33.642380] nix-grpc-daemon[780]: ts=2026-09-21T14:43:45Z level=debug event=path_info_miss path=p6lhqb373376nh9xdab84lh6d9781g76-nixos-system-client-test error="error: path '/nix/store/p6lhqb373376nh9xdab84lh6d9781g76-nixos-system-client-test' is required, but there is no substituter that can build it"4574client: must succeed: rm -r /var/lib/nix-grpc-store4575client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.01 seconds)4576worker1 # [ 33.783844] nix-grpc-daemon[780]: ts=2026-09-21T14:43:45Z level=info event=denied method=QueryMissing cn=- role=none4577worker1 # [ 33.921994] nix-grpc-daemon[780]: ts=2026-09-21T14:43:46Z level=info event=denied method=QueryMissing cn=stranger role=none4578(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 2.55 seconds)4579subtest: DAG build is scheduled across workers and lands in the cache4580client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&24581worker1 # [ 34.074162] nix-grpc-daemon[780]: ts=2026-09-21T14:43:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:485224582client # don't know how to build these paths:4583client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4584worker2 # [ 34.081911] nix-grpc-daemon[782]: ts=2026-09-21T14:43:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:373024585worker1 # [ 34.115126] nix-grpc-daemon[780]: ts=2026-09-21T14:43:46Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04586worker1 # [ 34.120938] nix-grpc-daemon[780]: ts=2026-09-21T14:43:46Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14587worker1 # [ 34.164529] nix-grpc-daemon[780]: ts=2026-09-21T14:43:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485224588worker1 # [ 34.181774] nix-grpc-daemon[780]: ts=2026-09-21T14:43:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485224589client # copying 1 paths to 'lb:50051'...4590worker1 # [ 34.195721] nix-grpc-daemon[780]: ts=2026-09-21T14:43:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:485224591client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4592worker1 # [ 34.214780] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4593worker1 # [ 34.238477] nix-grpc-daemon[780]: ts=2026-09-21T14:43:46Z level=info event=niks3_push_started pid=8764594worker1 # [ 34.389825] nix-grpc-daemon[876]: time=2026-09-21T14:43:46.556Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""4595worker1 # [ 34.482816] nix-daemon[862]: accepted connection from pid 889, user nix-grpc-daemon (trusted)4596worker1 # [ 34.496191] nix-daemon[862]: reaped child process 892, status = succeeded4597worker1 # [ 34.579230] nix-daemon[862]: accepted connection from pid 894, user nix-grpc-daemon (trusted)4598worker1 # [ 34.592587] nix-daemon[862]: reaped child process 897, status = succeeded4599lb # [ 34.599552] niks3-server[982]: 2026/09/21 14:43:46 INFO Received uploads request method=POST path=/api/pending_closures4600worker1 # [ 34.614161] nix-grpc-daemon[876]: time=2026-09-21T14:43:46.781Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4601worker1 # [ 34.615554] nix-grpc-daemon[876]: time=2026-09-21T14:43:46.781Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4602lb # [ 34.651334] niks3-server[982]: 2026/09/21 14:43:46 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4603lb # [ 34.658734] niks3-server[982]: 2026/09/21 14:43:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4604lb # [ 34.674301] niks3-server[982]: 2026/09/21 14:43:46 INFO Signed narinfos id=1 count=14605worker1 # [ 34.675642] nix-grpc-daemon[876]: time=2026-09-21T14:43:46.842Z level=INFO msg="Uploading 1 narinfos"4606lb # [ 34.685038] niks3-server[982]: 2026/09/21 14:43:46 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4607lb # [ 34.692559] niks3-server[982]: 2026/09/21 14:43:46 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4608lb # [ 34.705785] niks3-server[982]: 2026/09/21 14:43:46 INFO Completed upload id=14609worker1 # [ 34.705359] nix-grpc-daemon[876]: time=2026-09-21T14:43:46.872Z level=INFO msg="Upload complete. (311ms)"4610lb # [ 34.715223] niks3-server[982]: 2026/09/21 14:43:46 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4611worker1 # [ 34.710648] nix-grpc-daemon[780]: ts=2026-09-21T14:43:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=0 paths=1 nar_bytes_in=5044612worker1 # [ 34.718796] nix-grpc-daemon[780]: ts=2026-09-21T14:43:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485224613worker1 # [ 34.728673] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4614worker1 # [ 34.737885] nix-daemon[862]: reaped child process 874, status = succeeded4615worker1 # [ 34.746419] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4616client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4617client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4618client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4619client # farm-a> LOG-farm-a-t14620worker1 # [ 35.760681] nix-daemon[862]: reaped child process 901, status = succeeded4621worker1 # [ 35.837588] nix-daemon[862]: accepted connection from pid 917, user nix-grpc-daemon (trusted)4622worker1 # [ 35.849311] nix-daemon[862]: reaped child process 920, status = succeeded4623lb # [ 35.857770] niks3-server[982]: 2026/09/21 14:43:47 INFO Received uploads request method=POST path=/api/pending_closures4624worker1 # [ 35.868888] nix-grpc-daemon[876]: time=2026-09-21T14:43:48.036Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4625worker1 # [ 35.871051] nix-grpc-daemon[876]: time=2026-09-21T14:43:48.037Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4626lb # [ 35.899125] niks3-server[982]: 2026/09/21 14:43:47 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4627lb # [ 35.911971] niks3-server[982]: 2026/09/21 14:43:47 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4628lb # [ 35.919159] niks3-server[982]: 2026/09/21 14:43:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4629worker1 # [ 35.924428] nix-grpc-daemon[876]: time=2026-09-21T14:43:48.091Z level=INFO msg="Uploading 1 narinfos"4630lb # [ 35.924892] niks3-server[982]: 2026/09/21 14:43:47 INFO Signed narinfos id=2 count=14631lb # [ 35.931520] niks3-server[982]: 2026/09/21 14:43:47 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4632lb # [ 35.938057] niks3-server[982]: 2026/09/21 14:43:47 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4633lb # [ 35.945189] niks3-server[982]: 2026/09/21 14:43:47 INFO Completed upload id=24634worker1 # [ 35.944470] nix-grpc-daemon[876]: time=2026-09-21T14:43:48.111Z level=INFO msg="Upload complete. (191ms)"4635lb # [ 35.947510] niks3-server[982]: 2026/09/21 14:43:47 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4636worker1 # [ 35.953599] nix-grpc-daemon[780]: ts=2026-09-21T14:43:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14637worker1 # [ 35.965730] nix-grpc-daemon[780]: ts=2026-09-21T14:43:48Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04638worker1 # [ 35.969178] nix-grpc-daemon[780]: ts=2026-09-21T14:43:48Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14639worker1 # [ 35.978314] nix-daemon[862]: reaped child process 899, status = succeeded4640worker1 # [ 36.008638] nix-grpc-daemon[780]: ts=2026-09-21T14:43:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485224641worker1 # [ 36.021131] nix-grpc-daemon[780]: ts=2026-09-21T14:43:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485224642worker1 # [ 36.035423] nix-grpc-daemon[876]: time=2026-09-21T14:43:48.202Z level=INFO msg="All 1 paths already cached"4643client # copying 1 paths to 'lb:50051'...4644worker1 # [ 36.043737] nix-grpc-daemon[780]: ts=2026-09-21T14:43:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:485224645client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4646worker1 # [ 36.060784] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4647worker1 # [ 36.166086] nix-daemon[862]: accepted connection from pid 926, user nix-grpc-daemon (trusted)4648worker1 # [ 36.180236] nix-daemon[862]: reaped child process 929, status = succeeded4649worker1 # [ 36.263313] nix-daemon[862]: accepted connection from pid 931, user nix-grpc-daemon (trusted)4650worker1 # [ 36.276670] nix-daemon[862]: reaped child process 934, status = succeeded4651lb # [ 36.282679] niks3-server[982]: 2026/09/21 14:43:47 INFO Received uploads request method=POST path=/api/pending_closures4652worker1 # [ 36.297987] nix-grpc-daemon[876]: time=2026-09-21T14:43:48.465Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4653worker1 # [ 36.300097] nix-grpc-daemon[876]: time=2026-09-21T14:43:48.466Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4654lb # [ 36.318829] niks3-server[982]: 2026/09/21 14:43:47 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4655lb # [ 36.331300] niks3-server[982]: 2026/09/21 14:43:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4656lb # [ 36.338142] niks3-server[982]: 2026/09/21 14:43:47 INFO Signed narinfos id=3 count=14657worker1 # [ 36.340432] nix-grpc-daemon[876]: time=2026-09-21T14:43:48.507Z level=INFO msg="Uploading 1 narinfos"4658lb # [ 36.343142] niks3-server[982]: 2026/09/21 14:43:47 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4659lb # [ 36.355533] niks3-server[982]: 2026/09/21 14:43:47 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4660lb # [ 36.363794] niks3-server[982]: 2026/09/21 14:43:47 INFO Completed upload id=34661worker1 # [ 36.363069] nix-grpc-daemon[876]: time=2026-09-21T14:43:48.530Z level=INFO msg="Upload complete. (288ms)"4662lb # [ 36.369908] niks3-server[982]: 2026/09/21 14:43:47 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4663worker1 # [ 36.367325] nix-grpc-daemon[780]: ts=2026-09-21T14:43:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=0 paths=1 nar_bytes_in=6244664worker1 # [ 36.373377] nix-grpc-daemon[780]: ts=2026-09-21T14:43:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485224665worker1 # [ 36.385110] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4666worker1 # [ 36.394834] nix-daemon[862]: reaped child process 924, status = succeeded4667worker1 # [ 36.398932] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4668client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4669client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4670lb # [ 36.599417] envoy[917]: conn peer=[2001:db8:1::1]:55968 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=407 ms=18024671lb # [ 36.603054] envoy[917]: conn peer=[2001:db8:1::1]:53872 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=604672lb # [ 36.606970] envoy[917]: conn peer=[2001:db8:1::1]:53878 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=84673lb # [ 36.610469] envoy[917]: conn peer=[2001:db8:1::1]:53890 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=124674client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4675client # farm-b> LOG-farm-b-t14676worker1 # [ 37.233679] nix-daemon[862]: reaped child process 938, status = succeeded4677worker1 # [ 37.309076] nix-daemon[862]: accepted connection from pid 954, user nix-grpc-daemon (trusted)4678worker1 # [ 37.322343] nix-daemon[862]: reaped child process 957, status = succeeded4679lb # [ 37.330474] niks3-server[982]: 2026/09/21 14:43:48 INFO Received uploads request method=POST path=/api/pending_closures4680worker1 # [ 37.338120] nix-grpc-daemon[876]: time=2026-09-21T14:43:49.505Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4681worker1 # [ 37.339680] nix-grpc-daemon[876]: time=2026-09-21T14:43:49.506Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4682lb # [ 37.366358] niks3-server[982]: 2026/09/21 14:43:48 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4683lb # [ 37.375779] niks3-server[982]: 2026/09/21 14:43:48 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4684lb # [ 37.388893] niks3-server[982]: 2026/09/21 14:43:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4685lb # [ 37.394937] niks3-server[982]: 2026/09/21 14:43:48 INFO Signed narinfos id=4 count=14686worker1 # [ 37.396460] nix-grpc-daemon[876]: time=2026-09-21T14:43:49.563Z level=INFO msg="Uploading 1 narinfos"4687lb # [ 37.399227] niks3-server[982]: 2026/09/21 14:43:48 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4688lb # [ 37.408392] niks3-server[982]: 2026/09/21 14:43:48 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4689lb # [ 37.410809] niks3-server[982]: 2026/09/21 14:43:48 INFO Completed upload id=44690worker1 # [ 37.410879] nix-grpc-daemon[876]: time=2026-09-21T14:43:49.578Z level=INFO msg="Upload complete. (204ms)"4691worker1 # [ 37.413687] nix-grpc-daemon[876]: time=2026-09-21T14:43:49.580Z level=WARN msg="Failed to register uploaded object" key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo error="server returned 404: object is not pending upload\n"4692worker1 # [ 37.418443] nix-grpc-daemon[780]: ts=2026-09-21T14:43:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14693worker1 # [ 37.425983] nix-grpc-daemon[780]: ts=2026-09-21T14:43:49Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04694worker1 # [ 37.428405] nix-grpc-daemon[780]: ts=2026-09-21T14:43:49Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14695worker1 # [ 37.432234] nix-daemon[862]: reaped child process 936, status = succeeded4696worker1 # [ 37.469141] nix-grpc-daemon[780]: ts=2026-09-21T14:43:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485224697worker1 # [ 37.477549] nix-grpc-daemon[780]: ts=2026-09-21T14:43:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485224698worker1 # [ 37.487698] nix-grpc-daemon[876]: time=2026-09-21T14:43:49.654Z level=INFO msg="All 2 paths already cached"4699client # copying 1 paths to 'lb:50051'...4700client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4701worker1 # [ 37.495131] nix-grpc-daemon[780]: ts=2026-09-21T14:43:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:485224702worker1 # [ 37.508271] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4703worker1 # [ 37.595431] nix-daemon[862]: accepted connection from pid 964, user nix-grpc-daemon (trusted)4704worker1 # [ 37.608444] nix-daemon[862]: reaped child process 967, status = succeeded4705worker1 # [ 37.691922] nix-daemon[862]: accepted connection from pid 969, user nix-grpc-daemon (trusted)4706worker1 # [ 37.705103] nix-daemon[862]: reaped child process 972, status = succeeded4707lb # [ 37.712282] niks3-server[982]: 2026/09/21 14:43:49 INFO Received uploads request method=POST path=/api/pending_closures4708worker1 # [ 37.719915] nix-grpc-daemon[876]: time=2026-09-21T14:43:49.886Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4709worker1 # [ 37.721380] nix-grpc-daemon[876]: time=2026-09-21T14:43:49.886Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4710lb # [ 37.739544] niks3-server[982]: 2026/09/21 14:43:49 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4711lb # [ 37.750518] niks3-server[982]: 2026/09/21 14:43:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4712lb # [ 37.755443] niks3-server[982]: 2026/09/21 14:43:49 INFO Signed narinfos id=5 count=14713worker1 # [ 37.757875] nix-grpc-daemon[876]: time=2026-09-21T14:43:49.924Z level=INFO msg="Uploading 1 narinfos"4714lb # [ 37.764803] niks3-server[982]: 2026/09/21 14:43:49 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4715lb # [ 37.777112] niks3-server[982]: 2026/09/21 14:43:49 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4716lb # [ 37.785936] niks3-server[982]: 2026/09/21 14:43:49 INFO Completed upload id=54717worker1 # [ 37.785091] nix-grpc-daemon[876]: time=2026-09-21T14:43:49.952Z level=INFO msg="Upload complete. (268ms)"4718lb # [ 37.791516] niks3-server[982]: 2026/09/21 14:43:49 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4719worker1 # [ 37.788929] nix-grpc-daemon[780]: ts=2026-09-21T14:43:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=0 paths=1 nar_bytes_in=7524720worker1 # [ 37.796297] nix-grpc-daemon[780]: ts=2026-09-21T14:43:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485224721worker1 # [ 37.805140] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4722worker1 # [ 37.814384] nix-daemon[862]: reaped child process 962, status = succeeded4723worker1 # [ 37.817266] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4724client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4725client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4726client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4727client # farm-top> LOG-farm-top-t14728worker1 # [ 38.430902] nix-daemon[862]: reaped child process 976, status = succeeded4729worker1 # [ 38.505561] nix-daemon[862]: accepted connection from pid 992, user nix-grpc-daemon (trusted)4730worker1 # [ 38.518754] nix-daemon[862]: reaped child process 995, status = succeeded4731lb # [ 38.529875] niks3-server[982]: 2026/09/21 14:43:49 INFO Received uploads request method=POST path=/api/pending_closures4732worker1 # [ 38.542327] nix-grpc-daemon[876]: time=2026-09-21T14:43:50.708Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4733worker1 # [ 38.543685] nix-grpc-daemon[876]: time=2026-09-21T14:43:50.708Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4734lb # [ 38.570872] niks3-server[982]: 2026/09/21 14:43:49 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4735lb # [ 38.582282] niks3-server[982]: 2026/09/21 14:43:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4736lb # [ 38.583770] niks3-server[982]: 2026/09/21 14:43:50 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4737worker1 # [ 38.585741] nix-grpc-daemon[876]: time=2026-09-21T14:43:50.752Z level=INFO msg="Uploading 1 narinfos"4738lb # [ 38.586610] niks3-server[982]: 2026/09/21 14:43:50 INFO Signed narinfos id=6 count=14739lb # [ 38.591033] niks3-server[982]: 2026/09/21 14:43:50 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4740lb # [ 38.599352] niks3-server[982]: 2026/09/21 14:43:50 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4741lb # [ 38.601619] niks3-server[982]: 2026/09/21 14:43:50 INFO Completed upload id=64742worker1 # [ 38.600828] nix-grpc-daemon[876]: time=2026-09-21T14:43:50.768Z level=INFO msg="Upload complete. (177ms)"4743lb # [ 38.607288] niks3-server[982]: 2026/09/21 14:43:50 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4744worker1 # [ 38.606358] nix-grpc-daemon[780]: ts=2026-09-21T14:43:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=0 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14745worker1 # [ 38.618865] nix-daemon[862]: reaped child process 974, status = succeeded4746client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&2, in 4.69 seconds)4747client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath4748client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath, in 0.11 seconds)4749client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4750client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.01 seconds)4751client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4752client # copying 3 paths...4753client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4754client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4755client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4756client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.16 seconds)4757(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 5.04 seconds)4758subtest: repeat is answered Cached by the scheduler without building4759worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4760worker1 # finding garbage collector roots...4761worker1 # deleting specified paths...4762worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4763worker1 # deleting unused links...4764worker1 # note: hard linking is currently saving 0.0 KiB4765worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.15 seconds)4766worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4767worker2 # finding garbage collector roots...4768worker2 # deleting specified paths...4769worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4770worker2 # deleting unused links...4771worker2 # note: hard linking is currently saving 0.0 KiB4772worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.18 seconds)4773client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&24774client # don't know how to build these paths:4775client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4776worker2 # [ 39.460669] nix-grpc-daemon[782]: ts=2026-09-21T14:43:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:373024777worker1 # [ 39.467783] nix-grpc-daemon[780]: ts=2026-09-21T14:43:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485224778worker1 # [ 39.478935] nix-grpc-daemon[876]: time=2026-09-21T14:43:51.646Z level=INFO msg="All 2 paths already cached"4779worker1 # [ 39.493181] nix-grpc-daemon[780]: ts=2026-09-21T14:43:51Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14780client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&2, in 0.19 seconds)4781client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath4782client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath, in 0.10 seconds)4783lb: must succeed: curl -sf localhost:9901/clusters4784lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4785lb: must succeed: curl -sf localhost:9901/clusters4786lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4787worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4788worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4789(finished: subtest: repeat is answered Cached by the scheduler without building, in 0.81 seconds)4790subtest: inputs already in the cache are not sent to the scheduler4791lb: must succeed: curl -sf localhost:9901/clusters4792lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4793lb: must succeed: curl -sf localhost:9901/clusters4794lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)4795worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4796worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.04 seconds)4797client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b >&24798client # don't know how to build these paths:4799client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4800worker1 # [ 40.036123] nix-grpc-daemon[780]: ts=2026-09-21T14:43:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:485224801worker1 # [ 40.042300] nix-grpc-daemon[780]: ts=2026-09-21T14:43:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485224802worker1 # [ 40.050234] nix-grpc-daemon[876]: time=2026-09-21T14:43:52.217Z level=INFO msg="All 2 paths already cached"4803worker1 # [ 40.062225] nix-grpc-daemon[780]: ts=2026-09-21T14:43:52Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04804worker1 # [ 40.067456] nix-grpc-daemon[780]: ts=2026-09-21T14:43:52Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14805worker1 # [ 40.073520] nix-grpc-daemon[780]: ts=2026-09-21T14:43:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485224806worker1 # [ 40.085314] nix-grpc-daemon[780]: ts=2026-09-21T14:43:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485224807worker1 # [ 40.098201] nix-grpc-daemon[876]: time=2026-09-21T14:43:52.265Z level=INFO msg="All 2 paths already cached"4808client # copying 1 paths to 'lb:50051'...4809client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4810worker1 # [ 40.106823] nix-grpc-daemon[780]: ts=2026-09-21T14:43:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:485224811worker1 # [ 40.123394] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4812worker1 # [ 40.218965] nix-daemon[862]: accepted connection from pid 1025, user nix-grpc-daemon (trusted)4813worker1 # [ 40.232495] nix-daemon[862]: reaped child process 1028, status = succeeded4814worker1 # [ 40.313805] nix-daemon[862]: accepted connection from pid 1030, user nix-grpc-daemon (trusted)4815worker1 # [ 40.326555] nix-daemon[862]: reaped child process 1033, status = succeeded4816lb # [ 40.334875] niks3-server[982]: 2026/09/21 14:43:51 INFO Received uploads request method=POST path=/api/pending_closures4817worker1 # [ 40.342033] nix-grpc-daemon[876]: time=2026-09-21T14:43:52.509Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4818worker1 # [ 40.343599] nix-grpc-daemon[876]: time=2026-09-21T14:43:52.510Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4819lb # [ 40.365957] niks3-server[982]: 2026/09/21 14:43:51 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4820lb # [ 40.383152] niks3-server[982]: 2026/09/21 14:43:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4821lb # [ 40.389126] niks3-server[982]: 2026/09/21 14:43:51 INFO Signed narinfos id=7 count=14822worker1 # [ 40.391673] nix-grpc-daemon[876]: time=2026-09-21T14:43:52.558Z level=INFO msg="Uploading 1 narinfos"4823lb # [ 40.397869] niks3-server[982]: 2026/09/21 14:43:51 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4824lb # [ 40.411936] niks3-server[982]: 2026/09/21 14:43:51 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4825lb # [ 40.417331] niks3-server[982]: 2026/09/21 14:43:51 INFO Completed upload id=74826worker1 # [ 40.416513] nix-grpc-daemon[876]: time=2026-09-21T14:43:52.583Z level=INFO msg="Upload complete. (282ms)"4827lb # [ 40.419790] niks3-server[982]: 2026/09/21 14:43:51 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4828worker1 # [ 40.421211] nix-grpc-daemon[780]: ts=2026-09-21T14:43:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=0 paths=1 nar_bytes_in=7524829worker1 # [ 40.427599] nix-grpc-daemon[780]: ts=2026-09-21T14:43:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485224830worker1 # [ 40.439249] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4831worker1 # [ 40.448272] nix-daemon[862]: reaped child process 1023, status = succeeded4832worker1 # [ 40.452589] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4833client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4834client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4835client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4836client # farm-top> LOG-farm-top-t1b4837worker1 # [ 41.488930] nix-daemon[862]: reaped child process 1037, status = succeeded4838worker1 # [ 41.567539] nix-daemon[862]: accepted connection from pid 1054, user nix-grpc-daemon (trusted)4839worker1 # [ 41.581210] nix-daemon[862]: reaped child process 1057, status = succeeded4840lb # [ 41.589334] niks3-server[982]: 2026/09/21 14:43:53 INFO Received uploads request method=POST path=/api/pending_closures4841worker1 # [ 41.597054] nix-grpc-daemon[876]: time=2026-09-21T14:43:53.764Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4842worker1 # [ 41.599243] nix-grpc-daemon[876]: time=2026-09-21T14:43:53.765Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4843lb # [ 41.631156] niks3-server[982]: 2026/09/21 14:43:53 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4844lb # [ 41.644768] niks3-server[982]: 2026/09/21 14:43:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4845lb # [ 41.656100] niks3-server[982]: 2026/09/21 14:43:53 INFO Signed narinfos id=8 count=14846worker1 # [ 41.658209] nix-grpc-daemon[876]: time=2026-09-21T14:43:53.825Z level=INFO msg="Uploading 1 narinfos"4847lb # [ 41.659254] niks3-server[982]: 2026/09/21 14:43:53 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4848lb # [ 41.665604] niks3-server[982]: 2026/09/21 14:43:53 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4849lb # [ 41.674966] niks3-server[982]: 2026/09/21 14:43:53 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4850lb # [ 41.677698] niks3-server[982]: 2026/09/21 14:43:53 INFO Completed upload id=84851worker1 # [ 41.676895] nix-grpc-daemon[876]: time=2026-09-21T14:43:53.844Z level=INFO msg="Upload complete. (215ms)"4852lb # [ 41.680028] niks3-server[982]: 2026/09/21 14:43:53 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4853worker1 # [ 41.684521] nix-grpc-daemon[780]: ts=2026-09-21T14:43:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14854worker1 # [ 41.699400] nix-daemon[862]: reaped child process 1035, status = succeeded4855client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.82 seconds)4856client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b outPath4857client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.12 seconds)4858lb: must succeed: curl -sf localhost:9901/clusters4859lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4860lb: must succeed: curl -sf localhost:9901/clusters4861lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)4862worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4863worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.04 seconds)4864(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.14 seconds)4865subtest: two clients wanting the same drv build it once4866client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup4867client # [ 41.974747] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.4868client # Running as unit: dedup1.service; invocation ID: fd8f9b94888641739497efa1395c07b04869client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 0.06 seconds)4870client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup4871client # [ 42.113351] nix[888]: don't know how to build these paths:4872client # [ 42.114719] nix[888]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4873worker1 # [ 42.122302] nix-grpc-daemon[780]: ts=2026-09-21T14:43:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:485224874worker1 # [ 42.132904] nix-grpc-daemon[780]: ts=2026-09-21T14:43:54Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04875worker2 # [ 42.134720] nix-grpc-daemon[782]: ts=2026-09-21T14:43:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:373024876worker1 # [ 42.137727] nix-grpc-daemon[780]: ts=2026-09-21T14:43:54Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=14877worker2 # [ 42.147481] nix-grpc-daemon[782]: ts=2026-09-21T14:43:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:373024878client # [ 42.149073] nix[888]: copying 1 paths to 'lb:50051'...4879client # [ 42.149873] nix[888]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4880worker2 # [ 42.157482] nix-grpc-daemon[782]: ts=2026-09-21T14:43:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:373024881worker2 # [ 42.168630] nix-daemon[850]: accepted connection from pid 782, user nix-grpc-daemon (trusted)4882worker2 # [ 42.189958] nix-grpc-daemon[782]: ts=2026-09-21T14:43:54Z level=info event=niks3_push_started pid=8784883worker2 # [ 42.327774] nix-grpc-daemon[878]: time=2026-09-21T14:43:54.472Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""4884worker2 # [ 42.422671] nix-daemon[850]: accepted connection from pid 891, user nix-grpc-daemon (trusted)4885worker2 # [ 42.435892] nix-daemon[850]: reaped child process 894, status = succeeded4886worker2 # [ 42.521861] nix-daemon[850]: accepted connection from pid 896, user nix-grpc-daemon (trusted)4887worker2 # [ 42.535263] nix-daemon[850]: reaped child process 899, status = succeeded4888lb # [ 42.545075] niks3-server[982]: 2026/09/21 14:43:53 INFO Received uploads request method=POST path=/api/pending_closures4889worker2 # [ 42.549728] nix-grpc-daemon[878]: time=2026-09-21T14:43:54.695Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4890worker2 # [ 42.552703] nix-grpc-daemon[878]: time=2026-09-21T14:43:54.696Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4891lb # [ 42.586258] niks3-server[982]: 2026/09/21 14:43:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4892lb # [ 42.592875] niks3-server[982]: 2026/09/21 14:43:54 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4893lb # [ 42.599297] niks3-server[982]: 2026/09/21 14:43:54 INFO Signed narinfos id=9 count=14894worker2 # [ 42.599272] nix-grpc-daemon[878]: time=2026-09-21T14:43:54.744Z level=INFO msg="Uploading 1 narinfos"4895lb # [ 42.604129] niks3-server[982]: 2026/09/21 14:43:54 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4896lb # [ 42.619222] niks3-server[982]: 2026/09/21 14:43:54 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4897lb # [ 42.628204] niks3-server[982]: 2026/09/21 14:43:54 INFO Completed upload id=94898worker2 # [ 42.627032] nix-grpc-daemon[878]: time=2026-09-21T14:43:54.772Z level=INFO msg="Upload complete. (295ms)"4899lb # [ 42.632674] niks3-server[982]: 2026/09/21 14:43:54 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4900worker2 # [ 42.630699] nix-grpc-daemon[782]: ts=2026-09-21T14:43:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37302 duration_s=0 paths=1 nar_bytes_in=4324901worker2 # [ 42.635882] nix-grpc-daemon[782]: ts=2026-09-21T14:43:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:373024902worker2 # [ 42.644528] nix-daemon[850]: accepted connection from pid 782, user nix-grpc-daemon (trusted)4903worker2 # [ 42.652460] nix-daemon[850]: reaped child process 876, status = succeeded4904worker2 # [ 42.657170] nix-daemon[850]: accepted connection from pid 782, user nix-grpc-daemon (trusted)4905client # [ 42.649751] nix[888]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4906client # [ 43.036823] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.4907client # Running as unit: dedup2.service; invocation ID: bc15f20ec76747cc80748e80adf020964908client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 1.07 seconds)4909client # [ 43.208978] nix[904]: don't know how to build these paths:4910client # [ 43.209847] nix[904]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4911worker1 # [ 43.218336] nix-grpc-daemon[780]: ts=2026-09-21T14:43:55Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:485224912worker1 # [ 43.228645] nix-grpc-daemon[780]: ts=2026-09-21T14:43:55Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04913worker2 # [ 43.228976] nix-grpc-daemon[782]: ts=2026-09-21T14:43:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:373024914client # [ 43.224092] nix[904]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4915client: must succeed: sleep 24916client: (finished: must succeed: sleep 2, in 2.04 seconds)4917client: waiting for success: ! systemctl is-active dedup1 dedup24918worker2 # [ 46.476315] nix-daemon[850]: reaped child process 903, status = succeeded4919worker2 # [ 46.550655] nix-daemon[850]: accepted connection from pid 933, user nix-grpc-daemon (trusted)4920worker2 # [ 46.563637] nix-daemon[850]: reaped child process 936, status = succeeded4921lb # [ 46.576488] niks3-server[982]: 2026/09/21 14:43:57 INFO Received uploads request method=POST path=/api/pending_closures4922worker2 # [ 46.579952] nix-grpc-daemon[878]: time=2026-09-21T14:43:58.725Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4923worker2 # [ 46.581524] nix-grpc-daemon[878]: time=2026-09-21T14:43:58.726Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4924lb # [ 46.601146] envoy[917]: conn peer=[2001:db8:1::1]:53904 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9451 tx=4036 ms=45534925lb # [ 46.606400] envoy[917]: conn peer=[2001:db8:1::1]:53920 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=454926lb # [ 46.609951] envoy[917]: conn peer=[2001:db8:1::1]:53932 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4182 tx=1783 ms=16574927lb # [ 46.630469] niks3-server[982]: 2026/09/21 14:43:58 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4928lb # [ 46.642107] niks3-server[982]: 2026/09/21 14:43:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4929worker2 # [ 46.646416] nix-grpc-daemon[878]: time=2026-09-21T14:43:58.791Z level=INFO msg="Uploading 1 narinfos"4930lb # [ 46.649209] niks3-server[982]: 2026/09/21 14:43:58 INFO Signed narinfos id=10 count=14931lb # [ 46.654923] niks3-server[982]: 2026/09/21 14:43:58 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4932lb # [ 46.657454] niks3-server[982]: 2026/09/21 14:43:58 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4933lb # [ 46.661818] niks3-server[982]: 2026/09/21 14:43:58 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4934lb # [ 46.665640] niks3-server[982]: 2026/09/21 14:43:58 INFO Completed upload id=104935lb # [ 46.667139] niks3-server[982]: 2026/09/21 14:43:58 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo4936worker2 # [ 46.662599] nix-grpc-daemon[878]: time=2026-09-21T14:43:58.808Z level=INFO msg="Upload complete. (195ms)"4937worker2 # [ 46.669089] nix-grpc-daemon[782]: ts=2026-09-21T14:43:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37302 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14938worker2 # [ 46.675799] nix-grpc-daemon[782]: ts=2026-09-21T14:43:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37302 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14939worker2 # [ 46.688196] nix-daemon[850]: reaped child process 901, status = succeeded4940client # [ 46.692428] systemd[1]: dedup2.service: Deactivated successfully.4941client # [ 46.696120] systemd[1]: dedup2.service: Consumed 145ms CPU time over 3.658s wall clock time, 35.4M memory peak, 5.9K incoming IP traffic, 5.7K outgoing IP traffic.4942client # [ 46.730824] systemd[1]: dedup1.service: Deactivated successfully.4943client # [ 46.733619] systemd[1]: dedup1.service: Consumed 119ms CPU time over 4.758s wall clock time, 43.5M memory peak, 7.2K incoming IP traffic, 8.5K outgoing IP traffic.4944client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.09 seconds)4945client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4946client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)4947worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4948worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)4949worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4950worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)4951worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4952worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4953worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4954worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4955(finished: subtest: two clients wanting the same drv build it once, in 5.77 seconds)4956subtest: build hook: nix-daemon with builders = grpc://lb4957client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log4958client # [ 47.821502] systemd[1]: Started Nix Daemon.4959client # [ 47.912037] nix-daemon[946]: accepted connection from pid 943, user root (trusted)4960worker1 # [ 48.039981] nix-grpc-daemon[780]: ts=2026-09-21T14:44:00Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:485224961worker1 # [ 48.046120] nix-grpc-daemon[780]: ts=2026-09-21T14:44:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485224962worker1 # [ 48.057901] nix-grpc-daemon[780]: ts=2026-09-21T14:44:00Z level=debug event=want client=6 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=04963worker1 # [ 48.063876] nix-grpc-daemon[780]: ts=2026-09-21T14:44:00Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=14964worker1 # [ 48.101709] nix-grpc-daemon[780]: ts=2026-09-21T14:44:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485224965worker1 # [ 48.115965] nix-grpc-daemon[780]: ts=2026-09-21T14:44:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485224966worker1 # [ 48.129074] nix-grpc-daemon[780]: ts=2026-09-21T14:44:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:485224967worker1 # [ 48.138682] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4968worker1 # [ 48.241205] nix-daemon[862]: accepted connection from pid 1093, user nix-grpc-daemon (trusted)4969worker1 # [ 48.256237] nix-daemon[862]: reaped child process 1096, status = succeeded4970worker1 # [ 48.335470] nix-daemon[862]: accepted connection from pid 1098, user nix-grpc-daemon (trusted)4971worker1 # [ 48.349961] nix-daemon[862]: reaped child process 1101, status = succeeded4972lb # [ 48.355681] niks3-server[982]: 2026/09/21 14:43:59 INFO Received uploads request method=POST path=/api/pending_closures4973worker1 # [ 48.363303] nix-grpc-daemon[876]: time=2026-09-21T14:44:00.530Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4974worker1 # [ 48.364984] nix-grpc-daemon[876]: time=2026-09-21T14:44:00.530Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"4975lb # [ 48.380548] niks3-server[982]: 2026/09/21 14:43:59 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst4976lb # [ 48.391661] niks3-server[982]: 2026/09/21 14:43:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4977lb # [ 48.397121] niks3-server[982]: 2026/09/21 14:43:59 INFO Signed narinfos id=11 count=14978worker1 # [ 48.398959] nix-grpc-daemon[876]: time=2026-09-21T14:44:00.566Z level=INFO msg="Uploading 1 narinfos"4979lb # [ 48.407070] niks3-server[982]: 2026/09/21 14:43:59 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls4980lb # [ 48.417287] niks3-server[982]: 2026/09/21 14:43:59 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4981worker1 # [ 48.422203] nix-grpc-daemon[876]: time=2026-09-21T14:44:00.589Z level=INFO msg="Upload complete. (269ms)"4982lb # [ 48.423160] niks3-server[982]: 2026/09/21 14:43:59 INFO Completed upload id=114983worker1 # [ 48.424312] nix-grpc-daemon[780]: ts=2026-09-21T14:44:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=0 paths=1 nar_bytes_in=5124984lb # [ 48.429537] niks3-server[982]: 2026/09/21 14:43:59 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo4985worker1 # [ 48.429139] nix-grpc-daemon[780]: ts=2026-09-21T14:44:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485224986worker1 # [ 48.438059] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4987worker1 # [ 48.442503] nix-daemon[862]: reaped child process 1091, status = succeeded4988worker1 # [ 48.447440] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4989worker1 # [ 49.464121] nix-daemon[862]: reaped child process 1105, status = succeeded4990worker1 # [ 49.538580] nix-daemon[862]: accepted connection from pid 1121, user nix-grpc-daemon (trusted)4991worker1 # [ 49.552170] nix-daemon[862]: reaped child process 1124, status = succeeded4992lb # [ 49.559761] niks3-server[982]: 2026/09/21 14:44:00 INFO Received uploads request method=POST path=/api/pending_closures4993worker1 # [ 49.567421] nix-grpc-daemon[876]: time=2026-09-21T14:44:01.734Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4994worker1 # [ 49.568916] nix-grpc-daemon[876]: time=2026-09-21T14:44:01.736Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"4995lb # [ 49.598833] niks3-server[982]: 2026/09/21 14:44:01 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4996lb # [ 49.616273] niks3-server[982]: 2026/09/21 14:44:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4997lb # [ 49.621704] niks3-server[982]: 2026/09/21 14:44:01 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv4998worker1 # [ 49.629394] nix-grpc-daemon[876]: time=2026-09-21T14:44:01.796Z level=INFO msg="Uploading 1 narinfos"4999lb # [ 49.630287] niks3-server[982]: 2026/09/21 14:44:01 INFO Signed narinfos id=12 count=15000lb # [ 49.635601] niks3-server[982]: 2026/09/21 14:44:01 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5001lb # [ 49.645980] niks3-server[982]: 2026/09/21 14:44:01 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete5002lb # [ 49.648519] niks3-server[982]: 2026/09/21 14:44:01 INFO Completed upload id=125003worker1 # [ 49.649143] nix-grpc-daemon[876]: time=2026-09-21T14:44:01.816Z level=INFO msg="Upload complete. (194ms)"5004lb # [ 49.651934] niks3-server[982]: 2026/09/21 14:44:01 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5005worker1 # [ 49.657441] nix-grpc-daemon[780]: ts=2026-09-21T14:44:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=6 outputs=15006worker1 # [ 49.665350] nix-grpc-daemon[780]: ts=2026-09-21T14:44:01Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:485225007worker1 # [ 49.675172] nix-daemon[862]: reaped child process 1103, status = succeeded5008worker2 # [ 49.674771] nix-grpc-daemon[782]: ts=2026-09-21T14:44:01Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:373025009worker2 # [ 49.685536] nix-grpc-daemon[782]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5010worker2 # [ 49.700853] nix-grpc-daemon[782]: ts=2026-09-21T14:44:01Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:37302 duration_s=0 paths=1 nar_bytes_out=1285011worker1 # [ 49.910395] nix-grpc-daemon[780]: ts=2026-09-21T14:44:02Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:485225012worker2 # [ 49.910707] nix-grpc-daemon[782]: ts=2026-09-21T14:44:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:373025013worker2 # [ 49.920979] nix-grpc-daemon[878]: time=2026-09-21T14:44:02.065Z level=INFO msg="All 1 paths already cached"5014worker1 # [ 49.938196] nix-grpc-daemon[780]: ts=2026-09-21T14:44:02Z level=debug event=want client=7 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05015worker1 # [ 49.944326] nix-grpc-daemon[780]: ts=2026-09-21T14:44:02Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=15016worker1 # [ 49.949972] nix-grpc-daemon[780]: ts=2026-09-21T14:44:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485225017worker1 # [ 49.961669] nix-grpc-daemon[780]: ts=2026-09-21T14:44:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485225018worker1 # [ 49.974206] nix-grpc-daemon[876]: time=2026-09-21T14:44:02.141Z level=INFO msg="All 1 paths already cached"5019worker1 # [ 49.981666] nix-grpc-daemon[780]: ts=2026-09-21T14:44:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:485225020worker1 # [ 49.989978] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5021worker1 # [ 50.093684] nix-daemon[862]: accepted connection from pid 1130, user nix-grpc-daemon (trusted)5022worker1 # [ 50.108969] nix-daemon[862]: reaped child process 1133, status = succeeded5023worker1 # [ 50.193308] nix-daemon[862]: accepted connection from pid 1135, user nix-grpc-daemon (trusted)5024worker1 # [ 50.206840] nix-daemon[862]: reaped child process 1138, status = succeeded5025lb # [ 50.214100] niks3-server[982]: 2026/09/21 14:44:01 INFO Received uploads request method=POST path=/api/pending_closures5026worker1 # [ 50.221295] nix-grpc-daemon[876]: time=2026-09-21T14:44:02.388Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5027worker1 # [ 50.223355] nix-grpc-daemon[876]: time=2026-09-21T14:44:02.390Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5028lb # [ 50.247107] niks3-server[982]: 2026/09/21 14:44:01 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5029lb # [ 50.255096] niks3-server[982]: 2026/09/21 14:44:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign5030lb # [ 50.259606] niks3-server[982]: 2026/09/21 14:44:01 INFO Signed narinfos id=13 count=15031worker1 # [ 50.262307] nix-grpc-daemon[876]: time=2026-09-21T14:44:02.429Z level=INFO msg="Uploading 1 narinfos"5032lb # [ 50.266908] niks3-server[982]: 2026/09/21 14:44:01 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5033lb # [ 50.278177] niks3-server[982]: 2026/09/21 14:44:01 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete5034lb # [ 50.284222] niks3-server[982]: 2026/09/21 14:44:01 INFO Completed upload id=135035worker1 # [ 50.286101] nix-grpc-daemon[876]: time=2026-09-21T14:44:02.453Z level=INFO msg="Upload complete. (282ms)"5036worker1 # [ 50.288587] nix-grpc-daemon[780]: ts=2026-09-21T14:44:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=0 paths=1 nar_bytes_in=6245037lb # [ 50.293356] niks3-server[982]: 2026/09/21 14:44:01 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5038worker1 # [ 50.294494] nix-grpc-daemon[780]: ts=2026-09-21T14:44:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485225039worker1 # [ 50.305332] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5040worker1 # [ 50.309762] nix-daemon[862]: reaped child process 1128, status = succeeded5041worker1 # [ 50.314057] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5042worker1 # [ 51.760945] nix-daemon[862]: reaped child process 1142, status = succeeded5043worker1 # [ 51.836260] nix-daemon[862]: accepted connection from pid 1159, user nix-grpc-daemon (trusted)5044worker1 # [ 51.849254] nix-daemon[862]: reaped child process 1162, status = succeeded5045lb # [ 51.857885] niks3-server[982]: 2026/09/21 14:44:03 INFO Received uploads request method=POST path=/api/pending_closures5046worker1 # [ 51.865937] nix-grpc-daemon[876]: time=2026-09-21T14:44:04.033Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5047worker1 # [ 51.867652] nix-grpc-daemon[876]: time=2026-09-21T14:44:04.034Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5048lb # [ 51.894793] niks3-server[982]: 2026/09/21 14:44:03 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5049lb # [ 51.906068] niks3-server[982]: 2026/09/21 14:44:03 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5050lb # [ 51.917690] niks3-server[982]: 2026/09/21 14:44:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign5051lb # [ 51.922135] niks3-server[982]: 2026/09/21 14:44:03 INFO Signed narinfos id=14 count=15052worker1 # [ 51.921348] nix-grpc-daemon[876]: time=2026-09-21T14:44:04.088Z level=INFO msg="Uploading 1 narinfos"5053lb # [ 51.927461] niks3-server[982]: 2026/09/21 14:44:03 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5054lb # [ 51.936253] niks3-server[982]: 2026/09/21 14:44:03 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete5055lb # [ 51.942261] niks3-server[982]: 2026/09/21 14:44:03 INFO Completed upload id=145056lb # [ 51.944740] niks3-server[982]: 2026/09/21 14:44:03 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5057worker1 # [ 51.942973] nix-grpc-daemon[876]: time=2026-09-21T14:44:04.110Z level=INFO msg="Upload complete. (220ms)"5058worker1 # [ 51.949733] nix-grpc-daemon[780]: ts=2026-09-21T14:44:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=7 outputs=15059worker1 # [ 51.958774] nix-grpc-daemon[780]: ts=2026-09-21T14:44:04Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:485225060worker1 # [ 51.969372] nix-daemon[862]: reaped child process 1140, status = succeeded5061worker1 # [ 51.976106] nix-grpc-daemon[780]: ts=2026-09-21T14:44:04Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:485225062worker1 # [ 51.979150] nix-grpc-daemon[780]: ts=2026-09-21T14:44:04Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=0 paths=1 nar_bytes_out=1765063worker2 # [ 52.187222] nix-grpc-daemon[782]: ts=2026-09-21T14:44:04Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:373025064worker2 # [ 52.193133] nix-grpc-daemon[782]: ts=2026-09-21T14:44:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:373025065worker2 # [ 52.203831] nix-grpc-daemon[878]: time=2026-09-21T14:44:04.348Z level=INFO msg="All 1 paths already cached"5066worker1 # [ 52.220558] nix-grpc-daemon[780]: ts=2026-09-21T14:44:04Z level=debug event=want client=8 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05067worker1 # [ 52.225952] nix-grpc-daemon[780]: ts=2026-09-21T14:44:04Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=15068worker1 # [ 52.231523] nix-grpc-daemon[780]: ts=2026-09-21T14:44:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485225069worker1 # [ 52.243174] nix-grpc-daemon[780]: ts=2026-09-21T14:44:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485225070worker1 # [ 52.254843] nix-grpc-daemon[876]: time=2026-09-21T14:44:04.422Z level=INFO msg="All 2 paths already cached"5071worker1 # [ 52.262187] nix-grpc-daemon[780]: ts=2026-09-21T14:44:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:485225072worker1 # [ 52.267072] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5073worker1 # [ 52.353760] nix-daemon[862]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5074worker1 # [ 52.366961] nix-daemon[862]: reaped child process 1171, status = succeeded5075worker1 # [ 52.450590] nix-daemon[862]: accepted connection from pid 1173, user nix-grpc-daemon (trusted)5076worker1 # [ 52.464378] nix-daemon[862]: reaped child process 1176, status = succeeded5077lb # [ 52.471882] niks3-server[982]: 2026/09/21 14:44:03 INFO Received uploads request method=POST path=/api/pending_closures5078worker1 # [ 52.479948] nix-grpc-daemon[876]: time=2026-09-21T14:44:04.646Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5079worker1 # [ 52.481320] nix-grpc-daemon[876]: time=2026-09-21T14:44:04.647Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5080lb # [ 52.502508] niks3-server[982]: 2026/09/21 14:44:03 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5081lb # [ 52.512986] niks3-server[982]: 2026/09/21 14:44:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5082worker1 # [ 52.518295] nix-grpc-daemon[876]: time=2026-09-21T14:44:04.685Z level=INFO msg="Uploading 1 narinfos"5083lb # [ 52.519111] niks3-server[982]: 2026/09/21 14:44:03 INFO Signed narinfos id=15 count=15084lb # [ 52.529751] niks3-server[982]: 2026/09/21 14:44:03 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5085lb # [ 52.539339] niks3-server[982]: 2026/09/21 14:44:03 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5086lb # [ 52.542202] niks3-server[982]: 2026/09/21 14:44:03 INFO Completed upload id=155087worker1 # [ 52.541245] nix-grpc-daemon[876]: time=2026-09-21T14:44:04.708Z level=INFO msg="Upload complete. (268ms)"5088lb # [ 52.544557] niks3-server[982]: 2026/09/21 14:44:03 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5089worker1 # [ 52.544976] nix-grpc-daemon[780]: ts=2026-09-21T14:44:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=0 paths=1 nar_bytes_in=7525090worker1 # [ 52.549904] nix-grpc-daemon[780]: ts=2026-09-21T14:44:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485225091worker1 # [ 52.558992] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5092worker1 # [ 52.565576] nix-daemon[862]: reaped child process 1166, status = succeeded5093worker1 # [ 52.569326] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5094worker1 # [ 53.600469] nix-daemon[862]: reaped child process 1180, status = succeeded5095worker1 # [ 53.680405] nix-daemon[862]: accepted connection from pid 1196, user nix-grpc-daemon (trusted)5096worker1 # [ 53.692899] nix-daemon[862]: reaped child process 1199, status = succeeded5097lb # [ 53.703620] niks3-server[982]: 2026/09/21 14:44:05 INFO Received uploads request method=POST path=/api/pending_closures5098worker1 # [ 53.758181] nix-grpc-daemon[876]: time=2026-09-21T14:44:05.925Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5099worker1 # [ 53.763939] nix-grpc-daemon[876]: time=2026-09-21T14:44:05.928Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5100lb # [ 53.797441] niks3-server[982]: 2026/09/21 14:44:05 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5101lb # [ 53.813461] niks3-server[982]: 2026/09/21 14:44:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5102lb # [ 53.818370] niks3-server[982]: 2026/09/21 14:44:05 INFO Signed narinfos id=16 count=15103worker1 # [ 53.820772] nix-grpc-daemon[876]: time=2026-09-21T14:44:05.987Z level=INFO msg="Uploading 1 narinfos"5104lb # [ 53.824059] niks3-server[982]: 2026/09/21 14:44:05 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5105lb # [ 53.837894] niks3-server[982]: 2026/09/21 14:44:05 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5106lb # [ 53.843501] niks3-server[982]: 2026/09/21 14:44:05 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5107lb # [ 53.850850] niks3-server[982]: 2026/09/21 14:44:05 INFO Completed upload id=165108worker1 # [ 53.850365] nix-grpc-daemon[876]: time=2026-09-21T14:44:06.017Z level=INFO msg="Upload complete. (273ms)"5109lb # [ 53.854582] niks3-server[982]: 2026/09/21 14:44:05 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5110worker1 # [ 53.859268] nix-grpc-daemon[780]: ts=2026-09-21T14:44:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=8 outputs=15111worker1 # [ 53.865593] nix-grpc-daemon[780]: ts=2026-09-21T14:44:06Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:485225112worker1 # [ 53.877215] nix-daemon[862]: reaped child process 1178, status = succeeded5113worker2 # [ 53.880471] nix-grpc-daemon[782]: ts=2026-09-21T14:44:06Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:373025114worker2 # [ 53.891836] nix-grpc-daemon[782]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...5115worker2 # [ 53.905111] nix-grpc-daemon[782]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...5116worker2 # [ 53.918305] nix-grpc-daemon[782]: ts=2026-09-21T14:44:06Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:37302 duration_s=0 paths=1 nar_bytes_out=2325117client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.32 seconds)5118client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5119client # [ 54.028349] nix-daemon[946]: reaped child process 948, status = succeeded5120client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.02 seconds)5121client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5122client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.02 seconds)5123client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5124client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.02 seconds)5125(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 6.38 seconds)5126subtest: an input only one worker has still reaches the builder5127client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only5128client # this derivation will be built:5129client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5130client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5131client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only, in 0.26 seconds)5132client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5133client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.10 seconds)5134worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5135worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.13 seconds)5136worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5137worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)5138client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&25139client # [ 54.698151] nix-daemon[946]: accepted connection from pid 1029, user root (trusted)5140client # this derivation will be built:5141client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5142client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5143worker1 # [ 54.828904] nix-grpc-daemon[780]: ts=2026-09-21T14:44:06Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:485225144worker1 # [ 54.834515] nix-grpc-daemon[780]: ts=2026-09-21T14:44:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485225145worker1 # [ 54.939527] nix-daemon[862]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)5146worker1 # [ 54.950678] nix-daemon[862]: reaped child process 1209, status = succeeded5147lb # [ 54.959377] niks3-server[982]: 2026/09/21 14:44:06 INFO Received uploads request method=POST path=/api/pending_closures5148worker1 # [ 54.963424] nix-grpc-daemon[876]: time=2026-09-21T14:44:07.130Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5149worker1 # [ 54.965201] nix-grpc-daemon[876]: time=2026-09-21T14:44:07.132Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5150lb # [ 54.975442] niks3-server[982]: 2026/09/21 14:44:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5151lb # [ 54.977524] niks3-server[982]: 2026/09/21 14:44:06 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5152lb # [ 54.980249] niks3-server[982]: 2026/09/21 14:44:06 INFO Signed narinfos id=17 count=15153worker1 # [ 54.979480] nix-grpc-daemon[876]: time=2026-09-21T14:44:07.146Z level=INFO msg="Uploading 1 narinfos"5154lb # [ 54.986187] niks3-server[982]: 2026/09/21 14:44:06 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5155lb # [ 54.988257] niks3-server[982]: 2026/09/21 14:44:06 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5156lb # [ 54.992870] niks3-server[982]: 2026/09/21 14:44:06 INFO Completed upload id=175157worker1 # [ 54.991912] nix-grpc-daemon[876]: time=2026-09-21T14:44:07.159Z level=INFO msg="Upload complete. (153ms)"5158lb # [ 54.994226] niks3-server[982]: 2026/09/21 14:44:06 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5159worker1 # [ 55.006568] nix-grpc-daemon[780]: ts=2026-09-21T14:44:07Z level=debug event=want client=9 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05160worker1 # [ 55.008559] nix-grpc-daemon[780]: ts=2026-09-21T14:44:07Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=15161worker2 # [ 55.009409] nix-grpc-daemon[782]: ts=2026-09-21T14:44:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:373025162worker2 # [ 55.020238] nix-grpc-daemon[782]: ts=2026-09-21T14:44:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:373025163client # copying 2 paths to 'lb:50051'...5164client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5165worker2 # [ 55.031088] nix-grpc-daemon[782]: ts=2026-09-21T14:44:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:373025166client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5167worker2 # [ 55.043981] nix-daemon[850]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5168worker2 # [ 55.150591] nix-daemon[850]: accepted connection from pid 955, user nix-grpc-daemon (trusted)5169worker2 # [ 55.165412] nix-daemon[850]: reaped child process 958, status = succeeded5170worker2 # [ 55.249254] nix-daemon[850]: accepted connection from pid 960, user nix-grpc-daemon (trusted)5171worker2 # [ 55.265333] nix-daemon[850]: reaped child process 963, status = succeeded5172lb # [ 55.275529] niks3-server[982]: 2026/09/21 14:44:06 INFO Received uploads request method=POST path=/api/pending_closures5173lb # [ 55.284204] niks3-server[982]: 2026/09/21 14:44:06 INFO Received uploads request method=POST path=/api/pending_closures5174worker2 # [ 55.289331] nix-grpc-daemon[878]: time=2026-09-21T14:44:07.434Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5175worker2 # [ 55.290686] nix-grpc-daemon[878]: time=2026-09-21T14:44:07.434Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5176worker2 # [ 55.292540] nix-grpc-daemon[878]: time=2026-09-21T14:44:07.434Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5177lb # [ 55.331878] niks3-server[982]: 2026/09/21 14:44:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5178lb # [ 55.559521] niks3-server[982]: 2026/09/21 14:44:06 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5179lb # [ 55.563178] niks3-server[982]: 2026/09/21 14:44:06 INFO Signed narinfos id=18 count=15180lb # [ 55.565092] niks3-server[982]: 2026/09/21 14:44:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5181lb # [ 55.567135] niks3-server[982]: 2026/09/21 14:44:06 INFO Signed narinfos id=19 count=25182lb # [ 55.569482] niks3-server[982]: 2026/09/21 14:44:06 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5183worker2 # [ 55.566284] nix-grpc-daemon[878]: time=2026-09-21T14:44:07.710Z level=INFO msg="Uploading 3 narinfos"5184lb # [ 55.572056] niks3-server[982]: 2026/09/21 14:44:06 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5185lb # [ 55.580549] niks3-server[982]: 2026/09/21 14:44:07 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5186lb # [ 55.587191] niks3-server[982]: 2026/09/21 14:44:07 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5187lb # [ 55.599946] niks3-server[982]: 2026/09/21 14:44:07 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5188lb # [ 55.607144] niks3-server[982]: 2026/09/21 14:44:07 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5189lb # [ 55.620564] niks3-server[982]: 2026/09/21 14:44:07 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5190lb # [ 55.624057] niks3-server[982]: 2026/09/21 14:44:07 INFO Completed upload id=185191lb # [ 55.625492] niks3-server[982]: 2026/09/21 14:44:07 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5192lb # [ 55.632803] niks3-server[982]: 2026/09/21 14:44:07 INFO Completed upload id=195193worker2 # [ 55.629784] nix-grpc-daemon[878]: time=2026-09-21T14:44:07.775Z level=INFO msg="Upload complete. (572ms)"5194worker2 # [ 55.634665] nix-grpc-daemon[782]: ts=2026-09-21T14:44:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37302 duration_s=0 paths=2 nar_bytes_in=9765195worker2 # [ 55.641350] nix-grpc-daemon[782]: ts=2026-09-21T14:44:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:373025196worker2 # [ 55.651643] nix-daemon[850]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5197worker2 # [ 55.663461] nix-daemon[850]: reaped child process 953, status = succeeded5198worker2 # [ 55.675263] nix-grpc-daemon[782]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5199client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5200client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5201worker2 # [ 55.690218] nix-daemon[850]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5202client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5203worker2 # [ 56.305849] nix-daemon[850]: reaped child process 968, status = succeeded5204worker2 # [ 56.382323] nix-daemon[850]: accepted connection from pid 984, user nix-grpc-daemon (trusted)5205worker2 # [ 56.394950] nix-daemon[850]: reaped child process 987, status = succeeded5206lb # [ 56.406912] niks3-server[982]: 2026/09/21 14:44:07 INFO Received uploads request method=POST path=/api/pending_closures5207worker2 # [ 56.411435] nix-grpc-daemon[878]: time=2026-09-21T14:44:08.556Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5208lb # [ 56.437265] niks3-server[982]: 2026/09/21 14:44:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5209lb # [ 56.447256] niks3-server[982]: 2026/09/21 14:44:07 INFO Signed narinfos id=20 count=15210worker2 # [ 56.447151] nix-grpc-daemon[878]: time=2026-09-21T14:44:08.592Z level=INFO msg="Uploading 1 narinfos"5211lb # [ 56.453301] niks3-server[982]: 2026/09/21 14:44:07 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5212lb # [ 56.458064] niks3-server[982]: 2026/09/21 14:44:07 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5213lb # [ 56.470891] niks3-server[982]: 2026/09/21 14:44:07 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5214worker2 # [ 56.473806] nix-grpc-daemon[878]: time=2026-09-21T14:44:08.619Z level=INFO msg="Upload complete. (173ms)"5215lb # [ 56.477274] niks3-server[982]: 2026/09/21 14:44:07 INFO Completed upload id=205216client # copying 1 paths...5217worker2 # [ 56.478240] nix-grpc-daemon[782]: ts=2026-09-21T14:44:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37302 duration_s=0 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=9 outputs=15218worker1 # [ 56.485481] nix-grpc-daemon[780]: ts=2026-09-21T14:44:08Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:485225219lb # [ 56.486081] niks3-server[982]: 2026/09/21 14:44:07 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5220worker1 # [ 56.491282] nix-grpc-daemon[780]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5221worker2 # [ 56.489731] nix-daemon[850]: reaped child process 965, status = succeeded5222client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5223worker2 # [ 56.510121] nix-grpc-daemon[782]: ts=2026-09-21T14:44:08Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:373025224worker2 # [ 56.514211] nix-grpc-daemon[782]: ts=2026-09-21T14:44:08Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:37302 duration_s=0 paths=1 nar_bytes_out=1205225lb # [ 56.601652] envoy[917]: conn peer=[2001:db8:1::1]:56030 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1440 tx=1085 ms=34725226lb # [ 56.605404] envoy[917]: conn peer=[2001:db8:1::1]:53942 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3011 tx=1451 ms=46015227lb # [ 56.609070] envoy[917]: conn peer=[2001:db8:1::1]:56046 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3423 tx=1716 ms=17175228lb # [ 56.612576] envoy[917]: conn peer=[2001:db8:1::1]:56048 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=334 ms=795229lb # [ 56.616067] envoy[917]: conn peer=[2001:db8:1::1]:56070 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=515230lb # [ 56.620235] envoy[917]: conn peer=[2001:db8:1::1]:56062 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3893 tx=1933 ms=21185231lb # [ 56.623809] envoy[917]: conn peer=[2001:db8:1::1]:56084 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4341 tx=2160 ms=17845232client # [ 56.617331] nix-daemon[946]: reaped child process 1032, status = succeeded5233lb # [ 56.627310] envoy[917]: conn peer=[2001:db8:1::1]:32982 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=411 ms=905234lb # [ 56.630912] envoy[917]: conn peer=[2001:db8:1::1]:32996 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3862 tx=1934 ms=17415235client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.04 seconds)5236lb # [ 56.634372] envoy[917]: conn peer=[2001:db8:1::1]:33008 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=565237client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&25238client # [ 56.724624] nix-daemon[946]: accepted connection from pid 1050, user root (trusted)5239client # this derivation will be built:5240client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5241worker1 # [ 56.848425] nix-grpc-daemon[780]: ts=2026-09-21T14:44:09Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:485225242client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5243worker2 # [ 56.849154] nix-grpc-daemon[782]: ts=2026-09-21T14:44:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:373025244worker2 # [ 56.857484] nix-grpc-daemon[878]: time=2026-09-21T14:44:09.002Z level=INFO msg="All 1 paths already cached"5245worker1 # [ 56.870031] nix-grpc-daemon[780]: ts=2026-09-21T14:44:09Z level=debug event=want client=10 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05246worker1 # [ 56.872390] nix-grpc-daemon[780]: ts=2026-09-21T14:44:09Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15247worker1 # [ 56.874927] nix-grpc-daemon[780]: ts=2026-09-21T14:44:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485225248worker1 # [ 56.880685] nix-grpc-daemon[780]: ts=2026-09-21T14:44:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485225249client # copying 1 paths to 'lb:50051'...5250client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5251worker1 # [ 56.888320] nix-grpc-daemon[780]: ts=2026-09-21T14:44:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:485225252worker1 # [ 56.893458] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5253worker1 # [ 56.900434] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 43 ms (attempt 1/5)5254worker1 # [ 56.946708] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 191 ms (attempt 2/5)5255worker1 # [ 57.140175] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 124 ms (attempt 3/5)5256worker1 # [ 57.266062] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 256 ms (attempt 4/5)5257worker1 # [ 57.524959] nix-grpc-daemon[780]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5258worker1 # [ 57.528941] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5259worker1 # [ 57.534319] nix-grpc-daemon[780]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5260worker1 # [ 57.655089] nix-daemon[862]: accepted connection from pid 1223, user nix-grpc-daemon (trusted)5261worker1 # [ 57.671108] nix-daemon[862]: reaped child process 1226, status = succeeded5262worker1 # [ 57.751955] nix-daemon[862]: accepted connection from pid 1228, user nix-grpc-daemon (trusted)5263worker1 # [ 57.768102] nix-daemon[862]: reaped child process 1231, status = succeeded5264lb # [ 57.774605] niks3-server[982]: 2026/09/21 14:44:09 INFO Received uploads request method=POST path=/api/pending_closures5265worker1 # [ 57.778868] nix-grpc-daemon[876]: time=2026-09-21T14:44:09.945Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5266worker1 # [ 57.780622] nix-grpc-daemon[876]: time=2026-09-21T14:44:09.947Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5267lb # [ 57.791756] niks3-server[982]: 2026/09/21 14:44:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5268lb # [ 57.793869] niks3-server[982]: 2026/09/21 14:44:09 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5269lb # [ 57.796811] niks3-server[982]: 2026/09/21 14:44:09 INFO Signed narinfos id=21 count=15270worker1 # [ 57.795644] nix-grpc-daemon[876]: time=2026-09-21T14:44:09.962Z level=INFO msg="Uploading 1 narinfos"5271lb # [ 57.801064] niks3-server[982]: 2026/09/21 14:44:09 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5272lb # [ 57.803484] niks3-server[982]: 2026/09/21 14:44:09 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5273lb # [ 57.808311] niks3-server[982]: 2026/09/21 14:44:09 INFO Completed upload id=215274worker1 # [ 57.807377] nix-grpc-daemon[876]: time=2026-09-21T14:44:09.974Z level=INFO msg="Upload complete. (255ms)"5275lb # [ 57.810355] niks3-server[982]: 2026/09/21 14:44:09 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5276worker1 # [ 57.809270] nix-grpc-daemon[780]: ts=2026-09-21T14:44:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=0 paths=1 nar_bytes_in=5605277worker1 # [ 57.814149] nix-grpc-daemon[780]: ts=2026-09-21T14:44:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:485225278worker1 # [ 57.819397] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5279worker1 # [ 57.821608] nix-daemon[862]: reaped child process 1216, status = succeeded5280worker1 # [ 57.826792] nix-daemon[862]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5281client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5282client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5283client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5284worker1 # [ 59.267942] nix-daemon[862]: reaped child process 1235, status = succeeded5285worker1 # [ 59.343699] nix-daemon[862]: accepted connection from pid 1251, user nix-grpc-daemon (trusted)5286worker1 # [ 59.354869] nix-daemon[862]: reaped child process 1254, status = succeeded5287lb # [ 59.364072] niks3-server[982]: 2026/09/21 14:44:10 INFO Received uploads request method=POST path=/api/pending_closures5288worker1 # [ 59.371315] nix-grpc-daemon[876]: time=2026-09-21T14:44:11.538Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5289lb # [ 59.396107] niks3-server[982]: 2026/09/21 14:44:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5290lb # [ 59.401521] niks3-server[982]: 2026/09/21 14:44:10 INFO Signed narinfos id=22 count=15291worker1 # [ 59.403853] nix-grpc-daemon[876]: time=2026-09-21T14:44:11.571Z level=INFO msg="Uploading 1 narinfos"5292lb # [ 59.408862] niks3-server[982]: 2026/09/21 14:44:10 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5293lb # [ 59.414120] niks3-server[982]: 2026/09/21 14:44:10 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5294lb # [ 59.429216] niks3-server[982]: 2026/09/21 14:44:10 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5295lb # [ 59.435356] niks3-server[982]: 2026/09/21 14:44:10 INFO Completed upload id=225296worker1 # [ 59.436389] nix-grpc-daemon[876]: time=2026-09-21T14:44:11.603Z level=INFO msg="Upload complete. (201ms)"5297lb # [ 59.438083] niks3-server[982]: 2026/09/21 14:44:10 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5298worker1 # [ 59.438939] nix-grpc-daemon[780]: ts=2026-09-21T14:44:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=10 outputs=15299client # copying 1 paths...5300worker1 # [ 59.447357] nix-grpc-daemon[780]: ts=2026-09-21T14:44:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:485225301client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5302worker1 # [ 59.450438] nix-daemon[862]: reaped child process 1233, status = succeeded5303worker1 # [ 59.461597] nix-grpc-daemon[780]: ts=2026-09-21T14:44:11Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:485225304worker1 # [ 59.463862] nix-grpc-daemon[780]: ts=2026-09-21T14:44:11Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:48522 duration_s=0 paths=1 nar_bytes_out=1205305client # [ 59.563690] nix-daemon[946]: reaped child process 1053, status = succeeded5306client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.94 seconds)5307client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&25308client # [ 59.667896] nix-daemon[946]: accepted connection from pid 1072, user root (trusted)5309client # this derivation will be built:5310client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5311worker2 # [ 59.795702] nix-grpc-daemon[782]: ts=2026-09-21T14:44:11Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:373025312client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5313worker1 # [ 59.805167] nix-grpc-daemon[780]: ts=2026-09-21T14:44:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485225314worker1 # [ 59.814830] nix-grpc-daemon[876]: time=2026-09-21T14:44:11.982Z level=INFO msg="All 1 paths already cached"5315worker1 # [ 59.828237] nix-grpc-daemon[780]: ts=2026-09-21T14:44:11Z level=debug event=want client=11 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05316worker1 # [ 59.832427] nix-grpc-daemon[780]: ts=2026-09-21T14:44:11Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15317worker2 # [ 59.838112] nix-grpc-daemon[782]: ts=2026-09-21T14:44:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:373025318worker2 # [ 59.849698] nix-grpc-daemon[782]: ts=2026-09-21T14:44:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:373025319worker2 # [ 59.862564] nix-grpc-daemon[878]: time=2026-09-21T14:44:12.007Z level=INFO msg="All 1 paths already cached"5320client # copying 1 paths to 'lb:50051'...5321client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5322worker2 # [ 59.870713] nix-grpc-daemon[782]: ts=2026-09-21T14:44:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:373025323worker2 # [ 59.879736] nix-daemon[850]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5324worker2 # [ 59.981617] nix-daemon[850]: accepted connection from pid 991, user nix-grpc-daemon (trusted)5325worker2 # [ 59.995179] nix-daemon[850]: reaped child process 994, status = succeeded5326worker2 # [ 60.077380] nix-daemon[850]: accepted connection from pid 996, user nix-grpc-daemon (trusted)5327worker2 # [ 60.090758] nix-daemon[850]: reaped child process 999, status = succeeded5328lb # [ 60.100326] niks3-server[982]: 2026/09/21 14:44:11 INFO Received uploads request method=POST path=/api/pending_closures5329worker2 # [ 60.105431] nix-grpc-daemon[878]: time=2026-09-21T14:44:12.250Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5330worker2 # [ 60.106830] nix-grpc-daemon[878]: time=2026-09-21T14:44:12.250Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5331lb # [ 60.131991] niks3-server[982]: 2026/09/21 14:44:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5332lb # [ 60.138312] niks3-server[982]: 2026/09/21 14:44:11 INFO Signed narinfos id=23 count=15333worker2 # [ 60.138108] nix-grpc-daemon[878]: time=2026-09-21T14:44:12.283Z level=INFO msg="Uploading 1 narinfos"5334lb # [ 60.143654] niks3-server[982]: 2026/09/21 14:44:11 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5335lb # [ 60.157467] niks3-server[982]: 2026/09/21 14:44:11 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5336lb # [ 60.164356] niks3-server[982]: 2026/09/21 14:44:11 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5337lb # [ 60.169646] niks3-server[982]: 2026/09/21 14:44:11 INFO Completed upload id=235338worker2 # [ 60.168618] nix-grpc-daemon[878]: time=2026-09-21T14:44:12.314Z level=INFO msg="Upload complete. (274ms)"5339lb # [ 60.175331] niks3-server[982]: 2026/09/21 14:44:11 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5340worker2 # [ 60.172414] nix-grpc-daemon[782]: ts=2026-09-21T14:44:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37302 duration_s=0 paths=1 nar_bytes_in=5605341worker2 # [ 60.178705] nix-grpc-daemon[782]: ts=2026-09-21T14:44:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:373025342worker2 # [ 60.189578] nix-daemon[850]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5343worker2 # [ 60.192403] nix-daemon[850]: reaped child process 989, status = succeeded5344worker2 # [ 60.198540] nix-daemon[850]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5345client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5346client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5347client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5348worker2 # [ 61.038147] nix-daemon[850]: reaped child process 1003, status = succeeded5349worker2 # [ 61.121401] nix-daemon[850]: accepted connection from pid 1019, user nix-grpc-daemon (trusted)5350worker2 # [ 61.133526] nix-daemon[850]: reaped child process 1022, status = succeeded5351lb # [ 61.144402] niks3-server[982]: 2026/09/21 14:44:12 INFO Received uploads request method=POST path=/api/pending_closures5352worker2 # [ 61.148830] nix-grpc-daemon[878]: time=2026-09-21T14:44:13.294Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5353lb # [ 61.174121] niks3-server[982]: 2026/09/21 14:44:12 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5354lb # [ 61.181308] niks3-server[982]: 2026/09/21 14:44:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5355lb # [ 61.188663] niks3-server[982]: 2026/09/21 14:44:12 INFO Signed narinfos id=24 count=15356worker2 # [ 61.188331] nix-grpc-daemon[878]: time=2026-09-21T14:44:13.333Z level=INFO msg="Uploading 1 narinfos"5357lb # [ 61.196389] niks3-server[982]: 2026/09/21 14:44:12 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5358lb # [ 61.208734] niks3-server[982]: 2026/09/21 14:44:12 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5359lb # [ 61.214509] niks3-server[982]: 2026/09/21 14:44:12 INFO Completed upload id=245360worker2 # [ 61.211151] nix-grpc-daemon[878]: time=2026-09-21T14:44:13.356Z level=INFO msg="Upload complete. (205ms)"5361lb # [ 61.221771] niks3-server[982]: 2026/09/21 14:44:12 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5362worker2 # [ 61.215983] nix-grpc-daemon[782]: ts=2026-09-21T14:44:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37302 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=11 outputs=15363worker2 # [ 61.229666] nix-daemon[850]: reaped child process 1001, status = succeeded5364client # copying 1 paths...5365worker1 # [ 61.266984] nix-grpc-daemon[780]: ts=2026-09-21T14:44:13Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:485225366worker1 # [ 61.274377] nix-grpc-daemon[780]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5367client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5368worker2 # [ 61.305291] nix-grpc-daemon[782]: ts=2026-09-21T14:44:13Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:373025369worker2 # [ 61.309788] nix-grpc-daemon[782]: ts=2026-09-21T14:44:13Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:37302 duration_s=0 paths=1 nar_bytes_out=1205370client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&2, in 1.80 seconds)5371(finished: subtest: an input only one worker has still reaches the builder, in 7.29 seconds)5372subtest: upload whose reference the worker lacks is completed from the cache5373client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache5374client # [ 61.415507] nix-daemon[946]: reaped child process 1075, status = succeeded5375client # this derivation will be built:5376client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5377client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5378client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache, in 0.26 seconds)5379client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache5380client # this derivation will be built:5381client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5382client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5383client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache, in 0.24 seconds)5384client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5385client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.10 seconds)5386worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5387worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.12 seconds)5388worker2 # [ 62.147858] systemd[1]: Stopping Nix worker-protocol over gRPC...5389worker2 # [ 62.828824] nix-grpc-daemon[782]: ts=2026-09-21T14:44:14Z level=info event=signal_exit5390worker1 # [ 64.599747] nix-grpc-daemon[780]: ts=2026-09-21T14:44:16Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:485065391worker1 # [ 64.603864] nix-grpc-daemon[780]: ts=2026-09-21T14:44:16Z level=info event=worker_gone addr=192.168.1.4:500515392worker2 # [ 64.596781] nix-grpc-daemon[782]: ts=2026-09-21T14:44:16Z level=info event=scheduler_restarting5393worker2 # [ 64.612416] nix-grpc-daemon[782]: ts=2026-09-21T14:44:16Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5394worker2 # [ 64.632269] nix-daemon[850]: reaped child process 852, status = succeeded5395worker2 # [ 64.919606] nix-grpc-daemon[782]: E0000 00:00:1790001857.064724 827 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5396worker2 # [ 64.924493] nix-grpc-daemon[782]: E0000 00:00:1790001857.065044 827 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5397worker2 # [ 65.751568] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5398worker2 # [ 65.755389] systemd[1]: Stopped Nix worker-protocol over gRPC.5399worker2 # [ 65.759549] systemd[1]: nix-grpc-daemon.service: Consumed 1.295s CPU time over 49.073s wall clock time, 56.3M memory peak, 52K incoming IP traffic, 141.6K outgoing IP traffic.5400worker2 # [ 65.771190] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5401worker2 # [ 65.775420] systemd[1]: Closed Nix worker-protocol over gRPC.5402worker2 # [ 65.779270] systemd[1]: nix-grpc-daemon.socket: Consumed 304us CPU time over 58.153s wall clock time, 256K memory peak, 52.1K incoming IP traffic, 44K outgoing IP traffic.5403lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5404lb: must succeed: curl -sf localhost:9901/clusters5405lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5406lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.05 seconds)5407client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25408worker1 # [ 65.984444] nix-grpc-daemon[780]: ts=2026-09-21T14:44:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:485225409client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5410worker1 # [ 65.987581] nix-grpc-daemon[780]: ts=2026-09-21T14:44:18Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:485225411client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.16 seconds)5412worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5413worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.02 seconds)5414worker2 # [ 66.063777] systemd[1]: Listening on Nix worker-protocol over gRPC.5415worker1 # [ 66.109564] systemd[1]: Stopping Nix worker-protocol over gRPC...5416lb # [ 66.602171] envoy[917]: conn peer=[2001:db8:1::1]:33024 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3741 tx=1895 ms=26665417lb # [ 66.605743] envoy[917]: conn peer=[2001:db8:1::1]:33034 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=525418lb # [ 66.609126] envoy[917]: conn peer=[2001:db8:1::1]:33036 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3755 tx=1990 ms=15265419lb # [ 66.612846] envoy[917]: conn peer=[2001:db8:1::1]:33050 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=195420lb # [ 66.616500] envoy[917]: conn peer=[2001:db8:1::4]:59562 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=58688 tx=2730 ms=477395421lb # [ 66.620145] envoy[917]: conn peer=[2001:db8:1::1]:36956 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=519 ms=115422worker1 # [ 66.749146] nix-grpc-daemon[780]: ts=2026-09-21T14:44:18Z level=info event=signal_exit5423worker2 # [ 66.748682] systemd[1]: Starting scheduler-token.service...5424worker2 # [ 66.809210] systemd[1]: scheduler-token.service: Deactivated successfully.5425worker2 # [ 66.812462] systemd[1]: Finished scheduler-token.service.5426worker2 # [ 66.815329] systemd[1]: scheduler-token.service: Consumed 34ms CPU time over 63ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5427worker2 # [ 66.824138] systemd[1]: Starting Nix worker-protocol over gRPC...5428worker2 # [ 66.898672] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5429worker2 # [ 66.900608] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5430worker2 # [ 66.903677] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15431worker2 # [ 66.907695] nix-grpc-daemon[1046]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5432worker2 # [ 66.910196] nix-grpc-daemon[1046]: E0000 00:00:1790001859.053231 1046 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5433worker2 # [ 66.915075] nix-grpc-daemon[1046]: E0000 00:00:1790001859.058585 1046 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5434worker2 # [ 66.919192] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5435worker2 # [ 66.922554] systemd[1]: Started Nix worker-protocol over gRPC.5436worker2 # [ 66.926122] nix-grpc-daemon[1046]: E0000 00:00:1790001859.070783 1088 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5437worker2 # [ 66.939841] nix-grpc-daemon[1046]: E0000 00:00:1790001859.085293 1088 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5438worker2 # [ 66.944056] nix-grpc-daemon[1046]: E0000 00:00:1790001859.087677 1088 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5439worker2 # [ 66.953794] nix-grpc-daemon[1046]: E0000 00:00:1790001859.099285 1088 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5440worker2 # [ 66.962420] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=scheduler_connected addr=lb:500515441worker2 # [ 66.963873] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5442worker2 # [ 67.165224] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=scheduler_connected addr=lb:500515443worker2 # [ 67.169184] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5444worker2 # [ 67.370233] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=scheduler_connected addr=lb:500515445worker2 # [ 67.374265] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5446worker2 # [ 67.575414] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=scheduler_connected addr=lb:500515447worker2 # [ 67.579263] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5448lb # [ 67.594797] niks3-server[982]: 2026/09/21 14:44:19 INFO lead: released remote=192.168.1.3:487305449worker2 # [ 67.780331] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=scheduler_connected addr=lb:500515450worker2 # [ 67.784201] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5451worker2 # [ 67.986316] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:20Z level=info event=scheduler_connected addr=lb:500515452worker2 # [ 67.989529] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5453worker2 # [ 68.187537] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:20Z level=info event=scheduler_connected addr=lb:500515454worker2 # [ 68.191404] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5455worker2 # [ 68.392324] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:20Z level=info event=scheduler_connected addr=lb:500515456worker2 # [ 68.396192] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5457worker1 # [ 68.598951] nix-grpc-daemon[780]: ts=2026-09-21T14:44:20Z level=info event=scheduler_restarting5458worker2 # [ 68.597172] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:20Z level=info event=scheduler_connected addr=lb:500515459worker1 # [ 68.603501] nix-grpc-daemon[780]: ts=2026-09-21T14:44:20Z level=info event=scheduler_restarting addr=lb:500515460worker2 # [ 68.601174] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5461worker1 # [ 68.607266] nix-grpc-daemon[780]: ts=2026-09-21T14:44:20Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:485065462worker1 # [ 68.611433] nix-grpc-daemon[780]: ts=2026-09-21T14:44:20Z level=info event=worker_gone addr=192.168.1.3:500515463worker1 # [ 68.633275] nix-daemon[862]: reaped child process 864, status = succeeded5464worker2 # [ 68.802386] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:20Z level=info event=scheduler_connected addr=lb:500515465worker2 # [ 68.806511] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5466worker1 # [ 68.810836] nix-grpc-daemon[780]: ts=2026-09-21T14:44:20Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5467worker1 # [ 68.857242] nix-grpc-daemon[780]: E0000 00:00:1790001861.024192 825 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5468worker1 # [ 68.862092] nix-grpc-daemon[780]: E0000 00:00:1790001861.024520 825 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5469worker2 # [ 69.008806] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:21Z level=info event=scheduler_connected addr=lb:500515470worker2 # [ 69.011925] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5471worker2 # [ 69.209783] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:21Z level=info event=scheduler_connected addr=lb:500515472worker2 # [ 69.213667] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5473worker2 # [ 69.414611] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:21Z level=info event=scheduler_connected addr=lb:500515474worker2 # [ 69.418529] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5475worker2 # [ 69.619785] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:21Z level=info event=scheduler_connected addr=lb:500515476worker2 # [ 69.623606] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5477worker1 # [ 69.711132] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5478worker1 # [ 69.714963] systemd[1]: Stopped Nix worker-protocol over gRPC.5479worker1 # [ 69.718523] systemd[1]: nix-grpc-daemon.service: Consumed 2.623s CPU time over 53.116s wall clock time, 69.6M memory peak, 98.7K incoming IP traffic, 222.1K outgoing IP traffic.5480worker1 # [ 69.730585] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5481worker1 # [ 69.734850] systemd[1]: Closed Nix worker-protocol over gRPC.5482worker1 # [ 69.737933] systemd[1]: nix-grpc-daemon.socket: Consumed 317us CPU time over 1min 2.156s wall clock time, 256K memory peak, 92.3K incoming IP traffic, 76.9K outgoing IP traffic.5483lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5484lb: must succeed: curl -sf localhost:9901/clusters5485lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5486lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.03 seconds)5487client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25488worker2 # [ 69.824644] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:21Z level=info event=scheduler_connected addr=lb:500515489worker2 # [ 69.828574] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5490worker2 # [ 69.916984] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395905491worker2 # [ 69.928404] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)5492worker2 # [ 69.946888] nix-grpc-daemon[1046]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 50 ms (attempt 1/5)5493worker2 # [ 70.004374] nix-grpc-daemon[1046]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 35 ms (attempt 2/5)5494worker2 # [ 70.030089] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=info event=scheduler_connected addr=lb:500515495worker2 # [ 70.033640] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5496worker2 # [ 70.040831] nix-grpc-daemon[1046]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 21 ms (attempt 3/5)5497worker2 # [ 70.068983] nix-grpc-daemon[1046]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 563 ms (attempt 4/5)5498worker2 # [ 70.234921] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=info event=scheduler_connected addr=lb:500515499worker2 # [ 70.238848] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5500worker2 # [ 70.439761] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=info event=scheduler_connected addr=lb:500515501worker2 # [ 70.443780] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5502worker2 # [ 70.634658] nix-grpc-daemon[1046]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5503worker2 # [ 70.637993] nix-grpc-daemon[1046]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5504worker2 # [ 70.644513] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=info event=scheduler_connected addr=lb:500515505worker2 # [ 70.648284] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5506client # copying 2 paths...5507client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5508worker2 # [ 70.663474] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395905509client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5510worker2 # [ 70.673737] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)5511worker2 # [ 70.700765] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=info event=niks3_push_started pid=11065512worker2 # [ 70.795410] nix-grpc-daemon[1106]: time=2026-09-21T14:44:22.940Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""5513worker2 # [ 70.849474] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=info event=scheduler_connected addr=lb:500515514worker2 # [ 70.853172] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5515worker2 # [ 70.881085] nix-daemon[850]: accepted connection from pid 1119, user nix-grpc-daemon (trusted)5516worker2 # [ 70.893218] nix-daemon[850]: reaped child process 1122, status = succeeded5517lb # [ 70.902808] niks3-server[982]: 2026/09/21 14:44:22 INFO Received uploads request method=POST path=/api/pending_closures5518lb # [ 70.910225] niks3-server[982]: 2026/09/21 14:44:22 INFO Received uploads request method=POST path=/api/pending_closures5519worker2 # [ 70.910108] nix-grpc-daemon[1106]: time=2026-09-21T14:44:23.055Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5520worker2 # [ 70.911710] nix-grpc-daemon[1106]: time=2026-09-21T14:44:23.057Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5521worker2 # [ 70.913919] nix-grpc-daemon[1106]: time=2026-09-21T14:44:23.059Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5522lb # [ 70.939112] niks3-server[982]: 2026/09/21 14:44:22 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5523lb # [ 70.946459] niks3-server[982]: 2026/09/21 14:44:22 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5524lb # [ 70.949122] niks3-server[982]: 2026/09/21 14:44:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5525lb # [ 70.950745] niks3-server[982]: 2026/09/21 14:44:22 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5526lb # [ 70.953513] niks3-server[982]: 2026/09/21 14:44:22 INFO Signed narinfos id=25 count=15527lb # [ 70.955234] niks3-server[982]: 2026/09/21 14:44:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5528lb # [ 70.957257] niks3-server[982]: 2026/09/21 14:44:22 INFO Signed narinfos id=26 count=25529worker2 # [ 70.955670] nix-grpc-daemon[1106]: time=2026-09-21T14:44:23.100Z level=INFO msg="Uploading 3 narinfos"5530lb # [ 70.960531] niks3-server[982]: 2026/09/21 14:44:22 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5531lb # [ 70.970556] niks3-server[982]: 2026/09/21 14:44:22 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5532lb # [ 70.976260] niks3-server[982]: 2026/09/21 14:44:22 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5533lb # [ 70.979584] niks3-server[982]: 2026/09/21 14:44:22 INFO Completed upload id=255534lb # [ 70.981101] niks3-server[982]: 2026/09/21 14:44:22 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5535lb # [ 70.982747] niks3-server[982]: 2026/09/21 14:44:22 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5536lb # [ 70.986222] niks3-server[982]: 2026/09/21 14:44:22 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5537lb # [ 70.988462] niks3-server[982]: 2026/09/21 14:44:22 INFO Completed upload id=265538worker2 # [ 70.986209] nix-grpc-daemon[1106]: time=2026-09-21T14:44:23.131Z level=INFO msg="Upload complete. (187ms)"5539worker2 # [ 70.989946] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39590 duration_s=0 paths=2 nar_bytes_in=3125540worker2 # [ 70.999667] nix-daemon[850]: reaped child process 1104, status = succeeded5541client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.25 seconds)5542worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5543worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.01 seconds)5544worker2 # [ 71.052324] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:23Z level=info event=scheduler_connected addr=lb:500515545worker2 # [ 71.054068] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5546worker1 # [ 71.059200] systemd[1]: Listening on Nix worker-protocol over gRPC.5547worker1 # [ 71.063067] systemd[1]: Starting Nix worker-protocol over gRPC...5548worker1 # [ 71.138862] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5549worker1 # [ 71.140931] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5550worker1 # [ 71.144489] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15551worker1 # [ 71.149062] nix-grpc-daemon[1280]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5552worker1 # [ 71.150405] nix-grpc-daemon[1280]: E0000 00:00:1790001863.315521 1280 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5553worker1 # [ 71.155787] nix-grpc-daemon[1280]: E0000 00:00:1790001863.320929 1280 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5554worker1 # [ 71.159822] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5555worker1 # [ 71.162344] systemd[1]: Started Nix worker-protocol over gRPC.5556lb # [ 71.165292] niks3-server[982]: 2026/09/21 14:44:22 INFO lead: acquired remote=192.168.1.3:389805557worker1 # [ 71.164531] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=scheduler_take_over5558worker1 # [ 71.167077] nix-grpc-daemon[1280]: E0000 00:00:1790001863.333543 1319 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5559lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5560lb: must succeed: curl -sf localhost:9901/clusters5561worker1 # [ 71.179881] nix-grpc-daemon[1280]: E0000 00:00:1790001863.347172 1319 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5562worker1 # [ 71.183171] nix-grpc-daemon[1280]: E0000 00:00:1790001863.349341 1319 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5563worker1 # [ 71.194056] nix-grpc-daemon[1280]: E0000 00:00:1790001863.361337 1319 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5564worker1 # [ 71.202401] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=scheduler_connected addr=lb:500515565worker1 # [ 71.204095] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5566lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5567worker2 # [ 71.255917] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:23Z level=info event=scheduler_connected addr=lb:500515568worker2 # [ 71.259191] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5569worker1 # [ 71.404970] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=scheduler_connected addr=lb:500515570worker1 # [ 71.409138] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5571worker2 # [ 71.456690] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:23Z level=info event=scheduler_connected addr=lb:500515572worker2 # [ 71.460667] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5573worker1 # [ 71.609791] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=scheduler_connected addr=lb:500515574worker1 # [ 71.613796] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5575worker2 # [ 71.661137] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:23Z level=info event=scheduler_connected addr=lb:500515576worker2 # [ 71.663069] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5577worker1 # [ 71.814785] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=scheduler_connected addr=lb:500515578worker1 # [ 71.822102] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:377585579worker1 # [ 71.826198] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:23Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05580worker2 # [ 71.863689] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:24Z level=info event=scheduler_connected addr=lb:500515581worker1 # [ 71.870055] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:24Z level=info event=oidc_jwks provider=mock keys=15582worker1 # [ 71.871390] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:24Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:377585583worker1 # [ 71.873168] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:24Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05584lb: must succeed: curl -sf localhost:9901/clusters5585lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5586lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.08 seconds)5587(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.87 seconds)5588subtest: a niks3 restart does not cost the leader its role5589lb: must succeed: curl -sf localhost:9901/clusters5590lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5591lb: must succeed: curl -sf localhost:9901/clusters5592lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5593lb # [ 72.324585] niks3-server[982]: 2026/09/21 14:44:23 INFO Shutdown signal received, draining in-flight requests timeout=10s5594worker1 # [ 72.324371] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:24Z level=info event=scheduler_lock_lost status=2005595lb # [ 72.328284] niks3-server[982]: 2026/09/21 14:44:23 INFO lead: released remote=192.168.1.3:389805596worker2 # [ 72.322877] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:24Z level=info event=scheduler_lock_lost status=2005597lb # [ 72.329518] systemd[1]: Stopping niks3 server...5598lb # [ 72.339262] systemd[1]: niks3.service: Deactivated successfully.5599lb # [ 72.340818] systemd[1]: Stopped niks3 server.5600lb # [ 72.341605] systemd[1]: niks3.service: Consumed 552ms CPU time over 52.003s wall clock time, 26.3M memory peak, 69.5K incoming IP traffic, 99.4K outgoing IP traffic.5601lb # [ 72.348903] systemd[1]: Starting niks3 server...5602lb # [ 72.396937] niks3-server[1117]: 2026/09/21 14:44:23 goose: no migrations to run. current version: 202609200000005603lb # [ 72.403224] niks3-server[1117]: 2026/09/21 14:44:23 OK 1_commit_pending_closure.sql (4.32ms)5604lb # [ 72.406651] niks3-server[1117]: 2026/09/21 14:44:23 OK 2_object_stats_trigger.sql (3.11ms)5605lb # [ 72.407752] niks3-server[1117]: 2026/09/21 14:44:23 goose: up to current file version: 25606lb # [ 72.409026] niks3-server[1117]: 2026/09/21 14:44:23 INFO Loaded signing key name=farm-test-1 path=/nix/store/hk0nhypl37jvkn3gr36v3y7qx593p39i-key5607lb # [ 72.419732] niks3-server[1117]: 2026/09/21 14:44:23 INFO Read proxy enabled — serving cache objects from S35608lb # [ 72.420950] niks3-server[1117]: 2026/09/21 14:44:23 INFO Using socket-activated listener address=0.0.0.0:57515609lb # [ 72.423189] systemd[1]: Started niks3 server.5610lb # [ 72.423858] niks3-server[1117]: 2026/09/21 14:44:23 INFO systemd watchdog enabled interval=15s5611lb # [ 72.426100] niks3-server[1117]: 2026/09/21 14:44:23 INFO Starting HTTP server address=0.0.0.0:57515612lb: waiting for unit niks3.service5613lb: (finished: waiting for unit niks3.service, in 0.02 seconds)5614lb: must succeed: curl -sf localhost:9901/clusters5615lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5616worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5617worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5618lb: must succeed: curl -sf localhost:9901/clusters5619lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5620lb: must succeed: curl -sf localhost:9901/clusters5621lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5622worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5623worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.05 seconds)5624client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&25625worker2 # [ 72.743555] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:395905626client # don't know how to build these paths:5627client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5628worker2 # [ 72.752960] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395905629worker1 # [ 72.781419] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:24Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=05630worker1 # [ 72.786333] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:24Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.3:50051 clients=15631worker1 # [ 72.832092] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:605965632worker1 # [ 72.835964] nix-daemon[862]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5633worker1 # [ 72.844459] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 32 ms (attempt 1/5)5634worker1 # [ 72.880786] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 149 ms (attempt 2/5)5635worker1 # [ 73.032486] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 83 ms (attempt 3/5)5636worker1 # [ 73.117418] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 253 ms (attempt 4/5)5637lb # [ 73.332394] niks3-server[1117]: 2026/09/21 14:44:24 INFO lead: acquired remote=192.168.1.3:546205638worker1 # [ 73.378899] nix-grpc-daemon[1280]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5639worker1 # [ 73.383190] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5640worker1 # [ 73.399333] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:605965641client # copying 1 paths to 'lb:50051'...5642client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...5643worker1 # [ 73.413176] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:605965644worker1 # [ 73.423165] nix-daemon[862]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5645worker1 # [ 73.444882] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:25Z level=info event=niks3_push_started pid=13525646worker1 # [ 73.547367] nix-grpc-daemon[1352]: time=2026-09-21T14:44:25.714Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""5647worker1 # [ 73.644463] nix-daemon[862]: accepted connection from pid 1365, user nix-grpc-daemon (trusted)5648worker1 # [ 73.656455] nix-daemon[862]: reaped child process 1368, status = succeeded5649worker1 # [ 73.739858] nix-daemon[862]: accepted connection from pid 1370, user nix-grpc-daemon (trusted)5650worker1 # [ 73.753933] nix-daemon[862]: reaped child process 1373, status = succeeded5651lb # [ 73.759911] niks3-server[1117]: 2026/09/21 14:44:25 INFO Received uploads request method=POST path=/api/pending_closures5652worker1 # [ 73.771253] nix-grpc-daemon[1352]: time=2026-09-21T14:44:25.938Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5653worker1 # [ 73.774376] nix-grpc-daemon[1352]: time=2026-09-21T14:44:25.940Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"5654lb # [ 73.804826] niks3-server[1117]: 2026/09/21 14:44:25 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst5655lb # [ 73.811120] niks3-server[1117]: 2026/09/21 14:44:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5656lb # [ 73.830701] niks3-server[1117]: 2026/09/21 14:44:25 INFO Signed narinfos id=27 count=15657worker1 # [ 73.834510] nix-grpc-daemon[1352]: time=2026-09-21T14:44:26.001Z level=INFO msg="Uploading 1 narinfos"5658lb # [ 73.844293] niks3-server[1117]: 2026/09/21 14:44:25 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5659lb # [ 73.850335] niks3-server[1117]: 2026/09/21 14:44:25 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls5660lb # [ 73.856174] niks3-server[1117]: 2026/09/21 14:44:25 INFO Completed upload id=275661worker1 # [ 73.854863] nix-grpc-daemon[1352]: time=2026-09-21T14:44:26.022Z level=INFO msg="Upload complete. (304ms)"5662worker1 # [ 73.859934] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60596 duration_s=0 paths=1 nar_bytes_in=5365663worker1 # [ 73.867347] nix-grpc-daemon[1352]: time=2026-09-21T14:44:26.026Z level=WARN msg="Failed to register uploaded object" key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo error="server returned 404: object is not pending upload\n"5664worker1 # [ 73.871920] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:605965665worker1 # [ 73.885392] nix-daemon[862]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5666worker1 # [ 73.886750] nix-daemon[862]: reaped child process 1350, status = succeeded5667worker1 # [ 73.892043] nix-daemon[862]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5668client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...5669client # farm-a> node-a: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5670client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5671client # farm-a> LOG-farm-a-after-niks3-restart5672worker1 # [ 74.728889] nix-daemon[862]: reaped child process 1377, status = succeeded5673worker1 # [ 74.813349] nix-daemon[862]: accepted connection from pid 1393, user nix-grpc-daemon (trusted)5674worker1 # [ 74.825925] nix-daemon[862]: reaped child process 1396, status = succeeded5675lb # [ 74.834544] niks3-server[1117]: 2026/09/21 14:44:26 INFO Received uploads request method=POST path=/api/pending_closures5676worker1 # [ 74.845459] nix-grpc-daemon[1352]: time=2026-09-21T14:44:27.012Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5677worker1 # [ 74.846938] nix-grpc-daemon[1352]: time=2026-09-21T14:44:27.014Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"5678lb # [ 74.882342] niks3-server[1117]: 2026/09/21 14:44:26 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5679lb # [ 74.893985] niks3-server[1117]: 2026/09/21 14:44:26 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5680lb # [ 74.900296] niks3-server[1117]: 2026/09/21 14:44:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5681lb # [ 74.904688] niks3-server[1117]: 2026/09/21 14:44:26 INFO Signed narinfos id=28 count=15682worker1 # [ 74.907138] nix-grpc-daemon[1352]: time=2026-09-21T14:44:27.073Z level=INFO msg="Uploading 1 narinfos"5683lb # [ 74.913035] niks3-server[1117]: 2026/09/21 14:44:26 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls5684lb # [ 74.918320] niks3-server[1117]: 2026/09/21 14:44:26 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5685lb # [ 74.923896] niks3-server[1117]: 2026/09/21 14:44:26 INFO Completed upload id=285686worker1 # [ 74.923251] nix-grpc-daemon[1352]: time=2026-09-21T14:44:27.090Z level=INFO msg="Upload complete. (224ms)"5687lb # [ 74.926431] niks3-server[1117]: 2026/09/21 14:44:26 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo5688worker1 # [ 74.932650] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60596 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=15689worker1 # [ 74.944194] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:27Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=05690worker1 # [ 74.947349] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:27Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.3:50051 clients=15691worker1 # [ 74.957715] nix-daemon[862]: reaped child process 1375, status = succeeded5692worker1 # [ 74.987597] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:605965693worker1 # [ 74.999793] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:605965694worker1 # [ 75.013827] nix-grpc-daemon[1352]: time=2026-09-21T14:44:27.181Z level=INFO msg="All 1 paths already cached"5695client # copying 1 paths to 'lb:50051'...5696client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...5697worker1 # [ 75.022219] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:605965698worker1 # [ 75.030543] nix-daemon[862]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5699worker1 # [ 75.127174] nix-daemon[862]: accepted connection from pid 1402, user nix-grpc-daemon (trusted)5700worker1 # [ 75.139774] nix-daemon[862]: reaped child process 1405, status = succeeded5701worker1 # [ 75.221810] nix-daemon[862]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)5702worker1 # [ 75.234819] nix-daemon[862]: reaped child process 1410, status = succeeded5703lb # [ 75.241852] niks3-server[1117]: 2026/09/21 14:44:26 INFO Received uploads request method=POST path=/api/pending_closures5704worker1 # [ 75.249791] nix-grpc-daemon[1352]: time=2026-09-21T14:44:27.417Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5705worker1 # [ 75.251765] nix-grpc-daemon[1352]: time=2026-09-21T14:44:27.418Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"5706lb # [ 75.274343] niks3-server[1117]: 2026/09/21 14:44:26 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst5707lb # [ 75.283376] niks3-server[1117]: 2026/09/21 14:44:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5708worker1 # [ 75.289584] nix-grpc-daemon[1352]: time=2026-09-21T14:44:27.456Z level=INFO msg="Uploading 1 narinfos"5709lb # [ 75.290299] niks3-server[1117]: 2026/09/21 14:44:26 INFO Signed narinfos id=29 count=15710lb # [ 75.301573] niks3-server[1117]: 2026/09/21 14:44:26 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls5711lb # [ 75.310333] niks3-server[1117]: 2026/09/21 14:44:26 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5712lb # [ 75.318580] niks3-server[1117]: 2026/09/21 14:44:26 INFO Completed upload id=295713worker1 # [ 75.317873] nix-grpc-daemon[1352]: time=2026-09-21T14:44:27.485Z level=INFO msg="Upload complete. (274ms)"5714lb # [ 75.321296] niks3-server[1117]: 2026/09/21 14:44:26 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo5715worker1 # [ 75.321684] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60596 duration_s=0 paths=1 nar_bytes_in=6565716worker1 # [ 75.327274] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:605965717worker1 # [ 75.336490] nix-daemon[862]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5718worker1 # [ 75.344533] nix-daemon[862]: reaped child process 1400, status = succeeded5719worker1 # [ 75.348859] nix-daemon[862]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5720client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...5721client # farm-b> node-a: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5722client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5723client # farm-b> LOG-farm-b-after-niks3-restart5724worker1 # [ 76.574174] nix-daemon[862]: reaped child process 1414, status = succeeded5725lb # [ 76.602485] envoy[917]: conn peer=[2001:db8:1::3]:47536 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=15847 tx=3165 ms=520185726lb # [ 76.607154] envoy[917]: conn peer=[2001:db8:1::1]:36970 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1005 tx=281 ms=10865727worker1 # [ 76.652214] nix-daemon[862]: accepted connection from pid 1430, user nix-grpc-daemon (trusted)5728worker1 # [ 76.665342] nix-daemon[862]: reaped child process 1433, status = succeeded5729lb # [ 76.673196] niks3-server[1117]: 2026/09/21 14:44:28 INFO Received uploads request method=POST path=/api/pending_closures5730worker1 # [ 76.681236] nix-grpc-daemon[1352]: time=2026-09-21T14:44:28.848Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5731worker1 # [ 76.682946] nix-grpc-daemon[1352]: time=2026-09-21T14:44:28.850Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"5732lb # [ 76.711145] niks3-server[1117]: 2026/09/21 14:44:28 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5733lb # [ 76.718764] niks3-server[1117]: 2026/09/21 14:44:28 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst5734lb # [ 76.727435] niks3-server[1117]: 2026/09/21 14:44:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5735lb # [ 76.731550] niks3-server[1117]: 2026/09/21 14:44:28 INFO Signed narinfos id=30 count=15736worker1 # [ 76.733191] nix-grpc-daemon[1352]: time=2026-09-21T14:44:28.900Z level=INFO msg="Uploading 1 narinfos"5737lb # [ 76.738438] niks3-server[1117]: 2026/09/21 14:44:28 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls5738lb # [ 76.743145] niks3-server[1117]: 2026/09/21 14:44:28 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5739lb # [ 76.747392] niks3-server[1117]: 2026/09/21 14:44:28 INFO Completed upload id=305740worker1 # [ 76.746336] nix-grpc-daemon[1352]: time=2026-09-21T14:44:28.913Z level=INFO msg="Upload complete. (189ms)"5741lb # [ 76.749306] niks3-server[1117]: 2026/09/21 14:44:28 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo5742worker1 # [ 76.751789] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60596 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=15743worker1 # [ 76.761951] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:28Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=05744worker1 # [ 76.765958] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:28Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.3:50051 clients=15745worker1 # [ 76.773575] nix-daemon[862]: reaped child process 1412, status = succeeded5746worker1 # [ 76.806410] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:605965747worker1 # [ 76.818814] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:605965748worker1 # [ 76.832310] nix-grpc-daemon[1352]: time=2026-09-21T14:44:28.999Z level=INFO msg="All 2 paths already cached"5749client # copying 1 paths to 'lb:50051'...5750client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...5751worker1 # [ 76.841225] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:605965752worker1 # [ 76.851170] nix-daemon[862]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5753worker1 # [ 76.943615] nix-daemon[862]: accepted connection from pid 1439, user nix-grpc-daemon (trusted)5754worker1 # [ 76.956857] nix-daemon[862]: reaped child process 1442, status = succeeded5755worker1 # [ 77.037407] nix-daemon[862]: accepted connection from pid 1444, user nix-grpc-daemon (trusted)5756worker1 # [ 77.049907] nix-daemon[862]: reaped child process 1447, status = succeeded5757lb # [ 77.058815] niks3-server[1117]: 2026/09/21 14:44:28 INFO Received uploads request method=POST path=/api/pending_closures5758worker1 # [ 77.067094] nix-grpc-daemon[1352]: time=2026-09-21T14:44:29.234Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5759worker1 # [ 77.069084] nix-grpc-daemon[1352]: time=2026-09-21T14:44:29.236Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"5760lb # [ 77.089464] niks3-server[1117]: 2026/09/21 14:44:28 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst5761lb # [ 77.099814] niks3-server[1117]: 2026/09/21 14:44:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5762lb # [ 77.104538] niks3-server[1117]: 2026/09/21 14:44:28 INFO Signed narinfos id=31 count=15763worker1 # [ 77.106232] nix-grpc-daemon[1352]: time=2026-09-21T14:44:29.273Z level=INFO msg="Uploading 1 narinfos"5764lb # [ 77.109359] niks3-server[1117]: 2026/09/21 14:44:28 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls5765lb # [ 77.119941] niks3-server[1117]: 2026/09/21 14:44:28 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5766lb # [ 77.125886] niks3-server[1117]: 2026/09/21 14:44:28 INFO Completed upload id=315767worker1 # [ 77.124993] nix-grpc-daemon[1352]: time=2026-09-21T14:44:29.292Z level=INFO msg="Upload complete. (263ms)"5768lb # [ 77.127192] niks3-server[1117]: 2026/09/21 14:44:28 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo5769worker1 # [ 77.127247] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60596 duration_s=0 paths=1 nar_bytes_in=7845770worker1 # [ 77.130748] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:605965771worker1 # [ 77.139563] nix-daemon[862]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5772worker1 # [ 77.145708] nix-daemon[862]: reaped child process 1437, status = succeeded5773worker1 # [ 77.148638] nix-daemon[862]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5774client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...5775client # farm-top> node-a: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5776client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5777client # farm-top> LOG-farm-top-after-niks3-restart5778worker1 # [ 78.187292] nix-daemon[862]: reaped child process 1451, status = succeeded5779worker1 # [ 78.257238] nix-daemon[862]: accepted connection from pid 1467, user nix-grpc-daemon (trusted)5780worker1 # [ 78.269745] nix-daemon[862]: reaped child process 1470, status = succeeded5781lb # [ 78.279379] niks3-server[1117]: 2026/09/21 14:44:29 INFO Received uploads request method=POST path=/api/pending_closures5782worker1 # [ 78.287049] nix-grpc-daemon[1352]: time=2026-09-21T14:44:30.454Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5783worker1 # [ 78.288605] nix-grpc-daemon[1352]: time=2026-09-21T14:44:30.455Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"5784lb # [ 78.319598] niks3-server[1117]: 2026/09/21 14:44:29 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5785lb # [ 78.331131] niks3-server[1117]: 2026/09/21 14:44:29 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst5786lb # [ 78.345850] niks3-server[1117]: 2026/09/21 14:44:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5787worker1 # [ 78.349138] nix-grpc-daemon[1352]: time=2026-09-21T14:44:30.516Z level=INFO msg="Uploading 1 narinfos"5788lb # [ 78.350186] niks3-server[1117]: 2026/09/21 14:44:29 INFO Signed narinfos id=32 count=15789lb # [ 78.359593] niks3-server[1117]: 2026/09/21 14:44:29 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls5790lb # [ 78.362663] niks3-server[1117]: 2026/09/21 14:44:29 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5791lb # [ 78.366937] niks3-server[1117]: 2026/09/21 14:44:29 INFO Completed upload id=325792lb # [ 78.369126] niks3-server[1117]: 2026/09/21 14:44:29 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo5793worker1 # [ 78.367612] nix-grpc-daemon[1352]: time=2026-09-21T14:44:30.534Z level=INFO msg="Upload complete. (211ms)"5794worker1 # [ 78.374762] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60596 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=15795worker1 # [ 78.389711] nix-daemon[862]: reaped child process 1449, status = succeeded5796client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&2, in 5.81 seconds)5797client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath5798client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath, in 0.12 seconds)5799(finished: subtest: a niks3 restart does not cost the leader its role, in 6.28 seconds)5800subtest: scheduler down: the other node takes the lock and keeps it5801lb: must succeed: curl -sf localhost:9901/clusters5802lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5803lb: must succeed: curl -sf localhost:9901/clusters5804lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5805lb: must succeed: curl -sf localhost:9901/clusters5806lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5807lb: must succeed: curl -sf localhost:9901/clusters5808lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5809worker1 # [ 78.699183] systemd[1]: Stopping Nix worker-protocol over gRPC...5810worker1 # [ 79.157729] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:31Z level=info event=signal_exit5811lb # [ 79.335960] niks3-server[1117]: 2026/09/21 14:44:30 INFO lead: released remote=192.168.1.3:546205812worker2 # [ 80.333445] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:32Z level=info event=scheduler_restarting addr=lb:500515813worker1 # [ 80.339729] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:32Z level=info event=scheduler_restarting5814worker1 # [ 80.342994] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:32Z level=info event=scheduler_restarting addr=lb:500515815worker1 # [ 80.346503] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:32Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:377585816worker1 # [ 80.350287] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:32Z level=info event=worker_gone addr=192.168.1.4:500515817worker1 # [ 80.353403] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:32Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:377585818worker1 # [ 80.356863] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:32Z level=info event=worker_gone addr=192.168.1.3:500515819worker1 # [ 80.377710] nix-daemon[862]: reaped child process 1342, status = succeeded5820worker2 # [ 80.533745] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:32Z level=info event=scheduler_connected addr=lb:500515821worker2 # [ 80.537927] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5822worker1 # [ 80.546772] nix-grpc-daemon[1280]: ts=2026-09-21T14:44:32Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5823worker1 # [ 80.734642] nix-grpc-daemon[1280]: E0000 00:00:1790001872.901596 1325 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true5824worker2 # [ 80.738945] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:32Z level=info event=scheduler_connected addr=lb:500515825worker1 # [ 80.739992] nix-grpc-daemon[1280]: E0000 00:00:1790001872.906945 1325 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5826worker2 # [ 80.743204] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5827worker2 # [ 80.944169] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:33Z level=info event=scheduler_connected addr=lb:500515828worker2 # [ 80.948281] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5829worker2 # [ 81.149232] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:33Z level=info event=scheduler_connected addr=lb:500515830worker2 # [ 81.153322] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5831worker2 # [ 81.355495] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:33Z level=info event=scheduler_connected addr=lb:500515832worker2 # [ 81.358793] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5833worker2 # [ 81.560148] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:33Z level=info event=scheduler_connected addr=lb:500515834worker2 # [ 81.564166] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5835worker2 # [ 81.765365] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:33Z level=info event=scheduler_connected addr=lb:500515836worker2 # [ 81.769391] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5837worker2 # [ 81.970142] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:34Z level=info event=scheduler_connected addr=lb:500515838worker2 # [ 81.974150] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5839worker2 # [ 82.174967] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:34Z level=info event=scheduler_connected addr=lb:500515840worker2 # [ 82.179159] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5841worker2 # [ 82.381400] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:34Z level=info event=scheduler_connected addr=lb:500515842worker2 # [ 82.384997] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5843worker1 # [ 82.543119] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5844worker1 # [ 82.546188] systemd[1]: Stopped Nix worker-protocol over gRPC.5845worker1 # [ 82.550482] systemd[1]: nix-grpc-daemon.service: Consumed 970ms CPU time over 11.479s wall clock time, 38.5M memory peak, 36.6K incoming IP traffic, 59.3K outgoing IP traffic.5846worker1 # [ 82.561161] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5847worker1 # [ 82.565595] systemd[1]: Closed Nix worker-protocol over gRPC.5848worker1 # [ 82.569444] systemd[1]: nix-grpc-daemon.socket: Consumed 295us CPU time over 11.509s wall clock time, 256K memory peak, 33K incoming IP traffic, 23.8K outgoing IP traffic.5849worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5850worker2 # [ 82.586330] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:34Z level=info event=scheduler_connected addr=lb:500515851worker2 # [ 82.591172] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5852worker2 # [ 82.791976] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:34Z level=info event=scheduler_connected addr=lb:500515853worker2 # [ 82.796247] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5854worker2 # [ 82.997139] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:35Z level=info event=scheduler_connected addr=lb:500515855worker2 # [ 83.001355] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5856worker2 # [ 83.202256] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:35Z level=info event=scheduler_connected addr=lb:500515857worker2 # [ 83.206384] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5858worker2 # [ 83.330578] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:35Z level=info event=scheduler_take_over5859lb # [ 83.334439] niks3-server[1117]: 2026/09/21 14:44:34 INFO lead: acquired remote=192.168.1.4:412985860worker2 # [ 83.408438] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:35Z level=info event=scheduler_connected addr=lb:500515861worker2 # [ 83.412220] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5862worker2 # [ 83.613499] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:35Z level=info event=scheduler_connected addr=lb:500515863worker2 # [ 83.617545] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5864worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 1.12 seconds)5865worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5866worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)5867worker2 # [ 83.818691] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:35Z level=info event=scheduler_connected addr=lb:500515868worker2 # [ 83.822774] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5869worker2 # [ 84.023799] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:36Z level=info event=scheduler_connected addr=lb:500515870worker2 # [ 84.040159] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:36Z level=info event=oidc_jwks provider=mock keys=15871worker2 # [ 84.044178] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:36Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:596225872worker2 # [ 84.048314] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:36Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05873worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5874worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5875client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&25876client # don't know how to build these paths:5877client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5878worker2 # [ 84.904255] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:395905879worker2 # [ 84.909753] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395905880worker2 # [ 84.931244] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:37Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=05881worker2 # [ 84.935825] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:37Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.4:50051 clients=15882worker2 # [ 84.980257] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395905883worker2 # [ 84.992418] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395905884client # copying 1 paths to 'lb:50051'...5885client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...5886worker2 # [ 85.004919] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395905887worker2 # [ 85.016112] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)5888worker2 # [ 85.116570] nix-daemon[850]: accepted connection from pid 1151, user nix-grpc-daemon (trusted)5889worker2 # [ 85.129692] nix-daemon[850]: reaped child process 1154, status = succeeded5890worker2 # [ 85.213154] nix-daemon[850]: accepted connection from pid 1156, user nix-grpc-daemon (trusted)5891worker2 # [ 85.226881] nix-daemon[850]: reaped child process 1159, status = succeeded5892lb # [ 85.237635] niks3-server[1117]: 2026/09/21 14:44:36 INFO Received uploads request method=POST path=/api/pending_closures5893worker2 # [ 85.243860] nix-grpc-daemon[1106]: time=2026-09-21T14:44:37.389Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5894worker2 # [ 85.245762] nix-grpc-daemon[1106]: time=2026-09-21T14:44:37.390Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"5895lb # [ 85.272925] niks3-server[1117]: 2026/09/21 14:44:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5896lb # [ 85.285823] niks3-server[1117]: 2026/09/21 14:44:36 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst5897worker2 # [ 85.287546] nix-grpc-daemon[1106]: time=2026-09-21T14:44:37.432Z level=INFO msg="Uploading 1 narinfos"5898lb # [ 85.291144] niks3-server[1117]: 2026/09/21 14:44:36 INFO Signed narinfos id=33 count=15899lb # [ 85.314345] niks3-server[1117]: 2026/09/21 14:44:36 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5900lb # [ 85.320611] niks3-server[1117]: 2026/09/21 14:44:36 INFO Completed upload id=335901worker2 # [ 85.317110] nix-grpc-daemon[1106]: time=2026-09-21T14:44:37.462Z level=INFO msg="Upload complete. (289ms)"5902lb # [ 85.323162] niks3-server[1117]: 2026/09/21 14:44:36 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo5903lb # [ 85.326679] niks3-server[1117]: 2026/09/21 14:44:36 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls5904worker2 # [ 85.321493] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39590 duration_s=0 paths=1 nar_bytes_in=5205905worker2 # [ 85.328346] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395905906worker2 # [ 85.339134] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)5907worker2 # [ 85.350854] nix-daemon[850]: reaped child process 1149, status = succeeded5908worker2 # [ 85.354614] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)5909client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...5910client # farm-a> worker2: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5911client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5912client # farm-a> LOG-farm-a-on-standby5913worker2 # [ 86.570403] nix-daemon[850]: reaped child process 1163, status = succeeded5914lb # [ 86.603101] envoy[917]: conn peer=[2001:db8:1::1]:36978 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9696 tx=4083 ms=56445915lb # [ 86.606774] envoy[917]: conn peer=[2001:db8:1::3]:57362 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1726 tx=569 ms=93395916worker2 # [ 86.652628] nix-daemon[850]: accepted connection from pid 1179, user nix-grpc-daemon (trusted)5917worker2 # [ 86.664870] nix-daemon[850]: reaped child process 1182, status = succeeded5918lb # [ 86.677230] niks3-server[1117]: 2026/09/21 14:44:38 INFO Received uploads request method=POST path=/api/pending_closures5919worker2 # [ 86.693031] nix-grpc-daemon[1106]: time=2026-09-21T14:44:38.838Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5920worker2 # [ 86.694592] nix-grpc-daemon[1106]: time=2026-09-21T14:44:38.840Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"5921lb # [ 86.732536] niks3-server[1117]: 2026/09/21 14:44:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5922lb # [ 86.738228] niks3-server[1117]: 2026/09/21 14:44:38 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5923worker2 # [ 86.742181] nix-grpc-daemon[1106]: time=2026-09-21T14:44:38.887Z level=INFO msg="Uploading 1 narinfos"5924lb # [ 86.745624] niks3-server[1117]: 2026/09/21 14:44:38 INFO Signed narinfos id=34 count=15925lb # [ 86.751833] niks3-server[1117]: 2026/09/21 14:44:38 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5926lb # [ 86.759237] niks3-server[1117]: 2026/09/21 14:44:38 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls5927lb # [ 86.769167] niks3-server[1117]: 2026/09/21 14:44:38 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5928lb # [ 86.776315] niks3-server[1117]: 2026/09/21 14:44:38 INFO Completed upload id=345929worker2 # [ 86.773089] nix-grpc-daemon[1106]: time=2026-09-21T14:44:38.918Z level=INFO msg="Upload complete. (210ms)"5930lb # [ 86.779229] niks3-server[1117]: 2026/09/21 14:44:38 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo5931worker2 # [ 86.786856] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39590 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=15932worker2 # [ 86.793477] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:38Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=05933worker2 # [ 86.796568] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:38Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.4:50051 clients=15934worker2 # [ 86.806415] nix-daemon[850]: reaped child process 1161, status = succeeded5935worker2 # [ 86.836177] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395905936worker2 # [ 86.848764] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395905937worker2 # [ 86.862489] nix-grpc-daemon[1106]: time=2026-09-21T14:44:39.007Z level=INFO msg="All 1 paths already cached"5938client # copying 1 paths to 'lb:50051'...5939client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...5940worker2 # [ 86.870668] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395905941worker2 # [ 86.878962] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)5942worker2 # [ 86.980128] nix-daemon[850]: accepted connection from pid 1188, user nix-grpc-daemon (trusted)5943worker2 # [ 86.992771] nix-daemon[850]: reaped child process 1191, status = succeeded5944worker2 # [ 87.080463] nix-daemon[850]: accepted connection from pid 1193, user nix-grpc-daemon (trusted)5945worker2 # [ 87.093329] nix-daemon[850]: reaped child process 1196, status = succeeded5946lb # [ 87.104870] niks3-server[1117]: 2026/09/21 14:44:38 INFO Received uploads request method=POST path=/api/pending_closures5947worker2 # [ 87.109502] nix-grpc-daemon[1106]: time=2026-09-21T14:44:39.255Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5948worker2 # [ 87.111130] nix-grpc-daemon[1106]: time=2026-09-21T14:44:39.256Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"5949lb # [ 87.134985] niks3-server[1117]: 2026/09/21 14:44:38 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst5950lb # [ 87.143623] niks3-server[1117]: 2026/09/21 14:44:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5951lb # [ 87.151436] niks3-server[1117]: 2026/09/21 14:44:38 INFO Signed narinfos id=35 count=15952worker2 # [ 87.151223] nix-grpc-daemon[1106]: time=2026-09-21T14:44:39.296Z level=INFO msg="Uploading 1 narinfos"5953lb # [ 87.157691] niks3-server[1117]: 2026/09/21 14:44:38 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls5954lb # [ 87.169702] niks3-server[1117]: 2026/09/21 14:44:38 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5955worker2 # [ 87.176035] nix-grpc-daemon[1106]: time=2026-09-21T14:44:39.321Z level=INFO msg="Upload complete. (284ms)"5956lb # [ 87.179854] niks3-server[1117]: 2026/09/21 14:44:38 INFO Completed upload id=355957worker2 # [ 87.178785] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39590 duration_s=0 paths=1 nar_bytes_in=6405958lb # [ 87.185571] niks3-server[1117]: 2026/09/21 14:44:38 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo5959worker2 # [ 87.184273] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395905960worker2 # [ 87.192484] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)5961worker2 # [ 87.201486] nix-daemon[850]: reaped child process 1186, status = succeeded5962worker2 # [ 87.206296] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)5963client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...5964client # farm-b> worker2: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5965client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5966client # farm-b> LOG-farm-b-on-standby5967worker2 # [ 88.442655] nix-daemon[850]: reaped child process 1200, status = succeeded5968worker2 # [ 88.529042] nix-daemon[850]: accepted connection from pid 1216, user nix-grpc-daemon (trusted)5969worker2 # [ 88.540991] nix-daemon[850]: reaped child process 1219, status = succeeded5970lb # [ 88.552575] niks3-server[1117]: 2026/09/21 14:44:39 INFO Received uploads request method=POST path=/api/pending_closures5971worker2 # [ 88.557700] nix-grpc-daemon[1106]: time=2026-09-21T14:44:40.703Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5972worker2 # [ 88.560033] nix-grpc-daemon[1106]: time=2026-09-21T14:44:40.704Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"5973lb # [ 88.598470] niks3-server[1117]: 2026/09/21 14:44:40 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5974lb # [ 88.603451] niks3-server[1117]: 2026/09/21 14:44:40 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst5975lb # [ 88.616306] niks3-server[1117]: 2026/09/21 14:44:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5976lb # [ 88.622238] niks3-server[1117]: 2026/09/21 14:44:40 INFO Signed narinfos id=36 count=15977worker2 # [ 88.619229] nix-grpc-daemon[1106]: time=2026-09-21T14:44:40.764Z level=INFO msg="Uploading 1 narinfos"5978lb # [ 88.629764] niks3-server[1117]: 2026/09/21 14:44:40 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls5979lb # [ 88.634053] niks3-server[1117]: 2026/09/21 14:44:40 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5980lb # [ 88.637024] niks3-server[1117]: 2026/09/21 14:44:40 INFO Completed upload id=365981worker2 # [ 88.633775] nix-grpc-daemon[1106]: time=2026-09-21T14:44:40.779Z level=INFO msg="Upload complete. (219ms)"5982lb # [ 88.639859] niks3-server[1117]: 2026/09/21 14:44:40 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo5983worker2 # [ 88.641919] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39590 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=15984worker2 # [ 88.652301] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:40Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=05985worker2 # [ 88.656219] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:40Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.4:50051 clients=15986worker2 # [ 88.665671] nix-daemon[850]: reaped child process 1198, status = succeeded5987worker2 # [ 88.695278] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395905988worker2 # [ 88.707568] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395905989worker2 # [ 88.720491] nix-grpc-daemon[1106]: time=2026-09-21T14:44:40.865Z level=INFO msg="All 2 paths already cached"5990client # copying 1 paths to 'lb:50051'...5991client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...5992worker2 # [ 88.729241] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395905993worker2 # [ 88.737683] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)5994worker2 # [ 88.837096] nix-daemon[850]: accepted connection from pid 1225, user nix-grpc-daemon (trusted)5995worker2 # [ 88.851124] nix-daemon[850]: reaped child process 1228, status = succeeded5996worker2 # [ 88.935380] nix-daemon[850]: accepted connection from pid 1230, user nix-grpc-daemon (trusted)5997worker2 # [ 88.948648] nix-daemon[850]: reaped child process 1233, status = succeeded5998lb # [ 88.959356] niks3-server[1117]: 2026/09/21 14:44:40 INFO Received uploads request method=POST path=/api/pending_closures5999worker2 # [ 88.965107] nix-grpc-daemon[1106]: time=2026-09-21T14:44:41.110Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6000worker2 # [ 88.967039] nix-grpc-daemon[1106]: time=2026-09-21T14:44:41.112Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6001lb # [ 88.992347] niks3-server[1117]: 2026/09/21 14:44:40 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6002lb # [ 89.002690] niks3-server[1117]: 2026/09/21 14:44:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6003worker2 # [ 89.007280] nix-grpc-daemon[1106]: time=2026-09-21T14:44:41.152Z level=INFO msg="Uploading 1 narinfos"6004lb # [ 89.010437] niks3-server[1117]: 2026/09/21 14:44:40 INFO Signed narinfos id=37 count=16005lb # [ 89.014767] niks3-server[1117]: 2026/09/21 14:44:40 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6006lb # [ 89.026100] niks3-server[1117]: 2026/09/21 14:44:40 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6007lb # [ 89.029769] niks3-server[1117]: 2026/09/21 14:44:40 INFO Completed upload id=376008worker2 # [ 89.026284] nix-grpc-daemon[1106]: time=2026-09-21T14:44:41.171Z level=INFO msg="Upload complete. (275ms)"6009lb # [ 89.033084] niks3-server[1117]: 2026/09/21 14:44:40 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6010worker2 # [ 89.029813] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39590 duration_s=0 paths=1 nar_bytes_in=7686011worker2 # [ 89.034905] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395906012worker2 # [ 89.044860] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)6013worker2 # [ 89.049152] nix-daemon[850]: reaped child process 1223, status = succeeded6014worker2 # [ 89.053861] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)6015client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6016client # farm-top> worker2: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6017client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6018client # farm-top> LOG-farm-top-on-standby6019worker2 # [ 90.068319] nix-daemon[850]: reaped child process 1237, status = succeeded6020worker2 # [ 90.147556] nix-daemon[850]: accepted connection from pid 1253, user nix-grpc-daemon (trusted)6021worker2 # [ 90.159423] nix-daemon[850]: reaped child process 1256, status = succeeded6022lb # [ 90.171721] niks3-server[1117]: 2026/09/21 14:44:41 INFO Received uploads request method=POST path=/api/pending_closures6023worker2 # [ 90.176385] nix-grpc-daemon[1106]: time=2026-09-21T14:44:42.321Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6024worker2 # [ 90.178147] nix-grpc-daemon[1106]: time=2026-09-21T14:44:42.323Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6025lb # [ 90.212305] niks3-server[1117]: 2026/09/21 14:44:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6026lb # [ 90.219188] niks3-server[1117]: 2026/09/21 14:44:41 INFO Signed narinfos id=38 count=16027worker2 # [ 90.219602] nix-grpc-daemon[1106]: time=2026-09-21T14:44:42.364Z level=INFO msg="Uploading 1 narinfos"6028lb # [ 90.227415] niks3-server[1117]: 2026/09/21 14:44:41 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6029lb # [ 90.232500] niks3-server[1117]: 2026/09/21 14:44:41 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6030lb # [ 90.243859] niks3-server[1117]: 2026/09/21 14:44:41 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6031lb # [ 90.248076] niks3-server[1117]: 2026/09/21 14:44:41 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6032lb # [ 90.251530] niks3-server[1117]: 2026/09/21 14:44:41 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6033lb # [ 90.254800] niks3-server[1117]: 2026/09/21 14:44:41 INFO Completed upload id=386034worker2 # [ 90.251571] nix-grpc-daemon[1106]: time=2026-09-21T14:44:42.396Z level=INFO msg="Upload complete. (189ms)"6035worker2 # [ 90.259340] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39590 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16036worker2 # [ 90.272365] nix-daemon[850]: reaped child process 1235, status = succeeded6037client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&2, in 5.53 seconds)6038client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath6039client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath, in 0.12 seconds)6040worker1 # [ 90.462976] systemd[1]: Listening on Nix worker-protocol over gRPC.6041worker1 # [ 90.474105] systemd[1]: Starting Nix worker-protocol over gRPC...6042worker1 # [ 90.559657] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:42Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6043worker1 # [ 90.561661] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:42Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6044worker1 # [ 90.564811] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:42Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16045worker1 # [ 90.569053] nix-grpc-daemon[1488]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6046worker1 # [ 90.571152] nix-grpc-daemon[1488]: E0000 00:00:1790001882.736252 1488 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6047worker1 # [ 90.576776] nix-grpc-daemon[1488]: E0000 00:00:1790001882.741717 1488 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6048worker1 # [ 90.580523] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:42Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6049worker1 # [ 90.584084] systemd[1]: Started Nix worker-protocol over gRPC.6050worker1 # [ 90.590103] nix-grpc-daemon[1488]: E0000 00:00:1790001882.756392 1530 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6051lb: must succeed: curl -sf localhost:9901/clusters6052worker1 # [ 90.604110] nix-grpc-daemon[1488]: E0000 00:00:1790001882.771382 1530 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6053worker1 # [ 90.609055] nix-grpc-daemon[1488]: E0000 00:00:1790001882.773643 1530 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6054worker1 # [ 90.617983] nix-grpc-daemon[1488]: E0000 00:00:1790001882.785287 1530 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6055worker1 # [ 90.627879] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:42Z level=info event=scheduler_connected addr=lb:500516056worker2 # [ 90.627117] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:42Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:596226057worker2 # [ 90.631288] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:42Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06058lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6059worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6060worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6061lb: must succeed: curl -sf localhost:9901/clusters6062lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6063lb: must succeed: curl -sf localhost:9901/clusters6064lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6065worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6066worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6067(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.26 seconds)6068subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6069client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs6070client # [ 90.836537] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs.6071client # Running as unit: rs.service; invocation ID: adb5112fec3e4a05b8b1baf3a49221b06072client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs, in 0.06 seconds)6073worker2 # [ 90.981277] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:395906074client # [ 90.980648] nix[1199]: don't know how to build these paths:6075client # [ 90.981628] nix[1199]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6076worker2 # [ 90.995638] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:43Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06077worker2 # [ 91.000822] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:43Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.4:50051 clients=16078worker2 # [ 91.040301] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395906079worker2 # [ 91.052287] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:395906080client # [ 91.055217] nix[1199]: copying 1 paths to 'lb:50051'...6081worker2 # [ 91.063869] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:395906082client # [ 91.058149] nix[1199]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6083worker2 # [ 91.076807] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)6084worker2 # [ 91.180691] nix-daemon[850]: accepted connection from pid 1269, user nix-grpc-daemon (trusted)6085worker2 # [ 91.194239] nix-daemon[850]: reaped child process 1272, status = succeeded6086worker2 # [ 91.281211] nix-daemon[850]: accepted connection from pid 1274, user nix-grpc-daemon (trusted)6087worker2 # [ 91.296414] nix-daemon[850]: reaped child process 1277, status = succeeded6088lb # [ 91.305928] niks3-server[1117]: 2026/09/21 14:44:42 INFO Received uploads request method=POST path=/api/pending_closures6089worker2 # [ 91.326106] nix-grpc-daemon[1106]: time=2026-09-21T14:44:43.471Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6090worker2 # [ 91.327941] nix-grpc-daemon[1106]: time=2026-09-21T14:44:43.473Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6091lb # [ 91.353410] niks3-server[1117]: 2026/09/21 14:44:42 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6092lb # [ 91.363126] niks3-server[1117]: 2026/09/21 14:44:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6093lb # [ 91.368464] niks3-server[1117]: 2026/09/21 14:44:42 INFO Signed narinfos id=39 count=16094worker2 # [ 91.368773] nix-grpc-daemon[1106]: time=2026-09-21T14:44:43.513Z level=INFO msg="Uploading 1 narinfos"6095lb # [ 91.376297] niks3-server[1117]: 2026/09/21 14:44:42 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6096lb # [ 91.390947] niks3-server[1117]: 2026/09/21 14:44:42 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6097lb # [ 91.398066] niks3-server[1117]: 2026/09/21 14:44:42 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6098lb # [ 91.400729] niks3-server[1117]: 2026/09/21 14:44:42 INFO Completed upload id=396099worker2 # [ 91.398621] nix-grpc-daemon[1106]: time=2026-09-21T14:44:43.544Z level=INFO msg="Upload complete. (310ms)"6100worker2 # [ 91.403717] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39590 duration_s=0 paths=1 nar_bytes_in=4166101worker2 # [ 91.409963] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:395906102worker2 # [ 91.421581] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)6103worker2 # [ 91.431610] nix-daemon[850]: reaped child process 1267, status = succeeded6104client # [ 91.430776] nix[1199]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6105worker2 # [ 91.439377] nix-daemon[850]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)6106lb: must succeed: curl -sf localhost:9901/clusters6107lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6108lb: must succeed: curl -sf localhost:9901/clusters6109lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6110lb: must succeed: curl -sf localhost:9901/clusters6111lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6112lb: must succeed: curl -sf localhost:9901/clusters6113lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6114worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6115worker1 # [ 93.494332] systemd-journald[283]: Received client request to rotate journal, rotating.6116worker1 # [ 93.514757] systemd-journald[283]: Vacuuming done, freed 0B of archived journals from /var/log/journal/c6bff1b450e043bcb2fb427d125cbbfa.6117worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.06 seconds)6118worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service6119worker2 # [ 93.421230] systemd[1]: Stopping Nix worker-protocol over gRPC...6120worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.03 seconds)6121client: must succeed: sleep 5.06122worker2 # [ 93.925193] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:46Z level=info event=draining6123lb # [ 96.603128] envoy[917]: conn peer=[2001:db8:1::1]:58576 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9534 tx=4022 ms=53656124client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6125worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6126worker2 # [ 98.662469] nix-daemon[850]: reaped child process 1281, status = succeeded6127worker2 # [ 98.737595] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted)6128worker2 # [ 98.749839] nix-daemon[850]: reaped child process 1325, status = succeeded6129lb # [ 98.759882] niks3-server[1117]: 2026/09/21 14:44:50 INFO Received uploads request method=POST path=/api/pending_closures6130worker2 # [ 98.766658] nix-grpc-daemon[1106]: time=2026-09-21T14:44:50.911Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6131worker2 # [ 98.768165] nix-grpc-daemon[1106]: time=2026-09-21T14:44:50.913Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6132lb # [ 98.810251] niks3-server[1117]: 2026/09/21 14:44:50 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6133lb # [ 98.821080] niks3-server[1117]: 2026/09/21 14:44:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6134worker2 # [ 98.821968] nix-grpc-daemon[1106]: time=2026-09-21T14:44:50.967Z level=INFO msg="Uploading 1 narinfos"6135lb # [ 98.826132] niks3-server[1117]: 2026/09/21 14:44:50 INFO Signed narinfos id=40 count=16136lb # [ 98.833251] niks3-server[1117]: 2026/09/21 14:44:50 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6137lb # [ 98.838395] niks3-server[1117]: 2026/09/21 14:44:50 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6138lb # [ 98.846675] niks3-server[1117]: 2026/09/21 14:44:50 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6139lb # [ 98.851932] niks3-server[1117]: 2026/09/21 14:44:50 INFO Completed upload id=406140worker2 # [ 98.848718] nix-grpc-daemon[1106]: time=2026-09-21T14:44:50.994Z level=INFO msg="Upload complete. (194ms)"6141lb # [ 98.854286] niks3-server[1117]: 2026/09/21 14:44:50 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6142worker2 # [ 98.857451] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39590 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16143worker2 # [ 98.871151] nix-daemon[850]: reaped child process 1279, status = succeeded6144client # [ 98.879526] systemd[1]: rs.service: Deactivated successfully.6145client # [ 98.883296] systemd[1]: rs.service: Consumed 130ms CPU time over 8.046s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6146worker2 # [ 98.926299] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:51Z level=info event=signal_exit6147lb # [ 99.336248] niks3-server[1117]: 2026/09/21 14:44:50 INFO lead: released remote=192.168.1.4:412986148worker2 # [ 100.336526] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:52Z level=info event=scheduler_restarting6149worker1 # [ 100.338829] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:52Z level=info event=scheduler_restarting addr=lb:500516150worker2 # [ 100.341904] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:52Z level=info event=scheduler_restarting addr=lb:500516151worker2 # [ 100.345441] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:52Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:596226152worker2 # [ 100.349168] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:52Z level=info event=worker_gone addr=192.168.1.3:500516153worker2 # [ 100.352366] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:52Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:596226154worker2 # [ 100.355978] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:52Z level=info event=worker_gone addr=192.168.1.4:500516155worker2 # [ 100.372469] nix-daemon[850]: reaped child process 1098, status = succeeded6156worker1 # [ 100.538457] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:52Z level=info event=scheduler_connected addr=lb:500516157worker1 # [ 100.542912] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6158worker2 # [ 100.548149] nix-grpc-daemon[1046]: ts=2026-09-21T14:44:52Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6159worker1 # [ 100.589487] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:52Z level=info event=scheduler_take_over6160lb # [ 100.590776] niks3-server[1117]: 2026/09/21 14:44:52 INFO lead: acquired remote=192.168.1.3:441326161worker2 # [ 100.715343] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6162worker2 # [ 100.719290] systemd[1]: Stopped Nix worker-protocol over gRPC.6163worker2 # [ 100.722774] systemd[1]: nix-grpc-daemon.service: Consumed 1.443s CPU time over 33.895s wall clock time, 40.1M memory peak, 63.2K incoming IP traffic, 158.5K outgoing IP traffic.6164worker2 # [ 100.737466] systemd[1]: Starting scheduler-token.service...6165worker1 # [ 100.744149] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:52Z level=info event=scheduler_connected addr=lb:500516166worker1 # [ 100.752560] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:52Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:581026167worker1 # [ 100.756324] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:52Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06168worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.15 seconds)6169client: waiting for success: ! systemctl is-active rs6170worker2 # [ 100.781228] systemd[1]: scheduler-token.service: Deactivated successfully.6171worker2 # [ 100.783482] systemd[1]: Finished scheduler-token.service.6172worker2 # [ 100.785198] systemd[1]: scheduler-token.service: Consumed 23ms CPU time over 45ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6173worker2 # [ 100.788273] systemd[1]: Starting Nix worker-protocol over gRPC...6174client: (finished: waiting for success: ! systemctl is-active rs, in 0.04 seconds)6175client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6176client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)6177client: must fail: journalctl -u rs | grep -q 'reconnecting'6178worker2 # [ 100.864834] nix-grpc-daemon[1335]: ts=2026-09-21T14:44:53Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6179worker2 # [ 100.866822] nix-grpc-daemon[1335]: ts=2026-09-21T14:44:53Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6180client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.02 seconds)6181lb: must succeed: curl -sf localhost:9901/clusters6182worker2 # [ 100.869950] nix-grpc-daemon[1335]: ts=2026-09-21T14:44:53Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16183worker2 # [ 100.874053] nix-grpc-daemon[1335]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6184worker2 # [ 100.877187] nix-grpc-daemon[1335]: E0000 00:00:1790001893.019448 1335 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6185worker2 # [ 100.881889] nix-grpc-daemon[1335]: E0000 00:00:1790001893.022615 1335 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6186worker2 # [ 100.885450] nix-grpc-daemon[1335]: ts=2026-09-21T14:44:53Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6187worker2 # [ 100.888152] systemd[1]: Started Nix worker-protocol over gRPC.6188worker2 # [ 100.890076] nix-grpc-daemon[1335]: E0000 00:00:1790001893.035012 1377 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6189worker2 # [ 100.901689] nix-grpc-daemon[1335]: E0000 00:00:1790001893.047166 1377 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6190worker2 # [ 100.905577] nix-grpc-daemon[1335]: E0000 00:00:1790001893.049634 1377 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6191lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6192worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6193worker2 # [ 100.917181] nix-grpc-daemon[1335]: E0000 00:00:1790001893.062667 1377 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6194worker2 # [ 100.925610] nix-grpc-daemon[1335]: ts=2026-09-21T14:44:53Z level=info event=scheduler_connected addr=lb:500516195worker1 # [ 100.935819] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:53Z level=info event=oidc_jwks provider=mock keys=16196worker1 # [ 100.940272] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:53Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:581026197worker1 # [ 100.944211] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:53Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06198worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6199(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 10.18 seconds)6200subtest: scheduler restart mid-build: build finishes, no double build6201client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid6202client # [ 101.011859] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid.6203client # Running as unit: mid.service; invocation ID: 2c6273b34f4e4fb1b87fad78e4d154bf6204client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid, in 0.06 seconds)6205worker1 # [ 101.165169] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:581066206worker1 # [ 101.176981] nix-daemon[862]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6207client # [ 101.180180] nix[1235]: don't know how to build these paths:6208client # [ 101.181181] nix[1235]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6209worker1 # [ 101.196753] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:53Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06210worker1 # [ 101.202557] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:53Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16211worker1 # [ 101.246490] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:581066212worker1 # [ 101.257814] nix-grpc-daemon[1488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 37 ms (attempt 1/5)6213worker1 # [ 101.302201] nix-grpc-daemon[1488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 58 ms (attempt 2/5)6214worker1 # [ 101.363167] nix-grpc-daemon[1488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 350 ms (attempt 3/5)6215worker1 # [ 101.714585] nix-grpc-daemon[1488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 575 ms (attempt 4/5)6216worker1 # [ 102.292182] nix-grpc-daemon[1488]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6217worker1 # [ 102.295344] nix-grpc-daemon[1488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6218worker1 # [ 102.313928] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:581066219client # [ 102.316447] nix[1235]: copying 1 paths to 'lb:50051'...6220worker1 # [ 102.327108] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:581066221client # [ 102.318712] nix[1235]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6222worker1 # [ 102.340307] nix-daemon[862]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6223worker1 # [ 102.361289] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:54Z level=info event=niks3_push_started pid=16116224worker1 # [ 102.464252] nix-grpc-daemon[1611]: time=2026-09-21T14:44:54.630Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6225worker1 # [ 102.548888] nix-daemon[862]: accepted connection from pid 1624, user nix-grpc-daemon (trusted)6226worker1 # [ 102.561260] nix-daemon[862]: reaped child process 1627, status = succeeded6227worker1 # [ 102.645510] nix-daemon[862]: accepted connection from pid 1629, user nix-grpc-daemon (trusted)6228worker1 # [ 102.658361] nix-daemon[862]: reaped child process 1632, status = succeeded6229lb # [ 102.666929] niks3-server[1117]: 2026/09/21 14:44:54 INFO Received uploads request method=POST path=/api/pending_closures6230worker1 # [ 102.677085] nix-grpc-daemon[1611]: time=2026-09-21T14:44:54.844Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6231worker1 # [ 102.680173] nix-grpc-daemon[1611]: time=2026-09-21T14:44:54.845Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6232lb # [ 102.708101] niks3-server[1117]: 2026/09/21 14:44:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6233lb # [ 102.714474] niks3-server[1117]: 2026/09/21 14:44:54 INFO Signed narinfos id=41 count=16234worker1 # [ 102.717051] nix-grpc-daemon[1611]: time=2026-09-21T14:44:54.884Z level=INFO msg="Uploading 1 narinfos"6235lb # [ 102.725965] niks3-server[1117]: 2026/09/21 14:44:54 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6236lb # [ 102.735654] niks3-server[1117]: 2026/09/21 14:44:54 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6237lb # [ 102.744955] niks3-server[1117]: 2026/09/21 14:44:54 INFO Completed upload id=416238worker1 # [ 102.744116] nix-grpc-daemon[1611]: time=2026-09-21T14:44:54.911Z level=INFO msg="Upload complete. (280ms)"6239lb # [ 102.746634] niks3-server[1117]: 2026/09/21 14:44:54 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6240worker1 # [ 102.746557] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58106 duration_s=0 paths=1 nar_bytes_in=4246241worker1 # [ 102.750384] nix-grpc-daemon[1488]: ts=2026-09-21T14:44:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:581066242lb # [ 102.757492] niks3-server[1117]: 2026/09/21 14:44:54 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6243worker1 # [ 102.756890] nix-daemon[862]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6244worker1 # [ 102.759210] nix-daemon[862]: reaped child process 1609, status = succeeded6245worker1 # [ 102.764225] nix-daemon[862]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6246client # [ 102.754365] nix[1235]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6247lb: must succeed: curl -sf localhost:9901/clusters6248lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6249lb: must succeed: curl -sf localhost:9901/clusters6250lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6251worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6252worker1 # [ 104.602280] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1488 (nix-grpc-daemon) on client request.6253worker1 # [ 104.604884] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6254lb # [ 104.608144] niks3-server[1117]: 2026/09/21 14:44:56 INFO lead: released remote=192.168.1.3:441326255client # [ 104.603264] nix[1235]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6256worker2 # [ 104.610312] nix-grpc-daemon[1335]: ts=2026-09-21T14:44:56Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6257worker1 # [ 104.618186] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6258worker1 # [ 104.619651] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6259client # [ 104.607258] nix[1235]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6260worker1 # [ 104.622143] systemd[1]: nix-grpc-daemon.service: Consumed 422ms CPU time over 14.144s wall clock time, 35.7M memory peak, 15.1K incoming IP traffic, 22.2K outgoing IP traffic.6261worker1 # [ 104.627550] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6262worker1 # [ 104.633080] systemd[1]: Starting Nix worker-protocol over gRPC...6263worker1 # [ 104.667393] nix-daemon[862]: reaped child process 1634, status = succeeded6264worker1 # [ 104.668779] nix-daemon[862]: reaped child process 1597, status = succeeded6265worker1 # [ 104.674208] nix-daemon[1636]: unexpected Nix daemon error: error: interrupted by the user6266worker1 # [ 104.684223] nix-daemon[862]: reaped child process 1636, status = failed with exit code 16267worker1 # [ 104.725311] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:56Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6268worker1 # [ 104.727431] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:56Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6269worker1 # [ 104.730623] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:56Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16270worker1 # [ 104.734994] nix-grpc-daemon[1673]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6271worker1 # [ 104.736394] nix-grpc-daemon[1673]: E0000 00:00:1790001896.901912 1673 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6272worker1 # [ 104.744084] nix-grpc-daemon[1673]: E0000 00:00:1790001896.907340 1673 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6273worker1 # [ 104.747115] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:56Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6274worker1 # [ 104.750753] systemd[1]: Started Nix worker-protocol over gRPC.6275worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.18 seconds)6276client: must succeed: sleep 5.06277worker1 # [ 104.756157] nix-grpc-daemon[1673]: E0000 00:00:1790001896.922749 1712 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6278worker1 # [ 104.760060] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:56Z level=info event=scheduler_take_over6279lb # [ 104.760946] niks3-server[1117]: 2026/09/21 14:44:56 INFO lead: acquired remote=192.168.1.3:513586280worker1 # [ 104.770738] nix-grpc-daemon[1673]: E0000 00:00:1790001896.938042 1712 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6281worker1 # [ 104.775054] nix-grpc-daemon[1673]: E0000 00:00:1790001896.940299 1712 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6282worker1 # [ 104.784939] nix-grpc-daemon[1673]: E0000 00:00:1790001896.952234 1712 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6283worker1 # [ 104.793888] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:56Z level=info event=scheduler_connected addr=lb:500516284worker1 # [ 104.796921] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:56Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:555626285worker1 # [ 104.798689] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:56Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06286worker2 # [ 104.811158] nix-grpc-daemon[1335]: ts=2026-09-21T14:44:56Z level=info event=scheduler_connected addr=lb:500516287worker1 # [ 104.818569] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:56Z level=info event=oidc_jwks provider=mock keys=16288worker1 # [ 104.819918] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:56Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:555626289worker1 # [ 104.821602] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:56Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06290worker1 # [ 105.123159] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:57Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06291worker1 # [ 105.128821] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:57Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16292worker1 # [ 105.171834] nix-grpc-daemon[1673]: ts=2026-09-21T14:44:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:555786293worker1 # [ 105.182171] nix-daemon[862]: accepted connection from pid 1673, user nix-grpc-daemon (trusted)6294worker1 # [ 105.195898] nix-daemon[862]: accepted connection from pid 1673, user nix-grpc-daemon (trusted)6295client # [ 105.199480] nix[1235]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6296worker1 # [ 105.210762] nix-daemon[862]: accepted connection from pid 1673, user nix-grpc-daemon (trusted)6297lb # [ 106.603472] envoy[917]: conn peer=[2001:db8:1::1]:58586 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2960 tx=1405 ms=78836298lb # [ 106.607273] envoy[917]: conn peer=[2001:db8:1::4]:52166 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=36673 tx=1995 ms=335796299lb # [ 106.611080] envoy[917]: conn peer=[2001:db8:1::3]:42368 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=993 tx=363 ms=139896300client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6301client: waiting for success: ! systemctl is-active mid6302worker1 # [ 110.035551] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:02Z level=info event=niks3_push_started pid=17566303worker1 # [ 110.067439] nix-daemon[862]: reaped child process 1731, status = succeeded6304worker1 # [ 110.143879] nix-grpc-daemon[1756]: time=2026-09-21T14:45:02.310Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6305worker1 # [ 110.228907] nix-daemon[862]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6306worker1 # [ 110.241858] nix-daemon[862]: reaped child process 1771, status = succeeded6307lb # [ 110.251226] niks3-server[1117]: 2026/09/21 14:45:01 INFO Received uploads request method=POST path=/api/pending_closures6308worker1 # [ 110.259650] nix-grpc-daemon[1756]: time=2026-09-21T14:45:02.426Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6309worker1 # [ 110.261170] nix-grpc-daemon[1756]: time=2026-09-21T14:45:02.428Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6310lb # [ 110.290816] niks3-server[1117]: 2026/09/21 14:45:01 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6311lb # [ 110.304570] niks3-server[1117]: 2026/09/21 14:45:01 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6312lb # [ 110.322760] niks3-server[1117]: 2026/09/21 14:45:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6313lb # [ 110.330798] niks3-server[1117]: 2026/09/21 14:45:01 INFO Signed narinfos id=42 count=16314worker1 # [ 110.329960] nix-grpc-daemon[1756]: time=2026-09-21T14:45:02.497Z level=INFO msg="Uploading 1 narinfos"6315lb # [ 110.337159] niks3-server[1117]: 2026/09/21 14:45:01 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6316lb # [ 110.346096] niks3-server[1117]: 2026/09/21 14:45:01 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6317lb # [ 110.351783] niks3-server[1117]: 2026/09/21 14:45:01 INFO Completed upload id=426318worker1 # [ 110.351257] nix-grpc-daemon[1756]: time=2026-09-21T14:45:02.518Z level=INFO msg="Upload complete. (204ms)"6319lb # [ 110.354100] niks3-server[1117]: 2026/09/21 14:45:01 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6320worker1 # [ 110.359270] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55578 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16321worker1 # [ 110.374171] nix-daemon[862]: reaped child process 1729, status = succeeded6322client # [ 110.388331] systemd[1]: mid.service: Deactivated successfully.6323client # [ 110.390858] systemd[1]: mid.service: Consumed 147ms CPU time over 9.375s wall clock time, 43.4M memory peak, 8.7K incoming IP traffic, 10.4K outgoing IP traffic.6324client: (finished: waiting for success: ! systemctl is-active mid, in 1.09 seconds)6325client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6326client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.03 seconds)6327(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.10 seconds)6328subtest: interrupting the client stops the build on the worker6329client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr6330client # [ 111.088986] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr.6331client # Running as unit: intr.service; invocation ID: 77ff1daf9f084f7b893502429a3349306332client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr, in 0.03 seconds)6333client # [ 111.214069] nix[1271]: don't know how to build these paths:6334client # [ 111.214994] nix[1271]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6335worker1 # [ 111.222660] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:555786336worker1 # [ 111.234237] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:03Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06337worker1 # [ 111.238405] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:03Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16338worker2 # [ 111.287903] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337886339worker2 # [ 111.300595] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6340worker2 # [ 111.318403] nix-grpc-daemon[1335]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 69 ms (attempt 1/5)6341worker2 # [ 111.395103] nix-grpc-daemon[1335]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 55 ms (attempt 2/5)6342worker2 # [ 111.457956] nix-grpc-daemon[1335]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 238 ms (attempt 3/5)6343worker2 # [ 111.698196] nix-grpc-daemon[1335]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 320 ms (attempt 4/5)6344worker2 # [ 112.020393] nix-grpc-daemon[1335]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6345worker2 # [ 112.023921] nix-grpc-daemon[1335]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6346worker2 # [ 112.039960] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337886347client # [ 112.042178] nix[1271]: copying 1 paths to 'lb:50051'...6348worker2 # [ 112.050340] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:337886349client # [ 112.045156] nix[1271]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6350worker2 # [ 112.060402] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6351worker2 # [ 112.081698] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:04Z level=info event=niks3_push_started pid=14276352worker2 # [ 112.180160] nix-grpc-daemon[1427]: time=2026-09-21T14:45:04.325Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""6353worker2 # [ 112.267213] nix-daemon[850]: accepted connection from pid 1439, user nix-grpc-daemon (trusted)6354worker2 # [ 112.292638] nix-daemon[850]: reaped child process 1445, status = succeeded6355worker2 # [ 112.374549] nix-daemon[850]: accepted connection from pid 1448, user nix-grpc-daemon (trusted)6356worker2 # [ 112.386472] nix-daemon[850]: reaped child process 1451, status = succeeded6357lb # [ 112.396059] niks3-server[1117]: 2026/09/21 14:45:03 INFO Received uploads request method=POST path=/api/pending_closures6358worker2 # [ 112.397097] nix-grpc-daemon[1427]: time=2026-09-21T14:45:04.542Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6359worker2 # [ 112.398639] nix-grpc-daemon[1427]: time=2026-09-21T14:45:04.544Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6360lb # [ 112.417381] niks3-server[1117]: 2026/09/21 14:45:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6361lb # [ 112.421635] niks3-server[1117]: 2026/09/21 14:45:03 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6362lb # [ 112.425289] niks3-server[1117]: 2026/09/21 14:45:03 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6363lb # [ 112.429772] niks3-server[1117]: 2026/09/21 14:45:03 INFO Signed narinfos id=43 count=16364worker2 # [ 112.427350] nix-grpc-daemon[1427]: time=2026-09-21T14:45:04.572Z level=INFO msg="Uploading 1 narinfos"6365lb # [ 112.435432] niks3-server[1117]: 2026/09/21 14:45:03 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6366lb # [ 112.442856] niks3-server[1117]: 2026/09/21 14:45:03 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6367lb # [ 112.447051] niks3-server[1117]: 2026/09/21 14:45:03 INFO Completed upload id=436368worker2 # [ 112.444834] nix-grpc-daemon[1427]: time=2026-09-21T14:45:04.590Z level=INFO msg="Upload complete. (261ms)"6369worker2 # [ 112.447193] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=0 paths=1 nar_bytes_in=4246370worker2 # [ 112.449904] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337886371worker2 # [ 112.455383] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6372worker2 # [ 112.460396] nix-daemon[850]: reaped child process 1425, status = succeeded6373worker2 # [ 112.465043] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6374client # [ 112.458395] nix[1271]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6375client: must succeed: systemctl kill -s INT intr6376client # [ 113.454252] systemd[1]: intr.service: Sent signal SIGINT to main process 1271 (nix) on client request.6377worker2 # [ 113.463907] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:05Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6378client # [ 113.461347] nix[1271]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6379client: (finished: must succeed: systemctl kill -s INT intr, in 0.04 seconds)6380client # [ 113.467156] nix[1271]: error: interrupted by the user6381worker2 # [ 113.481582] nix-daemon[1455]: unexpected Nix daemon error: error: interrupted by the user6382worker2 # [ 113.484312] nix-daemon[850]: reaped child process 1453, status = succeeded6383worker2 # [ 113.487717] nix-daemon[850]: reaped child process 1455, status = failed with exit code 16384client # [ 113.495447] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6385client # [ 113.498307] systemd[1]: intr.service: Failed with result 'exit-code'.6386client # [ 113.501871] systemd[1]: intr.service: Consumed 118ms CPU time over 2.411s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.4K outgoing IP traffic.6387(finished: subtest: interrupting the client stops the build on the worker, in 2.46 seconds)6388subtest: a deploy mid-build drains: the build finishes, the new generation takes over6389client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw6390client # [ 113.563870] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw.6391client # Running as unit: sw.service; invocation ID: b3e2c024a64f48bdbebd5ac68f42cc576392client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw, in 0.04 seconds)6393client # [ 113.686685] nix[1291]: don't know how to build these paths:6394client # [ 113.687637] nix[1291]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6395worker1 # [ 113.694765] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:555786396worker1 # [ 113.707968] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:05Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06397worker1 # [ 113.713772] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:05Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16398worker1 # [ 113.752747] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:555786399worker1 # [ 113.763574] nix-grpc-daemon[1673]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 59 ms (attempt 1/5)6400worker1 # [ 113.829984] nix-grpc-daemon[1673]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 197 ms (attempt 2/5)6401worker1 # [ 114.029186] nix-grpc-daemon[1673]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 278 ms (attempt 3/5)6402worker1 # [ 114.309131] nix-grpc-daemon[1673]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 97 ms (attempt 4/5)6403worker1 # [ 114.408499] nix-grpc-daemon[1673]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6404worker1 # [ 114.412310] nix-grpc-daemon[1673]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6405worker1 # [ 114.429185] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:555786406client # [ 114.430081] nix[1291]: copying 1 paths to 'lb:50051'...6407client # [ 114.435136] nix[1291]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6408worker1 # [ 114.445285] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:555786409worker1 # [ 114.456107] nix-daemon[862]: accepted connection from pid 1673, user nix-grpc-daemon (trusted)6410worker1 # [ 114.558036] nix-daemon[862]: accepted connection from pid 1804, user nix-grpc-daemon (trusted)6411worker1 # [ 114.571233] nix-daemon[862]: reaped child process 1807, status = succeeded6412worker1 # [ 114.651446] nix-daemon[862]: accepted connection from pid 1809, user nix-grpc-daemon (trusted)6413worker1 # [ 114.665697] nix-daemon[862]: reaped child process 1812, status = succeeded6414lb # [ 114.671596] niks3-server[1117]: 2026/09/21 14:45:06 INFO Received uploads request method=POST path=/api/pending_closures6415worker1 # [ 114.681949] nix-grpc-daemon[1756]: time=2026-09-21T14:45:06.848Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6416worker1 # [ 114.683664] nix-grpc-daemon[1756]: time=2026-09-21T14:45:06.848Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6417lb # [ 114.703411] niks3-server[1117]: 2026/09/21 14:45:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6418lb # [ 114.709269] niks3-server[1117]: 2026/09/21 14:45:06 INFO Signed narinfos id=44 count=16419worker1 # [ 114.711707] nix-grpc-daemon[1756]: time=2026-09-21T14:45:06.878Z level=INFO msg="Uploading 1 narinfos"6420lb # [ 114.720987] niks3-server[1117]: 2026/09/21 14:45:06 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6421lb # [ 114.734803] niks3-server[1117]: 2026/09/21 14:45:06 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6422worker1 # [ 114.742343] nix-grpc-daemon[1756]: time=2026-09-21T14:45:06.909Z level=INFO msg="Upload complete. (274ms)"6423lb # [ 114.743343] niks3-server[1117]: 2026/09/21 14:45:06 INFO Completed upload id=446424worker1 # [ 114.744647] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55578 duration_s=0 paths=1 nar_bytes_in=4166425lb # [ 114.746294] niks3-server[1117]: 2026/09/21 14:45:06 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6426lb # [ 114.749068] niks3-server[1117]: 2026/09/21 14:45:06 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6427worker1 # [ 114.748228] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:555786428worker1 # [ 114.754828] nix-daemon[862]: accepted connection from pid 1673, user nix-grpc-daemon (trusted)6429worker1 # [ 114.758151] nix-daemon[862]: reaped child process 1802, status = succeeded6430worker1 # [ 114.764145] nix-daemon[862]: accepted connection from pid 1673, user nix-grpc-daemon (trusted)6431client # [ 114.754141] nix[1291]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6432worker1: must succeed: systemctl reload nix-grpc-daemon.service6433worker1 # [ 115.949804] systemd[1]: Reloading Nix worker-protocol over gRPC...6434worker1 # [ 115.976659] systemd[1]: Reloaded Nix worker-protocol over gRPC.6435worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.06 seconds)6436worker1: must succeed: pgrep -f 'read -t [9]0 x'6437worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.03 seconds)6438client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&26439client # don't know how to build these paths:6440client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6441worker2 # [ 116.150750] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:337886442worker1 # [ 116.156872] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:555786443worker1 # [ 116.169528] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:08Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06444worker1 # [ 116.171399] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:08Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16445worker2 # [ 116.219354] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337886446worker2 # [ 116.225763] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337886447client # copying 1 paths to 'lb:50051'...6448worker2 # [ 116.232539] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:337886449client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6450worker2 # [ 116.247443] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6451worker2 # [ 116.333348] nix-daemon[850]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6452worker2 # [ 116.347205] nix-daemon[850]: reaped child process 1501, status = succeeded6453worker2 # [ 116.427233] nix-daemon[850]: accepted connection from pid 1503, user nix-grpc-daemon (trusted)6454worker2 # [ 116.440424] nix-daemon[850]: reaped child process 1506, status = succeeded6455lb # [ 116.449612] niks3-server[1117]: 2026/09/21 14:45:07 INFO Received uploads request method=POST path=/api/pending_closures6456worker2 # [ 116.451672] nix-grpc-daemon[1427]: time=2026-09-21T14:45:08.596Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6457worker2 # [ 116.453135] nix-grpc-daemon[1427]: time=2026-09-21T14:45:08.596Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6458lb # [ 116.464201] niks3-server[1117]: 2026/09/21 14:45:07 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6459lb # [ 116.468240] niks3-server[1117]: 2026/09/21 14:45:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6460lb # [ 116.470682] niks3-server[1117]: 2026/09/21 14:45:07 INFO Signed narinfos id=45 count=16461worker2 # [ 116.467197] nix-grpc-daemon[1427]: time=2026-09-21T14:45:08.612Z level=INFO msg="Uploading 1 narinfos"6462lb # [ 116.473799] niks3-server[1117]: 2026/09/21 14:45:07 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6463lb # [ 116.479057] niks3-server[1117]: 2026/09/21 14:45:07 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6464lb # [ 116.482681] niks3-server[1117]: 2026/09/21 14:45:07 INFO Completed upload id=456465worker2 # [ 116.480613] nix-grpc-daemon[1427]: time=2026-09-21T14:45:08.626Z level=INFO msg="Upload complete. (227ms)"6466lb # [ 116.484858] niks3-server[1117]: 2026/09/21 14:45:07 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6467worker2 # [ 116.482659] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=0 paths=1 nar_bytes_in=5286468worker2 # [ 116.485803] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337886469worker2 # [ 116.491793] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6470worker2 # [ 116.496318] nix-daemon[850]: reaped child process 1496, status = succeeded6471worker2 # [ 116.500504] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6472client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6473client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6474lb # [ 116.603700] envoy[917]: conn peer=[2001:db8:1::1]:51980 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3934 tx=1852 ms=92196475lb # [ 116.605252] envoy[917]: conn peer=[2001:db8:1::1]:46320 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2993 tx=828 ms=22686476worker1 # [ 116.750676] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:08Z level=info event=draining6477client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6478client # farm-a> LOG-farm-a-during-drain6479worker2 # [ 117.336239] nix-daemon[850]: reaped child process 1510, status = succeeded6480worker2 # [ 117.416611] nix-daemon[850]: accepted connection from pid 1526, user nix-grpc-daemon (trusted)6481worker2 # [ 117.429210] nix-daemon[850]: reaped child process 1529, status = succeeded6482lb # [ 117.439375] niks3-server[1117]: 2026/09/21 14:45:08 INFO Received uploads request method=POST path=/api/pending_closures6483worker2 # [ 117.440699] nix-grpc-daemon[1427]: time=2026-09-21T14:45:09.586Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6484worker2 # [ 117.442240] nix-grpc-daemon[1427]: time=2026-09-21T14:45:09.587Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6485lb # [ 117.459398] niks3-server[1117]: 2026/09/21 14:45:08 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6486lb # [ 117.464564] niks3-server[1117]: 2026/09/21 14:45:08 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6487lb # [ 117.468715] niks3-server[1117]: 2026/09/21 14:45:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6488lb # [ 117.471652] niks3-server[1117]: 2026/09/21 14:45:08 INFO Signed narinfos id=46 count=16489worker2 # [ 117.468521] nix-grpc-daemon[1427]: time=2026-09-21T14:45:09.613Z level=INFO msg="Uploading 1 narinfos"6490lb # [ 117.473153] niks3-server[1117]: 2026/09/21 14:45:08 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6491lb # [ 117.478717] niks3-server[1117]: 2026/09/21 14:45:08 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6492lb # [ 117.483193] niks3-server[1117]: 2026/09/21 14:45:08 INFO Completed upload id=466493worker2 # [ 117.479823] nix-grpc-daemon[1427]: time=2026-09-21T14:45:09.625Z level=INFO msg="Upload complete. (171ms)"6494lb # [ 117.486216] niks3-server[1117]: 2026/09/21 14:45:08 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6495worker2 # [ 117.482567] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=0 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16496worker2 # [ 117.489602] nix-daemon[850]: reaped child process 1508, status = succeeded6497worker1 # [ 117.491561] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:09Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06498worker1 # [ 117.493354] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:09Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16499worker2 # [ 117.532513] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337886500worker2 # [ 117.543181] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337886501worker2 # [ 117.553595] nix-grpc-daemon[1427]: time=2026-09-21T14:45:09.699Z level=INFO msg="All 1 paths already cached"6502client # copying 1 paths to 'lb:50051'...6503worker2 # [ 117.560940] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:337886504client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6505worker2 # [ 117.580259] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6506worker2 # [ 117.674905] nix-daemon[850]: accepted connection from pid 1533, user nix-grpc-daemon (trusted)6507worker2 # [ 117.688133] nix-daemon[850]: reaped child process 1536, status = succeeded6508worker2 # [ 117.768956] nix-daemon[850]: accepted connection from pid 1538, user nix-grpc-daemon (trusted)6509worker2 # [ 117.782141] nix-daemon[850]: reaped child process 1541, status = succeeded6510lb # [ 117.791267] niks3-server[1117]: 2026/09/21 14:45:09 INFO Received uploads request method=POST path=/api/pending_closures6511worker2 # [ 117.792481] nix-grpc-daemon[1427]: time=2026-09-21T14:45:09.937Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6512worker2 # [ 117.794561] nix-grpc-daemon[1427]: time=2026-09-21T14:45:09.939Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6513lb # [ 117.830355] niks3-server[1117]: 2026/09/21 14:45:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6514lb # [ 117.834507] niks3-server[1117]: 2026/09/21 14:45:09 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6515lb # [ 117.840590] niks3-server[1117]: 2026/09/21 14:45:09 INFO Signed narinfos id=47 count=16516worker2 # [ 117.839997] nix-grpc-daemon[1427]: time=2026-09-21T14:45:09.985Z level=INFO msg="Uploading 1 narinfos"6517lb # [ 117.851755] niks3-server[1117]: 2026/09/21 14:45:09 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6518lb # [ 117.860984] niks3-server[1117]: 2026/09/21 14:45:09 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6519lb # [ 117.868102] niks3-server[1117]: 2026/09/21 14:45:09 INFO Completed upload id=476520worker2 # [ 117.864534] nix-grpc-daemon[1427]: time=2026-09-21T14:45:10.010Z level=INFO msg="Upload complete. (272ms)"6521lb # [ 117.870168] niks3-server[1117]: 2026/09/21 14:45:09 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6522worker2 # [ 117.867215] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=0 paths=1 nar_bytes_in=6406523worker2 # [ 117.871718] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337886524worker2 # [ 117.880729] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6525worker2 # [ 117.890397] nix-daemon[850]: reaped child process 1531, status = succeeded6526worker2 # [ 117.894668] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6527client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6528client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6529client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6530client # farm-b> LOG-farm-b-during-drain6531worker2 # [ 118.709413] nix-daemon[850]: reaped child process 1545, status = succeeded6532worker2 # [ 118.783265] nix-daemon[850]: accepted connection from pid 1561, user nix-grpc-daemon (trusted)6533worker2 # [ 118.795949] nix-daemon[850]: reaped child process 1564, status = succeeded6534lb # [ 118.807537] niks3-server[1117]: 2026/09/21 14:45:10 INFO Received uploads request method=POST path=/api/pending_closures6535worker2 # [ 118.811796] nix-grpc-daemon[1427]: time=2026-09-21T14:45:10.957Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6536worker2 # [ 118.813318] nix-grpc-daemon[1427]: time=2026-09-21T14:45:10.958Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6537lb # [ 118.843588] niks3-server[1117]: 2026/09/21 14:45:10 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6538lb # [ 118.855171] niks3-server[1117]: 2026/09/21 14:45:10 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6539lb # [ 118.862659] niks3-server[1117]: 2026/09/21 14:45:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6540worker2 # [ 118.866554] nix-grpc-daemon[1427]: time=2026-09-21T14:45:11.011Z level=INFO msg="Uploading 1 narinfos"6541lb # [ 118.870568] niks3-server[1117]: 2026/09/21 14:45:10 INFO Signed narinfos id=48 count=16542lb # [ 118.881954] niks3-server[1117]: 2026/09/21 14:45:10 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6543lb # [ 118.885093] niks3-server[1117]: 2026/09/21 14:45:10 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6544lb # [ 118.889332] niks3-server[1117]: 2026/09/21 14:45:10 INFO Completed upload id=486545worker2 # [ 118.885891] nix-grpc-daemon[1427]: time=2026-09-21T14:45:11.031Z level=INFO msg="Upload complete. (184ms)"6546lb # [ 118.891295] niks3-server[1117]: 2026/09/21 14:45:10 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6547worker2 # [ 118.890587] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16548worker1 # [ 118.897639] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:11Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06549worker1 # [ 118.899705] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:11Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16550worker2 # [ 118.903883] nix-daemon[850]: reaped child process 1543, status = succeeded6551worker2 # [ 118.942167] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337886552worker2 # [ 118.955318] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337886553worker2 # [ 118.969654] nix-grpc-daemon[1427]: time=2026-09-21T14:45:11.114Z level=INFO msg="All 2 paths already cached"6554client # copying 1 paths to 'lb:50051'...6555worker2 # [ 118.978368] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:337886556client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6557worker2 # [ 118.995508] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6558worker2 # [ 119.098860] nix-daemon[850]: accepted connection from pid 1568, user nix-grpc-daemon (trusted)6559worker2 # [ 119.111779] nix-daemon[850]: reaped child process 1571, status = succeeded6560worker2 # [ 119.193210] nix-daemon[850]: accepted connection from pid 1574, user nix-grpc-daemon (trusted)6561worker2 # [ 119.206388] nix-daemon[850]: reaped child process 1577, status = succeeded6562lb # [ 119.215772] niks3-server[1117]: 2026/09/21 14:45:10 INFO Received uploads request method=POST path=/api/pending_closures6563worker2 # [ 119.216879] nix-grpc-daemon[1427]: time=2026-09-21T14:45:11.362Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6564worker2 # [ 119.218687] nix-grpc-daemon[1427]: time=2026-09-21T14:45:11.363Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6565lb # [ 119.231714] niks3-server[1117]: 2026/09/21 14:45:10 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6566lb # [ 119.234599] niks3-server[1117]: 2026/09/21 14:45:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6567lb # [ 119.236647] niks3-server[1117]: 2026/09/21 14:45:10 INFO Signed narinfos id=49 count=16568worker2 # [ 119.234919] nix-grpc-daemon[1427]: time=2026-09-21T14:45:11.380Z level=INFO msg="Uploading 1 narinfos"6569lb # [ 119.240829] niks3-server[1117]: 2026/09/21 14:45:10 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6570lb # [ 119.245514] niks3-server[1117]: 2026/09/21 14:45:10 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6571lb # [ 119.248832] niks3-server[1117]: 2026/09/21 14:45:10 INFO Completed upload id=496572worker2 # [ 119.245429] nix-grpc-daemon[1427]: time=2026-09-21T14:45:11.390Z level=INFO msg="Upload complete. (235ms)"6573worker2 # [ 119.247501] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=0 paths=1 nar_bytes_in=7686574lb # [ 119.251732] niks3-server[1117]: 2026/09/21 14:45:10 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6575worker2 # [ 119.250747] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337886576worker2 # [ 119.257924] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6577worker2 # [ 119.260428] nix-daemon[850]: reaped child process 1566, status = succeeded6578worker2 # [ 119.265875] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6579client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6580client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6581client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6582client # farm-top> LOG-farm-top-during-drain6583worker2 # [ 120.507490] nix-daemon[850]: reaped child process 1581, status = succeeded6584worker2 # [ 120.587904] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6585worker2 # [ 120.602202] nix-daemon[850]: reaped child process 1600, status = succeeded6586lb # [ 120.610311] niks3-server[1117]: 2026/09/21 14:45:12 INFO Received uploads request method=POST path=/api/pending_closures6587worker2 # [ 120.611526] nix-grpc-daemon[1427]: time=2026-09-21T14:45:12.756Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6588worker2 # [ 120.613029] nix-grpc-daemon[1427]: time=2026-09-21T14:45:12.758Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6589lb # [ 120.631891] niks3-server[1117]: 2026/09/21 14:45:12 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6590lb # [ 120.635492] niks3-server[1117]: 2026/09/21 14:45:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6591lb # [ 120.641083] niks3-server[1117]: 2026/09/21 14:45:12 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6592lb # [ 120.644218] niks3-server[1117]: 2026/09/21 14:45:12 INFO Signed narinfos id=50 count=16593worker2 # [ 120.643609] nix-grpc-daemon[1427]: time=2026-09-21T14:45:12.788Z level=INFO msg="Uploading 1 narinfos"6594lb # [ 120.648449] niks3-server[1117]: 2026/09/21 14:45:12 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6595lb # [ 120.655224] niks3-server[1117]: 2026/09/21 14:45:12 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6596lb # [ 120.658822] niks3-server[1117]: 2026/09/21 14:45:12 INFO Completed upload id=506597worker2 # [ 120.655373] nix-grpc-daemon[1427]: time=2026-09-21T14:45:12.800Z level=INFO msg="Upload complete. (181ms)"6598lb # [ 120.662266] niks3-server[1117]: 2026/09/21 14:45:12 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6599worker2 # [ 120.659069] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16600worker2 # [ 120.665333] nix-daemon[850]: reaped child process 1579, status = succeeded6601client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&2, in 4.68 seconds)6602client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath6603client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath, in 0.12 seconds)6604worker1: must succeed: pkill -f 'read -t [9]0 x'6605worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.04 seconds)6606client: waiting for success: ! systemctl is-active sw6607worker1 # [ 121.009616] nix-daemon[862]: reaped child process 1820, status = succeeded6608worker1 # [ 121.086295] nix-daemon[862]: accepted connection from pid 1864, user nix-grpc-daemon (trusted)6609worker1 # [ 121.098899] nix-daemon[862]: reaped child process 1867, status = succeeded6610lb # [ 121.106639] niks3-server[1117]: 2026/09/21 14:45:12 INFO Received uploads request method=POST path=/api/pending_closures6611worker1 # [ 121.114581] nix-grpc-daemon[1756]: time=2026-09-21T14:45:13.281Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6612worker1 # [ 121.116098] nix-grpc-daemon[1756]: time=2026-09-21T14:45:13.283Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6613lb # [ 121.149618] niks3-server[1117]: 2026/09/21 14:45:12 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6614lb # [ 121.155448] niks3-server[1117]: 2026/09/21 14:45:12 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6615lb # [ 121.163425] niks3-server[1117]: 2026/09/21 14:45:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6616lb # [ 121.169098] niks3-server[1117]: 2026/09/21 14:45:12 INFO Signed narinfos id=51 count=16617worker1 # [ 121.172312] nix-grpc-daemon[1756]: time=2026-09-21T14:45:13.338Z level=INFO msg="Uploading 1 narinfos"6618lb # [ 121.176556] niks3-server[1117]: 2026/09/21 14:45:12 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6619lb # [ 121.186118] niks3-server[1117]: 2026/09/21 14:45:12 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6620lb # [ 121.192062] niks3-server[1117]: 2026/09/21 14:45:12 INFO Completed upload id=516621worker1 # [ 121.191345] nix-grpc-daemon[1756]: time=2026-09-21T14:45:13.358Z level=INFO msg="Upload complete. (214ms)"6622lb # [ 121.194510] niks3-server[1117]: 2026/09/21 14:45:12 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6623worker1 # [ 121.198811] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55578 duration_s=6 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16624worker1 # [ 121.213270] nix-daemon[862]: reaped child process 1818, status = succeeded6625client # [ 121.217636] systemd[1]: sw.service: Deactivated successfully.6626client # [ 121.221822] systemd[1]: sw.service: Consumed 121ms CPU time over 7.655s wall clock time, 39.4M memory peak, 7.5K incoming IP traffic, 8.5K outgoing IP traffic.6627worker1 # [ 121.752508] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:13Z level=info event=signal_exit6628lb # [ 121.761812] niks3-server[1117]: 2026/09/21 14:45:13 INFO lead: released remote=192.168.1.3:513586629client: (finished: waiting for success: ! systemctl is-active sw, in 1.06 seconds)6630client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6631client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.03 seconds)6632worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6633worker2 # [ 122.760290] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:14Z level=info event=scheduler_restarting addr=lb:500516634worker1 # [ 122.766594] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:14Z level=info event=scheduler_restarting6635worker1 # [ 122.770299] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:14Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:555626636worker1 # [ 122.774286] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:14Z level=info event=worker_gone addr=192.168.1.4:500516637worker1 # [ 122.777718] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:14Z level=info event=scheduler_restarting addr=lb:500516638worker1 # [ 122.781072] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:14Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:555626639worker1 # [ 122.784933] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:14Z level=info event=worker_gone addr=192.168.1.3:500516640worker1 # [ 122.794446] nix-grpc-daemon[1673]: E0000 00:00:1790001914.961520 1718 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true6641worker1 # [ 122.799266] nix-grpc-daemon[1673]: E0000 00:00:1790001914.961712 1718 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6642worker1 # [ 122.810309] nix-daemon[862]: reaped child process 1727, status = succeeded6643worker2 # [ 122.960593] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:15Z level=info event=scheduler_connected addr=lb:500516644worker2 # [ 122.964701] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6645worker1 # [ 122.976564] nix-grpc-daemon[1673]: ts=2026-09-21T14:45:15Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6646worker2 # [ 123.165767] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:15Z level=info event=scheduler_connected addr=lb:500516647worker2 # [ 123.169732] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6648worker2 # [ 123.370807] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:15Z level=info event=scheduler_connected addr=lb:500516649worker2 # [ 123.374715] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6650worker2 # [ 123.575782] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:15Z level=info event=scheduler_connected addr=lb:500516651worker2 # [ 123.579664] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6652worker2 # [ 123.781477] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:15Z level=info event=scheduler_connected addr=lb:500516653worker2 # [ 123.785367] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6654worker2 # [ 123.986542] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:16Z level=info event=scheduler_connected addr=lb:500516655worker2 # [ 123.990565] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6656worker2 # [ 124.191433] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:16Z level=info event=scheduler_connected addr=lb:500516657worker2 # [ 124.195357] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6658worker2 # [ 124.396291] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:16Z level=info event=scheduler_connected addr=lb:500516659worker2 # [ 124.400243] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6660worker2 # [ 124.601227] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:16Z level=info event=scheduler_connected addr=lb:500516661worker2 # [ 124.605170] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6662worker1 # [ 124.773448] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6663worker1 # [ 124.776637] systemd[1]: nix-grpc-daemon.service: Consumed 649ms CPU time over 20.157s wall clock time, 36.3M memory peak, 25.1K incoming IP traffic, 31.5K outgoing IP traffic.6664worker1 # [ 124.793960] systemd[1]: Starting Nix worker-protocol over gRPC...6665worker2 # [ 124.807344] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:16Z level=info event=scheduler_connected addr=lb:500516666worker2 # [ 124.810896] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6667worker1 # [ 124.883697] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6668worker1 # [ 124.888149] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6669worker1 # [ 124.892203] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16670worker1 # [ 124.898141] nix-grpc-daemon[1881]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6671worker1 # [ 124.900654] nix-grpc-daemon[1881]: E0000 00:00:1790001917.057932 1881 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6672worker1 # [ 124.904096] nix-grpc-daemon[1881]: E0000 00:00:1790001917.061805 1881 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6673lb # [ 124.907192] niks3-server[1117]: 2026/09/21 14:45:16 INFO lead: acquired remote=192.168.1.3:374106674worker1 # [ 124.908194] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6675worker1 # [ 124.911121] systemd[1]: Started Nix worker-protocol over gRPC.6676worker1 # [ 124.916081] nix-grpc-daemon[1881]: E0000 00:00:1790001917.074081 1923 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6677worker1 # [ 124.919115] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=scheduler_take_over6678worker1 # [ 124.924233] nix-grpc-daemon[1881]: E0000 00:00:1790001917.091577 1923 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6679worker1 # [ 124.929126] nix-grpc-daemon[1881]: E0000 00:00:1790001917.093876 1923 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6680worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.99 seconds)6681lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6682lb: must succeed: curl -sf localhost:9901/clusters6683worker1 # [ 124.939781] nix-grpc-daemon[1881]: E0000 00:00:1790001917.107120 1923 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6684worker1 # [ 124.951069] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=scheduler_connected addr=lb:500516685worker1 # [ 124.952723] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6686lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6687worker2 # [ 125.011972] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:17Z level=info event=scheduler_connected addr=lb:500516688worker2 # [ 125.015892] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6689worker1 # [ 125.153810] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=scheduler_connected addr=lb:500516690worker1 # [ 125.158243] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6691worker2 # [ 125.216965] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:17Z level=info event=scheduler_connected addr=lb:500516692worker2 # [ 125.221079] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6693worker1 # [ 125.358844] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=scheduler_connected addr=lb:500516694worker1 # [ 125.363104] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6695worker2 # [ 125.421839] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:17Z level=info event=scheduler_connected addr=lb:500516696worker2 # [ 125.425684] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6697worker1 # [ 125.563889] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=scheduler_connected addr=lb:500516698worker1 # [ 125.567989] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6699worker2 # [ 125.627114] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:17Z level=info event=scheduler_connected addr=lb:500516700worker2 # [ 125.631104] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6701worker1 # [ 125.769117] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=scheduler_connected addr=lb:500516702worker1 # [ 125.778841] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6703worker2 # [ 125.832717] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:17Z level=info event=scheduler_connected addr=lb:500516704worker1 # [ 125.845266] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:18Z level=info event=oidc_jwks provider=mock keys=16705worker1 # [ 125.849254] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:18Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:542186706worker1 # [ 125.853178] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:18Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06707worker1 # [ 125.974804] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:18Z level=info event=scheduler_connected addr=lb:500516708worker1 # [ 125.979598] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:18Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:542186709worker1 # [ 125.983283] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:18Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06710lb: must succeed: curl -sf localhost:9901/clusters6711lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6712lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.10 seconds)6713lb: must succeed: curl -sf localhost:9901/clusters6714lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6715worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6716worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6717(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 12.57 seconds)6718subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6719client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16720worker2 # [ 126.245072] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:337886721worker2 # [ 126.249944] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337886722worker1 # [ 126.298358] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:18Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06723worker1 # [ 126.302408] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:18Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16724worker1 # [ 126.307321] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:18Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06725worker1 # [ 126.312616] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:18Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06726worker1 # [ 126.356080] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306727worker1 # [ 126.367474] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6728worker1 # [ 126.396096] nix-grpc-daemon[1881]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 50 ms (attempt 1/5)6729worker1 # [ 126.453849] nix-grpc-daemon[1881]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 44 ms (attempt 2/5)6730worker1 # [ 126.499678] nix-grpc-daemon[1881]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 289 ms (attempt 3/5)6731lb # [ 126.604305] envoy[917]: conn peer=[2001:db8:1::1]:36264 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9581 tx=4045 ms=45206732lb # [ 126.608127] envoy[917]: conn peer=[2001:db8:1::1]:36256 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2992 tx=1447 ms=75176733lb # [ 126.611739] envoy[917]: conn peer=[2001:db8:1::3]:34688 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=958 tx=344 ms=181776734worker1 # [ 126.796773] nix-grpc-daemon[1881]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 446 ms (attempt 4/5)6735worker1 # [ 127.252257] nix-grpc-daemon[1881]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6736worker1 # [ 127.255580] nix-grpc-daemon[1881]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6737worker1 # [ 127.272142] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:542306738worker1 # [ 127.285085] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:542306739worker1 # [ 127.295247] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6740worker1 # [ 127.317979] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:19Z level=info event=niks3_push_started pid=19466741worker1 # [ 127.418200] nix-grpc-daemon[1946]: time=2026-09-21T14:45:19.585Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6742worker1 # [ 127.504580] nix-daemon[862]: accepted connection from pid 1959, user nix-grpc-daemon (trusted)6743worker1 # [ 127.516198] nix-daemon[862]: reaped child process 1962, status = succeeded6744worker1 # [ 127.599755] nix-daemon[862]: accepted connection from pid 1964, user nix-grpc-daemon (trusted)6745worker1 # [ 127.613287] nix-daemon[862]: reaped child process 1967, status = succeeded6746lb # [ 127.619945] niks3-server[1117]: 2026/09/21 14:45:19 INFO Received uploads request method=POST path=/api/pending_closures6747worker1 # [ 127.629636] nix-grpc-daemon[1946]: time=2026-09-21T14:45:19.795Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6748worker1 # [ 127.631027] nix-grpc-daemon[1946]: time=2026-09-21T14:45:19.795Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6749lb # [ 127.655396] niks3-server[1117]: 2026/09/21 14:45:19 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6750lb # [ 127.663163] niks3-server[1117]: 2026/09/21 14:45:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6751lb # [ 127.667644] niks3-server[1117]: 2026/09/21 14:45:19 INFO Signed narinfos id=52 count=16752worker1 # [ 127.669241] nix-grpc-daemon[1946]: time=2026-09-21T14:45:19.836Z level=INFO msg="Uploading 1 narinfos"6753lb # [ 127.680213] niks3-server[1117]: 2026/09/21 14:45:19 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6754lb # [ 127.683897] niks3-server[1117]: 2026/09/21 14:45:19 INFO Completed upload id=526755worker1 # [ 127.684922] nix-grpc-daemon[1946]: time=2026-09-21T14:45:19.852Z level=INFO msg="Upload complete. (263ms)"6756worker1 # [ 127.687264] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=0 paths=1 nar_bytes_in=5046757worker1 # [ 127.689279] nix-grpc-daemon[1946]: time=2026-09-21T14:45:19.854Z level=WARN msg="Failed to register uploaded object" key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo error="server returned 404: object is not pending upload\n"6758worker1 # [ 127.692435] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306759lb # [ 127.696819] niks3-server[1117]: 2026/09/21 14:45:19 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6760worker1 # [ 127.698838] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6761worker1 # [ 127.702833] nix-daemon[862]: reaped child process 1944, status = succeeded6762worker1 # [ 127.707507] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6763worker1 # [ 128.938974] nix-daemon[862]: reaped child process 1971, status = succeeded6764worker1 # [ 129.027660] nix-daemon[862]: accepted connection from pid 1987, user nix-grpc-daemon (trusted)6765worker1 # [ 129.038926] nix-daemon[862]: reaped child process 1990, status = succeeded6766lb # [ 129.048379] niks3-server[1117]: 2026/09/21 14:45:20 INFO Received uploads request method=POST path=/api/pending_closures6767worker1 # [ 129.059548] nix-grpc-daemon[1946]: time=2026-09-21T14:45:21.226Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6768worker1 # [ 129.061134] nix-grpc-daemon[1946]: time=2026-09-21T14:45:21.228Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6769lb # [ 129.092289] niks3-server[1117]: 2026/09/21 14:45:20 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6770lb # [ 129.099456] niks3-server[1117]: 2026/09/21 14:45:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6771lb # [ 129.103937] niks3-server[1117]: 2026/09/21 14:45:20 INFO Signed narinfos id=53 count=16772worker1 # [ 129.107112] nix-grpc-daemon[1946]: time=2026-09-21T14:45:21.273Z level=INFO msg="Uploading 1 narinfos"6773lb # [ 129.111460] niks3-server[1117]: 2026/09/21 14:45:20 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6774lb # [ 129.118498] niks3-server[1117]: 2026/09/21 14:45:20 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6775lb # [ 129.122971] niks3-server[1117]: 2026/09/21 14:45:20 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6776lb # [ 129.126359] niks3-server[1117]: 2026/09/21 14:45:20 INFO Completed upload id=536777lb # [ 129.129300] niks3-server[1117]: 2026/09/21 14:45:20 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6778worker1 # [ 129.125548] nix-grpc-daemon[1946]: time=2026-09-21T14:45:21.292Z level=INFO msg="Upload complete. (210ms)"6779worker1 # [ 129.132760] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=16780worker1 # [ 129.143787] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:21Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16781worker1 # [ 129.148223] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:21Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06782worker1 # [ 129.155241] nix-daemon[862]: reaped child process 1969, status = succeeded6783worker1 # [ 129.185912] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306784worker1 # [ 129.198462] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:542306785worker1 # [ 129.210183] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:542306786worker1 # [ 129.218432] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6787worker1 # [ 129.311724] nix-daemon[862]: accepted connection from pid 1996, user nix-grpc-daemon (trusted)6788worker1 # [ 129.324912] nix-daemon[862]: reaped child process 1999, status = succeeded6789worker1 # [ 129.404871] nix-daemon[862]: accepted connection from pid 2001, user nix-grpc-daemon (trusted)6790worker1 # [ 129.418677] nix-daemon[862]: reaped child process 2004, status = succeeded6791lb # [ 129.424321] niks3-server[1117]: 2026/09/21 14:45:20 INFO Received uploads request method=POST path=/api/pending_closures6792worker1 # [ 129.431367] nix-grpc-daemon[1946]: time=2026-09-21T14:45:21.598Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6793worker1 # [ 129.433568] nix-grpc-daemon[1946]: time=2026-09-21T14:45:21.600Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6794lb # [ 129.456201] niks3-server[1117]: 2026/09/21 14:45:20 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6795lb # [ 129.465968] niks3-server[1117]: 2026/09/21 14:45:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6796lb # [ 129.471544] niks3-server[1117]: 2026/09/21 14:45:20 INFO Signed narinfos id=54 count=16797worker1 # [ 129.473778] nix-grpc-daemon[1946]: time=2026-09-21T14:45:21.640Z level=INFO msg="Uploading 1 narinfos"6798lb # [ 129.485689] niks3-server[1117]: 2026/09/21 14:45:20 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6799lb # [ 129.492464] niks3-server[1117]: 2026/09/21 14:45:20 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6800worker1 # [ 129.498469] nix-grpc-daemon[1946]: time=2026-09-21T14:45:21.665Z level=INFO msg="Upload complete. (268ms)"6801lb # [ 129.499333] niks3-server[1117]: 2026/09/21 14:45:20 INFO Completed upload id=546802lb # [ 129.503934] niks3-server[1117]: 2026/09/21 14:45:20 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6803worker1 # [ 129.501701] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=0 paths=1 nar_bytes_in=5046804worker1 # [ 129.507355] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306805worker1 # [ 129.516472] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6806worker1 # [ 129.526090] nix-daemon[862]: reaped child process 1994, status = succeeded6807worker1 # [ 129.528772] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6808worker1 # [ 130.970839] nix-daemon[862]: reaped child process 2008, status = succeeded6809worker1 # [ 131.051531] nix-daemon[862]: accepted connection from pid 2024, user nix-grpc-daemon (trusted)6810worker1 # [ 131.064770] nix-daemon[862]: reaped child process 2027, status = succeeded6811lb # [ 131.072214] niks3-server[1117]: 2026/09/21 14:45:22 INFO Received uploads request method=POST path=/api/pending_closures6812worker1 # [ 131.079883] nix-grpc-daemon[1946]: time=2026-09-21T14:45:23.246Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6813worker1 # [ 131.081438] nix-grpc-daemon[1946]: time=2026-09-21T14:45:23.248Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6814lb # [ 131.110670] niks3-server[1117]: 2026/09/21 14:45:22 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6815lb # [ 131.124448] niks3-server[1117]: 2026/09/21 14:45:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6816lb # [ 131.131115] niks3-server[1117]: 2026/09/21 14:45:22 INFO Signed narinfos id=55 count=16817worker1 # [ 131.132991] nix-grpc-daemon[1946]: time=2026-09-21T14:45:23.300Z level=INFO msg="Uploading 1 narinfos"6818lb # [ 131.135518] niks3-server[1117]: 2026/09/21 14:45:22 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6819lb # [ 131.140222] niks3-server[1117]: 2026/09/21 14:45:22 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6820lb # [ 131.150368] niks3-server[1117]: 2026/09/21 14:45:22 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6821lb # [ 131.153365] niks3-server[1117]: 2026/09/21 14:45:22 INFO Completed upload id=556822lb # [ 131.156234] niks3-server[1117]: 2026/09/21 14:45:22 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6823worker1 # [ 131.153895] nix-grpc-daemon[1946]: time=2026-09-21T14:45:23.321Z level=INFO msg="Upload complete. (216ms)"6824worker1 # [ 131.161907] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=16825worker1 # [ 131.174240] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:23Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16826worker1 # [ 131.177813] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306827worker1 # [ 131.180881] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:23Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06828worker1 # [ 131.189389] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:542306829worker1 # [ 131.194301] nix-daemon[862]: reaped child process 2006, status = succeeded6830worker1 # [ 131.196283] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:542306831worker1 # [ 131.200888] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6832worker1 # [ 131.283589] nix-daemon[862]: accepted connection from pid 2033, user nix-grpc-daemon (trusted)6833worker1 # [ 131.296303] nix-daemon[862]: reaped child process 2036, status = succeeded6834worker1 # [ 131.378078] nix-daemon[862]: accepted connection from pid 2038, user nix-grpc-daemon (trusted)6835worker1 # [ 131.391549] nix-daemon[862]: reaped child process 2041, status = succeeded6836lb # [ 131.398353] niks3-server[1117]: 2026/09/21 14:45:22 INFO Received uploads request method=POST path=/api/pending_closures6837worker1 # [ 131.405333] nix-grpc-daemon[1946]: time=2026-09-21T14:45:23.572Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6838worker1 # [ 131.406823] nix-grpc-daemon[1946]: time=2026-09-21T14:45:23.574Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6839lb # [ 131.426416] niks3-server[1117]: 2026/09/21 14:45:22 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6840lb # [ 131.437287] niks3-server[1117]: 2026/09/21 14:45:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6841lb # [ 131.442491] niks3-server[1117]: 2026/09/21 14:45:22 INFO Signed narinfos id=56 count=16842worker1 # [ 131.444361] nix-grpc-daemon[1946]: time=2026-09-21T14:45:23.611Z level=INFO msg="Uploading 1 narinfos"6843lb # [ 131.450645] niks3-server[1117]: 2026/09/21 14:45:22 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6844lb # [ 131.464444] niks3-server[1117]: 2026/09/21 14:45:22 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6845worker1 # [ 131.469053] nix-grpc-daemon[1946]: time=2026-09-21T14:45:23.636Z level=INFO msg="Upload complete. (263ms)"6846lb # [ 131.469976] niks3-server[1117]: 2026/09/21 14:45:22 INFO Completed upload id=566847worker1 # [ 131.471968] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=0 paths=1 nar_bytes_in=5046848lb # [ 131.475519] niks3-server[1117]: 2026/09/21 14:45:22 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6849worker1 # [ 131.479073] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306850worker1 # [ 131.488963] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6851worker1 # [ 131.491289] nix-daemon[862]: reaped child process 2031, status = succeeded6852worker1 # [ 131.497401] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6853worker1 # [ 132.337914] nix-daemon[862]: reaped child process 2045, status = succeeded6854worker1 # [ 132.417681] nix-daemon[862]: accepted connection from pid 2061, user nix-grpc-daemon (trusted)6855worker1 # [ 132.428990] nix-daemon[862]: reaped child process 2064, status = succeeded6856lb # [ 132.437551] niks3-server[1117]: 2026/09/21 14:45:23 INFO Received uploads request method=POST path=/api/pending_closures6857worker1 # [ 132.444993] nix-grpc-daemon[1946]: time=2026-09-21T14:45:24.612Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6858worker1 # [ 132.447726] nix-grpc-daemon[1946]: time=2026-09-21T14:45:24.613Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"6859lb # [ 132.475826] niks3-server[1117]: 2026/09/21 14:45:23 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv6860lb # [ 132.488132] niks3-server[1117]: 2026/09/21 14:45:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6861worker1 # [ 132.491887] nix-grpc-daemon[1946]: time=2026-09-21T14:45:24.659Z level=INFO msg="Uploading 1 narinfos"6862lb # [ 132.492481] niks3-server[1117]: 2026/09/21 14:45:23 INFO Signed narinfos id=57 count=16863lb # [ 132.500698] niks3-server[1117]: 2026/09/21 14:45:23 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6864lb # [ 132.504618] niks3-server[1117]: 2026/09/21 14:45:23 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls6865lb # [ 132.514313] niks3-server[1117]: 2026/09/21 14:45:23 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6866lb # [ 132.518155] niks3-server[1117]: 2026/09/21 14:45:23 INFO Completed upload id=576867worker1 # [ 132.517258] nix-grpc-daemon[1946]: time=2026-09-21T14:45:24.684Z level=INFO msg="Upload complete. (213ms)"6868lb # [ 132.519809] niks3-server[1117]: 2026/09/21 14:45:23 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo6869worker1 # [ 132.525694] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=16870worker1 # [ 132.535270] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:24Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=16871worker1 # [ 132.539233] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:24Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=06872worker1 # [ 132.548489] nix-daemon[862]: reaped child process 2043, status = succeeded6873worker1 # [ 132.568204] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306874worker1 # [ 132.579717] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:542306875worker1 # [ 132.593645] nix-grpc-daemon[1946]: time=2026-09-21T14:45:24.760Z level=INFO msg="All 1 paths already cached"6876worker1 # [ 132.600382] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:542306877worker1 # [ 132.608134] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6878worker1 # [ 132.700301] nix-daemon[862]: accepted connection from pid 2070, user nix-grpc-daemon (trusted)6879worker1 # [ 132.713035] nix-daemon[862]: reaped child process 2073, status = succeeded6880worker1 # [ 132.795384] nix-daemon[862]: accepted connection from pid 2075, user nix-grpc-daemon (trusted)6881worker1 # [ 132.808302] nix-daemon[862]: reaped child process 2078, status = succeeded6882lb # [ 132.815311] niks3-server[1117]: 2026/09/21 14:45:24 INFO Received uploads request method=POST path=/api/pending_closures6883worker1 # [ 132.823602] nix-grpc-daemon[1946]: time=2026-09-21T14:45:24.990Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6884worker1 # [ 132.825301] nix-grpc-daemon[1946]: time=2026-09-21T14:45:24.992Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"6885lb # [ 132.844701] niks3-server[1117]: 2026/09/21 14:45:24 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst6886lb # [ 132.857272] niks3-server[1117]: 2026/09/21 14:45:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6887lb # [ 132.862241] niks3-server[1117]: 2026/09/21 14:45:24 INFO Signed narinfos id=58 count=16888worker1 # [ 132.865972] nix-grpc-daemon[1946]: time=2026-09-21T14:45:25.033Z level=INFO msg="Uploading 1 narinfos"6889lb # [ 132.871153] niks3-server[1117]: 2026/09/21 14:45:24 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls6890lb # [ 132.884956] niks3-server[1117]: 2026/09/21 14:45:24 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6891lb # [ 132.892484] niks3-server[1117]: 2026/09/21 14:45:24 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo6892lb # [ 132.896477] niks3-server[1117]: 2026/09/21 14:45:24 INFO Completed upload id=586893worker1 # [ 132.895699] nix-grpc-daemon[1946]: time=2026-09-21T14:45:25.062Z level=INFO msg="Upload complete. (276ms)"6894worker1 # [ 132.900579] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=0 paths=1 nar_bytes_in=6246895worker1 # [ 132.907307] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306896worker1 # [ 132.918235] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6897worker1 # [ 132.926641] nix-daemon[862]: reaped child process 2068, status = succeeded6898worker1 # [ 132.934380] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6899worker1 # [ 133.751355] nix-daemon[862]: reaped child process 2082, status = succeeded6900worker1 # [ 133.822890] nix-daemon[862]: accepted connection from pid 2098, user nix-grpc-daemon (trusted)6901worker1 # [ 133.836824] nix-daemon[862]: reaped child process 2101, status = succeeded6902lb # [ 133.842867] niks3-server[1117]: 2026/09/21 14:45:25 INFO Received uploads request method=POST path=/api/pending_closures6903worker1 # [ 133.849379] nix-grpc-daemon[1946]: time=2026-09-21T14:45:26.016Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6904worker1 # [ 133.850910] nix-grpc-daemon[1946]: time=2026-09-21T14:45:26.018Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"6905lb # [ 133.868092] niks3-server[1117]: 2026/09/21 14:45:25 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv6906lb # [ 133.871705] niks3-server[1117]: 2026/09/21 14:45:25 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst6907lb # [ 133.875722] niks3-server[1117]: 2026/09/21 14:45:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6908lb # [ 133.877745] niks3-server[1117]: 2026/09/21 14:45:25 INFO Signed narinfos id=59 count=16909worker1 # [ 133.877255] nix-grpc-daemon[1946]: time=2026-09-21T14:45:26.044Z level=INFO msg="Uploading 1 narinfos"6910lb # [ 133.884104] niks3-server[1117]: 2026/09/21 14:45:25 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6911lb # [ 133.886242] niks3-server[1117]: 2026/09/21 14:45:25 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls6912lb # [ 133.890188] niks3-server[1117]: 2026/09/21 14:45:25 INFO Completed upload id=596913worker1 # [ 133.889129] nix-grpc-daemon[1946]: time=2026-09-21T14:45:26.056Z level=INFO msg="Upload complete. (148ms)"6914lb # [ 133.892550] niks3-server[1117]: 2026/09/21 14:45:25 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo6915worker1 # [ 133.892147] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=0 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=16916worker1 # [ 133.896830] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:26Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=16917worker1 # [ 133.898834] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306918worker1 # [ 133.902091] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:26Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=06919worker1 # [ 133.908606] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:542306920worker1 # [ 133.910721] nix-daemon[862]: reaped child process 2080, status = succeeded6921worker1 # [ 133.914323] nix-grpc-daemon[1946]: time=2026-09-21T14:45:26.081Z level=INFO msg="All 1 paths already cached"6922worker1 # [ 133.918232] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:542306923worker1 # [ 133.922834] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6924worker1 # [ 134.005513] nix-daemon[862]: accepted connection from pid 2107, user nix-grpc-daemon (trusted)6925worker1 # [ 134.019629] nix-daemon[862]: reaped child process 2110, status = succeeded6926worker1 # [ 134.098697] nix-daemon[862]: accepted connection from pid 2112, user nix-grpc-daemon (trusted)6927worker1 # [ 134.111774] nix-daemon[862]: reaped child process 2115, status = succeeded6928lb # [ 134.118571] niks3-server[1117]: 2026/09/21 14:45:25 INFO Received uploads request method=POST path=/api/pending_closures6929worker1 # [ 134.122365] nix-grpc-daemon[1946]: time=2026-09-21T14:45:26.289Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6930worker1 # [ 134.123744] nix-grpc-daemon[1946]: time=2026-09-21T14:45:26.289Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"6931lb # [ 134.132822] niks3-server[1117]: 2026/09/21 14:45:25 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst6932lb # [ 134.134604] niks3-server[1117]: 2026/09/21 14:45:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6933lb # [ 134.137067] niks3-server[1117]: 2026/09/21 14:45:25 INFO Signed narinfos id=60 count=16934worker1 # [ 134.137722] nix-grpc-daemon[1946]: time=2026-09-21T14:45:26.304Z level=INFO msg="Uploading 1 narinfos"6935lb # [ 134.140145] niks3-server[1117]: 2026/09/21 14:45:25 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls6936lb # [ 134.145126] niks3-server[1117]: 2026/09/21 14:45:25 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6937lb # [ 134.147792] niks3-server[1117]: 2026/09/21 14:45:25 INFO Completed upload id=606938worker1 # [ 134.146848] nix-grpc-daemon[1946]: time=2026-09-21T14:45:26.314Z level=INFO msg="Upload complete. (219ms)"6939worker1 # [ 134.148776] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=0 paths=1 nar_bytes_in=6246940lb # [ 134.151760] niks3-server[1117]: 2026/09/21 14:45:25 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo6941worker1 # [ 134.152968] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306942worker1 # [ 134.159088] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6943worker1 # [ 134.162139] nix-daemon[862]: reaped child process 2105, status = succeeded6944worker1 # [ 134.165851] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6945worker1 # [ 135.405619] nix-daemon[862]: reaped child process 2119, status = succeeded6946worker1 # [ 135.485924] nix-daemon[862]: accepted connection from pid 2135, user nix-grpc-daemon (trusted)6947worker1 # [ 135.499568] nix-daemon[862]: reaped child process 2138, status = succeeded6948lb # [ 135.505966] niks3-server[1117]: 2026/09/21 14:45:26 INFO Received uploads request method=POST path=/api/pending_closures6949worker1 # [ 135.509836] nix-grpc-daemon[1946]: time=2026-09-21T14:45:27.676Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6950worker1 # [ 135.511392] nix-grpc-daemon[1946]: time=2026-09-21T14:45:27.678Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"6951lb # [ 135.530082] niks3-server[1117]: 2026/09/21 14:45:26 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst6952lb # [ 135.533091] niks3-server[1117]: 2026/09/21 14:45:26 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv6953lb # [ 135.536848] niks3-server[1117]: 2026/09/21 14:45:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6954lb # [ 135.538571] niks3-server[1117]: 2026/09/21 14:45:26 INFO Signed narinfos id=61 count=16955worker1 # [ 135.538985] nix-grpc-daemon[1946]: time=2026-09-21T14:45:27.706Z level=INFO msg="Uploading 1 narinfos"6956lb # [ 135.542635] niks3-server[1117]: 2026/09/21 14:45:26 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls6957lb # [ 135.548061] niks3-server[1117]: 2026/09/21 14:45:26 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6958lb # [ 135.550902] niks3-server[1117]: 2026/09/21 14:45:26 INFO Completed upload id=616959worker1 # [ 135.549880] nix-grpc-daemon[1946]: time=2026-09-21T14:45:27.717Z level=INFO msg="Upload complete. (176ms)"6960lb # [ 135.553620] niks3-server[1117]: 2026/09/21 14:45:26 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo6961worker1 # [ 135.554060] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=16962worker1 # [ 135.556734] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:27Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=16963worker1 # [ 135.560219] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306964worker1 # [ 135.563230] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:27Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=06965worker1 # [ 135.568825] nix-daemon[862]: reaped child process 2117, status = succeeded6966worker1 # [ 135.570227] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:542306967worker1 # [ 135.575488] nix-grpc-daemon[1946]: time=2026-09-21T14:45:27.742Z level=INFO msg="All 1 paths already cached"6968worker1 # [ 135.578593] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:542306969worker1 # [ 135.583174] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6970worker1 # [ 135.667132] nix-daemon[862]: accepted connection from pid 2144, user nix-grpc-daemon (trusted)6971worker1 # [ 135.679733] nix-daemon[862]: reaped child process 2147, status = succeeded6972worker1 # [ 135.761842] nix-daemon[862]: accepted connection from pid 2149, user nix-grpc-daemon (trusted)6973worker1 # [ 135.774780] nix-daemon[862]: reaped child process 2152, status = succeeded6974lb # [ 135.781672] niks3-server[1117]: 2026/09/21 14:45:27 INFO Received uploads request method=POST path=/api/pending_closures6975worker1 # [ 135.785237] nix-grpc-daemon[1946]: time=2026-09-21T14:45:27.952Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6976worker1 # [ 135.786772] nix-grpc-daemon[1946]: time=2026-09-21T14:45:27.954Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"6977lb # [ 135.799538] niks3-server[1117]: 2026/09/21 14:45:27 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst6978lb # [ 135.801836] niks3-server[1117]: 2026/09/21 14:45:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6979lb # [ 135.803954] niks3-server[1117]: 2026/09/21 14:45:27 INFO Signed narinfos id=62 count=16980worker1 # [ 135.802983] nix-grpc-daemon[1946]: time=2026-09-21T14:45:27.970Z level=INFO msg="Uploading 1 narinfos"6981lb # [ 135.807137] niks3-server[1117]: 2026/09/21 14:45:27 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls6982lb # [ 135.811884] niks3-server[1117]: 2026/09/21 14:45:27 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6983lb # [ 135.814717] niks3-server[1117]: 2026/09/21 14:45:27 INFO Completed upload id=626984worker1 # [ 135.813742] nix-grpc-daemon[1946]: time=2026-09-21T14:45:27.981Z level=INFO msg="Upload complete. (225ms)"6985worker1 # [ 135.815602] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=0 paths=1 nar_bytes_in=6246986lb # [ 135.818198] niks3-server[1117]: 2026/09/21 14:45:27 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo6987worker1 # [ 135.819660] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542306988worker1 # [ 135.825508] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6989worker1 # [ 135.827807] nix-daemon[862]: reaped child process 2142, status = succeeded6990worker1 # [ 135.832975] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6991worker1 # [ 136.870283] nix-daemon[862]: reaped child process 2156, status = succeeded6992worker1 # [ 136.934342] nix-daemon[862]: accepted connection from pid 2172, user nix-grpc-daemon (trusted)6993worker1 # [ 136.946736] nix-daemon[862]: reaped child process 2175, status = succeeded6994lb # [ 136.955413] niks3-server[1117]: 2026/09/21 14:45:28 INFO Received uploads request method=POST path=/api/pending_closures6995worker1 # [ 136.959136] nix-grpc-daemon[1946]: time=2026-09-21T14:45:29.126Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6996worker1 # [ 136.960795] nix-grpc-daemon[1946]: time=2026-09-21T14:45:29.128Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"6997lb # [ 136.976563] niks3-server[1117]: 2026/09/21 14:45:28 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst6998lb # [ 136.979839] niks3-server[1117]: 2026/09/21 14:45:28 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv6999lb # [ 136.985731] niks3-server[1117]: 2026/09/21 14:45:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign7000lb # [ 136.990941] niks3-server[1117]: 2026/09/21 14:45:28 INFO Signed narinfos id=63 count=17001worker1 # [ 136.992946] nix-grpc-daemon[1946]: time=2026-09-21T14:45:29.160Z level=INFO msg="Uploading 1 narinfos"7002lb # [ 136.995418] niks3-server[1117]: 2026/09/21 14:45:28 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls7003lb # [ 137.000287] niks3-server[1117]: 2026/09/21 14:45:28 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete7004lb # [ 137.003692] niks3-server[1117]: 2026/09/21 14:45:28 INFO Completed upload id=637005worker1 # [ 137.002539] nix-grpc-daemon[1946]: time=2026-09-21T14:45:29.169Z level=INFO msg="Upload complete. (162ms)"7006lb # [ 137.006105] niks3-server[1117]: 2026/09/21 14:45:28 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo7007worker1 # [ 137.005241] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=17008worker1 # [ 137.009841] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:29Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=17009worker1 # [ 137.013393] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:29Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=07010worker1 # [ 137.018088] nix-daemon[862]: reaped child process 2154, status = succeeded7011worker1 # [ 137.052402] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542307012worker1 # [ 137.063086] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:542307013worker1 # [ 137.074875] nix-grpc-daemon[1946]: time=2026-09-21T14:45:29.242Z level=INFO msg="All 2 paths already cached"7014worker1 # [ 137.080318] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:542307015worker1 # [ 137.088132] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)7016worker1 # [ 137.175145] nix-daemon[862]: accepted connection from pid 2181, user nix-grpc-daemon (trusted)7017worker1 # [ 137.187881] nix-daemon[862]: reaped child process 2184, status = succeeded7018worker1 # [ 137.269418] nix-daemon[862]: accepted connection from pid 2186, user nix-grpc-daemon (trusted)7019worker1 # [ 137.282687] nix-daemon[862]: reaped child process 2189, status = succeeded7020lb # [ 137.289593] niks3-server[1117]: 2026/09/21 14:45:28 INFO Received uploads request method=POST path=/api/pending_closures7021worker1 # [ 137.296735] nix-grpc-daemon[1946]: time=2026-09-21T14:45:29.464Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7022worker1 # [ 137.298594] nix-grpc-daemon[1946]: time=2026-09-21T14:45:29.465Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7023lb # [ 137.316585] niks3-server[1117]: 2026/09/21 14:45:28 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7024lb # [ 137.330139] niks3-server[1117]: 2026/09/21 14:45:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7025lb # [ 137.336802] niks3-server[1117]: 2026/09/21 14:45:28 INFO Signed narinfos id=64 count=17026worker1 # [ 137.338806] nix-grpc-daemon[1946]: time=2026-09-21T14:45:29.505Z level=INFO msg="Uploading 1 narinfos"7027lb # [ 137.340647] niks3-server[1117]: 2026/09/21 14:45:28 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7028lb # [ 137.354842] niks3-server[1117]: 2026/09/21 14:45:28 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7029lb # [ 137.362136] niks3-server[1117]: 2026/09/21 14:45:28 INFO Completed upload id=647030worker1 # [ 137.361109] nix-grpc-daemon[1946]: time=2026-09-21T14:45:29.528Z level=INFO msg="Upload complete. (264ms)"7031worker1 # [ 137.364958] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=0 paths=1 nar_bytes_in=7527032lb # [ 137.367162] niks3-server[1117]: 2026/09/21 14:45:28 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7033worker1 # [ 137.372241] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542307034worker1 # [ 137.383230] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)7035worker1 # [ 137.385492] nix-daemon[862]: reaped child process 2179, status = succeeded7036worker1 # [ 137.390750] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)7037worker1 # [ 138.206732] nix-daemon[862]: reaped child process 2193, status = succeeded7038worker1 # [ 138.278356] nix-daemon[862]: accepted connection from pid 2209, user nix-grpc-daemon (trusted)7039worker1 # [ 138.291328] nix-daemon[862]: reaped child process 2212, status = succeeded7040lb # [ 138.299145] niks3-server[1117]: 2026/09/21 14:45:29 INFO Received uploads request method=POST path=/api/pending_closures7041worker1 # [ 138.304279] nix-grpc-daemon[1946]: time=2026-09-21T14:45:30.469Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7042worker1 # [ 138.305689] nix-grpc-daemon[1946]: time=2026-09-21T14:45:30.469Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7043lb # [ 138.322821] niks3-server[1117]: 2026/09/21 14:45:29 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7044lb # [ 138.325522] niks3-server[1117]: 2026/09/21 14:45:29 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7045lb # [ 138.329526] niks3-server[1117]: 2026/09/21 14:45:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7046lb # [ 138.331298] niks3-server[1117]: 2026/09/21 14:45:29 INFO Signed narinfos id=65 count=17047worker1 # [ 138.331728] nix-grpc-daemon[1946]: time=2026-09-21T14:45:30.498Z level=INFO msg="Uploading 1 narinfos"7048lb # [ 138.335553] niks3-server[1117]: 2026/09/21 14:45:29 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7049lb # [ 138.341333] niks3-server[1117]: 2026/09/21 14:45:29 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7050lb # [ 138.343782] niks3-server[1117]: 2026/09/21 14:45:29 INFO Completed upload id=657051worker1 # [ 138.342868] nix-grpc-daemon[1946]: time=2026-09-21T14:45:30.510Z level=INFO msg="Upload complete. (146ms)"7052lb # [ 138.346657] niks3-server[1117]: 2026/09/21 14:45:29 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7053worker1 # [ 138.346154] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=0 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=17054worker1 # [ 138.349191] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:30Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17055worker1 # [ 138.352180] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542307056worker1 # [ 138.358293] nix-daemon[862]: reaped child process 2191, status = succeeded7057worker1 # [ 138.359615] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:542307058worker1 # [ 138.364825] nix-grpc-daemon[1946]: time=2026-09-21T14:45:30.532Z level=INFO msg="All 2 paths already cached"7059worker1 # [ 138.368459] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:542307060worker1 # [ 138.372978] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)7061worker1 # [ 138.455895] nix-daemon[862]: accepted connection from pid 2216, user nix-grpc-daemon (trusted)7062worker1 # [ 138.469910] nix-daemon[862]: reaped child process 2219, status = succeeded7063worker1 # [ 138.548941] nix-daemon[862]: accepted connection from pid 2221, user nix-grpc-daemon (trusted)7064worker1 # [ 138.562441] nix-daemon[862]: reaped child process 2224, status = succeeded7065lb # [ 138.568901] niks3-server[1117]: 2026/09/21 14:45:29 INFO Received uploads request method=POST path=/api/pending_closures7066worker1 # [ 138.576217] nix-grpc-daemon[1946]: time=2026-09-21T14:45:30.743Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7067worker1 # [ 138.577780] nix-grpc-daemon[1946]: time=2026-09-21T14:45:30.745Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7068lb # [ 138.600471] niks3-server[1117]: 2026/09/21 14:45:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7069lb # [ 138.608464] niks3-server[1117]: 2026/09/21 14:45:30 INFO Signed narinfos id=66 count=17070worker1 # [ 138.610805] nix-grpc-daemon[1946]: time=2026-09-21T14:45:30.778Z level=INFO msg="Uploading 1 narinfos"7071lb # [ 138.618563] niks3-server[1117]: 2026/09/21 14:45:30 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7072lb # [ 138.622996] niks3-server[1117]: 2026/09/21 14:45:30 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7073lb # [ 138.637967] niks3-server[1117]: 2026/09/21 14:45:30 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7074worker1 # [ 138.642556] nix-grpc-daemon[1946]: time=2026-09-21T14:45:30.809Z level=INFO msg="Upload complete. (265ms)"7075lb # [ 138.643843] niks3-server[1117]: 2026/09/21 14:45:30 INFO Completed upload id=667076lb # [ 138.647382] niks3-server[1117]: 2026/09/21 14:45:30 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7077worker1 # [ 138.646286] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=0 paths=1 nar_bytes_in=7527078worker1 # [ 138.651077] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542307079worker1 # [ 138.661134] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)7080worker1 # [ 138.667138] nix-daemon[862]: reaped child process 2214, status = succeeded7081worker1 # [ 138.670804] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)7082worker1 # [ 139.712466] nix-daemon[862]: reaped child process 2228, status = succeeded7083worker1 # [ 139.793749] nix-daemon[862]: accepted connection from pid 2244, user nix-grpc-daemon (trusted)7084worker1 # [ 139.808446] nix-daemon[862]: reaped child process 2247, status = succeeded7085lb # [ 139.815732] niks3-server[1117]: 2026/09/21 14:45:31 INFO Received uploads request method=POST path=/api/pending_closures7086worker1 # [ 139.824977] nix-grpc-daemon[1946]: time=2026-09-21T14:45:31.992Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7087worker1 # [ 139.826678] nix-grpc-daemon[1946]: time=2026-09-21T14:45:31.993Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7088lb # [ 139.858448] niks3-server[1117]: 2026/09/21 14:45:31 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7089lb # [ 139.866404] niks3-server[1117]: 2026/09/21 14:45:31 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7090lb # [ 139.873825] niks3-server[1117]: 2026/09/21 14:45:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7091worker1 # [ 139.879491] nix-grpc-daemon[1946]: time=2026-09-21T14:45:32.046Z level=INFO msg="Uploading 1 narinfos"7092lb # [ 139.880906] niks3-server[1117]: 2026/09/21 14:45:31 INFO Signed narinfos id=67 count=17093lb # [ 139.885878] niks3-server[1117]: 2026/09/21 14:45:31 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7094lb # [ 139.895748] niks3-server[1117]: 2026/09/21 14:45:31 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7095lb # [ 139.898430] niks3-server[1117]: 2026/09/21 14:45:31 INFO Completed upload id=677096worker1 # [ 139.897809] nix-grpc-daemon[1946]: time=2026-09-21T14:45:32.065Z level=INFO msg="Upload complete. (217ms)"7097lb # [ 139.901062] niks3-server[1117]: 2026/09/21 14:45:31 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7098worker1 # [ 139.906827] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=17099worker1 # [ 139.914262] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:32Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17100worker1 # [ 139.925185] nix-daemon[862]: reaped child process 2226, status = succeeded7101worker1 # [ 139.948613] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542307102worker1 # [ 139.959653] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:542307103worker1 # [ 139.972877] nix-grpc-daemon[1946]: time=2026-09-21T14:45:32.140Z level=INFO msg="All 2 paths already cached"7104worker1 # [ 139.980223] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:542307105worker1 # [ 139.988468] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)7106worker1 # [ 140.081949] nix-daemon[862]: accepted connection from pid 2251, user nix-grpc-daemon (trusted)7107worker1 # [ 140.094775] nix-daemon[862]: reaped child process 2254, status = succeeded7108worker1 # [ 140.177047] nix-daemon[862]: accepted connection from pid 2256, user nix-grpc-daemon (trusted)7109worker1 # [ 140.189478] nix-daemon[862]: reaped child process 2259, status = succeeded7110lb # [ 140.197805] niks3-server[1117]: 2026/09/21 14:45:31 INFO Received uploads request method=POST path=/api/pending_closures7111worker1 # [ 140.205078] nix-grpc-daemon[1946]: time=2026-09-21T14:45:32.372Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7112worker1 # [ 140.206486] nix-grpc-daemon[1946]: time=2026-09-21T14:45:32.372Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7113lb # [ 140.225123] niks3-server[1117]: 2026/09/21 14:45:31 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7114lb # [ 140.233767] niks3-server[1117]: 2026/09/21 14:45:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7115lb # [ 140.238295] niks3-server[1117]: 2026/09/21 14:45:31 INFO Signed narinfos id=68 count=17116worker1 # [ 140.240433] nix-grpc-daemon[1946]: time=2026-09-21T14:45:32.407Z level=INFO msg="Uploading 1 narinfos"7117lb # [ 140.248473] niks3-server[1117]: 2026/09/21 14:45:31 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7118lb # [ 140.258649] niks3-server[1117]: 2026/09/21 14:45:31 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7119lb # [ 140.264176] niks3-server[1117]: 2026/09/21 14:45:31 INFO Completed upload id=687120worker1 # [ 140.266120] nix-grpc-daemon[1946]: time=2026-09-21T14:45:32.433Z level=INFO msg="Upload complete. (266ms)"7121lb # [ 140.270182] niks3-server[1117]: 2026/09/21 14:45:31 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7122worker1 # [ 140.269227] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=0 paths=1 nar_bytes_in=7527123worker1 # [ 140.274265] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:542307124worker1 # [ 140.284065] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)7125worker1 # [ 140.289245] nix-daemon[862]: reaped child process 2249, status = succeeded7126worker1 # [ 140.293607] nix-daemon[862]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)7127worker1 # [ 141.538883] nix-daemon[862]: reaped child process 2263, status = succeeded7128worker1 # [ 141.609541] nix-daemon[862]: accepted connection from pid 2279, user nix-grpc-daemon (trusted)7129worker1 # [ 141.622431] nix-daemon[862]: reaped child process 2282, status = succeeded7130lb # [ 141.632559] niks3-server[1117]: 2026/09/21 14:45:33 INFO Received uploads request method=POST path=/api/pending_closures7131worker1 # [ 141.638278] nix-grpc-daemon[1946]: time=2026-09-21T14:45:33.805Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7132worker1 # [ 141.639792] nix-grpc-daemon[1946]: time=2026-09-21T14:45:33.807Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7133lb # [ 141.672667] niks3-server[1117]: 2026/09/21 14:45:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7134worker1 # [ 141.675903] nix-grpc-daemon[1946]: time=2026-09-21T14:45:33.842Z level=INFO msg="Uploading 1 narinfos"7135lb # [ 141.677348] niks3-server[1117]: 2026/09/21 14:45:33 INFO Signed narinfos id=69 count=17136lb # [ 141.686240] niks3-server[1117]: 2026/09/21 14:45:33 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7137lb # [ 141.692902] niks3-server[1117]: 2026/09/21 14:45:33 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7138lb # [ 141.703072] niks3-server[1117]: 2026/09/21 14:45:33 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7139lb # [ 141.705822] niks3-server[1117]: 2026/09/21 14:45:33 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7140lb # [ 141.711858] niks3-server[1117]: 2026/09/21 14:45:33 INFO Completed upload id=697141worker1 # [ 141.711259] nix-grpc-daemon[1946]: time=2026-09-21T14:45:33.878Z level=INFO msg="Upload complete. (209ms)"7142lb # [ 141.716868] niks3-server[1117]: 2026/09/21 14:45:33 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7143worker1 # [ 141.717334] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54230 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=17144worker1 # [ 141.731964] nix-daemon[862]: reaped child process 2261, status = succeeded7145client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 15.66 seconds)7146worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7147worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7148client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]'7149client # [ 141.846914] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7150client # Running as unit: gpu.service; invocation ID: 6889c06181434b6e94e105e001e26b797151client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.06 seconds)7152lb: must succeed: curl -sf localhost:9901/clusters7153lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7154lb: must succeed: curl -sf localhost:9901/clusters7155lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7156worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7157worker2 # [ 141.987593] nix-grpc-daemon[1335]: ts=2026-09-21T14:45:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:337887158client # [ 141.981430] nix[1371]: don't know how to build these paths:7159client # [ 141.982349] nix[1371]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7160worker1 # [ 141.992512] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:542307161worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7162client # [ 142.004248] nix[1371]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker7163worker1 # [ 142.013352] nix-grpc-daemon[1881]: ts=2026-09-21T14:45:34Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07164lb: must succeed: curl -sf localhost:9901/clusters7165lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)7166lb: must succeed: curl -sf localhost:9901/clusters7167lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7168worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7169worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7170client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7171lb # [ 146.604721] envoy[917]: conn peer=[2001:db8:1::1]:33594 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27085 tx=11349 ms=154887172client # [ 172.004996] nix[1371]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}7173client # [ 172.010876] nix[1371]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7174client # [ 172.028297] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7175client # [ 172.030344] systemd[1]: gpu.service: Failed with result 'exit-code'.7176client # [ 172.031771] systemd[1]: gpu.service: Consumed 141ms CPU time over 30.184s wall clock time, 35.6M memory peak, 5.1K incoming IP traffic, 5.1K outgoing IP traffic.7177client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.37 seconds)7178client: must succeed: journalctl -u gpu -o cat7179client: (finished: must succeed: journalctl -u gpu -o cat, in 0.01 seconds)7180lb: must succeed: curl -sf localhost:9901/clusters7181lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7182lb: must succeed: curl -sf localhost:9901/clusters7183lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7184worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7185worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7186(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 46.53 seconds)7187subtest: low disk drains a worker and builds go to the other7188worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7189worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.98 seconds)7190worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7191worker1 # [ 173.917068] nix-grpc-daemon[1881]: ts=2026-09-21T14:46:06Z level=info event=unhealthy reason=min_free7192worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.07 seconds)7193client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&27194worker1 # [ 174.832497] nix-grpc-daemon[1881]: ts=2026-09-21T14:46:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:542307195client # don't know how to build these paths:7196client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7197worker1 # [ 174.840554] nix-grpc-daemon[1881]: ts=2026-09-21T14:46:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:542307198worker1 # [ 174.858212] nix-grpc-daemon[1881]: ts=2026-09-21T14:46:07Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07199worker1 # [ 174.862598] nix-grpc-daemon[1881]: ts=2026-09-21T14:46:07Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17200worker2 # [ 174.905257] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337887201worker2 # [ 174.917380] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337887202client # copying 1 paths to 'lb:50051'...7203client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7204worker2 # [ 174.927246] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:337887205worker2 # [ 174.938200] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)7206worker2 # [ 175.039606] nix-daemon[850]: accepted connection from pid 1604, user nix-grpc-daemon (trusted)7207worker2 # [ 175.052447] nix-daemon[850]: reaped child process 1607, status = succeeded7208worker2 # [ 175.134994] nix-daemon[850]: accepted connection from pid 1609, user nix-grpc-daemon (trusted)7209worker2 # [ 175.152289] nix-daemon[850]: reaped child process 1612, status = succeeded7210lb # [ 175.160815] niks3-server[1117]: 2026/09/21 14:46:06 INFO Received uploads request method=POST path=/api/pending_closures7211worker2 # [ 175.299471] nix-grpc-daemon[1427]: time=2026-09-21T14:46:07.443Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7212worker2 # [ 175.303196] nix-grpc-daemon[1427]: time=2026-09-21T14:46:07.443Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7213lb # [ 175.331291] niks3-server[1117]: 2026/09/21 14:46:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7214lb # [ 175.339356] niks3-server[1117]: 2026/09/21 14:46:06 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7215lb # [ 175.345570] niks3-server[1117]: 2026/09/21 14:46:06 INFO Signed narinfos id=70 count=17216worker2 # [ 175.342575] nix-grpc-daemon[1427]: time=2026-09-21T14:46:07.487Z level=INFO msg="Uploading 1 narinfos"7217lb # [ 175.361041] niks3-server[1117]: 2026/09/21 14:46:06 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7218lb # [ 175.371901] niks3-server[1117]: 2026/09/21 14:46:06 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7219lb # [ 175.378803] niks3-server[1117]: 2026/09/21 14:46:06 INFO Completed upload id=707220worker2 # [ 175.375380] nix-grpc-daemon[1427]: time=2026-09-21T14:46:07.520Z level=INFO msg="Upload complete. (425ms)"7221lb # [ 175.381818] niks3-server[1117]: 2026/09/21 14:46:06 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7222worker2 # [ 175.378995] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=0 paths=1 nar_bytes_in=5127223worker2 # [ 175.383049] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337887224worker2 # [ 175.389707] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)7225worker2 # [ 175.391994] nix-daemon[850]: reaped child process 1602, status = succeeded7226client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7227worker2 # [ 175.397228] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)7228client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7229lb # [ 176.606557] envoy[917]: conn peer=[2001:db8:1::1]:59846 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300397230client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7231client # farm-a> LOG-farm-a-drain7232worker2 # [ 176.812958] nix-daemon[850]: reaped child process 1616, status = succeeded7233worker2 # [ 176.889680] nix-daemon[850]: accepted connection from pid 1632, user nix-grpc-daemon (trusted)7234worker2 # [ 176.903338] nix-daemon[850]: reaped child process 1635, status = succeeded7235lb # [ 176.913805] niks3-server[1117]: 2026/09/21 14:46:08 INFO Received uploads request method=POST path=/api/pending_closures7236worker2 # [ 176.919357] nix-grpc-daemon[1427]: time=2026-09-21T14:46:09.064Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7237worker2 # [ 176.921350] nix-grpc-daemon[1427]: time=2026-09-21T14:46:09.066Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7238lb # [ 176.947458] niks3-server[1117]: 2026/09/21 14:46:08 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7239lb # [ 176.952573] niks3-server[1117]: 2026/09/21 14:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7240lb # [ 176.955530] niks3-server[1117]: 2026/09/21 14:46:08 INFO Signed narinfos id=71 count=17241worker2 # [ 176.954052] nix-grpc-daemon[1427]: time=2026-09-21T14:46:09.099Z level=INFO msg="Uploading 1 narinfos"7242lb # [ 176.958959] niks3-server[1117]: 2026/09/21 14:46:08 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7243lb # [ 176.961791] niks3-server[1117]: 2026/09/21 14:46:08 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7244lb # [ 176.967080] niks3-server[1117]: 2026/09/21 14:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7245lb # [ 176.970477] niks3-server[1117]: 2026/09/21 14:46:08 INFO Completed upload id=717246worker2 # [ 176.968218] nix-grpc-daemon[1427]: time=2026-09-21T14:46:09.113Z level=INFO msg="Upload complete. (163ms)"7247lb # [ 176.972192] niks3-server[1117]: 2026/09/21 14:46:08 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7248worker2 # [ 176.970875] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=17249worker2 # [ 176.978049] nix-daemon[850]: reaped child process 1614, status = succeeded7250worker1 # [ 176.980325] nix-grpc-daemon[1881]: ts=2026-09-21T14:46:09Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07251worker1 # [ 176.984831] nix-grpc-daemon[1881]: ts=2026-09-21T14:46:09Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17252worker2 # [ 177.020709] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337887253worker2 # [ 177.032249] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337887254worker2 # [ 177.044468] nix-grpc-daemon[1427]: time=2026-09-21T14:46:09.189Z level=INFO msg="All 1 paths already cached"7255client # copying 1 paths to 'lb:50051'...7256client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7257worker2 # [ 177.053703] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:337887258worker2 # [ 177.061971] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)7259worker2 # [ 177.162345] nix-daemon[850]: accepted connection from pid 1639, user nix-grpc-daemon (trusted)7260worker2 # [ 177.175362] nix-daemon[850]: reaped child process 1642, status = succeeded7261worker2 # [ 177.259036] nix-daemon[850]: accepted connection from pid 1644, user nix-grpc-daemon (trusted)7262worker2 # [ 177.271214] nix-daemon[850]: reaped child process 1647, status = succeeded7263lb # [ 177.282680] niks3-server[1117]: 2026/09/21 14:46:08 INFO Received uploads request method=POST path=/api/pending_closures7264worker2 # [ 177.287737] nix-grpc-daemon[1427]: time=2026-09-21T14:46:09.433Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7265worker2 # [ 177.289156] nix-grpc-daemon[1427]: time=2026-09-21T14:46:09.433Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7266lb # [ 177.308111] niks3-server[1117]: 2026/09/21 14:46:08 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7267lb # [ 177.323452] niks3-server[1117]: 2026/09/21 14:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7268worker2 # [ 177.326337] nix-grpc-daemon[1427]: time=2026-09-21T14:46:09.471Z level=INFO msg="Uploading 1 narinfos"7269lb # [ 177.329678] niks3-server[1117]: 2026/09/21 14:46:08 INFO Signed narinfos id=72 count=17270lb # [ 177.337911] niks3-server[1117]: 2026/09/21 14:46:08 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7271lb # [ 177.347854] niks3-server[1117]: 2026/09/21 14:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7272lb # [ 177.354877] niks3-server[1117]: 2026/09/21 14:46:08 INFO Completed upload id=727273worker2 # [ 177.351590] nix-grpc-daemon[1427]: time=2026-09-21T14:46:09.497Z level=INFO msg="Upload complete. (275ms)"7274lb # [ 177.356232] niks3-server[1117]: 2026/09/21 14:46:08 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7275worker2 # [ 177.353869] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=0 paths=1 nar_bytes_in=6247276worker2 # [ 177.359285] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337887277worker2 # [ 177.368496] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)7278worker2 # [ 177.375118] nix-daemon[850]: reaped child process 1637, status = succeeded7279worker2 # [ 177.378344] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)7280client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7281client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7282client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7283client # farm-b> LOG-farm-b-drain7284worker2 # [ 177.993903] nix-daemon[850]: reaped child process 1651, status = succeeded7285worker2 # [ 178.069784] nix-daemon[850]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7286worker2 # [ 178.081405] nix-daemon[850]: reaped child process 1670, status = succeeded7287lb # [ 178.093243] niks3-server[1117]: 2026/09/21 14:46:09 INFO Received uploads request method=POST path=/api/pending_closures7288worker2 # [ 178.104266] nix-grpc-daemon[1427]: time=2026-09-21T14:46:10.249Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7289worker2 # [ 178.106048] nix-grpc-daemon[1427]: time=2026-09-21T14:46:10.251Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7290lb # [ 178.141953] niks3-server[1117]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7291lb # [ 178.148526] niks3-server[1117]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7292lb # [ 178.160898] niks3-server[1117]: 2026/09/21 14:46:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7293lb # [ 178.166312] niks3-server[1117]: 2026/09/21 14:46:09 INFO Signed narinfos id=73 count=17294worker2 # [ 178.162695] nix-grpc-daemon[1427]: time=2026-09-21T14:46:10.307Z level=INFO msg="Uploading 1 narinfos"7295lb # [ 178.172253] niks3-server[1117]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7296lb # [ 178.181802] niks3-server[1117]: 2026/09/21 14:46:09 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7297lb # [ 178.187026] niks3-server[1117]: 2026/09/21 14:46:09 INFO Completed upload id=737298lb # [ 178.188741] niks3-server[1117]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7299worker2 # [ 178.185142] nix-grpc-daemon[1427]: time=2026-09-21T14:46:10.330Z level=INFO msg="Upload complete. (200ms)"7300worker2 # [ 178.190980] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=0 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=17301worker1 # [ 178.200502] nix-grpc-daemon[1881]: ts=2026-09-21T14:46:10Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07302worker2 # [ 178.200252] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337887303worker1 # [ 178.205117] nix-grpc-daemon[1881]: ts=2026-09-21T14:46:10Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17304worker2 # [ 178.212544] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:337887305worker2 # [ 178.218544] nix-daemon[850]: reaped child process 1649, status = succeeded7306worker2 # [ 178.224692] nix-grpc-daemon[1427]: time=2026-09-21T14:46:10.370Z level=INFO msg="All 2 paths already cached"7307client # copying 1 paths to 'lb:50051'...7308client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7309worker2 # [ 178.232055] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:337887310worker2 # [ 178.238749] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)7311worker2 # [ 178.325097] nix-daemon[850]: accepted connection from pid 1674, user nix-grpc-daemon (trusted)7312worker2 # [ 178.339534] nix-daemon[850]: reaped child process 1677, status = succeeded7313worker2 # [ 178.423113] nix-daemon[850]: accepted connection from pid 1679, user nix-grpc-daemon (trusted)7314worker2 # [ 178.435521] nix-daemon[850]: reaped child process 1682, status = succeeded7315lb # [ 178.447612] niks3-server[1117]: 2026/09/21 14:46:09 INFO Received uploads request method=POST path=/api/pending_closures7316worker2 # [ 178.457465] nix-grpc-daemon[1427]: time=2026-09-21T14:46:10.603Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7317worker2 # [ 178.459036] nix-grpc-daemon[1427]: time=2026-09-21T14:46:10.604Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7318lb # [ 178.485525] niks3-server[1117]: 2026/09/21 14:46:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7319worker2 # [ 178.488301] nix-grpc-daemon[1427]: time=2026-09-21T14:46:10.633Z level=INFO msg="Uploading 1 narinfos"7320lb # [ 178.491853] niks3-server[1117]: 2026/09/21 14:46:09 INFO Signed narinfos id=74 count=17321lb # [ 178.496797] niks3-server[1117]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7322lb # [ 178.506360] niks3-server[1117]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7323lb # [ 178.518054] niks3-server[1117]: 2026/09/21 14:46:09 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7324lb # [ 178.524382] niks3-server[1117]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7325lb # [ 178.529398] niks3-server[1117]: 2026/09/21 14:46:09 INFO Completed upload id=747326worker2 # [ 178.527263] nix-grpc-daemon[1427]: time=2026-09-21T14:46:10.671Z level=INFO msg="Upload complete. (280ms)"7327worker2 # [ 178.529830] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=0 paths=1 nar_bytes_in=7527328worker2 # [ 178.537239] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:337887329worker2 # [ 178.545480] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)7330worker2 # [ 178.558563] nix-daemon[850]: reaped child process 1672, status = succeeded7331client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7332worker2 # [ 178.561458] nix-daemon[850]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)7333client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7334client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7335client # farm-top> LOG-farm-top-drain7336worker2 # [ 179.599898] nix-daemon[850]: reaped child process 1686, status = succeeded7337worker2 # [ 179.679364] nix-daemon[850]: accepted connection from pid 1702, user nix-grpc-daemon (trusted)7338worker2 # [ 179.690695] nix-daemon[850]: reaped child process 1705, status = succeeded7339lb # [ 179.703246] niks3-server[1117]: 2026/09/21 14:46:11 INFO Received uploads request method=POST path=/api/pending_closures7340worker2 # [ 179.708971] nix-grpc-daemon[1427]: time=2026-09-21T14:46:11.854Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7341worker2 # [ 179.710618] nix-grpc-daemon[1427]: time=2026-09-21T14:46:11.856Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7342lb # [ 179.743894] niks3-server[1117]: 2026/09/21 14:46:11 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7343lb # [ 179.753676] niks3-server[1117]: 2026/09/21 14:46:11 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7344lb # [ 179.761363] niks3-server[1117]: 2026/09/21 14:46:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7345worker2 # [ 179.765465] nix-grpc-daemon[1427]: time=2026-09-21T14:46:11.910Z level=INFO msg="Uploading 1 narinfos"7346lb # [ 179.769169] niks3-server[1117]: 2026/09/21 14:46:11 INFO Signed narinfos id=75 count=17347lb # [ 179.776095] niks3-server[1117]: 2026/09/21 14:46:11 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7348lb # [ 179.784596] niks3-server[1117]: 2026/09/21 14:46:11 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7349lb # [ 179.787799] niks3-server[1117]: 2026/09/21 14:46:11 INFO Completed upload id=757350lb # [ 179.790601] niks3-server[1117]: 2026/09/21 14:46:11 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7351worker2 # [ 179.785700] nix-grpc-daemon[1427]: time=2026-09-21T14:46:11.931Z level=INFO msg="Upload complete. (217ms)"7352worker2 # [ 179.792400] nix-grpc-daemon[1335]: ts=2026-09-21T14:46:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33788 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=17353worker2 # [ 179.805323] nix-daemon[850]: reaped child process 1684, status = succeeded7354client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&2, in 5.17 seconds)7355client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath7356client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath, in 0.11 seconds)7357worker1: must succeed: rm /nix/.rw-store/fill7358worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.09 seconds)7359worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7360worker1 # [ 180.920132] nix-grpc-daemon[1881]: ts=2026-09-21T14:46:13Z level=info event=healthy reason=min_free7361worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.06 seconds)7362(finished: subtest: low disk drains a worker and builds go to the other, in 8.57 seconds)7363(finished: run the VM test script, in 182.10 seconds)7364test script finished in 182.40s7365cleanup7366kill QemuMachine (pid 49)7367client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7368client # [2026-09-21T14:46:13Z INFO virtiofsd] Client disconnected, shutting down7369client # [2026-09-21T14:46:13Z INFO virtiofsd] Client disconnected, shutting down7370client # [2026-09-21T14:46:13Z INFO virtiofsd] Client disconnected, shutting down7371kill QemuMachine (pid 51)7372lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7373lb # [2026-09-21T14:46:13Z INFO virtiofsd] Client disconnected, shutting down7374lb # [2026-09-21T14:46:13Z INFO virtiofsd] Client disconnected, shutting down7375lb # [2026-09-21T14:46:13Z INFO virtiofsd] Client disconnected, shutting down7376kill QemuMachine (pid 50)7377worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7378worker1 # [2026-09-21T14:46:14Z INFO virtiofsd] Client disconnected, shutting down7379worker1 # [2026-09-21T14:46:14Z INFO virtiofsd] Client disconnected, shutting down7380worker1 # [2026-09-21T14:46:14Z INFO virtiofsd] Client disconnected, shutting down7381kill QemuMachine (pid 52)7382worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7383worker2 # [2026-09-21T14:46:14Z INFO virtiofsd] Client disconnected, shutting down7384worker2 # [2026-09-21T14:46:14Z INFO virtiofsd] Client disconnected, shutting down7385worker2 # [2026-09-21T14:46:14Z INFO virtiofsd] Client disconnected, shutting down7386(finished: cleanup, in 1.06 seconds)7387additionally exposed symbols:7388 client, lb, worker1, worker2,7389 vlan1,7390 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh