vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #535
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.01 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12client: QEMU running (pid 49)13worker2: QEMU running (pid 52)14worker2 # Disk image does not exist, creating the virtualisation disk image...15lb # Disk image does not exist, creating the virtualisation disk image...16worker2 # Formatting '/build/vm-state-worker2/tmp.cMiNlg0aK1', fmt=raw size=107374182417client # Disk image does not exist, creating the virtualisation disk image...18worker1: QEMU running (pid 51)19lb: QEMU running (pid 50)20worker1 # Disk image does not exist, creating the virtualisation disk image...21lb # Formatting '/build/vm-state-lb/tmp.ayn2XUb5BH', fmt=raw size=107374182422client # Formatting '/build/vm-state-client/tmp.bdTzhpTQ9m', fmt=raw size=107374182423worker2 # mke2fs 1.47.4 (6-Mar-2025)24(finished: start all VMs, in 0.21 seconds)25client # mke2fs 1.47.4 (6-Mar-2025)26worker2 # Discarding device blocks: 0/262144 done27client: waiting for unit network-addresses-eth1.service28client # Discarding device blocks: 0/262144 done29worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes30client: waiting for the VM to finish booting31client # Creating filesystem with 262144 4k blocks and 65536 inodes32worker2 # Filesystem UUID: 0df8a1e4-2b77-4449-8a89-10da3c9f98ba33lb # mke2fs 1.47.4 (6-Mar-2025)34worker1 # Formatting '/build/vm-state-worker1/tmp.5zp5DwjaEV', fmt=raw size=107374182435client # Filesystem UUID: b0304305-2254-42bb-91b1-71de4a72a3ed36worker2 # Superblock backups stored on blocks:37lb # Discarding device blocks: 0/262144 done38worker1 # mke2fs 1.47.4 (6-Mar-2025)39client # Superblock backups stored on blocks:40worker2 # 32768, 98304, 163840, 22937641lb # Creating filesystem with 262144 4k blocks and 65536 inodes42worker1 # Discarding device blocks: 0/262144 done43client # 32768, 98304, 163840, 22937644worker2 # 45lb # Filesystem UUID: f865ffbd-45df-49ba-a98a-fa2b73b6401c46worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes47client # 48worker2 # Allocating group tables: 0/8 done49lb # Superblock backups stored on blocks:50worker1 # Filesystem UUID: b6ead55f-799f-489d-8de3-4e78c873c87851client # Allocating group tables: 0/8 done52worker2 # Writing inode tables: 0/8 done53lb # 32768, 98304, 163840, 22937654worker1 # Superblock backups stored on blocks:55client # Writing inode tables: 0/8 done56worker2 # Creating journal (8192 blocks): done57lb # 58worker1 # 32768, 98304, 163840, 22937659client # Creating journal (8192 blocks): done60worker2 # Writing superblocks and filesystem accounting information: 0/8 done61lb # Allocating group tables: 0/8 done62worker1 # 63client # Writing superblocks and filesystem accounting information: 0/8 done64worker2 # 65lb # Writing inode tables: 0/8 done66worker1 # Allocating group tables: 0/8 done67client # 68worker2 # Virtualisation disk image created.69lb # Creating journal (8192 blocks): done70worker1 # Writing inode tables: 0/8 done71client # Virtualisation disk image created.72worker2 # Starting virtiofs daemons...73lb # Writing superblocks and filesystem accounting information: 0/8 done74worker1 # Creating journal (8192 blocks): done75client # Starting virtiofs daemons...76worker2 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)77lb # 78worker1 # Writing superblocks and filesystem accounting information: 0/8 done79client # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)80worker2 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether81lb # Virtualisation disk image created.82worker1 # 83client # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether84worker2 # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...85lb # Starting virtiofs daemons...86worker1 # Virtualisation disk image created.87client # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...88worker2 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)89lb # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)90worker1 # Starting virtiofs daemons...91client # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)92worker2 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether93lb # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether94worker1 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)95client # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether96worker2 # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...97lb # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...98worker1 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether99client # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...100worker2 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)101lb # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)102worker1 # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...103client # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)104worker2 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether105lb # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether106worker1 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)107client # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether108worker2 # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...109lb # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...110worker1 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether111client # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...112worker2 # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests113lb # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)114worker1 # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...115client # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests116worker2 # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests117lb # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether118worker1 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)119client # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests120worker2 # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests121lb # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...122worker1 # [2026-09-23T06:58:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether123lb # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests124worker1 # [2026-09-23T06:58:58Z INFO virtiofsd] Waiting for vhost-user socket connection...125lb # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests126worker1 # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests127lb # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests128worker1 # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests129client # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests130worker1 # [2026-09-23T06:58:58Z INFO virtiofsd] Client connected, servicing requests131client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)132worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)133worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)134lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)135client # 136client # 137client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00138client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...139client # 140client # 141client # 142client # 143client # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00144worker2 # 145worker2 # 146worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00147worker1 # 148worker1 # 149client # Press Ctrl-B to configure iPXE (PCI 00:05.0)...150worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00151client # 152client # 153worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...154worker2 # 155worker2 # 156worker2 # 157worker2 # 158lb # 159lb # 160worker2 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00161worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...162worker1 # 163worker1 # 164worker1 # 165worker1 # 166lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00167worker1 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00168lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...169lb # 170lb # 171lb # 172lb # 173worker2 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...174worker2 # 175worker2 # 176lb # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00177worker1 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...178worker1 # 179worker1 # 180lb # Press Ctrl-B to configure iPXE (PCI 00:05.0)...181lb # 182lb # 183client # Booting from ROM...184worker2 # Booting from ROM...185worker1 # Booting from ROM...186lb # Booting from ROM...187client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026188client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/8icb5zsbqv3dxvnzhf8ddq1cyhg5f86j-nixos-system-client-test/init regInfo=/nix/store/99yzymliyji70vhz8x5j5c4ckg24296s-closure-info/registration console=ttyS0,115200n8 console=tty0189client # [ 0.000000] BIOS-provided physical RAM map:190client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable191client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved192client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved193client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable194client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved195client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved196client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved197client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved198client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved199lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026200client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved201worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026202client # [ 0.000000] NX (Execute Disable) protection: active203client # [ 0.000000] APIC: Static calls initialized204client # [ 0.000000] SMBIOS 2.8 present.205worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026206client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014207client # [ 0.000000] DMI: Memory slots populated: 1/1208lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/5f2pyghh7bdk9s7schxljxgh1845dmij-nixos-system-lb-test/init regInfo=/nix/store/b17iww818kslhkx3acpxq2j13rz8h7sy-closure-info/registration console=ttyS0,115200n8 console=tty0209client # [ 0.000000] Hypervisor detected: KVM210worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/7ivsh1xpnbmbn73qmia5fjss23hffswz-nixos-system-worker1-test/init regInfo=/nix/store/0m6yj9v4lz5rh7gnbwj9ql3hx52rgzbd-closure-info/registration console=ttyS0,115200n8 console=tty0211lb # [ 0.000000] BIOS-provided physical RAM map:212worker1 # [ 0.000000] BIOS-provided physical RAM map:213client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000214lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable215client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00216worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable217lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved218client # [ 0.000000] kvm-clock: using sched offset of 486801697 cycles219worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/xis419qg6ssac1mxl23xpjki6jlxkhwd-nixos-system-worker2-test/init regInfo=/nix/store/5fllks9l0kij5jdjhq2g604s52nb8f4c-closure-info/registration console=ttyS0,115200n8 console=tty0220worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved221worker2 # [ 0.000000] BIOS-provided physical RAM map:222lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved223worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved224client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns225worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable226lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable227client # [ 0.000005] tsc: Detected 2400.012 MHz processor228worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable229worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved230lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved231client # [ 0.000811] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000232worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved233worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved234lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved235client # [ 0.000839] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs236worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved237worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable238lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved239client # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT240worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved241worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved242lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved243client # [ 0.002731] found SMP MP-table at [mem 0x000f5450-0x000f545f]244worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved245worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved246client # [ 0.002742] Using GB pages for direct mapping247lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved248client # [ 0.002884] RAMDISK: [mem 0x5e370000-0x5ffcffff]249worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved250worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved251lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved252client # [ 0.002893] ACPI: Early table checksum verification disabled253worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved254lb # [ 0.000000] NX (Execute Disable) protection: active255worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved256client # [ 0.002896] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )257lb # [ 0.000000] APIC: Static calls initialized258worker1 # [ 0.000000] NX (Execute Disable) protection: active259lb # [ 0.000000] SMBIOS 2.8 present.260worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved261worker1 # [ 0.000000] APIC: Static calls initialized262worker1 # [ 0.000000] SMBIOS 2.8 present.263client # [ 0.002900] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)264worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved265lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014266worker2 # [ 0.000000] NX (Execute Disable) protection: active267client # [ 0.002903] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)268worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014269lb # [ 0.000000] DMI: Memory slots populated: 1/1270worker2 # [ 0.000000] APIC: Static calls initialized271lb # [ 0.000000] Hypervisor detected: KVM272worker1 # [ 0.000000] DMI: Memory slots populated: 1/1273worker2 # [ 0.000000] SMBIOS 2.8 present.274worker1 # [ 0.000000] Hypervisor detected: KVM275client # [ 0.002910] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)276lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000277client # [ 0.002912] ACPI: FACS 0x000000005FFE0000 000040278worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000279lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00280worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014281worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00282worker2 # [ 0.000000] DMI: Memory slots populated: 1/1283client # [ 0.002914] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)284lb # [ 0.000001] kvm-clock: using sched offset of 494704706 cycles285worker2 # [ 0.000000] Hypervisor detected: KVM286worker1 # [ 0.000000] kvm-clock: using sched offset of 487491623 cycles287worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000288client # [ 0.002916] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)289lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns290worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00291worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns292lb # [ 0.000005] tsc: Detected 2400.012 MHz processor293client # [ 0.002917] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)294worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor295worker2 # [ 0.000000] kvm-clock: using sched offset of 490031191 cycles296lb # [ 0.000810] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000297worker1 # [ 0.000805] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000298client # [ 0.002919] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)299worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns300lb # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs301client # [ 0.002920] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]302worker1 # [ 0.000832] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs303worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor304lb # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT305worker2 # [ 0.000809] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000306client # [ 0.002921] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]307worker1 # [ 0.000834] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT308lb # [ 0.002733] found SMP MP-table at [mem 0x000f5450-0x000f545f]309worker1 # [ 0.002719] found SMP MP-table at [mem 0x000f5450-0x000f545f]310lb # [ 0.002744] Using GB pages for direct mapping311client # [ 0.002922] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]312worker2 # [ 0.000836] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs313worker1 # [ 0.002731] Using GB pages for direct mapping314lb # [ 0.002862] RAMDISK: [mem 0x5e36e000-0x5ffcffff]315client # [ 0.002922] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]316worker2 # [ 0.000839] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT317worker1 # [ 0.002877] RAMDISK: [mem 0x5e370000-0x5ffcffff]318lb # [ 0.002870] ACPI: Early table checksum verification disabled319worker2 # [ 0.002724] found SMP MP-table at [mem 0x000f5450-0x000f545f]320client # [ 0.002923] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]321worker1 # [ 0.002892] ACPI: Early table checksum verification disabled322lb # [ 0.002873] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )323worker2 # [ 0.002735] Using GB pages for direct mapping324worker1 # [ 0.002895] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )325client # [ 0.002923] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]326worker2 # [ 0.002851] RAMDISK: [mem 0x5e36f000-0x5ffcffff]327lb # [ 0.002877] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)328worker1 # [ 0.002899] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)329client # [ 0.002923] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]330worker2 # [ 0.002858] ACPI: Early table checksum verification disabled331lb # [ 0.002881] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)332client # [ 0.003149] No NUMA configuration found333worker2 # [ 0.002861] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )334worker1 # [ 0.002903] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)335client # [ 0.003150] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]336lb # [ 0.002888] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)337worker2 # [ 0.002865] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)338worker1 # [ 0.002910] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)339client # [ 0.003152] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]340lb # [ 0.002891] ACPI: FACS 0x000000005FFE0000 000040341client # [ 0.005406] Zone ranges:342worker1 # [ 0.002912] ACPI: FACS 0x000000005FFE0000 000040343worker2 # [ 0.002869] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)344client # [ 0.005407] DMA [mem 0x0000000000001000-0x0000000000ffffff]345lb # [ 0.002892] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)346worker1 # [ 0.002914] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)347worker2 # [ 0.002875] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)348client # [ 0.005409] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]349client # [ 0.005410] Normal empty350lb # [ 0.002894] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)351worker2 # [ 0.002877] ACPI: FACS 0x000000005FFE0000 000040352client # [ 0.005411] Device empty353worker1 # [ 0.002915] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)354lb # [ 0.002895] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)355client # [ 0.005412] Movable zone start for each node356worker2 # [ 0.002879] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)357client # [ 0.005413] Early memory node ranges358worker1 # [ 0.002917] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)359lb # [ 0.002897] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)360client # [ 0.005413] node 0: [mem 0x0000000000001000-0x000000000009efff]361worker2 # [ 0.002881] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)362worker1 # [ 0.002918] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)363client # [ 0.005414] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]364lb # [ 0.002898] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]365worker2 # [ 0.002882] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)366worker1 # [ 0.002920] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]367lb # [ 0.002899] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]368client # [ 0.005415] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]369worker1 # [ 0.002921] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]370worker2 # [ 0.002884] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)371client # [ 0.005434] On node 0, zone DMA: 1 pages in unavailable ranges372lb # [ 0.002900] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]373worker1 # [ 0.002921] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]374worker2 # [ 0.002885] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]375client # [ 0.005701] On node 0, zone DMA: 97 pages in unavailable ranges376lb # [ 0.002900] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]377client # [ 0.031973] On node 0, zone DMA32: 40 pages in unavailable ranges378worker1 # [ 0.002922] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]379worker2 # [ 0.002886] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]380lb # [ 0.002900] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]381client # [ 0.032429] ACPI: PM-Timer IO Port: 0x608382worker1 # [ 0.002922] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]383worker2 # [ 0.002887] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]384client # [ 0.032441] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])385lb # [ 0.002901] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]386worker1 # [ 0.002923] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]387worker2 # [ 0.002887] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]388client # [ 0.032470] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23389lb # [ 0.002901] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]390lb # [ 0.003126] No NUMA configuration found391worker1 # [ 0.002923] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]392worker2 # [ 0.002888] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]393client # [ 0.032473] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)394worker1 # [ 0.003150] No NUMA configuration found395worker2 # [ 0.002888] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]396lb # [ 0.003127] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]397client # [ 0.032474] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)398worker1 # [ 0.003151] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]399lb # [ 0.003129] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]400worker2 # [ 0.002888] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]401client # [ 0.032476] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)402worker1 # [ 0.003153] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]403lb # [ 0.005386] Zone ranges:404worker2 # [ 0.003112] No NUMA configuration found405worker1 # [ 0.005413] Zone ranges:406lb # [ 0.005387] DMA [mem 0x0000000000001000-0x0000000000ffffff]407client # [ 0.032477] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)408worker2 # [ 0.003113] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]409worker1 # [ 0.005414] DMA [mem 0x0000000000001000-0x0000000000ffffff]410lb # [ 0.005388] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]411client # [ 0.032477] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)412lb # [ 0.005389] Normal empty413worker1 # [ 0.005415] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]414worker2 # [ 0.003115] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]415worker2 # [ 0.005403] Zone ranges:416lb # [ 0.005390] Device empty417worker1 # [ 0.005416] Normal empty418client # [ 0.032480] ACPI: Using ACPI (MADT) for SMP configuration information419worker1 # [ 0.005417] Device empty420lb # [ 0.005391] Movable zone start for each node421worker2 # [ 0.005403] DMA [mem 0x0000000000001000-0x0000000000ffffff]422client # [ 0.032481] ACPI: HPET id: 0x8086a201 base: 0xfed00000423worker1 # [ 0.005417] Movable zone start for each node424lb # [ 0.005392] Early memory node ranges425client # [ 0.032485] TSC deadline timer available426worker1 # [ 0.005418] Early memory node ranges427worker2 # [ 0.005405] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]428lb # [ 0.005392] node 0: [mem 0x0000000000001000-0x000000000009efff]429client # [ 0.032489] CPU topo: Max. logical packages: 1430worker2 # [ 0.005406] Normal empty431worker1 # [ 0.005418] node 0: [mem 0x0000000000001000-0x000000000009efff]432worker2 # [ 0.005407] Device empty433lb # [ 0.005393] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]434client # [ 0.032489] CPU topo: Max. logical dies: 1435worker2 # [ 0.005407] Movable zone start for each node436worker1 # [ 0.005420] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]437client # [ 0.032490] CPU topo: Max. dies per package: 1438worker2 # [ 0.005408] Early memory node ranges439lb # [ 0.005394] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]440client # [ 0.032493] CPU topo: Max. threads per core: 1441worker1 # [ 0.005421] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]442worker2 # [ 0.005408] node 0: [mem 0x0000000000001000-0x000000000009efff]443lb # [ 0.005416] On node 0, zone DMA: 1 pages in unavailable ranges444client # [ 0.032493] CPU topo: Num. cores per package: 1445worker1 # [ 0.005440] On node 0, zone DMA: 1 pages in unavailable ranges446worker2 # [ 0.005409] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]447client # [ 0.032494] CPU topo: Num. threads per package: 1448lb # [ 0.005676] On node 0, zone DMA: 97 pages in unavailable ranges449worker1 # [ 0.005700] On node 0, zone DMA: 97 pages in unavailable ranges450worker2 # [ 0.005410] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]451lb # [ 0.032031] On node 0, zone DMA32: 40 pages in unavailable ranges452client # [ 0.032494] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs453worker1 # [ 0.031916] On node 0, zone DMA32: 40 pages in unavailable ranges454lb # [ 0.032496] ACPI: PM-Timer IO Port: 0x608455worker2 # [ 0.005429] On node 0, zone DMA: 1 pages in unavailable ranges456client # [ 0.032511] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()457worker1 # [ 0.032376] ACPI: PM-Timer IO Port: 0x608458lb # [ 0.032508] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])459worker2 # [ 0.005692] On node 0, zone DMA: 97 pages in unavailable ranges460worker1 # [ 0.032388] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])461client # [ 0.032541] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]462lb # [ 0.032538] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23463worker2 # [ 0.032846] On node 0, zone DMA32: 40 pages in unavailable ranges464worker1 # [ 0.032418] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23465worker2 # [ 0.033310] ACPI: PM-Timer IO Port: 0x608466client # [ 0.032542] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]467lb # [ 0.032540] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)468worker1 # [ 0.032421] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)469worker2 # [ 0.033322] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])470client # [ 0.032544] [mem 0x60000000-0xafffffff] available for PCI devices471lb # [ 0.032542] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)472worker1 # [ 0.032422] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)473client # [ 0.032545] Booting paravirtualized kernel on KVM474worker2 # [ 0.033351] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23475lb # [ 0.032543] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)476worker1 # [ 0.032423] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)477worker2 # [ 0.033353] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)478lb # [ 0.032544] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)479client # [ 0.032548] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns480worker1 # [ 0.032424] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)481worker2 # [ 0.033355] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)482lb # [ 0.032545] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)483client # [ 0.037048] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1484worker1 # [ 0.032425] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)485worker2 # [ 0.033356] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)486client # [ 0.039184] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152487lb # [ 0.032547] ACPI: Using ACPI (MADT) for SMP configuration information488worker1 # [ 0.032428] ACPI: Using ACPI (MADT) for SMP configuration information489worker2 # [ 0.033357] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)490lb # [ 0.032548] ACPI: HPET id: 0x8086a201 base: 0xfed00000491client # [ 0.039229] kvm-guest: PV spinlocks disabled, single CPU492worker1 # [ 0.032429] ACPI: HPET id: 0x8086a201 base: 0xfed00000493lb # [ 0.032551] TSC deadline timer available494worker2 # [ 0.033358] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)495worker1 # [ 0.032432] TSC deadline timer available496lb # [ 0.032555] CPU topo: Max. logical packages: 1497worker2 # [ 0.033360] ACPI: Using ACPI (MADT) for SMP configuration information498worker1 # [ 0.032436] CPU topo: Max. logical packages: 1499lb # [ 0.032555] CPU topo: Max. logical dies: 1500worker1 # [ 0.032437] CPU topo: Max. logical dies: 1501worker2 # [ 0.033361] ACPI: HPET id: 0x8086a201 base: 0xfed00000502lb # [ 0.032556] CPU topo: Max. dies per package: 1503worker2 # [ 0.033364] TSC deadline timer available504worker1 # [ 0.032437] CPU topo: Max. dies per package: 1505lb # [ 0.032559] CPU topo: Max. threads per core: 1506worker1 # [ 0.032441] CPU topo: Max. threads per core: 1507worker2 # [ 0.033368] CPU topo: Max. logical packages: 1508lb # [ 0.032560] CPU topo: Num. cores per package: 1509worker1 # [ 0.032441] CPU topo: Num. cores per package: 1510worker2 # [ 0.033369] CPU topo: Max. logical dies: 1511lb # [ 0.032560] CPU topo: Num. threads per package: 1512client # [ 0.039231] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/8icb5zsbqv3dxvnzhf8ddq1cyhg5f86j-nixos-system-client-test/init regInfo=/nix/store/99yzymliyji70vhz8x5j5c4ckg24296s-closure-info/registration console=ttyS0,115200n8 console=tty0513worker2 # [ 0.033369] CPU topo: Max. dies per package: 1514worker1 # [ 0.032442] CPU topo: Num. threads per package: 1515lb # [ 0.032560] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs516worker2 # [ 0.033373] CPU topo: Max. threads per core: 1517worker1 # [ 0.032442] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs518worker2 # [ 0.033373] CPU topo: Num. cores per package: 1519lb # [ 0.032579] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()520client # [ 0.039325] Unknown kernel command line parameters "regInfo=/nix/store/99yzymliyji70vhz8x5j5c4ckg24296s-closure-info/registration", will be passed to user space.521worker1 # [ 0.032462] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()522worker2 # [ 0.033373] CPU topo: Num. threads per package: 1523client # [ 0.039337] random: crng init done524lb # [ 0.032610] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]525worker2 # [ 0.033374] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs526worker1 # [ 0.032493] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]527client # [ 0.039338] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes528lb # [ 0.032611] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]529worker2 # [ 0.033391] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()530worker1 # [ 0.032495] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]531client # [ 0.041429] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)532lb # [ 0.032613] [mem 0x60000000-0xafffffff] available for PCI devices533worker2 # [ 0.033420] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]534worker1 # [ 0.032496] [mem 0x60000000-0xafffffff] available for PCI devices535lb # [ 0.032614] Booting paravirtualized kernel on KVM536client # [ 0.042486] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)537worker1 # [ 0.032497] Booting paravirtualized kernel on KVM538client # [ 0.042518] Fallback order for Node 0: 0539worker2 # [ 0.033422] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]540lb # [ 0.032617] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns541worker2 # [ 0.033423] [mem 0x60000000-0xafffffff] available for PCI devices542worker1 # [ 0.032500] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns543client # [ 0.042521] Built 1 zonelists, mobility grouping on. Total pages: 393078544client # [ 0.042522] Policy zone: DMA32545worker2 # [ 0.033425] Booting paravirtualized kernel on KVM546lb # [ 0.037077] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1547worker1 # [ 0.036975] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1548client # [ 0.045043] mem auto-init: stack:all(zero), heap alloc:on, heap free:off549lb # [ 0.037202] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152550worker2 # [ 0.033428] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns551worker1 # [ 0.039313] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152552client # [ 0.048149] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1553lb # [ 0.037248] kvm-guest: PV spinlocks disabled, single CPU554worker1 # [ 0.039359] kvm-guest: PV spinlocks disabled, single CPU555client # [ 0.051629] allocated 3145728 bytes of page_ext556worker2 # [ 0.037896] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1557client # [ 0.061666] ftrace: allocating 48787 entries in 192 pages558worker2 # [ 0.040390] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152559client # [ 0.061668] ftrace: allocated 192 pages with 2 groups560worker2 # [ 0.040434] kvm-guest: PV spinlocks disabled, single CPU561client # [ 0.062487] Dynamic Preempt: lazy562client # [ 0.062604] rcu: Preemptible hierarchical RCU implementation.563client # [ 0.062605] rcu: RCU event tracing is enabled.564lb # [ 0.037250] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/5f2pyghh7bdk9s7schxljxgh1845dmij-nixos-system-lb-test/init regInfo=/nix/store/b17iww818kslhkx3acpxq2j13rz8h7sy-closure-info/registration console=ttyS0,115200n8 console=tty0565worker1 # [ 0.039361] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/7ivsh1xpnbmbn73qmia5fjss23hffswz-nixos-system-worker1-test/init regInfo=/nix/store/0m6yj9v4lz5rh7gnbwj9ql3hx52rgzbd-closure-info/registration console=ttyS0,115200n8 console=tty0566client # [ 0.062606] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.567client # [ 0.062607] Trampoline variant of Tasks RCU enabled.568client # [ 0.062607] Rude variant of Tasks RCU enabled.569lb # [ 0.037346] Unknown kernel command line parameters "regInfo=/nix/store/b17iww818kslhkx3acpxq2j13rz8h7sy-closure-info/registration", will be passed to user space.570worker1 # [ 0.039455] Unknown kernel command line parameters "regInfo=/nix/store/0m6yj9v4lz5rh7gnbwj9ql3hx52rgzbd-closure-info/registration", will be passed to user space.571worker2 # [ 0.040435] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/xis419qg6ssac1mxl23xpjki6jlxkhwd-nixos-system-worker2-test/init regInfo=/nix/store/5fllks9l0kij5jdjhq2g604s52nb8f4c-closure-info/registration console=ttyS0,115200n8 console=tty0572lb # [ 0.037358] random: crng init done573client # [ 0.062608] Tracing variant of Tasks RCU enabled.574worker1 # [ 0.039467] random: crng init done575client # [ 0.062608] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.576lb # [ 0.037359] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes577worker1 # [ 0.039468] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes578worker2 # [ 0.040526] Unknown kernel command line parameters "regInfo=/nix/store/5fllks9l0kij5jdjhq2g604s52nb8f4c-closure-info/registration", will be passed to user space.579client # [ 0.062609] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1580worker2 # [ 0.040538] random: crng init done581lb # [ 0.037682] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)582worker1 # [ 0.041562] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)583worker2 # [ 0.040539] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes584client # [ 0.062673] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.585lb # [ 0.037707] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)586worker1 # [ 0.042606] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)587lb # [ 0.037731] Fallback order for Node 0: 0588worker1 # [ 0.042636] Fallback order for Node 0: 0589worker2 # [ 0.042692] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)590client # [ 0.062674] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.591lb # [ 0.037734] Built 1 zonelists, mobility grouping on. Total pages: 393078592worker1 # [ 0.042639] Built 1 zonelists, mobility grouping on. Total pages: 393078593lb # [ 0.037735] Policy zone: DMA32594worker2 # [ 0.043777] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)595worker1 # [ 0.042640] Policy zone: DMA32596client # [ 0.062675] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.597worker2 # [ 0.043808] Fallback order for Node 0: 0598lb # [ 0.037839] mem auto-init: stack:all(zero), heap alloc:on, heap free:off599worker1 # [ 0.045113] mem auto-init: stack:all(zero), heap alloc:on, heap free:off600client # [ 0.067045] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16601worker2 # [ 0.043811] Built 1 zonelists, mobility grouping on. Total pages: 393078602lb # [ 0.040870] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1603worker2 # [ 0.043812] Policy zone: DMA32604worker1 # [ 0.048271] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1605client # [ 0.067334] rcu: srcu_init: Setting srcu_struct sizes based on contention.606lb # [ 0.044179] allocated 3145728 bytes of page_ext607worker1 # [ 0.051769] allocated 3145728 bytes of page_ext608worker2 # [ 0.046324] mem auto-init: stack:all(zero), heap alloc:on, heap free:off609lb # [ 0.054038] ftrace: allocating 48787 entries in 192 pages610worker1 # [ 0.061687] ftrace: allocating 48787 entries in 192 pages611client # [ 0.067340] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns612worker2 # [ 0.049446] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1613lb # [ 0.054040] ftrace: allocated 192 pages with 2 groups614worker1 # [ 0.061689] ftrace: allocated 192 pages with 2 groups615lb # [ 0.054913] Dynamic Preempt: lazy616worker2 # [ 0.053086] allocated 3145728 bytes of page_ext617worker1 # [ 0.062511] Dynamic Preempt: lazy618client # [ 0.067443] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)619lb # [ 0.055104] rcu: Preemptible hierarchical RCU implementation.620worker2 # [ 0.062878] ftrace: allocating 48787 entries in 192 pages621worker1 # [ 0.062627] rcu: Preemptible hierarchical RCU implementation.622client # [ 0.070952] Console: colour VGA+ 80x25623lb # [ 0.055105] rcu: RCU event tracing is enabled.624worker2 # [ 0.062880] ftrace: allocated 192 pages with 2 groups625worker1 # [ 0.062628] rcu: RCU event tracing is enabled.626client # [ 0.070955] printk: legacy console [tty0] enabled627worker2 # [ 0.063716] Dynamic Preempt: lazy628lb # [ 0.055105] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.629worker1 # [ 0.062629] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.630client # [ 0.100361] printk: legacy console [ttyS0] enabled631worker2 # [ 0.063885] rcu: Preemptible hierarchical RCU implementation.632lb # [ 0.055107] Trampoline variant of Tasks RCU enabled.633worker1 # [ 0.062630] Trampoline variant of Tasks RCU enabled.634worker2 # [ 0.063885] rcu: RCU event tracing is enabled.635client # [ 0.196947] ACPI: Core revision 20250807636lb # [ 0.055107] Rude variant of Tasks RCU enabled.637worker1 # [ 0.062631] Rude variant of Tasks RCU enabled.638lb # [ 0.055107] Tracing variant of Tasks RCU enabled.639worker2 # [ 0.063886] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.640worker1 # [ 0.062631] Tracing variant of Tasks RCU enabled.641worker2 # [ 0.063888] Trampoline variant of Tasks RCU enabled.642lb # [ 0.055108] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.643client # [ 0.197856] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns644worker1 # [ 0.062632] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.645worker2 # [ 0.063888] Rude variant of Tasks RCU enabled.646worker2 # [ 0.063888] Tracing variant of Tasks RCU enabled.647lb # [ 0.055109] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1648worker1 # [ 0.062633] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1649client # [ 0.199378] APIC: Switch to symmetric I/O mode setup650client # [ 0.200355] x2apic enabled651worker2 # [ 0.063889] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.652lb # [ 0.055119] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.653worker1 # [ 0.062683] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.654worker2 # [ 0.063890] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1655client # [ 0.201132] APIC: Switched APIC routing to: physical x2apic656lb # [ 0.055120] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.657worker1 # [ 0.062685] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.658worker2 # [ 0.063906] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.659lb # [ 0.055121] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.660worker1 # [ 0.062685] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.661worker2 # [ 0.063907] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.662lb # [ 0.059758] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16663worker1 # [ 0.067040] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16664client # [ 0.202934] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1665lb # [ 0.060064] rcu: srcu_init: Setting srcu_struct sizes based on contention.666worker1 # [ 0.067312] rcu: srcu_init: Setting srcu_struct sizes based on contention.667worker2 # [ 0.063908] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.668client # [ 0.203906] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns669worker2 # [ 0.068278] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16670lb # [ 0.060072] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns671worker1 # [ 0.067320] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns672worker2 # [ 0.068564] rcu: srcu_init: Setting srcu_struct sizes based on contention.673client # [ 0.205544] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)674lb # [ 0.060181] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)675worker1 # [ 0.067422] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)676lb # [ 0.063714] Console: colour VGA+ 80x25677worker2 # [ 0.068571] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns678worker1 # [ 0.070985] Console: colour VGA+ 80x25679client # [ 0.206628] x86/cpu: User Mode Instruction Prevention (UMIP) activated680lb # [ 0.063717] printk: legacy console [tty0] enabled681worker1 # [ 0.070987] printk: legacy console [tty0] enabled682lb # [ 0.093225] printk: legacy console [ttyS0] enabled683worker2 # [ 0.068674] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)684worker1 # [ 0.100398] printk: legacy console [ttyS0] enabled685worker2 # [ 0.072175] Console: colour VGA+ 80x25686client # [ 0.207694] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127687lb # [ 0.189095] ACPI: Core revision 20250807688worker1 # [ 0.195974] ACPI: Core revision 20250807689worker2 # [ 0.072177] printk: legacy console [tty0] enabled690client # [ 0.208541] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0691worker2 # [ 0.101682] printk: legacy console [ttyS0] enabled692worker2 # [ 0.196780] ACPI: Core revision 20250807693worker1 # [ 0.196854] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns694lb # [ 0.189987] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns695client # [ 0.209544] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto696lb # [ 0.191553] APIC: Switch to symmetric I/O mode setup697worker1 # [ 0.198324] APIC: Switch to symmetric I/O mode setup698worker2 # [ 0.197655] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns699worker1 # [ 0.199292] x2apic enabled700lb # [ 0.192534] x2apic enabled701client # [ 0.211540] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl702worker2 # [ 0.199156] APIC: Switch to symmetric I/O mode setup703lb # [ 0.193310] APIC: Switched APIC routing to: physical x2apic704worker1 # [ 0.200026] APIC: Switched APIC routing to: physical x2apic705client # [ 0.212540] Transient Scheduler Attacks: Vulnerable: No microcode706worker2 # [ 0.200119] x2apic enabled707client # [ 0.213540] Spectre V2 : Mitigation: Enhanced / Automatic IBRS708worker2 # [ 0.200856] APIC: Switched APIC routing to: physical x2apic709lb # [ 0.195102] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1710worker1 # [ 0.201831] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1711client # [ 0.214541] Speculative Return Stack Overflow: Mitigation: Safe RET712worker2 # [ 0.202658] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1713lb # [ 0.196075] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns714worker1 # [ 0.202788] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns715worker1 # [ 0.204497] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)716lb # [ 0.197774] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)717worker2 # [ 0.203617] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns718lb # [ 0.198864] x86/cpu: User Mode Instruction Prevention (UMIP) activated719worker1 # [ 0.205586] x86/cpu: User Mode Instruction Prevention (UMIP) activated720client # [ 0.215540] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization721worker2 # [ 0.205295] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)722client # [ 0.217548] Spectre V2 : Enabling IBPB for BPF723worker1 # [ 0.206651] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127724lb # [ 0.199914] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127725worker2 # [ 0.206379] x86/cpu: User Mode Instruction Prevention (UMIP) activated726worker1 # [ 0.207493] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0727lb # [ 0.201447] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0728client # [ 0.218541] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier729worker2 # [ 0.207462] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127730client # [ 0.219541] active return thunk: srso_alias_return_thunk731worker2 # [ 0.208292] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0732worker1 # [ 0.208497] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto733lb # [ 0.201775] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto734client # [ 0.220562] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'735worker1 # [ 0.210493] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl736lb # [ 0.203771] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl737worker2 # [ 0.209296] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto738client # [ 0.221540] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'739worker1 # [ 0.211493] Transient Scheduler Attacks: Vulnerable: No microcode740lb # [ 0.204771] Transient Scheduler Attacks: Vulnerable: No microcode741client # [ 0.222540] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'742worker2 # [ 0.211292] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl743worker1 # [ 0.212493] Spectre V2 : Mitigation: Enhanced / Automatic IBRS744lb # [ 0.205771] Spectre V2 : Mitigation: Enhanced / Automatic IBRS745client # [ 0.223540] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'746worker2 # [ 0.212292] Transient Scheduler Attacks: Vulnerable: No microcode747worker1 # [ 0.213493] Speculative Return Stack Overflow: Mitigation: Safe RET748lb # [ 0.206771] Speculative Return Stack Overflow: Mitigation: Safe RET749client # [ 0.224540] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'750worker2 # [ 0.213292] Spectre V2 : Mitigation: Enhanced / Automatic IBRS751client # [ 0.225540] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'752worker2 # [ 0.214292] Speculative Return Stack Overflow: Mitigation: Safe RET753client # [ 0.226540] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'754client # [ 0.227541] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256755worker1 # [ 0.214493] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization756lb # [ 0.207771] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization757worker1 # [ 0.216500] Spectre V2 : Enabling IBPB for BPF758client # [ 0.229249] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64759lb # [ 0.209778] Spectre V2 : Enabling IBPB for BPF760worker2 # [ 0.215292] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization761client # [ 0.230252] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512762worker1 # [ 0.217207] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier763worker2 # [ 0.217300] Spectre V2 : Enabling IBPB for BPF764lb # [ 0.210771] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier765client # [ 0.230541] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024766worker1 # [ 0.218493] active return thunk: srso_alias_return_thunk767lb # [ 0.211771] active return thunk: srso_alias_return_thunk768client # [ 0.231540] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8769worker2 # [ 0.218293] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier770worker1 # [ 0.219515] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'771lb # [ 0.212793] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'772worker2 # [ 0.219293] active return thunk: srso_alias_return_thunk773worker1 # [ 0.220493] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'774client # [ 0.232541] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.775lb # [ 0.213770] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'776worker2 # [ 0.220313] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'777worker1 # [ 0.221493] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'778lb # [ 0.215502] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'779worker2 # [ 0.221292] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'780worker1 # [ 0.222493] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'781lb # [ 0.216524] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'782worker2 # [ 0.222292] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'783worker1 # [ 0.223493] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'784lb # [ 0.217527] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'785worker2 # [ 0.223292] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'786worker1 # [ 0.224493] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'787lb # [ 0.218517] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'788worker2 # [ 0.224292] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'789worker1 # [ 0.225493] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'790worker2 # [ 0.225292] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'791lb # [ 0.219554] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'792worker1 # [ 0.226494] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256793lb # [ 0.220771] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256794worker2 # [ 0.226292] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'795worker1 # [ 0.227493] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64796lb # [ 0.221771] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64797worker2 # [ 0.227293] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256798worker1 # [ 0.228493] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512799lb # [ 0.222770] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512800worker2 # [ 0.228292] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64801worker1 # [ 0.229493] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024802lb # [ 0.223770] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024803worker2 # [ 0.229292] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512804worker1 # [ 0.230493] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8805lb # [ 0.224771] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8806worker2 # [ 0.230292] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024807worker1 # [ 0.231493] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.808worker2 # [ 0.231292] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8809lb # [ 0.225720] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.810worker2 # [ 0.232292] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.811client # [ 0.266474] Freeing SMP alternatives memory: 44K812client # [ 0.267107] pid_max: default: 32768 minimum: 301813client # [ 0.267642] LSM: initializing lsm=capability,landlock,yama,bpf,ima814client # [ 0.268644] landlock: Up and running.815client # [ 0.269541] Yama: becoming mindful.816client # [ 0.270738] LSM support for eBPF active817client # [ 0.271482] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)818client # [ 0.272577] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)819client # [ 0.275011] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)820client # [ 0.276133] Performance Events: Fam17h+ core perfctr, AMD PMU driver.821worker1 # [ 0.265748] Freeing SMP alternatives memory: 44K822client # [ 0.276545] ... version: 2823lb # [ 0.259760] Freeing SMP alternatives memory: 44K824worker1 # [ 0.266496] pid_max: default: 32768 minimum: 301825client # [ 0.277254] ... bit width: 48826lb # [ 0.260554] pid_max: default: 32768 minimum: 301827worker1 # [ 0.267317] LSM: initializing lsm=capability,landlock,yama,bpf,ima828client # [ 0.277543] ... generic counters: 6829worker2 # [ 0.267103] Freeing SMP alternatives memory: 44K830worker1 # [ 0.268281] landlock: Up and running.831lb # [ 0.260892] LSM: initializing lsm=capability,landlock,yama,bpf,ima832worker2 # [ 0.267836] pid_max: default: 32768 minimum: 301833client # [ 0.278215] ... generic bitmap: 000000000000003f834worker1 # [ 0.268907] Yama: becoming mindful.835lb # [ 0.261868] landlock: Up and running.836client # [ 0.278554] ... fixed-purpose counters: 0837worker1 # [ 0.269706] LSM support for eBPF active838worker2 # [ 0.268388] LSM: initializing lsm=capability,landlock,yama,bpf,ima839lb # [ 0.262771] Yama: becoming mindful.840client # [ 0.279265] ... fixed-purpose bitmap: 0000000000000000841worker2 # [ 0.269386] landlock: Up and running.842lb # [ 0.263984] LSM support for eBPF active843worker1 # [ 0.270550] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)844worker2 # [ 0.270293] Yama: becoming mindful.845client # [ 0.279543] ... value mask: 0000ffffffffffff846worker2 # [ 0.271072] LSM support for eBPF active847lb # [ 0.264787] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)848client # [ 0.280415] ... max period: 00007fffffffffff849worker1 # [ 0.271533] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)850client # [ 0.281171] ... global_ctrl mask: 000000000000003f851worker2 # [ 0.271847] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)852lb # [ 0.265810] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)853client # [ 0.281654] signal: max sigframe size: 3376854worker2 # [ 0.273179] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)855client # [ 0.282442] rcu: Hierarchical SRCU implementation.856worker1 # [ 0.273985] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)857client # [ 0.283143] rcu: Max phase no-delay instances is 400.858lb # [ 0.268353] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)859worker1 # [ 0.275046] Performance Events: Fam17h+ core perfctr, AMD PMU driver.860worker2 # [ 0.275432] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)861worker1 # [ 0.275502] ... version: 2862lb # [ 0.269296] Performance Events: Fam17h+ core perfctr, AMD PMU driver.863worker1 # [ 0.276198] ... bit width: 48864lb # [ 0.269775] ... version: 2865worker1 # [ 0.276496] ... generic counters: 6866worker2 # [ 0.276877] Performance Events: Fam17h+ core perfctr, AMD PMU driver.867lb # [ 0.270493] ... bit width: 48868worker2 # [ 0.277301] ... version: 2869worker1 # [ 0.277172] ... generic bitmap: 000000000000003f870lb # [ 0.270774] ... generic counters: 6871worker2 # [ 0.277965] ... bit width: 48872worker1 # [ 0.277496] ... fixed-purpose counters: 0873lb # [ 0.271467] ... generic bitmap: 000000000000003f874worker2 # [ 0.278295] ... generic counters: 6875client # [ 0.288101] smp: Bringing up secondary CPUs ...876worker1 # [ 0.278187] ... fixed-purpose bitmap: 0000000000000000877lb # [ 0.271820] ... fixed-purpose counters: 0878client # [ 0.288557] smp: Brought up 1 node, 1 CPU879worker2 # [ 0.279004] ... generic bitmap: 000000000000003f880worker1 # [ 0.278496] ... value mask: 0000ffffffffffff881lb # [ 0.272555] ... fixed-purpose bitmap: 0000000000000000882worker2 # [ 0.279295] ... fixed-purpose counters: 0883client # [ 0.289219] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)884worker1 # [ 0.279335] ... max period: 00007fffffffffff885worker2 # [ 0.279954] ... fixed-purpose bitmap: 0000000000000000886lb # [ 0.272774] ... value mask: 0000ffffffffffff887worker1 # [ 0.280115] ... global_ctrl mask: 000000000000003f888worker2 # [ 0.280295] ... value mask: 0000ffffffffffff889lb # [ 0.273693] ... max period: 00007fffffffffff890worker1 # [ 0.280592] signal: max sigframe size: 3376891client # [ 0.289726] Memory: 1454604K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109304K reserved, 0K cma-reserved)892worker2 # [ 0.281159] ... max period: 00007fffffffffff893lb # [ 0.274477] ... global_ctrl mask: 000000000000003f894worker1 # [ 0.281369] rcu: Hierarchical SRCU implementation.895client # [ 0.290849] devtmpfs: initialized896worker2 # [ 0.281957] ... global_ctrl mask: 000000000000003f897lb # [ 0.274885] signal: max sigframe size: 3376898worker1 # [ 0.282092] rcu: Max phase no-delay instances is 400.899worker2 # [ 0.282403] signal: max sigframe size: 3376900client # [ 0.291629] x86/mm: Memory block size: 128MB901lb # [ 0.275638] rcu: Hierarchical SRCU implementation.902worker2 # [ 0.283229] rcu: Hierarchical SRCU implementation.903lb # [ 0.276341] rcu: Max phase no-delay instances is 400.904worker2 # [ 0.283845] rcu: Max phase no-delay instances is 400.905client # [ 0.293386] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)906client # [ 0.294463] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).907client # [ 0.295615] pinctrl core: initialized pinctrl subsystem908worker1 # [ 0.287002] smp: Bringing up secondary CPUs ...909worker1 # [ 0.287509] smp: Brought up 1 node, 1 CPU910client # [ 0.296710] PM: RTC time: 06:58:58, date: 2026-09-23911lb # [ 0.281288] smp: Bringing up secondary CPUs ...912worker1 # [ 0.288211] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)913lb # [ 0.281787] smp: Brought up 1 node, 1 CPU914worker2 # [ 0.288843] smp: Bringing up secondary CPUs ...915worker2 # [ 0.289309] smp: Brought up 1 node, 1 CPU916lb # [ 0.282453] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)917worker2 # [ 0.289944] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)918worker1 # [ 0.288678] Memory: 1454616K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)919worker1 # [ 0.289761] devtmpfs: initialized920client # [ 0.300453] NET: Registered PF_NETLINK/PF_ROUTE protocol family921lb # [ 0.282933] Memory: 1454584K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109316K reserved, 0K cma-reserved)922worker1 # [ 0.290558] x86/mm: Memory block size: 128MB923worker2 # [ 0.290502] Memory: 1454612K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109312K reserved, 0K cma-reserved)924lb # [ 0.284009] devtmpfs: initialized925client # [ 0.301723] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations926worker2 # [ 0.291528] devtmpfs: initialized927lb # [ 0.284800] x86/mm: Memory block size: 128MB928client # [ 0.302567] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations929worker2 # [ 0.292306] x86/mm: Memory block size: 128MB930worker1 # [ 0.292331] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)931client # [ 0.303816] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations932lb # [ 0.286610] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)933worker1 # [ 0.293449] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).934client # [ 0.304553] audit: initializing netlink subsys (disabled)935worker2 # [ 0.294057] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)936worker1 # [ 0.294586] pinctrl core: initialized pinctrl subsystem937lb # [ 0.287655] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).938client # [ 0.305717] thermal_sys: Registered thermal governor 'fair_share'939worker1 # [ 0.295659] PM: RTC time: 06:58:58, date: 2026-09-23940worker2 # [ 0.295167] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).941lb # [ 0.288857] pinctrl core: initialized pinctrl subsystem942client # [ 0.305720] thermal_sys: Registered thermal governor 'bang_bang'943worker2 # [ 0.296380] pinctrl core: initialized pinctrl subsystem944lb # [ 0.289987] PM: RTC time: 06:58:58, date: 2026-09-23945client # [ 0.306544] thermal_sys: Registered thermal governor 'step_wise'946worker2 # [ 0.297515] PM: RTC time: 06:58:58, date: 2026-09-23947client # [ 0.307509] thermal_sys: Registered thermal governor 'user_space'948worker1 # [ 0.299306] NET: Registered PF_NETLINK/PF_ROUTE protocol family949client # [ 0.308290] audit: type=2000 audit(1790146739.086:1): state=initialized audit_enabled=0 res=1950lb # [ 0.293531] NET: Registered PF_NETLINK/PF_ROUTE protocol family951client # [ 0.309546] thermal_sys: Registered thermal governor 'power_allocator'952worker1 # [ 0.300752] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations953client # [ 0.309559] cpuidle: using governor menu954worker2 # [ 0.301385] NET: Registered PF_NETLINK/PF_ROUTE protocol family955worker1 # [ 0.301520] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations956lb # [ 0.294266] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations957client # [ 0.311773] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5958worker2 # [ 0.302807] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations959lb # [ 0.294799] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations960worker1 # [ 0.302766] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations961worker2 # [ 0.303317] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations962client # [ 0.312774] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]963worker1 # [ 0.303507] audit: initializing netlink subsys (disabled)964lb # [ 0.296045] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations965client # [ 0.313546] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry966worker1 # [ 0.304606] thermal_sys: Registered thermal governor 'fair_share'967worker2 # [ 0.304573] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations968lb # [ 0.296791] audit: initializing netlink subsys (disabled)969client # [ 0.314556] PCI: Using configuration type 1 for base access970worker1 # [ 0.304609] thermal_sys: Registered thermal governor 'bang_bang'971worker2 # [ 0.305304] audit: initializing netlink subsys (disabled)972lb # [ 0.297775] audit: type=2000 audit(1790146739.098:1): state=initialized audit_enabled=0 res=1973worker1 # [ 0.305497] thermal_sys: Registered thermal governor 'step_wise'974worker2 # [ 0.306350] thermal_sys: Registered thermal governor 'fair_share'975client # [ 0.315670] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.976lb # [ 0.298887] thermal_sys: Registered thermal governor 'fair_share'977worker1 # [ 0.306434] thermal_sys: Registered thermal governor 'user_space'978worker2 # [ 0.306352] thermal_sys: Registered thermal governor 'bang_bang'979lb # [ 0.298890] thermal_sys: Registered thermal governor 'bang_bang'980worker2 # [ 0.307263] thermal_sys: Registered thermal governor 'step_wise'981worker1 # [ 0.307218] audit: type=2000 audit(1790146739.098:1): state=initialized audit_enabled=0 res=1982lb # [ 0.299774] thermal_sys: Registered thermal governor 'step_wise'983worker2 # [ 0.307971] thermal_sys: Registered thermal governor 'user_space'984worker1 # [ 0.308499] thermal_sys: Registered thermal governor 'power_allocator'985lb # [ 0.300693] thermal_sys: Registered thermal governor 'user_space'986worker1 # [ 0.308513] cpuidle: using governor menu987worker2 # [ 0.308298] audit: type=2000 audit(1790146739.101:1): state=initialized audit_enabled=0 res=1988lb # [ 0.301487] thermal_sys: Registered thermal governor 'power_allocator'989lb # [ 0.301785] cpuidle: using governor menu990worker2 # [ 0.310297] thermal_sys: Registered thermal governor 'power_allocator'991worker1 # [ 0.310732] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5992client # [ 0.320813] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages993worker2 # [ 0.310311] cpuidle: using governor menu994client # [ 0.321544] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page995lb # [ 0.303960] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5996worker1 # [ 0.311755] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]997worker2 # [ 0.312336] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5998worker1 # [ 0.312499] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry999lb # [ 0.305056] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]1000worker2 # [ 0.313519] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]1001worker1 # [ 0.313508] PCI: Using configuration type 1 for base access1002lb # [ 0.305777] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry1003worker2 # [ 0.314298] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry1004lb # [ 0.306785] PCI: Using configuration type 1 for base access1005worker1 # [ 0.314623] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.1006worker2 # [ 0.315306] PCI: Using configuration type 1 for base access1007client # [ 0.326544] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1008lb # [ 0.307852] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.1009worker2 # [ 0.316388] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.1010client # [ 0.327543] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1011worker1 # [ 0.319617] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1012worker1 # [ 0.320497] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1013worker2 # [ 0.321388] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1014worker2 # [ 0.322296] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1015lb # [ 0.314787] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1016lb # [ 0.315773] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1017worker1 # [ 0.325497] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1018client # [ 0.337992] ACPI: Added _OSI(Module Device)1019worker1 # [ 0.326495] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1020client # [ 0.338544] ACPI: Added _OSI(Processor Device)1021worker2 # [ 0.327296] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1022client # [ 0.339278] ACPI: Added _OSI(Processor Aggregator Device)1023worker2 # [ 0.328295] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1024lb # [ 0.320775] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1025lb # [ 0.321774] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1026client # [ 0.346920] ACPI: 1 ACPI AML tables successfully acquired and loaded1027worker1 # [ 0.336839] ACPI: Added _OSI(Module Device)1028worker1 # [ 0.337497] ACPI: Added _OSI(Processor Device)1029worker2 # [ 0.338679] ACPI: Added _OSI(Module Device)1030worker1 # [ 0.338257] ACPI: Added _OSI(Processor Aggregator Device)1031worker2 # [ 0.339296] ACPI: Added _OSI(Processor Device)1032worker2 # [ 0.340039] ACPI: Added _OSI(Processor Aggregator Device)1033client # [ 0.350950] ACPI: Interpreter enabled1034lb # [ 0.333064] ACPI: Added _OSI(Module Device)1035client # [ 0.351558] ACPI: PM: (supports S0 S3 S4 S5)1036lb # [ 0.333775] ACPI: Added _OSI(Processor Device)1037client # [ 0.352255] ACPI: Using IOAPIC for interrupt routing1038lb # [ 0.334580] ACPI: Added _OSI(Processor Aggregator Device)1039client # [ 0.354614] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1040client # [ 0.355543] PCI: Using E820 reservations for host bridge windows1041worker1 # [ 0.346580] ACPI: 1 ACPI AML tables successfully acquired and loaded1042worker2 # [ 0.348359] ACPI: 1 ACPI AML tables successfully acquired and loaded1043lb # [ 0.341340] ACPI: 1 ACPI AML tables successfully acquired and loaded1044client # [ 0.358643] ACPI: Enabled 2 GPEs in block 00 to 3F1045worker1 # [ 0.350899] ACPI: Interpreter enabled1046worker1 # [ 0.351511] ACPI: PM: (supports S0 S3 S4 S5)1047worker2 # [ 0.352720] ACPI: Interpreter enabled1048lb # [ 0.345204] ACPI: Interpreter enabled1049worker1 # [ 0.352263] ACPI: Using IOAPIC for interrupt routing1050worker2 # [ 0.353310] ACPI: PM: (supports S0 S3 S4 S5)1051lb # [ 0.345791] ACPI: PM: (supports S0 S3 S4 S5)1052worker2 # [ 0.354026] ACPI: Using IOAPIC for interrupt routing1053lb # [ 0.346514] ACPI: Using IOAPIC for interrupt routing1054worker1 # [ 0.354579] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1055client # [ 0.366445] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1056worker1 # [ 0.355497] PCI: Using E820 reservations for host bridge windows1057worker2 # [ 0.356360] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1058lb # [ 0.348817] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1059client # [ 0.367289] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1060worker2 # [ 0.357296] PCI: Using E820 reservations for host bridge windows1061lb # [ 0.349773] PCI: Using E820 reservations for host bridge windows1062client # [ 0.368622] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1063worker1 # [ 0.358538] ACPI: Enabled 2 GPEs in block 00 to 3F1064client # [ 0.369668] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1065worker2 # [ 0.360380] ACPI: Enabled 2 GPEs in block 00 to 3F1066lb # [ 0.352878] ACPI: Enabled 2 GPEs in block 00 to 3F1067client # [ 0.370932] PCI host bridge to bus 0000:001068client # [ 0.371548] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1069client # [ 0.372543] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1070client # [ 0.373543] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1071client # [ 0.374543] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1072client # [ 0.375543] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1073worker1 # [ 0.365923] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1074client # [ 0.376544] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1075lb # [ 0.359893] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1076worker2 # [ 0.367930] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1077worker1 # [ 0.366503] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1078client # [ 0.377554] pci_bus 0000:00: root bus resource [bus 00-ff]1079lb # [ 0.360779] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1080worker2 # [ 0.368302] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1081worker1 # [ 0.367575] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1082client # [ 0.378477] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1083lb # [ 0.361849] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1084worker1 # [ 0.368618] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1085worker2 # [ 0.369375] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1086client # [ 0.379962] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1087lb # [ 0.362896] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1088worker1 # [ 0.369862] PCI host bridge to bus 0000:001089worker2 # [ 0.370417] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1090lb # [ 0.364140] PCI host bridge to bus 0000:001091worker1 # [ 0.370500] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1092worker2 # [ 0.371646] PCI host bridge to bus 0000:001093lb # [ 0.364778] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1094worker1 # [ 0.371496] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1095worker2 # [ 0.372300] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1096lb # [ 0.365773] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1097worker2 # [ 0.373295] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1098worker1 # [ 0.372496] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1099lb # [ 0.366773] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1100worker2 # [ 0.374295] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1101worker1 # [ 0.373496] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1102lb # [ 0.367773] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1103worker2 # [ 0.375295] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1104worker1 # [ 0.374496] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1105lb # [ 0.368773] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1106worker2 # [ 0.376295] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1107worker1 # [ 0.375496] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1108client # [ 0.382636] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1109lb # [ 0.369773] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1110worker1 # [ 0.376502] pci_bus 0000:00: root bus resource [bus 00-ff]1111worker2 # [ 0.377299] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1112client # [ 0.383555] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1113lb # [ 0.370774] pci_bus 0000:00: root bus resource [bus 00-ff]1114worker2 # [ 0.378295] pci_bus 0000:00: root bus resource [bus 00-ff]1115worker1 # [ 0.377450] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1116client # [ 0.384504] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1117lb # [ 0.371765] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1118worker2 # [ 0.379238] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1119worker1 # [ 0.378989] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1120client # [ 0.385512] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1121lb # [ 0.373217] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1122worker2 # [ 0.380782] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1123client # [ 0.387283] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1124client # [ 0.390586] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1125client # [ 0.391482] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1126client # [ 0.392310] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1127worker1 # [ 0.381584] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1128client # [ 0.392578] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1129lb # [ 0.375839] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1130worker1 # [ 0.382508] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1131worker2 # [ 0.383434] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1132lb # [ 0.376786] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1133worker1 # [ 0.383449] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1134worker2 # [ 0.384308] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1135lb # [ 0.377740] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1136client # [ 0.394230] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1137worker2 # [ 0.385256] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1138worker1 # [ 0.384477] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1139lb # [ 0.378857] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1140worker2 # [ 0.386337] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1141client # [ 0.396583] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1142worker1 # [ 0.386234] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1143lb # [ 0.380662] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1144client # [ 0.397444] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1145worker2 # [ 0.388016] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1146client # [ 0.398293] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1147worker1 # [ 0.389532] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1148lb # [ 0.383790] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1149client # [ 0.399125] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1150worker1 # [ 0.390387] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1151worker2 # [ 0.391336] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1152lb # [ 0.384641] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1153worker1 # [ 0.391201] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1154worker2 # [ 0.392182] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1155lb # [ 0.385507] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1156worker1 # [ 0.391502] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1157worker2 # [ 0.393050] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1158lb # [ 0.385779] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1159worker2 # [ 0.393302] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1160client # [ 0.401852] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1161worker1 # [ 0.393023] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1162lb # [ 0.387316] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1163client # [ 0.403176] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1164worker2 # [ 0.394855] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1165client # [ 0.403564] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1166worker1 # [ 0.394532] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1167lb # [ 0.388791] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1168worker1 # [ 0.395403] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1169worker2 # [ 0.397334] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1170client # [ 0.405122] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1171lb # [ 0.389635] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1172worker2 # [ 0.398174] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1173worker1 # [ 0.396202] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1174lb # [ 0.390493] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1175worker2 # [ 0.399028] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1176client # [ 0.406580] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1177worker1 # [ 0.397072] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1178lb # [ 0.391319] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1179client # [ 0.407440] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1180worker2 # [ 0.399864] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1181client # [ 0.408269] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1182client # [ 0.408549] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1183worker1 # [ 0.399798] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1184lb # [ 0.394047] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1185client # [ 0.410082] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1186worker2 # [ 0.402303] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1187worker1 # [ 0.401174] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1188lb # [ 0.395401] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1189worker2 # [ 0.403930] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1190worker1 # [ 0.401518] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1191lb # [ 0.395795] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1192client # [ 0.411587] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1193worker2 # [ 0.404317] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1194client # [ 0.412543] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1195worker1 # [ 0.403070] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1196lb # [ 0.397331] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1197worker2 # [ 0.405876] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1198client # [ 0.414103] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1199worker1 # [ 0.404530] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1200lb # [ 0.398794] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1201worker2 # [ 0.407334] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1202worker1 # [ 0.405408] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1203lb # [ 0.399654] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1204client # [ 0.415586] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1205worker2 # [ 0.408181] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1206worker1 # [ 0.406255] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1207lb # [ 0.400497] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1208worker2 # [ 0.409022] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1209client # [ 0.416548] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1210lb # [ 0.400780] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1211worker1 # [ 0.406502] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1212worker2 # [ 0.409303] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1213client # [ 0.418148] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1214lb # [ 0.402681] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1215worker1 # [ 0.408043] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1216worker2 # [ 0.410872] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1217client # [ 0.419586] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1218lb # [ 0.404795] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1219worker1 # [ 0.409538] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1220worker2 # [ 0.412344] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1221client # [ 0.420565] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1222lb # [ 0.405745] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1223worker1 # [ 0.410496] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1224worker2 # [ 0.413285] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1225client # [ 0.422076] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1226lb # [ 0.407141] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1227worker1 # [ 0.412049] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1228worker2 # [ 0.414675] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1229client # [ 0.423586] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1230lb # [ 0.408795] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1231worker1 # [ 0.413538] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1232worker2 # [ 0.416341] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1233lb # [ 0.409747] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1234client # [ 0.424565] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1235worker1 # [ 0.414517] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1236worker2 # [ 0.417300] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1237client # [ 0.426073] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1238lb # [ 0.411868] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1239worker1 # [ 0.416133] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1240worker2 # [ 0.418935] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1241client # [ 0.427561] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1242lb # [ 0.413785] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1243worker1 # [ 0.417538] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1244worker2 # [ 0.420338] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1245client # [ 0.428464] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1246lb # [ 0.414742] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1247worker1 # [ 0.418518] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1248worker2 # [ 0.421251] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1249client # [ 0.429300] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1250lb # [ 0.416123] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1251worker1 # [ 0.420043] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1252worker2 # [ 0.422638] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1253client # [ 0.430110] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1254lb # [ 0.417787] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1255worker1 # [ 0.421538] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1256worker2 # [ 0.424342] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1257lb # [ 0.418773] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1258client # [ 0.432580] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1259worker1 # [ 0.422518] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1260worker2 # [ 0.425294] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1261client # [ 0.433411] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1262lb # [ 0.420325] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1263worker1 # [ 0.424039] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1264client # [ 0.434296] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1265worker2 # [ 0.426874] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1266lb # [ 0.421792] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1267client # [ 0.435141] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1268worker1 # [ 0.425513] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1269worker2 # [ 0.428869] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1270lb # [ 0.422647] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1271worker1 # [ 0.426378] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1272client # [ 0.436229] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1273worker2 # [ 0.429302] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1274lb # [ 0.423513] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1275worker1 # [ 0.427242] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1276worker2 # [ 0.430254] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1277client # [ 0.436752] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1278lb # [ 0.424321] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1279worker1 # [ 0.428046] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1280worker2 # [ 0.431654] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1281client # [ 0.438261] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1282client # [ 0.438781] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1283lb # [ 0.426151] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1284worker1 # [ 0.429505] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1285worker2 # [ 0.433303] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1286client # [ 0.440232] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1287lb # [ 0.427416] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1288worker1 # [ 0.431164] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1289worker2 # [ 0.434898] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1290lb # [ 0.427795] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1291worker1 # [ 0.431517] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1292worker2 # [ 0.435317] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1293lb # [ 0.429342] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1294worker1 # [ 0.433065] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1295worker2 # [ 0.436893] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1296lb # [ 0.430529] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1297worker1 # [ 0.434221] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1298worker2 # [ 0.438030] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1299lb # [ 0.430976] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1300worker1 # [ 0.434691] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1301worker2 # [ 0.438491] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1302lb # [ 0.432415] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1303worker1 # [ 0.436148] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1304worker2 # [ 0.439973] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1305lb # [ 0.432968] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1306worker1 # [ 0.436687] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1307worker2 # [ 0.440491] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1308lb # [ 0.434489] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1309worker2 # [ 0.441971] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1310worker1 # [ 0.438199] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1311client # [ 0.440757] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1312client # [ 0.443180] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1313client # [ 0.443949] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1314client # [ 0.444829] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1315client # [ 0.445789] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1316client # [ 0.447602] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1317client # [ 0.448513] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1318client # [ 0.449677] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1319worker2 # [ 0.442549] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1320lb # [ 0.434991] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1321worker1 # [ 0.438717] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1322client # [ 0.451210] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1323worker2 # [ 0.444945] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1324worker1 # [ 0.441120] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1325lb # [ 0.437397] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1326worker2 # [ 0.445586] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1327worker1 # [ 0.441801] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1328lb # [ 0.438066] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1329worker2 # [ 0.446625] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1330lb # [ 0.439066] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1331worker1 # [ 0.442794] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1332client # [ 0.455155] ACPI: PCI: Interrupt link LNKA configured for IRQ 101333lb # [ 0.440029] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1334worker1 # [ 0.443780] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1335worker2 # [ 0.447554] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1336client # [ 0.455685] ACPI: PCI: Interrupt link LNKB configured for IRQ 101337client # [ 0.456643] ACPI: PCI: Interrupt link LNKC configured for IRQ 111338lb # [ 0.441801] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1339worker1 # [ 0.445524] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1340worker2 # [ 0.449357] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1341client # [ 0.457645] ACPI: PCI: Interrupt link LNKD configured for IRQ 111342lb # [ 0.442642] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1343worker1 # [ 0.446371] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1344worker2 # [ 0.450199] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1345client # [ 0.458646] ACPI: PCI: Interrupt link LNKE configured for IRQ 101346lb # [ 0.443794] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1347worker1 # [ 0.447543] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1348worker2 # [ 0.451334] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1349client # [ 0.459643] ACPI: PCI: Interrupt link LNKF configured for IRQ 101350lb # [ 0.445425] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1351client # [ 0.460645] ACPI: PCI: Interrupt link LNKG configured for IRQ 111352worker1 # [ 0.449201] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1353worker2 # [ 0.452919] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1354client # [ 0.461640] ACPI: PCI: Interrupt link LNKH configured for IRQ 111355client # [ 0.462582] ACPI: PCI: Interrupt link GSIA configured for IRQ 161356client # [ 0.463560] ACPI: PCI: Interrupt link GSIB configured for IRQ 171357client # [ 0.464555] ACPI: PCI: Interrupt link GSIC configured for IRQ 181358lb # [ 0.448798] ACPI: PCI: Interrupt link LNKA configured for IRQ 101359worker1 # [ 0.452573] ACPI: PCI: Interrupt link LNKA configured for IRQ 101360worker2 # [ 0.456733] ACPI: PCI: Interrupt link LNKA configured for IRQ 101361client # [ 0.465504] ACPI: PCI: Interrupt link GSID configured for IRQ 191362lb # [ 0.449882] ACPI: PCI: Interrupt link LNKB configured for IRQ 101363worker2 # [ 0.457429] ACPI: PCI: Interrupt link LNKB configured for IRQ 101364worker1 # [ 0.453603] ACPI: PCI: Interrupt link LNKB configured for IRQ 101365client # [ 0.466273] ACPI: PCI: Interrupt link GSIE configured for IRQ 201366lb # [ 0.450873] ACPI: PCI: Interrupt link LNKC configured for IRQ 111367worker2 # [ 0.458427] ACPI: PCI: Interrupt link LNKC configured for IRQ 111368worker1 # [ 0.454605] ACPI: PCI: Interrupt link LNKC configured for IRQ 111369client # [ 0.466557] ACPI: PCI: Interrupt link GSIF configured for IRQ 211370lb # [ 0.451874] ACPI: PCI: Interrupt link LNKD configured for IRQ 111371worker2 # [ 0.459432] ACPI: PCI: Interrupt link LNKD configured for IRQ 111372worker1 # [ 0.455604] ACPI: PCI: Interrupt link LNKD configured for IRQ 111373client # [ 0.467514] ACPI: PCI: Interrupt link GSIG configured for IRQ 221374lb # [ 0.452871] ACPI: PCI: Interrupt link LNKE configured for IRQ 101375worker2 # [ 0.460398] ACPI: PCI: Interrupt link LNKE configured for IRQ 101376worker1 # [ 0.456596] ACPI: PCI: Interrupt link LNKE configured for IRQ 101377client # [ 0.468294] ACPI: PCI: Interrupt link GSIH configured for IRQ 231378lb # [ 0.453870] ACPI: PCI: Interrupt link LNKF configured for IRQ 101379worker2 # [ 0.461393] ACPI: PCI: Interrupt link LNKF configured for IRQ 101380worker1 # [ 0.457593] ACPI: PCI: Interrupt link LNKF configured for IRQ 101381lb # [ 0.454881] ACPI: PCI: Interrupt link LNKG configured for IRQ 111382worker2 # [ 0.462397] ACPI: PCI: Interrupt link LNKG configured for IRQ 111383worker1 # [ 0.458595] ACPI: PCI: Interrupt link LNKG configured for IRQ 111384client # [ 0.469660] iommu: Default domain type: Translated1385lb # [ 0.455876] ACPI: PCI: Interrupt link LNKH configured for IRQ 111386worker2 # [ 0.463395] ACPI: PCI: Interrupt link LNKH configured for IRQ 111387worker1 # [ 0.459592] ACPI: PCI: Interrupt link LNKH configured for IRQ 111388client # [ 0.470452] iommu: DMA domain TLB invalidation policy: lazy mode1389lb # [ 0.456817] ACPI: PCI: Interrupt link GSIA configured for IRQ 161390worker2 # [ 0.464333] ACPI: PCI: Interrupt link GSIA configured for IRQ 161391worker1 # [ 0.460531] ACPI: PCI: Interrupt link GSIA configured for IRQ 161392client # [ 0.471543] ACPI: bus type USB registered1393lb # [ 0.457757] ACPI: PCI: Interrupt link GSIB configured for IRQ 171394client # [ 0.472291] usbcore: registered new interface driver usbfs1395worker2 # [ 0.465272] ACPI: PCI: Interrupt link GSIB configured for IRQ 171396worker1 # [ 0.461473] ACPI: PCI: Interrupt link GSIB configured for IRQ 171397client # [ 0.472560] usbcore: registered new interface driver hub1398lb # [ 0.458497] ACPI: PCI: Interrupt link GSIC configured for IRQ 181399worker1 # [ 0.462451] ACPI: PCI: Interrupt link GSIC configured for IRQ 181400worker2 # [ 0.466260] ACPI: PCI: Interrupt link GSIC configured for IRQ 181401client # [ 0.473423] usbcore: registered new device driver usb1402lb # [ 0.459565] ACPI: PCI: Interrupt link GSID configured for IRQ 191403worker1 # [ 0.463259] ACPI: PCI: Interrupt link GSID configured for IRQ 191404worker2 # [ 0.467066] ACPI: PCI: Interrupt link GSID configured for IRQ 191405lb # [ 0.460526] ACPI: PCI: Interrupt link GSIE configured for IRQ 201406worker1 # [ 0.463514] ACPI: PCI: Interrupt link GSIE configured for IRQ 201407worker2 # [ 0.468056] ACPI: PCI: Interrupt link GSIE configured for IRQ 201408client # [ 0.475011] NetLabel: Initializing1409lb # [ 0.461504] ACPI: PCI: Interrupt link GSIF configured for IRQ 211410client # [ 0.475545] NetLabel: domain hash size = 1281411worker1 # [ 0.464481] ACPI: PCI: Interrupt link GSIF configured for IRQ 211412worker2 # [ 0.468308] ACPI: PCI: Interrupt link GSIF configured for IRQ 211413client # [ 0.476258] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1414lb # [ 0.461786] ACPI: PCI: Interrupt link GSIG configured for IRQ 221415worker1 # [ 0.465202] ACPI: PCI: Interrupt link GSIG configured for IRQ 221416worker2 # [ 0.469270] ACPI: PCI: Interrupt link GSIG configured for IRQ 221417client # [ 0.476579] NetLabel: unlabeled traffic allowed by default1418lb # [ 0.462741] ACPI: PCI: Interrupt link GSIH configured for IRQ 231419worker2 # [ 0.470027] ACPI: PCI: Interrupt link GSIH configured for IRQ 231420worker1 # [ 0.465508] ACPI: PCI: Interrupt link GSIH configured for IRQ 231421client # [ 0.477460] PCI: Using ACPI for IRQ routing1422lb # [ 0.464539] iommu: Default domain type: Translated1423worker1 # [ 0.467524] iommu: Default domain type: Translated1424worker2 # [ 0.471429] iommu: Default domain type: Translated1425lb # [ 0.464781] iommu: DMA domain TLB invalidation policy: lazy mode1426worker1 # [ 0.468321] iommu: DMA domain TLB invalidation policy: lazy mode1427worker2 # [ 0.472225] iommu: DMA domain TLB invalidation policy: lazy mode1428lb # [ 0.466010] ACPI: bus type USB registered1429worker1 # [ 0.469511] ACPI: bus type USB registered1430worker2 # [ 0.473299] ACPI: bus type USB registered1431lb # [ 0.466754] usbcore: registered new interface driver usbfs1432worker2 # [ 0.474050] usbcore: registered new interface driver usbfs1433worker1 # [ 0.470257] usbcore: registered new interface driver usbfs1434lb # [ 0.467462] usbcore: registered new interface driver hub1435worker2 # [ 0.474309] usbcore: registered new interface driver hub1436worker1 # [ 0.470516] usbcore: registered new interface driver hub1437lb # [ 0.467782] usbcore: registered new device driver usb1438worker2 # [ 0.475129] usbcore: registered new device driver usb1439worker1 # [ 0.471382] usbcore: registered new device driver usb1440lb # [ 0.469491] NetLabel: Initializing1441worker1 # [ 0.473007] NetLabel: Initializing1442worker2 # [ 0.476831] NetLabel: Initializing1443lb # [ 0.469780] NetLabel: domain hash size = 1281444worker2 # [ 0.477296] NetLabel: domain hash size = 1281445worker1 # [ 0.473497] NetLabel: domain hash size = 1281446lb # [ 0.470506] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1447worker2 # [ 0.477978] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1448worker1 # [ 0.474224] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1449lb # [ 0.470831] NetLabel: unlabeled traffic allowed by default1450worker2 # [ 0.478376] NetLabel: unlabeled traffic allowed by default1451worker1 # [ 0.474551] NetLabel: unlabeled traffic allowed by default1452lb # [ 0.471726] PCI: Using ACPI for IRQ routing1453worker2 # [ 0.479258] PCI: Using ACPI for IRQ routing1454worker1 # [ 0.475450] PCI: Using ACPI for IRQ routing1455client # [ 0.521360] pci 0000:00:01.0: vgaarb: setting as boot VGA device1456client # [ 0.521539] pci 0000:00:01.0: vgaarb: bridge control possible1457client # [ 0.521539] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1458client # [ 0.521548] vgaarb: loaded1459client # [ 0.522305] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01460client # [ 0.522543] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1461client # [ 0.526631] clocksource: Switched to clocksource kvm-clock1462lb # [ 0.515584] pci 0000:00:01.0: vgaarb: setting as boot VGA device1463lb # [ 0.515769] pci 0000:00:01.0: vgaarb: bridge control possible1464worker1 # [ 0.520402] pci 0000:00:01.0: vgaarb: setting as boot VGA device1465worker2 # [ 0.524110] pci 0000:00:01.0: vgaarb: setting as boot VGA device1466worker1 # [ 0.520492] pci 0000:00:01.0: vgaarb: bridge control possible1467lb # [ 0.515769] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1468client # [ 0.530159] VFS: Disk quotas dquot_6.6.01469lb # [ 0.515778] vgaarb: loaded1470worker2 # [ 0.524291] pci 0000:00:01.0: vgaarb: bridge control possible1471worker1 # [ 0.520492] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1472client # [ 0.530830] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1473lb # [ 0.516498] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01474worker1 # [ 0.520501] vgaarb: loaded1475worker2 # [ 0.524291] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1476client # [ 0.532176] pnp: PnP ACPI init1477worker2 # [ 0.524300] vgaarb: loaded1478lb # [ 0.516773] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1479worker1 # [ 0.521213] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01480client # [ 0.532934] ACPI: IRQ 4 override to edge(!), high(!)1481worker2 # [ 0.525001] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01482worker1 # [ 0.521497] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1483worker2 # [ 0.525296] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1484client # [ 0.533849] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1485client # [ 0.535396] pnp: PnP ACPI: found 5 devices1486lb # [ 0.519863] clocksource: Switched to clocksource kvm-clock1487worker1 # [ 0.524581] clocksource: Switched to clocksource kvm-clock1488lb # [ 0.521588] VFS: Disk quotas dquot_6.6.01489worker2 # [ 0.530385] clocksource: Switched to clocksource kvm-clock1490lb # [ 0.524120] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1491worker1 # [ 0.528436] VFS: Disk quotas dquot_6.6.01492lb # [ 0.525476] pnp: PnP ACPI init1493worker1 # [ 0.529187] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1494lb # [ 0.526305] ACPI: IRQ 4 override to edge(!), high(!)1495worker1 # [ 0.530640] pnp: PnP ACPI init1496worker2 # [ 0.533875] VFS: Disk quotas dquot_6.6.01497client # [ 0.543065] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1498worker1 # [ 0.531496] ACPI: IRQ 4 override to edge(!), high(!)1499lb # [ 0.527263] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1500worker2 # [ 0.534655] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1501client # [ 0.544799] clocksource: Switched to clocksource acpi_pm1502worker2 # [ 0.536018] pnp: PnP ACPI init1503lb # [ 0.529420] pnp: PnP ACPI: found 5 devices1504worker1 # [ 0.532491] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1505client # [ 0.545829] NET: Registered PF_INET protocol family1506worker2 # [ 0.536859] ACPI: IRQ 4 override to edge(!), high(!)1507worker1 # [ 0.534024] pnp: PnP ACPI: found 5 devices1508client # [ 0.547013] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1509worker2 # [ 0.537875] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1510worker2 # [ 0.539484] pnp: PnP ACPI: found 5 devices1511lb # [ 0.537328] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1512worker1 # [ 0.541604] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1513lb # [ 0.538791] clocksource: Switched to clocksource acpi_pm1514worker1 # [ 0.543063] clocksource: Switched to clocksource acpi_pm1515lb # [ 0.539936] NET: Registered PF_INET protocol family1516worker1 # [ 0.544068] NET: Registered PF_INET protocol family1517worker2 # [ 0.547133] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1518lb # [ 0.541143] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1519worker2 # [ 0.548596] clocksource: Switched to clocksource acpi_pm1520worker1 # [ 0.545344] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1521worker2 # [ 0.549558] NET: Registered PF_INET protocol family1522worker2 # [ 0.550811] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1523client # [ 0.564388] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1524client # [ 0.566073] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1525client # [ 0.567497] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1526client # [ 0.568916] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1527client # [ 0.570267] TCP: Hash tables configured (established 16384 bind 16384)1528client # [ 0.571470] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1529client # [ 0.572781] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1530client # [ 0.573908] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1531lb # [ 0.558134] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1532worker1 # [ 0.561721] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1533client # [ 0.575168] NET: Registered PF_UNIX/PF_LOCAL protocol family1534client # [ 0.576160] NET: Registered PF_XDP protocol family1535worker1 # [ 0.563311] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1536lb # [ 0.559755] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1537client # [ 0.577032] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1538worker2 # [ 0.567096] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1539worker1 # [ 0.564584] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1540lb # [ 0.561069] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1541client # [ 0.578012] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1542worker2 # [ 0.568581] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1543worker1 # [ 0.565853] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1544lb # [ 0.562377] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1545client # [ 0.579030] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1546worker2 # [ 0.569866] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1547worker1 # [ 0.567052] TCP: Hash tables configured (established 16384 bind 16384)1548lb # [ 0.563583] TCP: Hash tables configured (established 16384 bind 16384)1549client # [ 0.580090] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1550worker2 # [ 0.571164] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1551worker1 # [ 0.568143] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1552client # [ 0.581159] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1553lb # [ 0.564688] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1554worker2 # [ 0.572358] TCP: Hash tables configured (established 16384 bind 16384)1555worker1 # [ 0.569384] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1556lb # [ 0.565935] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1557client # [ 0.582224] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1558worker1 # [ 0.570433] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1559worker2 # [ 0.573508] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1560lb # [ 0.566996] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1561client # [ 0.584015] ACPI: \_SB_.GSIA: Enabled at IRQ 161562worker1 # [ 0.571627] NET: Registered PF_UNIX/PF_LOCAL protocol family1563lb # [ 0.568178] NET: Registered PF_UNIX/PF_LOCAL protocol family1564worker2 # [ 0.574821] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1565worker1 # [ 0.572573] NET: Registered PF_XDP protocol family1566lb # [ 0.569126] NET: Registered PF_XDP protocol family1567worker2 # [ 0.575948] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1568lb # [ 0.569952] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1569worker1 # [ 0.573415] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1570client # [ 0.586861] ACPI: \_SB_.GSIB: Enabled at IRQ 171571worker2 # [ 0.577132] NET: Registered PF_UNIX/PF_LOCAL protocol family1572lb # [ 0.570946] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1573worker1 # [ 0.574444] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1574worker2 # [ 0.578056] NET: Registered PF_XDP protocol family1575worker1 # [ 0.575455] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1576lb # [ 0.571951] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1577worker2 # [ 0.578910] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1578client # [ 0.588938] ACPI: \_SB_.GSIC: Enabled at IRQ 181579lb # [ 0.573024] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1580worker1 # [ 0.576537] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1581worker2 # [ 0.579881] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1582lb # [ 0.574063] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1583worker1 # [ 0.577595] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1584client # [ 0.590816] ACPI: \_SB_.GSID: Enabled at IRQ 191585worker2 # [ 0.580866] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1586worker1 # [ 0.578638] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1587lb # [ 0.575111] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1588worker2 # [ 0.581927] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1589client # [ 0.592499] PCI: CLS 0 bytes, default 641590worker2 # [ 0.583001] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1591worker1 # [ 0.580304] ACPI: \_SB_.GSIA: Enabled at IRQ 161592lb # [ 0.576910] ACPI: \_SB_.GSIA: Enabled at IRQ 161593worker2 # [ 0.584131] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1594client # [ 0.593438] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1595client # [ 0.595225] Trying to unpack rootfs image as initramfs...1596worker1 # [ 0.582556] ACPI: \_SB_.GSIB: Enabled at IRQ 171597worker2 # [ 0.585910] ACPI: \_SB_.GSIA: Enabled at IRQ 161598lb # [ 0.579855] ACPI: \_SB_.GSIB: Enabled at IRQ 171599worker1 # [ 0.584891] ACPI: \_SB_.GSIC: Enabled at IRQ 181600worker2 # [ 0.588524] ACPI: \_SB_.GSIB: Enabled at IRQ 171601lb # [ 0.581960] ACPI: \_SB_.GSIC: Enabled at IRQ 181602worker1 # [ 0.586998] ACPI: \_SB_.GSID: Enabled at IRQ 191603worker2 # [ 0.590451] ACPI: \_SB_.GSIC: Enabled at IRQ 181604lb # [ 0.583916] ACPI: \_SB_.GSID: Enabled at IRQ 191605worker1 # [ 0.588740] PCI: CLS 0 bytes, default 641606lb # [ 0.585599] PCI: CLS 0 bytes, default 641607worker2 # [ 0.592355] ACPI: \_SB_.GSID: Enabled at IRQ 191608worker1 # [ 0.589710] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1609lb # [ 0.586523] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1610worker2 # [ 0.594086] PCI: CLS 0 bytes, default 641611worker1 # [ 0.591443] Trying to unpack rootfs image as initramfs...1612lb # [ 0.588251] Trying to unpack rootfs image as initramfs...1613worker2 # [ 0.594981] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1614worker2 # [ 0.596804] Trying to unpack rootfs image as initramfs...1615client # [ 0.640690] Initialise system trusted keyrings1616client # [ 0.644277] workingset: timestamp_bits=40 max_order=19 bucket_order=01617worker1 # [ 0.635344] Initialise system trusted keyrings1618lb # [ 0.633117] Initialise system trusted keyrings1619worker1 # [ 0.638361] workingset: timestamp_bits=40 max_order=19 bucket_order=01620worker2 # [ 0.641832] Initialise system trusted keyrings1621lb # [ 0.635913] workingset: timestamp_bits=40 max_order=19 bucket_order=01622worker2 # [ 0.644800] workingset: timestamp_bits=40 max_order=19 bucket_order=01623client # [ 0.665530] Key type asymmetric registered1624client # [ 0.668235] Asymmetric key parser 'x509' registered1625client # [ 0.669145] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1626client # [ 0.672346] io scheduler mq-deadline registered1627client # [ 0.673147] io scheduler kyber registered1628worker1 # [ 0.661253] Key type asymmetric registered1629worker1 # [ 0.662005] Asymmetric key parser 'x509' registered1630worker2 # [ 0.666085] Key type asymmetric registered1631client # [ 0.675859] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1632lb # [ 0.659857] Key type asymmetric registered1633worker1 # [ 0.662902] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1634lb # [ 0.660587] Asymmetric key parser 'x509' registered1635client # [ 0.677116] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1636lb # [ 0.661474] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1637worker2 # [ 0.668755] Asymmetric key parser 'x509' registered1638worker1 # [ 0.666310] io scheduler mq-deadline registered1639worker2 # [ 0.669630] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1640worker1 # [ 0.667113] io scheduler kyber registered1641lb # [ 0.664861] io scheduler mq-deadline registered1642lb # [ 0.665647] io scheduler kyber registered1643worker2 # [ 0.672847] io scheduler mq-deadline registered1644client # [ 0.683087] Linux agpgart interface v0.1031645worker2 # [ 0.673666] io scheduler kyber registered1646worker1 # [ 0.670822] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1647client # [ 0.683930] ACPI: bus type drm_connector registered1648worker1 # [ 0.672095] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1649lb # [ 0.669919] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1650worker2 # [ 0.677326] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1651lb # [ 0.671282] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1652client # [ 0.688376] usbcore: registered new interface driver usbserial_generic1653worker1 # [ 0.676010] Linux agpgart interface v0.1031654worker2 # [ 0.678692] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1655worker1 # [ 0.676754] ACPI: bus type drm_connector registered1656client # [ 0.689465] usbserial: USB Serial support registered for generic1657worker2 # [ 0.682580] Linux agpgart interface v0.1031658lb # [ 0.676603] Linux agpgart interface v0.1031659client # [ 0.692233] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1660worker2 # [ 0.683370] ACPI: bus type drm_connector registered1661client # [ 0.692233] Please enable it if your BIOS has the CPPC option.1662lb # [ 0.677408] ACPI: bus type drm_connector registered1663worker1 # [ 0.681728] usbcore: registered new interface driver usbserial_generic1664client # [ 0.694459] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1665worker1 # [ 0.682844] usbserial: USB Serial support registered for generic1666client # [ 0.697384] drop_monitor: Initializing network drop monitor service1667worker2 # [ 0.688159] usbcore: registered new interface driver usbserial_generic1668lb # [ 0.681892] usbcore: registered new interface driver usbserial_generic1669client # [ 0.698601] NET: Registered PF_INET6 protocol family1670worker2 # [ 0.689256] usbserial: USB Serial support registered for generic1671lb # [ 0.683007] usbserial: USB Serial support registered for generic1672worker1 # [ 0.686212] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1673worker1 # [ 0.686212] Please enable it if your BIOS has the CPPC option.1674worker2 # [ 0.690218] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1675worker2 # [ 0.690218] Please enable it if your BIOS has the CPPC option.1676worker1 # [ 0.688458] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1677worker2 # [ 0.692475] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1678lb # [ 0.685731] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1679lb # [ 0.685731] Please enable it if your BIOS has the CPPC option.1680client # [ 0.703782] Segment Routing with IPv61681lb # [ 0.687977] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1682worker1 # [ 0.692360] drop_monitor: Initializing network drop monitor service1683client # [ 0.706292] In-situ OAM (IOAM) with IPv61684worker1 # [ 0.693575] NET: Registered PF_INET6 protocol family1685client # [ 0.707443] IPI shorthand broadcast: enabled1686lb # [ 0.690891] drop_monitor: Initializing network drop monitor service1687worker2 # [ 0.697881] drop_monitor: Initializing network drop monitor service1688lb # [ 0.692092] NET: Registered PF_INET6 protocol family1689worker2 # [ 0.699037] NET: Registered PF_INET6 protocol family1690worker1 # [ 0.696665] Segment Routing with IPv61691worker1 # [ 0.697367] In-situ OAM (IOAM) with IPv61692worker2 # [ 0.702221] Segment Routing with IPv61693worker1 # [ 0.699493] IPI shorthand broadcast: enabled1694lb # [ 0.697230] Segment Routing with IPv61695lb # [ 0.697926] In-situ OAM (IOAM) with IPv61696worker2 # [ 0.705778] In-situ OAM (IOAM) with IPv61697worker2 # [ 0.706907] IPI shorthand broadcast: enabled1698lb # [ 0.701039] IPI shorthand broadcast: enabled1699client # [ 0.716768] sched_clock: Marking stable (582014550, 134199039)->(791757681, -75544092)1700worker1 # [ 0.707869] sched_clock: Marking stable (574014539, 133172685)->(782748726, -75561502)1701client # [ 0.721384] registered taskstats version 11702client # [ 0.722402] Loading compiled-in X.509 certificates1703worker1 # [ 0.711391] registered taskstats version 11704worker1 # [ 0.712415] Loading compiled-in X.509 certificates1705worker2 # [ 0.715217] sched_clock: Marking stable (582014645, 132720716)->(786523767, -71788406)1706lb # [ 0.709445] sched_clock: Marking stable (575014719, 133698351)->(781070355, -72357285)1707lb # [ 0.712944] registered taskstats version 11708worker2 # [ 0.719853] registered taskstats version 11709lb # [ 0.713965] Loading compiled-in X.509 certificates1710worker2 # [ 0.720848] Loading compiled-in X.509 certificates1711client # [ 0.741236] Demotion targets for Node 0: null1712client # [ 0.742180] Key type .fscrypt registered1713worker1 # [ 0.730207] Demotion targets for Node 0: null1714client # [ 0.745220] Key type fscrypt-provisioning registered1715worker1 # [ 0.733233] Key type .fscrypt registered1716client # [ 0.746239] ima: No TPM chip found, activating TPM-bypass!1717worker1 # [ 0.733969] Key type fscrypt-provisioning registered1718client # [ 0.747142] ima: Allocated hash algorithm: sha11719worker1 # [ 0.734940] ima: No TPM chip found, activating TPM-bypass!1720lb # [ 0.732726] Demotion targets for Node 0: null1721client # [ 0.749256] ima: No architecture policies found1722worker2 # [ 0.739751] Demotion targets for Node 0: null1723lb # [ 0.733648] Key type .fscrypt registered1724worker2 # [ 0.740660] Key type .fscrypt registered1725worker1 # [ 0.739198] ima: Allocated hash algorithm: sha11726client # [ 0.752228] PM: Magic number: 10:823:9741727worker1 # [ 0.740021] ima: No architecture policies found1728lb # [ 0.736720] Key type fscrypt-provisioning registered1729worker2 # [ 0.743740] Key type fscrypt-provisioning registered1730worker1 # [ 0.741008] PM: Magic number: 10:823:9741731client # [ 0.753976] RAS: Correctable Errors collector initialized.1732lb # [ 0.737687] ima: No TPM chip found, activating TPM-bypass!1733worker2 # [ 0.744744] ima: No TPM chip found, activating TPM-bypass!1734lb # [ 0.738609] ima: Allocated hash algorithm: sha11735worker2 # [ 0.745650] ima: Allocated hash algorithm: sha11736worker2 # [ 0.746494] ima: No architecture policies found1737lb # [ 0.740753] ima: No architecture policies found1738worker1 # [ 0.744627] RAS: Correctable Errors collector initialized.1739worker2 # [ 0.748748] PM: Magic number: 10:823:9741740lb # [ 0.743725] PM: Magic number: 10:823:9741741worker2 # [ 0.750410] RAS: Correctable Errors collector initialized.1742lb # [ 0.745616] RAS: Correctable Errors collector initialized.1743client # [ 0.762928] clk: Disabling unused clocks1744client # [ 0.765241] PM: genpd: Disabling unused power domains1745worker1 # [ 0.753269] clk: Disabling unused clocks1746worker1 # [ 0.754001] PM: genpd: Disabling unused power domains1747worker2 # [ 0.760447] clk: Disabling unused clocks1748lb # [ 0.754593] clk: Disabling unused clocks1749worker2 # [ 0.762752] PM: genpd: Disabling unused power domains1750lb # [ 0.758743] PM: genpd: Disabling unused power domains1751worker1 # [ 0.918494] Freeing initrd memory: 29056K1752client # [ 0.933535] Freeing initrd memory: 29056K1753worker2 # [ 0.924203] Freeing initrd memory: 29060K1754worker1 # [ 0.921686] Freeing unused decrypted memory: 2028K1755client # [ 0.936928] Freeing unused decrypted memory: 2028K1756worker2 # [ 0.927390] Freeing unused decrypted memory: 2028K1757worker1 # [ 0.924275] Freeing unused kernel image (initmem) memory: 3652K1758worker1 # [ 0.925361] Write protecting the kernel read-only data: 32768k1759client # [ 0.939742] Freeing unused kernel image (initmem) memory: 3652K1760worker2 # [ 0.929989] Freeing unused kernel image (initmem) memory: 3652K1761worker1 # [ 0.927252] Freeing unused kernel image (text/rodata gap) memory: 1184K1762client # [ 0.940801] Write protecting the kernel read-only data: 32768k1763worker2 # [ 0.931139] Write protecting the kernel read-only data: 32768k1764worker1 # [ 0.928741] Freeing unused kernel image (rodata/data gap) memory: 720K1765client # [ 0.942730] Freeing unused kernel image (text/rodata gap) memory: 1184K1766worker2 # [ 0.933017] Freeing unused kernel image (text/rodata gap) memory: 1184K1767worker2 # [ 0.934542] Freeing unused kernel image (rodata/data gap) memory: 720K1768client # [ 0.944272] Freeing unused kernel image (rodata/data gap) memory: 720K1769lb # [ 0.929247] Freeing initrd memory: 29064K1770lb # [ 0.932545] Freeing unused decrypted memory: 2028K1771lb # [ 0.935447] Freeing unused kernel image (initmem) memory: 3652K1772lb # [ 0.936586] Write protecting the kernel read-only data: 32768k1773lb # [ 0.938533] Freeing unused kernel image (text/rodata gap) memory: 1184K1774lb # [ 0.940103] Freeing unused kernel image (rodata/data gap) memory: 720K1775worker1 # [ 0.979768] x86/mm: Checked W+X mappings: passed, no W+X pages found.1776worker1 # [ 0.980911] Run /init as init process1777client # [ 0.994970] x86/mm: Checked W+X mappings: passed, no W+X pages found.1778worker2 # [ 0.985577] x86/mm: Checked W+X mappings: passed, no W+X pages found.1779client # [ 0.996078] Run /init as init process1780worker2 # [ 0.986689] Run /init as init process1781worker1 # [ 0.991857] systemd[1]: Inserted module 'autofs4'1782worker2 # [ 0.997552] systemd[1]: Inserted module 'autofs4'1783lb # [ 0.991014] x86/mm: Checked W+X mappings: passed, no W+X pages found.1784client # [ 1.007824] systemd[1]: Inserted module 'autofs4'1785lb # [ 0.992077] Run /init as init process1786lb # [ 1.004008] systemd[1]: Inserted module 'autofs4'1787worker1 # [ 1.012046] fuse: init (API version 7.45)1788worker2 # [ 1.017813] fuse: init (API version 7.45)1789client # [ 1.028373] fuse: init (API version 7.45)1790worker1 # [ 1.019548] ACPI: \_SB_.GSIG: Enabled at IRQ 221791worker1 # [ 1.021940] ACPI: \_SB_.GSIH: Enabled at IRQ 231792worker2 # [ 1.025246] ACPI: \_SB_.GSIG: Enabled at IRQ 221793client # [ 1.036025] ACPI: \_SB_.GSIG: Enabled at IRQ 221794worker2 # [ 1.027959] ACPI: \_SB_.GSIH: Enabled at IRQ 231795worker1 # [ 1.024957] ACPI: \_SB_.GSIE: Enabled at IRQ 201796client # [ 1.038807] ACPI: \_SB_.GSIH: Enabled at IRQ 231797worker1 # [ 1.026888] ACPI: \_SB_.GSIF: Enabled at IRQ 211798lb # [ 1.024443] fuse: init (API version 7.45)1799worker2 # [ 1.031339] ACPI: \_SB_.GSIE: Enabled at IRQ 201800client # [ 1.042274] ACPI: \_SB_.GSIE: Enabled at IRQ 201801worker2 # [ 1.033594] ACPI: \_SB_.GSIF: Enabled at IRQ 211802client # [ 1.044562] ACPI: \_SB_.GSIF: Enabled at IRQ 211803worker1 # [ 1.032066] virtiofs virtio5: discovered new tag: nix-store1804worker1 # [ 1.033649] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1805lb # [ 1.032053] ACPI: \_SB_.GSIG: Enabled at IRQ 221806worker2 # [ 1.039220] virtiofs virtio5: discovered new tag: nix-store1807client # [ 1.050294] virtiofs virtio5: discovered new tag: nix-store1808worker2 # [ 1.040758] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1809lb # [ 1.034896] ACPI: \_SB_.GSIH: Enabled at IRQ 231810client # [ 1.051916] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1811worker1 # [ 1.040573] virtiofs virtio6: discovered new tag: shared1812lb # [ 1.038163] ACPI: \_SB_.GSIE: Enabled at IRQ 201813worker1 # [ 1.042278] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1814lb # [ 1.040526] ACPI: \_SB_.GSIF: Enabled at IRQ 211815worker1 # [ 1.045528] virtiofs virtio7: discovered new tag: xchg1816client # [ 1.058977] virtiofs virtio6: discovered new tag: shared1817worker2 # [ 1.050226] virtiofs virtio6: discovered new tag: shared1818worker1 # [ 1.046988] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1819client # [ 1.060561] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1820worker2 # [ 1.051841] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1821lb # [ 1.046243] virtiofs virtio5: discovered new tag: nix-store1822client # [ 1.064170] virtiofs virtio7: discovered new tag: xchg1823lb # [ 1.047900] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1824worker2 # [ 1.055138] virtiofs virtio7: discovered new tag: xchg1825client # [ 1.065671] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1826worker2 # [ 1.056662] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1827lb # [ 1.055936] virtiofs virtio6: discovered new tag: shared1828lb # [ 1.057554] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1829lb # [ 1.061306] virtiofs virtio7: discovered new tag: xchg1830lb # [ 1.062853] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1831worker1 # [ 1.067989] systemd[1]: Successfully made /usr/ read-only.1832client # [ 1.088276] systemd[1]: Successfully made /usr/ read-only.1833worker2 # [ 1.078850] systemd[1]: Successfully made /usr/ read-only.1834lb # [ 1.084604] systemd[1]: Successfully made /usr/ read-only.1835client # [ 1.423876] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1836worker1 # [ 1.404618] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1837client # [ 1.429067] systemd[1]: Detected virtualization kvm.1838client # [ 1.429942] systemd[1]: Detected architecture x86-64.1839worker1 # [ 1.416669] systemd[1]: Detected virtualization kvm.1840client # [ 1.430795] systemd[1]: Running in initrd.1841client # [ 1.431934] systemd[1]: Initializing machine ID from random generator.1842worker1 # [ 1.418907] systemd[1]: Detected architecture x86-64.1843client # [ 1.433175] systemd[1]: Hostname set to <client>.1844worker1 # [ 1.421118] systemd[1]: Running in initrd.1845worker2 # [ 1.415628] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1846worker1 # [ 1.423660] systemd[1]: Initializing machine ID from random generator.1847worker2 # [ 1.427671] systemd[1]: Detected virtualization kvm.1848worker1 # [ 1.426859] systemd[1]: Hostname set to <worker1>.1849worker2 # [ 1.429868] systemd[1]: Detected architecture x86-64.1850worker2 # [ 1.432227] systemd[1]: Running in initrd.1851worker2 # [ 1.434911] systemd[1]: Initializing machine ID from random generator.1852lb # [ 1.421227] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1853worker2 # [ 1.437881] systemd[1]: Hostname set to <worker2>.1854lb # [ 1.433238] systemd[1]: Detected virtualization kvm.1855lb # [ 1.435414] systemd[1]: Detected architecture x86-64.1856lb # [ 1.437573] systemd[1]: Running in initrd.1857lb # [ 1.440086] systemd[1]: Initializing machine ID from random generator.1858lb # [ 1.442947] systemd[1]: Hostname set to <lb>.1859lb # [ 1.546644] systemd[1]: bpf-restrict-fs: LSM BPF program attached1860lb # [ 1.584847] systemd[1]: Queued start job for default target Initrd Default Target.1861lb # [ 1.588030] systemd[1]: Created slice Slice /system/modprobe.1862lb # [ 1.589206] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1863lb # [ 1.590663] systemd[1]: Expecting device /dev/disk/by-label/nixos...1864lb # [ 1.591823] systemd[1]: Reached target Path Units.1865lb # [ 1.592738] systemd[1]: Reached target Slice Units.1866lb # [ 1.593630] systemd[1]: Reached target Swaps.1867lb # [ 1.594508] systemd[1]: Reached target Timer Units.1868lb # [ 1.595523] systemd[1]: Listening on D-Bus System Message Bus Socket.1869lb # [ 1.596799] systemd[1]: Listening on Journal Socket (/dev/log).1870lb # [ 1.597993] systemd[1]: Listening on Journal Sockets.1871lb # [ 1.599033] systemd[1]: Listening on udev Control Socket.1872lb # [ 1.600091] systemd[1]: Listening on udev Kernel Socket.1873lb # [ 1.601094] systemd[1]: Reached target Socket Units.1874lb # [ 1.602877] systemd[1]: Starting Create List of Static Device Nodes...1875lb # [ 1.606785] systemd[1]: Starting Load Kernel Module configfs...1876lb # [ 1.615834] systemd[1]: Starting Journal Service...1877client # [ 1.642451] systemd[1]: bpf-restrict-fs: LSM BPF program attached1878lb # [ 1.641309] systemd[1]: Starting Load Kernel Modules...1879lb # [ 1.646288] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1880worker1 # [ 1.649687] systemd[1]: bpf-restrict-fs: LSM BPF program attached1881lb # [ 1.660811] systemd[1]: Starting Coldplug All udev Devices...1882worker2 # [ 1.666406] systemd[1]: bpf-restrict-fs: LSM BPF program attached1883lb # [ 1.669490] systemd-journald[66]: Collecting audit messages is disabled.1884lb # [ 1.676673] systemd[1]: Finished Create List of Static Device Nodes.1885client # [ 1.694945] systemd[1]: Queued start job for default target Initrd Default Target.1886lb # [ 1.680359] systemd[1]: modprobe@configfs.service: Deactivated successfully.1887client # [ 1.699518] systemd[1]: Created slice Slice /system/modprobe.1888client # [ 1.700796] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1889client # [ 1.702287] systemd[1]: Expecting device /dev/disk/by-label/nixos...1890lb # [ 1.686491] systemd[1]: Finished Load Kernel Module configfs.1891client # [ 1.703430] systemd[1]: Reached target Path Units.1892client # [ 1.704346] systemd[1]: Reached target Slice Units.1893client # [ 1.705266] systemd[1]: Reached target Swaps.1894client # [ 1.706085] systemd[1]: Reached target Timer Units.1895client # [ 1.707097] systemd[1]: Listening on D-Bus System Message Bus Socket.1896client # [ 1.708367] systemd[1]: Listening on Journal Socket (/dev/log).1897client # [ 1.709593] systemd[1]: Listening on Journal Sockets.1898lb # [ 1.693145] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1899client # [ 1.710627] systemd[1]: Listening on udev Control Socket.1900client # [ 1.711671] systemd[1]: Listening on udev Kernel Socket.1901client # [ 1.712642] systemd[1]: Reached target Socket Units.1902client # [ 1.714484] systemd[1]: Starting Create List of Static Device Nodes...1903worker1 # [ 1.704163] systemd[1]: Queued start job for default target Initrd Default Target.1904client # [ 1.718288] systemd[1]: Starting Load Kernel Module configfs...1905lb # [ 1.704804] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1906worker1 # [ 1.708639] systemd[1]: Created slice Slice /system/modprobe.1907worker1 # [ 1.709805] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1908worker1 # [ 1.711260] systemd[1]: Expecting device /dev/disk/by-label/nixos...1909worker1 # [ 1.712468] systemd[1]: Reached target Path Units.1910lb # [ 1.708307] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1911worker1 # [ 1.713464] systemd[1]: Reached target Slice Units.1912worker1 # [ 1.714385] systemd[1]: Reached target Swaps.1913worker1 # [ 1.715273] systemd[1]: Reached target Timer Units.1914worker1 # [ 1.716293] systemd[1]: Listening on D-Bus System Message Bus Socket.1915worker1 # [ 1.717586] systemd[1]: Listening on Journal Socket (/dev/log).1916worker1 # [ 1.718755] systemd[1]: Listening on Journal Sockets.1917worker2 # [ 1.721545] systemd[1]: Queued start job for default target Initrd Default Target.1918worker1 # [ 1.719752] systemd[1]: Listening on udev Control Socket.1919worker1 # [ 1.720814] systemd[1]: Listening on udev Kernel Socket.1920worker1 # [ 1.721780] systemd[1]: Reached target Socket Units.1921worker2 # [ 1.726139] systemd[1]: Created slice Slice /system/modprobe.1922worker1 # [ 1.723714] systemd[1]: Starting Create List of Static Device Nodes...1923worker2 # [ 1.727365] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1924worker2 # [ 1.728857] systemd[1]: Expecting device /dev/disk/by-label/nixos...1925client # [ 1.727282] systemd[1]: Starting Journal Service...1926lb # [ 1.721918] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1927worker2 # [ 1.729948] systemd[1]: Reached target Path Units.1928worker2 # [ 1.730895] systemd[1]: Reached target Slice Units.1929worker1 # [ 1.728324] systemd[1]: Starting Load Kernel Module configfs...1930worker2 # [ 1.731803] systemd[1]: Reached target Swaps.1931worker2 # [ 1.732616] systemd[1]: Reached target Timer Units.1932worker2 # [ 1.733633] systemd[1]: Listening on D-Bus System Message Bus Socket.1933worker2 # [ 1.734902] systemd[1]: Listening on Journal Socket (/dev/log).1934worker2 # [ 1.736086] systemd[1]: Listening on Journal Sockets.1935worker2 # [ 1.737122] systemd[1]: Listening on udev Control Socket.1936worker2 # [ 1.738230] systemd[1]: Listening on udev Kernel Socket.1937worker2 # [ 1.739196] systemd[1]: Reached target Socket Units.1938worker2 # [ 1.741048] systemd[1]: Starting Create List of Static Device Nodes...1939client # [ 1.753432] systemd[1]: Starting Load Kernel Modules...1940worker2 # [ 1.744808] systemd[1]: Starting Load Kernel Module configfs...1941client # [ 1.757327] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1942lb # [ 1.744259] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1943worker1 # [ 1.737883] systemd[1]: Starting Journal Service...1944client # [ 1.766460] systemd[1]: Starting Coldplug All udev Devices...1945lb # [ 1.752921] systemd[1]: Starting Create Static Device Nodes in /dev...1946worker2 # [ 1.755440] systemd[1]: Starting Journal Service...1947worker1 # [ 1.763825] systemd[1]: Starting Load Kernel Modules...1948client # [ 1.778550] systemd-journald[66]: Collecting audit messages is disabled.1949client # [ 1.779885] systemd[1]: Finished Create List of Static Device Nodes.1950lb # [ 1.764250] systemd[1]: Finished Load Kernel Modules.1951worker1 # [ 1.768334] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1952client # [ 1.785831] systemd[1]: modprobe@configfs.service: Deactivated successfully.1953lb # [ 1.772933] systemd[1]: Starting Apply Kernel Variables...1954worker2 # [ 1.780347] systemd[1]: Starting Load Kernel Modules...1955worker1 # [ 1.778429] systemd[1]: Starting Coldplug All udev Devices...1956client # [ 1.794594] systemd[1]: Finished Load Kernel Module configfs.1957worker2 # [ 1.784871] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1958client # [ 1.800645] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1959worker1 # [ 1.789611] systemd-journald[66]: Collecting audit messages is disabled.1960worker2 # [ 1.792874] systemd[1]: Starting Coldplug All udev Devices...1961worker1 # [ 1.792275] systemd[1]: Finished Create List of Static Device Nodes.1962lb # [ 1.791022] systemd[1]: Started Journal Service.1963worker1 # [ 1.797980] systemd[1]: modprobe@configfs.service: Deactivated successfully.1964lb # [ 1.656851] systemd-modules-load[67]: Inserted module 'dm_mod'1965client # [ 1.811306] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1966lb # [ 1.662583] systemd-modules-load[67]: Inserted module 'virtio_balloon'1967lb # [ 1.663606] systemd-modules-load[67]: Inserted module 'virtio_gpu'1968worker2 # [ 1.805314] systemd-journald[66]: Collecting audit messages is disabled.1969lb # [ 1.666267] systemd[1]: Finished Create Static Device Nodes in /dev.1970worker2 # [ 1.807102] systemd[1]: Finished Create List of Static Device Nodes.1971worker1 # [ 1.804617] systemd[1]: Finished Load Kernel Module configfs.1972lb # [ 1.668441] systemd[1]: Reached target Preparation for Local File Systems.1973lb # [ 1.670362] systemd[1]: Reached target Local File Systems.1974worker2 # [ 1.810702] systemd[1]: modprobe@configfs.service: Deactivated successfully.1975client # [ 1.822598] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1976worker1 # [ 1.810615] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1977lb # [ 1.675070] systemd[1]: Starting Create System Files and Directories...1978worker2 # [ 1.819139] systemd[1]: Finished Load Kernel Module configfs.1979client # [ 1.832274] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1980lb # [ 1.682938] systemd[1]: Starting Rule-based Manager for Device Events and Files...1981worker1 # [ 1.822416] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1982worker2 # [ 1.825231] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1983lb # [ 1.690032] systemd[1]: Finished Apply Kernel Variables.1984worker1 # [ 1.833502] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1985worker2 # [ 1.838815] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1986client # [ 1.851853] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1987worker1 # [ 1.841445] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1988worker2 # [ 1.849881] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1989lb # [ 1.711028] systemd[1]: Finished Create System Files and Directories.1990client # [ 1.861414] systemd[1]: Starting Create Static Device Nodes in /dev...1991worker2 # [ 1.858776] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1992client # [ 1.873289] systemd[1]: Finished Load Kernel Modules.1993client # [ 1.881429] systemd[1]: Starting Apply Kernel Variables...1994worker1 # [ 1.871811] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1995lb # [ 1.738612] systemd-udevd[76]: Using default interface naming scheme 'v261'.1996worker2 # [ 1.880337] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1997worker1 # [ 1.880422] systemd[1]: Starting Create Static Device Nodes in /dev...1998worker2 # [ 1.888960] systemd[1]: Starting Create Static Device Nodes in /dev...1999client # [ 1.768227] systemd-modules-load[67]: Inserted module 'dm_mod'[ 1.903645] systemd[1]: Started Journal Service.2000worker1 # [ 1.891746] systemd[1]: Finished Load Kernel Modules.
loading 2,711 hidden lines…
4712lb # [ 59.154499] niks3-server[991]: 2026/09/23 06:59:58 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.ls4713lb # [ 59.160189] niks3-server[991]: 2026/09/23 06:59:58 INFO Registered completed upload object_key=log/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv4714lb # [ 59.169493] niks3-server[991]: 2026/09/23 06:59:58 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4715lb # [ 59.178106] niks3-server[991]: 2026/09/23 06:59:58 INFO Completed upload id=124716lb # [ 59.179789] niks3-server[991]: 2026/09/23 06:59:58 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.narinfo4717worker2 # [ 59.186057] nix-grpc-daemon[880]: time=2026-09-23T06:59:58.371Z level=INFO msg="Upload complete. (189ms)"4718worker2 # [ 59.192848] nix-grpc-daemon[783]: ts=2026-09-23T06:59:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=1 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv assign_id=6 outputs=14719worker1 # [ 59.195825] nix-grpc-daemon[783]: ts=2026-09-23T06:59:57Z level=debug event=want client=4 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv cached=04720worker1 # [ 59.200251] nix-grpc-daemon[783]: ts=2026-09-23T06:59:57Z level=debug event=assigned drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv worker=192.168.1.4:50051 clients=14721worker2 # [ 59.207681] nix-daemon[855]: reaped child process 928, status = succeeded4722worker2 # [ 59.244080] nix-grpc-daemon[783]: ts=2026-09-23T06:59:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408824723worker2 # [ 59.256679] nix-grpc-daemon[783]: ts=2026-09-23T06:59:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408824724worker2 # [ 59.269309] nix-grpc-daemon[880]: time=2026-09-23T06:59:58.454Z level=INFO msg="All 2 paths already cached"4725client # copying 1 paths to 'lb:50051'...4726worker2 # [ 59.278459] nix-grpc-daemon[783]: ts=2026-09-23T06:59:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:408824727client # copying path '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' to 'lb:50051'...4728worker2 # [ 59.303175] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4729worker2 # [ 59.403453] nix-daemon[855]: accepted connection from pid 953, user nix-grpc-daemon (trusted)4730worker2 # [ 59.415481] nix-daemon[855]: reaped child process 956, status = succeeded4731lb # [ 59.415854] niks3-server[991]: 2026/09/23 06:59:58 INFO Received uploads request method=POST path=/api/pending_closures4732worker2 # [ 59.431470] nix-grpc-daemon[880]: time=2026-09-23T06:59:58.617Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4733worker2 # [ 59.433424] nix-grpc-daemon[880]: time=2026-09-23T06:59:58.618Z level=INFO msg="Uploading cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv (752B)"4734lb # [ 59.448774] niks3-server[991]: 2026/09/23 06:59:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4735lb # [ 59.454263] niks3-server[991]: 2026/09/23 06:59:58 INFO Signed narinfos id=13 count=14736worker2 # [ 59.465153] nix-grpc-daemon[880]: time=2026-09-23T06:59:58.650Z level=INFO msg="Uploading 1 narinfos"4737lb # [ 59.458078] niks3-server[991]: 2026/09/23 06:59:58 INFO Registered completed upload object_key=nar/0w21abkvlzg18nqkl2x4zxs7d0376kbld1n0fkr68vcpqpq1g5zg.nar.zst4738lb # [ 59.472253] niks3-server[991]: 2026/09/23 06:59:58 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4739lb # [ 59.478967] niks3-server[991]: 2026/09/23 06:59:58 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.ls4740lb # [ 59.483538] niks3-server[991]: 2026/09/23 06:59:58 INFO Completed upload id=134741worker2 # [ 59.491218] nix-grpc-daemon[880]: time=2026-09-23T06:59:58.676Z level=INFO msg="Upload complete. (174ms)"4742lb # [ 59.486814] niks3-server[991]: 2026/09/23 06:59:58 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.narinfo4743worker2 # [ 59.495157] nix-grpc-daemon[783]: ts=2026-09-23T06:59:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=0 paths=1 nar_bytes_in=7524744worker2 # [ 59.501381] nix-grpc-daemon[783]: ts=2026-09-23T06:59:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408824745worker2 # [ 59.511202] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4746worker2 # [ 59.514739] nix-daemon[855]: reaped child process 951, status = succeeded4747worker2 # [ 59.519898] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4748client # building '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' on lb:50051...4749client # farm-top> worker2: building cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv4750client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4751client # farm-top> LOG-farm-top-sig4752worker2 # [ 60.735708] nix-daemon[855]: reaped child process 960, status = succeeded4753worker2 # [ 60.811090] nix-daemon[855]: accepted connection from pid 976, user nix-grpc-daemon (trusted)4754worker2 # [ 60.823225] nix-daemon[855]: reaped child process 979, status = succeeded4755lb # [ 60.822296] niks3-server[991]: 2026/09/23 07:00:00 INFO Received uploads request method=POST path=/api/pending_closures4756worker2 # [ 60.901114] nix-grpc-daemon[880]: time=2026-09-23T07:00:00.086Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4757worker2 # [ 60.905936] nix-grpc-daemon[880]: time=2026-09-23T07:00:00.089Z level=INFO msg="Uploading dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top (232B)"4758lb # [ 60.928923] niks3-server[991]: 2026/09/23 07:00:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4759lb # [ 60.937701] niks3-server[991]: 2026/09/23 07:00:00 INFO Signed narinfos id=14 count=14760worker2 # [ 60.949387] nix-grpc-daemon[880]: time=2026-09-23T07:00:00.134Z level=INFO msg="Uploading 1 narinfos"4761lb # [ 60.947512] niks3-server[991]: 2026/09/23 07:00:00 INFO Registered completed upload object_key=nar/0h4zvhqjxsvdqli2l6i5ajh5sh40hz57gzchdirnvbisq2aks0k8.nar.zst4762lb # [ 60.956424] niks3-server[991]: 2026/09/23 07:00:00 INFO Registered completed upload object_key=log/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv4763lb # [ 60.959509] niks3-server[991]: 2026/09/23 07:00:00 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4764lb # [ 60.977469] niks3-server[991]: 2026/09/23 07:00:00 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.ls4765lb # [ 60.986833] niks3-server[991]: 2026/09/23 07:00:00 INFO Completed upload id=144766worker2 # [ 60.994609] nix-grpc-daemon[880]: time=2026-09-23T07:00:00.180Z level=INFO msg="Upload complete. (267ms)"4767lb # [ 60.995710] niks3-server[991]: 2026/09/23 07:00:00 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.narinfo4768worker2 # [ 61.000594] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=1 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv assign_id=7 outputs=14769worker2 # [ 61.015775] nix-daemon[855]: reaped child process 958, status = succeeded4770client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig >&2, in 5.18 seconds)4771client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath4772client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath, in 0.12 seconds)4773client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top4774worker1 # [ 61.257955] nix-grpc-daemon[783]: ts=2026-09-23T06:59:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:524224775client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):4776client # /nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a4777client # /nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b4778client # /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top4779worker1 # [ 61.302315] nix-grpc-daemon[783]: ts=2026-09-23T06:59:59Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:524224780worker1 # [ 61.308855] nix-grpc-daemon[783]: copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'http://lb:5751'...4781worker1 # [ 61.322411] nix-grpc-daemon[783]: copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'http://lb:5751'...4782worker1 # [ 61.333910] nix-grpc-daemon[783]: copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'http://lb:5751'...4783client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.22 seconds)4784client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top4785worker2 # [ 61.490921] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:465404786worker2 # [ 61.497431] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:465404787client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.18 seconds)4788client: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top4789client # finding garbage collector roots...4790client # removing stale temporary roots file "/nix/var/nix/temproots/887"4791client # removing stale temporary roots file "/nix/var/nix/temproots/858"4792client # removing stale temporary roots file "/nix/var/nix/temproots/879"4793client # removing stale temporary roots file "/nix/var/nix/temproots/866"4794client # removing stale temporary roots file "/nix/var/nix/temproots/846"4795client # removing stale temporary roots file "/nix/var/nix/temproots/825"4796client # removing stale temporary roots file "/nix/var/nix/temproots/810"4797client # removing stale temporary roots file "/nix/var/nix/temproots/902"4798client # deleting specified paths...4799client # deleting '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top'4800client # deleting unused links...4801client # note: hard linking is currently saving 0.0 KiB4802worker1 # [ 61.792938] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:437004803worker1 # [ 61.800210] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:437004804worker1 # [ 61.810309] nix-grpc-daemon[883]: time=2026-09-23T07:00:00.089Z level=INFO msg="All 1 paths already cached"4805worker1 # [ 61.816537] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:437004806worker2 # [ 61.828800] nix-grpc-daemon[783]: ts=2026-09-23T07:00:01Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:408824807worker1 # [ 61.832177] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:437004808client # copying 3 paths...4809worker1 # [ 61.842339] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:437004810client # copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'grpc://lb:50051'...4811worker2 # [ 61.863391] nix-grpc-daemon[783]: ts=2026-09-23T07:00:01Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:408824812worker2 # [ 61.868827] nix-grpc-daemon[783]: ts=2026-09-23T07:00:01Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=0 paths=1 nar_bytes_out=2324813worker1 # [ 61.866626] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:437004814worker1 # [ 61.872181] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=0 paths=1 nar_bytes_out=1764815worker2 # [ 61.876915] nix-grpc-daemon[783]: ts=2026-09-23T07:00:01Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:408824816worker2 # [ 61.881528] nix-grpc-daemon[783]: ts=2026-09-23T07:00:01Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=0 paths=1 nar_bytes_out=1284817client # copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'grpc://lb:50051'...4818client # copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'grpc://lb:50051'...4819client: (finished: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.41 seconds)4820(finished: subtest: outputs read back through any worker carry the cache signature, in 6.11 seconds)4821subtest: two clients wanting the same drv build it once4822client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup4823client # [ 61.999667] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.4824client # Running as unit: dedup1.service; invocation ID: fbf4cd748318489f9cf43ecc7004074d4825client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 0.03 seconds)4826client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup4827worker2 # [ 62.121760] nix-grpc-daemon[783]: ts=2026-09-23T07:00:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:408824828client # [ 62.135704] nix[955]: don't know how to build these paths:4829client # [ 62.136617] nix[955]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4830worker1 # [ 62.131283] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04831worker1 # [ 62.135422] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.3:50051 clients=14832worker1 # [ 62.143349] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437004833worker1 # [ 62.155562] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:437004834client # [ 62.177094] nix[955]: copying 1 paths to 'lb:50051'...4835client # [ 62.177949] nix[955]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4836worker1 # [ 62.166591] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:437004837worker1 # [ 62.175735] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4838worker1 # [ 62.276668] nix-daemon[869]: accepted connection from pid 1061, user nix-grpc-daemon (trusted)4839worker1 # [ 62.290161] nix-daemon[869]: reaped child process 1064, status = succeeded4840lb # [ 62.293677] niks3-server[991]: 2026/09/23 07:00:01 INFO Received uploads request method=POST path=/api/pending_closures4841worker1 # [ 62.306208] nix-grpc-daemon[883]: time=2026-09-23T07:00:00.585Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4842worker1 # [ 62.307780] nix-grpc-daemon[883]: time=2026-09-23T07:00:00.587Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4843lb # [ 62.326653] niks3-server[991]: 2026/09/23 07:00:01 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4844lb # [ 62.332987] niks3-server[991]: 2026/09/23 07:00:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4845lb # [ 62.340762] niks3-server[991]: 2026/09/23 07:00:01 INFO Signed narinfos id=15 count=14846worker1 # [ 62.348081] nix-grpc-daemon[883]: time=2026-09-23T07:00:00.627Z level=INFO msg="Uploading 1 narinfos"4847lb # [ 62.349437] niks3-server[991]: 2026/09/23 07:00:01 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4848lb # [ 62.359566] niks3-server[991]: 2026/09/23 07:00:01 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4849lb # [ 62.366669] niks3-server[991]: 2026/09/23 07:00:01 INFO Completed upload id=154850worker1 # [ 62.370619] nix-grpc-daemon[883]: time=2026-09-23T07:00:00.650Z level=INFO msg="Upload complete. (181ms)"4851lb # [ 62.369387] niks3-server[991]: 2026/09/23 07:00:01 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4852worker1 # [ 62.374966] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=0 paths=1 nar_bytes_in=4324853worker1 # [ 62.381761] nix-grpc-daemon[783]: ts=2026-09-23T07:00:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437004854worker1 # [ 62.392250] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4855worker1 # [ 62.396690] nix-daemon[869]: reaped child process 1059, status = succeeded4856worker1 # [ 62.401660] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4857client # [ 62.413208] nix[955]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4858client # [ 63.055568] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.4859client # Running as unit: dedup2.service; invocation ID: 4e3e341583f64383aabc52791df66e134860client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 1.07 seconds)4861worker1 # [ 63.207962] nix-grpc-daemon[783]: ts=2026-09-23T07:00:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:437004862client # [ 63.228395] nix[971]: this derivation will be built:4863worker2 # [ 63.225667] nix-grpc-daemon[783]: ts=2026-09-23T07:00:02Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:408824864worker2 # [ 63.236808] nix-grpc-daemon[783]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' from 'http://lb:5751'...4865client # [ 63.261101] nix[971]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4866worker1 # [ 63.254385] nix-grpc-daemon[783]: ts=2026-09-23T07:00:01Z level=debug event=want client=6 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04867worker1 # [ 63.256691] nix-grpc-daemon[783]: ts=2026-09-23T07:00:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437004868client # [ 63.271096] nix[971]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4869client: must succeed: sleep 24870client: (finished: must succeed: sleep 2, in 2.02 seconds)4871client: waiting for success: ! systemctl is-active dedup1 dedup24872worker1 # [ 66.432635] nix-daemon[869]: reaped child process 1068, status = succeeded4873worker1 # [ 66.506213] nix-daemon[869]: accepted connection from pid 1100, user nix-grpc-daemon (trusted)4874worker1 # [ 66.518502] nix-daemon[869]: reaped child process 1103, status = succeeded4875lb # [ 66.522552] niks3-server[991]: 2026/09/23 07:00:05 INFO Received uploads request method=POST path=/api/pending_closures4876worker1 # [ 66.536122] nix-grpc-daemon[883]: time=2026-09-23T07:00:04.815Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4877worker1 # [ 66.540452] nix-grpc-daemon[883]: time=2026-09-23T07:00:04.817Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4878lb # [ 66.568692] niks3-server[991]: 2026/09/23 07:00:05 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4879lb # [ 66.585098] niks3-server[991]: 2026/09/23 07:00:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4880worker1 # [ 66.593422] nix-grpc-daemon[883]: time=2026-09-23T07:00:04.872Z level=INFO msg="Uploading 1 narinfos"4881lb # [ 66.589088] niks3-server[991]: 2026/09/23 07:00:05 INFO Signed narinfos id=16 count=14882lb # [ 66.598097] niks3-server[991]: 2026/09/23 07:00:05 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4883lb # [ 66.602086] niks3-server[991]: 2026/09/23 07:00:05 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4884lb # [ 66.607746] envoy[912]: conn peer=[2001:db8:1::1]:51694 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9507 tx=4354 ms=50184885lb # [ 66.609260] envoy[912]: conn peer=[2001:db8:1::1]:51700 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1108 tx=2928 ms=1514886lb # [ 66.610765] envoy[912]: conn peer=[2001:db8:1::1]:51720 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=330 ms=704887lb # [ 66.612180] envoy[912]: conn peer=[2001:db8:1::1]:51708 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=410 ms=844888lb # [ 66.613653] envoy[912]: conn peer=[2001:db8:1::1]:51710 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=376 ms=784889lb # [ 66.616495] niks3-server[991]: 2026/09/23 07:00:05 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4890lb # [ 66.624429] niks3-server[991]: 2026/09/23 07:00:05 INFO Completed upload id=164891worker1 # [ 66.628903] nix-grpc-daemon[883]: time=2026-09-23T07:00:04.908Z level=INFO msg="Upload complete. (217ms)"4892lb # [ 66.626724] niks3-server[991]: 2026/09/23 07:00:05 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo4893worker1 # [ 66.636793] nix-grpc-daemon[783]: ts=2026-09-23T07:00:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=14894worker1 # [ 66.644291] nix-grpc-daemon[783]: ts=2026-09-23T07:00:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=14895worker1 # [ 66.658380] nix-daemon[869]: reaped child process 1066, status = succeeded4896client # [ 66.698529] systemd[1]: dedup1.service: Deactivated successfully.4897client # [ 66.701828] systemd[1]: dedup1.service: Consumed 117ms CPU time over 4.701s wall clock time, 41.5M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.4898client # [ 66.707899] systemd[1]: dedup2.service: Deactivated successfully.4899client # [ 66.710606] systemd[1]: dedup2.service: Consumed 141ms CPU time over 3.654s wall clock time, 35.4M memory peak, 6.6K incoming IP traffic, 5.9K outgoing IP traffic.4900client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.09 seconds)4901client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4902client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)4903worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4904worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)4905worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4906worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)4907worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4908worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4909worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4910worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4911(finished: subtest: two clients wanting the same drv build it once, in 5.72 seconds)4912subtest: --timeout and --max-silent-time reach the builder4913worker1 # [ 67.831234] nix-grpc-daemon[783]: ts=2026-09-23T07:00:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:437004914worker1 # [ 67.842163] nix-grpc-daemon[783]: ts=2026-09-23T07:00:06Z level=debug event=want client=7 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv cached=04915worker1 # [ 67.846372] nix-grpc-daemon[783]: ts=2026-09-23T07:00:06Z level=debug event=assigned drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv worker=192.168.1.4:50051 clients=14916worker2 # [ 67.858597] nix-grpc-daemon[783]: ts=2026-09-23T07:00:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408824917worker2 # [ 67.870748] nix-grpc-daemon[783]: ts=2026-09-23T07:00:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408824918worker2 # [ 67.882479] nix-grpc-daemon[783]: ts=2026-09-23T07:00:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:408824919worker2 # [ 67.893249] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4920worker2 # [ 67.994539] nix-daemon[855]: accepted connection from pid 1005, user nix-grpc-daemon (trusted)4921worker2 # [ 68.007883] nix-daemon[855]: reaped child process 1008, status = succeeded4922lb # [ 68.007640] niks3-server[991]: 2026/09/23 07:00:07 INFO Received uploads request method=POST path=/api/pending_closures4923worker2 # [ 68.024171] nix-grpc-daemon[880]: time=2026-09-23T07:00:07.209Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4924worker2 # [ 68.026143] nix-grpc-daemon[880]: time=2026-09-23T07:00:07.211Z level=INFO msg="Uploading qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv (424B)"4925lb # [ 68.038832] niks3-server[991]: 2026/09/23 07:00:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4926worker2 # [ 68.054293] nix-grpc-daemon[880]: time=2026-09-23T07:00:07.239Z level=INFO msg="Uploading 1 narinfos"4927lb # [ 68.046923] niks3-server[991]: 2026/09/23 07:00:07 INFO Signed narinfos id=17 count=14928lb # [ 68.055603] niks3-server[991]: 2026/09/23 07:00:07 INFO Registered completed upload object_key=nar/1s4sciw269ksj1s7ymwz5cy4dw9n7bxv3k8wqw69h694spdh4ha2.nar.zst4929lb # [ 68.068954] niks3-server[991]: 2026/09/23 07:00:07 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4930lb # [ 68.074698] niks3-server[991]: 2026/09/23 07:00:07 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.ls4931lb # [ 68.079610] niks3-server[991]: 2026/09/23 07:00:07 INFO Completed upload id=174932worker2 # [ 68.087500] nix-grpc-daemon[880]: time=2026-09-23T07:00:07.273Z level=INFO msg="Upload complete. (183ms)"4933lb # [ 68.082887] niks3-server[991]: 2026/09/23 07:00:07 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.narinfo4934worker2 # [ 68.092342] nix-grpc-daemon[783]: ts=2026-09-23T07:00:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=0 paths=1 nar_bytes_in=4244935worker2 # [ 68.098292] nix-grpc-daemon[783]: ts=2026-09-23T07:00:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408824936worker2 # [ 68.108741] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4937worker2 # [ 68.119095] nix-daemon[855]: reaped child process 1003, status = succeeded4938worker2 # [ 68.122206] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4939worker2 # [ 74.134264] nix-grpc-daemon[783]: ts=2026-09-23T07:00:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=6 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv assign_id=9 outputs=04940worker2 # [ 74.145559] nix-daemon[855]: reaped child process 1010, status = succeeded4941worker2 # [ 74.146614] nix-daemon[855]: reaped child process 1012, status = succeeded4942worker1 # [ 74.339607] nix-grpc-daemon[783]: ts=2026-09-23T07:00:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:437004943worker1 # [ 74.350648] nix-grpc-daemon[783]: ts=2026-09-23T07:00:12Z level=debug event=want client=8 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv cached=04944worker1 # [ 74.357209] nix-grpc-daemon[783]: ts=2026-09-23T07:00:12Z level=debug event=assigned drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv worker=192.168.1.3:50051 clients=14945worker1 # [ 74.362065] nix-grpc-daemon[783]: ts=2026-09-23T07:00:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437004946worker1 # [ 74.369792] nix-grpc-daemon[783]: ts=2026-09-23T07:00:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:437004947worker1 # [ 74.380925] nix-grpc-daemon[783]: ts=2026-09-23T07:00:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:437004948worker1 # [ 74.391545] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4949worker1 # [ 74.487131] nix-daemon[869]: accepted connection from pid 1120, user nix-grpc-daemon (trusted)4950worker1 # [ 74.500430] nix-daemon[869]: reaped child process 1123, status = succeeded4951lb # [ 74.504408] niks3-server[991]: 2026/09/23 07:00:13 INFO Received uploads request method=POST path=/api/pending_closures4952worker1 # [ 74.517140] nix-grpc-daemon[883]: time=2026-09-23T07:00:12.796Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4953worker1 # [ 74.519063] nix-grpc-daemon[883]: time=2026-09-23T07:00:12.798Z level=INFO msg="Uploading fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv (424B)"4954lb # [ 74.539311] niks3-server[991]: 2026/09/23 07:00:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4955worker1 # [ 74.549385] nix-grpc-daemon[883]: time=2026-09-23T07:00:12.828Z level=INFO msg="Uploading 1 narinfos"4956lb # [ 74.545179] niks3-server[991]: 2026/09/23 07:00:13 INFO Signed narinfos id=18 count=14957lb # [ 74.558268] niks3-server[991]: 2026/09/23 07:00:13 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.ls4958lb # [ 74.564415] niks3-server[991]: 2026/09/23 07:00:13 INFO Registered completed upload object_key=nar/013y5sy2r6v0qdwv0dzpgvdfy08bds4mfh2qpvbgjazzrg2v71hp.nar.zst4959lb # [ 74.572247] niks3-server[991]: 2026/09/23 07:00:13 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4960lb # [ 74.579891] niks3-server[991]: 2026/09/23 07:00:13 INFO Completed upload id=184961worker1 # [ 74.586127] nix-grpc-daemon[883]: time=2026-09-23T07:00:12.865Z level=INFO msg="Upload complete. (183ms)"4962lb # [ 74.585282] niks3-server[991]: 2026/09/23 07:00:13 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.narinfo4963worker1 # [ 74.590195] nix-grpc-daemon[783]: ts=2026-09-23T07:00:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=0 paths=1 nar_bytes_in=4244964worker1 # [ 74.595450] nix-grpc-daemon[783]: ts=2026-09-23T07:00:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437004965worker1 # [ 74.605130] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4966worker1 # [ 74.617671] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4967worker1 # [ 74.619822] nix-daemon[869]: reaped child process 1118, status = succeeded4968lb # [ 76.608793] envoy[912]: conn peer=[2001:db8:1::1]:51736 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2981 tx=1543 ms=45354969lb # [ 76.612638] envoy[912]: conn peer=[2001:db8:1::1]:33796 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1589 tx=1598 ms=34584970lb # [ 76.616217] envoy[912]: conn peer=[2001:db8:1::1]:33806 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=935 ms=63574971worker1 # [ 80.630641] nix-grpc-daemon[783]: ts=2026-09-23T07:00:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=6 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv assign_id=10 outputs=04972worker1 # [ 80.642433] nix-daemon[869]: reaped child process 1125, status = succeeded4973worker1 # [ 80.643617] nix-daemon[869]: reaped child process 1127, status = succeeded4974(finished: subtest: --timeout and --max-silent-time reach the builder, in 13.12 seconds)4975subtest: the first client leaving does not fail the second4976client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share4977client # [ 80.852746] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.4978client # Running as unit: share1.service; invocation ID: 14aaa88bd1c84eeaaf2eeca1823c1a664979client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 0.06 seconds)4980client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share4981client # [ 80.999545] nix[1044]: don't know how to build these paths:4982client # [ 81.000516] nix[1044]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv4983worker1 # [ 80.986355] nix-grpc-daemon[783]: ts=2026-09-23T07:00:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:437004984worker1 # [ 80.997244] nix-grpc-daemon[783]: ts=2026-09-23T07:00:19Z level=debug event=want client=9 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=04985worker1 # [ 81.001721] nix-grpc-daemon[783]: ts=2026-09-23T07:00:19Z level=debug event=assigned drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv worker=192.168.1.4:50051 clients=14986worker2 # [ 81.003983] nix-grpc-daemon[783]: ts=2026-09-23T07:00:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408824987worker2 # [ 81.017925] nix-grpc-daemon[783]: ts=2026-09-23T07:00:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408824988client # [ 81.035842] nix[1044]: copying 1 paths to 'lb:50051'...4989client # [ 81.036743] nix[1044]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' to 'lb:50051'...4990worker2 # [ 81.028461] nix-grpc-daemon[783]: ts=2026-09-23T07:00:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:408824991worker2 # [ 81.039370] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4992worker2 # [ 81.137548] nix-daemon[855]: accepted connection from pid 1035, user nix-grpc-daemon (trusted)4993worker2 # [ 81.149550] nix-daemon[855]: reaped child process 1038, status = succeeded4994lb # [ 81.150337] niks3-server[991]: 2026/09/23 07:00:20 INFO Received uploads request method=POST path=/api/pending_closures4995worker2 # [ 81.167047] nix-grpc-daemon[880]: time=2026-09-23T07:00:20.352Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4996worker2 # [ 81.169023] nix-grpc-daemon[880]: time=2026-09-23T07:00:20.354Z level=INFO msg="Uploading j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv (432B)"4997lb # [ 81.189866] niks3-server[991]: 2026/09/23 07:00:20 INFO Registered completed upload object_key=nar/0s0b0gfhn36p0aylk1qa64cgw01bi7a8ca93iv4zg2q7wqcjcqhs.nar.zst4998lb # [ 81.197317] niks3-server[991]: 2026/09/23 07:00:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4999lb # [ 81.203683] niks3-server[991]: 2026/09/23 07:00:20 INFO Signed narinfos id=19 count=15000worker2 # [ 81.214997] nix-grpc-daemon[880]: time=2026-09-23T07:00:20.400Z level=INFO msg="Uploading 1 narinfos"5001lb # [ 81.218249] niks3-server[991]: 2026/09/23 07:00:20 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.ls5002lb # [ 81.226512] niks3-server[991]: 2026/09/23 07:00:20 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5003lb # [ 81.232808] niks3-server[991]: 2026/09/23 07:00:20 INFO Completed upload id=195004lb # [ 81.234276] niks3-server[991]: 2026/09/23 07:00:20 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.narinfo5005worker2 # [ 81.240651] nix-grpc-daemon[880]: time=2026-09-23T07:00:20.426Z level=INFO msg="Upload complete. (190ms)"5006worker2 # [ 81.245355] nix-grpc-daemon[783]: ts=2026-09-23T07:00:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=0 paths=1 nar_bytes_in=4325007worker2 # [ 81.252317] nix-grpc-daemon[783]: ts=2026-09-23T07:00:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408825008worker2 # [ 81.262944] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5009worker2 # [ 81.272440] nix-daemon[855]: reaped child process 1033, status = succeeded5010worker2 # [ 81.279709] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5011client # [ 81.287764] nix[1044]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5012client # [ 81.920962] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.5013client # Running as unit: share2.service; invocation ID: 6bbdea94b5e14775989fe63181888af45014client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 1.07 seconds)5015client: must succeed: sleep 2; systemctl kill -s INT share15016worker2 # [ 82.068668] nix-grpc-daemon[783]: ts=2026-09-23T07:00:21Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:408825017client # [ 82.084653] nix[1060]: this derivation will be built:5018worker1 # [ 82.075238] nix-grpc-daemon[783]: ts=2026-09-23T07:00:20Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:437005019worker1 # [ 82.086363] nix-grpc-daemon[783]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' from 'http://lb:5751'...5020client # [ 82.112770] nix[1060]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5021worker1 # [ 82.107450] nix-grpc-daemon[783]: ts=2026-09-23T07:00:20Z level=debug event=want client=10 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05022worker2 # [ 82.117848] nix-grpc-daemon[783]: ts=2026-09-23T07:00:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408825023client # [ 82.128615] nix[1060]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5024client # [ 84.082066] systemd[1]: share1.service: Sent signal SIGINT to main process 1044 (nix) on client request.5025client # [ 84.088845] nix[1044]: shutting downj2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5026client # [ 84.095734] nix[1044]: error: interrupted by the user5027client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.06 seconds)5028client: waiting for success: ! systemctl is-active share15029client # [ 84.139935] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE5030client # [ 84.141419] systemd[1]: share1.service: Failed with result 'exit-code'.5031client # [ 84.143365] systemd[1]: share1.service: Consumed 118ms CPU time over 3.290s wall clock time, 41.5M memory peak, 6.1K incoming IP traffic, 8.1K outgoing IP traffic.5032client: (finished: waiting for success: ! systemctl is-active share1, in 1.08 seconds)5033client: waiting for success: ! systemctl is-active share25034worker2 # [ 85.520875] nix-daemon[855]: reaped child process 1042, status = succeeded5035worker2 # [ 85.598246] nix-daemon[855]: accepted connection from pid 1072, user nix-grpc-daemon (trusted)5036worker2 # [ 85.610435] nix-daemon[855]: reaped child process 1075, status = succeeded5037lb # [ 85.610823] niks3-server[991]: 2026/09/23 07:00:24 INFO Received uploads request method=POST path=/api/pending_closures5038worker2 # [ 85.629567] nix-grpc-daemon[880]: time=2026-09-23T07:00:24.815Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5039worker2 # [ 85.631336] nix-grpc-daemon[880]: time=2026-09-23T07:00:24.816Z level=INFO msg="Uploading b9lg6znd0rinl36gy6ixq1dcj6b8bhhw-slow-share (120B)"5040lb # [ 85.655407] niks3-server[991]: 2026/09/23 07:00:24 INFO Registered completed upload object_key=log/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5041lb # [ 85.664852] niks3-server[991]: 2026/09/23 07:00:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5042lb # [ 85.674646] niks3-server[991]: 2026/09/23 07:00:24 INFO Signed narinfos id=20 count=15043worker2 # [ 85.682700] nix-grpc-daemon[880]: time=2026-09-23T07:00:24.868Z level=INFO msg="Uploading 1 narinfos"5044lb # [ 85.679882] niks3-server[991]: 2026/09/23 07:00:24 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst5045lb # [ 85.684539] niks3-server[991]: 2026/09/23 07:00:24 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.ls5046lb # [ 85.694553] niks3-server[991]: 2026/09/23 07:00:24 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5047lb # [ 85.698866] niks3-server[991]: 2026/09/23 07:00:24 INFO Completed upload id=205048lb # [ 85.701467] niks3-server[991]: 2026/09/23 07:00:24 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.narinfo5049worker2 # [ 85.708127] nix-grpc-daemon[880]: time=2026-09-23T07:00:24.893Z level=INFO msg="Upload complete. (218ms)"5050worker2 # [ 85.714832] nix-grpc-daemon[783]: ts=2026-09-23T07:00:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=4 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15051worker2 # [ 85.721261] nix-grpc-daemon[783]: ts=2026-09-23T07:00:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=3 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15052worker2 # [ 85.734387] nix-daemon[855]: reaped child process 1040, status = succeeded5053client # [ 85.796307] systemd[1]: share2.service: Deactivated successfully.5054client # [ 85.800221] systemd[1]: share2.service: Consumed 120ms CPU time over 3.879s wall clock time, 35.4M memory peak, 6.5K incoming IP traffic, 6K outgoing IP traffic.5055client: (finished: waiting for success: ! systemctl is-active share2, in 1.08 seconds)5056client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }5057client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.03 seconds)5058(finished: subtest: the first client leaving does not fail the second, in 5.66 seconds)5059subtest: build hook: nix-daemon with builders = grpc://lb5060client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log5061client # [ 86.557351] systemd[1]: Started Nix Daemon.5062lb # [ 86.609680] envoy[912]: conn peer=[2001:db8:1::1]:50002 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2965 tx=933 ms=63475063lb # [ 86.613531] envoy[912]: conn peer=[2001:db8:1::1]:50008 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2965 tx=796 ms=31115064lb # [ 86.617090] envoy[912]: conn peer=[2001:db8:1::1]:50016 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1602 tx=1598 ms=36975065client # [ 86.649943] nix-daemon[1112]: accepted connection from pid 1109, user root (trusted)5066worker2 # [ 86.759572] nix-grpc-daemon[783]: ts=2026-09-23T07:00:25Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:408825067worker2 # [ 86.764880] nix-grpc-daemon[783]: ts=2026-09-23T07:00:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408825068worker1 # [ 86.772810] nix-grpc-daemon[783]: ts=2026-09-23T07:00:25Z level=debug event=want client=11 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05069worker1 # [ 86.778495] nix-grpc-daemon[783]: ts=2026-09-23T07:00:25Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=15070worker1 # [ 86.785241] nix-grpc-daemon[783]: ts=2026-09-23T07:00:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437005071worker1 # [ 86.796492] nix-grpc-daemon[783]: ts=2026-09-23T07:00:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:437005072worker1 # [ 86.808161] nix-grpc-daemon[783]: ts=2026-09-23T07:00:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:437005073worker1 # [ 86.819867] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5074worker1 # [ 86.919592] nix-daemon[869]: accepted connection from pid 1169, user nix-grpc-daemon (trusted)5075worker1 # [ 86.931690] nix-daemon[869]: reaped child process 1172, status = succeeded5076lb # [ 86.936812] niks3-server[991]: 2026/09/23 07:00:26 INFO Received uploads request method=POST path=/api/pending_closures5077worker1 # [ 86.951322] nix-grpc-daemon[883]: time=2026-09-23T07:00:25.230Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5078worker1 # [ 86.953354] nix-grpc-daemon[883]: time=2026-09-23T07:00:25.232Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5079lb # [ 86.972443] niks3-server[991]: 2026/09/23 07:00:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5080worker1 # [ 86.982637] nix-grpc-daemon[883]: time=2026-09-23T07:00:25.262Z level=INFO msg="Uploading 1 narinfos"5081lb # [ 86.978760] niks3-server[991]: 2026/09/23 07:00:26 INFO Signed narinfos id=21 count=15082lb # [ 86.983090] niks3-server[991]: 2026/09/23 07:00:26 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5083lb # [ 86.994025] niks3-server[991]: 2026/09/23 07:00:26 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5084lb # [ 86.996952] niks3-server[991]: 2026/09/23 07:00:26 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5085lb # [ 87.002771] niks3-server[991]: 2026/09/23 07:00:26 INFO Completed upload id=215086worker1 # [ 87.007078] nix-grpc-daemon[883]: time=2026-09-23T07:00:25.286Z level=INFO msg="Upload complete. (175ms)"5087lb # [ 87.004387] niks3-server[991]: 2026/09/23 07:00:26 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo5088worker1 # [ 87.010974] nix-grpc-daemon[783]: ts=2026-09-23T07:00:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=0 paths=1 nar_bytes_in=5125089worker1 # [ 87.015782] nix-grpc-daemon[783]: ts=2026-09-23T07:00:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437005090worker1 # [ 87.026992] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5091worker1 # [ 87.030383] nix-daemon[869]: reaped child process 1167, status = succeeded5092worker1 # [ 87.035225] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5093worker1 # [ 87.874375] nix-daemon[869]: reaped child process 1176, status = succeeded5094worker1 # [ 87.951073] nix-daemon[869]: accepted connection from pid 1192, user nix-grpc-daemon (trusted)5095worker1 # [ 87.964079] nix-daemon[869]: reaped child process 1195, status = succeeded5096lb # [ 87.967703] niks3-server[991]: 2026/09/23 07:00:27 INFO Received uploads request method=POST path=/api/pending_closures5097worker1 # [ 87.982206] nix-grpc-daemon[883]: time=2026-09-23T07:00:26.261Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5098worker1 # [ 87.983786] nix-grpc-daemon[883]: time=2026-09-23T07:00:26.263Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5099lb # [ 88.006682] niks3-server[991]: 2026/09/23 07:00:27 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5100lb # [ 88.022173] niks3-server[991]: 2026/09/23 07:00:27 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5101lb # [ 88.024959] niks3-server[991]: 2026/09/23 07:00:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5102lb # [ 88.030863] niks3-server[991]: 2026/09/23 07:00:27 INFO Signed narinfos id=22 count=15103worker1 # [ 88.035391] nix-grpc-daemon[883]: time=2026-09-23T07:00:26.314Z level=INFO msg="Uploading 1 narinfos"5104lb # [ 88.037478] niks3-server[991]: 2026/09/23 07:00:27 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5105lb # [ 88.045113] niks3-server[991]: 2026/09/23 07:00:27 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5106lb # [ 88.050289] niks3-server[991]: 2026/09/23 07:00:27 INFO Completed upload id=225107worker1 # [ 88.054427] nix-grpc-daemon[883]: time=2026-09-23T07:00:26.333Z level=INFO msg="Upload complete. (212ms)"5108lb # [ 88.052154] niks3-server[991]: 2026/09/23 07:00:27 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5109worker1 # [ 88.061679] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=12 outputs=15110worker1 # [ 88.070380] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:437005111worker1 # [ 88.080778] nix-daemon[869]: reaped child process 1174, status = succeeded5112worker1 # [ 88.087934] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:437005113worker1 # [ 88.091885] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=0 paths=1 nar_bytes_out=1285114worker1 # [ 88.299331] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:437005115worker2 # [ 88.306921] nix-grpc-daemon[783]: ts=2026-09-23T07:00:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408825116worker1 # [ 88.324592] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=debug event=want client=12 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05117worker1 # [ 88.329889] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=15118worker1 # [ 88.335541] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437005119worker1 # [ 88.346931] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:437005120worker1 # [ 88.359644] nix-grpc-daemon[883]: time=2026-09-23T07:00:26.639Z level=INFO msg="All 1 paths already cached"5121worker1 # [ 88.367285] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:437005122worker1 # [ 88.375558] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5123worker1 # [ 88.475638] nix-daemon[869]: accepted connection from pid 1201, user nix-grpc-daemon (trusted)5124worker1 # [ 88.488712] nix-daemon[869]: reaped child process 1204, status = succeeded5125lb # [ 88.491646] niks3-server[991]: 2026/09/23 07:00:27 INFO Received uploads request method=POST path=/api/pending_closures5126worker1 # [ 88.504714] nix-grpc-daemon[883]: time=2026-09-23T07:00:26.784Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5127worker1 # [ 88.506144] nix-grpc-daemon[883]: time=2026-09-23T07:00:26.785Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5128lb # [ 88.526793] niks3-server[991]: 2026/09/23 07:00:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5129worker1 # [ 88.537845] nix-grpc-daemon[883]: time=2026-09-23T07:00:26.817Z level=INFO msg="Uploading 1 narinfos"5130lb # [ 88.533689] niks3-server[991]: 2026/09/23 07:00:27 INFO Signed narinfos id=23 count=15131lb # [ 88.542178] niks3-server[991]: 2026/09/23 07:00:27 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5132lb # [ 88.547662] niks3-server[991]: 2026/09/23 07:00:27 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5133lb # [ 88.558307] niks3-server[991]: 2026/09/23 07:00:27 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5134lb # [ 88.564068] niks3-server[991]: 2026/09/23 07:00:27 INFO Completed upload id=235135worker1 # [ 88.570619] nix-grpc-daemon[883]: time=2026-09-23T07:00:26.850Z level=INFO msg="Upload complete. (181ms)"5136lb # [ 88.571933] niks3-server[991]: 2026/09/23 07:00:27 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5137worker1 # [ 88.575285] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=0 paths=1 nar_bytes_in=6245138worker1 # [ 88.581517] nix-grpc-daemon[783]: ts=2026-09-23T07:00:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437005139worker1 # [ 88.592259] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5140worker1 # [ 88.601880] nix-daemon[869]: reaped child process 1199, status = succeeded5141worker1 # [ 88.606220] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5142worker1 # [ 90.044536] nix-daemon[869]: reaped child process 1208, status = succeeded5143worker1 # [ 90.128502] nix-daemon[869]: accepted connection from pid 1224, user nix-grpc-daemon (trusted)5144worker1 # [ 90.142580] nix-daemon[869]: reaped child process 1227, status = succeeded5145lb # [ 90.144686] niks3-server[991]: 2026/09/23 07:00:29 INFO Received uploads request method=POST path=/api/pending_closures5146worker1 # [ 90.157862] nix-grpc-daemon[883]: time=2026-09-23T07:00:28.437Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5147worker1 # [ 90.160209] nix-grpc-daemon[883]: time=2026-09-23T07:00:28.439Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5148lb # [ 90.190179] niks3-server[991]: 2026/09/23 07:00:29 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5149lb # [ 90.195636] niks3-server[991]: 2026/09/23 07:00:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5150lb # [ 90.200312] niks3-server[991]: 2026/09/23 07:00:29 INFO Signed narinfos id=24 count=15151worker1 # [ 90.208036] nix-grpc-daemon[883]: time=2026-09-23T07:00:28.487Z level=INFO msg="Uploading 1 narinfos"5152lb # [ 90.206584] niks3-server[991]: 2026/09/23 07:00:29 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5153lb # [ 90.219124] niks3-server[991]: 2026/09/23 07:00:29 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5154lb # [ 90.221655] niks3-server[991]: 2026/09/23 07:00:29 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5155lb # [ 90.227371] niks3-server[991]: 2026/09/23 07:00:29 INFO Completed upload id=245156lb # [ 90.229029] niks3-server[991]: 2026/09/23 07:00:29 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5157worker1 # [ 90.231827] nix-grpc-daemon[883]: time=2026-09-23T07:00:28.511Z level=INFO msg="Upload complete. (217ms)"5158worker1 # [ 90.238665] nix-grpc-daemon[783]: ts=2026-09-23T07:00:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=13 outputs=15159worker2 # [ 90.254491] nix-grpc-daemon[783]: ts=2026-09-23T07:00:29Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:408825160worker1 # [ 90.254345] nix-daemon[869]: reaped child process 1206, status = succeeded5161worker2 # [ 90.264205] nix-grpc-daemon[783]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5162worker2 # [ 90.274834] nix-grpc-daemon[783]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...5163worker2 # [ 90.302065] nix-grpc-daemon[783]: ts=2026-09-23T07:00:29Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:408825164worker2 # [ 90.307462] nix-grpc-daemon[783]: ts=2026-09-23T07:00:29Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=0 paths=1 nar_bytes_out=1765165worker2 # [ 90.518270] nix-grpc-daemon[783]: ts=2026-09-23T07:00:29Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:408825166worker2 # [ 90.525530] nix-grpc-daemon[783]: ts=2026-09-23T07:00:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408825167worker2 # [ 90.532560] nix-grpc-daemon[880]: time=2026-09-23T07:00:29.717Z level=INFO msg="All 2 paths already cached"5168worker1 # [ 90.543582] nix-grpc-daemon[783]: ts=2026-09-23T07:00:28Z level=debug event=want client=13 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05169worker1 # [ 90.549833] nix-grpc-daemon[783]: ts=2026-09-23T07:00:28Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=15170worker1 # [ 90.554959] nix-grpc-daemon[783]: ts=2026-09-23T07:00:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437005171worker1 # [ 90.565997] nix-grpc-daemon[783]: ts=2026-09-23T07:00:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:437005172worker1 # [ 90.578525] nix-grpc-daemon[883]: time=2026-09-23T07:00:28.858Z level=INFO msg="All 2 paths already cached"5173worker1 # [ 90.586608] nix-grpc-daemon[783]: ts=2026-09-23T07:00:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:437005174worker1 # [ 90.594914] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5175worker1 # [ 90.693569] nix-daemon[869]: accepted connection from pid 1233, user nix-grpc-daemon (trusted)5176worker1 # [ 90.706386] nix-daemon[869]: reaped child process 1236, status = succeeded5177lb # [ 90.710609] niks3-server[991]: 2026/09/23 07:00:29 INFO Received uploads request method=POST path=/api/pending_closures5178worker1 # [ 90.725270] nix-grpc-daemon[883]: time=2026-09-23T07:00:29.004Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5179worker1 # [ 90.726749] nix-grpc-daemon[883]: time=2026-09-23T07:00:29.006Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5180lb # [ 90.746810] niks3-server[991]: 2026/09/23 07:00:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5181lb # [ 90.754449] niks3-server[991]: 2026/09/23 07:00:29 INFO Signed narinfos id=25 count=15182worker1 # [ 90.762024] nix-grpc-daemon[883]: time=2026-09-23T07:00:29.041Z level=INFO msg="Uploading 1 narinfos"5183lb # [ 90.759082] niks3-server[991]: 2026/09/23 07:00:29 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5184lb # [ 90.766705] niks3-server[991]: 2026/09/23 07:00:29 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5185lb # [ 90.778893] niks3-server[991]: 2026/09/23 07:00:29 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5186lb # [ 90.784260] niks3-server[991]: 2026/09/23 07:00:29 INFO Completed upload id=255187worker1 # [ 90.791310] nix-grpc-daemon[883]: time=2026-09-23T07:00:29.070Z level=INFO msg="Upload complete. (182ms)"5188worker1 # [ 90.796369] nix-grpc-daemon[783]: ts=2026-09-23T07:00:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=0 paths=1 nar_bytes_in=7525189worker1 # [ 90.802323] nix-grpc-daemon[783]: ts=2026-09-23T07:00:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437005190worker1 # [ 90.814351] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5191worker1 # [ 90.822137] nix-daemon[869]: reaped child process 1231, status = succeeded5192worker1 # [ 90.829736] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5193worker1 # [ 92.261705] nix-daemon[869]: reaped child process 1240, status = succeeded5194worker1 # [ 92.348856] nix-daemon[869]: accepted connection from pid 1256, user nix-grpc-daemon (trusted)5195worker1 # [ 92.361292] nix-daemon[869]: reaped child process 1259, status = succeeded5196lb # [ 92.366336] niks3-server[991]: 2026/09/23 07:00:31 INFO Received uploads request method=POST path=/api/pending_closures5197worker1 # [ 92.380708] nix-grpc-daemon[883]: time=2026-09-23T07:00:30.660Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5198worker1 # [ 92.383121] nix-grpc-daemon[883]: time=2026-09-23T07:00:30.662Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5199lb # [ 92.412713] niks3-server[991]: 2026/09/23 07:00:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5200worker1 # [ 92.423176] nix-grpc-daemon[883]: time=2026-09-23T07:00:30.702Z level=INFO msg="Uploading 1 narinfos"5201lb # [ 92.419371] niks3-server[991]: 2026/09/23 07:00:31 INFO Signed narinfos id=26 count=15202lb # [ 92.428287] niks3-server[991]: 2026/09/23 07:00:31 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5203lb # [ 92.434143] niks3-server[991]: 2026/09/23 07:00:31 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5204lb # [ 92.438204] niks3-server[991]: 2026/09/23 07:00:31 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5205lb # [ 92.443777] niks3-server[991]: 2026/09/23 07:00:31 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5206lb # [ 92.448643] niks3-server[991]: 2026/09/23 07:00:31 INFO Completed upload id=265207lb # [ 92.450450] niks3-server[991]: 2026/09/23 07:00:31 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5208worker1 # [ 92.453081] nix-grpc-daemon[883]: time=2026-09-23T07:00:30.732Z level=INFO msg="Upload complete. (213ms)"5209worker1 # [ 92.459960] nix-grpc-daemon[783]: ts=2026-09-23T07:00:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=14 outputs=15210worker1 # [ 92.473505] nix-daemon[869]: reaped child process 1238, status = succeeded5211worker2 # [ 92.510706] nix-grpc-daemon[783]: ts=2026-09-23T07:00:31Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:408825212worker2 # [ 92.522376] nix-grpc-daemon[783]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...5213worker1 # [ 92.547255] nix-grpc-daemon[783]: ts=2026-09-23T07:00:30Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:437005214worker1 # [ 92.552446] nix-grpc-daemon[783]: ts=2026-09-23T07:00:30Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=0 paths=1 nar_bytes_out=2325215client # [ 92.632629] nix-daemon[1112]: reaped child process 1114, status = succeeded5216client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.18 seconds)5217client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5218client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.02 seconds)5219client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5220client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.02 seconds)5221client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5222client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.02 seconds)5223(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 6.24 seconds)5224subtest: an input only one worker has still reaches the builder5225client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only5226client # this derivation will be built:5227client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5228client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5229client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only, in 0.25 seconds)5230client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5231client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.10 seconds)5232worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5233worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.12 seconds)5234worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5235worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)5236client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&25237client # [ 93.317095] nix-daemon[1112]: accepted connection from pid 1194, user root (trusted)5238client # this derivation will be built:5239client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5240client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5241worker2 # [ 93.433113] nix-grpc-daemon[783]: ts=2026-09-23T07:00:32Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:408825242worker2 # [ 93.438321] nix-grpc-daemon[783]: ts=2026-09-23T07:00:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408825243client # copying 1 paths...5244client # copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' to 'grpc://lb:50051'...5245worker1 # [ 93.447996] nix-grpc-daemon[783]: ts=2026-09-23T07:00:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:437005246worker1 # [ 93.458502] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5247worker1 # [ 93.564497] nix-daemon[869]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5248worker1 # [ 93.577475] nix-daemon[869]: reaped child process 1271, status = succeeded5249lb # [ 93.581205] niks3-server[991]: 2026/09/23 07:00:32 INFO Received uploads request method=POST path=/api/pending_closures5250worker1 # [ 93.593825] nix-grpc-daemon[883]: time=2026-09-23T07:00:31.873Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5251worker1 # [ 93.595866] nix-grpc-daemon[883]: time=2026-09-23T07:00:31.875Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5252lb # [ 93.612179] niks3-server[991]: 2026/09/23 07:00:32 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5253lb # [ 93.619500] niks3-server[991]: 2026/09/23 07:00:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5254worker1 # [ 93.632075] nix-grpc-daemon[883]: time=2026-09-23T07:00:31.911Z level=INFO msg="Uploading 1 narinfos"5255lb # [ 93.627825] niks3-server[991]: 2026/09/23 07:00:32 INFO Signed narinfos id=27 count=15256lb # [ 93.634896] niks3-server[991]: 2026/09/23 07:00:32 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5257lb # [ 93.646215] niks3-server[991]: 2026/09/23 07:00:32 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5258lb # [ 93.654290] niks3-server[991]: 2026/09/23 07:00:32 INFO Completed upload id=275259worker1 # [ 93.658829] nix-grpc-daemon[883]: time=2026-09-23T07:00:31.938Z level=INFO msg="Upload complete. (190ms)"5260lb # [ 93.656659] niks3-server[991]: 2026/09/23 07:00:32 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5261worker1 # [ 93.665321] nix-grpc-daemon[783]: ts=2026-09-23T07:00:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=0 paths=1 nar_bytes_in=1205262worker1 # [ 93.680240] nix-grpc-daemon[783]: ts=2026-09-23T07:00:31Z level=debug event=want client=14 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05263worker1 # [ 93.683842] nix-grpc-daemon[783]: ts=2026-09-23T07:00:31Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=15264worker2 # [ 93.688226] nix-grpc-daemon[783]: ts=2026-09-23T07:00:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408825265worker1 # [ 93.689263] nix-daemon[869]: reaped child process 1266, status = succeeded5266worker2 # [ 93.697872] nix-grpc-daemon[783]: ts=2026-09-23T07:00:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408825267client # copying 2 paths to 'lb:50051'...5268client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5269worker2 # [ 93.712458] nix-grpc-daemon[783]: ts=2026-09-23T07:00:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:408825270client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5271worker2 # [ 93.723601] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5272worker2 # [ 93.826742] nix-daemon[855]: accepted connection from pid 1085, user nix-grpc-daemon (trusted)5273worker2 # [ 93.839054] nix-daemon[855]: reaped child process 1088, status = succeeded5274lb # [ 93.838995] niks3-server[991]: 2026/09/23 07:00:33 INFO Received uploads request method=POST path=/api/pending_closures5275lb # [ 93.848645] niks3-server[991]: 2026/09/23 07:00:33 INFO Received uploads request method=POST path=/api/pending_closures5276worker2 # [ 93.865267] nix-grpc-daemon[880]: time=2026-09-23T07:00:33.049Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5277worker2 # [ 93.866627] nix-grpc-daemon[880]: time=2026-09-23T07:00:33.049Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5278worker2 # [ 93.869688] nix-grpc-daemon[880]: time=2026-09-23T07:00:33.049Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5279lb # [ 93.888549] niks3-server[991]: 2026/09/23 07:00:33 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5280lb # [ 93.894154] niks3-server[991]: 2026/09/23 07:00:33 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5281lb # [ 93.905133] niks3-server[991]: 2026/09/23 07:00:33 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5282lb # [ 93.907237] niks3-server[991]: 2026/09/23 07:00:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5283lb # [ 93.909340] niks3-server[991]: 2026/09/23 07:00:33 INFO Signed narinfos id=29 count=25284lb # [ 93.910894] niks3-server[991]: 2026/09/23 07:00:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5285lb # [ 93.913493] niks3-server[991]: 2026/09/23 07:00:33 INFO Signed narinfos id=28 count=15286worker2 # [ 93.922307] nix-grpc-daemon[880]: time=2026-09-23T07:00:33.107Z level=INFO msg="Uploading 3 narinfos"5287lb # [ 93.916676] niks3-server[991]: 2026/09/23 07:00:33 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5288lb # [ 93.931309] niks3-server[991]: 2026/09/23 07:00:33 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5289lb # [ 93.934043] niks3-server[991]: 2026/09/23 07:00:33 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5290lb # [ 93.937515] niks3-server[991]: 2026/09/23 07:00:33 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5291lb # [ 93.939869] niks3-server[991]: 2026/09/23 07:00:33 INFO Completed upload id=285292lb # [ 93.941256] niks3-server[991]: 2026/09/23 07:00:33 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5293lb # [ 93.943622] niks3-server[991]: 2026/09/23 07:00:33 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5294lb # [ 93.946458] niks3-server[991]: 2026/09/23 07:00:33 INFO Completed upload id=295295worker2 # [ 93.954613] nix-grpc-daemon[880]: time=2026-09-23T07:00:33.140Z level=INFO msg="Upload complete. (217ms)"5296worker2 # [ 93.959854] nix-grpc-daemon[783]: ts=2026-09-23T07:00:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=0 paths=2 nar_bytes_in=9765297worker2 # [ 93.965443] nix-grpc-daemon[783]: ts=2026-09-23T07:00:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408825298worker2 # [ 93.979111] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5299worker2 # [ 93.986193] nix-daemon[855]: reaped child process 1083, status = succeeded5300worker2 # [ 93.996063] nix-grpc-daemon[783]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5301worker2 # [ 94.002788] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5302client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5303client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5304client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5305worker2 # [ 95.416267] nix-daemon[855]: reaped child process 1093, status = succeeded5306worker2 # [ 95.491188] nix-daemon[855]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5307worker2 # [ 95.503273] nix-daemon[855]: reaped child process 1112, status = succeeded5308lb # [ 95.504472] niks3-server[991]: 2026/09/23 07:00:34 INFO Received uploads request method=POST path=/api/pending_closures5309worker2 # [ 95.520913] nix-grpc-daemon[880]: time=2026-09-23T07:00:34.706Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5310lb # [ 95.538622] niks3-server[991]: 2026/09/23 07:00:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5311lb # [ 95.544083] niks3-server[991]: 2026/09/23 07:00:34 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5312worker2 # [ 95.559185] nix-grpc-daemon[880]: time=2026-09-23T07:00:34.744Z level=INFO msg="Uploading 1 narinfos"5313lb # [ 95.552305] niks3-server[991]: 2026/09/23 07:00:34 INFO Signed narinfos id=30 count=15314lb # [ 95.558863] niks3-server[991]: 2026/09/23 07:00:34 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5315lb # [ 95.570485] niks3-server[991]: 2026/09/23 07:00:34 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5316lb # [ 95.580153] niks3-server[991]: 2026/09/23 07:00:34 INFO Completed upload id=305317worker2 # [ 95.589344] nix-grpc-daemon[880]: time=2026-09-23T07:00:34.775Z level=INFO msg="Upload complete. (177ms)"5318lb # [ 95.584685] niks3-server[991]: 2026/09/23 07:00:34 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5319worker2 # [ 95.593976] nix-grpc-daemon[783]: ts=2026-09-23T07:00:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=15 outputs=15320client # copying 1 paths...5321worker2 # [ 95.606538] nix-grpc-daemon[783]: ts=2026-09-23T07:00:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:408825322worker2 # [ 95.611582] nix-daemon[855]: reaped child process 1090, status = succeeded5323client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5324worker2 # [ 95.627575] nix-grpc-daemon[783]: ts=2026-09-23T07:00:34Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:408825325worker2 # [ 95.631894] nix-grpc-daemon[783]: ts=2026-09-23T07:00:34Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=0 paths=1 nar_bytes_out=1205326client # [ 95.752738] nix-daemon[1112]: reaped child process 1197, status = succeeded5327client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.56 seconds)5328client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&25329client # [ 95.859884] nix-daemon[1112]: accepted connection from pid 1216, user root (trusted)5330client # this derivation will be built:5331client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5332worker2 # [ 95.973852] nix-grpc-daemon[783]: ts=2026-09-23T07:00:35Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:408825333client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5334worker1 # [ 95.975715] nix-grpc-daemon[783]: ts=2026-09-23T07:00:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:437005335worker1 # [ 95.982504] nix-grpc-daemon[883]: time=2026-09-23T07:00:34.261Z level=INFO msg="All 1 paths already cached"5336worker1 # [ 95.996643] nix-grpc-daemon[783]: ts=2026-09-23T07:00:34Z level=debug event=want client=15 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05337worker1 # [ 96.002401] nix-grpc-daemon[783]: ts=2026-09-23T07:00:34Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15338worker1 # [ 96.040879] nix-grpc-daemon[783]: ts=2026-09-23T07:00:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437005339worker1 # [ 96.053852] nix-grpc-daemon[783]: ts=2026-09-23T07:00:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:437005340client # copying 1 paths to 'lb:50051'...5341client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5342worker1 # [ 96.071577] nix-grpc-daemon[783]: ts=2026-09-23T07:00:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:437005343worker1 # [ 96.083120] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5344worker1 # [ 96.098929] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 23 ms (attempt 1/5)5345worker1 # [ 96.129385] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 152 ms (attempt 2/5)5346worker1 # [ 96.283673] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 394 ms (attempt 3/5)5347lb # [ 96.610373] envoy[912]: conn peer=[2001:db8:1::1]:35414 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3405 tx=1923 ms=13905348lb # [ 96.614168] envoy[912]: conn peer=[2001:db8:1::1]:35420 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=555349lb # [ 96.617565] envoy[912]: conn peer=[2001:db8:1::1]:35422 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3863 tx=2137 ms=20565350lb # [ 96.621118] envoy[912]: conn peer=[2001:db8:1::1]:35424 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=585351lb # [ 96.624537] envoy[912]: conn peer=[2001:db8:1::1]:35428 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4310 tx=2371 ms=20495352lb # [ 96.628049] envoy[912]: conn peer=[2001:db8:1::1]:35440 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=195353lb # [ 96.632360] envoy[912]: conn peer=[2001:db8:1::1]:40164 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4198 tx=2015 ms=22545354lb # [ 96.635879] envoy[912]: conn peer=[2001:db8:1::1]:40170 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=575355worker1 # [ 96.685255] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 175 ms (attempt 4/5)5356worker1 # [ 96.862569] nix-grpc-daemon[783]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5357worker1 # [ 96.866265] nix-grpc-daemon[783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5358worker1 # [ 96.871773] nix-grpc-daemon[783]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5359worker1 # [ 96.985509] nix-daemon[869]: accepted connection from pid 1284, user nix-grpc-daemon (trusted)5360worker1 # [ 97.001972] nix-daemon[869]: reaped child process 1287, status = succeeded5361lb # [ 97.005773] niks3-server[991]: 2026/09/23 07:00:36 INFO Received uploads request method=POST path=/api/pending_closures5362worker1 # [ 97.019406] nix-grpc-daemon[883]: time=2026-09-23T07:00:35.298Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5363worker1 # [ 97.021371] nix-grpc-daemon[883]: time=2026-09-23T07:00:35.300Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5364lb # [ 97.041297] niks3-server[991]: 2026/09/23 07:00:36 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5365lb # [ 97.045768] niks3-server[991]: 2026/09/23 07:00:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5366worker1 # [ 97.057629] nix-grpc-daemon[883]: time=2026-09-23T07:00:35.337Z level=INFO msg="Uploading 1 narinfos"5367lb # [ 97.053452] niks3-server[991]: 2026/09/23 07:00:36 INFO Signed narinfos id=31 count=15368lb # [ 97.059509] niks3-server[991]: 2026/09/23 07:00:36 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5369lb # [ 97.069333] niks3-server[991]: 2026/09/23 07:00:36 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5370lb # [ 97.075413] niks3-server[991]: 2026/09/23 07:00:36 INFO Completed upload id=315371worker1 # [ 97.079622] nix-grpc-daemon[883]: time=2026-09-23T07:00:35.359Z level=INFO msg="Upload complete. (188ms)"5372lb # [ 97.078339] niks3-server[991]: 2026/09/23 07:00:36 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5373worker1 # [ 97.085391] nix-grpc-daemon[783]: ts=2026-09-23T07:00:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=1 paths=1 nar_bytes_in=5605374worker1 # [ 97.091776] nix-grpc-daemon[783]: ts=2026-09-23T07:00:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:437005375worker1 # [ 97.103879] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5376worker1 # [ 97.114134] nix-daemon[869]: reaped child process 1277, status = succeeded5377worker1 # [ 97.116705] nix-daemon[869]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5378client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5379client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5380client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5381worker1 # [ 97.743536] nix-daemon[869]: reaped child process 1291, status = succeeded5382worker1 # [ 97.835417] nix-daemon[869]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5383worker1 # [ 97.846884] nix-daemon[869]: reaped child process 1310, status = succeeded5384lb # [ 97.850787] niks3-server[991]: 2026/09/23 07:00:37 INFO Received uploads request method=POST path=/api/pending_closures5385worker1 # [ 97.863104] nix-grpc-daemon[883]: time=2026-09-23T07:00:36.142Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5386lb # [ 97.880892] niks3-server[991]: 2026/09/23 07:00:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5387worker1 # [ 97.890327] nix-grpc-daemon[883]: time=2026-09-23T07:00:36.169Z level=INFO msg="Uploading 1 narinfos"5388lb # [ 97.886739] niks3-server[991]: 2026/09/23 07:00:37 INFO Signed narinfos id=32 count=15389lb # [ 97.893773] niks3-server[991]: 2026/09/23 07:00:37 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5390lb # [ 97.901822] niks3-server[991]: 2026/09/23 07:00:37 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5391lb # [ 97.910885] niks3-server[991]: 2026/09/23 07:00:37 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5392lb # [ 97.918285] niks3-server[991]: 2026/09/23 07:00:37 INFO Completed upload id=325393worker1 # [ 97.925697] nix-grpc-daemon[883]: time=2026-09-23T07:00:36.205Z level=INFO msg="Upload complete. (202ms)"5394lb # [ 97.922567] niks3-server[991]: 2026/09/23 07:00:37 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5395worker1 # [ 97.931136] nix-grpc-daemon[783]: ts=2026-09-23T07:00:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=0 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=16 outputs=15396client # copying 1 paths...5397worker2 # [ 97.946529] nix-grpc-daemon[783]: ts=2026-09-23T07:00:37Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:408825398worker1 # [ 97.945567] nix-daemon[869]: reaped child process 1289, status = succeeded5399worker2 # [ 97.954252] nix-grpc-daemon[783]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5400client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5401worker1 # [ 97.977566] nix-grpc-daemon[783]: ts=2026-09-23T07:00:36Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:437005402worker1 # [ 97.982599] nix-grpc-daemon[783]: ts=2026-09-23T07:00:36Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43700 duration_s=0 paths=1 nar_bytes_out=1205403client # [ 98.062821] nix-daemon[1112]: reaped child process 1219, status = succeeded5404client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.31 seconds)5405client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&25406client # [ 98.169303] nix-daemon[1112]: accepted connection from pid 1238, user root (trusted)5407client # this derivation will be built:5408client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5409worker2 # [ 98.281152] nix-grpc-daemon[783]: ts=2026-09-23T07:00:37Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:408825410client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5411worker2 # [ 98.287935] nix-grpc-daemon[783]: ts=2026-09-23T07:00:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408825412worker2 # [ 98.298384] nix-grpc-daemon[880]: time=2026-09-23T07:00:37.483Z level=INFO msg="All 1 paths already cached"5413worker1 # [ 98.309895] nix-grpc-daemon[783]: ts=2026-09-23T07:00:36Z level=debug event=want client=16 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05414worker1 # [ 98.314769] nix-grpc-daemon[783]: ts=2026-09-23T07:00:36Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15415worker2 # [ 98.362954] nix-grpc-daemon[783]: ts=2026-09-23T07:00:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408825416worker2 # [ 98.375890] nix-grpc-daemon[783]: ts=2026-09-23T07:00:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408825417worker2 # [ 98.389111] nix-grpc-daemon[880]: time=2026-09-23T07:00:37.574Z level=INFO msg="All 1 paths already cached"5418client # copying 1 paths to 'lb:50051'...5419client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5420worker2 # [ 98.397295] nix-grpc-daemon[783]: ts=2026-09-23T07:00:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:408825421worker2 # [ 98.407879] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5422worker2 # [ 98.504293] nix-daemon[855]: accepted connection from pid 1117, user nix-grpc-daemon (trusted)5423worker2 # [ 98.516454] nix-daemon[855]: reaped child process 1120, status = succeeded5424lb # [ 98.516849] niks3-server[991]: 2026/09/23 07:00:37 INFO Received uploads request method=POST path=/api/pending_closures5425worker2 # [ 98.533371] nix-grpc-daemon[880]: time=2026-09-23T07:00:37.719Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5426worker2 # [ 98.534896] nix-grpc-daemon[880]: time=2026-09-23T07:00:37.720Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5427lb # [ 98.548616] niks3-server[991]: 2026/09/23 07:00:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5428lb # [ 98.554414] niks3-server[991]: 2026/09/23 07:00:37 INFO Signed narinfos id=33 count=15429worker2 # [ 98.565370] nix-grpc-daemon[880]: time=2026-09-23T07:00:37.750Z level=INFO msg="Uploading 1 narinfos"5430lb # [ 98.564564] niks3-server[991]: 2026/09/23 07:00:37 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5431lb # [ 98.571870] niks3-server[991]: 2026/09/23 07:00:37 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5432lb # [ 98.581744] niks3-server[991]: 2026/09/23 07:00:37 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5433lb # [ 98.589127] niks3-server[991]: 2026/09/23 07:00:37 INFO Completed upload id=335434worker2 # [ 98.597242] nix-grpc-daemon[880]: time=2026-09-23T07:00:37.782Z level=INFO msg="Upload complete. (178ms)"5435worker2 # [ 98.601542] nix-grpc-daemon[783]: ts=2026-09-23T07:00:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=0 paths=1 nar_bytes_in=5605436lb # [ 98.598343] niks3-server[991]: 2026/09/23 07:00:37 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5437worker2 # [ 98.607430] nix-grpc-daemon[783]: ts=2026-09-23T07:00:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408825438worker2 # [ 98.617747] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5439worker2 # [ 98.626343] nix-daemon[855]: reaped child process 1115, status = succeeded5440worker2 # [ 98.631438] nix-daemon[855]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5441client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5442client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5443client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5444worker2 # [ 99.863650] nix-daemon[855]: reaped child process 1124, status = succeeded5445worker2 # [ 99.948080] nix-daemon[855]: accepted connection from pid 1140, user nix-grpc-daemon (trusted)5446worker2 # [ 99.960775] nix-daemon[855]: reaped child process 1143, status = succeeded5447lb # [ 99.960521] niks3-server[991]: 2026/09/23 07:00:39 INFO Received uploads request method=POST path=/api/pending_closures5448worker2 # [ 99.977573] nix-grpc-daemon[880]: time=2026-09-23T07:00:39.163Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5449lb # [ 99.995595] niks3-server[991]: 2026/09/23 07:00:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5450lb # [ 100.003921] niks3-server[991]: 2026/09/23 07:00:39 INFO Signed narinfos id=34 count=15451worker2 # [ 100.015294] nix-grpc-daemon[880]: time=2026-09-23T07:00:39.201Z level=INFO msg="Uploading 1 narinfos"5452lb # [ 100.009315] niks3-server[991]: 2026/09/23 07:00:39 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5453lb # [ 100.017491] niks3-server[991]: 2026/09/23 07:00:39 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5454lb # [ 100.029819] niks3-server[991]: 2026/09/23 07:00:39 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5455lb # [ 100.034679] niks3-server[991]: 2026/09/23 07:00:39 INFO Completed upload id=345456worker2 # [ 100.044405] nix-grpc-daemon[880]: time=2026-09-23T07:00:39.230Z level=INFO msg="Upload complete. (205ms)"5457lb # [ 100.038721] niks3-server[991]: 2026/09/23 07:00:39 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5458worker2 # [ 100.050789] nix-grpc-daemon[783]: ts=2026-09-23T07:00:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=17 outputs=15459worker2 # [ 100.064402] nix-daemon[855]: reaped child process 1122, status = succeeded5460client # copying 1 paths...5461worker2 # [ 100.096226] nix-grpc-daemon[783]: ts=2026-09-23T07:00:39Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:408825462client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5463worker2 # [ 100.118233] nix-grpc-daemon[783]: ts=2026-09-23T07:00:39Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:408825464worker2 # [ 100.122056] nix-grpc-daemon[783]: ts=2026-09-23T07:00:39Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40882 duration_s=0 paths=1 nar_bytes_out=1205465client # [ 100.235327] nix-daemon[1112]: reaped child process 1241, status = succeeded5466client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.17 seconds)5467(finished: subtest: an input only one worker has still reaches the builder, in 7.54 seconds)5468subtest: upload whose reference the worker lacks is completed from the cache5469client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache5470client # this derivation will be built:5471client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5472client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5473client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache, in 0.25 seconds)5474client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache5475client # this derivation will be built:5476client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5477client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5478client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache, in 0.24 seconds)5479client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5480client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.10 seconds)5481worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5482worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.13 seconds)5483worker2 # [ 100.995465] systemd[1]: Stopping Nix worker-protocol over gRPC...5484worker2 # [ 101.862630] nix-grpc-daemon[783]: ts=2026-09-23T07:00:41Z level=info event=signal_exit5485worker1 # [ 103.390751] nix-grpc-daemon[783]: ts=2026-09-23T07:00:41Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:530525486worker1 # [ 103.397268] nix-grpc-daemon[783]: ts=2026-09-23T07:00:41Z level=info event=worker_gone addr=192.168.1.4:500515487worker2 # [ 103.397214] nix-grpc-daemon[783]: ts=2026-09-23T07:00:42Z level=info event=scheduler_restarting5488worker2 # [ 103.411450] nix-grpc-daemon[783]: ts=2026-09-23T07:00:42Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5489worker2 # [ 103.431475] nix-daemon[855]: reaped child process 857, status = succeeded5490worker2 # [ 103.742913] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5491worker2 # [ 103.745681] systemd[1]: Stopped Nix worker-protocol over gRPC.5492worker2 # [ 103.749551] systemd[1]: nix-grpc-daemon.service: Consumed 1.848s CPU time over 1min 27.042s wall clock time, 60.9M memory peak, 68.2K incoming IP traffic, 118.8K outgoing IP traffic.5493worker2 # [ 103.761356] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5494worker2 # [ 103.765401] systemd[1]: Closed Nix worker-protocol over gRPC.5495worker2 # [ 103.768745] systemd[1]: nix-grpc-daemon.socket: Consumed 311us CPU time over 1min 35.168s wall clock time, 256K memory peak, 87K incoming IP traffic, 78.7K outgoing IP traffic.5496lb: waiting until exactly ['worker1'] are healthy members of x86_64-linux5497lb: must succeed: curl -sf localhost:9901/clusters5498lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5499lb: (finished: waiting until exactly ['worker1'] are healthy members of x86_64-linux, in 0.05 seconds)5500client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25501worker1 # [ 103.965763] nix-grpc-daemon[783]: ts=2026-09-23T07:00:42Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:437005502worker1 # [ 103.968071] nix-grpc-daemon[783]: ts=2026-09-23T07:00:42Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:437005503client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5504client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.16 seconds)5505worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5506worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.02 seconds)5507worker2 # [ 104.052843] systemd[1]: Listening on Nix worker-protocol over gRPC.5508worker1 # [ 104.072668] systemd[1]: Stopping Nix worker-protocol over gRPC...5509worker1 # [ 104.822279] nix-grpc-daemon[783]: ts=2026-09-23T07:00:43Z level=info event=signal_exit5510worker2 # [ 105.043070] systemd[1]: Starting scheduler-token.service...5511worker2 # [ 105.104553] systemd[1]: scheduler-token.service: Deactivated successfully.5512worker2 # [ 105.108378] systemd[1]: Finished scheduler-token.service.5513worker2 # [ 105.110353] systemd[1]: scheduler-token.service: Consumed 34ms CPU time over 64ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5514worker2 # [ 105.117752] systemd[1]: Starting Nix worker-protocol over gRPC...5515worker2 # [ 105.195759] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:44Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5516worker2 # [ 105.197875] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:44Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5517worker2 # [ 105.201075] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:44Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15518worker2 # [ 105.205700] nix-grpc-daemon[1167]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5519worker2 # [ 105.207065] nix-grpc-daemon[1167]: E0000 00:00:1790146844.391203 1167 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5520worker2 # [ 105.212760] nix-grpc-daemon[1167]: E0000 00:00:1790146844.396749 1167 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5521worker2 # [ 105.216693] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:44Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5522worker2 # [ 105.218320] systemd[1]: Started Nix worker-protocol over gRPC.5523worker2 # [ 105.224065] nix-grpc-daemon[1167]: E0000 00:00:1790146844.409170 1209 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5524worker2 # [ 105.235759] nix-grpc-daemon[1167]: E0000 00:00:1790146844.421455 1209 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5525worker2 # [ 105.240054] nix-grpc-daemon[1167]: E0000 00:00:1790146844.423718 1209 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5526worker2 # [ 105.249937] nix-grpc-daemon[1167]: E0000 00:00:1790146844.435649 1209 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5527worker2 # [ 105.258812] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:44Z level=info event=scheduler_connected addr=lb:500515528worker2 # [ 105.260404] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5529lb # [ 105.385913] niks3-server[991]: 2026/09/23 07:00:44 INFO lead: released remote=192.168.1.3:560085530worker2 # [ 105.461689] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:44Z level=info event=scheduler_connected addr=lb:500515531worker2 # [ 105.465700] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5532worker2 # [ 105.666798] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:44Z level=info event=scheduler_connected addr=lb:500515533worker2 # [ 105.670756] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5534worker2 # [ 105.871795] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:45Z level=info event=scheduler_connected addr=lb:500515535worker2 # [ 105.875711] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5536worker2 # [ 106.076842] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:45Z level=info event=scheduler_connected addr=lb:500515537worker2 # [ 106.080748] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5538worker2 # [ 106.283203] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:45Z level=info event=scheduler_connected addr=lb:500515539worker2 # [ 106.286489] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5540worker1 # [ 106.390485] nix-grpc-daemon[783]: ts=2026-09-23T07:00:44Z level=info event=scheduler_restarting5541lb # [ 106.387143] envoy[912]: [2026-09-23 07:00:45.592][912][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream}5542worker1 # [ 106.395603] nix-grpc-daemon[783]: ts=2026-09-23T07:00:44Z level=info event=scheduler_restarting addr=lb:500515543worker1 # [ 106.401818] nix-grpc-daemon[783]: ts=2026-09-23T07:00:44Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:530525544worker1 # [ 106.405359] nix-grpc-daemon[783]: ts=2026-09-23T07:00:44Z level=info event=worker_gone addr=192.168.1.3:500515545worker1 # [ 106.429603] nix-daemon[869]: reaped child process 871, status = succeeded5546worker2 # [ 106.484476] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:45Z level=info event=scheduler_connected addr=lb:500515547worker2 # [ 106.488492] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5548worker1 # [ 106.609718] nix-grpc-daemon[783]: ts=2026-09-23T07:00:44Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5549lb # [ 106.609940] envoy[912]: conn peer=[2001:db8:1::1]:40184 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=421 tx=328 ms=165550lb # [ 106.613692] envoy[912]: conn peer=[2001:db8:1::1]:40174 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3710 tx=2105 ms=20245551lb # [ 106.617230] envoy[912]: conn peer=[2001:db8:1::1]:40204 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=545552lb # [ 106.620596] envoy[912]: conn peer=[2001:db8:1::1]:40196 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3768 tx=2099 ms=18905553lb # [ 106.624268] envoy[912]: conn peer=[2001:db8:1::4]:54106 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=2607 tx=851 ms=707455554lb # [ 106.627907] envoy[912]: conn peer=[2001:db8:1::1]:33916 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=115555lb # [ 106.631390] envoy[912]: conn peer=[2001:db8:1::3]:33612 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2581 tx=1124 ms=740485556worker2 # [ 106.689747] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:45Z level=info event=scheduler_connected addr=lb:500515557worker2 # [ 106.693677] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5558worker2 # [ 106.894841] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:46Z level=info event=scheduler_connected addr=lb:500515559worker2 # [ 106.898862] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5560worker1 # [ 107.023457] nix-grpc-daemon[783]: E0000 00:00:1790146845.302811 829 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 20, started true, shutdown true5561worker2 # [ 107.100187] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:46Z level=info event=scheduler_connected addr=lb:500515562worker2 # [ 107.104169] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5563worker1 # [ 107.273924] nix-grpc-daemon[783]: E0000 00:00:1790146845.553306 829 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5564worker2 # [ 107.305950] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:46Z level=info event=scheduler_connected addr=lb:500515565worker2 # [ 107.309832] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5566worker2 # [ 107.511233] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:46Z level=info event=scheduler_connected addr=lb:500515567worker2 # [ 107.515178] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5568worker1 # [ 107.700339] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5569worker1 # [ 107.703332] systemd[1]: Stopped Nix worker-protocol over gRPC.5570worker1 # [ 107.707400] systemd[1]: nix-grpc-daemon.service: Consumed 2.673s CPU time over 1min 31.085s wall clock time, 67.7M memory peak, 102K incoming IP traffic, 218.8K outgoing IP traffic.5571worker2 # [ 107.716306] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:46Z level=info event=scheduler_connected addr=lb:500515572worker2 # [ 107.720212] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5573worker1 # [ 107.719208] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5574worker1 # [ 107.723451] systemd[1]: Closed Nix worker-protocol over gRPC.5575worker1 # [ 107.727458] systemd[1]: nix-grpc-daemon.socket: Consumed 312us CPU time over 1min 40.301s wall clock time, 256K memory peak, 299.1K incoming IP traffic, 108.8K outgoing IP traffic.5576lb: waiting until exactly ['worker2'] are healthy members of x86_64-linux5577lb: must succeed: curl -sf localhost:9901/clusters5578lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5579worker2 # [ 107.921466] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:47Z level=info event=scheduler_connected addr=lb:500515580worker2 # [ 107.925516] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5581worker2 # [ 108.126708] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:47Z level=info event=scheduler_connected addr=lb:500515582worker2 # [ 108.130714] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5583worker2 # [ 108.332594] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:47Z level=info event=scheduler_connected addr=lb:500515584worker2 # [ 108.336409] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5585worker2 # [ 108.537874] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:47Z level=info event=scheduler_connected addr=lb:500515586worker2 # [ 108.541879] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5587worker2 # [ 108.742994] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:47Z level=info event=scheduler_connected addr=lb:500515588worker2 # [ 108.747064] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5589lb: must succeed: curl -sf localhost:9901/clusters5590lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5591worker2 # [ 108.948268] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:48Z level=info event=scheduler_connected addr=lb:500515592worker2 # [ 108.952242] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5593worker2 # [ 109.153242] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:48Z level=info event=scheduler_connected addr=lb:500515594worker2 # [ 109.157210] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5595worker2 # [ 109.359116] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:48Z level=info event=scheduler_connected addr=lb:500515596worker2 # [ 109.362869] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5597worker2 # [ 109.564321] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:48Z level=info event=scheduler_connected addr=lb:500515598worker2 # [ 109.568342] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5599worker2 # [ 109.769407] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:48Z level=info event=scheduler_connected addr=lb:500515600worker2 # [ 109.773339] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5601lb: must succeed: curl -sf localhost:9901/clusters5602lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5603worker2 # [ 109.974474] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:49Z level=info event=scheduler_connected addr=lb:500515604worker2 # [ 109.978433] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5605worker2 # [ 110.179612] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:49Z level=info event=scheduler_connected addr=lb:500515606worker2 # [ 110.183542] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5607lb # [ 110.217229] niks3-server[991]: 2026/09/23 07:00:49 INFO lead: acquired remote=192.168.1.4:540745608worker2 # [ 110.228261] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:49Z level=info event=scheduler_take_over5609worker2 # [ 110.385394] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:49Z level=info event=scheduler_connected addr=lb:500515610worker2 # [ 110.398141] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:49Z level=info event=oidc_jwks provider=mock keys=15611worker2 # [ 110.402397] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:49Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:456925612worker2 # [ 110.406353] nix-grpc-daemon[1167]: ts=2026-09-23T07:00:49Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05613lb: must succeed: curl -sf localhost:9901/clusters5614lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5615lb: must succeed: curl -sf localhost:9901/clusters5616lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5617lb: must succeed: curl -sf localhost:9901/clusters5618lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5619lb: must succeed: curl -sf localhost:9901/clusters5620lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5621lb: must succeed: curl -sf localhost:9901/clusters5622lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5623lb: must succeed: curl -sf localhost:9901/clusters5624lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5625lb: must succeed: curl -sf localhost:9901/clusters5626lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5627lb: must succeed: curl -sf localhost:9901/clusters5628lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5629lb: must succeed: curl -sf localhost:9901/clusters5630lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5631lb: must succeed: curl -sf localhost:9901/clusters5632lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5633lb: must succeed: curl -sf localhost:9901/clusters5634lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5635lb: must succeed: curl -sf localhost:9901/clusters5636lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5637lb: must succeed: curl -sf localhost:9901/clusters5638lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5639lb: must succeed: curl -sf localhost:9901/clusters5640lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5641lb: must succeed: curl -sf localhost:9901/clusters5642lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5643lb: must succeed: curl -sf localhost:9901/clusters5644lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5645lb: must succeed: curl -sf localhost:9901/clusters5646worker2 # [ 127.662736] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:06Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:393745647lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5648lb: (finished: waiting until exactly ['worker2'] are healthy members of x86_64-linux, in 19.97 seconds)5649client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25650worker2 # [ 127.865798] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393965651worker2 # [ 127.877489] nix-daemon[855]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5652worker2 # [ 127.893829] nix-grpc-daemon[1167]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 1/5)5653worker2 # [ 127.916963] nix-grpc-daemon[1167]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 114 ms (attempt 2/5)5654worker2 # [ 128.038751] nix-grpc-daemon[1167]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 309 ms (attempt 3/5)5655worker2 # [ 128.349975] nix-grpc-daemon[1167]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 208 ms (attempt 4/5)5656worker2 # [ 128.560244] nix-grpc-daemon[1167]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5657worker2 # [ 128.563635] nix-grpc-daemon[1167]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5658client # copying 2 paths...5659client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5660worker2 # [ 128.586611] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393965661client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5662worker2 # [ 128.597540] nix-daemon[855]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5663worker2 # [ 128.621912] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:07Z level=info event=niks3_push_started pid=12285664worker2 # [ 128.725486] nix-grpc-daemon[1228]: time=2026-09-23T07:01:07.910Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""5665worker2 # [ 128.811082] nix-daemon[855]: accepted connection from pid 1241, user nix-grpc-daemon (trusted)5666worker2 # [ 128.822348] nix-daemon[855]: reaped child process 1244, status = succeeded5667lb # [ 128.823746] niks3-server[991]: 2026/09/23 07:01:08 INFO Received uploads request method=POST path=/api/pending_closures5668lb # [ 128.835113] niks3-server[991]: 2026/09/23 07:01:08 INFO Received uploads request method=POST path=/api/pending_closures5669worker2 # [ 128.850561] nix-grpc-daemon[1228]: time=2026-09-23T07:01:08.036Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5670worker2 # [ 128.852181] nix-grpc-daemon[1228]: time=2026-09-23T07:01:08.037Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5671worker2 # [ 128.854131] nix-grpc-daemon[1228]: time=2026-09-23T07:01:08.039Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5672lb # [ 128.883830] niks3-server[991]: 2026/09/23 07:01:08 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5673lb # [ 128.898421] niks3-server[991]: 2026/09/23 07:01:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5674lb # [ 128.905127] niks3-server[991]: 2026/09/23 07:01:08 INFO Signed narinfos id=36 count=25675lb # [ 128.909131] niks3-server[991]: 2026/09/23 07:01:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5676lb # [ 128.911550] niks3-server[991]: 2026/09/23 07:01:08 INFO Signed narinfos id=35 count=15677worker2 # [ 128.922259] nix-grpc-daemon[1228]: time=2026-09-23T07:01:08.106Z level=INFO msg="Uploading 3 narinfos"5678lb # [ 128.917697] niks3-server[991]: 2026/09/23 07:01:08 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5679lb # [ 128.922561] niks3-server[991]: 2026/09/23 07:01:08 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5680lb # [ 128.936303] niks3-server[991]: 2026/09/23 07:01:08 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5681lb # [ 128.953118] niks3-server[991]: 2026/09/23 07:01:08 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5682lb # [ 128.954664] niks3-server[991]: 2026/09/23 07:01:08 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5683lb # [ 128.956171] niks3-server[991]: 2026/09/23 07:01:08 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5684lb # [ 128.958709] niks3-server[991]: 2026/09/23 07:01:08 INFO Completed upload id=365685lb # [ 128.960800] niks3-server[991]: 2026/09/23 07:01:08 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5686lb # [ 128.964056] niks3-server[991]: 2026/09/23 07:01:08 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5687lb # [ 128.966596] niks3-server[991]: 2026/09/23 07:01:08 INFO Completed upload id=355688worker2 # [ 128.974636] nix-grpc-daemon[1228]: time=2026-09-23T07:01:08.160Z level=INFO msg="Upload complete. (245ms)"5689worker2 # [ 128.980597] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39396 duration_s=0 paths=2 nar_bytes_in=3125690worker2 # [ 128.995130] nix-daemon[855]: reaped child process 1226, status = succeeded5691client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.30 seconds)5692worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5693worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.02 seconds)5694worker1 # [ 129.075807] systemd[1]: Listening on Nix worker-protocol over gRPC.5695worker1 # [ 129.083579] systemd[1]: Starting Nix worker-protocol over gRPC...5696worker1 # [ 129.176365] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:07Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5697worker1 # [ 129.178544] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:07Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5698worker1 # [ 129.181849] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:07Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15699worker1 # [ 129.187058] nix-grpc-daemon[1335]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5700worker1 # [ 129.188399] nix-grpc-daemon[1335]: E0000 00:00:1790146867.466114 1335 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5701worker1 # [ 129.193677] nix-grpc-daemon[1335]: E0000 00:00:1790146867.471654 1335 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5702worker1 # [ 129.197593] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:07Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5703worker1 # [ 129.200073] systemd[1]: Started Nix worker-protocol over gRPC.5704worker1 # [ 129.204258] nix-grpc-daemon[1335]: E0000 00:00:1790146867.483929 1378 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5705lb: waiting until exactly ['worker1', 'worker2'] are healthy members of x86_64-linux5706lb: must succeed: curl -sf localhost:9901/clusters5707worker1 # [ 129.217066] nix-grpc-daemon[1335]: E0000 00:00:1790146867.496666 1378 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5708worker1 # [ 129.221139] nix-grpc-daemon[1335]: E0000 00:00:1790146867.498994 1378 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5709worker1 # [ 129.231074] nix-grpc-daemon[1335]: E0000 00:00:1790146867.510744 1378 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5710worker1 # [ 129.241950] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:07Z level=info event=scheduler_connected addr=lb:500515711worker2 # [ 129.248137] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:08Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:456925712worker2 # [ 129.251862] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:08Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05713lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5714lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of x86_64-linux, in 0.06 seconds)5715(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 29.04 seconds)5716subtest: a niks3 restart does not cost the leader its role5717lb: must succeed: curl -sf localhost:9901/clusters5718lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5719lb: must succeed: curl -sf localhost:9901/clusters5720lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5721lb # [ 129.337335] systemd[1]: Stopping niks3 server...5722lb # [ 129.338338] niks3-server[991]: 2026/09/23 07:01:08 INFO Shutdown signal received, draining in-flight requests timeout=10s5723lb # [ 129.341543] niks3-server[991]: 2026/09/23 07:01:08 INFO lead: released remote=192.168.1.4:540745724worker1 # [ 129.345692] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:07Z level=info event=scheduler_lock_lost status=2005725worker2 # [ 129.349801] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:08Z level=info event=scheduler_lock_lost status=2005726lb # [ 129.359428] systemd[1]: niks3.service: Deactivated successfully.5727lb # [ 129.360343] systemd[1]: Stopped niks3 server.5728lb # [ 129.361713] systemd[1]: niks3.service: Consumed 804ms CPU time over 1min 47.197s wall clock time, 28.8M memory peak, 100.5K incoming IP traffic, 143.5K outgoing IP traffic.5729lb # [ 129.369466] systemd[1]: Starting niks3 server...5730lb # [ 129.426360] niks3-server[1225]: 2026/09/23 07:01:08 goose: no migrations to run. current version: 202609200000005731lb # [ 129.432201] niks3-server[1225]: 2026/09/23 07:01:08 OK 1_commit_pending_closure.sql (6.68ms)5732lb # [ 129.436298] niks3-server[1225]: 2026/09/23 07:01:08 OK 2_object_stats_trigger.sql (3.67ms)5733lb # [ 129.437451] niks3-server[1225]: 2026/09/23 07:01:08 goose: up to current file version: 25734lb # [ 129.439417] niks3-server[1225]: 2026/09/23 07:01:08 INFO Loaded signing key name=farm-test-1 path=/nix/store/hk0nhypl37jvkn3gr36v3y7qx593p39i-key5735lb # [ 129.451507] niks3-server[1225]: 2026/09/23 07:01:08 INFO Read proxy enabled — serving cache objects from S35736lb # [ 129.452833] niks3-server[1225]: 2026/09/23 07:01:08 INFO Using socket-activated listener address=0.0.0.0:57515737lb # [ 129.454651] niks3-server[1225]: 2026/09/23 07:01:08 INFO systemd watchdog enabled interval=15s5738lb # [ 129.457084] systemd[1]: Started niks3 server.5739lb # [ 129.458433] niks3-server[1225]: 2026/09/23 07:01:08 INFO Starting HTTP server address=0.0.0.0:57515740lb: waiting for unit niks3.service5741lb: (finished: waiting for unit niks3.service, in 0.02 seconds)5742lb: must succeed: curl -sf localhost:9901/clusters5743lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5744worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5745worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5746lb: must succeed: curl -sf localhost:9901/clusters5747lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5748lb: must succeed: curl -sf localhost:9901/clusters5749lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5750worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5751worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.04 seconds)5752client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&25753worker1 # [ 129.803062] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:445585754worker1 # [ 129.814586] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)5755client # don't know how to build these paths:5756client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5757worker2 # [ 129.833858] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393965758worker2 # [ 129.861174] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:09Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=05759worker2 # [ 129.866168] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:09Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.4:50051 clients=15760worker2 # [ 129.906435] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393965761worker2 # [ 129.919780] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393965762client # copying 1 paths to 'lb:50051'...5763client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...5764worker2 # [ 129.932826] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393965765worker2 # [ 129.942344] nix-daemon[855]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5766worker2 # [ 130.045214] nix-daemon[855]: accepted connection from pid 1261, user nix-grpc-daemon (trusted)5767worker2 # [ 130.058025] nix-daemon[855]: reaped child process 1264, status = succeeded5768lb # [ 130.056630] niks3-server[1225]: 2026/09/23 07:01:09 INFO Received uploads request method=POST path=/api/pending_closures5769worker2 # [ 130.076712] nix-grpc-daemon[1228]: time=2026-09-23T07:01:09.262Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5770worker2 # [ 130.078913] nix-grpc-daemon[1228]: time=2026-09-23T07:01:09.264Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"5771lb # [ 130.096240] niks3-server[1225]: 2026/09/23 07:01:09 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst5772lb # [ 130.102979] niks3-server[1225]: 2026/09/23 07:01:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5773lb # [ 130.118131] niks3-server[1225]: 2026/09/23 07:01:09 INFO Signed narinfos id=37 count=15774worker2 # [ 130.129315] nix-grpc-daemon[1228]: time=2026-09-23T07:01:09.314Z level=INFO msg="Uploading 1 narinfos"5775lb # [ 130.134990] niks3-server[1225]: 2026/09/23 07:01:09 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5776lb # [ 130.144494] niks3-server[1225]: 2026/09/23 07:01:09 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo5777lb # [ 130.147393] niks3-server[1225]: 2026/09/23 07:01:09 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls5778lb # [ 130.153657] niks3-server[1225]: 2026/09/23 07:01:09 INFO Completed upload id=375779worker2 # [ 130.161629] nix-grpc-daemon[1228]: time=2026-09-23T07:01:09.347Z level=INFO msg="Upload complete. (206ms)"5780worker2 # [ 130.166876] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39396 duration_s=0 paths=1 nar_bytes_in=5365781worker2 # [ 130.172951] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393965782worker2 # [ 130.185108] nix-daemon[855]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5783worker2 # [ 130.196243] nix-daemon[855]: reaped child process 1259, status = succeeded5784client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...5785worker2 # [ 130.201712] nix-daemon[855]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5786client # farm-a> worker2: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5787lb # [ 130.349327] niks3-server[1225]: 2026/09/23 07:01:09 INFO lead: acquired remote=192.168.1.4:562105788client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5789client # farm-a> LOG-farm-a-after-niks3-restart5790worker2 # [ 131.216330] nix-daemon[855]: reaped child process 1268, status = succeeded5791worker2 # [ 131.293700] nix-daemon[855]: accepted connection from pid 1286, user nix-grpc-daemon (trusted)5792worker2 # [ 131.306869] nix-daemon[855]: reaped child process 1289, status = succeeded5793lb # [ 131.305939] niks3-server[1225]: 2026/09/23 07:01:10 INFO Received uploads request method=POST path=/api/pending_closures5794worker2 # [ 131.327275] nix-grpc-daemon[1228]: time=2026-09-23T07:01:10.512Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5795worker2 # [ 131.328630] nix-grpc-daemon[1228]: time=2026-09-23T07:01:10.512Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"5796lb # [ 131.352443] niks3-server[1225]: 2026/09/23 07:01:10 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls5797lb # [ 131.360317] niks3-server[1225]: 2026/09/23 07:01:10 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5798lb # [ 131.366902] niks3-server[1225]: 2026/09/23 07:01:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5799worker2 # [ 131.379576] nix-grpc-daemon[1228]: time=2026-09-23T07:01:10.565Z level=INFO msg="Uploading 1 narinfos"5800lb # [ 131.373249] niks3-server[1225]: 2026/09/23 07:01:10 INFO Signed narinfos id=38 count=15801lb # [ 131.381521] niks3-server[1225]: 2026/09/23 07:01:10 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5802lb # [ 131.383512] niks3-server[1225]: 2026/09/23 07:01:10 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5803lb # [ 131.390822] niks3-server[1225]: 2026/09/23 07:01:10 INFO Completed upload id=385804worker2 # [ 131.398757] nix-grpc-daemon[1228]: time=2026-09-23T07:01:10.584Z level=INFO msg="Upload complete. (186ms)"5805lb # [ 131.394572] niks3-server[1225]: 2026/09/23 07:01:10 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo5806worker2 # [ 131.406512] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39396 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=15807worker2 # [ 131.418469] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:10Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=05808worker2 # [ 131.423199] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:10Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.4:50051 clients=15809worker2 # [ 131.433242] nix-daemon[855]: reaped child process 1266, status = succeeded5810worker2 # [ 131.462706] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393965811worker2 # [ 131.476314] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393965812worker2 # [ 131.491804] nix-grpc-daemon[1228]: time=2026-09-23T07:01:10.677Z level=INFO msg="All 1 paths already cached"5813client # copying 1 paths to 'lb:50051'...5814client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...5815worker2 # [ 131.500918] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393965816worker2 # [ 131.511179] nix-daemon[855]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5817worker2 # [ 131.605984] nix-daemon[855]: accepted connection from pid 1295, user nix-grpc-daemon (trusted)5818worker2 # [ 131.619927] nix-daemon[855]: reaped child process 1298, status = succeeded5819lb # [ 131.616874] niks3-server[1225]: 2026/09/23 07:01:10 INFO Received uploads request method=POST path=/api/pending_closures5820worker2 # [ 131.636391] nix-grpc-daemon[1228]: time=2026-09-23T07:01:10.822Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5821worker2 # [ 131.637885] nix-grpc-daemon[1228]: time=2026-09-23T07:01:10.823Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"5822lb # [ 131.655176] niks3-server[1225]: 2026/09/23 07:01:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5823lb # [ 131.661677] niks3-server[1225]: 2026/09/23 07:01:10 INFO Signed narinfos id=39 count=15824worker2 # [ 131.672247] nix-grpc-daemon[1228]: time=2026-09-23T07:01:10.858Z level=INFO msg="Uploading 1 narinfos"5825lb # [ 131.673181] niks3-server[1225]: 2026/09/23 07:01:10 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst5826lb # [ 131.677836] niks3-server[1225]: 2026/09/23 07:01:10 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls5827lb # [ 131.693718] niks3-server[1225]: 2026/09/23 07:01:10 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5828lb # [ 131.696673] niks3-server[1225]: 2026/09/23 07:01:10 INFO Completed upload id=395829worker2 # [ 131.704660] nix-grpc-daemon[1228]: time=2026-09-23T07:01:10.890Z level=INFO msg="Upload complete. (182ms)"5830lb # [ 131.702081] niks3-server[1225]: 2026/09/23 07:01:10 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo5831worker2 # [ 131.709452] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39396 duration_s=0 paths=1 nar_bytes_in=6565832worker2 # [ 131.715376] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393965833worker2 # [ 131.725755] nix-daemon[855]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5834worker2 # [ 131.734451] nix-daemon[855]: reaped child process 1293, status = succeeded5835client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...5836worker2 # [ 131.740895] nix-daemon[855]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5837client # farm-b> worker2: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5838client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5839client # farm-b> LOG-farm-b-after-niks3-restart5840worker2 # [ 132.555123] nix-daemon[855]: reaped child process 1302, status = succeeded5841worker2 # [ 132.630040] nix-daemon[855]: accepted connection from pid 1318, user nix-grpc-daemon (trusted)5842worker2 # [ 132.642243] nix-daemon[855]: reaped child process 1321, status = succeeded5843lb # [ 132.640655] niks3-server[1225]: 2026/09/23 07:01:11 INFO Received uploads request method=POST path=/api/pending_closures5844worker2 # [ 132.658413] nix-grpc-daemon[1228]: time=2026-09-23T07:01:11.843Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5845worker2 # [ 132.659891] nix-grpc-daemon[1228]: time=2026-09-23T07:01:11.845Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"5846lb # [ 132.687228] niks3-server[1225]: 2026/09/23 07:01:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5847worker2 # [ 132.705128] nix-grpc-daemon[1228]: time=2026-09-23T07:01:11.890Z level=INFO msg="Uploading 1 narinfos"5848lb # [ 132.697266] niks3-server[1225]: 2026/09/23 07:01:11 INFO Signed narinfos id=40 count=15849lb # [ 132.703610] niks3-server[1225]: 2026/09/23 07:01:11 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst5850lb # [ 132.712252] niks3-server[1225]: 2026/09/23 07:01:11 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5851lb # [ 132.720570] niks3-server[1225]: 2026/09/23 07:01:11 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5852lb # [ 132.728168] niks3-server[1225]: 2026/09/23 07:01:11 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls5853lb # [ 132.733398] niks3-server[1225]: 2026/09/23 07:01:11 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo5854lb # [ 132.737527] niks3-server[1225]: 2026/09/23 07:01:11 INFO Completed upload id=405855worker2 # [ 132.745536] nix-grpc-daemon[1228]: time=2026-09-23T07:01:11.931Z level=INFO msg="Upload complete. (199ms)"5856worker2 # [ 132.755106] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39396 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=15857worker2 # [ 132.763962] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:11Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=05858worker2 # [ 132.769403] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:11Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.4:50051 clients=15859worker2 # [ 132.779213] nix-daemon[855]: reaped child process 1300, status = succeeded5860worker2 # [ 132.808761] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393965861worker2 # [ 132.821774] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:393965862worker2 # [ 132.835740] nix-grpc-daemon[1228]: time=2026-09-23T07:01:12.021Z level=INFO msg="All 2 paths already cached"5863client # copying 1 paths to 'lb:50051'...5864client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...5865worker2 # [ 132.845144] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:393965866worker2 # [ 132.853563] nix-daemon[855]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5867worker2 # [ 132.949830] nix-daemon[855]: accepted connection from pid 1327, user nix-grpc-daemon (trusted)5868worker2 # [ 132.962710] nix-daemon[855]: reaped child process 1330, status = succeeded5869lb # [ 132.962148] niks3-server[1225]: 2026/09/23 07:01:12 INFO Received uploads request method=POST path=/api/pending_closures5870worker2 # [ 132.981480] nix-grpc-daemon[1228]: time=2026-09-23T07:01:12.167Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5871worker2 # [ 132.982998] nix-grpc-daemon[1228]: time=2026-09-23T07:01:12.168Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"5872lb # [ 133.002794] niks3-server[1225]: 2026/09/23 07:01:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5873lb # [ 133.007499] niks3-server[1225]: 2026/09/23 07:01:12 INFO Signed narinfos id=41 count=15874worker2 # [ 133.018315] nix-grpc-daemon[1228]: time=2026-09-23T07:01:12.204Z level=INFO msg="Uploading 1 narinfos"5875lb # [ 133.013793] niks3-server[1225]: 2026/09/23 07:01:12 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst5876lb # [ 133.026806] niks3-server[1225]: 2026/09/23 07:01:12 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls5877lb # [ 133.035143] niks3-server[1225]: 2026/09/23 07:01:12 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5878lb # [ 133.040821] niks3-server[1225]: 2026/09/23 07:01:12 INFO Completed upload id=415879worker2 # [ 133.050330] nix-grpc-daemon[1228]: time=2026-09-23T07:01:12.235Z level=INFO msg="Upload complete. (183ms)"5880lb # [ 133.044826] niks3-server[1225]: 2026/09/23 07:01:12 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo5881worker2 # [ 133.055413] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39396 duration_s=0 paths=1 nar_bytes_in=7845882worker2 # [ 133.061176] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:393965883worker2 # [ 133.071626] nix-daemon[855]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5884worker2 # [ 133.081300] nix-daemon[855]: reaped child process 1325, status = succeeded5885worker2 # [ 133.088990] nix-daemon[855]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5886client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...5887client # farm-top> worker2: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5888client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5889client # farm-top> LOG-farm-top-after-niks3-restart5890worker2 # [ 133.703915] nix-daemon[855]: reaped child process 1334, status = succeeded5891worker2 # [ 133.776499] nix-daemon[855]: accepted connection from pid 1350, user nix-grpc-daemon (trusted)5892worker2 # [ 133.788963] nix-daemon[855]: reaped child process 1353, status = succeeded5893lb # [ 133.789314] niks3-server[1225]: 2026/09/23 07:01:12 INFO Received uploads request method=POST path=/api/pending_closures5894worker2 # [ 133.807096] nix-grpc-daemon[1228]: time=2026-09-23T07:01:12.992Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5895worker2 # [ 133.808543] nix-grpc-daemon[1228]: time=2026-09-23T07:01:12.994Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"5896lb # [ 133.836184] niks3-server[1225]: 2026/09/23 07:01:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5897worker2 # [ 133.855739] nix-grpc-daemon[1228]: time=2026-09-23T07:01:13.041Z level=INFO msg="Uploading 1 narinfos"5898lb # [ 133.847724] niks3-server[1225]: 2026/09/23 07:01:13 INFO Signed narinfos id=42 count=15899lb # [ 133.852089] niks3-server[1225]: 2026/09/23 07:01:13 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5900lb # [ 133.856342] niks3-server[1225]: 2026/09/23 07:01:13 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls5901lb # [ 133.864560] niks3-server[1225]: 2026/09/23 07:01:13 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst5902lb # [ 133.868680] niks3-server[1225]: 2026/09/23 07:01:13 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5903lb # [ 133.873191] niks3-server[1225]: 2026/09/23 07:01:13 INFO Completed upload id=425904worker2 # [ 133.881256] nix-grpc-daemon[1228]: time=2026-09-23T07:01:13.066Z level=INFO msg="Upload complete. (186ms)"5905lb # [ 133.875527] niks3-server[1225]: 2026/09/23 07:01:13 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo5906worker2 # [ 133.888769] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39396 duration_s=0 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=15907worker2 # [ 133.903336] nix-daemon[855]: reaped child process 1332, status = succeeded5908client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&2, in 4.27 seconds)5909client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath5910client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath, in 0.12 seconds)5911(finished: subtest: a niks3 restart does not cost the leader its role, in 4.78 seconds)5912subtest: scheduler down: the other node takes the lock and keeps it5913lb: must succeed: curl -sf localhost:9901/clusters5914lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5915lb: must succeed: curl -sf localhost:9901/clusters5916lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5917lb: must succeed: curl -sf localhost:9901/clusters5918lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5919lb: must succeed: curl -sf localhost:9901/clusters5920lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5921worker2 # [ 134.216495] systemd[1]: Stopping Nix worker-protocol over gRPC...5922worker2 # [ 134.222939] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:13Z level=info event=signal_exit5923lb # [ 134.360262] niks3-server[1225]: 2026/09/23 07:01:13 INFO lead: released remote=192.168.1.4:562105924worker2 # [ 135.369096] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:14Z level=info event=scheduler_restarting5925lb # [ 135.360567] envoy[912]: [2026-09-23 07:01:14.566][912][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream}5926worker2 # [ 135.375691] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:14Z level=info event=scheduler_restarting addr=lb:500515927worker1 # [ 135.372917] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:13Z level=info event=scheduler_restarting addr=lb:500515928worker2 # [ 135.382191] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:14Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:456925929worker2 # [ 135.386253] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:14Z level=info event=worker_gone addr=192.168.1.4:500515930worker2 # [ 135.389495] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:14Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:456925931worker2 # [ 135.393196] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:14Z level=info event=worker_gone addr=192.168.1.3:500515932worker2 # [ 135.414408] nix-daemon[855]: reaped child process 1220, status = succeeded5933worker1 # [ 135.577376] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:13Z level=info event=scheduler_connected addr=lb:500515934worker1 # [ 135.581480] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5935worker2 # [ 135.588230] nix-grpc-daemon[1167]: ts=2026-09-23T07:01:14Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5936worker2 # [ 135.706324] nix-grpc-daemon[1167]: E0000 00:00:1790146874.891738 1213 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true5937worker1 # [ 135.782478] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:14Z level=info event=scheduler_connected addr=lb:500515938worker1 # [ 135.786494] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5939worker1 # [ 135.987374] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:14Z level=info event=scheduler_connected addr=lb:500515940worker1 # [ 135.991415] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5941worker1 # [ 136.192288] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:14Z level=info event=scheduler_connected addr=lb:500515942worker1 # [ 136.196723] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5943worker2 # [ 136.286082] nix-grpc-daemon[1167]: E0000 00:00:1790146875.471510 1213 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5944worker1 # [ 136.398589] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:14Z level=info event=scheduler_connected addr=lb:500515945worker1 # [ 136.402536] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5946worker1 # [ 136.603352] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:14Z level=info event=scheduler_connected addr=lb:500515947worker1 # [ 136.607417] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5948lb # [ 136.612309] envoy[912]: conn peer=[2001:db8:1::1]:36534 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1002 tx=281 ms=11285949lb # [ 136.616231] envoy[912]: conn peer=[2001:db8:1::1]:36550 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9636 tx=4362 ms=41015950lb # [ 136.619715] envoy[912]: conn peer=[2001:db8:1::4]:44882 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=22440 tx=1311 ms=303255951worker1 # [ 136.808232] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:15Z level=info event=scheduler_connected addr=lb:500515952worker1 # [ 136.812289] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5953worker1 # [ 137.013312] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:15Z level=info event=scheduler_connected addr=lb:500515954worker1 # [ 137.017313] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5955worker1 # [ 137.217914] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:15Z level=info event=scheduler_connected addr=lb:500515956worker1 # [ 137.221898] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5957worker1 # [ 137.423572] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:15Z level=info event=scheduler_connected addr=lb:500515958worker1 # [ 137.427492] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5959worker2 # [ 137.590577] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5960worker2 # [ 137.594584] systemd[1]: Stopped Nix worker-protocol over gRPC.5961worker2 # [ 137.598201] systemd[1]: nix-grpc-daemon.service: Consumed 971ms CPU time over 32.476s wall clock time, 38.7M memory peak, 50.5K incoming IP traffic, 106.2K outgoing IP traffic.5962worker2 # [ 137.609960] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5963worker2 # [ 137.614353] systemd[1]: Closed Nix worker-protocol over gRPC.5964worker2 # [ 137.618679] systemd[1]: nix-grpc-daemon.socket: Consumed 286us CPU time over 33.564s wall clock time, 256K memory peak, 218.7K incoming IP traffic, 34.6K outgoing IP traffic.5965worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5966worker1 # [ 137.628438] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:15Z level=info event=scheduler_connected addr=lb:500515967worker1 # [ 137.633465] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5968worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.06 seconds)5969worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5970worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)5971worker1 # [ 137.831204] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:16Z level=info event=scheduler_connected addr=lb:500515972worker1 # [ 137.835206] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5973worker1 # [ 138.035946] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:16Z level=info event=scheduler_connected addr=lb:500515974worker1 # [ 138.040067] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5975worker1 # [ 138.241430] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:16Z level=info event=scheduler_connected addr=lb:500515976worker1 # [ 138.245473] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5977worker1 # [ 138.447183] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:16Z level=info event=scheduler_connected addr=lb:500515978worker1 # [ 138.451118] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5979worker1 # [ 138.652138] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:16Z level=info event=scheduler_connected addr=lb:500515980worker1 # [ 138.656135] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5981worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5982worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5983worker1 # [ 138.856915] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:17Z level=info event=scheduler_connected addr=lb:500515984worker1 # [ 138.861106] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5985worker1 # [ 139.061729] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:17Z level=info event=scheduler_connected addr=lb:500515986worker1 # [ 139.065713] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5987worker1 # [ 139.266583] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:17Z level=info event=scheduler_connected addr=lb:500515988worker1 # [ 139.270576] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5989worker1 # [ 139.472287] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:17Z level=info event=scheduler_connected addr=lb:500515990worker1 # [ 139.476174] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5991worker1 # [ 139.676844] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:17Z level=info event=scheduler_connected addr=lb:500515992worker1 # [ 139.680927] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5993worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5994worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5995worker1 # [ 139.881792] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:18Z level=info event=scheduler_connected addr=lb:500515996worker1 # [ 139.885809] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5997worker1 # [ 140.086476] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:18Z level=info event=scheduler_connected addr=lb:500515998worker1 # [ 140.090440] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5999worker1 # [ 140.291463] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:18Z level=info event=scheduler_connected addr=lb:500516000worker1 # [ 140.296535] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6001worker1 # [ 140.355139] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:18Z level=info event=scheduler_take_over6002lb # [ 140.351245] niks3-server[1225]: 2026/09/23 07:01:19 INFO lead: acquired remote=192.168.1.3:477226003worker1 # [ 140.498209] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:18Z level=info event=scheduler_connected addr=lb:500516004worker1 # [ 140.502068] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6005worker1 # [ 140.702910] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:18Z level=info event=scheduler_connected addr=lb:500516006worker1 # [ 140.706940] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6007worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6008worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6009worker1 # [ 140.907960] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:19Z level=info event=scheduler_connected addr=lb:500516010worker1 # [ 140.915127] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:19Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:495266011worker1 # [ 140.919717] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:19Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06012worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6013worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6014client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&26015worker1 # [ 142.003746] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:20Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:445586016client # don't know how to build these paths:6017client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6018worker1 # [ 142.012600] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:445586019worker1 # [ 142.022931] nix-grpc-daemon[1335]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 83 ms (attempt 1/5)6020worker1 # [ 142.113913] nix-grpc-daemon[1335]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 149 ms (attempt 2/5)6021worker1 # [ 142.270844] nix-grpc-daemon[1335]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 22 ms (attempt 3/5)6022worker1 # [ 142.294500] nix-grpc-daemon[1335]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 235 ms (attempt 4/5)6023worker1 # [ 142.532193] nix-grpc-daemon[1335]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6024worker1 # [ 142.536109] nix-grpc-daemon[1335]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6025worker1 # [ 142.562522] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:20Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=06026worker1 # [ 142.568666] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:20Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=16027worker1 # [ 142.611939] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:445586028worker1 # [ 142.624603] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:445586029client # copying 1 paths to 'lb:50051'...6030client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...6031worker1 # [ 142.636181] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:445586032worker1 # [ 142.647204] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6033worker1 # [ 142.666194] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:20Z level=info event=niks3_push_started pid=14216034worker1 # [ 142.767844] nix-grpc-daemon[1421]: time=2026-09-23T07:01:21.047Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6035worker1 # [ 142.855578] nix-daemon[869]: accepted connection from pid 1434, user nix-grpc-daemon (trusted)6036worker1 # [ 142.866798] nix-daemon[869]: reaped child process 1437, status = succeeded6037lb # [ 142.872489] niks3-server[1225]: 2026/09/23 07:01:22 INFO Received uploads request method=POST path=/api/pending_closures6038worker1 # [ 142.885861] nix-grpc-daemon[1421]: time=2026-09-23T07:01:21.165Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6039worker1 # [ 142.887369] nix-grpc-daemon[1421]: time=2026-09-23T07:01:21.167Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"6040lb # [ 142.913719] niks3-server[1225]: 2026/09/23 07:01:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6041lb # [ 142.919988] niks3-server[1225]: 2026/09/23 07:01:22 INFO Signed narinfos id=43 count=16042worker1 # [ 142.927732] nix-grpc-daemon[1421]: time=2026-09-23T07:01:21.207Z level=INFO msg="Uploading 1 narinfos"6043lb # [ 142.933046] niks3-server[1225]: 2026/09/23 07:01:22 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls6044lb # [ 142.940866] niks3-server[1225]: 2026/09/23 07:01:22 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst6045lb # [ 142.950122] niks3-server[1225]: 2026/09/23 07:01:22 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6046lb # [ 142.954534] niks3-server[1225]: 2026/09/23 07:01:22 INFO Completed upload id=436047worker1 # [ 142.958983] nix-grpc-daemon[1421]: time=2026-09-23T07:01:21.238Z level=INFO msg="Upload complete. (187ms)"6048lb # [ 142.957370] niks3-server[1225]: 2026/09/23 07:01:22 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo6049worker1 # [ 142.963468] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44558 duration_s=0 paths=1 nar_bytes_in=5206050worker1 # [ 142.969099] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:445586051worker1 # [ 142.978268] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6052worker1 # [ 142.983284] nix-daemon[869]: reaped child process 1419, status = succeeded6053client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...6054worker1 # [ 142.988298] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6055client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6056client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6057worker1 # [ 143.705685] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:21Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:532426058client # farm-a> LOG-farm-a-on-standby6059worker1 # [ 143.819616] nix-daemon[869]: reaped child process 1441, status = succeeded6060worker1 # [ 143.890689] nix-daemon[869]: accepted connection from pid 1457, user nix-grpc-daemon (trusted)6061worker1 # [ 143.901524] nix-daemon[869]: reaped child process 1460, status = succeeded6062lb # [ 143.907446] niks3-server[1225]: 2026/09/23 07:01:23 INFO Received uploads request method=POST path=/api/pending_closures6063worker1 # [ 143.921547] nix-grpc-daemon[1421]: time=2026-09-23T07:01:22.200Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6064worker1 # [ 143.923196] nix-grpc-daemon[1421]: time=2026-09-23T07:01:22.202Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"6065lb # [ 143.939094] niks3-server[1225]: 2026/09/23 07:01:23 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls6066lb # [ 143.941469] niks3-server[1225]: 2026/09/23 07:01:23 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst6067lb # [ 143.944754] niks3-server[1225]: 2026/09/23 07:01:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6068lb # [ 143.947177] niks3-server[1225]: 2026/09/23 07:01:23 INFO Signed narinfos id=44 count=16069worker1 # [ 143.952877] nix-grpc-daemon[1421]: time=2026-09-23T07:01:22.232Z level=INFO msg="Uploading 1 narinfos"6070lb # [ 143.950742] niks3-server[1225]: 2026/09/23 07:01:23 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6071lb # [ 143.956148] niks3-server[1225]: 2026/09/23 07:01:23 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6072lb # [ 143.960151] niks3-server[1225]: 2026/09/23 07:01:23 INFO Completed upload id=446073worker1 # [ 143.964292] nix-grpc-daemon[1421]: time=2026-09-23T07:01:22.243Z level=INFO msg="Upload complete. (169ms)"6074lb # [ 143.962055] niks3-server[1225]: 2026/09/23 07:01:23 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo6075worker1 # [ 143.966650] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44558 duration_s=0 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=16076worker1 # [ 143.974485] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:22Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=06077worker1 # [ 143.976498] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:22Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=16078worker1 # [ 143.980802] nix-daemon[869]: reaped child process 1439, status = succeeded6079worker1 # [ 144.018809] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:445586080worker1 # [ 144.029778] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:445586081worker1 # [ 144.042132] nix-grpc-daemon[1421]: time=2026-09-23T07:01:22.321Z level=INFO msg="All 1 paths already cached"6082client # copying 1 paths to 'lb:50051'...6083client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...6084worker1 # [ 144.049525] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:445586085worker1 # [ 144.058128] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6086worker1 # [ 144.150243] nix-daemon[869]: accepted connection from pid 1466, user nix-grpc-daemon (trusted)6087worker1 # [ 144.162459] nix-daemon[869]: reaped child process 1469, status = succeeded6088lb # [ 144.167169] niks3-server[1225]: 2026/09/23 07:01:23 INFO Received uploads request method=POST path=/api/pending_closures6089worker1 # [ 144.179708] nix-grpc-daemon[1421]: time=2026-09-23T07:01:22.459Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6090worker1 # [ 144.181162] nix-grpc-daemon[1421]: time=2026-09-23T07:01:22.460Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"6091lb # [ 144.203279] niks3-server[1225]: 2026/09/23 07:01:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6092worker1 # [ 144.211762] nix-grpc-daemon[1421]: time=2026-09-23T07:01:22.491Z level=INFO msg="Uploading 1 narinfos"6093lb # [ 144.207951] niks3-server[1225]: 2026/09/23 07:01:23 INFO Signed narinfos id=45 count=16094lb # [ 144.213765] niks3-server[1225]: 2026/09/23 07:01:23 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst6095lb # [ 144.219091] niks3-server[1225]: 2026/09/23 07:01:23 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls6096lb # [ 144.229325] niks3-server[1225]: 2026/09/23 07:01:23 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6097worker1 # [ 144.240379] nix-grpc-daemon[1421]: time=2026-09-23T07:01:22.519Z level=INFO msg="Upload complete. (172ms)"6098lb # [ 144.236133] niks3-server[1225]: 2026/09/23 07:01:23 INFO Completed upload id=456099lb # [ 144.241844] niks3-server[1225]: 2026/09/23 07:01:23 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo6100worker1 # [ 144.243442] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44558 duration_s=0 paths=1 nar_bytes_in=6406101worker1 # [ 144.249284] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:445586102worker1 # [ 144.258249] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6103worker1 # [ 144.263854] nix-daemon[869]: reaped child process 1464, status = succeeded6104worker1 # [ 144.268655] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6105client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...6106client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6107client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6108client # farm-b> LOG-farm-b-on-standby6109worker1 # [ 145.092660] nix-daemon[869]: reaped child process 1473, status = succeeded6110worker1 # [ 145.166201] nix-daemon[869]: accepted connection from pid 1489, user nix-grpc-daemon (trusted)6111worker1 # [ 145.179347] nix-daemon[869]: reaped child process 1492, status = succeeded6112lb # [ 145.182896] niks3-server[1225]: 2026/09/23 07:01:24 INFO Received uploads request method=POST path=/api/pending_closures6113worker1 # [ 145.195906] nix-grpc-daemon[1421]: time=2026-09-23T07:01:23.475Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6114worker1 # [ 145.197405] nix-grpc-daemon[1421]: time=2026-09-23T07:01:23.477Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"6115lb # [ 145.226268] niks3-server[1225]: 2026/09/23 07:01:24 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst6116lb # [ 145.231906] niks3-server[1225]: 2026/09/23 07:01:24 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6117lb # [ 145.238897] niks3-server[1225]: 2026/09/23 07:01:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6118lb # [ 145.243943] niks3-server[1225]: 2026/09/23 07:01:24 INFO Signed narinfos id=46 count=16119worker1 # [ 145.251883] nix-grpc-daemon[1421]: time=2026-09-23T07:01:23.531Z level=INFO msg="Uploading 1 narinfos"6120lb # [ 145.256661] niks3-server[1225]: 2026/09/23 07:01:24 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls6121lb # [ 145.259805] niks3-server[1225]: 2026/09/23 07:01:24 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6122lb # [ 145.263700] niks3-server[1225]: 2026/09/23 07:01:24 INFO Completed upload id=466123worker1 # [ 145.269312] nix-grpc-daemon[1421]: time=2026-09-23T07:01:23.548Z level=INFO msg="Upload complete. (194ms)"6124lb # [ 145.266956] niks3-server[1225]: 2026/09/23 07:01:24 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo6125worker1 # [ 145.276993] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44558 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=16126worker1 # [ 145.288962] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:23Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=06127worker1 # [ 145.293211] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:23Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=16128worker1 # [ 145.301633] nix-daemon[869]: reaped child process 1471, status = succeeded6129worker1 # [ 145.333508] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:445586130worker1 # [ 145.346219] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:445586131worker1 # [ 145.360581] nix-grpc-daemon[1421]: time=2026-09-23T07:01:23.640Z level=INFO msg="All 2 paths already cached"6132client # copying 1 paths to 'lb:50051'...6133client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...6134worker1 # [ 145.369605] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:445586135worker1 # [ 145.377665] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6136worker1 # [ 145.475949] nix-daemon[869]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6137worker1 # [ 145.488376] nix-daemon[869]: reaped child process 1501, status = succeeded6138lb # [ 145.492946] niks3-server[1225]: 2026/09/23 07:01:24 INFO Received uploads request method=POST path=/api/pending_closures6139worker1 # [ 145.505593] nix-grpc-daemon[1421]: time=2026-09-23T07:01:23.785Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6140worker1 # [ 145.507080] nix-grpc-daemon[1421]: time=2026-09-23T07:01:23.786Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6141lb # [ 145.525689] niks3-server[1225]: 2026/09/23 07:01:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6142lb # [ 145.533716] niks3-server[1225]: 2026/09/23 07:01:24 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6143lb # [ 145.544643] niks3-server[1225]: 2026/09/23 07:01:24 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6144lb # [ 145.555864] niks3-server[1225]: 2026/09/23 07:01:24 INFO Signed narinfos id=47 count=16145worker1 # [ 145.560138] nix-grpc-daemon[1421]: time=2026-09-23T07:01:23.839Z level=INFO msg="Uploading 1 narinfos"6146lb # [ 145.565959] niks3-server[1225]: 2026/09/23 07:01:24 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6147lb # [ 145.574424] niks3-server[1225]: 2026/09/23 07:01:24 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6148lb # [ 145.577898] niks3-server[1225]: 2026/09/23 07:01:24 INFO Completed upload id=476149worker1 # [ 145.582379] nix-grpc-daemon[1421]: time=2026-09-23T07:01:23.861Z level=INFO msg="Upload complete. (191ms)"6150worker1 # [ 145.586915] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44558 duration_s=0 paths=1 nar_bytes_in=7686151worker1 # [ 145.592812] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:445586152worker1 # [ 145.602658] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6153worker1 # [ 145.614411] nix-daemon[869]: reaped child process 1496, status = succeeded6154client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6155worker1 # [ 145.622391] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6156client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6157client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6158client # farm-top> LOG-farm-top-on-standby6159worker1 # [ 146.839423] nix-daemon[869]: reaped child process 1505, status = succeeded6160worker1 # [ 146.916802] nix-daemon[869]: accepted connection from pid 1521, user nix-grpc-daemon (trusted)6161worker1 # [ 146.928181] nix-daemon[869]: reaped child process 1524, status = succeeded6162lb # [ 146.933234] niks3-server[1225]: 2026/09/23 07:01:26 INFO Received uploads request method=POST path=/api/pending_closures6163worker1 # [ 146.948071] nix-grpc-daemon[1421]: time=2026-09-23T07:01:25.227Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6164worker1 # [ 146.949822] nix-grpc-daemon[1421]: time=2026-09-23T07:01:25.229Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6165lb # [ 146.980624] niks3-server[1225]: 2026/09/23 07:01:26 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6166lb # [ 146.987538] niks3-server[1225]: 2026/09/23 07:01:26 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6167lb # [ 146.991734] niks3-server[1225]: 2026/09/23 07:01:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6168lb # [ 146.995611] niks3-server[1225]: 2026/09/23 07:01:26 INFO Signed narinfos id=48 count=16169worker1 # [ 147.002234] nix-grpc-daemon[1421]: time=2026-09-23T07:01:25.281Z level=INFO msg="Uploading 1 narinfos"6170lb # [ 147.001367] niks3-server[1225]: 2026/09/23 07:01:26 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6171lb # [ 147.010218] niks3-server[1225]: 2026/09/23 07:01:26 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6172lb # [ 147.013154] niks3-server[1225]: 2026/09/23 07:01:26 INFO Completed upload id=486173worker1 # [ 147.017587] nix-grpc-daemon[1421]: time=2026-09-23T07:01:25.297Z level=INFO msg="Upload complete. (185ms)"6174lb # [ 147.016240] niks3-server[1225]: 2026/09/23 07:01:26 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6175worker1 # [ 147.023933] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44558 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16176worker1 # [ 147.037100] nix-daemon[869]: reaped child process 1503, status = succeeded6177client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&2, in 5.18 seconds)6178client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath6179client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath, in 0.12 seconds)6180worker2 # [ 147.239780] systemd[1]: Listening on Nix worker-protocol over gRPC.6181worker2 # [ 147.246556] systemd[1]: Starting scheduler-token.service...6182worker2 # [ 147.292830] systemd[1]: scheduler-token.service: Deactivated successfully.6183worker2 # [ 147.294539] systemd[1]: Finished scheduler-token.service.6184worker2 # [ 147.295794] systemd[1]: scheduler-token.service: Consumed 22ms CPU time over 49ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6185worker2 # [ 147.300044] systemd[1]: Starting Nix worker-protocol over gRPC...6186worker2 # [ 147.375632] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:26Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6187worker2 # [ 147.377653] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:26Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6188worker2 # [ 147.380782] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:26Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16189worker2 # [ 147.385054] nix-grpc-daemon[1374]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6190worker2 # [ 147.386866] nix-grpc-daemon[1374]: E0000 00:00:1790146886.570467 1374 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6191worker2 # [ 147.393040] nix-grpc-daemon[1374]: E0000 00:00:1790146886.575919 1374 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6192worker2 # [ 147.396671] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:26Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6193worker2 # [ 147.399609] systemd[1]: Started Nix worker-protocol over gRPC.6194worker2 # [ 147.403082] nix-grpc-daemon[1374]: E0000 00:00:1790146886.587599 1417 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6195lb: must succeed: curl -sf localhost:9901/clusters6196worker2 # [ 147.414853] nix-grpc-daemon[1374]: E0000 00:00:1790146886.600555 1417 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6197worker2 # [ 147.418129] nix-grpc-daemon[1374]: E0000 00:00:1790146886.602782 1417 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6198worker2 # [ 147.428983] nix-grpc-daemon[1374]: E0000 00:00:1790146886.614693 1417 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6199worker2 # [ 147.438831] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:26Z level=info event=scheduler_connected addr=lb:500516200worker1 # [ 147.443180] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:25Z level=info event=oidc_jwks provider=mock keys=16201worker1 # [ 147.447515] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:25Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:495266202worker1 # [ 147.454267] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:25Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06203lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6204worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6205worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6206lb: must succeed: curl -sf localhost:9901/clusters6207lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6208lb: must succeed: curl -sf localhost:9901/clusters6209lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6210worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6211worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6212(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 13.57 seconds)6213subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6214client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs6215client # [ 147.673780] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs.6216client # Running as unit: rs.service; invocation ID: 303da3cf3d464e6a86b70c8a4975d3266217client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs, in 0.06 seconds)6218worker2 # [ 147.808203] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504566219worker2 # [ 147.819194] nix-daemon[855]: accepted connection from pid 1374, user nix-grpc-daemon (trusted)6220client # [ 147.839908] nix[1365]: don't know how to build these paths:6221client # [ 147.840947] nix[1365]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6222worker1 # [ 147.835108] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:26Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06223worker1 # [ 147.840559] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:26Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=16224worker1 # [ 147.879945] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:445586225worker1 # [ 147.893124] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:445586226client # [ 147.915194] nix[1365]: copying 1 paths to 'lb:50051'...6227worker1 # [ 147.904710] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:445586228client # [ 147.918278] nix[1365]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6229worker1 # [ 147.917740] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6230worker1 # [ 148.019476] nix-daemon[869]: accepted connection from pid 1537, user nix-grpc-daemon (trusted)6231worker1 # [ 148.032453] nix-daemon[869]: reaped child process 1540, status = succeeded6232lb # [ 148.036438] niks3-server[1225]: 2026/09/23 07:01:27 INFO Received uploads request method=POST path=/api/pending_closures6233worker1 # [ 148.049723] nix-grpc-daemon[1421]: time=2026-09-23T07:01:26.328Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6234worker1 # [ 148.051101] nix-grpc-daemon[1421]: time=2026-09-23T07:01:26.328Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6235lb # [ 148.066776] niks3-server[1225]: 2026/09/23 07:01:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6236worker1 # [ 148.078077] nix-grpc-daemon[1421]: time=2026-09-23T07:01:26.357Z level=INFO msg="Uploading 1 narinfos"6237lb # [ 148.073717] niks3-server[1225]: 2026/09/23 07:01:27 INFO Signed narinfos id=49 count=16238lb # [ 148.084561] niks3-server[1225]: 2026/09/23 07:01:27 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6239lb # [ 148.089112] niks3-server[1225]: 2026/09/23 07:01:27 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6240lb # [ 148.098224] niks3-server[1225]: 2026/09/23 07:01:27 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6241lb # [ 148.104748] niks3-server[1225]: 2026/09/23 07:01:27 INFO Completed upload id=496242worker1 # [ 148.108975] nix-grpc-daemon[1421]: time=2026-09-23T07:01:26.388Z level=INFO msg="Upload complete. (180ms)"6243lb # [ 148.108248] niks3-server[1225]: 2026/09/23 07:01:27 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6244worker1 # [ 148.113499] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44558 duration_s=0 paths=1 nar_bytes_in=4166245worker1 # [ 148.118929] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:445586246worker1 # [ 148.128122] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6247worker1 # [ 148.135449] nix-daemon[869]: reaped child process 1535, status = succeeded6248worker1 # [ 148.140033] nix-daemon[869]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)6249client # [ 148.151576] nix[1365]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6250lb: must succeed: curl -sf localhost:9901/clusters6251lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6252lb: must succeed: curl -sf localhost:9901/clusters6253lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6254lb: must succeed: curl -sf localhost:9901/clusters6255lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6256lb: must succeed: curl -sf localhost:9901/clusters6257lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6258worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6259worker2 # [ 150.320639] systemd-journald[284]: Received client request to rotate journal, rotating.6260worker2 # [ 150.342462] systemd-journald[284]: Vacuuming done, freed 0B of archived journals from /var/log/journal/60cc630c2c0b4ac48804561462d92acc.6261worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.06 seconds)6262worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service6263worker1 # [ 150.248667] systemd[1]: Stopping Nix worker-protocol over gRPC...6264worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.03 seconds)6265client: must succeed: sleep 5.06266worker1 # [ 151.202491] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:29Z level=info event=draining6267client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6268worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6269worker1 # [ 155.581515] nix-daemon[869]: reaped child process 1544, status = succeeded6270worker1 # [ 155.666669] nix-daemon[869]: accepted connection from pid 1585, user nix-grpc-daemon (trusted)6271worker1 # [ 155.679228] nix-daemon[869]: reaped child process 1588, status = succeeded6272lb # [ 155.681302] niks3-server[1225]: 2026/09/23 07:01:34 INFO Received uploads request method=POST path=/api/pending_closures6273worker1 # [ 155.695444] nix-grpc-daemon[1421]: time=2026-09-23T07:01:33.974Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6274worker1 # [ 155.696993] nix-grpc-daemon[1421]: time=2026-09-23T07:01:33.976Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6275lb # [ 155.727168] niks3-server[1225]: 2026/09/23 07:01:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6276worker1 # [ 155.738320] nix-grpc-daemon[1421]: time=2026-09-23T07:01:34.017Z level=INFO msg="Uploading 1 narinfos"6277lb # [ 155.733920] niks3-server[1225]: 2026/09/23 07:01:34 INFO Signed narinfos id=50 count=16278lb # [ 155.742869] niks3-server[1225]: 2026/09/23 07:01:34 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6279lb # [ 155.747646] niks3-server[1225]: 2026/09/23 07:01:34 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6280lb # [ 155.756424] niks3-server[1225]: 2026/09/23 07:01:34 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6281lb # [ 155.762071] niks3-server[1225]: 2026/09/23 07:01:34 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6282lb # [ 155.767128] niks3-server[1225]: 2026/09/23 07:01:34 INFO Completed upload id=506283worker1 # [ 155.771572] nix-grpc-daemon[1421]: time=2026-09-23T07:01:34.051Z level=INFO msg="Upload complete. (217ms)"6284lb # [ 155.769629] niks3-server[1225]: 2026/09/23 07:01:34 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6285worker1 # [ 155.780584] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44558 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16286worker1 # [ 155.793474] nix-daemon[869]: reaped child process 1542, status = succeeded6287client # [ 155.821991] systemd[1]: rs.service: Deactivated successfully.6288client # [ 155.825971] systemd[1]: rs.service: Consumed 131ms CPU time over 8.148s wall clock time, 41.4M memory peak, 7.1K incoming IP traffic, 8.3K outgoing IP traffic.6289worker1 # [ 156.203692] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:34Z level=info event=signal_exit6290lb # [ 156.353311] niks3-server[1225]: 2026/09/23 07:01:35 INFO lead: released remote=192.168.1.3:477226291lb # [ 156.613786] envoy[912]: conn peer=[2001:db8:1::1]:55008 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9560 tx=4394 ms=50376292lb # [ 156.617867] envoy[912]: conn peer=[2001:db8:1::1]:51202 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2943 tx=1498 ms=79876293worker1 # [ 157.359761] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:35Z level=info event=scheduler_restarting6294worker2 # [ 157.363594] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:36Z level=info event=scheduler_restarting addr=lb:500516295lb # [ 157.354204] envoy[912]: [2026-09-23 07:01:36.561][912][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream}6296worker1 # [ 157.362717] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:35Z level=info event=scheduler_restarting addr=lb:500516297worker1 # [ 157.369650] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:35Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:495266298worker1 # [ 157.373647] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:35Z level=info event=worker_gone addr=192.168.1.3:500516299worker1 # [ 157.377211] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:35Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:495266300worker1 # [ 157.381171] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:35Z level=info event=worker_gone addr=192.168.1.4:500516301worker2 # [ 157.401855] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:36Z level=info event=scheduler_take_over6302worker1 # [ 157.399257] nix-daemon[869]: reaped child process 1388, status = succeeded6303lb # [ 157.394992] niks3-server[1225]: 2026/09/23 07:01:36 INFO lead: acquired remote=192.168.1.4:477706304worker2 # [ 157.564485] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:36Z level=info event=scheduler_connected addr=lb:500516305worker2 # [ 157.568542] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6306worker1 # [ 157.572504] nix-grpc-daemon[1335]: ts=2026-09-23T07:01:35Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6307worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.11 seconds)6308client: waiting for success: ! systemctl is-active rs6309client: (finished: waiting for success: ! systemctl is-active rs, in 0.04 seconds)6310client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6311client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)6312client: must fail: journalctl -u rs | grep -q 'reconnecting'6313client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.02 seconds)6314lb: must succeed: curl -sf localhost:9901/clusters6315lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6316worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6317worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6318worker2 # [ 157.769230] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:36Z level=info event=scheduler_connected addr=lb:500516319worker2 # [ 157.772525] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6320worker1 # [ 157.781595] nix-grpc-daemon[1335]: E0000 00:00:1790146896.060961 1381 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 23, started true, shutdown true6321worker2 # [ 157.973699] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=scheduler_connected addr=lb:500516322worker2 # [ 157.977635] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6323worker2 # [ 158.178830] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=scheduler_connected addr=lb:500516324worker2 # [ 158.182768] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6325worker1 # [ 158.328398] nix-grpc-daemon[1335]: E0000 00:00:1790146896.607823 1381 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6326worker2 # [ 158.384658] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=scheduler_connected addr=lb:500516327worker2 # [ 158.388730] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6328worker2 # [ 158.590121] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=scheduler_connected addr=lb:500516329worker2 # [ 158.593905] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6330lb: must succeed: curl -sf localhost:9901/clusters6331worker2 # [ 158.795155] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=scheduler_connected addr=lb:500516332worker2 # [ 158.809136] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=oidc_jwks provider=mock keys=16333lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6334worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6335worker2 # [ 158.813313] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:528606336worker2 # [ 158.817288] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:37Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06337worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6338worker2 # [ 159.309621] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:38Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:528666339worker1 # [ 159.541956] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6340worker1 # [ 159.544642] systemd[1]: Stopped Nix worker-protocol over gRPC.6341worker1 # [ 159.547566] systemd[1]: nix-grpc-daemon.service: Consumed 1.075s CPU time over 30.458s wall clock time, 37.6M memory peak, 52.3K incoming IP traffic, 107.5K outgoing IP traffic.6342worker1 # [ 159.563280] systemd[1]: Starting Nix worker-protocol over gRPC...6343worker1 # [ 159.654734] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:37Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6344worker1 # [ 159.659186] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:37Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6345worker1 # [ 159.663924] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:37Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16346worker1 # [ 159.668025] nix-grpc-daemon[1595]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6347worker1 # [ 159.671367] nix-grpc-daemon[1595]: E0000 00:00:1790146897.941327 1595 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6348worker1 # [ 159.674394] nix-grpc-daemon[1595]: E0000 00:00:1790146897.943483 1595 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6349worker1 # [ 159.676859] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:37Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6350worker1 # [ 159.679212] systemd[1]: Started Nix worker-protocol over gRPC.6351worker1 # [ 159.682126] nix-grpc-daemon[1595]: E0000 00:00:1790146897.958294 1637 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6352worker1 # [ 159.691587] nix-grpc-daemon[1595]: E0000 00:00:1790146897.971195 1637 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6353worker1 # [ 159.696054] nix-grpc-daemon[1595]: E0000 00:00:1790146897.973398 1637 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6354worker1 # [ 159.705406] nix-grpc-daemon[1595]: E0000 00:00:1790146897.985032 1637 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6355worker1 # [ 159.714773] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:37Z level=info event=scheduler_connected addr=lb:500516356worker2 # [ 159.720652] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:38Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:528606357worker2 # [ 159.725139] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:38Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06358lb: must succeed: curl -sf localhost:9901/clusters6359lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6360worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6361worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6362(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.33 seconds)6363subtest: scheduler restart mid-build: build finishes, no double build6364client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid6365client # [ 159.993721] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid.6366client # Running as unit: mid.service; invocation ID: 4cae723ad45d4bbbbc571f781ef404ba6367client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid, in 0.05 seconds)6368worker2 # [ 160.120060] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504566369client # [ 160.135078] nix[1401]: don't know how to build these paths:6370client # [ 160.135988] nix[1401]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6371worker2 # [ 160.134242] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:39Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06372worker2 # [ 160.138240] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:39Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=16373worker2 # [ 160.188118] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504566374worker2 # [ 160.198824] nix-grpc-daemon[1374]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 1/5)6375worker2 # [ 160.232214] nix-grpc-daemon[1374]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 120 ms (attempt 2/5)6376worker2 # [ 160.354486] nix-grpc-daemon[1374]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 224 ms (attempt 3/5)6377worker2 # [ 160.580649] nix-grpc-daemon[1374]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 680 ms (attempt 4/5)6378worker2 # [ 161.263885] nix-grpc-daemon[1374]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6379worker2 # [ 161.267759] nix-grpc-daemon[1374]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6380worker2 # [ 161.284906] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:504566381client # [ 161.304332] nix[1401]: copying 1 paths to 'lb:50051'...6382client # [ 161.308298] nix[1401]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6383worker2 # [ 161.300667] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:504566384worker2 # [ 161.312164] nix-daemon[855]: accepted connection from pid 1374, user nix-grpc-daemon (trusted)6385worker2 # [ 161.333184] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:40Z level=info event=niks3_push_started pid=15046386worker2 # [ 161.433944] nix-grpc-daemon[1504]: time=2026-09-23T07:01:40.619Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""6387worker2 # [ 161.519136] nix-daemon[855]: accepted connection from pid 1516, user nix-grpc-daemon (trusted)6388worker2 # [ 161.530146] nix-daemon[855]: reaped child process 1519, status = succeeded6389lb # [ 161.531289] niks3-server[1225]: 2026/09/23 07:01:40 INFO Received uploads request method=POST path=/api/pending_closures6390worker2 # [ 161.555219] nix-grpc-daemon[1504]: time=2026-09-23T07:01:40.740Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6391worker2 # [ 161.556757] nix-grpc-daemon[1504]: time=2026-09-23T07:01:40.742Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6392lb # [ 161.576549] niks3-server[1225]: 2026/09/23 07:01:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6393lb # [ 161.583150] niks3-server[1225]: 2026/09/23 07:01:40 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6394lb # [ 161.595573] niks3-server[1225]: 2026/09/23 07:01:40 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6395lb # [ 161.600153] niks3-server[1225]: 2026/09/23 07:01:40 INFO Signed narinfos id=51 count=16396worker2 # [ 161.608791] nix-grpc-daemon[1504]: time=2026-09-23T07:01:40.793Z level=INFO msg="Uploading 1 narinfos"6397lb # [ 161.606058] niks3-server[1225]: 2026/09/23 07:01:40 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6398lb # [ 161.610669] niks3-server[1225]: 2026/09/23 07:01:40 INFO Completed upload id=516399worker2 # [ 161.620110] nix-grpc-daemon[1504]: time=2026-09-23T07:01:40.805Z level=INFO msg="Upload complete. (182ms)"6400lb # [ 161.615825] niks3-server[1225]: 2026/09/23 07:01:40 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6401worker2 # [ 161.625175] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50456 duration_s=0 paths=1 nar_bytes_in=4246402worker2 # [ 161.630167] nix-grpc-daemon[1374]: ts=2026-09-23T07:01:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:504566403worker2 # [ 161.639803] nix-daemon[855]: accepted connection from pid 1374, user nix-grpc-daemon (trusted)6404worker2 # [ 161.643766] nix-daemon[855]: reaped child process 1502, status = succeeded6405worker2 # [ 161.648489] nix-daemon[855]: accepted connection from pid 1374, user nix-grpc-daemon (trusted)6406client # [ 161.656475] nix[1401]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6407lb: must succeed: curl -sf localhost:9901/clusters6408lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6409lb: must succeed: curl -sf localhost:9901/clusters6410lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6411worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6412worker2 # [ 163.499506] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1374 (nix-grpc-daemon) on client request.6413worker2 # [ 163.502561] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6414lb # [ 163.505742] niks3-server[1225]: 2026/09/23 07:01:42 INFO lead: released remote=192.168.1.4:477706415worker2 # [ 163.520791] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6416worker1 # [ 163.514718] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:41Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6417lb # [ 163.510313] envoy[912]: [2026-09-23 07:01:42.716][912][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 13, upstream_reset_after_response_started{connection_termination}6418client # [ 163.527257] nix[1401]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6419worker2 # [ 163.522199] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6420worker2 # [ 163.523368] systemd[1]: nix-grpc-daemon.service: Consumed 389ms CPU time over 16.217s wall clock time, 36M memory peak, 16.6K incoming IP traffic, 36.8K outgoing IP traffic.6421client # [ 163.531736] nix[1401]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6422worker2 # [ 163.534367] systemd[1]: Starting scheduler-token.service...6423worker2 # [ 163.549127] nix-daemon[855]: reaped child process 1434, status = succeeded6424worker2 # [ 163.550635] nix-daemon[855]: reaped child process 1522, status = succeeded6425worker2 # [ 163.561240] nix-daemon[1524]: unexpected Nix daemon error: error: interrupted by the user6426worker2 # [ 163.565812] nix-daemon[855]: reaped child process 1524, status = failed with exit code 16427worker2 # [ 163.585918] systemd[1]: scheduler-token.service: Deactivated successfully.6428worker2 # [ 163.587084] systemd[1]: Finished scheduler-token.service.6429worker2 # [ 163.588932] systemd[1]: scheduler-token.service: Consumed 16ms CPU time over 52ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6430worker2 # [ 163.592086] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6431worker2 # [ 163.594590] systemd[1]: Starting scheduler-token.service...6432worker2 # [ 163.628049] systemd[1]: scheduler-token.service: Deactivated successfully.6433worker2 # [ 163.628994] systemd[1]: Finished scheduler-token.service.6434worker2 # [ 163.630384] systemd[1]: scheduler-token.service: Consumed 15ms CPU time over 33ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6435worker2 # [ 163.635155] systemd[1]: Starting Nix worker-protocol over gRPC...6436worker2 # [ 163.709967] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:42Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6437worker2 # [ 163.711928] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:42Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6438worker2 # [ 163.715091] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:42Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16439worker2 # [ 163.719463] nix-grpc-daemon[1567]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6440worker1 # [ 163.715248] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:41Z level=info event=scheduler_connected addr=lb:500516441worker2 # [ 163.722099] nix-grpc-daemon[1567]: E0000 00:00:1790146902.905076 1567 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6442worker2 # [ 163.727616] nix-grpc-daemon[1567]: E0000 00:00:1790146902.910801 1567 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6443worker2 # [ 163.732093] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:42Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6444worker2 # [ 163.735213] systemd[1]: Started Nix worker-protocol over gRPC.6445worker2 # [ 163.740555] nix-grpc-daemon[1567]: E0000 00:00:1790146902.926314 1609 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6446lb # [ 163.734487] niks3-server[1225]: 2026/09/23 07:01:42 INFO lead: acquired remote=192.168.1.4:498466447worker2 # [ 163.745125] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:42Z level=info event=scheduler_take_over6448worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.27 seconds)6449client: must succeed: sleep 5.06450worker2 # [ 163.749791] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:42Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:440966451worker2 # [ 163.752120] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:42Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06452worker2 # [ 163.760412] nix-grpc-daemon[1567]: E0000 00:00:1790146902.946154 1609 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6453worker2 # [ 163.764053] nix-grpc-daemon[1567]: E0000 00:00:1790146902.948408 1609 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6454worker2 # [ 163.774419] nix-grpc-daemon[1567]: E0000 00:00:1790146902.960131 1609 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6455worker2 # [ 163.783384] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:42Z level=info event=scheduler_connected addr=lb:500516456worker2 # [ 163.785051] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6457worker2 # [ 163.986373] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:43Z level=info event=scheduler_connected addr=lb:500516458worker2 # [ 163.990333] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6459client # [ 164.029925] nix[1401]: scheduler at lb:50051: no healthy upstream, reconnecting6460worker2 # [ 164.191520] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:43Z level=info event=scheduler_connected addr=lb:500516461worker2 # [ 164.195501] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6462worker2 # [ 164.396704] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:43Z level=info event=scheduler_connected addr=lb:500516463worker2 # [ 164.400619] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6464worker2 # [ 164.601919] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:43Z level=info event=scheduler_connected addr=lb:500516465worker2 # [ 164.606195] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6466worker2 # [ 164.807606] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:43Z level=info event=scheduler_connected addr=lb:500516467worker2 # [ 164.818144] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:44Z level=info event=oidc_jwks provider=mock keys=16468worker2 # [ 164.821654] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:44Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:440966469worker2 # [ 164.825530] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:44Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06470worker2 # [ 165.031262] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:44Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06471worker2 # [ 165.035857] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:44Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16472worker1 # [ 165.080672] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:438326473worker1 # [ 165.091855] nix-daemon[869]: accepted connection from pid 1595, user nix-grpc-daemon (trusted)6474worker1 # [ 165.108114] nix-grpc-daemon[1595]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 34 ms (attempt 1/5)6475worker1 # [ 165.149441] nix-grpc-daemon[1595]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 190 ms (attempt 2/5)6476worker1 # [ 165.341668] nix-grpc-daemon[1595]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 39 ms (attempt 3/5)6477worker1 # [ 165.382330] nix-grpc-daemon[1595]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 444 ms (attempt 4/5)6478worker1 # [ 165.828913] nix-grpc-daemon[1595]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6479worker1 # [ 165.832283] nix-grpc-daemon[1595]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6480worker1 # [ 165.847587] nix-grpc-daemon[1595]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' from 'http://lb:5751'...6481worker1 # [ 165.871157] nix-daemon[869]: accepted connection from pid 1595, user nix-grpc-daemon (trusted)6482worker1 # [ 165.884755] nix-daemon[869]: accepted connection from pid 1595, user nix-grpc-daemon (trusted)6483client # [ 165.895961] nix[1401]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6484worker2 # [ 165.912618] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:45Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:441166485lb # [ 166.614308] envoy[912]: conn peer=[2001:db8:1::3]:38378 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=6804 tx=1494 ms=283266486lb # [ 166.618689] envoy[912]: conn peer=[2001:db8:1::4]:53868 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=7013 tx=556 ms=160846487client: (finished: must succeed: sleep 5.0, in 5.04 seconds)6488client: waiting for success: ! systemctl is-active mid6489worker1 # [ 169.110575] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:47Z level=info event=niks3_push_started pid=17026490worker1 # [ 169.145360] nix-daemon[869]: reaped child process 1677, status = succeeded6491worker1 # [ 169.231735] nix-grpc-daemon[1702]: time=2026-09-23T07:01:47.511Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6492worker1 # [ 169.318273] nix-daemon[869]: accepted connection from pid 1715, user nix-grpc-daemon (trusted)6493worker1 # [ 169.330861] nix-daemon[869]: reaped child process 1718, status = succeeded6494lb # [ 169.336448] niks3-server[1225]: 2026/09/23 07:01:48 INFO Received uploads request method=POST path=/api/pending_closures6495worker1 # [ 169.349721] nix-grpc-daemon[1702]: time=2026-09-23T07:01:47.629Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6496worker1 # [ 169.351241] nix-grpc-daemon[1702]: time=2026-09-23T07:01:47.630Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6497lb # [ 169.385516] niks3-server[1225]: 2026/09/23 07:01:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6498lb # [ 169.390114] niks3-server[1225]: 2026/09/23 07:01:48 INFO Signed narinfos id=52 count=16499worker1 # [ 169.397144] nix-grpc-daemon[1702]: time=2026-09-23T07:01:47.676Z level=INFO msg="Uploading 1 narinfos"6500lb # [ 169.399662] niks3-server[1225]: 2026/09/23 07:01:48 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6501lb # [ 169.404115] niks3-server[1225]: 2026/09/23 07:01:48 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6502lb # [ 169.414559] niks3-server[1225]: 2026/09/23 07:01:48 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6503lb # [ 169.418316] niks3-server[1225]: 2026/09/23 07:01:48 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6504lb # [ 169.422063] niks3-server[1225]: 2026/09/23 07:01:48 INFO Completed upload id=526505worker1 # [ 169.426618] nix-grpc-daemon[1702]: time=2026-09-23T07:01:47.706Z level=INFO msg="Upload complete. (191ms)"6506lb # [ 169.424819] niks3-server[1225]: 2026/09/23 07:01:48 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6507worker1 # [ 169.431479] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43832 duration_s=4 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16508worker1 # [ 169.443843] nix-daemon[869]: reaped child process 1675, status = succeeded6509client # [ 169.477603] systemd[1]: mid.service: Deactivated successfully.6510client # [ 169.481341] systemd[1]: mid.service: Consumed 149ms CPU time over 9.486s wall clock time, 41.4M memory peak, 9K incoming IP traffic, 10.8K outgoing IP traffic.6511client: (finished: waiting for success: ! systemctl is-active mid, in 1.08 seconds)6512client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6513client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.03 seconds)6514(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.13 seconds)6515subtest: interrupting the client stops the build on the worker6516client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr6517client # [ 170.102948] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr.6518client # Running as unit: intr.service; invocation ID: e0cf5ac9c89148f0995b6dde377acaa66519client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr, in 0.03 seconds)6520worker2 # [ 170.226764] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:441226521worker2 # [ 170.237798] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6522client # [ 170.258876] nix[1437]: don't know how to build these paths:6523client # [ 170.259842] nix[1437]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6524worker2 # [ 170.256699] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:49Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06525worker2 # [ 170.260786] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:49Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16526worker2 # [ 170.305138] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226527worker2 # [ 170.316340] nix-grpc-daemon[1567]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 22 ms (attempt 1/5)6528worker2 # [ 170.345546] nix-grpc-daemon[1567]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 161 ms (attempt 2/5)6529worker2 # [ 170.508761] nix-grpc-daemon[1567]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 41 ms (attempt 3/5)6530worker2 # [ 170.551527] nix-grpc-daemon[1567]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 123 ms (attempt 4/5)6531worker2 # [ 170.682538] nix-grpc-daemon[1567]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6532worker2 # [ 170.685985] nix-grpc-daemon[1567]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6533worker2 # [ 170.703947] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:441226534client # [ 170.723337] nix[1437]: copying 1 paths to 'lb:50051'...6535worker2 # [ 170.716169] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:441226536client # [ 170.726127] nix[1437]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6537worker2 # [ 170.728455] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6538worker2 # [ 170.749460] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:49Z level=info event=niks3_push_started pid=16446539worker2 # [ 170.849726] nix-grpc-daemon[1644]: time=2026-09-23T07:01:50.035Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""6540worker2 # [ 170.934780] nix-daemon[855]: accepted connection from pid 1657, user nix-grpc-daemon (trusted)6541worker2 # [ 170.945905] nix-daemon[855]: reaped child process 1660, status = succeeded6542lb # [ 170.947584] niks3-server[1225]: 2026/09/23 07:01:50 INFO Received uploads request method=POST path=/api/pending_closures6543worker2 # [ 170.963680] nix-grpc-daemon[1644]: time=2026-09-23T07:01:50.149Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6544worker2 # [ 170.965202] nix-grpc-daemon[1644]: time=2026-09-23T07:01:50.150Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6545lb # [ 170.986139] niks3-server[1225]: 2026/09/23 07:01:50 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6546lb # [ 170.992860] niks3-server[1225]: 2026/09/23 07:01:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6547lb # [ 170.999815] niks3-server[1225]: 2026/09/23 07:01:50 INFO Signed narinfos id=53 count=16548worker2 # [ 171.010621] nix-grpc-daemon[1644]: time=2026-09-23T07:01:50.196Z level=INFO msg="Uploading 1 narinfos"6549lb # [ 171.009381] niks3-server[1225]: 2026/09/23 07:01:50 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6550lb # [ 171.018882] niks3-server[1225]: 2026/09/23 07:01:50 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6551lb # [ 171.024977] niks3-server[1225]: 2026/09/23 07:01:50 INFO Completed upload id=536552worker2 # [ 171.033074] nix-grpc-daemon[1644]: time=2026-09-23T07:01:50.218Z level=INFO msg="Upload complete. (180ms)"6553lb # [ 171.028480] niks3-server[1225]: 2026/09/23 07:01:50 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6554worker2 # [ 171.035552] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=0 paths=1 nar_bytes_in=4246555worker2 # [ 171.039158] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226556worker2 # [ 171.047252] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6557worker2 # [ 171.052638] nix-daemon[855]: reaped child process 1642, status = succeeded6558worker2 # [ 171.056689] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6559client # [ 171.064956] nix[1437]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6560client: must succeed: systemctl kill -s INT intr6561client # [ 172.426239] systemd[1]: intr.service: Sent signal SIGINT to main process 1437 (nix) on client request.6562worker2 # [ 172.420365] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:51Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6563client # [ 172.434797] nix[1437]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6564client: (finished: must succeed: systemctl kill -s INT intr, in 0.05 seconds)6565client # [ 172.440578] nix[1437]: error: interrupted by the user6566client # [ 172.465903] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6567client # [ 172.467192] systemd[1]: intr.service: Failed with result 'exit-code'.6568client # [ 172.468737] systemd[1]: intr.service: Consumed 117ms CPU time over 2.362s wall clock time, 41.4M memory peak, 6.3K incoming IP traffic, 8K outgoing IP traffic.6569worker2 # [ 172.472934] nix-daemon[1664]: unexpected Nix daemon error:6570worker2 # [ 172.474342] nix-daemon[855]: reaped child process 1662, status = succeeded6571worker2 # [ 172.476113] nix-daemon[1664]: error: interrupted by the user6572worker2 # [ 172.480419] nix-daemon[855]: reaped child process 1664, status = failed with exit code 16573(finished: subtest: interrupting the client stops the build on the worker, in 2.42 seconds)6574subtest: a deploy mid-build drains: the build finishes, the new generation takes over6575client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw6576client # [ 172.541473] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw.6577client # Running as unit: sw.service; invocation ID: e5b15a62358646bd9f640082f440db996578client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw, in 0.04 seconds)6579worker2 # [ 172.656138] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:441226580client # [ 172.669975] nix[1458]: don't know how to build these paths:6581client # [ 172.670984] nix[1458]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6582worker2 # [ 172.668638] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:51Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06583worker2 # [ 172.673265] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:51Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16584worker1 # [ 172.710776] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:438326585worker1 # [ 172.725102] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:438326586client # [ 172.747262] nix[1458]: copying 1 paths to 'lb:50051'...6587worker1 # [ 172.736782] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:438326588client # [ 172.749853] nix[1458]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6589worker1 # [ 172.749604] nix-daemon[869]: accepted connection from pid 1595, user nix-grpc-daemon (trusted)6590worker1 # [ 172.853468] nix-daemon[869]: accepted connection from pid 1742, user nix-grpc-daemon (trusted)6591worker1 # [ 172.866485] nix-daemon[869]: reaped child process 1745, status = succeeded6592lb # [ 172.870569] niks3-server[1225]: 2026/09/23 07:01:52 INFO Received uploads request method=POST path=/api/pending_closures6593worker1 # [ 172.883769] nix-grpc-daemon[1702]: time=2026-09-23T07:01:51.163Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6594worker1 # [ 172.886122] nix-grpc-daemon[1702]: time=2026-09-23T07:01:51.164Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6595lb # [ 172.902556] niks3-server[1225]: 2026/09/23 07:01:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6596worker1 # [ 172.915803] nix-grpc-daemon[1702]: time=2026-09-23T07:01:51.195Z level=INFO msg="Uploading 1 narinfos"6597lb # [ 172.910984] niks3-server[1225]: 2026/09/23 07:01:52 INFO Signed narinfos id=54 count=16598lb # [ 172.917288] niks3-server[1225]: 2026/09/23 07:01:52 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6599lb # [ 172.921744] niks3-server[1225]: 2026/09/23 07:01:52 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6600lb # [ 172.931273] niks3-server[1225]: 2026/09/23 07:01:52 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6601lb # [ 172.935088] niks3-server[1225]: 2026/09/23 07:01:52 INFO Completed upload id=546602worker1 # [ 172.939367] nix-grpc-daemon[1702]: time=2026-09-23T07:01:51.219Z level=INFO msg="Upload complete. (178ms)"6603lb # [ 172.937944] niks3-server[1225]: 2026/09/23 07:01:52 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6604worker1 # [ 172.943162] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43832 duration_s=0 paths=1 nar_bytes_in=4166605worker1 # [ 172.947933] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:438326606worker1 # [ 172.958162] nix-daemon[869]: accepted connection from pid 1595, user nix-grpc-daemon (trusted)6607worker1 # [ 172.966035] nix-daemon[869]: accepted connection from pid 1595, user nix-grpc-daemon (trusted)6608client # [ 172.977477] nix[1458]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6609worker1 # [ 172.968536] nix-daemon[869]: reaped child process 1740, status = succeeded6610worker1: must succeed: systemctl reload nix-grpc-daemon.service6611worker1 # [ 174.931545] systemd[1]: Reloading Nix worker-protocol over gRPC...6612worker1 # [ 174.962660] systemd[1]: Reloaded Nix worker-protocol over gRPC.6613worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.06 seconds)6614worker1: must succeed: pgrep -f 'read -t [9]0 x'6615worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.04 seconds)6616client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&26617client # don't know how to build these paths:6618client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6619worker2 # [ 175.157790] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:441226620worker2 # [ 175.163472] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:441226621worker2 # [ 175.186712] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:54Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06622worker2 # [ 175.191811] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:54Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16623worker2 # [ 175.237750] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226624worker2 # [ 175.250956] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:441226625client # copying 1 paths to 'lb:50051'...6626worker2 # [ 175.264267] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:441226627client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6628worker2 # [ 175.285178] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6629worker2 # [ 175.385314] nix-daemon[855]: accepted connection from pid 1715, user nix-grpc-daemon (trusted)6630worker2 # [ 175.397452] nix-daemon[855]: reaped child process 1718, status = succeeded6631lb # [ 175.400082] niks3-server[1225]: 2026/09/23 07:01:54 INFO Received uploads request method=POST path=/api/pending_closures6632worker2 # [ 175.415272] nix-grpc-daemon[1644]: time=2026-09-23T07:01:54.600Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6633worker2 # [ 175.417564] nix-grpc-daemon[1644]: time=2026-09-23T07:01:54.602Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6634worker2 # [ 175.438789] nix-grpc-daemon[1644]: time=2026-09-23T07:01:54.624Z level=INFO msg="Uploading 1 narinfos"6635lb # [ 175.430756] niks3-server[1225]: 2026/09/23 07:01:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6636lb # [ 175.434692] niks3-server[1225]: 2026/09/23 07:01:54 INFO Signed narinfos id=55 count=16637lb # [ 175.444914] niks3-server[1225]: 2026/09/23 07:01:54 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6638lb # [ 175.457822] niks3-server[1225]: 2026/09/23 07:01:54 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6639lb # [ 175.463234] niks3-server[1225]: 2026/09/23 07:01:54 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6640lb # [ 175.470466] niks3-server[1225]: 2026/09/23 07:01:54 INFO Completed upload id=556641lb # [ 175.473391] niks3-server[1225]: 2026/09/23 07:01:54 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6642worker2 # [ 175.480372] nix-grpc-daemon[1644]: time=2026-09-23T07:01:54.666Z level=INFO msg="Upload complete. (183ms)"6643worker2 # [ 175.484478] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=0 paths=1 nar_bytes_in=5286644worker2 # [ 175.489273] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226645worker2 # [ 175.498727] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6646worker2 # [ 175.506849] nix-daemon[855]: reaped child process 1713, status = succeeded6647worker2 # [ 175.511786] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6648client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6649client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6650worker1 # [ 175.672201] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:53Z level=info event=draining6651client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6652client # farm-a> LOG-farm-a-during-drain6653worker2 # [ 176.354180] nix-daemon[855]: reaped child process 1722, status = succeeded6654worker2 # [ 176.429537] nix-daemon[855]: accepted connection from pid 1738, user nix-grpc-daemon (trusted)6655worker2 # [ 176.441162] nix-daemon[855]: reaped child process 1741, status = succeeded6656lb # [ 176.440494] niks3-server[1225]: 2026/09/23 07:01:55 INFO Received uploads request method=POST path=/api/pending_closures6657worker2 # [ 176.457406] nix-grpc-daemon[1644]: time=2026-09-23T07:01:55.642Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6658worker2 # [ 176.458915] nix-grpc-daemon[1644]: time=2026-09-23T07:01:55.644Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6659lb # [ 176.475164] niks3-server[1225]: 2026/09/23 07:01:55 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6660lb # [ 176.491190] niks3-server[1225]: 2026/09/23 07:01:55 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6661lb # [ 176.498861] niks3-server[1225]: 2026/09/23 07:01:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6662lb # [ 176.504780] niks3-server[1225]: 2026/09/23 07:01:55 INFO Signed narinfos id=56 count=16663worker2 # [ 176.513428] nix-grpc-daemon[1644]: time=2026-09-23T07:01:55.698Z level=INFO msg="Uploading 1 narinfos"6664lb # [ 176.508357] niks3-server[1225]: 2026/09/23 07:01:55 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6665lb # [ 176.518582] niks3-server[1225]: 2026/09/23 07:01:55 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6666lb # [ 176.522758] niks3-server[1225]: 2026/09/23 07:01:55 INFO Completed upload id=566667worker2 # [ 176.530465] nix-grpc-daemon[1644]: time=2026-09-23T07:01:55.716Z level=INFO msg="Upload complete. (211ms)"6668lb # [ 176.525262] niks3-server[1225]: 2026/09/23 07:01:55 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6669worker2 # [ 176.536808] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16670worker2 # [ 176.549642] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:55Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06671worker2 # [ 176.553166] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:55Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16672worker2 # [ 176.561535] nix-daemon[855]: reaped child process 1720, status = succeeded6673worker2 # [ 176.594226] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226674worker2 # [ 176.607493] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:441226675worker2 # [ 176.621761] nix-grpc-daemon[1644]: time=2026-09-23T07:01:55.807Z level=INFO msg="All 1 paths already cached"6676lb # [ 176.614965] envoy[912]: conn peer=[2001:db8:1::1]:41910 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4106 tx=2034 ms=93286677client # copying 1 paths to 'lb:50051'...6678lb # [ 176.619508] envoy[912]: conn peer=[2001:db8:1::1]:33246 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2924 tx=812 ms=22246679worker2 # [ 176.636833] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:441226680client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6681worker2 # [ 176.656230] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6682worker2 # [ 176.757932] nix-daemon[855]: accepted connection from pid 1747, user nix-grpc-daemon (trusted)6683worker2 # [ 176.770231] nix-daemon[855]: reaped child process 1750, status = succeeded6684lb # [ 176.770386] niks3-server[1225]: 2026/09/23 07:01:55 INFO Received uploads request method=POST path=/api/pending_closures6685worker2 # [ 176.788082] nix-grpc-daemon[1644]: time=2026-09-23T07:01:55.973Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6686worker2 # [ 176.789597] nix-grpc-daemon[1644]: time=2026-09-23T07:01:55.975Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6687lb # [ 176.804898] niks3-server[1225]: 2026/09/23 07:01:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6688worker2 # [ 176.818854] nix-grpc-daemon[1644]: time=2026-09-23T07:01:56.004Z level=INFO msg="Uploading 1 narinfos"6689lb # [ 176.811223] niks3-server[1225]: 2026/09/23 07:01:56 INFO Signed narinfos id=57 count=16690lb # [ 176.815661] niks3-server[1225]: 2026/09/23 07:01:56 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6691lb # [ 176.825192] niks3-server[1225]: 2026/09/23 07:01:56 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6692lb # [ 176.834882] niks3-server[1225]: 2026/09/23 07:01:56 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6693lb # [ 176.841940] niks3-server[1225]: 2026/09/23 07:01:56 INFO Completed upload id=576694worker2 # [ 176.849781] nix-grpc-daemon[1644]: time=2026-09-23T07:01:56.035Z level=INFO msg="Upload complete. (181ms)"6695lb # [ 176.847181] niks3-server[1225]: 2026/09/23 07:01:56 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6696worker2 # [ 176.853989] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=0 paths=1 nar_bytes_in=6406697worker2 # [ 176.860576] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226698worker2 # [ 176.869936] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6699worker2 # [ 176.880224] nix-daemon[855]: reaped child process 1745, status = succeeded6700worker2 # [ 176.883145] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6701client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6702client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6703client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6704client # farm-b> LOG-farm-b-during-drain6705worker2 # [ 178.116921] nix-daemon[855]: reaped child process 1754, status = succeeded6706worker2 # [ 178.196341] nix-daemon[855]: accepted connection from pid 1770, user nix-grpc-daemon (trusted)6707worker2 # [ 178.207269] nix-daemon[855]: reaped child process 1773, status = succeeded6708lb # [ 178.207982] niks3-server[1225]: 2026/09/23 07:01:57 INFO Received uploads request method=POST path=/api/pending_closures6709worker2 # [ 178.227713] nix-grpc-daemon[1644]: time=2026-09-23T07:01:57.413Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6710worker2 # [ 178.229171] nix-grpc-daemon[1644]: time=2026-09-23T07:01:57.414Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6711lb # [ 178.253184] niks3-server[1225]: 2026/09/23 07:01:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6712lb # [ 178.259530] niks3-server[1225]: 2026/09/23 07:01:57 INFO Signed narinfos id=58 count=16713worker2 # [ 178.272074] nix-grpc-daemon[1644]: time=2026-09-23T07:01:57.457Z level=INFO msg="Uploading 1 narinfos"6714lb # [ 178.271552] niks3-server[1225]: 2026/09/23 07:01:57 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6715lb # [ 178.277039] niks3-server[1225]: 2026/09/23 07:01:57 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6716lb # [ 178.285744] niks3-server[1225]: 2026/09/23 07:01:57 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6717lb # [ 178.289863] niks3-server[1225]: 2026/09/23 07:01:57 INFO Completed upload id=586718worker2 # [ 178.297661] nix-grpc-daemon[1644]: time=2026-09-23T07:01:57.483Z level=INFO msg="Upload complete. (205ms)"6719lb # [ 178.292734] niks3-server[1225]: 2026/09/23 07:01:57 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6720lb # [ 178.295248] niks3-server[1225]: 2026/09/23 07:01:57 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6721worker2 # [ 178.304471] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16722worker2 # [ 178.316505] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:57Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06723worker2 # [ 178.320427] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:57Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16724worker2 # [ 178.329598] nix-daemon[855]: reaped child process 1752, status = succeeded6725worker2 # [ 178.361232] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226726worker2 # [ 178.374296] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:441226727worker2 # [ 178.388233] nix-grpc-daemon[1644]: time=2026-09-23T07:01:57.573Z level=INFO msg="All 2 paths already cached"6728client # copying 1 paths to 'lb:50051'...6729client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6730worker2 # [ 178.397843] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:441226731worker2 # [ 178.405982] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6732worker2 # [ 178.500936] nix-daemon[855]: accepted connection from pid 1779, user nix-grpc-daemon (trusted)6733worker2 # [ 178.513864] nix-daemon[855]: reaped child process 1782, status = succeeded6734lb # [ 178.512960] niks3-server[1225]: 2026/09/23 07:01:57 INFO Received uploads request method=POST path=/api/pending_closures6735worker2 # [ 178.529141] nix-grpc-daemon[1644]: time=2026-09-23T07:01:57.714Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6736worker2 # [ 178.530604] nix-grpc-daemon[1644]: time=2026-09-23T07:01:57.716Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6737lb # [ 178.547774] niks3-server[1225]: 2026/09/23 07:01:57 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6738lb # [ 178.558154] niks3-server[1225]: 2026/09/23 07:01:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6739lb # [ 178.563940] niks3-server[1225]: 2026/09/23 07:01:57 INFO Signed narinfos id=59 count=16740worker2 # [ 178.575028] nix-grpc-daemon[1644]: time=2026-09-23T07:01:57.760Z level=INFO msg="Uploading 1 narinfos"6741lb # [ 178.569743] niks3-server[1225]: 2026/09/23 07:01:57 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6742lb # [ 178.583440] niks3-server[1225]: 2026/09/23 07:01:57 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6743lb # [ 178.589940] niks3-server[1225]: 2026/09/23 07:01:57 INFO Completed upload id=596744worker2 # [ 178.597704] nix-grpc-daemon[1644]: time=2026-09-23T07:01:57.783Z level=INFO msg="Upload complete. (178ms)"6745lb # [ 178.593126] niks3-server[1225]: 2026/09/23 07:01:57 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6746worker2 # [ 178.602402] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=0 paths=1 nar_bytes_in=7686747worker2 # [ 178.609336] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226748worker2 # [ 178.620085] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6749worker2 # [ 178.629287] nix-daemon[855]: reaped child process 1777, status = succeeded6750client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6751worker2 # [ 178.635593] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6752client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6753client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6754client # farm-top> LOG-farm-top-during-drain6755worker2 # [ 179.872752] nix-daemon[855]: reaped child process 1786, status = succeeded6756worker2 # [ 179.948144] nix-daemon[855]: accepted connection from pid 1802, user nix-grpc-daemon (trusted)6757worker2 # [ 179.960760] nix-daemon[855]: reaped child process 1805, status = succeeded6758lb # [ 179.961276] niks3-server[1225]: 2026/09/23 07:01:59 INFO Received uploads request method=POST path=/api/pending_closures6759worker2 # [ 179.978069] nix-grpc-daemon[1644]: time=2026-09-23T07:01:59.163Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6760worker2 # [ 179.979522] nix-grpc-daemon[1644]: time=2026-09-23T07:01:59.165Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6761lb # [ 180.002443] niks3-server[1225]: 2026/09/23 07:01:59 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6762lb # [ 180.016065] niks3-server[1225]: 2026/09/23 07:01:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6763worker2 # [ 180.033083] nix-grpc-daemon[1644]: time=2026-09-23T07:01:59.218Z level=INFO msg="Uploading 1 narinfos"6764lb # [ 180.025458] niks3-server[1225]: 2026/09/23 07:01:59 INFO Signed narinfos id=60 count=16765lb # [ 180.030057] niks3-server[1225]: 2026/09/23 07:01:59 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6766lb # [ 180.045451] niks3-server[1225]: 2026/09/23 07:01:59 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6767lb # [ 180.049682] niks3-server[1225]: 2026/09/23 07:01:59 INFO Completed upload id=606768worker2 # [ 180.057378] nix-grpc-daemon[1644]: time=2026-09-23T07:01:59.243Z level=INFO msg="Upload complete. (215ms)"6769lb # [ 180.051766] niks3-server[1225]: 2026/09/23 07:01:59 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6770lb # [ 180.057288] niks3-server[1225]: 2026/09/23 07:01:59 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6771worker2 # [ 180.063621] nix-grpc-daemon[1567]: ts=2026-09-23T07:01:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16772worker2 # [ 180.080425] nix-daemon[855]: reaped child process 1784, status = succeeded6773client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&2, in 5.14 seconds)6774client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath6775client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath, in 0.12 seconds)6776worker1: must succeed: pkill -f 'read -t [9]0 x'6777worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.04 seconds)6778client: waiting for success: ! systemctl is-active sw6779worker1 # [ 180.388296] nix-daemon[869]: reaped child process 1749, status = succeeded6780worker1 # [ 180.466297] nix-daemon[869]: accepted connection from pid 1790, user nix-grpc-daemon (trusted)6781worker1 # [ 180.478145] nix-daemon[869]: reaped child process 1793, status = succeeded6782lb # [ 180.481155] niks3-server[1225]: 2026/09/23 07:01:59 INFO Received uploads request method=POST path=/api/pending_closures6783worker1 # [ 180.497331] nix-grpc-daemon[1702]: time=2026-09-23T07:01:58.776Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6784worker1 # [ 180.499065] nix-grpc-daemon[1702]: time=2026-09-23T07:01:58.778Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6785lb # [ 180.527359] niks3-server[1225]: 2026/09/23 07:01:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6786lb # [ 180.533277] niks3-server[1225]: 2026/09/23 07:01:59 INFO Signed narinfos id=61 count=16787worker1 # [ 180.540233] nix-grpc-daemon[1702]: time=2026-09-23T07:01:58.819Z level=INFO msg="Uploading 1 narinfos"6788lb # [ 180.538644] niks3-server[1225]: 2026/09/23 07:01:59 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6789lb # [ 180.543091] niks3-server[1225]: 2026/09/23 07:01:59 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6790lb # [ 180.550551] niks3-server[1225]: 2026/09/23 07:01:59 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6791lb # [ 180.556060] niks3-server[1225]: 2026/09/23 07:01:59 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6792lb # [ 180.559364] niks3-server[1225]: 2026/09/23 07:01:59 INFO Completed upload id=616793worker1 # [ 180.563539] nix-grpc-daemon[1702]: time=2026-09-23T07:01:58.843Z level=INFO msg="Upload complete. (183ms)"6794lb # [ 180.562797] niks3-server[1225]: 2026/09/23 07:01:59 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6795worker1 # [ 180.570695] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43832 duration_s=7 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16796worker1 # [ 180.584582] nix-daemon[869]: reaped child process 1747, status = succeeded6797client # [ 180.615188] systemd[1]: sw.service: Deactivated successfully.6798client # [ 180.618377] systemd[1]: sw.service: Consumed 120ms CPU time over 8.072s wall clock time, 43.4M memory peak, 7.2K incoming IP traffic, 8.3K outgoing IP traffic.6799worker1 # [ 180.674756] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:58Z level=info event=signal_exit6800client: (finished: waiting for success: ! systemctl is-active sw, in 1.06 seconds)6801client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6802client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.03 seconds)6803worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6804worker2 # [ 181.688774] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:00Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:440966805worker2 # [ 181.695734] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:00Z level=info event=worker_gone addr=192.168.1.3:500516806worker1 # [ 181.680998] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:59Z level=info event=scheduler_restarting6807worker1 # [ 181.699931] nix-grpc-daemon[1595]: ts=2026-09-23T07:01:59Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6808worker1 # [ 181.719444] nix-daemon[869]: reaped child process 1668, status = succeeded6809worker1 # [ 182.597847] nix-grpc-daemon[1595]: E0000 00:00:1790146920.877342 1641 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true6810worker1 # [ 183.709295] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6811worker1 # [ 183.712371] systemd[1]: nix-grpc-daemon.service: Consumed 566ms CPU time over 24.146s wall clock time, 35M memory peak, 21.4K incoming IP traffic, 28.5K outgoing IP traffic.6812worker1 # [ 183.728188] systemd[1]: Starting Nix worker-protocol over gRPC...6813worker1 # [ 183.819157] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:02Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6814worker1 # [ 183.821219] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:02Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6815worker1 # [ 183.825309] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:02Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16816worker1 # [ 183.830066] nix-grpc-daemon[1807]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6817worker1 # [ 183.831434] nix-grpc-daemon[1807]: E0000 00:00:1790146922.108253 1807 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6818worker1 # [ 183.837061] nix-grpc-daemon[1807]: E0000 00:00:1790146922.113601 1807 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6819worker1 # [ 183.839203] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:02Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6820worker1 # [ 183.845081] systemd[1]: Started Nix worker-protocol over gRPC.6821worker1 # [ 183.847134] nix-grpc-daemon[1807]: E0000 00:00:1790146922.123865 1850 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6822worker1 # [ 183.859501] nix-grpc-daemon[1807]: E0000 00:00:1790146922.138429 1850 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6823worker1 # [ 183.863138] nix-grpc-daemon[1807]: E0000 00:00:1790146922.138547 1850 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6824worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.47 seconds)6825lb: waiting until exactly ['worker1', 'worker2'] are healthy members of x86_64-linux6826lb: must succeed: curl -sf localhost:9901/clusters6827worker1 # [ 183.874062] nix-grpc-daemon[1807]: E0000 00:00:1790146922.153674 1850 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6828worker1 # [ 183.882060] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:02Z level=info event=scheduler_connected addr=lb:500516829worker2 # [ 183.887457] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:440966830worker2 # [ 183.892780] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06831lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6832lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of x86_64-linux, in 0.03 seconds)6833lb: must succeed: curl -sf localhost:9901/clusters6834lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6835worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6836worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6837(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 11.47 seconds)6838subtest: queue gauges show a burst while every build blocks6839lb: must succeed: curl -sf localhost:9901/clusters6840lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6841lb: must succeed: curl -sf localhost:9901/clusters6842lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6843worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6844worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6845client: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]'6846client # [ 184.110717] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto --impure --expr "map (tag: import /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix { inherit tag; }) [ \"q0\" \"q1\" \"q2\" \"q3\" ]".6847client # Running as unit: burst.service; invocation ID: 970769757c51462f8dd720ec3195b9766848client: (finished: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]', in 0.07 seconds)6849lb: must succeed: curl -sf localhost:9901/clusters6850lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6851lb: must succeed: curl -sf localhost:9901/clusters6852lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6853worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6854worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6855worker1 # [ 184.245168] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:513726856worker1 # [ 184.256722] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)6857client # [ 184.280660] nix[1514]: don't know how to build these paths:6858client # [ 184.281535] nix[1514]: /nix/store/2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv6859client # [ 184.282507] nix[1514]: /nix/store/885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv6860client # [ 184.284100] nix[1514]: /nix/store/8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv6861client # [ 184.285291] nix[1514]: /nix/store/qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv6862worker2 # [ 184.283337] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=debug event=want client=5 drv=2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv cached=06863worker2 # [ 184.289219] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=debug event=assigned drv=2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv worker=192.168.1.4:50051 clients=16864worker2 # [ 184.293643] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=debug event=want client=5 drv=885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv cached=06865worker2 # [ 184.297493] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=debug event=assigned drv=885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv worker=192.168.1.3:50051 clients=16866worker2 # [ 184.301777] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=debug event=want client=5 drv=8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv cached=06867worker2 # [ 184.305762] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=debug event=want client=5 drv=qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv cached=06868worker1 # [ 184.328699] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513726869worker2 # [ 184.332508] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226870worker1 # [ 184.339810] nix-grpc-daemon[1807]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 82 ms (attempt 1/5)6871worker2 # [ 184.346333] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:441226872client # [ 184.365348] nix[1514]: copying 1 paths to 'lb:50051'...6873worker2 # [ 184.358140] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:441226874client # [ 184.368171] nix[1514]: copying path '/nix/store/2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv' to 'lb:50051'...6875worker2 # [ 184.370950] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6876worker1 # [ 184.429964] nix-grpc-daemon[1807]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 129 ms (attempt 2/5)6877worker2 # [ 184.466429] nix-daemon[855]: accepted connection from pid 1820, user nix-grpc-daemon (trusted)6878worker2 # [ 184.479348] nix-daemon[855]: reaped child process 1823, status = succeeded6879lb # [ 184.478612] niks3-server[1225]: 2026/09/23 07:02:03 INFO Received uploads request method=POST path=/api/pending_closures6880worker2 # [ 184.495267] nix-grpc-daemon[1644]: time=2026-09-23T07:02:03.680Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6881worker2 # [ 184.497264] nix-grpc-daemon[1644]: time=2026-09-23T07:02:03.682Z level=INFO msg="Uploading 2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv (416B)"6882lb # [ 184.509270] niks3-server[1225]: 2026/09/23 07:02:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6883worker2 # [ 184.522163] nix-grpc-daemon[1644]: time=2026-09-23T07:02:03.707Z level=INFO msg="Uploading 1 narinfos"6884lb # [ 184.514190] niks3-server[1225]: 2026/09/23 07:02:03 INFO Signed narinfos id=62 count=16885lb # [ 184.528379] niks3-server[1225]: 2026/09/23 07:02:03 INFO Registered completed upload object_key=2s9psgqm7awsyhlfyaclma7yr7qmrk0g.ls6886lb # [ 184.533297] niks3-server[1225]: 2026/09/23 07:02:03 INFO Registered completed upload object_key=nar/0raymp7y3ngiv7lj2qq6nkp63sms2510yhhlnr7hnz7ldnl7b3qc.nar.zst6887lb # [ 184.539905] niks3-server[1225]: 2026/09/23 07:02:03 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6888lb # [ 184.547819] niks3-server[1225]: 2026/09/23 07:02:03 INFO Completed upload id=626889worker2 # [ 184.555529] nix-grpc-daemon[1644]: time=2026-09-23T07:02:03.741Z level=INFO msg="Upload complete. (173ms)"6890lb # [ 184.550351] niks3-server[1225]: 2026/09/23 07:02:03 INFO Registered completed upload object_key=2s9psgqm7awsyhlfyaclma7yr7qmrk0g.narinfo6891worker2 # [ 184.559725] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=0 paths=1 nar_bytes_in=4166892worker2 # [ 184.565267] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226893worker1 # [ 184.566924] nix-grpc-daemon[1807]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 318 ms (attempt 3/5)6894worker2 # [ 184.575471] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6895worker2 # [ 184.581127] nix-daemon[855]: reaped child process 1818, status = succeeded6896worker2 # [ 184.583941] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6897client # [ 184.592127] nix[1514]: building '/nix/store/2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv' on lb:50051...6898worker1 # [ 184.887770] nix-grpc-daemon[1807]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 426 ms (attempt 4/5)6899lb: must succeed: curl -sf localhost:9901/clusters6900lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6901lb: must succeed: curl -sf localhost:9901/clusters6902lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6903worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6904worker1 # [ 185.315483] nix-grpc-daemon[1807]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6905worker1 # [ 185.319382] nix-grpc-daemon[1807]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6906worker1 # [ 185.335271] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513726907client # [ 185.356738] nix[1514]: copying 1 paths to 'lb:50051'...6908client # [ 185.360631] nix[1514]: copying path '/nix/store/885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv' to 'lb:50051'...6909worker1 # [ 185.349330] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:513726910worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6911worker1 # [ 185.361901] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)6912worker1 # [ 185.388989] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:03Z level=info event=niks3_push_started pid=18726913worker1 # [ 185.491570] nix-grpc-daemon[1872]: time=2026-09-23T07:02:03.770Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6914worker1 # [ 185.580199] nix-daemon[869]: accepted connection from pid 1885, user nix-grpc-daemon (trusted)6915worker1 # [ 185.593210] nix-daemon[869]: reaped child process 1888, status = succeeded6916lb # [ 185.596610] niks3-server[1225]: 2026/09/23 07:02:04 INFO Received uploads request method=POST path=/api/pending_closures6917worker1 # [ 185.609627] nix-grpc-daemon[1872]: time=2026-09-23T07:02:03.889Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6918worker1 # [ 185.611291] nix-grpc-daemon[1872]: time=2026-09-23T07:02:03.890Z level=INFO msg="Uploading 885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv (416B)"6919lb # [ 185.635514] niks3-server[1225]: 2026/09/23 07:02:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6920lb # [ 185.645871] niks3-server[1225]: 2026/09/23 07:02:04 INFO Signed narinfos id=63 count=16921worker1 # [ 185.654081] nix-grpc-daemon[1872]: time=2026-09-23T07:02:03.933Z level=INFO msg="Uploading 1 narinfos"6922lb # [ 185.649821] niks3-server[1225]: 2026/09/23 07:02:04 INFO Registered completed upload object_key=885n55y9srkrjs4p487lr255wvhla7hq.ls6923lb # [ 185.656996] niks3-server[1225]: 2026/09/23 07:02:04 INFO Registered completed upload object_key=nar/0mmcg3cvna05ngbx6v5vspkmc5gj8lna2zc48rllhxr0g8i9mg6y.nar.zst6924lb # [ 185.672943] niks3-server[1225]: 2026/09/23 07:02:04 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6925lb # [ 185.678028] niks3-server[1225]: 2026/09/23 07:02:04 INFO Completed upload id=636926worker1 # [ 185.682218] nix-grpc-daemon[1872]: time=2026-09-23T07:02:03.961Z level=INFO msg="Upload complete. (187ms)"6927lb # [ 185.680206] niks3-server[1225]: 2026/09/23 07:02:04 INFO Registered completed upload object_key=885n55y9srkrjs4p487lr255wvhla7hq.narinfo6928worker1 # [ 185.685211] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 paths=1 nar_bytes_in=4166929worker1 # [ 185.690407] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513726930worker1 # [ 185.701106] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)6931worker1 # [ 185.705158] nix-daemon[869]: reaped child process 1868, status = succeeded6932worker1 # [ 185.710821] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)6933client # [ 185.722552] nix[1514]: building '/nix/store/885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv' on lb:50051...6934lb # [ 186.615869] envoy[912]: conn peer=[2001:db8:1::1]:55506 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9581 tx=4382 ms=49756935lb # [ 186.619734] envoy[912]: conn peer=[2001:db8:1::1]:33260 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2958 tx=1498 ms=79276936lb # [ 186.623507] envoy[912]: conn peer=[2001:db8:1::3]:37246 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1148 tx=482 ms=219746937worker2 # [ 186.823516] nix-daemon[855]: reaped child process 1827, status = succeeded6938worker2 # [ 186.905969] nix-daemon[855]: accepted connection from pid 1855, user nix-grpc-daemon (trusted)6939worker2 # [ 186.917116] nix-daemon[855]: reaped child process 1858, status = succeeded6940lb # [ 186.917337] niks3-server[1225]: 2026/09/23 07:02:06 INFO Received uploads request method=POST path=/api/pending_closures6941worker2 # [ 186.934264] nix-grpc-daemon[1644]: time=2026-09-23T07:02:06.119Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6942worker2 # [ 186.935729] nix-grpc-daemon[1644]: time=2026-09-23T07:02:06.121Z level=INFO msg="Uploading vn3s5fgnxq5d2l4d4nn96l2q9wfdg4mq-slow-q3 (120B)"6943lb # [ 186.958741] niks3-server[1225]: 2026/09/23 07:02:06 INFO Registered completed upload object_key=vn3s5fgnxq5d2l4d4nn96l2q9wfdg4mq.ls6944lb # [ 186.969997] niks3-server[1225]: 2026/09/23 07:02:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6945lb # [ 186.977266] niks3-server[1225]: 2026/09/23 07:02:06 INFO Signed narinfos id=64 count=16946worker2 # [ 186.988344] nix-grpc-daemon[1644]: time=2026-09-23T07:02:06.173Z level=INFO msg="Uploading 1 narinfos"6947lb # [ 186.986410] niks3-server[1225]: 2026/09/23 07:02:06 INFO Registered completed upload object_key=log/2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv6948lb # [ 186.991812] niks3-server[1225]: 2026/09/23 07:02:06 INFO Registered completed upload object_key=nar/1v1grzkvw3yj09x0dkdm2avihvqvnri156qzp768hk85zbw7lkn9.nar.zst6949lb # [ 186.999906] niks3-server[1225]: 2026/09/23 07:02:06 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6950lb # [ 187.003942] niks3-server[1225]: 2026/09/23 07:02:06 INFO Completed upload id=646951worker2 # [ 187.011938] nix-grpc-daemon[1644]: time=2026-09-23T07:02:06.197Z level=INFO msg="Upload complete. (219ms)"6952lb # [ 187.006256] niks3-server[1225]: 2026/09/23 07:02:06 INFO Registered completed upload object_key=vn3s5fgnxq5d2l4d4nn96l2q9wfdg4mq.narinfo6953worker2 # [ 187.018598] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=2 drv=2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv assign_id=7 outputs=16954worker2 # [ 187.027372] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:06Z level=debug event=assigned drv=8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv worker=192.168.1.4:50051 clients=16955worker2 # [ 187.038801] nix-daemon[855]: reaped child process 1825, status = succeeded6956worker2 # [ 187.070598] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226957worker2 # [ 187.084183] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:441226958client # [ 187.104068] nix[1514]: copying 1 paths to 'lb:50051'...6959worker2 # [ 187.096171] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:441226960client # [ 187.106387] nix[1514]: copying path '/nix/store/8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv' to 'lb:50051'...6961worker2 # [ 187.106642] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6962worker2 # [ 187.202388] nix-daemon[855]: accepted connection from pid 1862, user nix-grpc-daemon (trusted)6963worker2 # [ 187.214761] nix-daemon[855]: reaped child process 1865, status = succeeded6964lb # [ 187.214644] niks3-server[1225]: 2026/09/23 07:02:06 INFO Received uploads request method=POST path=/api/pending_closures6965worker2 # [ 187.230918] nix-grpc-daemon[1644]: time=2026-09-23T07:02:06.416Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6966worker2 # [ 187.232382] nix-grpc-daemon[1644]: time=2026-09-23T07:02:06.418Z level=INFO msg="Uploading 8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv (416B)"6967lb # [ 187.250203] niks3-server[1225]: 2026/09/23 07:02:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6968worker2 # [ 187.265034] nix-grpc-daemon[1644]: time=2026-09-23T07:02:06.450Z level=INFO msg="Uploading 1 narinfos"6969lb # [ 187.257332] niks3-server[1225]: 2026/09/23 07:02:06 INFO Signed narinfos id=65 count=16970lb # [ 187.264149] niks3-server[1225]: 2026/09/23 07:02:06 INFO Registered completed upload object_key=8v867csm7d9sip4xd4nlg9jwvrpqqwv9.ls6971lb # [ 187.268388] niks3-server[1225]: 2026/09/23 07:02:06 INFO Registered completed upload object_key=nar/1xibay372y9by9a678y9h7bfzwahsxi0vq51acgywak31q6q2m87.nar.zst6972lb # [ 187.283448] niks3-server[1225]: 2026/09/23 07:02:06 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6973lb # [ 187.288887] niks3-server[1225]: 2026/09/23 07:02:06 INFO Completed upload id=656974worker2 # [ 187.296858] nix-grpc-daemon[1644]: time=2026-09-23T07:02:06.482Z level=INFO msg="Upload complete. (176ms)"6975lb # [ 187.290666] niks3-server[1225]: 2026/09/23 07:02:06 INFO Registered completed upload object_key=8v867csm7d9sip4xd4nlg9jwvrpqqwv9.narinfo6976worker2 # [ 187.301216] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=0 paths=1 nar_bytes_in=4166977worker2 # [ 187.306927] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441226978worker2 # [ 187.318984] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6979worker2 # [ 187.328905] nix-daemon[855]: reaped child process 1860, status = succeeded6980worker2 # [ 187.331714] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6981client # [ 187.339777] nix[1514]: building '/nix/store/8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv' on lb:50051...6982worker1 # [ 187.727338] nix-daemon[869]: reaped child process 1892, status = succeeded6983worker1 # [ 187.802914] nix-daemon[869]: accepted connection from pid 1917, user nix-grpc-daemon (trusted)6984worker1 # [ 187.815608] nix-daemon[869]: reaped child process 1920, status = succeeded6985lb # [ 187.818968] niks3-server[1225]: 2026/09/23 07:02:07 INFO Received uploads request method=POST path=/api/pending_closures6986worker1 # [ 187.832309] nix-grpc-daemon[1872]: time=2026-09-23T07:02:06.111Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6987worker1 # [ 187.834030] nix-grpc-daemon[1872]: time=2026-09-23T07:02:06.113Z level=INFO msg="Uploading nv0kp5b571hsjibnd9v06981i024iima-slow-q2 (120B)"6988lb # [ 187.858549] niks3-server[1225]: 2026/09/23 07:02:07 INFO Registered completed upload object_key=log/885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv6989lb # [ 187.871060] niks3-server[1225]: 2026/09/23 07:02:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6990lb # [ 187.876432] niks3-server[1225]: 2026/09/23 07:02:07 INFO Signed narinfos id=66 count=16991worker1 # [ 187.884420] nix-grpc-daemon[1872]: time=2026-09-23T07:02:06.163Z level=INFO msg="Uploading 1 narinfos"6992lb # [ 187.884417] niks3-server[1225]: 2026/09/23 07:02:07 INFO Registered completed upload object_key=nv0kp5b571hsjibnd9v06981i024iima.ls6993lb # [ 187.897061] niks3-server[1225]: 2026/09/23 07:02:07 INFO Registered completed upload object_key=nar/15imdx7zyy6mzjjv6qvdzywh0d9w1n6ikc0rq4rq95aw9x6swfzj.nar.zst6994lb # [ 187.898827] niks3-server[1225]: 2026/09/23 07:02:07 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6995lb # [ 187.902944] niks3-server[1225]: 2026/09/23 07:02:07 INFO Completed upload id=666996worker1 # [ 187.907218] nix-grpc-daemon[1872]: time=2026-09-23T07:02:06.186Z level=INFO msg="Upload complete. (188ms)"6997lb # [ 187.904681] niks3-server[1225]: 2026/09/23 07:02:07 INFO Registered completed upload object_key=nv0kp5b571hsjibnd9v06981i024iima.narinfo6998worker1 # [ 187.912926] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=2 drv=885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv assign_id=8 outputs=16999worker2 # [ 187.917275] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:07Z level=debug event=assigned drv=qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv worker=192.168.1.3:50051 clients=17000worker1 # [ 187.921290] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727001worker1 # [ 187.931293] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727002worker1 # [ 187.934342] nix-daemon[869]: reaped child process 1890, status = succeeded7003client # [ 187.949179] nix[1514]: copying 1 paths to 'lb:50051'...7004worker1 # [ 187.940230] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:513727005client # [ 187.951736] nix[1514]: copying path '/nix/store/qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv' to 'lb:50051'...7006worker1 # [ 187.946172] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7007worker1 # [ 188.031504] nix-daemon[869]: accepted connection from pid 1924, user nix-grpc-daemon (trusted)7008worker1 # [ 188.044567] nix-daemon[869]: reaped child process 1927, status = succeeded7009lb # [ 188.048379] niks3-server[1225]: 2026/09/23 07:02:07 INFO Received uploads request method=POST path=/api/pending_closures7010worker1 # [ 188.060981] nix-grpc-daemon[1872]: time=2026-09-23T07:02:06.340Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7011worker1 # [ 188.062461] nix-grpc-daemon[1872]: time=2026-09-23T07:02:06.342Z level=INFO msg="Uploading qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv (416B)"7012lb # [ 188.082622] niks3-server[1225]: 2026/09/23 07:02:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7013lb # [ 188.088379] niks3-server[1225]: 2026/09/23 07:02:07 INFO Signed narinfos id=67 count=17014worker1 # [ 188.096649] nix-grpc-daemon[1872]: time=2026-09-23T07:02:06.376Z level=INFO msg="Uploading 1 narinfos"7015lb # [ 188.098204] niks3-server[1225]: 2026/09/23 07:02:07 INFO Registered completed upload object_key=qs0cgdr560r5cllf2djj9bb4gz0vrp4p.ls7016lb # [ 188.102581] niks3-server[1225]: 2026/09/23 07:02:07 INFO Registered completed upload object_key=nar/1spxkiwzvlj9sgbjal4jf4p68h5kbylxymfww73s5va157qxgafi.nar.zst7017lb # [ 188.113388] niks3-server[1225]: 2026/09/23 07:02:07 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7018lb # [ 188.119315] niks3-server[1225]: 2026/09/23 07:02:07 INFO Completed upload id=677019worker1 # [ 188.123694] nix-grpc-daemon[1872]: time=2026-09-23T07:02:06.403Z level=INFO msg="Upload complete. (171ms)"7020lb # [ 188.121489] niks3-server[1225]: 2026/09/23 07:02:07 INFO Registered completed upload object_key=qs0cgdr560r5cllf2djj9bb4gz0vrp4p.narinfo7021worker1 # [ 188.127849] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 paths=1 nar_bytes_in=4167022worker1 # [ 188.133096] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727023worker1 # [ 188.142468] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7024worker1 # [ 188.151118] nix-daemon[869]: reaped child process 1922, status = succeeded7025worker1 # [ 188.154395] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7026client # [ 188.166825] nix[1514]: building '/nix/store/qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv' on lb:50051...7027worker2 # [ 188.957484] nix-daemon[855]: reaped child process 1869, status = succeeded7028worker2 # [ 189.036039] nix-daemon[855]: accepted connection from pid 1894, user nix-grpc-daemon (trusted)7029worker2 # [ 189.046967] nix-daemon[855]: reaped child process 1897, status = succeeded7030lb # [ 189.046860] niks3-server[1225]: 2026/09/23 07:02:08 INFO Received uploads request method=POST path=/api/pending_closures7031worker2 # [ 189.063263] nix-grpc-daemon[1644]: time=2026-09-23T07:02:08.248Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7032worker2 # [ 189.064747] nix-grpc-daemon[1644]: time=2026-09-23T07:02:08.250Z level=INFO msg="Uploading wmw8972gq2nf739xhg7is7lylxf8299y-slow-q0 (120B)"7033lb # [ 189.094301] niks3-server[1225]: 2026/09/23 07:02:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7034worker2 # [ 189.106873] nix-grpc-daemon[1644]: time=2026-09-23T07:02:08.292Z level=INFO msg="Uploading 1 narinfos"7035lb # [ 189.099565] niks3-server[1225]: 2026/09/23 07:02:08 INFO Signed narinfos id=68 count=17036lb # [ 189.107953] niks3-server[1225]: 2026/09/23 07:02:08 INFO Registered completed upload object_key=log/8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv7037lb # [ 189.113448] niks3-server[1225]: 2026/09/23 07:02:08 INFO Registered completed upload object_key=nar/0xr809jj32p5n6mnc6i7lqsz48wrm5f9jp14m77k724041711iq7.nar.zst7038lb # [ 189.118563] niks3-server[1225]: 2026/09/23 07:02:08 INFO Registered completed upload object_key=wmw8972gq2nf739xhg7is7lylxf8299y.ls7039lb # [ 189.128245] niks3-server[1225]: 2026/09/23 07:02:08 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7040lb # [ 189.131850] niks3-server[1225]: 2026/09/23 07:02:08 INFO Completed upload id=687041lb # [ 189.135727] niks3-server[1225]: 2026/09/23 07:02:08 INFO Registered completed upload object_key=wmw8972gq2nf739xhg7is7lylxf8299y.narinfo7042worker2 # [ 189.142055] nix-grpc-daemon[1644]: time=2026-09-23T07:02:08.327Z level=INFO msg="Upload complete. (202ms)"7043worker2 # [ 189.148590] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=1 drv=8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv assign_id=9 outputs=17044worker2 # [ 189.163155] nix-daemon[855]: reaped child process 1867, status = succeeded7045worker1 # [ 189.971306] nix-daemon[869]: reaped child process 1931, status = succeeded7046worker1 # [ 190.046712] nix-daemon[869]: accepted connection from pid 1956, user nix-grpc-daemon (trusted)7047worker1 # [ 190.059321] nix-daemon[869]: reaped child process 1959, status = succeeded7048lb # [ 190.063290] niks3-server[1225]: 2026/09/23 07:02:09 INFO Received uploads request method=POST path=/api/pending_closures7049worker1 # [ 190.076833] nix-grpc-daemon[1872]: time=2026-09-23T07:02:08.356Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7050worker1 # [ 190.078521] nix-grpc-daemon[1872]: time=2026-09-23T07:02:08.358Z level=INFO msg="Uploading 106i40yblbkazayvn9ppswrq6kqs26sj-slow-q1 (120B)"7051lb # [ 190.107981] niks3-server[1225]: 2026/09/23 07:02:09 INFO Registered completed upload object_key=106i40yblbkazayvn9ppswrq6kqs26sj.ls7052lb # [ 190.112564] niks3-server[1225]: 2026/09/23 07:02:09 INFO Registered completed upload object_key=nar/0c1c367r3msa4fz6imwp70zxi4fxvrkb86v3hv921wsrf13yd8v3.nar.zst7053lb # [ 190.122768] niks3-server[1225]: 2026/09/23 07:02:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7054lb # [ 190.127729] niks3-server[1225]: 2026/09/23 07:02:09 INFO Signed narinfos id=69 count=17055worker1 # [ 190.132043] nix-grpc-daemon[1872]: time=2026-09-23T07:02:08.411Z level=INFO msg="Uploading 1 narinfos"7056lb # [ 190.134117] niks3-server[1225]: 2026/09/23 07:02:09 INFO Registered completed upload object_key=log/qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv7057lb # [ 190.144159] niks3-server[1225]: 2026/09/23 07:02:09 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7058lb # [ 190.148652] niks3-server[1225]: 2026/09/23 07:02:09 INFO Completed upload id=697059lb # [ 190.150906] niks3-server[1225]: 2026/09/23 07:02:09 INFO Registered completed upload object_key=106i40yblbkazayvn9ppswrq6kqs26sj.narinfo7060worker1 # [ 190.154351] nix-grpc-daemon[1872]: time=2026-09-23T07:02:08.433Z level=INFO msg="Upload complete. (192ms)"7061worker1 # [ 190.160741] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=2 drv=qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv assign_id=10 outputs=17062worker1 # [ 190.175461] nix-daemon[869]: reaped child process 1929, status = succeeded7063client # [ 190.205984] systemd[1]: burst.service: Deactivated successfully.7064client # [ 190.210712] systemd[1]: burst.service: Consumed 162ms CPU time over 6.096s wall clock time, 53.9M memory peak, 14K incoming IP traffic, 18.8K outgoing IP traffic.7065client: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }7066client: (finished: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }, in 0.04 seconds)7067lb: must succeed: curl -sf localhost:9901/clusters7068lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7069lb: must succeed: curl -sf localhost:9901/clusters7070lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7071worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7072worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7073(finished: subtest: queue gauges show a burst while every build blocks, in 7.35 seconds)7074subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does7075client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&17076worker1 # [ 191.462682] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:09Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:513727077worker1 # [ 191.470073] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727078worker2 # [ 191.513787] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:10Z level=debug event=want client=6 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=07079worker2 # [ 191.518267] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:10Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=17080worker2 # [ 191.522705] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:10Z level=debug event=want client=6 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=07081worker2 # [ 191.526662] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:10Z level=debug event=want client=6 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=07082worker1 # [ 191.554720] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727083worker1 # [ 191.567830] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727084worker1 # [ 191.579808] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:513727085worker1 # [ 191.590946] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7086worker1 # [ 191.692819] nix-daemon[869]: accepted connection from pid 1967, user nix-grpc-daemon (trusted)7087worker1 # [ 191.705340] nix-daemon[869]: reaped child process 1970, status = succeeded7088lb # [ 191.710820] niks3-server[1225]: 2026/09/23 07:02:10 INFO Received uploads request method=POST path=/api/pending_closures7089worker1 # [ 191.727407] nix-grpc-daemon[1872]: time=2026-09-23T07:02:10.006Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7090worker1 # [ 191.728881] nix-grpc-daemon[1872]: time=2026-09-23T07:02:10.006Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"7091lb # [ 191.744796] niks3-server[1225]: 2026/09/23 07:02:10 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst7092lb # [ 191.753913] niks3-server[1225]: 2026/09/23 07:02:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7093worker1 # [ 191.765713] nix-grpc-daemon[1872]: time=2026-09-23T07:02:10.045Z level=INFO msg="Uploading 1 narinfos"7094lb # [ 191.761425] niks3-server[1225]: 2026/09/23 07:02:10 INFO Signed narinfos id=70 count=17095lb # [ 191.768367] niks3-server[1225]: 2026/09/23 07:02:10 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls7096lb # [ 191.776396] niks3-server[1225]: 2026/09/23 07:02:10 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7097lb # [ 191.785124] niks3-server[1225]: 2026/09/23 07:02:10 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo7098lb # [ 191.788584] niks3-server[1225]: 2026/09/23 07:02:10 INFO Completed upload id=707099worker1 # [ 191.792995] nix-grpc-daemon[1872]: time=2026-09-23T07:02:10.072Z level=INFO msg="Upload complete. (190ms)"7100worker1 # [ 191.797888] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 paths=1 nar_bytes_in=5047101worker1 # [ 191.803569] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727102worker1 # [ 191.813847] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7103worker1 # [ 191.825746] nix-daemon[869]: reaped child process 1965, status = succeeded7104worker1 # [ 191.829355] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7105worker1 # [ 192.454962] nix-daemon[869]: reaped child process 1974, status = succeeded7106worker1 # [ 192.531451] nix-daemon[869]: accepted connection from pid 1990, user nix-grpc-daemon (trusted)7107worker1 # [ 192.544075] nix-daemon[869]: reaped child process 1993, status = succeeded7108lb # [ 192.548319] niks3-server[1225]: 2026/09/23 07:02:11 INFO Received uploads request method=POST path=/api/pending_closures7109worker1 # [ 192.563365] nix-grpc-daemon[1872]: time=2026-09-23T07:02:10.843Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7110worker1 # [ 192.564970] nix-grpc-daemon[1872]: time=2026-09-23T07:02:10.844Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"7111lb # [ 192.593097] niks3-server[1225]: 2026/09/23 07:02:11 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls7112lb # [ 192.597530] niks3-server[1225]: 2026/09/23 07:02:11 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv7113lb # [ 192.604666] niks3-server[1225]: 2026/09/23 07:02:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7114lb # [ 192.607946] niks3-server[1225]: 2026/09/23 07:02:11 INFO Signed narinfos id=71 count=17115worker1 # [ 192.612162] nix-grpc-daemon[1872]: time=2026-09-23T07:02:10.891Z level=INFO msg="Uploading 1 narinfos"7116lb # [ 192.619592] niks3-server[1225]: 2026/09/23 07:02:11 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst7117lb # [ 192.623270] niks3-server[1225]: 2026/09/23 07:02:11 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7118lb # [ 192.627674] niks3-server[1225]: 2026/09/23 07:02:11 INFO Completed upload id=717119lb # [ 192.629557] niks3-server[1225]: 2026/09/23 07:02:11 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo7120worker1 # [ 192.633398] nix-grpc-daemon[1872]: time=2026-09-23T07:02:10.912Z level=INFO msg="Upload complete. (198ms)"7121worker1 # [ 192.638679] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=11 outputs=17122worker2 # [ 192.643432] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:11Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=17123worker1 # [ 192.646270] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727124worker2 # [ 192.653510] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:11Z level=debug event=want client=6 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=07125worker1 # [ 192.657527] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727126worker1 # [ 192.664718] nix-daemon[869]: reaped child process 1972, status = succeeded7127worker1 # [ 192.666209] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:513727128worker1 # [ 192.670893] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7129worker1 # [ 192.756178] nix-daemon[869]: accepted connection from pid 1997, user nix-grpc-daemon (trusted)7130worker1 # [ 192.768635] nix-daemon[869]: reaped child process 2000, status = succeeded7131lb # [ 192.773445] niks3-server[1225]: 2026/09/23 07:02:11 INFO Received uploads request method=POST path=/api/pending_closures7132worker1 # [ 192.786337] nix-grpc-daemon[1872]: time=2026-09-23T07:02:11.065Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7133worker1 # [ 192.788361] nix-grpc-daemon[1872]: time=2026-09-23T07:02:11.067Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"7134lb # [ 192.807464] niks3-server[1225]: 2026/09/23 07:02:12 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst7135lb # [ 192.815322] niks3-server[1225]: 2026/09/23 07:02:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7136worker1 # [ 192.825431] nix-grpc-daemon[1872]: time=2026-09-23T07:02:11.104Z level=INFO msg="Uploading 1 narinfos"7137lb # [ 192.821428] niks3-server[1225]: 2026/09/23 07:02:12 INFO Signed narinfos id=72 count=17138lb # [ 192.832503] niks3-server[1225]: 2026/09/23 07:02:12 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls7139lb # [ 192.842648] niks3-server[1225]: 2026/09/23 07:02:12 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7140lb # [ 192.845811] niks3-server[1225]: 2026/09/23 07:02:12 INFO Completed upload id=727141worker1 # [ 192.851567] nix-grpc-daemon[1872]: time=2026-09-23T07:02:11.131Z level=INFO msg="Upload complete. (176ms)"7142lb # [ 192.848564] niks3-server[1225]: 2026/09/23 07:02:12 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo7143worker1 # [ 192.855603] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 paths=1 nar_bytes_in=5047144worker1 # [ 192.860777] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727145worker1 # [ 192.870106] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7146worker1 # [ 192.877781] nix-daemon[869]: reaped child process 1995, status = succeeded7147worker1 # [ 192.882490] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7148worker1 # [ 193.898821] nix-daemon[869]: reaped child process 2004, status = succeeded7149worker1 # [ 193.978091] nix-daemon[869]: accepted connection from pid 2021, user nix-grpc-daemon (trusted)7150worker1 # [ 193.989427] nix-daemon[869]: reaped child process 2024, status = succeeded7151lb # [ 193.995559] niks3-server[1225]: 2026/09/23 07:02:13 INFO Received uploads request method=POST path=/api/pending_closures7152worker1 # [ 194.008334] nix-grpc-daemon[1872]: time=2026-09-23T07:02:12.287Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7153worker1 # [ 194.009903] nix-grpc-daemon[1872]: time=2026-09-23T07:02:12.289Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"7154lb # [ 194.039916] niks3-server[1225]: 2026/09/23 07:02:13 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls7155lb # [ 194.042937] niks3-server[1225]: 2026/09/23 07:02:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7156lb # [ 194.050227] niks3-server[1225]: 2026/09/23 07:02:13 INFO Signed narinfos id=73 count=17157worker1 # [ 194.057770] nix-grpc-daemon[1872]: time=2026-09-23T07:02:12.337Z level=INFO msg="Uploading 1 narinfos"7158lb # [ 194.057158] niks3-server[1225]: 2026/09/23 07:02:13 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv7159lb # [ 194.059919] niks3-server[1225]: 2026/09/23 07:02:13 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst7160lb # [ 194.065723] niks3-server[1225]: 2026/09/23 07:02:13 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7161lb # [ 194.068832] niks3-server[1225]: 2026/09/23 07:02:13 INFO Completed upload id=737162worker1 # [ 194.074613] nix-grpc-daemon[1872]: time=2026-09-23T07:02:12.354Z level=INFO msg="Upload complete. (185ms)"7163lb # [ 194.071303] niks3-server[1225]: 2026/09/23 07:02:13 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo7164worker1 # [ 194.079449] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=12 outputs=17165worker1 # [ 194.086611] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727166worker2 # [ 194.087975] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:13Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=17167worker2 # [ 194.093929] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:13Z level=debug event=want client=6 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=07168worker1 # [ 194.097285] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727169worker1 # [ 194.099812] nix-daemon[869]: reaped child process 2002, status = succeeded7170worker1 # [ 194.103462] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:513727171worker1 # [ 194.107401] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7172worker1 # [ 194.192331] nix-daemon[869]: accepted connection from pid 2028, user nix-grpc-daemon (trusted)7173worker1 # [ 194.205786] nix-daemon[869]: reaped child process 2031, status = succeeded7174lb # [ 194.208078] niks3-server[1225]: 2026/09/23 07:02:13 INFO Received uploads request method=POST path=/api/pending_closures7175worker1 # [ 194.222136] nix-grpc-daemon[1872]: time=2026-09-23T07:02:12.501Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7176worker1 # [ 194.223588] nix-grpc-daemon[1872]: time=2026-09-23T07:02:12.503Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"7177lb # [ 194.240139] niks3-server[1225]: 2026/09/23 07:02:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7178worker1 # [ 194.249067] nix-grpc-daemon[1872]: time=2026-09-23T07:02:12.528Z level=INFO msg="Uploading 1 narinfos"7179lb # [ 194.244841] niks3-server[1225]: 2026/09/23 07:02:13 INFO Signed narinfos id=74 count=17180lb # [ 194.254231] niks3-server[1225]: 2026/09/23 07:02:13 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls7181lb # [ 194.257836] niks3-server[1225]: 2026/09/23 07:02:13 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst7182lb # [ 194.266684] niks3-server[1225]: 2026/09/23 07:02:13 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7183lb # [ 194.269463] niks3-server[1225]: 2026/09/23 07:02:13 INFO Completed upload id=747184worker1 # [ 194.273862] nix-grpc-daemon[1872]: time=2026-09-23T07:02:12.553Z level=INFO msg="Upload complete. (160ms)"7185lb # [ 194.271410] niks3-server[1225]: 2026/09/23 07:02:13 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo7186worker1 # [ 194.276615] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 paths=1 nar_bytes_in=5047187worker1 # [ 194.280904] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727188worker1 # [ 194.287924] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7189worker1 # [ 194.290257] nix-daemon[869]: reaped child process 2026, status = succeeded7190worker1 # [ 194.295346] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7191worker1 # [ 195.333750] nix-daemon[869]: reaped child process 2035, status = succeeded7192worker1 # [ 195.412498] nix-daemon[869]: accepted connection from pid 2051, user nix-grpc-daemon (trusted)7193worker1 # [ 195.423403] nix-daemon[869]: reaped child process 2054, status = succeeded7194lb # [ 195.429427] niks3-server[1225]: 2026/09/23 07:02:14 INFO Received uploads request method=POST path=/api/pending_closures7195worker1 # [ 195.442266] nix-grpc-daemon[1872]: time=2026-09-23T07:02:13.721Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7196worker1 # [ 195.445057] nix-grpc-daemon[1872]: time=2026-09-23T07:02:13.723Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"7197lb # [ 195.474607] niks3-server[1225]: 2026/09/23 07:02:14 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst7198lb # [ 195.482808] niks3-server[1225]: 2026/09/23 07:02:14 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls7199lb # [ 195.489502] niks3-server[1225]: 2026/09/23 07:02:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7200lb # [ 195.495521] niks3-server[1225]: 2026/09/23 07:02:14 INFO Signed narinfos id=75 count=17201worker1 # [ 195.503069] nix-grpc-daemon[1872]: time=2026-09-23T07:02:13.782Z level=INFO msg="Uploading 1 narinfos"7202lb # [ 195.509492] niks3-server[1225]: 2026/09/23 07:02:14 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7203lb # [ 195.512507] niks3-server[1225]: 2026/09/23 07:02:14 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv7204lb # [ 195.515112] niks3-server[1225]: 2026/09/23 07:02:14 INFO Completed upload id=757205worker1 # [ 195.519588] nix-grpc-daemon[1872]: time=2026-09-23T07:02:13.799Z level=INFO msg="Upload complete. (217ms)"7206lb # [ 195.518272] niks3-server[1225]: 2026/09/23 07:02:14 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo7207worker1 # [ 195.525674] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=13 outputs=17208worker2 # [ 195.533924] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:14Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=17209worker1 # [ 195.534729] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727210worker2 # [ 195.539641] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:14Z level=debug event=want client=6 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=07211worker1 # [ 195.546523] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727212worker1 # [ 195.552792] nix-daemon[869]: reaped child process 2033, status = succeeded7213worker1 # [ 195.558396] nix-grpc-daemon[1872]: time=2026-09-23T07:02:13.837Z level=INFO msg="All 1 paths already cached"7214worker1 # [ 195.564384] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:513727215worker1 # [ 195.572167] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7216worker1 # [ 195.656459] nix-daemon[869]: accepted connection from pid 2058, user nix-grpc-daemon (trusted)7217worker1 # [ 195.669243] nix-daemon[869]: reaped child process 2061, status = succeeded7218lb # [ 195.672815] niks3-server[1225]: 2026/09/23 07:02:14 INFO Received uploads request method=POST path=/api/pending_closures7219worker1 # [ 195.685309] nix-grpc-daemon[1872]: time=2026-09-23T07:02:13.964Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7220worker1 # [ 195.687357] nix-grpc-daemon[1872]: time=2026-09-23T07:02:13.966Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"7221lb # [ 195.705175] niks3-server[1225]: 2026/09/23 07:02:14 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls7222lb # [ 195.712394] niks3-server[1225]: 2026/09/23 07:02:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign7223lb # [ 195.718232] niks3-server[1225]: 2026/09/23 07:02:14 INFO Signed narinfos id=76 count=17224worker1 # [ 195.725384] nix-grpc-daemon[1872]: time=2026-09-23T07:02:14.004Z level=INFO msg="Uploading 1 narinfos"7225lb # [ 195.732436] niks3-server[1225]: 2026/09/23 07:02:14 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst7226lb # [ 195.739856] niks3-server[1225]: 2026/09/23 07:02:14 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete7227lb # [ 195.745389] niks3-server[1225]: 2026/09/23 07:02:14 INFO Completed upload id=767228lb # [ 195.747656] niks3-server[1225]: 2026/09/23 07:02:14 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo7229worker1 # [ 195.749793] nix-grpc-daemon[1872]: time=2026-09-23T07:02:14.029Z level=INFO msg="Upload complete. (172ms)"7230worker1 # [ 195.753850] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 paths=1 nar_bytes_in=6247231worker1 # [ 195.758577] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727232worker1 # [ 195.768076] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7233worker1 # [ 195.774156] nix-daemon[869]: reaped child process 2056, status = succeeded7234worker1 # [ 195.777931] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7235lb # [ 196.616480] envoy[912]: conn peer=[2001:db8:1::1]:39848 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9604 tx=5186 ms=59307236worker1 # [ 197.210774] nix-daemon[869]: reaped child process 2065, status = succeeded7237worker1 # [ 197.303453] nix-daemon[869]: accepted connection from pid 2081, user nix-grpc-daemon (trusted)7238worker1 # [ 197.314990] nix-daemon[869]: reaped child process 2084, status = succeeded7239lb # [ 197.320470] niks3-server[1225]: 2026/09/23 07:02:16 INFO Received uploads request method=POST path=/api/pending_closures7240worker1 # [ 197.332505] nix-grpc-daemon[1872]: time=2026-09-23T07:02:15.611Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7241worker1 # [ 197.334156] nix-grpc-daemon[1872]: time=2026-09-23T07:02:15.613Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"7242lb # [ 197.360688] niks3-server[1225]: 2026/09/23 07:02:16 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst7243lb # [ 197.368868] niks3-server[1225]: 2026/09/23 07:02:16 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv7244lb # [ 197.372609] niks3-server[1225]: 2026/09/23 07:02:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign7245worker1 # [ 197.384708] nix-grpc-daemon[1872]: time=2026-09-23T07:02:15.664Z level=INFO msg="Uploading 1 narinfos"7246lb # [ 197.380187] niks3-server[1225]: 2026/09/23 07:02:16 INFO Signed narinfos id=77 count=17247lb # [ 197.388132] niks3-server[1225]: 2026/09/23 07:02:16 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls7248lb # [ 197.397203] niks3-server[1225]: 2026/09/23 07:02:16 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete7249lb # [ 197.402983] niks3-server[1225]: 2026/09/23 07:02:16 INFO Completed upload id=777250worker1 # [ 197.407336] nix-grpc-daemon[1872]: time=2026-09-23T07:02:15.686Z level=INFO msg="Upload complete. (220ms)"7251lb # [ 197.406164] niks3-server[1225]: 2026/09/23 07:02:16 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo7252worker1 # [ 197.413820] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=14 outputs=17253worker2 # [ 197.419653] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:16Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=17254worker1 # [ 197.422355] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727255worker2 # [ 197.427771] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:16Z level=debug event=want client=6 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=07256worker1 # [ 197.432893] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727257worker1 # [ 197.441783] nix-daemon[869]: reaped child process 2063, status = succeeded7258worker1 # [ 197.444996] nix-grpc-daemon[1872]: time=2026-09-23T07:02:15.723Z level=INFO msg="All 1 paths already cached"7259worker1 # [ 197.448952] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:513727260worker1 # [ 197.454372] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7261worker1 # [ 197.539558] nix-daemon[869]: accepted connection from pid 2088, user nix-grpc-daemon (trusted)7262worker1 # [ 197.551909] nix-daemon[869]: reaped child process 2091, status = succeeded7263lb # [ 197.556379] niks3-server[1225]: 2026/09/23 07:02:16 INFO Received uploads request method=POST path=/api/pending_closures7264worker1 # [ 197.568803] nix-grpc-daemon[1872]: time=2026-09-23T07:02:15.848Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7265worker1 # [ 197.570775] nix-grpc-daemon[1872]: time=2026-09-23T07:02:15.850Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"7266lb # [ 197.586799] niks3-server[1225]: 2026/09/23 07:02:16 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst7267lb # [ 197.595889] niks3-server[1225]: 2026/09/23 07:02:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign7268lb # [ 197.600493] niks3-server[1225]: 2026/09/23 07:02:16 INFO Signed narinfos id=78 count=17269worker1 # [ 197.607709] nix-grpc-daemon[1872]: time=2026-09-23T07:02:15.887Z level=INFO msg="Uploading 1 narinfos"7270lb # [ 197.612290] niks3-server[1225]: 2026/09/23 07:02:16 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls7271lb # [ 197.622057] niks3-server[1225]: 2026/09/23 07:02:16 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete7272lb # [ 197.628343] niks3-server[1225]: 2026/09/23 07:02:16 INFO Completed upload id=787273worker1 # [ 197.632128] nix-grpc-daemon[1872]: time=2026-09-23T07:02:15.911Z level=INFO msg="Upload complete. (172ms)"7274lb # [ 197.632078] niks3-server[1225]: 2026/09/23 07:02:16 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo7275worker1 # [ 197.635229] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 paths=1 nar_bytes_in=6247276worker1 # [ 197.640503] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727277worker1 # [ 197.651139] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7278worker1 # [ 197.656164] nix-daemon[869]: reaped child process 2086, status = succeeded7279worker1 # [ 197.660850] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7280worker1 # [ 198.691948] nix-daemon[869]: reaped child process 2095, status = succeeded7281worker1 # [ 198.782649] nix-daemon[869]: accepted connection from pid 2111, user nix-grpc-daemon (trusted)7282worker1 # [ 198.793750] nix-daemon[869]: reaped child process 2114, status = succeeded7283lb # [ 198.798887] niks3-server[1225]: 2026/09/23 07:02:18 INFO Received uploads request method=POST path=/api/pending_closures7284worker1 # [ 198.812637] nix-grpc-daemon[1872]: time=2026-09-23T07:02:17.092Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7285worker1 # [ 198.816608] nix-grpc-daemon[1872]: time=2026-09-23T07:02:17.093Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"7286lb # [ 198.848417] niks3-server[1225]: 2026/09/23 07:02:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign7287worker1 # [ 198.858735] nix-grpc-daemon[1872]: time=2026-09-23T07:02:17.138Z level=INFO msg="Uploading 1 narinfos"7288lb # [ 198.854658] niks3-server[1225]: 2026/09/23 07:02:18 INFO Signed narinfos id=79 count=17289lb # [ 198.856699] niks3-server[1225]: 2026/09/23 07:02:18 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv7290lb # [ 198.863319] niks3-server[1225]: 2026/09/23 07:02:18 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls7291lb # [ 198.868910] niks3-server[1225]: 2026/09/23 07:02:18 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst7292lb # [ 198.878492] niks3-server[1225]: 2026/09/23 07:02:18 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete7293lb # [ 198.881437] niks3-server[1225]: 2026/09/23 07:02:18 INFO Completed upload id=797294lb # [ 198.883922] niks3-server[1225]: 2026/09/23 07:02:18 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo7295worker1 # [ 198.887104] nix-grpc-daemon[1872]: time=2026-09-23T07:02:17.166Z level=INFO msg="Upload complete. (216ms)"7296worker2 # [ 198.898253] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:18Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=17297worker1 # [ 198.894262] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=15 outputs=17298worker1 # [ 198.908381] nix-daemon[869]: reaped child process 2093, status = succeeded7299worker1 # [ 198.937765] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727300worker2 # [ 198.944926] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:18Z level=debug event=want client=6 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=07301worker1 # [ 198.950506] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727302worker1 # [ 198.964605] nix-grpc-daemon[1872]: time=2026-09-23T07:02:17.244Z level=INFO msg="All 1 paths already cached"7303worker1 # [ 198.972310] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:513727304worker1 # [ 198.980417] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7305worker1 # [ 199.077869] nix-daemon[869]: accepted connection from pid 2118, user nix-grpc-daemon (trusted)7306worker1 # [ 199.090925] nix-daemon[869]: reaped child process 2121, status = succeeded7307lb # [ 199.093953] niks3-server[1225]: 2026/09/23 07:02:18 INFO Received uploads request method=POST path=/api/pending_closures7308worker1 # [ 199.106691] nix-grpc-daemon[1872]: time=2026-09-23T07:02:17.386Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7309worker1 # [ 199.108272] nix-grpc-daemon[1872]: time=2026-09-23T07:02:17.387Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"7310lb # [ 199.129529] niks3-server[1225]: 2026/09/23 07:02:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign7311lb # [ 199.134346] niks3-server[1225]: 2026/09/23 07:02:18 INFO Signed narinfos id=80 count=17312worker1 # [ 199.141330] nix-grpc-daemon[1872]: time=2026-09-23T07:02:17.420Z level=INFO msg="Uploading 1 narinfos"7313lb # [ 199.140085] niks3-server[1225]: 2026/09/23 07:02:18 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst7314lb # [ 199.148675] niks3-server[1225]: 2026/09/23 07:02:18 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls7315lb # [ 199.160946] niks3-server[1225]: 2026/09/23 07:02:18 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete7316lb # [ 199.168188] niks3-server[1225]: 2026/09/23 07:02:18 INFO Completed upload id=807317worker1 # [ 199.172587] nix-grpc-daemon[1872]: time=2026-09-23T07:02:17.452Z level=INFO msg="Upload complete. (179ms)"7318lb # [ 199.171554] niks3-server[1225]: 2026/09/23 07:02:18 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo7319worker1 # [ 199.177341] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 paths=1 nar_bytes_in=6247320worker1 # [ 199.185256] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727321worker1 # [ 199.194534] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7322worker1 # [ 199.203390] nix-daemon[869]: reaped child process 2116, status = succeeded7323worker1 # [ 199.208489] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7324worker1 # [ 200.223481] nix-daemon[869]: reaped child process 2125, status = succeeded7325worker1 # [ 200.299251] nix-daemon[869]: accepted connection from pid 2141, user nix-grpc-daemon (trusted)7326worker1 # [ 200.312551] nix-daemon[869]: reaped child process 2144, status = succeeded7327lb # [ 200.315738] niks3-server[1225]: 2026/09/23 07:02:19 INFO Received uploads request method=POST path=/api/pending_closures7328worker1 # [ 200.329254] nix-grpc-daemon[1872]: time=2026-09-23T07:02:18.608Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7329worker1 # [ 200.331050] nix-grpc-daemon[1872]: time=2026-09-23T07:02:18.610Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"7330lb # [ 200.359773] niks3-server[1225]: 2026/09/23 07:02:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign7331lb # [ 200.364970] niks3-server[1225]: 2026/09/23 07:02:19 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls7332worker1 # [ 200.374767] nix-grpc-daemon[1872]: time=2026-09-23T07:02:18.654Z level=INFO msg="Uploading 1 narinfos"7333lb # [ 200.370252] niks3-server[1225]: 2026/09/23 07:02:19 INFO Signed narinfos id=81 count=17334lb # [ 200.376832] niks3-server[1225]: 2026/09/23 07:02:19 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst7335lb # [ 200.382215] niks3-server[1225]: 2026/09/23 07:02:19 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv7336lb # [ 200.391715] niks3-server[1225]: 2026/09/23 07:02:19 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete7337lb # [ 200.395097] niks3-server[1225]: 2026/09/23 07:02:19 INFO Completed upload id=817338worker1 # [ 200.399515] nix-grpc-daemon[1872]: time=2026-09-23T07:02:18.679Z level=INFO msg="Upload complete. (184ms)"7339lb # [ 200.398330] niks3-server[1225]: 2026/09/23 07:02:19 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo7340worker1 # [ 200.405801] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=16 outputs=17341worker2 # [ 200.413868] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:19Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=17342worker1 # [ 200.414627] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727343worker2 # [ 200.419552] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:19Z level=debug event=want client=6 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=07344worker1 # [ 200.425541] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727345worker1 # [ 200.433366] nix-daemon[869]: reaped child process 2123, status = succeeded7346worker1 # [ 200.436528] nix-grpc-daemon[1872]: time=2026-09-23T07:02:18.715Z level=INFO msg="All 2 paths already cached"7347worker1 # [ 200.441553] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:513727348worker1 # [ 200.446377] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7349worker1 # [ 200.530494] nix-daemon[869]: accepted connection from pid 2148, user nix-grpc-daemon (trusted)7350worker1 # [ 200.543460] nix-daemon[869]: reaped child process 2151, status = succeeded7351lb # [ 200.547072] niks3-server[1225]: 2026/09/23 07:02:19 INFO Received uploads request method=POST path=/api/pending_closures7352worker1 # [ 200.560031] nix-grpc-daemon[1872]: time=2026-09-23T07:02:18.839Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7353worker1 # [ 200.562064] nix-grpc-daemon[1872]: time=2026-09-23T07:02:18.841Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7354lb # [ 200.579774] niks3-server[1225]: 2026/09/23 07:02:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign7355worker1 # [ 200.593360] nix-grpc-daemon[1872]: time=2026-09-23T07:02:18.872Z level=INFO msg="Uploading 1 narinfos"7356lb # [ 200.588994] niks3-server[1225]: 2026/09/23 07:02:19 INFO Signed narinfos id=82 count=17357lb # [ 200.593787] niks3-server[1225]: 2026/09/23 07:02:19 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7358lb # [ 200.599587] niks3-server[1225]: 2026/09/23 07:02:19 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7359lb # [ 200.613251] niks3-server[1225]: 2026/09/23 07:02:19 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete7360lb # [ 200.618660] niks3-server[1225]: 2026/09/23 07:02:19 INFO Completed upload id=827361worker1 # [ 200.622713] nix-grpc-daemon[1872]: time=2026-09-23T07:02:18.902Z level=INFO msg="Upload complete. (171ms)"7362lb # [ 200.621333] niks3-server[1225]: 2026/09/23 07:02:19 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7363worker1 # [ 200.626624] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 paths=1 nar_bytes_in=7527364worker1 # [ 200.631594] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727365worker1 # [ 200.641109] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7366worker1 # [ 200.648332] nix-daemon[869]: reaped child process 2146, status = succeeded7367worker1 # [ 200.653310] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7368worker1 # [ 201.488630] nix-daemon[869]: reaped child process 2155, status = succeeded7369worker1 # [ 201.564763] nix-daemon[869]: accepted connection from pid 2171, user nix-grpc-daemon (trusted)7370worker1 # [ 201.577589] nix-daemon[869]: reaped child process 2174, status = succeeded7371lb # [ 201.581713] niks3-server[1225]: 2026/09/23 07:02:20 INFO Received uploads request method=POST path=/api/pending_closures7372worker1 # [ 201.595041] nix-grpc-daemon[1872]: time=2026-09-23T07:02:19.874Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7373worker1 # [ 201.598071] nix-grpc-daemon[1872]: time=2026-09-23T07:02:19.876Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7374lb # [ 201.624235] niks3-server[1225]: 2026/09/23 07:02:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign7375worker1 # [ 201.636027] nix-grpc-daemon[1872]: time=2026-09-23T07:02:19.915Z level=INFO msg="Uploading 1 narinfos"7376lb # [ 201.632156] niks3-server[1225]: 2026/09/23 07:02:20 INFO Signed narinfos id=83 count=17377lb # [ 201.641347] niks3-server[1225]: 2026/09/23 07:02:20 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7378lb # [ 201.645632] niks3-server[1225]: 2026/09/23 07:02:20 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7379lb # [ 201.652199] niks3-server[1225]: 2026/09/23 07:02:20 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7380lb # [ 201.659947] niks3-server[1225]: 2026/09/23 07:02:20 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete7381lb # [ 201.663727] niks3-server[1225]: 2026/09/23 07:02:20 INFO Completed upload id=837382worker1 # [ 201.668062] nix-grpc-daemon[1872]: time=2026-09-23T07:02:19.947Z level=INFO msg="Upload complete. (207ms)"7383lb # [ 201.665890] niks3-server[1225]: 2026/09/23 07:02:20 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7384worker2 # [ 201.678819] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:20Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17385worker1 # [ 201.674946] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=17 outputs=17386worker1 # [ 201.689295] nix-daemon[869]: reaped child process 2153, status = succeeded7387worker1 # [ 201.719379] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727388worker1 # [ 201.732768] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727389worker1 # [ 201.746283] nix-grpc-daemon[1872]: time=2026-09-23T07:02:20.025Z level=INFO msg="All 2 paths already cached"7390worker1 # [ 201.754323] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:513727391worker1 # [ 201.764739] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7392worker1 # [ 201.864465] nix-daemon[869]: accepted connection from pid 2178, user nix-grpc-daemon (trusted)7393worker1 # [ 201.879576] nix-daemon[869]: reaped child process 2181, status = succeeded7394lb # [ 201.881558] niks3-server[1225]: 2026/09/23 07:02:21 INFO Received uploads request method=POST path=/api/pending_closures7395worker1 # [ 201.894954] nix-grpc-daemon[1872]: time=2026-09-23T07:02:20.174Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7396worker1 # [ 201.896456] nix-grpc-daemon[1872]: time=2026-09-23T07:02:20.176Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7397lb # [ 201.914322] niks3-server[1225]: 2026/09/23 07:02:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign7398worker1 # [ 201.926120] nix-grpc-daemon[1872]: time=2026-09-23T07:02:20.205Z level=INFO msg="Uploading 1 narinfos"7399lb # [ 201.921807] niks3-server[1225]: 2026/09/23 07:02:21 INFO Signed narinfos id=84 count=17400lb # [ 201.929131] niks3-server[1225]: 2026/09/23 07:02:21 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7401lb # [ 201.941452] niks3-server[1225]: 2026/09/23 07:02:21 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7402lb # [ 201.947627] niks3-server[1225]: 2026/09/23 07:02:21 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete7403lb # [ 201.955131] niks3-server[1225]: 2026/09/23 07:02:21 INFO Completed upload id=847404worker1 # [ 201.959225] nix-grpc-daemon[1872]: time=2026-09-23T07:02:20.238Z level=INFO msg="Upload complete. (181ms)"7405lb # [ 201.957276] niks3-server[1225]: 2026/09/23 07:02:21 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7406worker1 # [ 201.963133] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 paths=1 nar_bytes_in=7527407worker1 # [ 201.969139] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727408worker1 # [ 201.977488] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7409worker1 # [ 201.986457] nix-daemon[869]: reaped child process 2176, status = succeeded7410worker1 # [ 201.991546] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7411worker1 # [ 202.805582] nix-daemon[869]: reaped child process 2185, status = succeeded7412worker1 # [ 202.882262] nix-daemon[869]: accepted connection from pid 2201, user nix-grpc-daemon (trusted)7413worker1 # [ 202.894081] nix-daemon[869]: reaped child process 2204, status = succeeded7414lb # [ 202.898299] niks3-server[1225]: 2026/09/23 07:02:22 INFO Received uploads request method=POST path=/api/pending_closures7415worker1 # [ 202.912257] nix-grpc-daemon[1872]: time=2026-09-23T07:02:21.191Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7416worker1 # [ 202.913961] nix-grpc-daemon[1872]: time=2026-09-23T07:02:21.193Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7417lb # [ 202.943443] niks3-server[1225]: 2026/09/23 07:02:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign7418lb # [ 202.948259] niks3-server[1225]: 2026/09/23 07:02:22 INFO Signed narinfos id=85 count=17419worker1 # [ 202.955507] nix-grpc-daemon[1872]: time=2026-09-23T07:02:21.235Z level=INFO msg="Uploading 1 narinfos"7420lb # [ 202.956408] niks3-server[1225]: 2026/09/23 07:02:22 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7421lb # [ 202.961339] niks3-server[1225]: 2026/09/23 07:02:22 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7422lb # [ 202.971173] niks3-server[1225]: 2026/09/23 07:02:22 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete7423lb # [ 202.977125] niks3-server[1225]: 2026/09/23 07:02:22 INFO Completed upload id=857424worker1 # [ 202.981435] nix-grpc-daemon[1872]: time=2026-09-23T07:02:21.261Z level=INFO msg="Upload complete. (184ms)"7425lb # [ 202.979414] niks3-server[1225]: 2026/09/23 07:02:22 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7426lb # [ 202.981827] niks3-server[1225]: 2026/09/23 07:02:22 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7427worker1 # [ 202.987778] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=18 outputs=17428worker2 # [ 202.993887] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:22Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17429worker1 # [ 202.996424] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727430worker1 # [ 203.007538] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727431worker1 # [ 203.014336] nix-daemon[869]: reaped child process 2183, status = succeeded7432worker1 # [ 203.016603] nix-grpc-daemon[1872]: time=2026-09-23T07:02:21.296Z level=INFO msg="All 2 paths already cached"7433worker1 # [ 203.021100] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:513727434worker1 # [ 203.025214] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7435worker1 # [ 203.110745] nix-daemon[869]: accepted connection from pid 2208, user nix-grpc-daemon (trusted)7436worker1 # [ 203.124566] nix-daemon[869]: reaped child process 2211, status = succeeded7437lb # [ 203.127332] niks3-server[1225]: 2026/09/23 07:02:22 INFO Received uploads request method=POST path=/api/pending_closures7438worker1 # [ 203.141193] nix-grpc-daemon[1872]: time=2026-09-23T07:02:21.420Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7439worker1 # [ 203.143176] nix-grpc-daemon[1872]: time=2026-09-23T07:02:21.422Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7440lb # [ 203.159499] niks3-server[1225]: 2026/09/23 07:02:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/86/sign7441lb # [ 203.167162] niks3-server[1225]: 2026/09/23 07:02:22 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7442lb # [ 203.171953] niks3-server[1225]: 2026/09/23 07:02:22 INFO Signed narinfos id=86 count=17443worker1 # [ 203.179494] nix-grpc-daemon[1872]: time=2026-09-23T07:02:21.459Z level=INFO msg="Uploading 1 narinfos"7444lb # [ 203.185123] niks3-server[1225]: 2026/09/23 07:02:22 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7445lb # [ 203.191696] niks3-server[1225]: 2026/09/23 07:02:22 INFO Received complete upload request method=POST path=/api/pending_closures/86/complete7446lb # [ 203.196327] niks3-server[1225]: 2026/09/23 07:02:22 INFO Completed upload id=867447worker1 # [ 203.202861] nix-grpc-daemon[1872]: time=2026-09-23T07:02:21.482Z level=INFO msg="Upload complete. (172ms)"7448lb # [ 203.201846] niks3-server[1225]: 2026/09/23 07:02:22 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7449worker1 # [ 203.206874] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=0 paths=1 nar_bytes_in=7527450worker1 # [ 203.211514] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:513727451worker1 # [ 203.223117] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7452worker1 # [ 203.227387] nix-daemon[869]: reaped child process 2206, status = succeeded7453worker1 # [ 203.232487] nix-daemon[869]: accepted connection from pid 1807, user nix-grpc-daemon (trusted)7454worker1 # [ 204.248237] nix-daemon[869]: reaped child process 2215, status = succeeded7455worker1 # [ 204.323182] nix-daemon[869]: accepted connection from pid 2231, user nix-grpc-daemon (trusted)7456worker1 # [ 204.337142] nix-daemon[869]: reaped child process 2234, status = succeeded7457lb # [ 204.339736] niks3-server[1225]: 2026/09/23 07:02:23 INFO Received uploads request method=POST path=/api/pending_closures7458worker1 # [ 204.354350] nix-grpc-daemon[1872]: time=2026-09-23T07:02:22.633Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7459worker1 # [ 204.357064] nix-grpc-daemon[1872]: time=2026-09-23T07:02:22.635Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7460lb # [ 204.383814] niks3-server[1225]: 2026/09/23 07:02:23 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7461lb # [ 204.394098] niks3-server[1225]: 2026/09/23 07:02:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/87/sign7462lb # [ 204.403514] niks3-server[1225]: 2026/09/23 07:02:23 INFO Signed narinfos id=87 count=17463worker1 # [ 204.410475] nix-grpc-daemon[1872]: time=2026-09-23T07:02:22.689Z level=INFO msg="Uploading 1 narinfos"7464lb # [ 204.408532] niks3-server[1225]: 2026/09/23 07:02:23 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7465lb # [ 204.413494] niks3-server[1225]: 2026/09/23 07:02:23 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7466lb # [ 204.424900] niks3-server[1225]: 2026/09/23 07:02:23 INFO Received complete upload request method=POST path=/api/pending_closures/87/complete7467lb # [ 204.427449] niks3-server[1225]: 2026/09/23 07:02:23 INFO Completed upload id=877468worker1 # [ 204.431887] nix-grpc-daemon[1872]: time=2026-09-23T07:02:22.711Z level=INFO msg="Upload complete. (192ms)"7469lb # [ 204.431116] niks3-server[1225]: 2026/09/23 07:02:23 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7470worker1 # [ 204.437569] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51372 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=19 outputs=17471worker1 # [ 204.451871] nix-daemon[869]: reaped child process 2213, status = succeeded7472client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 13.17 seconds)7473worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7474worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7475client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]'7476client # [ 204.583570] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7477client # Running as unit: gpu.service; invocation ID: 86b21e4a4da54ddc8236c6f28317fbad7478client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.06 seconds)7479lb: must succeed: curl -sf localhost:9901/clusters7480lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7481lb: must succeed: curl -sf localhost:9901/clusters7482lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7483worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7484worker2 # [ 204.708945] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:441227485client # [ 204.722696] nix[1588]: don't know how to build these paths:7486client # [ 204.723614] nix[1588]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7487worker1 # [ 204.712371] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:513727488worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7489worker2 # [ 204.736415] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:23Z level=debug event=want client=7 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07490client # [ 204.745372] nix[1588]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,kvm,nixos-test,vip}, waiting up to 30s for a worker7491lb: must succeed: curl -sf localhost:9901/clusters7492lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7493lb: must succeed: curl -sf localhost:9901/clusters7494lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7495worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7496worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7497client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7498lb # [ 206.617779] envoy[912]: conn peer=[2001:db8:1::1]:39864 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27107 tx=12278 ms=129957499client # [ 234.746615] nix[1588]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,kvm,nixos-test,vip}7500client # [ 234.756573] nix[1588]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7501client # [ 234.789364] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7502client # [ 234.792666] systemd[1]: gpu.service: Failed with result 'exit-code'.7503client # [ 234.796608] systemd[1]: gpu.service: Consumed 134ms CPU time over 30.206s wall clock time, 37.6M memory peak, 5.6K incoming IP traffic, 5.3K outgoing IP traffic.7504client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.39 seconds)7505client: must succeed: journalctl -u gpu -o cat7506client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)7507lb: must succeed: curl -sf localhost:9901/clusters7508lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7509lb: must succeed: curl -sf localhost:9901/clusters7510lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7511worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7512worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7513(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 44.06 seconds)7514subtest: low disk drains a worker and builds go to the other7515worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7516worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.96 seconds)7517worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7518lb # [ 236.620231] envoy[912]: conn peer=[2001:db8:1::1]:60360 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=929 tx=596 ms=300487519worker1 # [ 236.854113] nix-grpc-daemon[1807]: ts=2026-09-23T07:02:55Z level=info event=unhealthy reason=min_free7520worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.09 seconds)7521client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&27522client # don't know how to build these paths:7523client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7524worker2 # [ 237.578634] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:56Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:441227525worker2 # [ 237.584380] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:441227526worker2 # [ 237.605672] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:56Z level=debug event=want client=8 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07527worker2 # [ 237.611137] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:56Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17528worker2 # [ 237.650453] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441227529worker2 # [ 237.663239] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:441227530client # copying 1 paths to 'lb:50051'...7531client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7532worker2 # [ 237.676079] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:441227533worker2 # [ 237.687344] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)7534worker2 # [ 237.797086] nix-daemon[855]: accepted connection from pid 1939, user nix-grpc-daemon (trusted)7535worker2 # [ 237.811929] nix-daemon[855]: reaped child process 1942, status = succeeded7536lb # [ 237.810748] niks3-server[1225]: 2026/09/23 07:02:57 INFO Received uploads request method=POST path=/api/pending_closures7537worker2 # [ 237.829664] nix-grpc-daemon[1644]: time=2026-09-23T07:02:57.015Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7538worker2 # [ 237.831649] nix-grpc-daemon[1644]: time=2026-09-23T07:02:57.016Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7539lb # [ 237.850630] niks3-server[1225]: 2026/09/23 07:02:57 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7540lb # [ 237.859969] niks3-server[1225]: 2026/09/23 07:02:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/88/sign7541worker2 # [ 237.874513] nix-grpc-daemon[1644]: time=2026-09-23T07:02:57.060Z level=INFO msg="Uploading 1 narinfos"7542lb # [ 237.866621] niks3-server[1225]: 2026/09/23 07:02:57 INFO Signed narinfos id=88 count=17543lb # [ 237.878539] niks3-server[1225]: 2026/09/23 07:02:57 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7544lb # [ 237.888573] niks3-server[1225]: 2026/09/23 07:02:57 INFO Received complete upload request method=POST path=/api/pending_closures/88/complete7545lb # [ 237.891564] niks3-server[1225]: 2026/09/23 07:02:57 INFO Completed upload id=887546worker2 # [ 237.899618] nix-grpc-daemon[1644]: time=2026-09-23T07:02:57.085Z level=INFO msg="Upload complete. (198ms)"7547worker2 # [ 237.904144] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=0 paths=1 nar_bytes_in=5127548lb # [ 237.901112] niks3-server[1225]: 2026/09/23 07:02:57 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7549worker2 # [ 237.910910] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441227550worker2 # [ 237.925157] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)7551worker2 # [ 237.930208] nix-daemon[855]: reaped child process 1937, status = succeeded7552worker2 # [ 237.939500] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)7553client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7554client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7555client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7556client # farm-a> LOG-farm-a-drain7557worker2 # [ 239.177418] nix-daemon[855]: reaped child process 1946, status = succeeded7558worker2 # [ 239.259325] nix-daemon[855]: accepted connection from pid 1962, user nix-grpc-daemon (trusted)7559worker2 # [ 239.271041] nix-daemon[855]: reaped child process 1965, status = succeeded7560lb # [ 239.270388] niks3-server[1225]: 2026/09/23 07:02:58 INFO Received uploads request method=POST path=/api/pending_closures7561worker2 # [ 239.289273] nix-grpc-daemon[1644]: time=2026-09-23T07:02:58.474Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7562worker2 # [ 239.290758] nix-grpc-daemon[1644]: time=2026-09-23T07:02:58.476Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7563lb # [ 239.313180] niks3-server[1225]: 2026/09/23 07:02:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/89/sign7564worker2 # [ 239.325090] nix-grpc-daemon[1644]: time=2026-09-23T07:02:58.510Z level=INFO msg="Uploading 1 narinfos"7565lb # [ 239.317556] niks3-server[1225]: 2026/09/23 07:02:58 INFO Signed narinfos id=89 count=17566lb # [ 239.328159] niks3-server[1225]: 2026/09/23 07:02:58 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7567lb # [ 239.331609] niks3-server[1225]: 2026/09/23 07:02:58 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7568lb # [ 239.336540] niks3-server[1225]: 2026/09/23 07:02:58 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7569lb # [ 239.340394] niks3-server[1225]: 2026/09/23 07:02:58 INFO Received complete upload request method=POST path=/api/pending_closures/89/complete7570lb # [ 239.342758] niks3-server[1225]: 2026/09/23 07:02:58 INFO Completed upload id=897571worker2 # [ 239.352212] nix-grpc-daemon[1644]: time=2026-09-23T07:02:58.537Z level=INFO msg="Upload complete. (205ms)"7572lb # [ 239.346464] niks3-server[1225]: 2026/09/23 07:02:58 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7573worker2 # [ 239.358666] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=20 outputs=17574worker2 # [ 239.369513] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:58Z level=debug event=want client=8 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07575worker2 # [ 239.373590] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:58Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17576worker2 # [ 239.383228] nix-daemon[855]: reaped child process 1944, status = succeeded7577worker2 # [ 239.413288] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441227578worker2 # [ 239.425704] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:441227579worker2 # [ 239.438750] nix-grpc-daemon[1644]: time=2026-09-23T07:02:58.624Z level=INFO msg="All 1 paths already cached"7580client # copying 1 paths to 'lb:50051'...7581client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7582worker2 # [ 239.447231] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:441227583worker2 # [ 239.455400] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)7584worker2 # [ 239.549572] nix-daemon[855]: accepted connection from pid 1971, user nix-grpc-daemon (trusted)7585worker2 # [ 239.562585] nix-daemon[855]: reaped child process 1974, status = succeeded7586lb # [ 239.561362] niks3-server[1225]: 2026/09/23 07:02:58 INFO Received uploads request method=POST path=/api/pending_closures7587worker2 # [ 239.578379] nix-grpc-daemon[1644]: time=2026-09-23T07:02:58.764Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7588worker2 # [ 239.579944] nix-grpc-daemon[1644]: time=2026-09-23T07:02:58.765Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7589lb # [ 239.596266] niks3-server[1225]: 2026/09/23 07:02:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/90/sign7590worker2 # [ 239.608240] nix-grpc-daemon[1644]: time=2026-09-23T07:02:58.793Z level=INFO msg="Uploading 1 narinfos"7591lb # [ 239.600436] niks3-server[1225]: 2026/09/23 07:02:58 INFO Signed narinfos id=90 count=17592lb # [ 239.607952] niks3-server[1225]: 2026/09/23 07:02:58 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7593lb # [ 239.612733] niks3-server[1225]: 2026/09/23 07:02:58 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7594lb # [ 239.622879] niks3-server[1225]: 2026/09/23 07:02:58 INFO Received complete upload request method=POST path=/api/pending_closures/90/complete7595lb # [ 239.627279] niks3-server[1225]: 2026/09/23 07:02:58 INFO Completed upload id=907596worker2 # [ 239.635215] nix-grpc-daemon[1644]: time=2026-09-23T07:02:58.820Z level=INFO msg="Upload complete. (167ms)"7597lb # [ 239.629561] niks3-server[1225]: 2026/09/23 07:02:58 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7598worker2 # [ 239.639509] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=0 paths=1 nar_bytes_in=6247599worker2 # [ 239.645128] nix-grpc-daemon[1567]: ts=2026-09-23T07:02:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441227600worker2 # [ 239.653746] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)7601worker2 # [ 239.659167] nix-daemon[855]: reaped child process 1969, status = succeeded7602worker2 # [ 239.663884] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)7603client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7604client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7605client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7606client # farm-b> LOG-farm-b-drain7607worker2 # [ 241.079601] nix-daemon[855]: reaped child process 1978, status = succeeded7608worker2 # [ 241.153388] nix-daemon[855]: accepted connection from pid 1994, user nix-grpc-daemon (trusted)7609worker2 # [ 241.165432] nix-daemon[855]: reaped child process 1997, status = succeeded7610lb # [ 241.166746] niks3-server[1225]: 2026/09/23 07:03:00 INFO Received uploads request method=POST path=/api/pending_closures7611worker2 # [ 241.183620] nix-grpc-daemon[1644]: time=2026-09-23T07:03:00.369Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7612worker2 # [ 241.185388] nix-grpc-daemon[1644]: time=2026-09-23T07:03:00.370Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7613lb # [ 241.211772] niks3-server[1225]: 2026/09/23 07:03:00 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7614lb # [ 241.215595] niks3-server[1225]: 2026/09/23 07:03:00 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7615lb # [ 241.222844] niks3-server[1225]: 2026/09/23 07:03:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/91/sign7616worker2 # [ 241.236596] nix-grpc-daemon[1644]: time=2026-09-23T07:03:00.422Z level=INFO msg="Uploading 1 narinfos"7617lb # [ 241.229109] niks3-server[1225]: 2026/09/23 07:03:00 INFO Signed narinfos id=91 count=17618lb # [ 241.232174] niks3-server[1225]: 2026/09/23 07:03:00 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7619lb # [ 241.243060] niks3-server[1225]: 2026/09/23 07:03:00 INFO Received complete upload request method=POST path=/api/pending_closures/91/complete7620lb # [ 241.247386] niks3-server[1225]: 2026/09/23 07:03:00 INFO Completed upload id=917621worker2 # [ 241.255301] nix-grpc-daemon[1644]: time=2026-09-23T07:03:00.440Z level=INFO msg="Upload complete. (183ms)"7622lb # [ 241.249753] niks3-server[1225]: 2026/09/23 07:03:00 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7623worker2 # [ 241.261935] nix-grpc-daemon[1567]: ts=2026-09-23T07:03:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=21 outputs=17624worker2 # [ 241.273857] nix-grpc-daemon[1567]: ts=2026-09-23T07:03:00Z level=debug event=want client=8 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07625worker2 # [ 241.278456] nix-grpc-daemon[1567]: ts=2026-09-23T07:03:00Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17626worker2 # [ 241.282252] nix-grpc-daemon[1567]: ts=2026-09-23T07:03:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441227627worker2 # [ 241.291048] nix-daemon[855]: reaped child process 1976, status = succeeded7628worker2 # [ 241.292689] nix-grpc-daemon[1567]: ts=2026-09-23T07:03:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:441227629worker2 # [ 241.299492] nix-grpc-daemon[1644]: time=2026-09-23T07:03:00.485Z level=INFO msg="All 2 paths already cached"7630client # copying 1 paths to 'lb:50051'...7631worker2 # [ 241.305284] nix-grpc-daemon[1567]: ts=2026-09-23T07:03:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:441227632client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7633worker2 # [ 241.310091] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)7634worker2 # [ 241.394360] nix-daemon[855]: accepted connection from pid 2003, user nix-grpc-daemon (trusted)7635worker2 # [ 241.407097] nix-daemon[855]: reaped child process 2006, status = succeeded7636lb # [ 241.406723] niks3-server[1225]: 2026/09/23 07:03:00 INFO Received uploads request method=POST path=/api/pending_closures7637worker2 # [ 241.425096] nix-grpc-daemon[1644]: time=2026-09-23T07:03:00.610Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7638worker2 # [ 241.427059] nix-grpc-daemon[1644]: time=2026-09-23T07:03:00.612Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7639lb # [ 241.438716] niks3-server[1225]: 2026/09/23 07:03:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/92/sign7640worker2 # [ 241.455380] nix-grpc-daemon[1644]: time=2026-09-23T07:03:00.640Z level=INFO msg="Uploading 1 narinfos"7641lb # [ 241.446973] niks3-server[1225]: 2026/09/23 07:03:00 INFO Signed narinfos id=92 count=17642lb # [ 241.454549] niks3-server[1225]: 2026/09/23 07:03:00 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7643lb # [ 241.461312] niks3-server[1225]: 2026/09/23 07:03:00 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7644lb # [ 241.471885] niks3-server[1225]: 2026/09/23 07:03:00 INFO Received complete upload request method=POST path=/api/pending_closures/92/complete7645lb # [ 241.478481] niks3-server[1225]: 2026/09/23 07:03:00 INFO Completed upload id=927646lb # [ 241.480503] niks3-server[1225]: 2026/09/23 07:03:00 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7647worker2 # [ 241.485969] nix-grpc-daemon[1644]: time=2026-09-23T07:03:00.671Z level=INFO msg="Upload complete. (170ms)"7648worker2 # [ 241.490322] nix-grpc-daemon[1567]: ts=2026-09-23T07:03:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=0 paths=1 nar_bytes_in=7527649worker2 # [ 241.495963] nix-grpc-daemon[1567]: ts=2026-09-23T07:03:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:441227650worker2 # [ 241.505902] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)7651worker2 # [ 241.511428] nix-daemon[855]: reaped child process 2001, status = succeeded7652worker2 # [ 241.516409] nix-daemon[855]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)7653client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7654client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7655client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7656client # farm-top> LOG-farm-top-drain7657worker2 # [ 242.354809] nix-daemon[855]: reaped child process 2010, status = succeeded7658worker2 # [ 242.430411] nix-daemon[855]: accepted connection from pid 2026, user nix-grpc-daemon (trusted)7659worker2 # [ 242.442403] nix-daemon[855]: reaped child process 2029, status = succeeded7660lb # [ 242.441919] niks3-server[1225]: 2026/09/23 07:03:01 INFO Received uploads request method=POST path=/api/pending_closures7661worker2 # [ 242.459654] nix-grpc-daemon[1644]: time=2026-09-23T07:03:01.645Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7662worker2 # [ 242.461198] nix-grpc-daemon[1644]: time=2026-09-23T07:03:01.646Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7663lb # [ 242.486234] niks3-server[1225]: 2026/09/23 07:03:01 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7664lb # [ 242.489844] niks3-server[1225]: 2026/09/23 07:03:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/93/sign7665worker2 # [ 242.505964] nix-grpc-daemon[1644]: time=2026-09-23T07:03:01.691Z level=INFO msg="Uploading 1 narinfos"7666lb # [ 242.498447] niks3-server[1225]: 2026/09/23 07:03:01 INFO Signed narinfos id=93 count=17667lb # [ 242.504738] niks3-server[1225]: 2026/09/23 07:03:01 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7668lb # [ 242.510719] niks3-server[1225]: 2026/09/23 07:03:01 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7669lb # [ 242.519822] niks3-server[1225]: 2026/09/23 07:03:01 INFO Received complete upload request method=POST path=/api/pending_closures/93/complete7670lb # [ 242.523267] niks3-server[1225]: 2026/09/23 07:03:01 INFO Completed upload id=937671lb # [ 242.525209] niks3-server[1225]: 2026/09/23 07:03:01 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7672worker2 # [ 242.532520] nix-grpc-daemon[1644]: time=2026-09-23T07:03:01.718Z level=INFO msg="Upload complete. (209ms)"7673worker2 # [ 242.539372] nix-grpc-daemon[1567]: ts=2026-09-23T07:03:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44122 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=22 outputs=17674worker2 # [ 242.554377] nix-daemon[855]: reaped child process 2008, status = succeeded7675client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&2, in 5.15 seconds)7676client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath7677client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath, in 0.12 seconds)7678worker1: must succeed: rm /nix/.rw-store/fill7679worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.12 seconds)7680worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7681worker1 # [ 243.856509] nix-grpc-daemon[1807]: ts=2026-09-23T07:03:02Z level=info event=healthy reason=min_free7682worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.09 seconds)7683(finished: subtest: low disk drains a worker and builds go to the other, in 8.57 seconds)7684(finished: run the VM test script, in 244.85 seconds)7685test script finished in 245.04s7686cleanup7687kill QemuMachine (pid 49)7688client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7689client # [2026-09-23T07:03:03Z INFO virtiofsd] Client disconnected, shutting down7690client # [2026-09-23T07:03:03Z INFO virtiofsd] Client disconnected, shutting down7691client # [2026-09-23T07:03:03Z INFO virtiofsd] Client disconnected, shutting down7692kill QemuMachine (pid 50)7693lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7694lb # [2026-09-23T07:03:03Z INFO virtiofsd] Client disconnected, shutting down7695lb # [2026-09-23T07:03:03Z INFO virtiofsd] Client disconnected, shutting down7696lb # [2026-09-23T07:03:03Z INFO virtiofsd] Client disconnected, shutting down7697kill QemuMachine (pid 51)7698worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7699worker1 # [2026-09-23T07:03:03Z INFO virtiofsd] Client disconnected, shutting down7700worker1 # [2026-09-23T07:03:03Z INFO virtiofsd] Client disconnected, shutting down7701worker1 # [2026-09-23T07:03:03Z INFO virtiofsd] Client disconnected, shutting down7702kill QemuMachine (pid 52)7703worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7704worker2 # [2026-09-23T07:03:04Z INFO virtiofsd] Client disconnected, shutting down7705worker2 # [2026-09-23T07:03:04Z INFO virtiofsd] Client disconnected, shutting down7706worker2 # [2026-09-23T07:03:04Z INFO virtiofsd] Client disconnected, shutting down7707(finished: cleanup, in 1.07 seconds)7708additionally exposed symbols:7709 client, lb, worker1, worker2,7710 vlan1,7711 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh