vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #504
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12lb # Disk image does not exist, creating the virtualisation disk image...13worker1 # Disk image does not exist, creating the virtualisation disk image...14lb # Formatting '/build/vm-state-lb/tmp.VdtB9J5e93', fmt=raw size=107374182415worker1: QEMU running (pid 51)16lb: QEMU running (pid 50)17worker1 # Formatting '/build/vm-state-worker1/tmp.lCCwydoBYO', fmt=raw size=107374182418lb # mke2fs 1.47.4 (6-Mar-2025)19worker2 # Disk image does not exist, creating the virtualisation disk image...20worker2: QEMU running (pid 52)21worker1 # mke2fs 1.47.4 (6-Mar-2025)22lb # Discarding device blocks: 0/262144 done23worker1 # Discarding device blocks: 0/262144 done24lb # Creating filesystem with 262144 4k blocks and 65536 inodes25worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes26lb # Filesystem UUID: 22e726a0-e7f4-4688-80db-8bb40e56334427worker1 # Filesystem UUID: 1ca88e7c-42bb-4927-a3da-39301db55bc228lb # Superblock backups stored on blocks:29worker1 # Superblock backups stored on blocks:30lb # 32768, 98304, 163840, 22937631worker1 # 32768, 98304, 163840, 22937632lb # 33worker1 # 34lb # Allocating group tables: 0/8 done35worker1 # Allocating group tables: 0/8 done36lb # Writing inode tables: 0/8 done37worker1 # Writing inode tables: 0/8 done38lb # Creating journal (8192 blocks): done39worker1 # Creating journal (8192 blocks): done40lb # Writing superblocks and filesystem accounting information: 0/8 done41worker1 # Writing superblocks and filesystem accounting information: 0/8 done42lb # 43worker1 # 44lb # Virtualisation disk image created.45client: QEMU running (pid 49)46worker1 # Virtualisation disk image created.47lb # Starting virtiofs daemons...48worker2 # Formatting '/build/vm-state-worker2/tmp.ZMT1BFxo7B', fmt=raw size=107374182449client # Disk image does not exist, creating the virtualisation disk image...50worker1 # Starting virtiofs daemons...51(finished: start all VMs, in 0.19 seconds)52client # Formatting '/build/vm-state-client/tmp.9IvFQTVXK0', fmt=raw size=107374182453worker1 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)54worker1 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether55client # mke2fs 1.47.4 (6-Mar-2025)56client: waiting for unit network-addresses-eth1.service57worker1 # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...58client # Discarding device blocks: 0/262144 done59client: waiting for the VM to finish booting60worker1 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)61client # Creating filesystem with 262144 4k blocks and 65536 inodes62worker2 # mke2fs 1.47.4 (6-Mar-2025)63lb # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)64worker1 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether65client # Filesystem UUID: 16b2f4da-c94c-4ce2-8baa-ad719692729766worker2 # Discarding device blocks: 0/262144 done67lb # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether68worker1 # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...69client # Superblock backups stored on blocks:70worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes71lb # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...72worker1 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)73client # 32768, 98304, 163840, 22937674worker2 # Filesystem UUID: fa10f76d-de7f-4c39-bee2-95b4ab0b213375lb # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)76worker1 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether77client # 78worker2 # Superblock backups stored on blocks:79lb # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether80worker1 # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...81client # Allocating group tables: 0/8 done82worker2 # 32768, 98304, 163840, 22937683lb # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...84worker1 # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests85client # Writing inode tables: 0/8 done86worker2 # 87lb # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)88worker1 # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests89client # Creating journal (8192 blocks): done90worker2 # Allocating group tables: 0/8 done91lb # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether92worker1 # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests93client # Writing superblocks and filesystem accounting information: 0/8 done94worker2 # Writing inode tables: 0/8 done95client # 96worker2 # Creating journal (8192 blocks): done97client # Virtualisation disk image created.98worker2 # Writing superblocks and filesystem accounting information: 0/8 done99client # Starting virtiofs daemons...100worker2 # 101client # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)102worker2 # Virtualisation disk image created.103client # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether104worker2 # Starting virtiofs daemons...105client # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...106worker2 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)107client # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)108worker2 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether109client # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether110worker2 # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...111client # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...112worker2 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)113client # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)114worker2 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)115client # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether116worker2 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether117client # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...118worker2 # [2026-09-22T16:04:09Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether119client # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests120worker2 # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...121client # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests122worker2 # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...123client # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests124worker2 # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests125lb # [2026-09-22T16:04:09Z INFO virtiofsd] Waiting for vhost-user socket connection...126worker2 # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests127lb # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests128worker2 # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests129lb # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests130lb # [2026-09-22T16:04:09Z INFO virtiofsd] Client connected, servicing requests131lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)132worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)133worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)134client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)135lb # 136lb # 137lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00138lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...139lb # 140lb # 141lb # 142lb # 143lb # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00144lb # Press Ctrl-B to configure iPXE (PCI 00:05.0)...145lb # 146lb # 147worker1 # 148worker1 # 149worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00150worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...151worker1 # 152worker1 # 153worker1 # 154worker1 # 155worker1 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00156worker2 # 157worker2 # 158worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00159worker1 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...160worker1 # 161worker1 # 162worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...163worker2 # 164worker2 # 165worker2 # 166worker2 # 167client # 168client # 169worker2 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00170client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00171client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...172client # 173client # 174worker2 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...175client # 176worker2 # 177client # 178worker2 # 179client # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00180lb # Booting from ROM...181client # Press Ctrl-B to configure iPXE (PCI 00:05.0)...182client # 183client # 184worker1 # Booting from ROM...185worker2 # Booting from ROM...186client # Booting from ROM...187lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026188lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/nry04znx19amfwpgmmym379f7hcf2spd-nixos-system-lb-test/init regInfo=/nix/store/ax20dpiqqpa2zqr4ahb7bspnj25x5j22-closure-info/registration console=ttyS0,115200n8 console=tty0189lb # [ 0.000000] BIOS-provided physical RAM map:190lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable191lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved192lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved193lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable194lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved195lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved196lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved197lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved198lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved199lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved200lb # [ 0.000000] NX (Execute Disable) protection: active201lb # [ 0.000000] APIC: Static calls initialized202lb # [ 0.000000] SMBIOS 2.8 present.203lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014204lb # [ 0.000000] DMI: Memory slots populated: 1/1205lb # [ 0.000000] Hypervisor detected: KVM206lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000207worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026208lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00209lb # [ 0.000000] kvm-clock: using sched offset of 469913780 cycles210lb # [ 0.000007] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns211lb # [ 0.000010] tsc: Detected 2400.012 MHz processor212lb # [ 0.000809] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000213worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/7hj4sk4i9w6mp1h0l38acx95n0smfk2s-nixos-system-worker1-test/init regInfo=/nix/store/1ra8h8rvaqsaasm1v4s2g2fsrrsb0g1s-closure-info/registration console=ttyS0,115200n8 console=tty0214worker1 # [ 0.000000] BIOS-provided physical RAM map:215lb # [ 0.000836] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs216worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable217lb # [ 0.000838] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT218lb # [ 0.002721] found SMP MP-table at [mem 0x000f5450-0x000f545f]219worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved220lb # [ 0.002731] Using GB pages for direct mapping221worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved222lb # [ 0.002891] RAMDISK: [mem 0x5e36e000-0x5ffcffff]223lb # [ 0.002898] ACPI: Early table checksum verification disabled224worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable225lb # [ 0.002901] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )226worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved227lb # [ 0.002904] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)228worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved229worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved230lb # [ 0.002908] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)231worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved232lb # [ 0.002915] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)233lb # [ 0.002917] ACPI: FACS 0x000000005FFE0000 000040234worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved235worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved236lb # [ 0.002919] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)237worker1 # [ 0.000000] NX (Execute Disable) protection: active238lb # [ 0.002920] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)239worker1 # [ 0.000000] APIC: Static calls initialized240worker1 # [ 0.000000] SMBIOS 2.8 present.241lb # [ 0.002922] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)242worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014243lb # [ 0.002924] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)244worker1 # [ 0.000000] DMI: Memory slots populated: 1/1245worker1 # [ 0.000000] Hypervisor detected: KVM246lb # [ 0.002925] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]247worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000248lb # [ 0.002926] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]249worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00250lb # [ 0.002926] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]251worker1 # [ 0.000000] kvm-clock: using sched offset of 465155104 cycles252lb # [ 0.002927] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]253client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026254worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns255lb # [ 0.002927] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]256worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor257lb # [ 0.002928] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]258worker1 # [ 0.000809] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000259lb # [ 0.002929] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]260worker1 # [ 0.000835] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs261lb # [ 0.003156] No NUMA configuration found262worker1 # [ 0.000838] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT263lb # [ 0.003158] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]264client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/nwrxkcyq2rabn5jfqyww8chp4fbw12l7-nixos-system-client-test/init regInfo=/nix/store/9mbchw87wm67kspwy4r01ypax3r3mych-closure-info/registration console=ttyS0,115200n8 console=tty0265client # [ 0.000000] BIOS-provided physical RAM map:266worker1 # [ 0.002730] found SMP MP-table at [mem 0x000f5450-0x000f545f]267worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026268lb # [ 0.003160] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]269client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable270lb # [ 0.005477] Zone ranges:271worker1 # [ 0.002741] Using GB pages for direct mapping272lb # [ 0.005477] DMA [mem 0x0000000000001000-0x0000000000ffffff]273worker1 # [ 0.002814] RAMDISK: [mem 0x5e370000-0x5ffcffff]274client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved275lb # [ 0.005479] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]276worker1 # [ 0.002822] ACPI: Early table checksum verification disabled277lb # [ 0.005480] Normal empty278client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved279worker1 # [ 0.002825] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )280lb # [ 0.005481] Device empty281lb # [ 0.005481] Movable zone start for each node282client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable283worker1 # [ 0.002829] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)284lb # [ 0.005482] Early memory node ranges285worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/6bw4n0f8bls8vkb4018phy0as0l93xnb-nixos-system-worker2-test/init regInfo=/nix/store/mnhq4clcym8k7q31bph6wbcnxjd6nnha-closure-info/registration console=ttyS0,115200n8 console=tty0286client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved287lb # [ 0.005482] node 0: [mem 0x0000000000001000-0x000000000009efff]288worker2 # [ 0.000000] BIOS-provided physical RAM map:289worker1 # [ 0.002833] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)290client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved291lb # [ 0.005483] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]292worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable293worker1 # [ 0.002840] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)294client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved295lb # [ 0.005484] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]296worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved297worker1 # [ 0.002842] ACPI: FACS 0x000000005FFE0000 000040298lb # [ 0.005507] On node 0, zone DMA: 1 pages in unavailable ranges299client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved300worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved301worker1 # [ 0.002843] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)302lb # [ 0.005769] On node 0, zone DMA: 97 pages in unavailable ranges303client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved304worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable305lb # [ 0.032428] On node 0, zone DMA32: 40 pages in unavailable ranges306worker1 # [ 0.002845] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)307client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved308lb # [ 0.032878] ACPI: PM-Timer IO Port: 0x608309worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved310client # [ 0.000000] NX (Execute Disable) protection: active311lb # [ 0.032889] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])312worker1 # [ 0.002846] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)313client # [ 0.000000] APIC: Static calls initialized314worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved315client # [ 0.000000] SMBIOS 2.8 present.316lb # [ 0.032915] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23317worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved318worker1 # [ 0.002848] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)319lb # [ 0.032917] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)320client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014321worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved322worker1 # [ 0.002849] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]323lb # [ 0.032919] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)324client # [ 0.000000] DMI: Memory slots populated: 1/1325client # [ 0.000000] Hypervisor detected: KVM326worker1 # [ 0.002850] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]327worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved328lb # [ 0.032920] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)329client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000330worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved331worker1 # [ 0.002851] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]332lb # [ 0.032921] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)333client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00334worker2 # [ 0.000000] NX (Execute Disable) protection: active335worker1 # [ 0.002851] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]336worker2 # [ 0.000000] APIC: Static calls initialized337lb # [ 0.032922] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)338client # [ 0.000000] kvm-clock: using sched offset of 506173500 cycles339worker2 # [ 0.000000] SMBIOS 2.8 present.340worker1 # [ 0.002852] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]341lb # [ 0.032924] ACPI: Using ACPI (MADT) for SMP configuration information342client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns343lb # [ 0.032925] ACPI: HPET id: 0x8086a201 base: 0xfed00000344worker1 # [ 0.002852] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]345worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014346client # [ 0.000005] tsc: Detected 2400.012 MHz processor347lb # [ 0.032928] TSC deadline timer available348worker2 # [ 0.000000] DMI: Memory slots populated: 1/1349lb # [ 0.032932] CPU topo: Max. logical packages: 1350worker1 # [ 0.002853] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]351worker2 # [ 0.000000] Hypervisor detected: KVM352client # [ 0.000803] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000353worker1 # [ 0.003075] No NUMA configuration found354lb # [ 0.032933] CPU topo: Max. logical dies: 1355worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000356lb # [ 0.032933] CPU topo: Max. dies per package: 1357client # [ 0.000835] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs358worker1 # [ 0.003076] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]359worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00360lb # [ 0.032937] CPU topo: Max. threads per core: 1361client # [ 0.000837] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT362worker1 # [ 0.003079] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]363worker2 # [ 0.000000] kvm-clock: using sched offset of 506920496 cycles364lb # [ 0.032937] CPU topo: Num. cores per package: 1365worker1 # [ 0.005423] Zone ranges:366client # [ 0.002716] found SMP MP-table at [mem 0x000f5450-0x000f545f]367lb # [ 0.032938] CPU topo: Num. threads per package: 1368worker1 # [ 0.005423] DMA [mem 0x0000000000001000-0x0000000000ffffff]369client # [ 0.002727] Using GB pages for direct mapping370worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns371lb # [ 0.032938] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs372worker1 # [ 0.005425] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]373client # [ 0.002807] RAMDISK: [mem 0x5e370000-0x5ffcffff]374worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor375worker1 # [ 0.005426] Normal empty376lb # [ 0.032954] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()377client # [ 0.002814] ACPI: Early table checksum verification disabled378worker2 # [ 0.000805] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000379worker1 # [ 0.005427] Device empty380worker1 # [ 0.005427] Movable zone start for each node381client # [ 0.002817] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )382lb # [ 0.032982] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]383worker1 # [ 0.005428] Early memory node ranges384worker2 # [ 0.000832] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs385client # [ 0.002821] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)386lb # [ 0.032984] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]387worker1 # [ 0.005428] node 0: [mem 0x0000000000001000-0x000000000009efff]388worker2 # [ 0.000834] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT389worker1 # [ 0.005429] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]390lb # [ 0.032986] [mem 0x60000000-0xafffffff] available for PCI devices391worker2 # [ 0.002723] found SMP MP-table at [mem 0x000f5450-0x000f545f]392client # [ 0.002825] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)393lb # [ 0.032987] Booting paravirtualized kernel on KVM394worker2 # [ 0.002734] Using GB pages for direct mapping395worker1 # [ 0.005430] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]396client # [ 0.002836] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)397worker2 # [ 0.002846] RAMDISK: [mem 0x5e36f000-0x5ffcffff]398worker1 # [ 0.005451] On node 0, zone DMA: 1 pages in unavailable ranges399lb # [ 0.032990] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns400client # [ 0.002838] ACPI: FACS 0x000000005FFE0000 000040401worker2 # [ 0.002855] ACPI: Early table checksum verification disabled402worker1 # [ 0.005722] On node 0, zone DMA: 97 pages in unavailable ranges403worker2 # [ 0.002858] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )404lb # [ 0.037468] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1405client # [ 0.002839] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)406worker1 # [ 0.031702] On node 0, zone DMA32: 40 pages in unavailable ranges407worker1 # [ 0.032154] ACPI: PM-Timer IO Port: 0x608408lb # [ 0.039564] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152409worker2 # [ 0.002862] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)410client # [ 0.002841] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)411lb # [ 0.039606] kvm-guest: PV spinlocks disabled, single CPU412worker1 # [ 0.032165] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])413worker2 # [ 0.002866] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)414client # [ 0.002842] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)415worker1 # [ 0.032196] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23416worker2 # [ 0.002873] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)417worker1 # [ 0.032198] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)418client # [ 0.002844] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)419worker2 # [ 0.002875] ACPI: FACS 0x000000005FFE0000 000040420worker1 # [ 0.032200] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)421client # [ 0.002845] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]422worker1 # [ 0.032201] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)423worker2 # [ 0.002877] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)424client # [ 0.002846] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]425lb # [ 0.039607] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/nry04znx19amfwpgmmym379f7hcf2spd-nixos-system-lb-test/init regInfo=/nix/store/ax20dpiqqpa2zqr4ahb7bspnj25x5j22-closure-info/registration console=ttyS0,115200n8 console=tty0426worker1 # [ 0.032202] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)427worker2 # [ 0.002878] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)428client # [ 0.002847] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]429worker1 # [ 0.032203] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)430client # [ 0.002847] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]431worker2 # [ 0.002880] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)432lb # [ 0.039700] Unknown kernel command line parameters "regInfo=/nix/store/ax20dpiqqpa2zqr4ahb7bspnj25x5j22-closure-info/registration", will be passed to user space.433worker1 # [ 0.032205] ACPI: Using ACPI (MADT) for SMP configuration information434lb # [ 0.039712] random: crng init done435client # [ 0.002848] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]436worker1 # [ 0.032206] ACPI: HPET id: 0x8086a201 base: 0xfed00000437worker2 # [ 0.002882] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)438lb # [ 0.039713] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes439worker1 # [ 0.032209] TSC deadline timer available440client # [ 0.002848] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]441worker2 # [ 0.002883] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]442worker1 # [ 0.032212] CPU topo: Max. logical packages: 1443lb # [ 0.041786] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)444client # [ 0.002849] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]445worker1 # [ 0.032213] CPU topo: Max. logical dies: 1446worker2 # [ 0.002884] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]447client # [ 0.003074] No NUMA configuration found448worker1 # [ 0.032213] CPU topo: Max. dies per package: 1449lb # [ 0.042997] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)450worker2 # [ 0.002884] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]451client # [ 0.003075] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]452worker1 # [ 0.032217] CPU topo: Max. threads per core: 1453lb # [ 0.043033] Fallback order for Node 0: 0454worker1 # [ 0.032217] CPU topo: Num. cores per package: 1455worker2 # [ 0.002885] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]456lb # [ 0.043036] Built 1 zonelists, mobility grouping on. Total pages: 393078457client # [ 0.003077] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]458worker1 # [ 0.032218] CPU topo: Num. threads per package: 1459client # [ 0.005396] Zone ranges:460lb # [ 0.043037] Policy zone: DMA32461worker2 # [ 0.002885] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]462client # [ 0.005397] DMA [mem 0x0000000000001000-0x0000000000ffffff]463worker1 # [ 0.032218] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs464lb # [ 0.045765] mem auto-init: stack:all(zero), heap alloc:on, heap free:off465worker2 # [ 0.002886] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]466client # [ 0.005398] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]467worker1 # [ 0.032235] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()468client # [ 0.005399] Normal empty469lb # [ 0.048788] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1470client # [ 0.005400] Device empty471worker2 # [ 0.002886] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]472lb # [ 0.052027] allocated 3145728 bytes of page_ext473client # [ 0.005401] Movable zone start for each node474worker2 # [ 0.003130] No NUMA configuration found475worker1 # [ 0.032265] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]476lb # [ 0.062032] ftrace: allocating 48787 entries in 192 pages477client # [ 0.005401] Early memory node ranges478worker2 # [ 0.003131] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]479worker1 # [ 0.032267] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]480lb # [ 0.062034] ftrace: allocated 192 pages with 2 groups481client # [ 0.005402] node 0: [mem 0x0000000000001000-0x000000000009efff]482lb # [ 0.062849] Dynamic Preempt: lazy483worker2 # [ 0.003133] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]484worker1 # [ 0.032270] [mem 0x60000000-0xafffffff] available for PCI devices485worker2 # [ 0.005547] Zone ranges:486lb # [ 0.063006] rcu: Preemptible hierarchical RCU implementation.487client # [ 0.005403] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]488worker1 # [ 0.032272] Booting paravirtualized kernel on KVM489worker2 # [ 0.005548] DMA [mem 0x0000000000001000-0x0000000000ffffff]490lb # [ 0.063007] rcu: RCU event tracing is enabled.491client # [ 0.005404] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]492worker2 # [ 0.005549] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]493lb # [ 0.063008] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.494worker1 # [ 0.032275] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns495client # [ 0.005423] On node 0, zone DMA: 1 pages in unavailable ranges496worker2 # [ 0.005550] Normal empty497worker2 # [ 0.005551] Device empty498lb # [ 0.063009] Trampoline variant of Tasks RCU enabled.499client # [ 0.005692] On node 0, zone DMA: 97 pages in unavailable ranges500worker1 # [ 0.036744] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1501worker2 # [ 0.005552] Movable zone start for each node502lb # [ 0.063009] Rude variant of Tasks RCU enabled.503worker2 # [ 0.005553] Early memory node ranges504client # [ 0.031453] On node 0, zone DMA32: 40 pages in unavailable ranges505lb # [ 0.063009] Tracing variant of Tasks RCU enabled.506worker1 # [ 0.038839] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152507client # [ 0.031913] ACPI: PM-Timer IO Port: 0x608508worker2 # [ 0.005553] node 0: [mem 0x0000000000001000-0x000000000009efff]509worker1 # [ 0.038883] kvm-guest: PV spinlocks disabled, single CPU510lb # [ 0.063010] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.511client # [ 0.031923] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])512worker2 # [ 0.005554] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]513lb # [ 0.063011] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1514client # [ 0.031948] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23515worker2 # [ 0.005555] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]516lb # [ 0.063032] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.517client # [ 0.031950] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)518worker2 # [ 0.005575] On node 0, zone DMA: 1 pages in unavailable ranges519client # [ 0.031952] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)520worker2 # [ 0.005850] On node 0, zone DMA: 97 pages in unavailable ranges521lb # [ 0.063034] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.522client # [ 0.031953] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)523worker2 # [ 0.033965] On node 0, zone DMA32: 40 pages in unavailable ranges524worker1 # [ 0.038885] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/7hj4sk4i9w6mp1h0l38acx95n0smfk2s-nixos-system-worker1-test/init regInfo=/nix/store/1ra8h8rvaqsaasm1v4s2g2fsrrsb0g1s-closure-info/registration console=ttyS0,115200n8 console=tty0525worker2 # [ 0.034431] ACPI: PM-Timer IO Port: 0x608526lb # [ 0.063035] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.527client # [ 0.031954] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)528worker2 # [ 0.034444] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])529lb # [ 0.067401] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16530client # [ 0.031954] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)531worker1 # [ 0.038975] Unknown kernel command line parameters "regInfo=/nix/store/1ra8h8rvaqsaasm1v4s2g2fsrrsb0g1s-closure-info/registration", will be passed to user space.532worker2 # [ 0.034474] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23533lb # [ 0.067684] rcu: srcu_init: Setting srcu_struct sizes based on contention.534worker1 # [ 0.038987] random: crng init done535client # [ 0.031957] ACPI: Using ACPI (MADT) for SMP configuration information536worker2 # [ 0.034477] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)537client # [ 0.031958] ACPI: HPET id: 0x8086a201 base: 0xfed00000538worker1 # [ 0.038988] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes539lb # [ 0.067692] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns540client # [ 0.031962] TSC deadline timer available541worker2 # [ 0.034478] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)542client # [ 0.031965] CPU topo: Max. logical packages: 1543worker1 # [ 0.041073] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)544worker2 # [ 0.034480] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)545lb # [ 0.067794] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)546client # [ 0.031966] CPU topo: Max. logical dies: 1547lb # [ 0.071495] Console: colour VGA+ 80x25548worker2 # [ 0.034481] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)549worker1 # [ 0.042298] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)550client # [ 0.031967] CPU topo: Max. dies per package: 1551lb # [ 0.071498] printk: legacy console [tty0] enabled552worker1 # [ 0.042327] Fallback order for Node 0: 0553client # [ 0.031970] CPU topo: Max. threads per core: 1554worker2 # [ 0.034481] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)555lb # [ 0.100853] printk: legacy console [ttyS0] enabled556client # [ 0.031970] CPU topo: Num. cores per package: 1557worker1 # [ 0.042330] Built 1 zonelists, mobility grouping on. Total pages: 393078558worker2 # [ 0.034484] ACPI: Using ACPI (MADT) for SMP configuration information559worker1 # [ 0.042331] Policy zone: DMA32560lb # [ 0.196173] ACPI: Core revision 20250807561client # [ 0.031971] CPU topo: Num. threads per package: 1562worker2 # [ 0.034485] ACPI: HPET id: 0x8086a201 base: 0xfed00000563worker1 # [ 0.044755] mem auto-init: stack:all(zero), heap alloc:on, heap free:off564client # [ 0.031971] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs565worker2 # [ 0.034488] TSC deadline timer available566worker2 # [ 0.034492] CPU topo: Max. logical packages: 1567worker1 # [ 0.047710] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1568client # [ 0.031988] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()569lb # [ 0.197097] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns570worker2 # [ 0.034493] CPU topo: Max. logical dies: 1571worker1 # [ 0.050935] allocated 3145728 bytes of page_ext572client # [ 0.032017] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]573lb # [ 0.198557] APIC: Switch to symmetric I/O mode setup574worker2 # [ 0.034493] CPU topo: Max. dies per package: 1575worker1 # [ 0.060779] ftrace: allocating 48787 entries in 192 pages576worker2 # [ 0.034497] CPU topo: Max. threads per core: 1577lb # [ 0.199521] x2apic enabled578worker1 # [ 0.060781] ftrace: allocated 192 pages with 2 groups579client # [ 0.032019] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]580worker2 # [ 0.034497] CPU topo: Num. cores per package: 1581worker1 # [ 0.061586] Dynamic Preempt: lazy582client # [ 0.032020] [mem 0x60000000-0xafffffff] available for PCI devices583worker2 # [ 0.034498] CPU topo: Num. threads per package: 1584lb # [ 0.200250] APIC: Switched APIC routing to: physical x2apic585worker1 # [ 0.061759] rcu: Preemptible hierarchical RCU implementation.586client # [ 0.032021] Booting paravirtualized kernel on KVM587worker1 # [ 0.061759] rcu: RCU event tracing is enabled.588worker2 # [ 0.034498] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs589worker1 # [ 0.061760] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.590worker2 # [ 0.034518] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()591client # [ 0.032024] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns592lb # [ 0.202024] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1593worker1 # [ 0.061762] Trampoline variant of Tasks RCU enabled.594worker2 # [ 0.034549] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]595client # [ 0.036487] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1596worker1 # [ 0.061762] Rude variant of Tasks RCU enabled.597worker1 # [ 0.061762] Tracing variant of Tasks RCU enabled.598client # [ 0.038581] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152599worker2 # [ 0.034551] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]600lb # [ 0.202975] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns601client # [ 0.038624] kvm-guest: PV spinlocks disabled, single CPU602worker2 # [ 0.034552] [mem 0x60000000-0xafffffff] available for PCI devices603worker1 # [ 0.061763] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.604worker2 # [ 0.034554] Booting paravirtualized kernel on KVM605worker1 # [ 0.061764] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1606lb # [ 0.204549] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)607worker2 # [ 0.034557] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns608lb # [ 0.206633] x86/cpu: User Mode Instruction Prevention (UMIP) activated609worker1 # [ 0.061784] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.610worker2 # [ 0.039050] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1611lb # [ 0.207676] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127612worker1 # [ 0.061786] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.613worker2 # [ 0.041354] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152614client # [ 0.038626] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/nwrxkcyq2rabn5jfqyww8chp4fbw12l7-nixos-system-client-test/init regInfo=/nix/store/9mbchw87wm67kspwy4r01ypax3r3mych-closure-info/registration console=ttyS0,115200n8 console=tty0615worker1 # [ 0.061787] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.616worker2 # [ 0.041403] kvm-guest: PV spinlocks disabled, single CPU617lb # [ 0.208547] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0618worker1 # [ 0.066309] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16619client # [ 0.038717] Unknown kernel command line parameters "regInfo=/nix/store/9mbchw87wm67kspwy4r01ypax3r3mych-closure-info/registration", will be passed to user space.620worker1 # [ 0.066588] rcu: srcu_init: Setting srcu_struct sizes based on contention.621client # [ 0.038729] random: crng init done622lb # [ 0.209550] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto623client # [ 0.038729] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes624worker1 # [ 0.066594] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns625lb # [ 0.210547] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl626client # [ 0.040797] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)627worker2 # [ 0.041405] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/6bw4n0f8bls8vkb4018phy0as0l93xnb-nixos-system-worker2-test/init regInfo=/nix/store/mnhq4clcym8k7q31bph6wbcnxjd6nnha-closure-info/registration console=ttyS0,115200n8 console=tty0628worker1 # [ 0.066691] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)629lb # [ 0.211547] Transient Scheduler Attacks: Vulnerable: No microcode630worker1 # [ 0.070212] Console: colour VGA+ 80x25631client # [ 0.042009] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)632client # [ 0.042039] Fallback order for Node 0: 0633worker1 # [ 0.070215] printk: legacy console [tty0] enabled634lb # [ 0.212546] Spectre V2 : Mitigation: Enhanced / Automatic IBRS635worker2 # [ 0.041501] Unknown kernel command line parameters "regInfo=/nix/store/mnhq4clcym8k7q31bph6wbcnxjd6nnha-closure-info/registration", will be passed to user space.636worker1 # [ 0.099639] printk: legacy console [ttyS0] enabled637client # [ 0.042042] Built 1 zonelists, mobility grouping on. Total pages: 393078638worker2 # [ 0.041513] random: crng init done639lb # [ 0.213547] Speculative Return Stack Overflow: Mitigation: Safe RET640client # [ 0.042043] Policy zone: DMA32641worker1 # [ 0.196581] ACPI: Core revision 20250807642worker2 # [ 0.041514] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes643client # [ 0.044452] mem auto-init: stack:all(zero), heap alloc:on, heap free:off644worker2 # [ 0.043765] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)645client # [ 0.047419] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1646worker1 # [ 0.197474] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns647client # [ 0.050655] allocated 3145728 bytes of page_ext648worker2 # [ 0.045220] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)649client # [ 0.060530] ftrace: allocating 48787 entries in 192 pages650worker1 # [ 0.199004] APIC: Switch to symmetric I/O mode setup651worker2 # [ 0.045253] Fallback order for Node 0: 0652client # [ 0.060532] ftrace: allocated 192 pages with 2 groups653worker1 # [ 0.199992] x2apic enabled654worker2 # [ 0.045256] Built 1 zonelists, mobility grouping on. Total pages: 393078655client # [ 0.061354] Dynamic Preempt: lazy656lb # [ 0.214546] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization657worker2 # [ 0.045257] Policy zone: DMA32658client # [ 0.061497] rcu: Preemptible hierarchical RCU implementation.659lb # [ 0.216554] Spectre V2 : Enabling IBPB for BPF660worker1 # [ 0.200771] APIC: Switched APIC routing to: physical x2apic661worker2 # [ 0.047874] mem auto-init: stack:all(zero), heap alloc:on, heap free:off662client # [ 0.061497] rcu: RCU event tracing is enabled.663worker2 # [ 0.051161] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1664client # [ 0.061498] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.665lb # [ 0.217253] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier666worker2 # [ 0.054637] allocated 3145728 bytes of page_ext667client # [ 0.061499] Trampoline variant of Tasks RCU enabled.668worker1 # [ 0.202556] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1669worker2 # [ 0.064601] ftrace: allocating 48787 entries in 192 pages670lb # [ 0.218547] active return thunk: srso_alias_return_thunk671client # [ 0.061500] Rude variant of Tasks RCU enabled.672worker2 # [ 0.064604] ftrace: allocated 192 pages with 2 groups673client # [ 0.061500] Tracing variant of Tasks RCU enabled.674worker2 # [ 0.065498] Dynamic Preempt: lazy675lb # [ 0.219568] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'676worker1 # [ 0.203533] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns677client # [ 0.061501] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.678worker2 # [ 0.065698] rcu: Preemptible hierarchical RCU implementation.679lb # [ 0.220547] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'680worker2 # [ 0.065699] rcu: RCU event tracing is enabled.681client # [ 0.061501] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1682worker1 # [ 0.205179] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)683worker2 # [ 0.065700] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.684lb # [ 0.221546] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'685client # [ 0.061558] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.686worker2 # [ 0.065701] Trampoline variant of Tasks RCU enabled.687worker1 # [ 0.206262] x86/cpu: User Mode Instruction Prevention (UMIP) activated688worker2 # [ 0.065702] Rude variant of Tasks RCU enabled.689lb # [ 0.222547] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'690client # [ 0.061560] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.691worker2 # [ 0.065702] Tracing variant of Tasks RCU enabled.692worker1 # [ 0.207302] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127693lb # [ 0.223547] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'694worker2 # [ 0.065703] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.695client # [ 0.061561] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.696worker1 # [ 0.208177] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0697lb # [ 0.224547] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'698worker2 # [ 0.065704] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1699client # [ 0.066123] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16700client # [ 0.066402] rcu: srcu_init: Setting srcu_struct sizes based on contention.701worker2 # [ 0.065729] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.702lb # [ 0.225546] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'703worker1 # [ 0.209179] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto704client # [ 0.066408] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns705worker2 # [ 0.065730] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.706lb # [ 0.226547] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256707worker1 # [ 0.211177] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl708client # [ 0.066507] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)709worker2 # [ 0.065731] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.710lb # [ 0.227546] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64711client # [ 0.070033] Console: colour VGA+ 80x25712worker1 # [ 0.212177] Transient Scheduler Attacks: Vulnerable: No microcode713worker2 # [ 0.070453] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16714client # [ 0.070036] printk: legacy console [tty0] enabled715lb # [ 0.228547] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512716worker2 # [ 0.070735] rcu: srcu_init: Setting srcu_struct sizes based on contention.717worker1 # [ 0.213176] Spectre V2 : Mitigation: Enhanced / Automatic IBRS718client # [ 0.099480] printk: legacy console [ttyS0] enabled719lb # [ 0.229547] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024720client # [ 0.195313] ACPI: Core revision 20250807721worker1 # [ 0.214176] Speculative Return Stack Overflow: Mitigation: Safe RET722worker2 # [ 0.070741] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns723lb # [ 0.230546] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8724worker2 # [ 0.070849] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)725client # [ 0.196182] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns726worker2 # [ 0.074416] Console: colour VGA+ 80x25727lb # [ 0.231547] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.728worker2 # [ 0.074419] printk: legacy console [tty0] enabled729client # [ 0.197685] APIC: Switch to symmetric I/O mode setup730worker2 # [ 0.104008] printk: legacy console [ttyS0] enabled731client # [ 0.198641] x2apic enabled732worker2 # [ 0.199229] ACPI: Core revision 20250807733worker1 # [ 0.215176] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization734client # [ 0.199419] APIC: Switched APIC routing to: physical x2apic735worker1 # [ 0.217184] Spectre V2 : Enabling IBPB for BPF736worker2 # [ 0.200228] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns737client # [ 0.201261] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1738worker1 # [ 0.218177] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier739worker2 # [ 0.201755] APIC: Switch to symmetric I/O mode setup740worker2 # [ 0.202751] x2apic enabled741worker1 # [ 0.219177] active return thunk: srso_alias_return_thunk742client # [ 0.202251] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns743worker2 # [ 0.203540] APIC: Switched APIC routing to: physical x2apic744worker1 # [ 0.220830] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'745client # [ 0.203925] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)746worker1 # [ 0.222085] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'747worker2 # [ 0.205380] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1748client # [ 0.205010] x86/cpu: User Mode Instruction Prevention (UMIP) activated749worker1 # [ 0.222939] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'750client # [ 0.206037] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127751worker1 # [ 0.223921] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'752worker2 # [ 0.206346] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns753client # [ 0.207606] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0754worker1 # [ 0.224925] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'755worker2 # [ 0.208005] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)756worker1 # [ 0.225928] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'757client # [ 0.207927] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto758worker2 # [ 0.209091] x86/cpu: User Mode Instruction Prevention (UMIP) activated759worker1 # [ 0.226973] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'760worker2 # [ 0.210148] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127761client # [ 0.209923] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl762worker1 # [ 0.228176] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256763worker2 # [ 0.211678] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0764client # [ 0.210922] Transient Scheduler Attacks: Vulnerable: No microcode765worker1 # [ 0.229176] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64766client # [ 0.212648] Spectre V2 : Mitigation: Enhanced / Automatic IBRS767worker1 # [ 0.230176] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512768worker2 # [ 0.212006] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto769client # [ 0.212922] Speculative Return Stack Overflow: Mitigation: Safe RET770worker1 # [ 0.231176] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024771worker2 # [ 0.214002] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl772worker1 # [ 0.232176] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8773worker2 # [ 0.215003] Transient Scheduler Attacks: Vulnerable: No microcode774worker1 # [ 0.233176] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.775worker2 # [ 0.216721] Spectre V2 : Mitigation: Enhanced / Automatic IBRS776client # [ 0.213922] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization777worker2 # [ 0.217689] Speculative Return Stack Overflow: Mitigation: Safe RET778client # [ 0.215929] Spectre V2 : Enabling IBPB for BPF779client # [ 0.216923] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier780client # [ 0.217922] active return thunk: srso_alias_return_thunk781worker2 # [ 0.218002] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization782client # [ 0.219615] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'783worker2 # [ 0.220010] Spectre V2 : Enabling IBPB for BPF784client # [ 0.220873] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'785worker2 # [ 0.221003] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier786client # [ 0.221922] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'787worker2 # [ 0.222003] active return thunk: srso_alias_return_thunk788client # [ 0.222922] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'789worker2 # [ 0.223647] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'790client # [ 0.223922] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'791worker2 # [ 0.224890] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'792client # [ 0.224922] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'793worker2 # [ 0.225745] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'794client # [ 0.225922] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'795worker2 # [ 0.226745] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'796client # [ 0.226922] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256797worker2 # [ 0.227750] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'798client # [ 0.227922] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64799worker2 # [ 0.228748] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'800lb # [ 0.266237] Freeing SMP alternatives memory: 44K801client # [ 0.228922] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512802lb # [ 0.266550] pid_max: default: 32768 minimum: 301803worker2 # [ 0.229775] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'804client # [ 0.229922] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024805worker2 # [ 0.231002] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256806lb # [ 0.267641] LSM: initializing lsm=capability,landlock,yama,bpf,ima807client # [ 0.230922] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8808lb # [ 0.268653] landlock: Up and running.809worker2 # [ 0.232002] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64810lb # [ 0.269547] Yama: becoming mindful.811client # [ 0.231922] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.812worker2 # [ 0.232921] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512813lb # [ 0.270348] LSM support for eBPF active814worker2 # [ 0.233708] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024815lb # [ 0.271121] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)816worker2 # [ 0.234717] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8817lb # [ 0.271582] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)818worker2 # [ 0.235719] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.819lb # [ 0.274809] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)820lb # [ 0.276096] Performance Events: Fam17h+ core perfctr, AMD PMU driver.821lb # [ 0.276556] ... version: 2822lb # [ 0.277301] ... bit width: 48823lb # [ 0.277549] ... generic counters: 6824lb # [ 0.278343] ... generic bitmap: 000000000000003f825lb # [ 0.278549] ... fixed-purpose counters: 0826lb # [ 0.279303] ... fixed-purpose bitmap: 0000000000000000827lb # [ 0.279549] ... value mask: 0000ffffffffffff828lb # [ 0.280505] ... max period: 00007fffffffffff829lb # [ 0.281288] ... global_ctrl mask: 000000000000003f830worker1 # [ 0.267826] Freeing SMP alternatives memory: 44K831lb # [ 0.281645] signal: max sigframe size: 3376832worker1 # [ 0.268179] pid_max: default: 32768 minimum: 301833lb # [ 0.282508] rcu: Hierarchical SRCU implementation.834worker1 # [ 0.269271] LSM: initializing lsm=capability,landlock,yama,bpf,ima835lb # [ 0.283177] rcu: Max phase no-delay instances is 400.836worker1 # [ 0.270284] landlock: Up and running.837worker1 # [ 0.270931] Yama: becoming mindful.838worker1 # [ 0.272193] LSM support for eBPF active839worker1 # [ 0.272963] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)840worker1 # [ 0.274218] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)841lb # [ 0.288045] smp: Bringing up secondary CPUs ...842lb # [ 0.288564] smp: Brought up 1 node, 1 CPU843lb # [ 0.289265] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)844worker1 # [ 0.276621] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)845worker1 # [ 0.277739] Performance Events: Fam17h+ core perfctr, AMD PMU driver.846lb # [ 0.289749] Memory: 1454620K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109316K reserved, 0K cma-reserved)847worker1 # [ 0.278181] ... version: 2848lb # [ 0.290765] devtmpfs: initialized849worker1 # [ 0.278916] ... bit width: 48850worker1 # [ 0.279179] ... generic counters: 6851lb # [ 0.291647] x86/mm: Memory block size: 128MB852worker1 # [ 0.279886] ... generic bitmap: 000000000000003f853worker1 # [ 0.280185] ... fixed-purpose counters: 0854lb # [ 0.293439] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)855worker1 # [ 0.280959] ... fixed-purpose bitmap: 0000000000000000856worker1 # [ 0.281178] ... value mask: 0000ffffffffffff857lb # [ 0.294478] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).858worker1 # [ 0.282132] ... max period: 00007fffffffffff859lb # [ 0.295647] pinctrl core: initialized pinctrl subsystem860client # [ 0.267008] Freeing SMP alternatives memory: 44K861worker1 # [ 0.282882] ... global_ctrl mask: 000000000000003f862client # [ 0.267782] pid_max: default: 32768 minimum: 301863lb # [ 0.296727] PM: RTC time: 16:04:09, date: 2026-09-22864worker1 # [ 0.283276] signal: max sigframe size: 3376865client # [ 0.268547] LSM: initializing lsm=capability,landlock,yama,bpf,ima866worker1 # [ 0.284081] rcu: Hierarchical SRCU implementation.867client # [ 0.269032] landlock: Up and running.868worker1 # [ 0.284753] rcu: Max phase no-delay instances is 400.869client # [ 0.269923] Yama: becoming mindful.870worker2 # [ 0.270870] Freeing SMP alternatives memory: 44K871client # [ 0.271141] LSM support for eBPF active872worker2 # [ 0.271566] pid_max: default: 32768 minimum: 301873lb # [ 0.300203] NET: Registered PF_NETLINK/PF_ROUTE protocol family874client # [ 0.271940] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)875worker2 # [ 0.272100] LSM: initializing lsm=capability,landlock,yama,bpf,ima876lb # [ 0.301057] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations877worker2 # [ 0.273132] landlock: Up and running.878client # [ 0.272958] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)879worker2 # [ 0.274002] Yama: becoming mindful.880lb # [ 0.301571] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations881worker2 # [ 0.275187] LSM support for eBPF active882worker1 # [ 0.289673] smp: Bringing up secondary CPUs ...883worker1 # [ 0.290193] smp: Brought up 1 node, 1 CPU884lb # [ 0.302819] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations885client # [ 0.275388] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)886worker2 # [ 0.275959] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)887lb # [ 0.303559] audit: initializing netlink subsys (disabled)888worker1 # [ 0.290833] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)889client # [ 0.276472] Performance Events: Fam17h+ core perfctr, AMD PMU driver.890worker2 # [ 0.277044] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)891lb # [ 0.304715] thermal_sys: Registered thermal governor 'fair_share'892client # [ 0.276927] ... version: 2893client # [ 0.277619] ... bit width: 48894worker1 # [ 0.291360] Memory: 1454604K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)895lb # [ 0.304717] thermal_sys: Registered thermal governor 'bang_bang'896client # [ 0.277925] ... generic counters: 6897worker1 # [ 0.292422] devtmpfs: initialized898lb # [ 0.305550] thermal_sys: Registered thermal governor 'step_wise'899worker2 # [ 0.279426] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)900client # [ 0.278617] ... generic bitmap: 000000000000003f901worker1 # [ 0.293155] x86/mm: Memory block size: 128MB902lb # [ 0.306476] thermal_sys: Registered thermal governor 'user_space'903client # [ 0.278929] ... fixed-purpose counters: 0904worker2 # [ 0.280552] Performance Events: Fam17h+ core perfctr, AMD PMU driver.905client # [ 0.279628] ... fixed-purpose bitmap: 0000000000000000906lb # [ 0.307256] audit: type=2000 audit(1790093050.006:1): state=initialized audit_enabled=0 res=1907worker2 # [ 0.281007] ... version: 2908worker1 # [ 0.294706] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)909client # [ 0.279924] ... value mask: 0000ffffffffffff910worker2 # [ 0.281742] ... bit width: 48911lb # [ 0.308560] thermal_sys: Registered thermal governor 'power_allocator'912worker2 # [ 0.282004] ... generic counters: 6913client # [ 0.280783] ... max period: 00007fffffffffff914lb # [ 0.308575] cpuidle: using governor menu915worker1 # [ 0.295203] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).916worker2 # [ 0.282673] ... generic bitmap: 000000000000003f917client # [ 0.281559] ... global_ctrl mask: 000000000000003f918worker2 # [ 0.283053] ... fixed-purpose counters: 0919worker1 # [ 0.296262] pinctrl core: initialized pinctrl subsystem920client # [ 0.282036] signal: max sigframe size: 3376921lb # [ 0.310746] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5922worker2 # [ 0.283758] ... fixed-purpose bitmap: 0000000000000000923worker1 # [ 0.297377] PM: RTC time: 16:04:09, date: 2026-09-22924client # [ 0.282831] rcu: Hierarchical SRCU implementation.925lb # [ 0.311778] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]926worker2 # [ 0.284005] ... value mask: 0000ffffffffffff927client # [ 0.283492] rcu: Max phase no-delay instances is 400.928worker2 # [ 0.284876] ... max period: 00007fffffffffff929lb # [ 0.312552] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry930worker2 # [ 0.285685] ... global_ctrl mask: 000000000000003f931lb # [ 0.313562] PCI: Using configuration type 1 for base access932worker2 # [ 0.286119] signal: max sigframe size: 3376933worker1 # [ 0.300860] NET: Registered PF_NETLINK/PF_ROUTE protocol family934lb # [ 0.314700] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.935worker2 # [ 0.286922] rcu: Hierarchical SRCU implementation.936worker2 # [ 0.287599] rcu: Max phase no-delay instances is 400.937worker1 # [ 0.301676] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations938client # [ 0.288519] smp: Bringing up secondary CPUs ...939worker1 # [ 0.302203] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations940client # [ 0.288939] smp: Brought up 1 node, 1 CPU941client # [ 0.289619] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)942worker1 # [ 0.303445] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations943worker1 # [ 0.304188] audit: initializing netlink subsys (disabled)944lb # [ 0.319773] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages945client # [ 0.290180] Memory: 1454576K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)946worker1 # [ 0.305373] thermal_sys: Registered thermal governor 'fair_share'947worker2 # [ 0.292654] smp: Bringing up secondary CPUs ...948lb # [ 0.320550] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page949client # [ 0.291150] devtmpfs: initialized950worker1 # [ 0.305376] thermal_sys: Registered thermal governor 'bang_bang'951worker2 # [ 0.293018] smp: Brought up 1 node, 1 CPU952client # [ 0.291973] x86/mm: Memory block size: 128MB953worker1 # [ 0.306180] thermal_sys: Registered thermal governor 'step_wise'954worker2 # [ 0.293709] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)955worker1 # [ 0.307128] thermal_sys: Registered thermal governor 'user_space'956client # [ 0.293755] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)957worker1 # [ 0.307945] audit: type=2000 audit(1790093050.023:1): state=initialized audit_enabled=0 res=1958worker2 # [ 0.294173] Memory: 1454560K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109312K reserved, 0K cma-reserved)959worker2 # [ 0.295323] devtmpfs: initialized960client # [ 0.294820] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).961lb # [ 0.325551] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages962worker1 # [ 0.309181] thermal_sys: Registered thermal governor 'power_allocator'963worker1 # [ 0.309195] cpuidle: using governor menu964worker2 # [ 0.296086] x86/mm: Memory block size: 128MB965client # [ 0.296008] pinctrl core: initialized pinctrl subsystem966lb # [ 0.326549] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page967client # [ 0.297035] PM: RTC time: 16:04:09, date: 2026-09-22968worker1 # [ 0.311168] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5969worker2 # [ 0.297869] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)970worker1 # [ 0.312208] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]971worker2 # [ 0.298993] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).972worker1 # [ 0.313181] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry973client # [ 0.300526] NET: Registered PF_NETLINK/PF_ROUTE protocol family974worker2 # [ 0.300083] pinctrl core: initialized pinctrl subsystem975worker1 # [ 0.314190] PCI: Using configuration type 1 for base access976worker2 # [ 0.301291] PM: RTC time: 16:04:09, date: 2026-09-22977client # [ 0.301407] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations978worker1 # [ 0.315248] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.979client # [ 0.301947] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations980client # [ 0.303195] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations981worker2 # [ 0.304886] NET: Registered PF_NETLINK/PF_ROUTE protocol family982client # [ 0.303934] audit: initializing netlink subsys (disabled)983lb # [ 0.336904] ACPI: Added _OSI(Module Device)984lb # [ 0.337551] ACPI: Added _OSI(Processor Device)985client # [ 0.305003] thermal_sys: Registered thermal governor 'fair_share'986worker2 # [ 0.306278] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations987lb # [ 0.338303] ACPI: Added _OSI(Processor Aggregator Device)988client # [ 0.305005] thermal_sys: Registered thermal governor 'bang_bang'989worker2 # [ 0.307029] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations990client # [ 0.305926] thermal_sys: Registered thermal governor 'step_wise'991worker1 # [ 0.322366] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages992client # [ 0.306863] thermal_sys: Registered thermal governor 'user_space'993worker2 # [ 0.308291] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations994worker1 # [ 0.323179] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page995worker2 # [ 0.309017] audit: initializing netlink subsys (disabled)996client # [ 0.307647] audit: type=2000 audit(1790093050.043:1): state=initialized audit_enabled=0 res=1997client # [ 0.308927] thermal_sys: Registered thermal governor 'power_allocator'998worker2 # [ 0.310007] audit: type=2000 audit(1790093050.048:1): state=initialized audit_enabled=0 res=1999client # [ 0.308949] cpuidle: using governor menu1000worker2 # [ 0.311202] thermal_sys: Registered thermal governor 'fair_share'1001client # [ 0.311151] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.51002worker2 # [ 0.311204] thermal_sys: Registered thermal governor 'bang_bang'1003worker2 # [ 0.312006] thermal_sys: Registered thermal governor 'step_wise'1004worker1 # [ 0.328180] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1005client # [ 0.312157] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]1006worker2 # [ 0.312974] thermal_sys: Registered thermal governor 'user_space'1007worker1 # [ 0.329179] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1008client # [ 0.312928] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry1009lb # [ 0.347118] ACPI: 1 ACPI AML tables successfully acquired and loaded1010worker2 # [ 0.313716] thermal_sys: Registered thermal governor 'power_allocator'1011client # [ 0.313936] PCI: Using configuration type 1 for base access1012worker2 # [ 0.314017] cpuidle: using governor menu1013lb # [ 0.348459] ACPI: Interpreter enabled1014lb # [ 0.349053] ACPI: PM: (supports S0 S3 S4 S5)1015client # [ 0.315050] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.1016worker2 # [ 0.316260] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.51017worker2 # [ 0.317250] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]1018lb # [ 0.351551] ACPI: Using IOAPIC for interrupt routing1019worker2 # [ 0.318008] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry1020worker2 # [ 0.319018] PCI: Using configuration type 1 for base access1021lb # [ 0.352413] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1022client # [ 0.320147] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1023worker2 # [ 0.320090] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.1024client # [ 0.320925] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1025worker1 # [ 0.339586] ACPI: Added _OSI(Module Device)1026lb # [ 0.355550] PCI: Using E820 reservations for host bridge windows1027worker1 # [ 0.340181] ACPI: Added _OSI(Processor Device)1028lb # [ 0.356660] ACPI: Enabled 2 GPEs in block 00 to 3F1029worker1 # [ 0.340918] ACPI: Added _OSI(Processor Aggregator Device)1030client # [ 0.325926] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1031client # [ 0.326925] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1032worker2 # [ 0.327021] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1033worker2 # [ 0.328005] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1034worker1 # [ 0.347757] ACPI: 1 ACPI AML tables successfully acquired and loaded1035lb # [ 0.365318] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1036worker2 # [ 0.333005] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1037lb # [ 0.366175] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1038worker2 # [ 0.334006] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1039lb # [ 0.366643] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1040worker1 # [ 0.351578] ACPI: Interpreter enabled1041worker1 # [ 0.352193] ACPI: PM: (supports S0 S3 S4 S5)1042lb # [ 0.367675] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1043worker1 # [ 0.352908] ACPI: Using IOAPIC for interrupt routing1044client # [ 0.337272] ACPI: Added _OSI(Module Device)1045lb # [ 0.368931] PCI host bridge to bus 0000:001046client # [ 0.337926] ACPI: Added _OSI(Processor Device)1047lb # [ 0.369554] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1048client # [ 0.338664] ACPI: Added _OSI(Processor Aggregator Device)1049lb # [ 0.370550] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1050worker1 # [ 0.355259] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1051lb # [ 0.371549] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1052worker1 # [ 0.356179] PCI: Using E820 reservations for host bridge windows1053lb # [ 0.372550] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1054lb # [ 0.373549] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1055worker1 # [ 0.359296] ACPI: Enabled 2 GPEs in block 00 to 3F1056lb # [ 0.374549] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1057lb # [ 0.375550] pci_bus 0000:00: root bus resource [bus 00-ff]1058worker2 # [ 0.345243] ACPI: Added _OSI(Module Device)1059worker2 # [ 0.345953] ACPI: Added _OSI(Processor Device)1060lb # [ 0.376531] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1061worker2 # [ 0.346545] ACPI: Added _OSI(Processor Aggregator Device)1062client # [ 0.347458] ACPI: 1 ACPI AML tables successfully acquired and loaded1063lb # [ 0.378046] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1064client # [ 0.348811] ACPI: Interpreter enabled1065client # [ 0.349437] ACPI: PM: (supports S0 S3 S4 S5)1066worker1 # [ 0.366102] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1067worker1 # [ 0.366978] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1068client # [ 0.351925] ACPI: Using IOAPIC for interrupt routing1069worker2 # [ 0.352454] ACPI: 1 ACPI AML tables successfully acquired and loaded1070worker1 # [ 0.368257] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1071client # [ 0.352767] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1072worker1 # [ 0.369301] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1073worker1 # [ 0.370524] PCI host bridge to bus 0000:001074lb # [ 0.380630] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1075worker1 # [ 0.371183] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1076worker2 # [ 0.356414] ACPI: Interpreter enabled1077client # [ 0.355925] PCI: Using E820 reservations for host bridge windows1078lb # [ 0.381562] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1079worker1 # [ 0.372179] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1080worker2 # [ 0.357021] ACPI: PM: (supports S0 S3 S4 S5)1081client # [ 0.356988] ACPI: Enabled 2 GPEs in block 00 to 3F1082lb # [ 0.382500] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1083worker2 # [ 0.357693] ACPI: Using IOAPIC for interrupt routing1084worker1 # [ 0.373179] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1085lb # [ 0.383453] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1086worker1 # [ 0.374178] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1087worker1 # [ 0.375178] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1088worker2 # [ 0.360042] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1089lb # [ 0.385279] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1090worker1 # [ 0.376179] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1091worker2 # [ 0.361005] PCI: Using E820 reservations for host bridge windows1092worker1 # [ 0.377179] pci_bus 0000:00: root bus resource [bus 00-ff]1093lb # [ 0.387577] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1094worker1 # [ 0.378180] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1095lb # [ 0.388437] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1096worker2 # [ 0.364143] ACPI: Enabled 2 GPEs in block 00 to 3F1097worker1 # [ 0.379617] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1098client # [ 0.365611] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1099lb # [ 0.389329] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1100lb # [ 0.390367] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1101client # [ 0.366483] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1102client # [ 0.367005] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1103lb # [ 0.391918] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1104client # [ 0.368048] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1105client # [ 0.369275] PCI host bridge to bus 0000:001106lb # [ 0.393582] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1107lb # [ 0.394466] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1108client # [ 0.369929] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1109worker2 # [ 0.371653] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1110client # [ 0.370925] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1111lb # [ 0.395277] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1112worker2 # [ 0.372011] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1113worker1 # [ 0.382321] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1114client # [ 0.371925] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1115worker2 # [ 0.373084] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1116worker1 # [ 0.383191] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1117lb # [ 0.396094] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1118client # [ 0.372925] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1119worker1 # [ 0.384142] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1120worker2 # [ 0.374128] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1121client # [ 0.373925] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1122worker2 # [ 0.375402] PCI host bridge to bus 0000:001123worker1 # [ 0.385314] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1124client # [ 0.374925] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1125worker2 # [ 0.376009] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1126client # [ 0.375925] pci_bus 0000:00: root bus resource [bus 00-ff]1127lb # [ 0.398586] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1128worker2 # [ 0.377005] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1129worker1 # [ 0.387084] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1130lb # [ 0.399556] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1131client # [ 0.376936] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1132worker2 # [ 0.378005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1133lb # [ 0.400480] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1134worker2 # [ 0.379005] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1135client # [ 0.378392] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1136worker1 # [ 0.389219] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1137lb # [ 0.401952] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1138worker2 # [ 0.380005] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1139worker1 # [ 0.390090] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1140worker2 # [ 0.381004] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1141worker1 # [ 0.391017] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1142worker2 # [ 0.382005] pci_bus 0000:00: root bus resource [bus 00-ff]1143lb # [ 0.403586] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1144worker1 # [ 0.392053] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1145lb # [ 0.404562] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1146worker2 # [ 0.382993] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1147lb # [ 0.405529] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1148worker1 # [ 0.393799] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1149worker2 # [ 0.384462] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1150lb # [ 0.406433] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1151worker1 # [ 0.395218] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1152lb # [ 0.408119] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1153client # [ 0.381017] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1154worker1 # [ 0.396088] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1155client # [ 0.381937] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1156lb # [ 0.409380] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1157worker1 # [ 0.396946] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1158client # [ 0.382912] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1159lb # [ 0.410314] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1160worker1 # [ 0.398592] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1161client # [ 0.384203] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1162lb # [ 0.411126] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1163worker2 # [ 0.387102] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1164client # [ 0.385845] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1165lb # [ 0.412562] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1166worker1 # [ 0.401795] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1167worker2 # [ 0.388283] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1168lb # [ 0.413530] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1169worker1 # [ 0.402186] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1170worker2 # [ 0.389036] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1171client # [ 0.388962] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1172worker1 # [ 0.403200] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1173lb # [ 0.415124] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1174client # [ 0.389818] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1175worker2 # [ 0.390394] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1176client # [ 0.390673] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1177lb # [ 0.416269] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1178worker1 # [ 0.404790] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1179lb # [ 0.416572] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1180client # [ 0.390930] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1181worker2 # [ 0.391765] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1182worker1 # [ 0.406186] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1183lb # [ 0.418142] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1184worker1 # [ 0.407069] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1185client # [ 0.392511] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1186worker1 # [ 0.407927] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1187lb # [ 0.419532] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1188worker2 # [ 0.395047] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1189worker1 # [ 0.408184] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1190lb # [ 0.420288] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1191worker2 # [ 0.395926] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1192client # [ 0.394832] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1193client # [ 0.395539] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1194worker1 # [ 0.409709] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1195worker2 # [ 0.396767] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1196client # [ 0.395946] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1197lb # [ 0.422400] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1198worker2 # [ 0.397012] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1199worker1 # [ 0.410989] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1200client # [ 0.397500] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1201worker1 # [ 0.411883] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1202lb # [ 0.424607] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1203worker2 # [ 0.398556] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1204lb # [ 0.425442] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1205worker1 # [ 0.412813] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1206lb # [ 0.426338] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1207worker2 # [ 0.401046] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1208worker1 # [ 0.414222] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1209client # [ 0.400966] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1210lb # [ 0.428017] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1211worker2 # [ 0.401959] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1212client # [ 0.401783] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1213worker1 # [ 0.415131] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1214worker2 # [ 0.402743] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1215client # [ 0.402640] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1216lb # [ 0.429557] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1217worker1 # [ 0.416990] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1218worker2 # [ 0.403587] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1219client # [ 0.403524] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1220lb # [ 0.430433] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1221worker1 # [ 0.418995] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1222lb # [ 0.431323] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1223worker1 # [ 0.419913] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1224client # [ 0.405884] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1225worker2 # [ 0.406041] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1226lb # [ 0.433001] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1227client # [ 0.406543] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1228worker2 # [ 0.406868] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1229worker1 # [ 0.420710] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1230lb # [ 0.434143] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1231worker2 # [ 0.407717] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1232client # [ 0.406948] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1233worker1 # [ 0.422222] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1234lb # [ 0.434753] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1235client # [ 0.407932] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1236worker2 # [ 0.408641] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1237worker1 # [ 0.423160] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1238lb # [ 0.436115] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1239client # [ 0.409453] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1240lb # [ 0.436744] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1241worker1 # [ 0.424517] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1242lb # [ 0.438582] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1243client # [ 0.410977] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1244worker1 # [ 0.427197] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1245worker2 # [ 0.411065] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1246client # [ 0.411941] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1247worker1 # [ 0.428041] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1248worker2 # [ 0.412623] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1249worker1 # [ 0.428905] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1250worker2 # [ 0.413026] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1251client # [ 0.413482] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1252worker2 # [ 0.414010] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1253worker1 # [ 0.429747] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1254worker2 # [ 0.415563] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1255client # [ 0.415925] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1256worker1 # [ 0.431216] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1257client # [ 0.416947] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1258worker2 # [ 0.417064] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1259worker1 # [ 0.432066] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1260worker2 # [ 0.418026] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1261worker1 # [ 0.433200] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1262client # [ 0.418569] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1263worker1 # [ 0.434771] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1264worker2 # [ 0.419564] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1265client # [ 0.419969] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1266lb # [ 0.439796] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1267worker1 # [ 0.435731] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1268client # [ 0.420947] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1269worker2 # [ 0.422048] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1270lb # [ 0.442109] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1271worker1 # [ 0.436439] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1272client # [ 0.422463] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1273worker2 # [ 0.423021] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1274worker1 # [ 0.437715] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1275lb # [ 0.442820] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1276worker2 # [ 0.424622] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1277worker1 # [ 0.438376] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1278lb # [ 0.443846] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1279client # [ 0.424969] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1280lb # [ 0.444824] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1281worker1 # [ 0.439710] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1282client # [ 0.425947] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1283worker2 # [ 0.426019] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1284worker2 # [ 0.427027] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1285lb # [ 0.446370] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1286client # [ 0.427456] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1287lb # [ 0.447190] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1288worker2 # [ 0.429297] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1289lb # [ 0.447879] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1290client # [ 0.429695] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1291client # [ 0.429932] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1292lb # [ 0.449148] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1293worker2 # [ 0.431017] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1294client # [ 0.430904] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1295worker2 # [ 0.432027] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1296client # [ 0.432333] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1297worker2 # [ 0.433767] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1298lb # [ 0.452638] ACPI: PCI: Interrupt link LNKA configured for IRQ 101299lb # [ 0.454040] ACPI: PCI: Interrupt link LNKB configured for IRQ 101300client # [ 0.435570] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1301lb # [ 0.454673] ACPI: PCI: Interrupt link LNKC configured for IRQ 111302worker1 # [ 0.440430] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1303client # [ 0.435932] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1304worker2 # [ 0.436978] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1305lb # [ 0.455675] ACPI: PCI: Interrupt link LNKD configured for IRQ 111306client # [ 0.436896] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1307worker2 # [ 0.438013] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1308lb # [ 0.456658] ACPI: PCI: Interrupt link LNKE configured for IRQ 101309worker1 # [ 0.443746] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1310worker2 # [ 0.438965] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1311lb # [ 0.457647] ACPI: PCI: Interrupt link LNKF configured for IRQ 101312client # [ 0.438353] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1313worker1 # [ 0.444418] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1314lb # [ 0.458648] ACPI: PCI: Interrupt link LNKG configured for IRQ 111315worker2 # [ 0.440387] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1316client # [ 0.439596] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1317lb # [ 0.459652] ACPI: PCI: Interrupt link LNKH configured for IRQ 111318worker1 # [ 0.445471] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1319lb # [ 0.460593] ACPI: PCI: Interrupt link GSIA configured for IRQ 161320lb # [ 0.461565] ACPI: PCI: Interrupt link GSIB configured for IRQ 171321client # [ 0.440126] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1322worker1 # [ 0.446428] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1323worker2 # [ 0.442748] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1324lb # [ 0.462525] ACPI: PCI: Interrupt link GSIC configured for IRQ 181325worker2 # [ 0.443012] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1326client # [ 0.441762] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1327lb # [ 0.463485] ACPI: PCI: Interrupt link GSID configured for IRQ 191328worker2 # [ 0.444027] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1329worker1 # [ 0.448208] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1330lb # [ 0.464327] ACPI: PCI: Interrupt link GSIE configured for IRQ 201331client # [ 0.442778] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1332worker1 # [ 0.449052] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1333lb # [ 0.464563] ACPI: PCI: Interrupt link GSIF configured for IRQ 211334client # [ 0.444663] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1335worker2 # [ 0.445602] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1336lb # [ 0.465568] ACPI: PCI: Interrupt link GSIG configured for IRQ 221337worker1 # [ 0.450242] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1338lb # [ 0.466537] ACPI: PCI: Interrupt link GSIH configured for IRQ 231339worker1 # [ 0.451768] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1340worker2 # [ 0.446860] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1341lb # [ 0.468450] iommu: Default domain type: Translated1342worker2 # [ 0.447871] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1343lb # [ 0.469185] iommu: DMA domain TLB invalidation policy: lazy mode1344worker2 # [ 0.449711] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1345lb # [ 0.469802] ACPI: bus type USB registered1346worker1 # [ 0.455201] ACPI: PCI: Interrupt link LNKA configured for IRQ 101347lb # [ 0.470575] usbcore: registered new interface driver usbfs1348worker2 # [ 0.450224] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1349worker1 # [ 0.456221] ACPI: PCI: Interrupt link LNKB configured for IRQ 101350lb # [ 0.471492] usbcore: registered new interface driver hub1351worker1 # [ 0.457257] ACPI: PCI: Interrupt link LNKC configured for IRQ 111352lb # [ 0.472238] usbcore: registered new device driver usb1353worker2 # [ 0.451699] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1354worker1 # [ 0.458250] ACPI: PCI: Interrupt link LNKD configured for IRQ 111355lb # [ 0.473416] NetLabel: Initializing1356worker1 # [ 0.459289] ACPI: PCI: Interrupt link LNKE configured for IRQ 101357lb # [ 0.473994] NetLabel: domain hash size = 1281358worker1 # [ 0.460281] ACPI: PCI: Interrupt link LNKF configured for IRQ 101359lb # [ 0.474550] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1360worker1 # [ 0.461285] ACPI: PCI: Interrupt link LNKG configured for IRQ 111361lb # [ 0.475514] NetLabel: unlabeled traffic allowed by default1362lb # [ 0.476264] PCI: Using ACPI for IRQ routing1363worker1 # [ 0.462277] ACPI: PCI: Interrupt link LNKH configured for IRQ 111364client # [ 0.445155] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1365worker1 # [ 0.463216] ACPI: PCI: Interrupt link GSIA configured for IRQ 161366client # [ 0.447663] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1367worker1 # [ 0.464164] ACPI: PCI: Interrupt link GSIB configured for IRQ 171368worker1 # [ 0.464919] ACPI: PCI: Interrupt link GSIC configured for IRQ 181369client # [ 0.448995] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1370worker1 # [ 0.465191] ACPI: PCI: Interrupt link GSID configured for IRQ 191371worker1 # [ 0.466146] ACPI: PCI: Interrupt link GSIE configured for IRQ 201372client # [ 0.450249] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1373worker1 # [ 0.466919] ACPI: PCI: Interrupt link GSIF configured for IRQ 211374worker1 # [ 0.467195] ACPI: PCI: Interrupt link GSIG configured for IRQ 221375client # [ 0.451175] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1376worker1 # [ 0.468193] ACPI: PCI: Interrupt link GSIH configured for IRQ 231377worker2 # [ 0.452228] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1378client # [ 0.452994] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1379worker1 # [ 0.470096] iommu: Default domain type: Translated1380client # [ 0.453897] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1381worker1 # [ 0.470745] iommu: DMA domain TLB invalidation policy: lazy mode1382worker1 # [ 0.471450] ACPI: bus type USB registered1383client # [ 0.455002] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1384worker1 # [ 0.472164] usbcore: registered new interface driver usbfs1385worker2 # [ 0.455655] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1386worker1 # [ 0.472870] usbcore: registered new interface driver hub1387client # [ 0.456992] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1388worker1 # [ 0.473188] usbcore: registered new device driver usb1389worker2 # [ 0.456304] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1390worker1 # [ 0.474917] NetLabel: Initializing1391worker1 # [ 0.475184] NetLabel: domain hash size = 1281392worker2 # [ 0.457327] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1393worker1 # [ 0.475916] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1394client # [ 0.460929] ACPI: PCI: Interrupt link LNKA configured for IRQ 101395worker2 # [ 0.458263] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1396worker1 # [ 0.476235] NetLabel: unlabeled traffic allowed by default1397client # [ 0.461978] ACPI: PCI: Interrupt link LNKB configured for IRQ 101398worker1 # [ 0.477193] PCI: Using ACPI for IRQ routing1399client # [ 0.463025] ACPI: PCI: Interrupt link LNKC configured for IRQ 111400worker2 # [ 0.461071] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1401client # [ 0.464025] ACPI: PCI: Interrupt link LNKD configured for IRQ 111402client # [ 0.465024] ACPI: PCI: Interrupt link LNKE configured for IRQ 101403worker2 # [ 0.461980] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1404client # [ 0.466025] ACPI: PCI: Interrupt link LNKF configured for IRQ 101405worker2 # [ 0.463359] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1406client # [ 0.467025] ACPI: PCI: Interrupt link LNKG configured for IRQ 111407client # [ 0.468020] ACPI: PCI: Interrupt link LNKH configured for IRQ 111408worker2 # [ 0.464903] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1409client # [ 0.468958] ACPI: PCI: Interrupt link GSIA configured for IRQ 161410client # [ 0.469937] ACPI: PCI: Interrupt link GSIB configured for IRQ 171411client # [ 0.470940] ACPI: PCI: Interrupt link GSIC configured for IRQ 181412client # [ 0.471939] ACPI: PCI: Interrupt link GSID configured for IRQ 191413worker2 # [ 0.468838] ACPI: PCI: Interrupt link LNKA configured for IRQ 101414client # [ 0.472935] ACPI: PCI: Interrupt link GSIE configured for IRQ 201415worker2 # [ 0.469874] ACPI: PCI: Interrupt link LNKB configured for IRQ 101416client # [ 0.473904] ACPI: PCI: Interrupt link GSIF configured for IRQ 211417worker2 # [ 0.470864] ACPI: PCI: Interrupt link LNKC configured for IRQ 111418client # [ 0.474713] ACPI: PCI: Interrupt link GSIG configured for IRQ 221419worker2 # [ 0.472103] ACPI: PCI: Interrupt link LNKD configured for IRQ 111420client # [ 0.475748] ACPI: PCI: Interrupt link GSIH configured for IRQ 231421worker2 # [ 0.473107] ACPI: PCI: Interrupt link LNKE configured for IRQ 101422client # [ 0.477695] iommu: Default domain type: Translated1423worker2 # [ 0.474102] ACPI: PCI: Interrupt link LNKF configured for IRQ 101424client # [ 0.477932] iommu: DMA domain TLB invalidation policy: lazy mode1425worker2 # [ 0.475113] ACPI: PCI: Interrupt link LNKG configured for IRQ 111426client # [ 0.479173] ACPI: bus type USB registered1427worker2 # [ 0.476103] ACPI: PCI: Interrupt link LNKH configured for IRQ 111428client # [ 0.479942] usbcore: registered new interface driver usbfs1429worker2 # [ 0.477043] ACPI: PCI: Interrupt link GSIA configured for IRQ 161430client # [ 0.480854] usbcore: registered new interface driver hub1431worker2 # [ 0.478021] ACPI: PCI: Interrupt link GSIB configured for IRQ 171432client # [ 0.481579] usbcore: registered new device driver usb1433worker2 # [ 0.479009] ACPI: PCI: Interrupt link GSIC configured for IRQ 181434client # [ 0.482918] NetLabel: Initializing1435worker2 # [ 0.480021] ACPI: PCI: Interrupt link GSID configured for IRQ 191436client # [ 0.483574] NetLabel: domain hash size = 1281437worker2 # [ 0.481017] ACPI: PCI: Interrupt link GSIE configured for IRQ 201438client # [ 0.483925] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1439worker2 # [ 0.482017] ACPI: PCI: Interrupt link GSIF configured for IRQ 211440client # [ 0.484989] NetLabel: unlabeled traffic allowed by default1441worker2 # [ 0.483017] ACPI: PCI: Interrupt link GSIG configured for IRQ 221442lb # [ 0.521329] pci 0000:00:01.0: vgaarb: setting as boot VGA device1443client # [ 0.485939] PCI: Using ACPI for IRQ routing1444worker2 # [ 0.484013] ACPI: PCI: Interrupt link GSIH configured for IRQ 231445lb # [ 0.521546] pci 0000:00:01.0: vgaarb: bridge control possible1446lb # [ 0.521546] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1447worker2 # [ 0.486045] iommu: Default domain type: Translated1448lb # [ 0.521554] vgaarb: loaded1449lb # [ 0.522321] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01450lb # [ 0.522551] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1451worker2 # [ 0.486894] iommu: DMA domain TLB invalidation policy: lazy mode1452worker2 # [ 0.488318] ACPI: bus type USB registered1453worker2 # [ 0.489077] usbcore: registered new interface driver usbfs1454lb # [ 0.526663] clocksource: Switched to clocksource kvm-clock1455worker2 # [ 0.490020] usbcore: registered new interface driver hub1456worker2 # [ 0.490988] usbcore: registered new device driver usb1457worker2 # [ 0.492664] NetLabel: Initializing1458worker2 # [ 0.493007] NetLabel: domain hash size = 1281459lb # [ 0.530756] VFS: Disk quotas dquot_6.6.01460worker2 # [ 0.493745] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1461lb # [ 0.531501] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1462worker2 # [ 0.494068] NetLabel: unlabeled traffic allowed by default1463lb # [ 0.532844] pnp: PnP ACPI init1464worker2 # [ 0.494981] PCI: Using ACPI for IRQ routing1465lb # [ 0.533712] ACPI: IRQ 4 override to edge(!), high(!)1466lb # [ 0.534727] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1467lb # [ 0.536307] pnp: PnP ACPI: found 5 devices1468worker1 # [ 0.521472] pci 0000:00:01.0: vgaarb: setting as boot VGA device1469worker1 # [ 0.522175] pci 0000:00:01.0: vgaarb: bridge control possible1470worker1 # [ 0.522175] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1471worker1 # [ 0.522183] vgaarb: loaded1472worker1 # [ 0.522933] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01473worker1 # [ 0.523180] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1474lb # [ 0.543852] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1475lb # [ 0.545372] clocksource: Switched to clocksource acpi_pm1476lb # [ 0.546377] NET: Registered PF_INET protocol family1477worker1 # [ 0.528270] clocksource: Switched to clocksource kvm-clock1478lb # [ 0.547612] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1479worker1 # [ 0.531866] VFS: Disk quotas dquot_6.6.01480worker1 # [ 0.532567] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1481worker1 # [ 0.533893] pnp: PnP ACPI init1482worker1 # [ 0.534720] ACPI: IRQ 4 override to edge(!), high(!)1483worker1 # [ 0.535666] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1484worker1 # [ 0.537149] pnp: PnP ACPI: found 5 devices1485client # [ 0.530187] pci 0000:00:01.0: vgaarb: setting as boot VGA device1486lb # [ 0.564001] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1487worker1 # [ 0.544713] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1488client # [ 0.530921] pci 0000:00:01.0: vgaarb: bridge control possible1489worker1 # [ 0.546134] clocksource: Switched to clocksource acpi_pm1490lb # [ 0.565533] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1491client # [ 0.530921] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1492worker1 # [ 0.547071] NET: Registered PF_INET protocol family1493client # [ 0.530930] vgaarb: loaded1494lb # [ 0.566818] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1495client # [ 0.531698] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01496worker1 # [ 0.548201] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1497lb # [ 0.568194] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1498client # [ 0.531926] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1499lb # [ 0.569438] TCP: Hash tables configured (established 16384 bind 16384)1500lb # [ 0.570610] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1501lb # [ 0.571932] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1502lb # [ 0.573041] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1503client # [ 0.536010] clocksource: Switched to clocksource kvm-clock1504lb # [ 0.574303] NET: Registered PF_UNIX/PF_LOCAL protocol family1505lb # [ 0.575295] NET: Registered PF_XDP protocol family1506lb # [ 0.576194] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1507client # [ 0.539520] VFS: Disk quotas dquot_6.6.01508lb # [ 0.577253] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1509worker2 # [ 0.539255] pci 0000:00:01.0: vgaarb: setting as boot VGA device1510client # [ 0.540213] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1511lb # [ 0.578283] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1512worker2 # [ 0.540001] pci 0000:00:01.0: vgaarb: bridge control possible1513client # [ 0.541557] pnp: PnP ACPI init1514lb # [ 0.579354] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1515client # [ 0.542342] ACPI: IRQ 4 override to edge(!), high(!)1516worker2 # [ 0.540001] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1517lb # [ 0.580452] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1518worker2 # [ 0.540009] vgaarb: loaded1519client # [ 0.543298] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1520lb # [ 0.581561] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1521worker2 # [ 0.540775] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01522client # [ 0.544860] pnp: PnP ACPI: found 5 devices1523worker2 # [ 0.541005] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1524lb # [ 0.583236] ACPI: \_SB_.GSIA: Enabled at IRQ 161525worker1 # [ 0.564767] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1526lb # [ 0.585394] ACPI: \_SB_.GSIB: Enabled at IRQ 171527worker1 # [ 0.566274] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1528worker2 # [ 0.544131] clocksource: Switched to clocksource kvm-clock1529worker1 # [ 0.567538] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1530lb # [ 0.587422] ACPI: \_SB_.GSIC: Enabled at IRQ 181531worker1 # [ 0.568854] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1532lb # [ 0.589380] ACPI: \_SB_.GSID: Enabled at IRQ 191533worker1 # [ 0.570064] TCP: Hash tables configured (established 16384 bind 16384)1534worker2 # [ 0.547855] VFS: Disk quotas dquot_6.6.01535client # [ 0.552388] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1536worker1 # [ 0.571159] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1537lb # [ 0.591132] PCI: CLS 0 bytes, default 641538worker2 # [ 0.548579] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1539client # [ 0.553879] clocksource: Switched to clocksource acpi_pm1540worker1 # [ 0.572366] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1541lb # [ 0.591976] Trying to unpack rootfs image as initramfs...1542worker2 # [ 0.549901] pnp: PnP ACPI init1543client # [ 0.554834] NET: Registered PF_INET protocol family1544worker1 # [ 0.573405] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1545worker2 # [ 0.550732] ACPI: IRQ 4 override to edge(!), high(!)1546worker1 # [ 0.574568] NET: Registered PF_UNIX/PF_LOCAL protocol family1547client # [ 0.555968] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1548worker1 # [ 0.575520] NET: Registered PF_XDP protocol family1549worker2 # [ 0.551711] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1550worker1 # [ 0.576358] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1551worker2 # [ 0.553253] pnp: PnP ACPI: found 5 devices1552worker1 # [ 0.577341] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1553worker1 # [ 0.578335] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1554lb # [ 0.598175] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1555worker1 # [ 0.579405] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1556worker1 # [ 0.580476] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1557worker1 # [ 0.581544] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1558worker1 # [ 0.583238] ACPI: \_SB_.GSIA: Enabled at IRQ 161559worker2 # [ 0.560911] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1560worker2 # [ 0.562538] clocksource: Switched to clocksource acpi_pm1561worker1 # [ 0.586055] ACPI: \_SB_.GSIB: Enabled at IRQ 171562worker2 # [ 0.563528] NET: Registered PF_INET protocol family1563worker1 # [ 0.588270] ACPI: \_SB_.GSIC: Enabled at IRQ 181564worker2 # [ 0.564746] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1565worker1 # [ 0.590284] ACPI: \_SB_.GSID: Enabled at IRQ 191566client # [ 0.572499] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1567worker1 # [ 0.592197] PCI: CLS 0 bytes, default 641568client # [ 0.574079] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1569worker1 # [ 0.593082] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1570client # [ 0.575413] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1571worker1 # [ 0.594769] Trying to unpack rootfs image as initramfs...1572client # [ 0.576800] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1573client # [ 0.578074] TCP: Hash tables configured (established 16384 bind 16384)1574client # [ 0.579220] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1575client # [ 0.580538] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1576client # [ 0.581662] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1577client # [ 0.582925] NET: Registered PF_UNIX/PF_LOCAL protocol family1578client # [ 0.583905] NET: Registered PF_XDP protocol family1579client # [ 0.584783] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1580client # [ 0.585820] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1581client # [ 0.586870] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1582worker2 # [ 0.581845] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1583client # [ 0.587965] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1584worker2 # [ 0.583397] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1585client # [ 0.589067] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1586worker2 # [ 0.584748] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1587client # [ 0.590181] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1588worker2 # [ 0.586125] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1589client # [ 0.591873] ACPI: \_SB_.GSIA: Enabled at IRQ 161590worker2 # [ 0.587436] TCP: Hash tables configured (established 16384 bind 16384)1591worker2 # [ 0.588683] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1592client # [ 0.594351] ACPI: \_SB_.GSIB: Enabled at IRQ 171593worker2 # [ 0.590000] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1594client # [ 0.596262] ACPI: \_SB_.GSIC: Enabled at IRQ 181595worker2 # [ 0.591148] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1596worker2 # [ 0.592438] NET: Registered PF_UNIX/PF_LOCAL protocol family1597client # [ 0.598162] ACPI: \_SB_.GSID: Enabled at IRQ 191598worker2 # [ 0.593465] NET: Registered PF_XDP protocol family1599worker2 # [ 0.594329] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1600client # [ 0.599828] PCI: CLS 0 bytes, default 641601worker2 # [ 0.595331] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1602lb # [ 0.638768] Initialise system trusted keyrings1603client # [ 0.600689] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1604worker2 # [ 0.596341] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1605client # [ 0.602339] Trying to unpack rootfs image as initramfs...1606worker2 # [ 0.597400] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1607lb # [ 0.640925] workingset: timestamp_bits=40 max_order=19 bucket_order=01608worker2 # [ 0.598521] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1609worker2 # [ 0.599689] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1610worker2 # [ 0.601565] ACPI: \_SB_.GSIA: Enabled at IRQ 161611worker2 # [ 0.604146] ACPI: \_SB_.GSIB: Enabled at IRQ 171612worker2 # [ 0.606175] ACPI: \_SB_.GSIC: Enabled at IRQ 181613worker2 # [ 0.608208] ACPI: \_SB_.GSID: Enabled at IRQ 191614worker2 # [ 0.609985] PCI: CLS 0 bytes, default 641615worker2 # [ 0.610901] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1616worker1 # [ 0.636566] Initialise system trusted keyrings1617worker2 # [ 0.612701] Trying to unpack rootfs image as initramfs...1618worker1 # [ 0.638658] workingset: timestamp_bits=40 max_order=19 bucket_order=01619lb # [ 0.664237] Key type asymmetric registered1620lb # [ 0.664958] Asymmetric key parser 'x509' registered1621lb # [ 0.665809] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1622lb # [ 0.671017] io scheduler mq-deadline registered1623lb # [ 0.673887] io scheduler kyber registered1624lb # [ 0.677052] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1625lb # [ 0.678387] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1626worker1 # [ 0.660628] Key type asymmetric registered1627worker1 # [ 0.661390] Asymmetric key parser 'x509' registered1628worker1 # [ 0.662295] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1629lb # [ 0.683640] Linux agpgart interface v0.1031630client # [ 0.646474] Initialise system trusted keyrings1631lb # [ 0.684453] ACPI: bus type drm_connector registered1632worker1 # [ 0.667747] io scheduler mq-deadline registered1633client # [ 0.649592] workingset: timestamp_bits=40 max_order=19 bucket_order=01634lb # [ 0.687341] usbcore: registered new interface driver usbserial_generic1635worker1 # [ 0.668499] io scheduler kyber registered1636lb # [ 0.688412] usbserial: USB Serial support registered for generic1637worker1 # [ 0.671126] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1638worker1 # [ 0.672432] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1639lb # [ 0.691887] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1640lb # [ 0.691887] Please enable it if your BIOS has the CPPC option.1641lb # [ 0.694190] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1642worker1 # [ 0.678397] Linux agpgart interface v0.1031643lb # [ 0.698018] drop_monitor: Initializing network drop monitor service1644worker1 # [ 0.679222] ACPI: bus type drm_connector registered1645lb # [ 0.699216] NET: Registered PF_INET6 protocol family1646worker2 # [ 0.658251] Initialise system trusted keyrings1647worker1 # [ 0.682011] usbcore: registered new interface driver usbserial_generic1648lb # [ 0.702319] Segment Routing with IPv61649worker1 # [ 0.683138] usbserial: USB Serial support registered for generic1650worker2 # [ 0.661344] workingset: timestamp_bits=40 max_order=19 bucket_order=01651lb # [ 0.704888] In-situ OAM (IOAM) with IPv61652lb # [ 0.705958] IPI shorthand broadcast: enabled1653worker1 # [ 0.687617] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1654client # [ 0.670775] Key type asymmetric registered1655worker1 # [ 0.687617] Please enable it if your BIOS has the CPPC option.1656client # [ 0.671463] Asymmetric key parser 'x509' registered1657worker1 # [ 0.689908] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1658client # [ 0.672329] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1659worker1 # [ 0.692741] drop_monitor: Initializing network drop monitor service1660worker1 # [ 0.693920] NET: Registered PF_INET6 protocol family1661lb # [ 0.714135] sched_clock: Marking stable (581014425, 132854836)->(785380353, -71511092)1662client # [ 0.677652] io scheduler mq-deadline registered1663client # [ 0.678429] io scheduler kyber registered1664worker1 # [ 0.698708] Segment Routing with IPv61665worker1 # [ 0.699399] In-situ OAM (IOAM) with IPv61666lb # [ 0.718963] registered taskstats version 11667client # [ 0.681107] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1668lb # [ 0.719888] Loading compiled-in X.509 certificates1669client # [ 0.682387] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1670worker1 # [ 0.701757] IPI shorthand broadcast: enabled1671worker2 # [ 0.680582] Key type asymmetric registered1672client # [ 0.688319] Linux agpgart interface v0.1031673client # [ 0.689147] ACPI: bus type drm_connector registered1674worker2 # [ 0.685294] Asymmetric key parser 'x509' registered1675worker2 # [ 0.686358] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1676client # [ 0.691959] usbcore: registered new interface driver usbserial_generic1677worker1 # [ 0.710119] sched_clock: Marking stable (575016833, 134581197)->(782683377, -73085347)1678client # [ 0.693030] usbserial: USB Serial support registered for generic1679worker2 # [ 0.689399] io scheduler mq-deadline registered1680worker2 # [ 0.690146] io scheduler kyber registered1681worker1 # [ 0.714695] registered taskstats version 11682client # [ 0.696562] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1683worker1 # [ 0.715715] Loading compiled-in X.509 certificates1684client # [ 0.696562] Please enable it if your BIOS has the CPPC option.1685lb # [ 0.736314] Demotion targets for Node 0: null1686worker2 # [ 0.693861] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1687client # [ 0.698847] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1688worker2 # [ 0.695174] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1689lb # [ 0.739922] Key type .fscrypt registered1690client # [ 0.702689] drop_monitor: Initializing network drop monitor service1691lb # [ 0.740615] Key type fscrypt-provisioning registered1692client # [ 0.703856] NET: Registered PF_INET6 protocol family1693lb # [ 0.741567] ima: No TPM chip found, activating TPM-bypass!1694worker2 # [ 0.700634] Linux agpgart interface v0.1031695lb # [ 0.743883] ima: Allocated hash algorithm: sha11696worker2 # [ 0.701419] ACPI: bus type drm_connector registered1697client # [ 0.706997] Segment Routing with IPv61698lb # [ 0.744708] ima: No architecture policies found1699lb # [ 0.746891] PM: Magic number: 10:881:851700client # [ 0.709568] In-situ OAM (IOAM) with IPv61701worker2 # [ 0.704706] usbcore: registered new interface driver usbserial_generic1702client # [ 0.710693] IPI shorthand broadcast: enabled1703lb # [ 0.748562] RAS: Correctable Errors collector initialized.1704worker2 # [ 0.705838] usbserial: USB Serial support registered for generic1705worker1 # [ 0.732096] Demotion targets for Node 0: null1706worker2 # [ 0.709294] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1707worker2 # [ 0.709294] Please enable it if your BIOS has the CPPC option.1708worker1 # [ 0.735644] Key type .fscrypt registered1709worker2 # [ 0.711659] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1710worker1 # [ 0.736320] Key type fscrypt-provisioning registered1711worker1 # [ 0.737297] ima: No TPM chip found, activating TPM-bypass!1712worker2 # [ 0.714446] drop_monitor: Initializing network drop monitor service1713client # [ 0.719843] sched_clock: Marking stable (586015220, 133513702)->(791615712, -72086790)1714worker2 # [ 0.715698] NET: Registered PF_INET6 protocol family1715worker1 # [ 0.739607] ima: Allocated hash algorithm: sha11716lb # [ 0.759581] clk: Disabling unused clocks1717worker1 # [ 0.740402] ima: No architecture policies found1718client # [ 0.722630] registered taskstats version 11719client # [ 0.723599] Loading compiled-in X.509 certificates1720worker1 # [ 0.742607] PM: Magic number: 10:881:851721lb # [ 0.761884] PM: genpd: Disabling unused power domains1722worker2 # [ 0.720576] Segment Routing with IPv61723worker1 # [ 0.744201] RAS: Correctable Errors collector initialized.1724worker2 # [ 0.721224] In-situ OAM (IOAM) with IPv61725worker2 # [ 0.724672] IPI shorthand broadcast: enabled1726worker1 # [ 0.753293] clk: Disabling unused clocks1727worker2 # [ 0.733021] sched_clock: Marking stable (599015315, 133259743)->(809557322, -77282264)1728worker1 # [ 0.757628] PM: genpd: Disabling unused power domains1729worker2 # [ 0.736496] registered taskstats version 11730client # [ 0.741544] Demotion targets for Node 0: null1731worker2 # [ 0.737471] Loading compiled-in X.509 certificates1732client # [ 0.744572] Key type .fscrypt registered1733client # [ 0.745261] Key type fscrypt-provisioning registered1734client # [ 0.746236] ima: No TPM chip found, activating TPM-bypass!1735client # [ 0.748545] ima: Allocated hash algorithm: sha11736client # [ 0.749344] ima: No architecture policies found1737client # [ 0.751540] PM: Magic number: 10:881:851738client # [ 0.753171] RAS: Correctable Errors collector initialized.1739worker2 # [ 0.755288] Demotion targets for Node 0: null1740worker2 # [ 0.756159] Key type .fscrypt registered1741client # [ 0.764279] clk: Disabling unused clocks1742worker2 # [ 0.759279] Key type fscrypt-provisioning registered1743worker2 # [ 0.760220] ima: No TPM chip found, activating TPM-bypass!1744worker2 # [ 0.761196] ima: Allocated hash algorithm: sha11745client # [ 0.766546] PM: genpd: Disabling unused power domains1746worker2 # [ 0.763307] ima: No architecture policies found1747worker2 # [ 0.766288] PM: Magic number: 10:881:851748worker2 # [ 0.768029] RAS: Correctable Errors collector initialized.1749worker2 # [ 0.777010] clk: Disabling unused clocks1750worker2 # [ 0.779300] PM: genpd: Disabling unused power domains1751worker1 # [ 0.920107] Freeing initrd memory: 29056K1752worker1 # [ 0.923289] Freeing unused decrypted memory: 2028K1753worker1 # [ 0.925965] Freeing unused kernel image (initmem) memory: 3652K1754worker1 # [ 0.927086] Write protecting the kernel read-only data: 32768k1755worker1 # [ 0.928969] Freeing unused kernel image (text/rodata gap) memory: 1184K1756lb # [ 0.949921] Freeing initrd memory: 29064K1757worker1 # [ 0.930502] Freeing unused kernel image (rodata/data gap) memory: 720K1758lb # [ 0.953516] Freeing unused decrypted memory: 2028K1759lb # [ 0.956639] Freeing unused kernel image (initmem) memory: 3652K1760lb # [ 0.957810] Write protecting the kernel read-only data: 32768k1761lb # [ 0.959906] Freeing unused kernel image (text/rodata gap) memory: 1184K1762lb # [ 0.961497] Freeing unused kernel image (rodata/data gap) memory: 720K1763client # [ 0.926816] Freeing initrd memory: 29056K1764client # [ 0.930014] Freeing unused decrypted memory: 2028K1765client # [ 0.932686] Freeing unused kernel image (initmem) memory: 3652K1766client # [ 0.933805] Write protecting the kernel read-only data: 32768k1767client # [ 0.935726] Freeing unused kernel image (text/rodata gap) memory: 1184K1768client # [ 0.937275] Freeing unused kernel image (rodata/data gap) memory: 720K1769worker2 # [ 0.949791] Freeing initrd memory: 29060K1770worker2 # [ 0.953071] Freeing unused decrypted memory: 2028K1771worker2 # [ 0.955867] Freeing unused kernel image (initmem) memory: 3652K1772worker2 # [ 0.956998] Write protecting the kernel read-only data: 32768k1773worker1 # [ 0.981334] x86/mm: Checked W+X mappings: passed, no W+X pages found.1774worker1 # [ 0.982428] Run /init as init process1775worker2 # [ 0.958955] Freeing unused kernel image (text/rodata gap) memory: 1184K1776worker2 # [ 0.960500] Freeing unused kernel image (rodata/data gap) memory: 720K1777worker1 # [ 0.993147] systemd[1]: Inserted module 'autofs4'1778lb # [ 1.012618] x86/mm: Checked W+X mappings: passed, no W+X pages found.1779lb # [ 1.013801] Run /init as init process1780lb # [ 1.025981] systemd[1]: Inserted module 'autofs4'1781client # [ 0.988262] x86/mm: Checked W+X mappings: passed, no W+X pages found.1782client # [ 0.989380] Run /init as init process1783worker1 # [ 1.012848] fuse: init (API version 7.45)1784client # [ 1.000157] systemd[1]: Inserted module 'autofs4'1785worker1 # [ 1.020198] ACPI: \_SB_.GSIG: Enabled at IRQ 221786worker1 # [ 1.022638] ACPI: \_SB_.GSIH: Enabled at IRQ 231787worker1 # [ 1.025665] ACPI: \_SB_.GSIE: Enabled at IRQ 201788lb # [ 1.046110] fuse: init (API version 7.45)1789worker1 # [ 1.027632] ACPI: \_SB_.GSIF: Enabled at IRQ 211790worker1 # [ 1.032519] virtiofs virtio5: discovered new tag: nix-store1791worker1 # [ 1.034091] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1792lb # [ 1.053729] ACPI: \_SB_.GSIG: Enabled at IRQ 221793worker2 # [ 1.011388] x86/mm: Checked W+X mappings: passed, no W+X pages found.1794worker2 # [ 1.012478] Run /init as init process1795lb # [ 1.056598] ACPI: \_SB_.GSIH: Enabled at IRQ 231796client # [ 1.019962] fuse: init (API version 7.45)1797lb # [ 1.060135] ACPI: \_SB_.GSIE: Enabled at IRQ 201798worker1 # [ 1.040705] virtiofs virtio6: discovered new tag: shared1799worker1 # [ 1.042146] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1800lb # [ 1.062378] ACPI: \_SB_.GSIF: Enabled at IRQ 211801worker1 # [ 1.045351] virtiofs virtio7: discovered new tag: xchg1802client # [ 1.027368] ACPI: \_SB_.GSIG: Enabled at IRQ 221803worker2 # [ 1.023533] systemd[1]: Inserted module 'autofs4'1804worker1 # [ 1.046839] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1805client # [ 1.029787] ACPI: \_SB_.GSIH: Enabled at IRQ 231806lb # [ 1.068621] virtiofs virtio5: discovered new tag: nix-store1807lb # [ 1.070276] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1808client # [ 1.032901] ACPI: \_SB_.GSIE: Enabled at IRQ 201809client # [ 1.034917] ACPI: \_SB_.GSIF: Enabled at IRQ 211810client # [ 1.040569] virtiofs virtio5: discovered new tag: nix-store1811lb # [ 1.078562] virtiofs virtio6: discovered new tag: shared1812client # [ 1.042159] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1813lb # [ 1.080165] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1814lb # [ 1.083986] virtiofs virtio7: discovered new tag: xchg1815lb # [ 1.085530] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1816worker2 # [ 1.043418] fuse: init (API version 7.45)1817client # [ 1.048774] virtiofs virtio6: discovered new tag: shared1818worker1 # [ 1.067769] systemd[1]: Successfully made /usr/ read-only.1819client # [ 1.050212] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1820client # [ 1.053704] virtiofs virtio7: discovered new tag: xchg1821client # [ 1.055152] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1822worker2 # [ 1.050975] ACPI: \_SB_.GSIG: Enabled at IRQ 221823worker2 # [ 1.053952] ACPI: \_SB_.GSIH: Enabled at IRQ 231824worker2 # [ 1.057450] ACPI: \_SB_.GSIE: Enabled at IRQ 201825worker2 # [ 1.059699] ACPI: \_SB_.GSIF: Enabled at IRQ 211826lb # [ 1.106924] systemd[1]: Successfully made /usr/ read-only.1827worker2 # [ 1.065884] virtiofs virtio5: discovered new tag: nix-store1828worker2 # [ 1.067527] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1829client # [ 1.076694] systemd[1]: Successfully made /usr/ read-only.1830worker2 # [ 1.075111] virtiofs virtio6: discovered new tag: shared1831worker2 # [ 1.076702] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1832worker2 # [ 1.080285] virtiofs virtio7: discovered new tag: xchg1833worker2 # [ 1.081798] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1834worker2 # [ 1.102019] systemd[1]: Successfully made /usr/ read-only.1835worker1 # [ 1.404534] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1836worker1 # [ 1.416550] systemd[1]: Detected virtualization kvm.1837worker1 # [ 1.418707] systemd[1]: Detected architecture x86-64.1838worker1 # [ 1.420783] systemd[1]: Running in initrd.1839worker1 # [ 1.423260] systemd[1]: Initializing machine ID from random generator.1840lb # [ 1.442601] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1841lb # [ 1.447533] systemd[1]: Detected virtualization kvm.1842worker1 # [ 1.426729] systemd[1]: Hostname set to <worker1>.1843lb # [ 1.448415] systemd[1]: Detected architecture x86-64.1844lb # [ 1.449331] systemd[1]: Running in initrd.1845lb # [ 1.450429] systemd[1]: Initializing machine ID from random generator.1846lb # [ 1.451646] systemd[1]: Hostname set to <lb>.1847client # [ 1.413458] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1848client # [ 1.425475] systemd[1]: Detected virtualization kvm.1849client # [ 1.427639] systemd[1]: Detected architecture x86-64.1850client # [ 1.429774] systemd[1]: Running in initrd.1851client # [ 1.432296] systemd[1]: Initializing machine ID from random generator.1852client # [ 1.435397] systemd[1]: Hostname set to <client>.1853worker2 # [ 1.438798] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1854worker2 # [ 1.450932] systemd[1]: Detected virtualization kvm.1855worker2 # [ 1.453073] systemd[1]: Detected architecture x86-64.1856worker2 # [ 1.455237] systemd[1]: Running in initrd.1857worker2 # [ 1.457739] systemd[1]: Initializing machine ID from random generator.1858worker2 # [ 1.460610] systemd[1]: Hostname set to <worker2>.1859lb # [ 1.662340] systemd[1]: bpf-restrict-fs: LSM BPF program attached1860worker1 # [ 1.655840] systemd[1]: bpf-restrict-fs: LSM BPF program attached1861client # [ 1.657249] systemd[1]: bpf-restrict-fs: LSM BPF program attached1862lb # [ 1.722954] systemd[1]: Queued start job for default target Initrd Default Target.1863lb # [ 1.726340] systemd[1]: Created slice Slice /system/modprobe.1864lb # [ 1.727501] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1865worker1 # [ 1.708500] systemd[1]: Queued start job for default target Initrd Default Target.1866worker2 # [ 1.684498] systemd[1]: bpf-restrict-fs: LSM BPF program attached1867lb # [ 1.728841] systemd[1]: Expecting device /dev/disk/by-label/nixos...1868lb # [ 1.729940] systemd[1]: Reached target Path Units.1869lb # [ 1.730795] systemd[1]: Reached target Slice Units.1870lb # [ 1.731640] systemd[1]: Reached target Swaps.1871lb # [ 1.732420] systemd[1]: Reached target Timer Units.1872worker1 # [ 1.712998] systemd[1]: Created slice Slice /system/modprobe.1873lb # [ 1.733371] systemd[1]: Listening on D-Bus System Message Bus Socket.1874worker1 # [ 1.714248] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1875lb # [ 1.734601] systemd[1]: Listening on Journal Socket (/dev/log).1876worker1 # [ 1.715705] systemd[1]: Expecting device /dev/disk/by-label/nixos...1877lb # [ 1.735723] systemd[1]: Listening on Journal Sockets.1878worker1 # [ 1.716847] systemd[1]: Reached target Path Units.1879lb # [ 1.736688] systemd[1]: Listening on udev Control Socket.1880worker1 # [ 1.717752] systemd[1]: Reached target Slice Units.1881lb # [ 1.737689] systemd[1]: Listening on udev Kernel Socket.1882worker1 # [ 1.718634] systemd[1]: Reached target Swaps.1883lb # [ 1.738616] systemd[1]: Reached target Socket Units.1884worker1 # [ 1.719455] systemd[1]: Reached target Timer Units.1885worker1 # [ 1.720485] systemd[1]: Listening on D-Bus System Message Bus Socket.1886lb # [ 1.740492] systemd[1]: Starting Create List of Static Device Nodes...1887worker1 # [ 1.721767] systemd[1]: Listening on Journal Socket (/dev/log).1888worker1 # [ 1.722956] systemd[1]: Listening on Journal Sockets.1889worker1 # [ 1.723966] systemd[1]: Listening on udev Control Socket.1890worker1 # [ 1.724988] systemd[1]: Listening on udev Kernel Socket.1891lb # [ 1.744941] systemd[1]: Starting Load Kernel Module configfs...1892worker1 # [ 1.725954] systemd[1]: Reached target Socket Units.1893worker1 # [ 1.727846] systemd[1]: Starting Create List of Static Device Nodes...1894client # [ 1.711670] systemd[1]: Queued start job for default target Initrd Default Target.1895worker1 # [ 1.732665] systemd[1]: Starting Load Kernel Module configfs...1896client # [ 1.716032] systemd[1]: Created slice Slice /system/modprobe.1897client # [ 1.717254] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1898client # [ 1.718852] systemd[1]: Expecting device /dev/disk/by-label/nixos...1899client # [ 1.719988] systemd[1]: Reached target Path Units.1900client # [ 1.720928] systemd[1]: Reached target Slice Units.1901client # [ 1.721845] systemd[1]: Reached target Swaps.1902client # [ 1.722663] systemd[1]: Reached target Timer Units.1903client # [ 1.723667] systemd[1]: Listening on D-Bus System Message Bus Socket.1904client # [ 1.724935] systemd[1]: Listening on Journal Socket (/dev/log).1905client # [ 1.726128] systemd[1]: Listening on Journal Sockets.1906client # [ 1.727171] systemd[1]: Listening on udev Control Socket.1907lb # [ 1.754416] systemd[1]: Starting Journal Service...1908client # [ 1.728239] systemd[1]: Listening on udev Kernel Socket.1909client # [ 1.729230] systemd[1]: Reached target Socket Units.1910client # [ 1.731152] systemd[1]: Starting Create List of Static Device Nodes...1911client # [ 1.735603] systemd[1]: Starting Load Kernel Module configfs...1912worker1 # [ 1.742303] systemd[1]: Starting Journal Service...1913lb # [ 1.777948] systemd[1]: Starting Load Kernel Modules...1914worker2 # [ 1.737701] systemd[1]: Queued start job for default target Initrd Default Target.1915lb # [ 1.783947] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1916worker2 # [ 1.741623] systemd[1]: Created slice Slice /system/modprobe.1917worker2 # [ 1.742868] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1918worker2 # [ 1.744326] systemd[1]: Expecting device /dev/disk/by-label/nixos...1919worker2 # [ 1.745478] systemd[1]: Reached target Path Units.1920lb # [ 1.789058] systemd[1]: Starting Coldplug All udev Devices...1921worker2 # [ 1.746391] systemd[1]: Reached target Slice Units.1922worker1 # [ 1.770378] systemd[1]: Starting Load Kernel Modules...1923worker2 # [ 1.747225] systemd[1]: Reached target Swaps.1924worker2 # [ 1.748049] systemd[1]: Reached target Timer Units.1925worker1 # [ 1.771540] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1926worker2 # [ 1.749054] systemd[1]: Listening on D-Bus System Message Bus Socket.1927worker2 # [ 1.750315] systemd[1]: Listening on Journal Socket (/dev/log).1928worker2 # [ 1.751498] systemd[1]: Listening on Journal Sockets.1929client # [ 1.745246] systemd[1]: Starting Journal Service...1930worker2 # [ 1.752490] systemd[1]: Listening on udev Control Socket.1931worker2 # [ 1.753502] systemd[1]: Listening on udev Kernel Socket.1932worker2 # [ 1.754464] systemd[1]: Reached target Socket Units.1933worker2 # [ 1.756240] systemd[1]: Starting Create List of Static Device Nodes...1934lb # [ 1.801747] systemd-journald[66]: Collecting audit messages is disabled.1935worker2 # [ 1.760097] systemd[1]: Starting Load Kernel Module configfs...1936lb # [ 1.804457] systemd[1]: Finished Create List of Static Device Nodes.1937worker1 # [ 1.787701] systemd[1]: Starting Coldplug All udev Devices...1938client # [ 1.771721] systemd[1]: Starting Load Kernel Modules...1939lb # [ 1.809557] systemd[1]: modprobe@configfs.service: Deactivated successfully.1940client # [ 1.775631] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1941lb # [ 1.816295] systemd[1]: Finished Load Kernel Module configfs.1942worker1 # [ 1.800902] systemd-journald[66]: Collecting audit messages is disabled.1943client # [ 1.783808] systemd[1]: Starting Coldplug All udev Devices...1944worker1 # [ 1.802718] systemd[1]: Finished Create List of Static Device Nodes.1945lb # [ 1.822330] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1946worker2 # [ 1.769739] systemd[1]: Starting Journal Service...1947worker1 # [ 1.811282] systemd[1]: modprobe@configfs.service: Deactivated successfully.1948client # [ 1.794205] systemd-journald[66]: Collecting audit messages is disabled.1949client # [ 1.797346] systemd[1]: Finished Create List of Static Device Nodes.1950lb # [ 1.835170] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1951worker2 # [ 1.795459] systemd[1]: Starting Load Kernel Modules...1952worker1 # [ 1.819000] systemd[1]: Finished Load Kernel Module configfs.1953client # [ 1.804314] systemd[1]: modprobe@configfs.service: Deactivated successfully.1954worker2 # [ 1.799389] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1955worker1 # [ 1.823045] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1956lb # [ 1.843445] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1957client # [ 1.811282] systemd[1]: Finished Load Kernel Module configfs.1958worker2 # [ 1.807461] systemd[1]: Starting Coldplug All udev Devices...1959worker1 # [ 1.835798] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1960lb # [ 1.855076] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1961client # [ 1.816986] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1962worker1 # [ 1.839628] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1963worker2 # [ 1.818593] systemd-journald[66]: Collecting audit messages is disabled.1964worker2 # [ 1.820502] systemd[1]: Finished Create List of Static Device Nodes.1965client # [ 1.829614] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1966worker1 # [ 1.850795] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1967worker2 # [ 1.827762] systemd[1]: modprobe@configfs.service: Deactivated successfully.1968lb # [ 1.876518] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1969worker2 # [ 1.835664] systemd[1]: Finished Load Kernel Module configfs.1970client # [ 1.840723] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1971worker2 # [ 1.839691] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1972lb # [ 1.885084] systemd[1]: Starting Create Static Device Nodes in /dev...1973client # [ 1.851744] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1974worker2 # [ 1.852349] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1975lb # [ 1.899004] systemd[1]: Finished Load Kernel Modules.1976worker1 # [ 1.879230] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1977worker2 # [ 1.858053] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1978worker1 # [ 1.889262] systemd[1]: Starting Create Static Device Nodes in /dev...1979lb # [ 1.908969] systemd[1]: Starting Apply Kernel Variables...1980client # [ 1.874156] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1981worker2 # [ 1.869464] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1982worker1 # [ 1.900195] systemd[1]: Finished Load Kernel Modules.1983client # [ 1.883831] systemd[1]: Starting Create Static Device Nodes in /dev...1984lb # [ 1.928214] systemd[1]: Finished Create Static Device Nodes in /dev.1985lb # [ 1.796749] systemd-modules-load[67]: Inserted module 'dm_mod'1986worker1 # [ 1.911681] systemd[1]: Starting Apply Kernel Variables...1987lb # [ 1.800203] systemd-modules-load[67]: Inserted module 'virtio_balloon'1988client # [ 1.896116] systemd[1]: Finished Load Kernel Modules.1989lb # [ 1.801304] systemd-modules-load[67]: Inserted module 'virtio_gpu'1990lb # [ 1.935193] systemd[1]: Started Journal Service.1991worker2 # [ 1.893436] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1992client # [ 1.906622] systemd[1]: Starting Apply Kernel Variables...1993worker2 # [ 1.901460] systemd[1]: Starting Create Static Device Nodes in /dev...1994worker1 # [ 1.788676] systemd-modules-load[67]: Inserted module 'dm_mod'1995lb # [ 1.813079] systemd[1]: Reached target Preparation for Local File Systems.1996worker1 # [ 1.792930] systemd-modules-load[67]: Inserted module 'virtio_balloon'1997worker1 # [ 1.794070] systemd-modules-load[67]: Inserted module 'virtio_gpu'1998lb # [ 1.815969] systemd[1]: Reached target Local File Systems.1999worker1 # [ 1.930275] systemd[1]: Started Journal Service.2000lb # [ 1.819069] systemd[1]: Starting Create System Files and Directories...
loading 2,739 hidden lines…
4740worker1 # [ 40.355700] nix-grpc-daemon[876]: time=2026-09-22T16:04:49.564Z level=INFO msg="Uploading 1 narinfos"4741lb # [ 40.382606] niks3-server[989]: 2026/09/22 16:04:49 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4742lb # [ 40.392445] niks3-server[989]: 2026/09/22 16:04:49 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4743lb # [ 40.395157] niks3-server[989]: 2026/09/22 16:04:49 INFO Completed upload id=64744lb # [ 40.397257] niks3-server[989]: 2026/09/22 16:04:49 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4745worker1 # [ 40.375675] nix-grpc-daemon[876]: time=2026-09-22T16:04:49.584Z level=INFO msg="Upload complete. (211ms)"4746worker1 # [ 40.381589] nix-grpc-daemon[779]: ts=2026-09-22T16:04:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14747worker1 # [ 40.397611] nix-daemon[861]: reaped child process 974, status = succeeded4748client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&2, in 5.25 seconds)4749client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath4750client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath, in 0.13 seconds)4751client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4752client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.01 seconds)4753client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4754client # copying 3 paths...4755client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4756client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4757client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4758client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.17 seconds)4759(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 5.59 seconds)4760subtest: repeat is answered Cached by the scheduler without building4761worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4762worker1 # finding garbage collector roots...4763worker1 # deleting specified paths...4764worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4765worker1 # deleting unused links...4766worker1 # note: hard linking is currently saving 0.0 KiB4767worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.15 seconds)4768worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4769worker2 # finding garbage collector roots...4770worker2 # deleting specified paths...4771worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4772worker2 # deleting unused links...4773worker2 # note: hard linking is currently saving 0.0 KiB4774worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.18 seconds)4775client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&24776worker1 # [ 41.285516] nix-grpc-daemon[779]: ts=2026-09-22T16:04:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:584324777client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4778client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4779worker2 # [ 41.282498] nix-grpc-daemon[786]: ts=2026-09-22T16:04:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343084780worker1 # [ 41.330136] nix-grpc-daemon[779]: ts=2026-09-22T16:04:50Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14781client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&2, in 0.21 seconds)4782client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath4783client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath, in 0.10 seconds)4784lb: must succeed: curl -sf localhost:9901/clusters4785lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4786lb: must succeed: curl -sf localhost:9901/clusters4787lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4788worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4789worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4790(finished: subtest: repeat is answered Cached by the scheduler without building, in 0.82 seconds)4791subtest: inputs already in the cache are not sent to the scheduler4792lb: must succeed: curl -sf localhost:9901/clusters4793lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4794lb: must succeed: curl -sf localhost:9901/clusters4795lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)4796worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4797worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.04 seconds)4798client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b >&24799worker2 # [ 41.818748] nix-grpc-daemon[786]: ts=2026-09-22T16:04:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:343084800client # don't know how to build these paths:4801client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4802worker2 # [ 41.824876] nix-grpc-daemon[786]: ts=2026-09-22T16:04:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343084803worker1 # [ 41.859275] nix-grpc-daemon[779]: ts=2026-09-22T16:04:51Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04804worker1 # [ 41.864466] nix-grpc-daemon[779]: ts=2026-09-22T16:04:51Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14805worker1 # [ 41.868875] nix-grpc-daemon[779]: ts=2026-09-22T16:04:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584324806worker1 # [ 41.880207] nix-grpc-daemon[779]: ts=2026-09-22T16:04:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584324807worker1 # [ 41.892419] nix-grpc-daemon[876]: time=2026-09-22T16:04:51.100Z level=INFO msg="All 2 paths already cached"4808client # copying 1 paths to 'lb:50051'...4809client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4810worker1 # [ 41.899727] nix-grpc-daemon[779]: ts=2026-09-22T16:04:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:584324811worker1 # [ 41.910579] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4812worker1 # [ 42.004142] nix-daemon[861]: accepted connection from pid 1025, user nix-grpc-daemon (trusted)4813worker1 # [ 42.017934] nix-daemon[861]: reaped child process 1028, status = succeeded4814worker1 # [ 42.098659] nix-daemon[861]: accepted connection from pid 1030, user nix-grpc-daemon (trusted)4815worker1 # [ 42.110898] nix-daemon[861]: reaped child process 1033, status = succeeded4816lb # [ 42.137521] niks3-server[989]: 2026/09/22 16:04:51 INFO Received uploads request method=POST path=/api/pending_closures4817worker1 # [ 42.125055] nix-grpc-daemon[876]: time=2026-09-22T16:04:51.334Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4818worker1 # [ 42.126554] nix-grpc-daemon[876]: time=2026-09-22T16:04:51.335Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4819lb # [ 42.178959] niks3-server[989]: 2026/09/22 16:04:51 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4820lb # [ 42.188620] niks3-server[989]: 2026/09/22 16:04:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4821lb # [ 42.198062] niks3-server[989]: 2026/09/22 16:04:51 INFO Signed narinfos id=7 count=14822worker1 # [ 42.181612] nix-grpc-daemon[876]: time=2026-09-22T16:04:51.390Z level=INFO msg="Uploading 1 narinfos"4823lb # [ 42.206338] niks3-server[989]: 2026/09/22 16:04:51 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4824lb # [ 42.219372] niks3-server[989]: 2026/09/22 16:04:51 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4825lb # [ 42.226928] niks3-server[989]: 2026/09/22 16:04:51 INFO Completed upload id=74826worker1 # [ 42.206255] nix-grpc-daemon[876]: time=2026-09-22T16:04:51.415Z level=INFO msg="Upload complete. (284ms)"4827lb # [ 42.229419] niks3-server[989]: 2026/09/22 16:04:51 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4828worker1 # [ 42.211295] nix-grpc-daemon[779]: ts=2026-09-22T16:04:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=0 paths=1 nar_bytes_in=7524829worker1 # [ 42.217319] nix-grpc-daemon[779]: ts=2026-09-22T16:04:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584324830worker1 # [ 42.229459] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4831worker1 # [ 42.238498] nix-daemon[861]: reaped child process 1023, status = succeeded4832client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4833client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4834worker1 # [ 42.246966] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4835client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4836client # farm-top> LOG-farm-top-t1b4837worker1 # [ 43.664702] nix-daemon[861]: reaped child process 1037, status = succeeded4838worker1 # [ 43.737812] nix-daemon[861]: accepted connection from pid 1054, user nix-grpc-daemon (trusted)4839worker1 # [ 43.750411] nix-daemon[861]: reaped child process 1057, status = succeeded4840lb # [ 43.780887] niks3-server[989]: 2026/09/22 16:04:53 INFO Received uploads request method=POST path=/api/pending_closures4841worker1 # [ 43.766222] nix-grpc-daemon[876]: time=2026-09-22T16:04:52.975Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4842worker1 # [ 43.767871] nix-grpc-daemon[876]: time=2026-09-22T16:04:52.976Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4843lb # [ 43.817873] niks3-server[989]: 2026/09/22 16:04:53 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4844lb # [ 43.828370] niks3-server[989]: 2026/09/22 16:04:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4845lb # [ 43.835233] niks3-server[989]: 2026/09/22 16:04:53 INFO Signed narinfos id=8 count=14846worker1 # [ 43.817307] nix-grpc-daemon[876]: time=2026-09-22T16:04:53.026Z level=INFO msg="Uploading 1 narinfos"4847lb # [ 43.842278] niks3-server[989]: 2026/09/22 16:04:53 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4848lb # [ 43.846942] niks3-server[989]: 2026/09/22 16:04:53 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4849lb # [ 43.858444] niks3-server[989]: 2026/09/22 16:04:53 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4850lb # [ 43.862173] niks3-server[989]: 2026/09/22 16:04:53 INFO Completed upload id=84851lb # [ 43.864753] niks3-server[989]: 2026/09/22 16:04:53 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4852worker1 # [ 43.842696] nix-grpc-daemon[876]: time=2026-09-22T16:04:53.051Z level=INFO msg="Upload complete. (186ms)"4853worker1 # [ 43.849247] nix-grpc-daemon[779]: ts=2026-09-22T16:04:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14854worker1 # [ 43.863508] nix-daemon[861]: reaped child process 1035, status = succeeded4855client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.16 seconds)4856client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b outPath4857client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.11 seconds)4858lb: must succeed: curl -sf localhost:9901/clusters4859lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4860lb: must succeed: curl -sf localhost:9901/clusters4861lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4862worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4863worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.04 seconds)4864(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.48 seconds)4865subtest: outputs read back through any worker carry the cache signature4866client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig >&24867client # don't know how to build these paths:4868client # /nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv4869worker2 # [ 44.222980] nix-grpc-daemon[786]: ts=2026-09-22T16:04:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:343084870worker2 # [ 44.230737] nix-grpc-daemon[786]: ts=2026-09-22T16:04:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343084871worker1 # [ 44.270141] nix-grpc-daemon[779]: ts=2026-09-22T16:04:53Z level=debug event=want client=4 drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv cached=04872worker2 # [ 44.250757] nix-grpc-daemon[786]: ts=2026-09-22T16:04:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343084873worker1 # [ 44.274714] nix-grpc-daemon[779]: ts=2026-09-22T16:04:53Z level=debug event=assigned drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv worker=192.168.1.4:50051 clients=14874worker2 # [ 44.263742] nix-grpc-daemon[786]: ts=2026-09-22T16:04:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343084875client # copying 1 paths to 'lb:50051'...4876client # copying path '/nix/store/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv' to 'lb:50051'...4877worker2 # [ 44.275403] nix-grpc-daemon[786]: ts=2026-09-22T16:04:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:343084878worker2 # [ 44.287298] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4879worker2 # [ 44.308486] nix-grpc-daemon[786]: ts=2026-09-22T16:04:53Z level=info event=niks3_push_started pid=8824880worker2 # [ 44.437991] nix-grpc-daemon[882]: time=2026-09-22T16:04:53.666Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""4881worker2 # [ 44.532367] nix-daemon[854]: accepted connection from pid 894, user nix-grpc-daemon (trusted)4882worker2 # [ 44.549685] nix-daemon[854]: reaped child process 897, status = succeeded4883worker2 # [ 44.644059] nix-daemon[854]: accepted connection from pid 899, user nix-grpc-daemon (trusted)4884worker2 # [ 44.659200] nix-daemon[854]: reaped child process 902, status = succeeded4885lb # [ 44.706281] niks3-server[989]: 2026/09/22 16:04:54 INFO Received uploads request method=POST path=/api/pending_closures4886worker2 # [ 44.669192] nix-grpc-daemon[882]: time=2026-09-22T16:04:53.897Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4887worker2 # [ 44.671728] nix-grpc-daemon[882]: time=2026-09-22T16:04:53.899Z level=INFO msg="Uploading lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv (504B)"4888lb # [ 44.728153] niks3-server[989]: 2026/09/22 16:04:54 INFO Registered completed upload object_key=nar/0bddczcnpndyzwmafx9yqfqrkrp1jg9rr5hsixz6rpy99kg84g1g.nar.zst4889lb # [ 44.732334] niks3-server[989]: 2026/09/22 16:04:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4890lb # [ 44.735090] niks3-server[989]: 2026/09/22 16:04:54 INFO Signed narinfos id=9 count=14891worker2 # [ 44.692092] nix-grpc-daemon[882]: time=2026-09-22T16:04:53.920Z level=INFO msg="Uploading 1 narinfos"4892lb # [ 44.739447] niks3-server[989]: 2026/09/22 16:04:54 INFO Registered completed upload object_key=lqi1rvcqjj8q9l279yzgrgaghp394zs5.ls4893lb # [ 44.745217] niks3-server[989]: 2026/09/22 16:04:54 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4894lb # [ 44.750116] niks3-server[989]: 2026/09/22 16:04:54 INFO Completed upload id=94895worker2 # [ 44.708328] nix-grpc-daemon[882]: time=2026-09-22T16:04:53.936Z level=INFO msg="Upload complete. (266ms)"4896lb # [ 44.753762] niks3-server[989]: 2026/09/22 16:04:54 INFO Registered completed upload object_key=lqi1rvcqjj8q9l279yzgrgaghp394zs5.narinfo4897worker2 # [ 44.710822] nix-grpc-daemon[786]: ts=2026-09-22T16:04:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=1 nar_bytes_in=5044898worker2 # [ 44.716494] nix-grpc-daemon[786]: ts=2026-09-22T16:04:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343084899worker2 # [ 44.726074] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4900worker2 # [ 44.727319] nix-daemon[854]: reaped child process 880, status = succeeded4901worker2 # [ 44.732913] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4902client # building '/nix/store/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv' on lb:50051...4903client # farm-a> worker2: building lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv4904client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4905client # farm-a> LOG-farm-a-sig4906worker2 # [ 45.367812] nix-daemon[854]: reaped child process 906, status = succeeded4907worker2 # [ 45.442095] nix-daemon[854]: accepted connection from pid 922, user nix-grpc-daemon (trusted)4908worker2 # [ 45.455427] nix-daemon[854]: reaped child process 925, status = succeeded4909lb # [ 45.505776] niks3-server[989]: 2026/09/22 16:04:54 INFO Received uploads request method=POST path=/api/pending_closures4910worker2 # [ 45.471252] nix-grpc-daemon[882]: time=2026-09-22T16:04:54.699Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4911worker2 # [ 45.475054] nix-grpc-daemon[882]: time=2026-09-22T16:04:54.701Z level=INFO msg="Uploading qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a (128B)"4912lb # [ 45.537623] niks3-server[989]: 2026/09/22 16:04:54 INFO Registered completed upload object_key=log/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv4913lb # [ 45.554145] niks3-server[989]: 2026/09/22 16:04:55 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst4914lb # [ 45.564335] niks3-server[989]: 2026/09/22 16:04:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4915worker2 # [ 45.526513] nix-grpc-daemon[882]: time=2026-09-22T16:04:54.754Z level=INFO msg="Uploading 1 narinfos"4916lb # [ 45.568956] niks3-server[989]: 2026/09/22 16:04:55 INFO Signed narinfos id=10 count=14917lb # [ 45.579965] niks3-server[989]: 2026/09/22 16:04:55 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4918lb # [ 45.586303] niks3-server[989]: 2026/09/22 16:04:55 INFO Completed upload id=104919worker2 # [ 45.543160] nix-grpc-daemon[882]: time=2026-09-22T16:04:54.771Z level=INFO msg="Upload complete. (203ms)"4920lb # [ 45.587614] niks3-server[989]: 2026/09/22 16:04:55 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.ls4921lb # [ 45.590703] niks3-server[989]: 2026/09/22 16:04:55 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.narinfo4922worker2 # [ 45.547871] nix-grpc-daemon[786]: ts=2026-09-22T16:04:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv assign_id=5 outputs=14923worker1 # [ 45.577316] nix-grpc-daemon[779]: ts=2026-09-22T16:04:54Z level=debug event=want client=4 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv cached=04924worker2 # [ 45.560817] nix-daemon[854]: reaped child process 904, status = succeeded4925worker1 # [ 45.581333] nix-grpc-daemon[779]: ts=2026-09-22T16:04:54Z level=debug event=assigned drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv worker=192.168.1.4:50051 clients=14926worker2 # [ 45.598740] nix-grpc-daemon[786]: ts=2026-09-22T16:04:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343084927worker2 # [ 45.611167] nix-grpc-daemon[786]: ts=2026-09-22T16:04:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343084928worker2 # [ 45.624913] nix-grpc-daemon[882]: time=2026-09-22T16:04:54.853Z level=INFO msg="All 1 paths already cached"4929client # copying 1 paths to 'lb:50051'...4930client # copying path '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' to 'lb:50051'...4931worker2 # [ 45.633759] nix-grpc-daemon[786]: ts=2026-09-22T16:04:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:343084932worker2 # [ 45.644159] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4933worker2 # [ 45.747509] nix-daemon[854]: accepted connection from pid 929, user nix-grpc-daemon (trusted)4934worker2 # [ 45.762542] nix-daemon[854]: reaped child process 932, status = succeeded4935worker2 # [ 45.850080] nix-daemon[854]: accepted connection from pid 934, user nix-grpc-daemon (trusted)4936worker2 # [ 45.864645] nix-daemon[854]: reaped child process 937, status = succeeded4937lb # [ 45.914409] niks3-server[989]: 2026/09/22 16:04:55 INFO Received uploads request method=POST path=/api/pending_closures4938worker2 # [ 45.882595] nix-grpc-daemon[882]: time=2026-09-22T16:04:55.110Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4939worker2 # [ 45.884111] nix-grpc-daemon[882]: time=2026-09-22T16:04:55.110Z level=INFO msg="Uploading 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv (624B)"4940lb # [ 45.942725] niks3-server[989]: 2026/09/22 16:04:55 INFO Registered completed upload object_key=nar/1m1mvw8bmb91glp26qj699lxqv98q2iz8wm4ik3miz0xj6vi3mw6.nar.zst4941lb # [ 45.954085] niks3-server[989]: 2026/09/22 16:04:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4942lb # [ 45.958607] niks3-server[989]: 2026/09/22 16:04:55 INFO Signed narinfos id=11 count=14943worker2 # [ 45.918653] nix-grpc-daemon[882]: time=2026-09-22T16:04:55.147Z level=INFO msg="Uploading 1 narinfos"4944lb # [ 45.969469] niks3-server[989]: 2026/09/22 16:04:55 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.ls4945lb # [ 45.977373] niks3-server[989]: 2026/09/22 16:04:55 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4946lb # [ 45.984441] niks3-server[989]: 2026/09/22 16:04:55 INFO Completed upload id=114947worker2 # [ 45.941436] nix-grpc-daemon[882]: time=2026-09-22T16:04:55.170Z level=INFO msg="Upload complete. (285ms)"4948lb # [ 45.990546] niks3-server[989]: 2026/09/22 16:04:55 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.narinfo4949worker2 # [ 45.945857] nix-grpc-daemon[786]: ts=2026-09-22T16:04:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=1 nar_bytes_in=6244950worker2 # [ 45.950752] nix-grpc-daemon[786]: ts=2026-09-22T16:04:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343084951worker2 # [ 45.963388] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4952worker2 # [ 45.965611] nix-daemon[854]: reaped child process 927, status = succeeded4953worker2 # [ 45.970837] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4954client # building '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' on lb:50051...4955client # farm-b> worker2: building 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv4956client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4957client # farm-b> LOG-farm-b-sig4958worker2 # [ 46.787913] nix-daemon[854]: reaped child process 941, status = succeeded4959worker2 # [ 46.867948] nix-daemon[854]: accepted connection from pid 957, user nix-grpc-daemon (trusted)4960worker2 # [ 46.882250] nix-daemon[854]: reaped child process 960, status = succeeded4961lb # [ 46.934415] niks3-server[989]: 2026/09/22 16:04:56 INFO Received uploads request method=POST path=/api/pending_closures4962worker2 # [ 46.898251] nix-grpc-daemon[882]: time=2026-09-22T16:04:56.126Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4963worker2 # [ 46.899733] nix-grpc-daemon[882]: time=2026-09-22T16:04:56.128Z level=INFO msg="Uploading 2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b (176B)"4964lb # [ 46.966276] niks3-server[989]: 2026/09/22 16:04:56 INFO Registered completed upload object_key=log/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv4965lb # [ 46.977667] niks3-server[989]: 2026/09/22 16:04:56 INFO Registered completed upload object_key=nar/0y3xlf15z3z540fgrc9sgnn2rl0aj3w6x9mjqd50hhqgmaxbiivd.nar.zst4966lb # [ 46.992901] niks3-server[989]: 2026/09/22 16:04:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4967lb # [ 46.998447] niks3-server[989]: 2026/09/22 16:04:56 INFO Signed narinfos id=12 count=14968worker2 # [ 46.955225] nix-grpc-daemon[882]: time=2026-09-22T16:04:56.183Z level=INFO msg="Uploading 1 narinfos"4969lb # [ 47.007990] niks3-server[989]: 2026/09/22 16:04:56 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.ls4970lb # [ 47.014162] niks3-server[989]: 2026/09/22 16:04:56 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4971lb # [ 47.018321] niks3-server[989]: 2026/09/22 16:04:56 INFO Completed upload id=124972worker2 # [ 46.975641] nix-grpc-daemon[882]: time=2026-09-22T16:04:56.204Z level=INFO msg="Upload complete. (196ms)"4973lb # [ 47.021762] niks3-server[989]: 2026/09/22 16:04:56 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.narinfo4974worker2 # [ 46.982581] nix-grpc-daemon[786]: ts=2026-09-22T16:04:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=1 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv assign_id=6 outputs=14975worker1 # [ 47.011320] nix-grpc-daemon[779]: ts=2026-09-22T16:04:56Z level=debug event=want client=4 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv cached=04976worker1 # [ 47.015507] nix-grpc-daemon[779]: ts=2026-09-22T16:04:56Z level=debug event=assigned drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv worker=192.168.1.4:50051 clients=14977worker2 # [ 46.998885] nix-daemon[854]: reaped child process 939, status = succeeded4978worker2 # [ 47.032187] nix-grpc-daemon[786]: ts=2026-09-22T16:04:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343084979worker2 # [ 47.044209] nix-grpc-daemon[786]: ts=2026-09-22T16:04:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343084980worker2 # [ 47.057450] nix-grpc-daemon[882]: time=2026-09-22T16:04:56.285Z level=INFO msg="All 2 paths already cached"4981client # copying 1 paths to 'lb:50051'...4982client # copying path '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' to 'lb:50051'...4983worker2 # [ 47.067186] nix-grpc-daemon[786]: ts=2026-09-22T16:04:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:343084984worker2 # [ 47.077138] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4985lb # [ 47.139260] envoy[925]: conn peer=[2001:db8:1::1]:33214 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9464 tx=4345 ms=51184986lb # [ 47.144231] envoy[925]: conn peer=[2001:db8:1::1]:33226 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=584987lb # [ 47.148091] envoy[925]: conn peer=[2001:db8:1::1]:33232 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4139 tx=1876 ms=20264988worker2 # [ 47.175883] nix-daemon[854]: accepted connection from pid 964, user nix-grpc-daemon (trusted)4989worker2 # [ 47.190614] nix-daemon[854]: reaped child process 967, status = succeeded4990worker2 # [ 47.277892] nix-daemon[854]: accepted connection from pid 969, user nix-grpc-daemon (trusted)4991worker2 # [ 47.294198] nix-daemon[854]: reaped child process 972, status = succeeded4992lb # [ 47.343227] niks3-server[989]: 2026/09/22 16:04:56 INFO Received uploads request method=POST path=/api/pending_closures4993worker2 # [ 47.313217] nix-grpc-daemon[882]: time=2026-09-22T16:04:56.541Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4994worker2 # [ 47.315140] nix-grpc-daemon[882]: time=2026-09-22T16:04:56.543Z level=INFO msg="Uploading cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv (752B)"4995lb # [ 47.374875] niks3-server[989]: 2026/09/22 16:04:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4996lb # [ 47.378887] niks3-server[989]: 2026/09/22 16:04:56 INFO Signed narinfos id=13 count=14997worker2 # [ 47.335818] nix-grpc-daemon[882]: time=2026-09-22T16:04:56.564Z level=INFO msg="Uploading 1 narinfos"4998lb # [ 47.384216] niks3-server[989]: 2026/09/22 16:04:56 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4999lb # [ 47.443485] niks3-server[989]: 2026/09/22 16:04:56 INFO Registered completed upload object_key=nar/0w21abkvlzg18nqkl2x4zxs7d0376kbld1n0fkr68vcpqpq1g5zg.nar.zst5000lb # [ 47.452291] niks3-server[989]: 2026/09/22 16:04:56 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.ls5001lb # [ 47.458225] niks3-server[989]: 2026/09/22 16:04:56 INFO Completed upload id=135002worker2 # [ 47.414974] nix-grpc-daemon[882]: time=2026-09-22T16:04:56.643Z level=INFO msg="Upload complete. (327ms)"5003worker2 # [ 47.420111] nix-grpc-daemon[786]: ts=2026-09-22T16:04:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=1 nar_bytes_in=7525004lb # [ 47.467155] niks3-server[989]: 2026/09/22 16:04:56 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.narinfo5005worker2 # [ 47.426322] nix-grpc-daemon[786]: ts=2026-09-22T16:04:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343085006worker2 # [ 47.438135] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5007worker2 # [ 47.449408] nix-daemon[854]: reaped child process 962, status = succeeded5008client # building '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' on lb:50051...5009client # farm-top> worker2: building cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv5010worker2 # [ 47.458997] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5011client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5012client # farm-top> LOG-farm-top-sig5013worker2 # [ 48.507281] nix-daemon[854]: reaped child process 976, status = succeeded5014worker2 # [ 48.588944] nix-daemon[854]: accepted connection from pid 992, user nix-grpc-daemon (trusted)5015worker2 # [ 48.603362] nix-daemon[854]: reaped child process 995, status = succeeded5016lb # [ 48.653407] niks3-server[989]: 2026/09/22 16:04:58 INFO Received uploads request method=POST path=/api/pending_closures5017worker2 # [ 48.614646] nix-grpc-daemon[882]: time=2026-09-22T16:04:57.843Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5018worker2 # [ 48.616126] nix-grpc-daemon[882]: time=2026-09-22T16:04:57.844Z level=INFO msg="Uploading dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top (232B)"5019lb # [ 48.674108] niks3-server[989]: 2026/09/22 16:04:58 INFO Registered completed upload object_key=nar/0h4zvhqjxsvdqli2l6i5ajh5sh40hz57gzchdirnvbisq2aks0k8.nar.zst5020lb # [ 48.680568] niks3-server[989]: 2026/09/22 16:04:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign5021lb # [ 48.682161] niks3-server[989]: 2026/09/22 16:04:58 INFO Signed narinfos id=14 count=15022worker2 # [ 48.640411] nix-grpc-daemon[882]: time=2026-09-22T16:04:57.868Z level=INFO msg="Uploading 1 narinfos"5023lb # [ 48.686931] niks3-server[989]: 2026/09/22 16:04:58 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.ls5024lb # [ 48.689423] niks3-server[989]: 2026/09/22 16:04:58 INFO Registered completed upload object_key=log/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv5025lb # [ 48.693986] niks3-server[989]: 2026/09/22 16:04:58 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete5026lb # [ 48.697041] niks3-server[989]: 2026/09/22 16:04:58 INFO Completed upload id=145027worker2 # [ 48.653960] nix-grpc-daemon[882]: time=2026-09-22T16:04:57.882Z level=INFO msg="Upload complete. (182ms)"5028lb # [ 48.699419] niks3-server[989]: 2026/09/22 16:04:58 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.narinfo5029worker2 # [ 48.656991] nix-grpc-daemon[786]: ts=2026-09-22T16:04:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=1 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv assign_id=7 outputs=15030worker2 # [ 48.664757] nix-daemon[854]: reaped child process 974, status = succeeded5031client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig >&2, in 4.65 seconds)5032client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath5033client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath, in 0.12 seconds)5034client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5035worker1 # [ 48.976253] nix-grpc-daemon[779]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:518445036client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):5037client # /nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a5038client # /nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b5039client # /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5040worker1 # [ 49.020420] nix-grpc-daemon[779]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:518445041worker1 # [ 49.026859] nix-grpc-daemon[779]: copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'http://lb:5751'...5042worker1 # [ 49.040879] nix-grpc-daemon[779]: copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'http://lb:5751'...5043worker1 # [ 49.052518] nix-grpc-daemon[779]: copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'http://lb:5751'...5044client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.22 seconds)5045client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5046worker2 # [ 49.174681] nix-grpc-daemon[786]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:594905047worker2 # [ 49.215727] nix-grpc-daemon[786]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:594905048client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.21 seconds)5049client: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5050client # finding garbage collector roots...5051client # removing stale temporary roots file "/nix/var/nix/temproots/808"5052client # removing stale temporary roots file "/nix/var/nix/temproots/879"5053client # removing stale temporary roots file "/nix/var/nix/temproots/858"5054client # removing stale temporary roots file "/nix/var/nix/temproots/846"5055client # removing stale temporary roots file "/nix/var/nix/temproots/887"5056client # removing stale temporary roots file "/nix/var/nix/temproots/825"5057client # removing stale temporary roots file "/nix/var/nix/temproots/902"5058client # removing stale temporary roots file "/nix/var/nix/temproots/866"5059client # deleting specified paths...5060client # deleting '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top'5061client # deleting unused links...5062client # note: hard linking is currently saving 0.0 KiB5063worker2 # [ 49.523708] nix-grpc-daemon[786]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:343085064worker2 # [ 49.526718] nix-grpc-daemon[786]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343085065worker2 # [ 49.531497] nix-grpc-daemon[882]: time=2026-09-22T16:04:58.760Z level=INFO msg="All 1 paths already cached"5066worker2 # [ 49.534956] nix-grpc-daemon[786]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:343085067worker1 # [ 49.561491] nix-grpc-daemon[779]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:584325068client # copying 3 paths...5069worker1 # [ 49.571404] nix-grpc-daemon[779]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:584325070client # copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'grpc://lb:50051'...5071worker2 # [ 49.562885] nix-grpc-daemon[786]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:343085072worker2 # [ 49.565552] nix-grpc-daemon[786]: ts=2026-09-22T16:04:58Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=1 nar_bytes_out=2325073worker2 # [ 49.568631] nix-grpc-daemon[786]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:343085074worker2 # [ 49.570207] nix-grpc-daemon[786]: ts=2026-09-22T16:04:58Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=1 nar_bytes_out=1765075worker1 # [ 49.595834] nix-grpc-daemon[779]: ts=2026-09-22T16:04:58Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:584325076worker1 # [ 49.600899] nix-grpc-daemon[779]: ts=2026-09-22T16:04:58Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=0 paths=1 nar_bytes_out=1285077client # copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'grpc://lb:50051'...5078client # copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'grpc://lb:50051'...5079client: (finished: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.37 seconds)5080(finished: subtest: outputs read back through any worker carry the cache signature, in 5.57 seconds)5081subtest: two clients wanting the same drv build it once5082client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup5083client # [ 49.681840] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.5084client # Running as unit: dedup1.service; invocation ID: 58b3410743654d09881b45615c7bfbfa5085client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 0.03 seconds)5086client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup5087client # [ 49.806326] nix[955]: don't know how to build these paths:5088client # [ 49.807822] nix[955]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5089worker2 # [ 49.801492] nix-grpc-daemon[786]: ts=2026-09-22T16:04:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:343085090worker1 # [ 49.831722] nix-grpc-daemon[779]: ts=2026-09-22T16:04:59Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05091worker1 # [ 49.837365] nix-grpc-daemon[779]: ts=2026-09-22T16:04:59Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.3:50051 clients=15092worker1 # [ 49.875173] nix-grpc-daemon[779]: ts=2026-09-22T16:04:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325093worker1 # [ 49.885456] nix-grpc-daemon[779]: ts=2026-09-22T16:04:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584325094client # [ 49.878200] nix[955]: copying 1 paths to 'lb:50051'...5095worker1 # [ 49.896354] nix-grpc-daemon[779]: ts=2026-09-22T16:04:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:584325096client # [ 49.880122] nix[955]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...5097worker1 # [ 49.908692] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5098worker1 # [ 50.011986] nix-daemon[861]: accepted connection from pid 1077, user nix-grpc-daemon (trusted)5099worker1 # [ 50.024460] nix-daemon[861]: reaped child process 1080, status = succeeded5100worker1 # [ 50.109398] nix-daemon[861]: accepted connection from pid 1082, user nix-grpc-daemon (trusted)5101worker1 # [ 50.121426] nix-daemon[861]: reaped child process 1085, status = succeeded5102lb # [ 50.151481] niks3-server[989]: 2026/09/22 16:04:59 INFO Received uploads request method=POST path=/api/pending_closures5103worker1 # [ 50.138863] nix-grpc-daemon[876]: time=2026-09-22T16:04:59.347Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5104worker1 # [ 50.140564] nix-grpc-daemon[876]: time=2026-09-22T16:04:59.349Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"5105lb # [ 50.186493] niks3-server[989]: 2026/09/22 16:04:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5106lb # [ 50.193624] niks3-server[989]: 2026/09/22 16:04:59 INFO Signed narinfos id=15 count=15107worker1 # [ 50.175730] nix-grpc-daemon[876]: time=2026-09-22T16:04:59.384Z level=INFO msg="Uploading 1 narinfos"5108lb # [ 50.199825] niks3-server[989]: 2026/09/22 16:04:59 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst5109lb # [ 50.205800] niks3-server[989]: 2026/09/22 16:04:59 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls5110lb # [ 50.219600] niks3-server[989]: 2026/09/22 16:04:59 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5111lb # [ 50.227266] niks3-server[989]: 2026/09/22 16:04:59 INFO Completed upload id=155112worker1 # [ 50.206309] nix-grpc-daemon[876]: time=2026-09-22T16:04:59.415Z level=INFO msg="Upload complete. (286ms)"5113lb # [ 50.229671] niks3-server[989]: 2026/09/22 16:04:59 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo5114worker1 # [ 50.210837] nix-grpc-daemon[779]: ts=2026-09-22T16:04:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=0 paths=1 nar_bytes_in=4325115worker1 # [ 50.216253] nix-grpc-daemon[779]: ts=2026-09-22T16:04:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325116worker1 # [ 50.226107] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5117worker1 # [ 50.233906] nix-daemon[861]: reaped child process 1075, status = succeeded5118worker1 # [ 50.238801] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5119client # [ 50.221194] nix[955]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5120client # [ 50.724587] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.5121client # Running as unit: dedup2.service; invocation ID: c4f0a12854f54af5aa8ea1862fa7d9805122client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 1.04 seconds)5123worker1 # [ 50.889857] nix-grpc-daemon[779]: ts=2026-09-22T16:05:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:584325124client # [ 50.884930] nix[971]: this derivation will be built:5125worker1 # [ 50.903508] nix-grpc-daemon[779]: ts=2026-09-22T16:05:00Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:584325126client # [ 50.892233] nix[971]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5127worker1 # [ 50.916973] nix-grpc-daemon[779]: ts=2026-09-22T16:05:00Z level=debug event=want client=6 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05128client # [ 50.905624] nix[971]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5129worker1 # [ 50.922513] nix-grpc-daemon[779]: ts=2026-09-22T16:05:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325130client: must succeed: sleep 25131client: (finished: must succeed: sleep 2, in 2.03 seconds)5132worker1 # [ 54.058429] nix-daemon[861]: reaped child process 1089, status = succeeded5133client: waiting for success: ! systemctl is-active dedup1 dedup25134worker1 # [ 54.140568] nix-daemon[861]: accepted connection from pid 1121, user nix-grpc-daemon (trusted)5135worker1 # [ 54.156637] nix-daemon[861]: reaped child process 1124, status = succeeded5136lb # [ 54.183889] niks3-server[989]: 2026/09/22 16:05:03 INFO Received uploads request method=POST path=/api/pending_closures5137worker1 # [ 54.172262] nix-grpc-daemon[876]: time=2026-09-22T16:05:03.381Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5138worker1 # [ 54.173991] nix-grpc-daemon[876]: time=2026-09-22T16:05:03.382Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"5139lb # [ 54.224144] niks3-server[989]: 2026/09/22 16:05:03 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5140lb # [ 54.232774] niks3-server[989]: 2026/09/22 16:05:03 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst5141lb # [ 54.245919] niks3-server[989]: 2026/09/22 16:05:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5142lb # [ 54.251129] niks3-server[989]: 2026/09/22 16:05:03 INFO Signed narinfos id=16 count=15143worker1 # [ 54.233247] nix-grpc-daemon[876]: time=2026-09-22T16:05:03.441Z level=INFO msg="Uploading 1 narinfos"5144lb # [ 54.258250] niks3-server[989]: 2026/09/22 16:05:03 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls5145lb # [ 54.267632] niks3-server[989]: 2026/09/22 16:05:03 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5146lb # [ 54.273491] niks3-server[989]: 2026/09/22 16:05:03 INFO Completed upload id=165147worker1 # [ 54.252741] nix-grpc-daemon[876]: time=2026-09-22T16:05:03.461Z level=INFO msg="Upload complete. (204ms)"5148lb # [ 54.275398] niks3-server[989]: 2026/09/22 16:05:03 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo5149worker1 # [ 54.260766] nix-grpc-daemon[779]: ts=2026-09-22T16:05:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15150worker1 # [ 54.268570] nix-grpc-daemon[779]: ts=2026-09-22T16:05:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15151worker1 # [ 54.282702] nix-daemon[861]: reaped child process 1087, status = succeeded5152client # [ 54.280288] systemd[1]: dedup1.service: Deactivated successfully.5153client # [ 54.283442] systemd[1]: dedup1.service: Consumed 117ms CPU time over 4.597s wall clock time, 43.5M memory peak, 7.2K incoming IP traffic, 8.5K outgoing IP traffic.5154client # [ 54.318692] systemd[1]: dedup2.service: Deactivated successfully.5155client # [ 54.319743] systemd[1]: dedup2.service: Consumed 134ms CPU time over 3.594s wall clock time, 35.4M memory peak, 6.6K incoming IP traffic, 6K outgoing IP traffic.5156client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.06 seconds)5157client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }5158client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)5159worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5160worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5161worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5162worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5163worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5164worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)5165worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5166worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5167(finished: subtest: two clients wanting the same drv build it once, in 5.66 seconds)5168subtest: --timeout and --max-silent-time reach the builder5169worker1 # [ 55.484717] nix-grpc-daemon[779]: ts=2026-09-22T16:05:04Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:584325170worker1 # [ 55.495133] nix-grpc-daemon[779]: ts=2026-09-22T16:05:04Z level=debug event=want client=7 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv cached=05171worker1 # [ 55.499183] nix-grpc-daemon[779]: ts=2026-09-22T16:05:04Z level=debug event=assigned drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv worker=192.168.1.4:50051 clients=15172worker2 # [ 55.488131] nix-grpc-daemon[786]: ts=2026-09-22T16:05:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343085173worker2 # [ 55.496458] nix-grpc-daemon[786]: ts=2026-09-22T16:05:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343085174worker2 # [ 55.507615] nix-grpc-daemon[786]: ts=2026-09-22T16:05:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:343085175worker2 # [ 55.519367] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5176worker2 # [ 55.631406] nix-daemon[854]: accepted connection from pid 1020, user nix-grpc-daemon (trusted)5177worker2 # [ 55.647937] nix-daemon[854]: reaped child process 1023, status = succeeded5178worker2 # [ 55.738713] nix-daemon[854]: accepted connection from pid 1025, user nix-grpc-daemon (trusted)5179worker2 # [ 55.751165] nix-daemon[854]: reaped child process 1028, status = succeeded5180lb # [ 55.801780] niks3-server[989]: 2026/09/22 16:05:05 INFO Received uploads request method=POST path=/api/pending_closures5181worker2 # [ 55.763245] nix-grpc-daemon[882]: time=2026-09-22T16:05:04.991Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5182worker2 # [ 55.764706] nix-grpc-daemon[882]: time=2026-09-22T16:05:04.993Z level=INFO msg="Uploading qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv (424B)"5183lb # [ 55.818715] niks3-server[989]: 2026/09/22 16:05:05 INFO Registered completed upload object_key=nar/1s4sciw269ksj1s7ymwz5cy4dw9n7bxv3k8wqw69h694spdh4ha2.nar.zst5184lb # [ 55.822831] niks3-server[989]: 2026/09/22 16:05:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5185lb # [ 55.825230] niks3-server[989]: 2026/09/22 16:05:05 INFO Signed narinfos id=17 count=15186worker2 # [ 55.782405] nix-grpc-daemon[882]: time=2026-09-22T16:05:05.010Z level=INFO msg="Uploading 1 narinfos"5187lb # [ 55.828721] niks3-server[989]: 2026/09/22 16:05:05 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.ls5188lb # [ 55.833121] niks3-server[989]: 2026/09/22 16:05:05 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5189lb # [ 55.836147] niks3-server[989]: 2026/09/22 16:05:05 INFO Completed upload id=175190worker2 # [ 55.793134] nix-grpc-daemon[882]: time=2026-09-22T16:05:05.021Z level=INFO msg="Upload complete. (261ms)"5191worker2 # [ 55.795343] nix-grpc-daemon[786]: ts=2026-09-22T16:05:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=1 nar_bytes_in=4245192lb # [ 55.839858] niks3-server[989]: 2026/09/22 16:05:05 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.narinfo5193worker2 # [ 55.799274] nix-grpc-daemon[786]: ts=2026-09-22T16:05:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343085194worker2 # [ 55.806262] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5195worker2 # [ 55.808608] nix-daemon[854]: reaped child process 1018, status = succeeded5196worker2 # [ 55.813798] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5197lb # [ 57.140434] envoy[925]: conn peer=[2001:db8:1::1]:41812 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9438 tx=4344 ms=44885198lb # [ 57.144221] envoy[925]: conn peer=[2001:db8:1::1]:41826 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2890 ms=1225199lb # [ 57.147738] envoy[925]: conn peer=[2001:db8:1::1]:41856 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=329 ms=675200lb # [ 57.151836] envoy[925]: conn peer=[2001:db8:1::1]:41838 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=410 ms=785201lb # [ 57.155648] envoy[925]: conn peer=[2001:db8:1::1]:41852 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=377 ms=745202lb # [ 57.159714] envoy[925]: conn peer=[2001:db8:1::1]:41868 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2998 tx=1535 ms=44535203lb # [ 57.163743] envoy[925]: conn peer=[2001:db8:1::1]:41880 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1589 tx=1598 ms=34335204worker2 # [ 61.426712] nix-grpc-daemon[786]: ts=2026-09-22T16:05:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=5 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv assign_id=9 outputs=05205worker2 # [ 61.450279] nix-daemon[854]: reaped child process 1032, status = succeeded5206worker2 # [ 61.452151] nix-daemon[854]: reaped child process 1030, status = succeeded5207worker1 # [ 61.647892] nix-grpc-daemon[779]: ts=2026-09-22T16:05:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:584325208worker1 # [ 61.658783] nix-grpc-daemon[779]: ts=2026-09-22T16:05:10Z level=debug event=want client=8 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv cached=05209worker1 # [ 61.660935] nix-grpc-daemon[779]: ts=2026-09-22T16:05:10Z level=debug event=assigned drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv worker=192.168.1.3:50051 clients=15210worker1 # [ 61.663892] nix-grpc-daemon[779]: ts=2026-09-22T16:05:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325211worker1 # [ 61.669506] nix-grpc-daemon[779]: ts=2026-09-22T16:05:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584325212worker1 # [ 61.676355] nix-grpc-daemon[779]: ts=2026-09-22T16:05:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:584325213worker1 # [ 61.681293] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5214worker1 # [ 61.765757] nix-daemon[861]: accepted connection from pid 1141, user nix-grpc-daemon (trusted)5215worker1 # [ 61.781296] nix-daemon[861]: reaped child process 1144, status = succeeded5216worker1 # [ 61.854856] nix-daemon[861]: accepted connection from pid 1146, user nix-grpc-daemon (trusted)5217worker1 # [ 61.867489] nix-daemon[861]: reaped child process 1149, status = succeeded5218lb # [ 61.894552] niks3-server[989]: 2026/09/22 16:05:11 INFO Received uploads request method=POST path=/api/pending_closures5219worker1 # [ 61.882089] nix-grpc-daemon[876]: time=2026-09-22T16:05:11.090Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5220worker1 # [ 61.883590] nix-grpc-daemon[876]: time=2026-09-22T16:05:11.092Z level=INFO msg="Uploading fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv (424B)"5221lb # [ 61.925478] niks3-server[989]: 2026/09/22 16:05:11 INFO Registered completed upload object_key=nar/013y5sy2r6v0qdwv0dzpgvdfy08bds4mfh2qpvbgjazzrg2v71hp.nar.zst5222lb # [ 61.944859] niks3-server[989]: 2026/09/22 16:05:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5223lb # [ 61.950334] niks3-server[989]: 2026/09/22 16:05:11 INFO Signed narinfos id=18 count=15224worker1 # [ 61.932114] nix-grpc-daemon[876]: time=2026-09-22T16:05:11.140Z level=INFO msg="Uploading 1 narinfos"5225lb # [ 61.957143] niks3-server[989]: 2026/09/22 16:05:11 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.ls5226lb # [ 61.968474] niks3-server[989]: 2026/09/22 16:05:11 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5227lb # [ 61.972469] niks3-server[989]: 2026/09/22 16:05:11 INFO Completed upload id=185228worker1 # [ 61.951615] nix-grpc-daemon[876]: time=2026-09-22T16:05:11.160Z level=INFO msg="Upload complete. (265ms)"5229lb # [ 61.976234] niks3-server[989]: 2026/09/22 16:05:11 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.narinfo5230worker1 # [ 61.955747] nix-grpc-daemon[779]: ts=2026-09-22T16:05:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=0 paths=1 nar_bytes_in=4245231worker1 # [ 61.960595] nix-grpc-daemon[779]: ts=2026-09-22T16:05:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325232worker1 # [ 61.970198] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5233worker1 # [ 61.978565] nix-daemon[861]: reaped child process 1139, status = succeeded5234worker1 # [ 61.983485] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5235lb # [ 67.139394] envoy[925]: conn peer=[2001:db8:1::1]:50504 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=960 ms=60185236worker1 # [ 67.999857] nix-grpc-daemon[779]: ts=2026-09-22T16:05:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=6 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv assign_id=10 outputs=05237worker1 # [ 68.028161] nix-daemon[861]: reaped child process 1151, status = succeeded5238worker1 # [ 68.030933] nix-daemon[861]: reaped child process 1153, status = succeeded5239(finished: subtest: --timeout and --max-silent-time reach the builder, in 12.78 seconds)5240subtest: the first client leaving does not fail the second5241client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share5242client # [ 68.117886] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.5243client # Running as unit: share1.service; invocation ID: 4010da72cce34dd7b94fc26037f9f9585244client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 0.03 seconds)5245client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share5246client # [ 68.245523] nix[1043]: don't know how to build these paths:5247client # [ 68.246530] nix[1043]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5248worker1 # [ 68.261647] nix-grpc-daemon[779]: ts=2026-09-22T16:05:17Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:584325249worker1 # [ 68.272495] nix-grpc-daemon[779]: ts=2026-09-22T16:05:17Z level=debug event=want client=9 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05250worker1 # [ 68.277107] nix-grpc-daemon[779]: ts=2026-09-22T16:05:17Z level=debug event=assigned drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv worker=192.168.1.4:50051 clients=15251worker2 # [ 68.255580] nix-grpc-daemon[786]: ts=2026-09-22T16:05:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343085252worker2 # [ 68.267077] nix-grpc-daemon[786]: ts=2026-09-22T16:05:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343085253client # [ 68.281551] nix[1043]: copying 1 paths to 'lb:50051'...5254client # [ 68.282485] nix[1043]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' to 'lb:50051'...5255worker2 # [ 68.277584] nix-grpc-daemon[786]: ts=2026-09-22T16:05:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:343085256worker2 # [ 68.289239] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5257worker2 # [ 68.396556] nix-daemon[854]: accepted connection from pid 1055, user nix-grpc-daemon (trusted)5258worker2 # [ 68.410689] nix-daemon[854]: reaped child process 1058, status = succeeded5259worker2 # [ 68.502153] nix-daemon[854]: accepted connection from pid 1060, user nix-grpc-daemon (trusted)5260worker2 # [ 68.516237] nix-daemon[854]: reaped child process 1063, status = succeeded5261lb # [ 68.569937] niks3-server[989]: 2026/09/22 16:05:18 INFO Received uploads request method=POST path=/api/pending_closures5262worker2 # [ 68.535711] nix-grpc-daemon[882]: time=2026-09-22T16:05:17.764Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5263worker2 # [ 68.537243] nix-grpc-daemon[882]: time=2026-09-22T16:05:17.765Z level=INFO msg="Uploading j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv (432B)"5264lb # [ 68.607437] niks3-server[989]: 2026/09/22 16:05:18 INFO Registered completed upload object_key=nar/0s0b0gfhn36p0aylk1qa64cgw01bi7a8ca93iv4zg2q7wqcjcqhs.nar.zst5265lb # [ 68.617896] niks3-server[989]: 2026/09/22 16:05:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5266worker2 # [ 68.582703] nix-grpc-daemon[882]: time=2026-09-22T16:05:17.811Z level=INFO msg="Uploading 1 narinfos"5267lb # [ 68.625501] niks3-server[989]: 2026/09/22 16:05:18 INFO Signed narinfos id=19 count=15268lb # [ 68.633722] niks3-server[989]: 2026/09/22 16:05:18 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.ls5269lb # [ 68.643046] niks3-server[989]: 2026/09/22 16:05:18 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5270lb # [ 68.649221] niks3-server[989]: 2026/09/22 16:05:18 INFO Completed upload id=195271worker2 # [ 68.606232] nix-grpc-daemon[882]: time=2026-09-22T16:05:17.834Z level=INFO msg="Upload complete. (305ms)"5272lb # [ 68.651814] niks3-server[989]: 2026/09/22 16:05:18 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.narinfo5273worker2 # [ 68.610974] nix-grpc-daemon[786]: ts=2026-09-22T16:05:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=1 nar_bytes_in=4325274worker2 # [ 68.617123] nix-grpc-daemon[786]: ts=2026-09-22T16:05:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343085275worker2 # [ 68.629255] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5276worker2 # [ 68.637733] nix-daemon[854]: reaped child process 1053, status = succeeded5277worker2 # [ 68.643089] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5278client # [ 68.647643] nix[1043]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5279client # [ 69.158724] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.5280client # Running as unit: share2.service; invocation ID: 93e068a24bf844c399b3812175721d0e5281client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 1.04 seconds)5282client: must succeed: sleep 2; systemctl kill -s INT share15283worker2 # [ 69.275057] nix-grpc-daemon[786]: ts=2026-09-22T16:05:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:343085284client # [ 69.289633] nix[1059]: this derivation will be built:5285worker1 # [ 69.308881] nix-grpc-daemon[779]: ts=2026-09-22T16:05:18Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:584325286worker1 # [ 69.320878] nix-grpc-daemon[779]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' from 'http://lb:5751'...5287client # [ 69.317884] nix[1059]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5288worker1 # [ 69.341210] nix-grpc-daemon[779]: ts=2026-09-22T16:05:18Z level=debug event=want client=10 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05289worker2 # [ 69.325961] nix-grpc-daemon[786]: ts=2026-09-22T16:05:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343085290client # [ 69.333799] nix[1059]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5291client # [ 71.289430] systemd[1]: share1.service: Sent signal SIGINT to main process 1043 (nix) on client request.5292client # [ 71.292890] nix[1043]: shutting downj2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5293client # [ 71.295890] nix[1043]: error: interrupted by the user5294client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.04 seconds)5295client: waiting for success: ! systemctl is-active share15296client # [ 71.324825] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE5297client # [ 71.326494] systemd[1]: share1.service: Failed with result 'exit-code'.5298client # [ 71.328910] systemd[1]: share1.service: Consumed 104ms CPU time over 3.210s wall clock time, 43.5M memory peak, 6.1K incoming IP traffic, 8.1K outgoing IP traffic.5299client: (finished: waiting for success: ! systemctl is-active share1, in 1.06 seconds)5300client: waiting for success: ! systemctl is-active share25301worker2 # [ 72.691457] nix-daemon[854]: reaped child process 1067, status = succeeded5302worker2 # [ 72.762095] nix-daemon[854]: accepted connection from pid 1097, user nix-grpc-daemon (trusted)5303worker2 # [ 72.774834] nix-daemon[854]: reaped child process 1100, status = succeeded5304lb # [ 72.825295] niks3-server[989]: 2026/09/22 16:05:22 INFO Received uploads request method=POST path=/api/pending_closures5305worker2 # [ 72.787625] nix-grpc-daemon[882]: time=2026-09-22T16:05:22.016Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5306worker2 # [ 72.789214] nix-grpc-daemon[882]: time=2026-09-22T16:05:22.017Z level=INFO msg="Uploading b9lg6znd0rinl36gy6ixq1dcj6b8bhhw-slow-share (120B)"5307lb # [ 72.849642] niks3-server[989]: 2026/09/22 16:05:22 INFO Registered completed upload object_key=log/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5308lb # [ 72.859240] niks3-server[989]: 2026/09/22 16:05:22 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst5309lb # [ 72.871456] niks3-server[989]: 2026/09/22 16:05:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5310worker2 # [ 72.833361] nix-grpc-daemon[882]: time=2026-09-22T16:05:22.061Z level=INFO msg="Uploading 1 narinfos"5311lb # [ 72.876362] niks3-server[989]: 2026/09/22 16:05:22 INFO Signed narinfos id=20 count=15312lb # [ 72.882073] niks3-server[989]: 2026/09/22 16:05:22 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.ls5313lb # [ 72.884149] niks3-server[989]: 2026/09/22 16:05:22 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5314lb # [ 72.886966] niks3-server[989]: 2026/09/22 16:05:22 INFO Completed upload id=205315worker2 # [ 72.844105] nix-grpc-daemon[882]: time=2026-09-22T16:05:22.072Z level=INFO msg="Upload complete. (191ms)"5316lb # [ 72.889468] niks3-server[989]: 2026/09/22 16:05:22 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.narinfo5317worker2 # [ 72.848362] nix-grpc-daemon[786]: ts=2026-09-22T16:05:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=3 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15318worker2 # [ 72.852150] nix-grpc-daemon[786]: ts=2026-09-22T16:05:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=4 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15319worker2 # [ 72.858972] nix-daemon[854]: reaped child process 1065, status = succeeded5320client # [ 72.924386] systemd[1]: share2.service: Deactivated successfully.5321client # [ 72.928175] systemd[1]: share2.service: Consumed 110ms CPU time over 3.768s wall clock time, 35.4M memory peak, 6.5K incoming IP traffic, 6K outgoing IP traffic.5322client: (finished: waiting for success: ! systemctl is-active share2, in 1.08 seconds)5323client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }5324client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.03 seconds)5325(finished: subtest: the first client leaving does not fail the second, in 5.56 seconds)5326subtest: build hook: nix-daemon with builders = grpc://lb5327client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log5328client # [ 73.747247] systemd[1]: Started Nix Daemon.5329client # [ 73.834322] nix-daemon[1111]: accepted connection from pid 1108, user root (trusted)5330worker2 # [ 73.943596] nix-grpc-daemon[786]: ts=2026-09-22T16:05:23Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:343085331worker2 # [ 73.949066] nix-grpc-daemon[786]: ts=2026-09-22T16:05:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343085332worker1 # [ 73.985602] nix-grpc-daemon[779]: ts=2026-09-22T16:05:23Z level=debug event=want client=11 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05333worker1 # [ 73.989883] nix-grpc-daemon[779]: ts=2026-09-22T16:05:23Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=15334worker1 # [ 73.994997] nix-grpc-daemon[779]: ts=2026-09-22T16:05:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325335worker1 # [ 74.006366] nix-grpc-daemon[779]: ts=2026-09-22T16:05:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584325336worker1 # [ 74.017451] nix-grpc-daemon[779]: ts=2026-09-22T16:05:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:584325337worker1 # [ 74.025899] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5338worker1 # [ 74.124533] nix-daemon[861]: accepted connection from pid 1195, user nix-grpc-daemon (trusted)5339worker1 # [ 74.136464] nix-daemon[861]: reaped child process 1198, status = succeeded5340worker1 # [ 74.219589] nix-daemon[861]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)5341worker1 # [ 74.231435] nix-daemon[861]: reaped child process 1203, status = succeeded5342lb # [ 74.261170] niks3-server[989]: 2026/09/22 16:05:23 INFO Received uploads request method=POST path=/api/pending_closures5343worker1 # [ 74.248437] nix-grpc-daemon[876]: time=2026-09-22T16:05:23.457Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5344worker1 # [ 74.249966] nix-grpc-daemon[876]: time=2026-09-22T16:05:23.459Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5345lb # [ 74.294671] niks3-server[989]: 2026/09/22 16:05:23 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5346lb # [ 74.303410] niks3-server[989]: 2026/09/22 16:05:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5347lb # [ 74.310136] niks3-server[989]: 2026/09/22 16:05:23 INFO Signed narinfos id=21 count=15348worker1 # [ 74.291786] nix-grpc-daemon[876]: time=2026-09-22T16:05:23.500Z level=INFO msg="Uploading 1 narinfos"5349lb # [ 74.316608] niks3-server[989]: 2026/09/22 16:05:23 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5350lb # [ 74.328071] niks3-server[989]: 2026/09/22 16:05:23 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5351lb # [ 74.331606] niks3-server[989]: 2026/09/22 16:05:23 INFO Completed upload id=215352worker1 # [ 74.310732] nix-grpc-daemon[876]: time=2026-09-22T16:05:23.519Z level=INFO msg="Upload complete. (271ms)"5353lb # [ 74.333303] niks3-server[989]: 2026/09/22 16:05:23 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo5354worker1 # [ 74.314949] nix-grpc-daemon[779]: ts=2026-09-22T16:05:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=0 paths=1 nar_bytes_in=5125355worker1 # [ 74.321157] nix-grpc-daemon[779]: ts=2026-09-22T16:05:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325356worker1 # [ 74.329082] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5357worker1 # [ 74.337991] nix-daemon[861]: reaped child process 1193, status = succeeded5358worker1 # [ 74.340713] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5359worker1 # [ 75.357443] nix-daemon[861]: reaped child process 1207, status = succeeded5360worker1 # [ 75.427402] nix-daemon[861]: accepted connection from pid 1223, user nix-grpc-daemon (trusted)5361worker1 # [ 75.439285] nix-daemon[861]: reaped child process 1226, status = succeeded5362lb # [ 75.465921] niks3-server[989]: 2026/09/22 16:05:24 INFO Received uploads request method=POST path=/api/pending_closures5363worker1 # [ 75.454191] nix-grpc-daemon[876]: time=2026-09-22T16:05:24.663Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5364worker1 # [ 75.455886] nix-grpc-daemon[876]: time=2026-09-22T16:05:24.664Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5365lb # [ 75.499789] niks3-server[989]: 2026/09/22 16:05:24 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5366lb # [ 75.514291] niks3-server[989]: 2026/09/22 16:05:24 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5367lb # [ 75.526076] niks3-server[989]: 2026/09/22 16:05:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5368lb # [ 75.530677] niks3-server[989]: 2026/09/22 16:05:24 INFO Signed narinfos id=22 count=15369worker1 # [ 75.512307] nix-grpc-daemon[876]: time=2026-09-22T16:05:24.721Z level=INFO msg="Uploading 1 narinfos"5370lb # [ 75.540313] niks3-server[989]: 2026/09/22 16:05:24 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5371lb # [ 75.545730] niks3-server[989]: 2026/09/22 16:05:24 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5372lb # [ 75.548722] niks3-server[989]: 2026/09/22 16:05:25 INFO Completed upload id=225373worker1 # [ 75.527606] nix-grpc-daemon[876]: time=2026-09-22T16:05:24.736Z level=INFO msg="Upload complete. (180ms)"5374worker1 # [ 75.530170] nix-grpc-daemon[779]: ts=2026-09-22T16:05:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=12 outputs=15375lb # [ 75.553629] niks3-server[989]: 2026/09/22 16:05:25 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5376worker1 # [ 75.537949] nix-daemon[861]: reaped child process 1205, status = succeeded5377worker2 # [ 75.516573] nix-grpc-daemon[786]: ts=2026-09-22T16:05:24Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:343085378worker2 # [ 75.525901] nix-grpc-daemon[786]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5379worker2 # [ 75.550288] nix-grpc-daemon[786]: ts=2026-09-22T16:05:24Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:343085380worker2 # [ 75.555531] nix-grpc-daemon[786]: ts=2026-09-22T16:05:24Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=1 nar_bytes_out=1285381worker1 # [ 75.783655] nix-grpc-daemon[779]: ts=2026-09-22T16:05:24Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:584325382worker2 # [ 75.764931] nix-grpc-daemon[786]: ts=2026-09-22T16:05:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343085383worker2 # [ 75.773746] nix-grpc-daemon[882]: time=2026-09-22T16:05:25.001Z level=INFO msg="All 1 paths already cached"5384worker1 # [ 75.807351] nix-grpc-daemon[779]: ts=2026-09-22T16:05:25Z level=debug event=want client=12 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05385worker1 # [ 75.811556] nix-grpc-daemon[779]: ts=2026-09-22T16:05:25Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=15386worker1 # [ 75.815792] nix-grpc-daemon[779]: ts=2026-09-22T16:05:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325387worker1 # [ 75.824336] nix-grpc-daemon[779]: ts=2026-09-22T16:05:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584325388worker1 # [ 75.832684] nix-grpc-daemon[876]: time=2026-09-22T16:05:25.041Z level=INFO msg="All 1 paths already cached"5389worker1 # [ 75.838183] nix-grpc-daemon[779]: ts=2026-09-22T16:05:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:584325390worker1 # [ 75.844731] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5391worker1 # [ 75.937199] nix-daemon[861]: accepted connection from pid 1232, user nix-grpc-daemon (trusted)5392worker1 # [ 75.949378] nix-daemon[861]: reaped child process 1235, status = succeeded5393worker1 # [ 76.032369] nix-daemon[861]: accepted connection from pid 1237, user nix-grpc-daemon (trusted)5394worker1 # [ 76.044383] nix-daemon[861]: reaped child process 1240, status = succeeded5395lb # [ 76.072817] niks3-server[989]: 2026/09/22 16:05:25 INFO Received uploads request method=POST path=/api/pending_closures5396worker1 # [ 76.056348] nix-grpc-daemon[876]: time=2026-09-22T16:05:25.265Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5397worker1 # [ 76.057797] nix-grpc-daemon[876]: time=2026-09-22T16:05:25.265Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5398lb # [ 76.085901] niks3-server[989]: 2026/09/22 16:05:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5399lb # [ 76.088818] niks3-server[989]: 2026/09/22 16:05:25 INFO Signed narinfos id=23 count=15400worker1 # [ 76.068078] nix-grpc-daemon[876]: time=2026-09-22T16:05:25.277Z level=INFO msg="Uploading 1 narinfos"5401lb # [ 76.096110] niks3-server[989]: 2026/09/22 16:05:25 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5402lb # [ 76.100069] niks3-server[989]: 2026/09/22 16:05:25 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5403lb # [ 76.110827] niks3-server[989]: 2026/09/22 16:05:25 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5404worker1 # [ 76.094452] nix-grpc-daemon[876]: time=2026-09-22T16:05:25.303Z level=INFO msg="Upload complete. (239ms)"5405lb # [ 76.116149] niks3-server[989]: 2026/09/22 16:05:25 INFO Completed upload id=235406worker1 # [ 76.096509] nix-grpc-daemon[779]: ts=2026-09-22T16:05:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=0 paths=1 nar_bytes_in=6245407worker1 # [ 76.099865] nix-grpc-daemon[779]: ts=2026-09-22T16:05:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325408lb # [ 76.121433] niks3-server[989]: 2026/09/22 16:05:25 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5409worker1 # [ 76.106306] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5410worker1 # [ 76.108582] nix-daemon[861]: reaped child process 1230, status = succeeded5411worker1 # [ 76.113590] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5412lb # [ 77.139497] envoy[925]: conn peer=[2001:db8:1::1]:50518 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3008 tx=971 ms=63635413lb # [ 77.143294] envoy[925]: conn peer=[2001:db8:1::1]:57598 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2969 tx=813 ms=30615414lb # [ 77.147008] envoy[925]: conn peer=[2001:db8:1::1]:57608 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1589 tx=1598 ms=36245415lb # [ 77.150760] envoy[925]: conn peer=[2001:db8:1::1]:53162 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3423 tx=1940 ms=16655416lb # [ 77.154320] envoy[925]: conn peer=[2001:db8:1::1]:53176 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=585417worker1 # [ 77.330093] nix-daemon[861]: reaped child process 1244, status = succeeded5418worker1 # [ 77.400155] nix-daemon[861]: accepted connection from pid 1260, user nix-grpc-daemon (trusted)5419worker1 # [ 77.413118] nix-daemon[861]: reaped child process 1263, status = succeeded5420lb # [ 77.439781] niks3-server[989]: 2026/09/22 16:05:26 INFO Received uploads request method=POST path=/api/pending_closures5421worker1 # [ 77.423516] nix-grpc-daemon[876]: time=2026-09-22T16:05:26.632Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5422worker1 # [ 77.425144] nix-grpc-daemon[876]: time=2026-09-22T16:05:26.634Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5423lb # [ 77.464727] niks3-server[989]: 2026/09/22 16:05:26 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5424lb # [ 77.469271] niks3-server[989]: 2026/09/22 16:05:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5425lb # [ 77.472341] niks3-server[989]: 2026/09/22 16:05:26 INFO Signed narinfos id=24 count=15426worker1 # [ 77.451650] nix-grpc-daemon[876]: time=2026-09-22T16:05:26.660Z level=INFO msg="Uploading 1 narinfos"5427lb # [ 77.474645] niks3-server[989]: 2026/09/22 16:05:26 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5428lb # [ 77.481368] niks3-server[989]: 2026/09/22 16:05:26 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5429lb # [ 77.485600] niks3-server[989]: 2026/09/22 16:05:26 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5430lb # [ 77.488814] niks3-server[989]: 2026/09/22 16:05:26 INFO Completed upload id=245431worker1 # [ 77.467682] nix-grpc-daemon[876]: time=2026-09-22T16:05:26.676Z level=INFO msg="Upload complete. (147ms)"5432worker1 # [ 77.470148] nix-grpc-daemon[779]: ts=2026-09-22T16:05:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=13 outputs=15433lb # [ 77.492202] niks3-server[989]: 2026/09/22 16:05:26 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5434worker1 # [ 77.477263] nix-daemon[861]: reaped child process 1242, status = succeeded5435worker1 # [ 77.514285] nix-grpc-daemon[779]: ts=2026-09-22T16:05:26Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:584325436worker1 # [ 77.528843] nix-grpc-daemon[779]: ts=2026-09-22T16:05:26Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:584325437worker1 # [ 77.530973] nix-grpc-daemon[779]: ts=2026-09-22T16:05:26Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=0 paths=1 nar_bytes_out=1765438worker1 # [ 77.732631] nix-grpc-daemon[779]: ts=2026-09-22T16:05:26Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:584325439worker1 # [ 77.735459] nix-grpc-daemon[779]: ts=2026-09-22T16:05:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584325440worker1 # [ 77.737946] nix-grpc-daemon[876]: time=2026-09-22T16:05:26.946Z level=INFO msg="All 2 paths already cached"5441worker1 # [ 77.745682] nix-grpc-daemon[779]: ts=2026-09-22T16:05:26Z level=debug event=want client=13 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05442worker1 # [ 77.747826] nix-grpc-daemon[779]: ts=2026-09-22T16:05:26Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=15443worker1 # [ 77.750399] nix-grpc-daemon[779]: ts=2026-09-22T16:05:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325444worker1 # [ 77.756064] nix-grpc-daemon[779]: ts=2026-09-22T16:05:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584325445worker1 # [ 77.761778] nix-grpc-daemon[876]: time=2026-09-22T16:05:26.970Z level=INFO msg="All 2 paths already cached"5446worker1 # [ 77.765368] nix-grpc-daemon[779]: ts=2026-09-22T16:05:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:584325447worker1 # [ 77.768727] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5448worker1 # [ 77.857299] nix-daemon[861]: accepted connection from pid 1269, user nix-grpc-daemon (trusted)5449worker1 # [ 77.871815] nix-daemon[861]: reaped child process 1272, status = succeeded5450worker1 # [ 77.952518] nix-daemon[861]: accepted connection from pid 1274, user nix-grpc-daemon (trusted)5451worker1 # [ 77.966140] nix-daemon[861]: reaped child process 1277, status = succeeded5452lb # [ 77.992787] niks3-server[989]: 2026/09/22 16:05:27 INFO Received uploads request method=POST path=/api/pending_closures5453worker1 # [ 77.979962] nix-grpc-daemon[876]: time=2026-09-22T16:05:27.189Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5454worker1 # [ 77.981778] nix-grpc-daemon[876]: time=2026-09-22T16:05:27.190Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5455lb # [ 78.021087] niks3-server[989]: 2026/09/22 16:05:27 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5456lb # [ 78.029662] niks3-server[989]: 2026/09/22 16:05:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5457lb # [ 78.034235] niks3-server[989]: 2026/09/22 16:05:27 INFO Signed narinfos id=25 count=15458worker1 # [ 78.016777] nix-grpc-daemon[876]: time=2026-09-22T16:05:27.225Z level=INFO msg="Uploading 1 narinfos"5459lb # [ 78.049748] niks3-server[989]: 2026/09/22 16:05:27 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5460lb # [ 78.058949] niks3-server[989]: 2026/09/22 16:05:27 INFO Completed upload id=255461worker1 # [ 78.041271] nix-grpc-daemon[876]: time=2026-09-22T16:05:27.250Z level=INFO msg="Upload complete. (266ms)"5462lb # [ 78.062746] niks3-server[989]: 2026/09/22 16:05:27 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5463worker1 # [ 78.043701] nix-grpc-daemon[779]: ts=2026-09-22T16:05:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=0 paths=1 nar_bytes_in=7525464lb # [ 78.067886] niks3-server[989]: 2026/09/22 16:05:27 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5465worker1 # [ 78.049864] nix-grpc-daemon[779]: ts=2026-09-22T16:05:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325466worker1 # [ 78.058282] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5467worker1 # [ 78.062919] nix-daemon[861]: reaped child process 1267, status = succeeded5468worker1 # [ 78.067697] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5469worker1 # [ 79.309501] nix-daemon[861]: reaped child process 1281, status = succeeded5470worker1 # [ 79.386053] nix-daemon[861]: accepted connection from pid 1297, user nix-grpc-daemon (trusted)5471worker1 # [ 79.399314] nix-daemon[861]: reaped child process 1300, status = succeeded5472lb # [ 79.424686] niks3-server[989]: 2026/09/22 16:05:28 INFO Received uploads request method=POST path=/api/pending_closures5473worker1 # [ 79.409178] nix-grpc-daemon[876]: time=2026-09-22T16:05:28.618Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5474worker1 # [ 79.410755] nix-grpc-daemon[876]: time=2026-09-22T16:05:28.619Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5475lb # [ 79.443218] niks3-server[989]: 2026/09/22 16:05:28 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5476lb # [ 79.449417] niks3-server[989]: 2026/09/22 16:05:28 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5477lb # [ 79.453909] niks3-server[989]: 2026/09/22 16:05:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5478lb # [ 79.456886] niks3-server[989]: 2026/09/22 16:05:28 INFO Signed narinfos id=26 count=15479worker1 # [ 79.436129] nix-grpc-daemon[876]: time=2026-09-22T16:05:28.645Z level=INFO msg="Uploading 1 narinfos"5480lb # [ 79.459975] niks3-server[989]: 2026/09/22 16:05:28 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5481lb # [ 79.465810] niks3-server[989]: 2026/09/22 16:05:28 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5482lb # [ 79.471843] niks3-server[989]: 2026/09/22 16:05:28 INFO Completed upload id=265483worker1 # [ 79.452182] nix-grpc-daemon[876]: time=2026-09-22T16:05:28.661Z level=INFO msg="Upload complete. (177ms)"5484lb # [ 79.474055] niks3-server[989]: 2026/09/22 16:05:28 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5485worker1 # [ 79.456045] nix-grpc-daemon[779]: ts=2026-09-22T16:05:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=14 outputs=15486worker1 # [ 79.462310] nix-daemon[861]: reaped child process 1279, status = succeeded5487worker1 # [ 79.501219] nix-grpc-daemon[779]: ts=2026-09-22T16:05:28Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:584325488worker1 # [ 79.518349] nix-grpc-daemon[779]: ts=2026-09-22T16:05:28Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:584325489worker1 # [ 79.523353] nix-grpc-daemon[779]: ts=2026-09-22T16:05:28Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=0 paths=1 nar_bytes_out=2325490client # [ 79.614318] nix-daemon[1111]: reaped child process 1113, status = succeeded5491client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log, in 5.97 seconds)5492client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5493client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.02 seconds)5494client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5495client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.02 seconds)5496client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5497client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.02 seconds)5498(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 6.03 seconds)5499subtest: an input only one worker has still reaches the builder5500client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only5501client # this derivation will be built:5502client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5503client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5504client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only, in 0.23 seconds)5505client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5506client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.09 seconds)5507worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5508worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.10 seconds)5509worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5510worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)5511client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&25512client # [ 80.239541] nix-daemon[1111]: accepted connection from pid 1194, user root (trusted)5513client # this derivation will be built:5514client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5515worker1 # [ 80.367757] nix-grpc-daemon[779]: ts=2026-09-22T16:05:29Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:584325516worker1 # [ 80.370179] nix-grpc-daemon[779]: ts=2026-09-22T16:05:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584325517client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5518worker1 # [ 80.451329] nix-daemon[861]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5519worker1 # [ 80.463707] nix-daemon[861]: reaped child process 1310, status = succeeded5520lb # [ 80.492420] niks3-server[989]: 2026/09/22 16:05:29 INFO Received uploads request method=POST path=/api/pending_closures5521worker1 # [ 80.479639] nix-grpc-daemon[876]: time=2026-09-22T16:05:29.688Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5522worker1 # [ 80.481409] nix-grpc-daemon[876]: time=2026-09-22T16:05:29.690Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5523lb # [ 80.523284] niks3-server[989]: 2026/09/22 16:05:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5524lb # [ 80.531158] niks3-server[989]: 2026/09/22 16:05:29 INFO Signed narinfos id=27 count=15525worker1 # [ 80.513272] nix-grpc-daemon[876]: time=2026-09-22T16:05:29.722Z level=INFO msg="Uploading 1 narinfos"5526lb # [ 80.539432] niks3-server[989]: 2026/09/22 16:05:29 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5527lb # [ 80.544824] niks3-server[989]: 2026/09/22 16:05:29 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5528lb # [ 80.555744] niks3-server[989]: 2026/09/22 16:05:30 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5529lb # [ 80.561217] niks3-server[989]: 2026/09/22 16:05:30 INFO Completed upload id=275530worker1 # [ 80.541463] nix-grpc-daemon[876]: time=2026-09-22T16:05:29.750Z level=INFO msg="Upload complete. (169ms)"5531lb # [ 80.563902] niks3-server[989]: 2026/09/22 16:05:30 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5532worker1 # [ 80.555347] nix-grpc-daemon[779]: ts=2026-09-22T16:05:29Z level=debug event=want client=14 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05533worker1 # [ 80.557398] nix-grpc-daemon[779]: ts=2026-09-22T16:05:29Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=15534worker2 # [ 80.537808] nix-grpc-daemon[786]: ts=2026-09-22T16:05:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343085535worker2 # [ 80.544163] nix-grpc-daemon[786]: ts=2026-09-22T16:05:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343085536client # copying 2 paths to 'lb:50051'...5537worker2 # [ 80.551793] nix-grpc-daemon[786]: ts=2026-09-22T16:05:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:343085538client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5539client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5540worker2 # [ 80.562764] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5541worker2 # [ 80.658943] nix-daemon[854]: accepted connection from pid 1108, user nix-grpc-daemon (trusted)5542worker2 # [ 80.673915] nix-daemon[854]: reaped child process 1111, status = succeeded5543worker2 # [ 80.764719] nix-daemon[854]: accepted connection from pid 1113, user nix-grpc-daemon (trusted)5544worker2 # [ 80.779276] nix-daemon[854]: reaped child process 1116, status = succeeded5545lb # [ 80.827741] niks3-server[989]: 2026/09/22 16:05:30 INFO Received uploads request method=POST path=/api/pending_closures5546lb # [ 80.837587] niks3-server[989]: 2026/09/22 16:05:30 INFO Received uploads request method=POST path=/api/pending_closures5547worker2 # [ 80.803428] nix-grpc-daemon[882]: time=2026-09-22T16:05:30.031Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5548worker2 # [ 80.804847] nix-grpc-daemon[882]: time=2026-09-22T16:05:30.031Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5549worker2 # [ 80.806464] nix-grpc-daemon[882]: time=2026-09-22T16:05:30.031Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5550lb # [ 80.869484] niks3-server[989]: 2026/09/22 16:05:30 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5551lb # [ 80.882761] niks3-server[989]: 2026/09/22 16:05:30 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5552lb # [ 80.893233] niks3-server[989]: 2026/09/22 16:05:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5553lb # [ 80.896403] niks3-server[989]: 2026/09/22 16:05:30 INFO Signed narinfos id=28 count=15554lb # [ 80.897774] niks3-server[989]: 2026/09/22 16:05:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5555lb # [ 80.899743] niks3-server[989]: 2026/09/22 16:05:30 INFO Signed narinfos id=29 count=25556worker2 # [ 80.858909] nix-grpc-daemon[882]: time=2026-09-22T16:05:30.086Z level=INFO msg="Uploading 3 narinfos"5557lb # [ 80.903821] niks3-server[989]: 2026/09/22 16:05:30 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5558lb # [ 80.906251] niks3-server[989]: 2026/09/22 16:05:30 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5559lb # [ 80.919512] niks3-server[989]: 2026/09/22 16:05:30 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5560lb # [ 80.922530] niks3-server[989]: 2026/09/22 16:05:30 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5561lb # [ 80.925146] niks3-server[989]: 2026/09/22 16:05:30 INFO Completed upload id=285562lb # [ 80.927119] niks3-server[989]: 2026/09/22 16:05:30 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5563lb # [ 80.929367] niks3-server[989]: 2026/09/22 16:05:30 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5564lb # [ 80.931765] niks3-server[989]: 2026/09/22 16:05:30 INFO Completed upload id=295565worker2 # [ 80.888717] nix-grpc-daemon[882]: time=2026-09-22T16:05:30.117Z level=INFO msg="Upload complete. (320ms)"5566lb # [ 80.933222] niks3-server[989]: 2026/09/22 16:05:30 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5567worker2 # [ 80.891347] nix-grpc-daemon[786]: ts=2026-09-22T16:05:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=2 nar_bytes_in=9765568worker2 # [ 80.894541] nix-grpc-daemon[786]: ts=2026-09-22T16:05:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343085569worker2 # [ 80.902544] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5570worker2 # [ 80.904969] nix-daemon[854]: reaped child process 1106, status = succeeded5571worker2 # [ 80.911849] nix-grpc-daemon[786]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5572client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5573client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5574worker2 # [ 80.919520] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5575client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5576worker2 # [ 81.973601] nix-daemon[854]: reaped child process 1121, status = succeeded5577worker2 # [ 82.036567] nix-daemon[854]: accepted connection from pid 1137, user nix-grpc-daemon (trusted)5578worker2 # [ 82.053861] nix-daemon[854]: reaped child process 1140, status = succeeded5579lb # [ 82.098204] niks3-server[989]: 2026/09/22 16:05:31 INFO Received uploads request method=POST path=/api/pending_closures5580worker2 # [ 82.059244] nix-grpc-daemon[882]: time=2026-09-22T16:05:31.287Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5581lb # [ 82.112152] niks3-server[989]: 2026/09/22 16:05:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5582lb # [ 82.114988] niks3-server[989]: 2026/09/22 16:05:31 INFO Signed narinfos id=30 count=15583worker2 # [ 82.072131] nix-grpc-daemon[882]: time=2026-09-22T16:05:31.300Z level=INFO msg="Uploading 1 narinfos"5584lb # [ 82.122651] niks3-server[989]: 2026/09/22 16:05:31 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5585lb # [ 82.130825] niks3-server[989]: 2026/09/22 16:05:31 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5586lb # [ 82.138509] niks3-server[989]: 2026/09/22 16:05:31 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5587lb # [ 82.141779] niks3-server[989]: 2026/09/22 16:05:31 INFO Completed upload id=305588worker2 # [ 82.100024] nix-grpc-daemon[882]: time=2026-09-22T16:05:31.328Z level=INFO msg="Upload complete. (172ms)"5589lb # [ 82.146585] niks3-server[989]: 2026/09/22 16:05:31 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5590worker2 # [ 82.102893] nix-grpc-daemon[786]: ts=2026-09-22T16:05:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=15 outputs=15591client # copying 1 paths...5592worker2 # [ 82.110566] nix-daemon[854]: reaped child process 1118, status = succeeded5593worker1 # [ 82.132871] nix-grpc-daemon[779]: ts=2026-09-22T16:05:31Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:584325594worker1 # [ 82.142118] nix-grpc-daemon[779]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5595client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5596worker1 # [ 82.166387] nix-grpc-daemon[779]: ts=2026-09-22T16:05:31Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:584325597worker1 # [ 82.172135] nix-grpc-daemon[779]: ts=2026-09-22T16:05:31Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=0 paths=1 nar_bytes_out=1205598client # [ 82.263333] nix-daemon[1111]: reaped child process 1197, status = succeeded5599client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.14 seconds)5600client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&25601client # [ 82.363348] nix-daemon[1111]: accepted connection from pid 1216, user root (trusted)5602client # this derivation will be built:5603client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5604worker2 # [ 82.471192] nix-grpc-daemon[786]: ts=2026-09-22T16:05:31Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:343085605client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5606worker1 # [ 82.501681] nix-grpc-daemon[779]: ts=2026-09-22T16:05:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584325607worker1 # [ 82.511361] nix-grpc-daemon[876]: time=2026-09-22T16:05:31.719Z level=INFO msg="All 1 paths already cached"5608worker1 # [ 82.524980] nix-grpc-daemon[779]: ts=2026-09-22T16:05:31Z level=debug event=want client=15 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05609worker1 # [ 82.531430] nix-grpc-daemon[779]: ts=2026-09-22T16:05:31Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15610worker1 # [ 82.536471] nix-grpc-daemon[779]: ts=2026-09-22T16:05:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325611worker1 # [ 82.544132] nix-grpc-daemon[779]: ts=2026-09-22T16:05:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584325612client # copying 1 paths to 'lb:50051'...5613client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5614worker1 # [ 82.562600] nix-grpc-daemon[779]: ts=2026-09-22T16:05:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:584325615worker1 # [ 82.572618] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5616worker1 # [ 82.586471] nix-grpc-daemon[779]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 36 ms (attempt 1/5)5617worker1 # [ 82.628470] nix-grpc-daemon[779]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 75 ms (attempt 2/5)5618worker1 # [ 82.705454] nix-grpc-daemon[779]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 377 ms (attempt 3/5)5619worker1 # [ 83.084460] nix-grpc-daemon[779]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 703 ms (attempt 4/5)5620worker1 # [ 83.789107] nix-grpc-daemon[779]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5621worker1 # [ 83.790916] nix-grpc-daemon[779]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5622worker1 # [ 83.793122] nix-grpc-daemon[779]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5623worker1 # [ 83.884739] nix-daemon[861]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)5624worker1 # [ 83.900648] nix-daemon[861]: reaped child process 1327, status = succeeded5625worker1 # [ 83.981957] nix-daemon[861]: accepted connection from pid 1329, user nix-grpc-daemon (trusted)5626worker1 # [ 83.997232] nix-daemon[861]: reaped child process 1332, status = succeeded5627lb # [ 84.024795] niks3-server[989]: 2026/09/22 16:05:33 INFO Received uploads request method=POST path=/api/pending_closures5628worker1 # [ 84.013170] nix-grpc-daemon[876]: time=2026-09-22T16:05:33.221Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5629worker1 # [ 84.014519] nix-grpc-daemon[876]: time=2026-09-22T16:05:33.221Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5630lb # [ 84.055155] niks3-server[989]: 2026/09/22 16:05:33 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5631lb # [ 84.067210] niks3-server[989]: 2026/09/22 16:05:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5632worker1 # [ 84.051862] nix-grpc-daemon[876]: time=2026-09-22T16:05:33.260Z level=INFO msg="Uploading 1 narinfos"5633lb # [ 84.073197] niks3-server[989]: 2026/09/22 16:05:33 INFO Signed narinfos id=31 count=15634lb # [ 84.078428] niks3-server[989]: 2026/09/22 16:05:33 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5635lb # [ 84.092347] niks3-server[989]: 2026/09/22 16:05:33 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5636lb # [ 84.099672] niks3-server[989]: 2026/09/22 16:05:33 INFO Completed upload id=315637worker1 # [ 84.080976] nix-grpc-daemon[876]: time=2026-09-22T16:05:33.290Z level=INFO msg="Upload complete. (276ms)"5638lb # [ 84.103427] niks3-server[989]: 2026/09/22 16:05:33 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5639worker1 # [ 84.083323] nix-grpc-daemon[779]: ts=2026-09-22T16:05:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=1 paths=1 nar_bytes_in=5605640worker1 # [ 84.087572] nix-grpc-daemon[779]: ts=2026-09-22T16:05:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:584325641worker1 # [ 84.092919] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5642worker1 # [ 84.097158] nix-daemon[861]: reaped child process 1317, status = succeeded5643worker1 # [ 84.101054] nix-daemon[861]: accepted connection from pid 779, user nix-grpc-daemon (trusted)5644client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5645client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5646client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5647worker1 # [ 84.919532] nix-daemon[861]: reaped child process 1336, status = succeeded5648worker1 # [ 84.984925] nix-daemon[861]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)5649worker1 # [ 84.997764] nix-daemon[861]: reaped child process 1355, status = succeeded5650lb # [ 85.023348] niks3-server[989]: 2026/09/22 16:05:34 INFO Received uploads request method=POST path=/api/pending_closures5651worker1 # [ 85.012489] nix-grpc-daemon[876]: time=2026-09-22T16:05:34.221Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5652lb # [ 85.054370] niks3-server[989]: 2026/09/22 16:05:34 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5653lb # [ 85.064088] niks3-server[989]: 2026/09/22 16:05:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5654lb # [ 85.069227] niks3-server[989]: 2026/09/22 16:05:34 INFO Signed narinfos id=32 count=15655worker1 # [ 85.051096] nix-grpc-daemon[876]: time=2026-09-22T16:05:34.259Z level=INFO msg="Uploading 1 narinfos"5656lb # [ 85.075585] niks3-server[989]: 2026/09/22 16:05:34 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5657lb # [ 85.084459] niks3-server[989]: 2026/09/22 16:05:34 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5658worker1 # [ 85.066805] nix-grpc-daemon[876]: time=2026-09-22T16:05:34.275Z level=INFO msg="Upload complete. (159ms)"5659lb # [ 85.087700] niks3-server[989]: 2026/09/22 16:05:34 INFO Completed upload id=325660worker1 # [ 85.069875] nix-grpc-daemon[779]: ts=2026-09-22T16:05:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58432 duration_s=0 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=16 outputs=15661lb # [ 85.092137] niks3-server[989]: 2026/09/22 16:05:34 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5662worker1 # [ 85.076578] nix-daemon[861]: reaped child process 1334, status = succeeded5663client # copying 1 paths...5664worker1 # [ 85.117156] nix-grpc-daemon[779]: ts=2026-09-22T16:05:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:584325665client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5666worker2 # [ 85.115993] nix-grpc-daemon[786]: ts=2026-09-22T16:05:34Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:343085667worker2 # [ 85.128530] nix-grpc-daemon[786]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5668worker2 # [ 85.143513] nix-grpc-daemon[786]: ts=2026-09-22T16:05:34Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=1 nar_bytes_out=1205669client # [ 85.256182] nix-daemon[1111]: reaped child process 1219, status = succeeded5670client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.99 seconds)5671client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&25672client # [ 85.358481] nix-daemon[1111]: accepted connection from pid 1238, user root (trusted)5673client # this derivation will be built:5674client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5675worker2 # [ 85.463272] nix-grpc-daemon[786]: ts=2026-09-22T16:05:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:343085676client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5677worker1 # [ 85.487810] nix-grpc-daemon[779]: ts=2026-09-22T16:05:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:584325678worker1 # [ 85.495548] nix-grpc-daemon[876]: time=2026-09-22T16:05:34.704Z level=INFO msg="All 1 paths already cached"5679worker1 # [ 85.509178] nix-grpc-daemon[779]: ts=2026-09-22T16:05:34Z level=debug event=want client=16 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05680worker1 # [ 85.513680] nix-grpc-daemon[779]: ts=2026-09-22T16:05:34Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15681worker2 # [ 85.531518] nix-grpc-daemon[786]: ts=2026-09-22T16:05:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343085682worker2 # [ 85.544990] nix-grpc-daemon[786]: ts=2026-09-22T16:05:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:343085683client # copying 1 paths to 'lb:50051'...5684worker2 # [ 85.559760] nix-grpc-daemon[882]: time=2026-09-22T16:05:34.787Z level=INFO msg="All 1 paths already cached"5685client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5686worker2 # [ 85.565874] nix-grpc-daemon[786]: ts=2026-09-22T16:05:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:343085687worker2 # [ 85.577943] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5688worker2 # [ 85.685205] nix-daemon[854]: accepted connection from pid 1145, user nix-grpc-daemon (trusted)5689worker2 # [ 85.702421] nix-daemon[854]: reaped child process 1148, status = succeeded5690worker2 # [ 85.784664] nix-daemon[854]: accepted connection from pid 1150, user nix-grpc-daemon (trusted)5691worker2 # [ 85.801311] nix-daemon[854]: reaped child process 1153, status = succeeded5692lb # [ 85.847404] niks3-server[989]: 2026/09/22 16:05:35 INFO Received uploads request method=POST path=/api/pending_closures5693worker2 # [ 85.812607] nix-grpc-daemon[882]: time=2026-09-22T16:05:35.041Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5694worker2 # [ 85.814477] nix-grpc-daemon[882]: time=2026-09-22T16:05:35.042Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5695lb # [ 85.884138] niks3-server[989]: 2026/09/22 16:05:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5696lb # [ 85.889259] niks3-server[989]: 2026/09/22 16:05:35 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5697lb # [ 85.896882] niks3-server[989]: 2026/09/22 16:05:35 INFO Signed narinfos id=33 count=15698worker2 # [ 85.859376] nix-grpc-daemon[882]: time=2026-09-22T16:05:35.087Z level=INFO msg="Uploading 1 narinfos"5699lb # [ 85.904663] niks3-server[989]: 2026/09/22 16:05:35 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5700lb # [ 85.917359] niks3-server[989]: 2026/09/22 16:05:35 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5701lb # [ 85.926670] niks3-server[989]: 2026/09/22 16:05:35 INFO Completed upload id=335702worker2 # [ 85.883874] nix-grpc-daemon[882]: time=2026-09-22T16:05:35.112Z level=INFO msg="Upload complete. (293ms)"5703lb # [ 85.928194] niks3-server[989]: 2026/09/22 16:05:35 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5704worker2 # [ 85.888688] nix-grpc-daemon[786]: ts=2026-09-22T16:05:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=1 nar_bytes_in=5605705worker2 # [ 85.895508] nix-grpc-daemon[786]: ts=2026-09-22T16:05:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:343085706worker2 # [ 85.906109] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5707worker2 # [ 85.914123] nix-daemon[854]: reaped child process 1143, status = succeeded5708client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5709worker2 # [ 85.919368] nix-daemon[854]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5710client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5711client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5712worker2 # [ 86.761537] nix-daemon[854]: reaped child process 1157, status = succeeded5713worker2 # [ 86.844905] nix-daemon[854]: accepted connection from pid 1173, user nix-grpc-daemon (trusted)5714worker2 # [ 86.858918] nix-daemon[854]: reaped child process 1176, status = succeeded5715lb # [ 86.910476] niks3-server[989]: 2026/09/22 16:05:36 INFO Received uploads request method=POST path=/api/pending_closures5716worker2 # [ 86.875615] nix-grpc-daemon[882]: time=2026-09-22T16:05:36.104Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5717lb # [ 86.940930] niks3-server[989]: 2026/09/22 16:05:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5718lb # [ 86.947536] niks3-server[989]: 2026/09/22 16:05:36 INFO Signed narinfos id=34 count=15719worker2 # [ 86.907524] nix-grpc-daemon[882]: time=2026-09-22T16:05:36.136Z level=INFO msg="Uploading 1 narinfos"5720lb # [ 86.956310] niks3-server[989]: 2026/09/22 16:05:36 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5721lb # [ 86.965957] niks3-server[989]: 2026/09/22 16:05:36 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5722lb # [ 86.974538] niks3-server[989]: 2026/09/22 16:05:36 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5723lb # [ 86.981625] niks3-server[989]: 2026/09/22 16:05:36 INFO Completed upload id=345724worker2 # [ 86.938776] nix-grpc-daemon[882]: time=2026-09-22T16:05:36.167Z level=INFO msg="Upload complete. (211ms)"5725lb # [ 86.986246] niks3-server[989]: 2026/09/22 16:05:36 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5726worker2 # [ 86.944669] nix-grpc-daemon[786]: ts=2026-09-22T16:05:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=17 outputs=15727client # copying 1 paths...5728worker2 # [ 86.955136] nix-grpc-daemon[786]: ts=2026-09-22T16:05:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:343085729client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5730worker2 # [ 86.962874] nix-daemon[854]: reaped child process 1155, status = succeeded5731worker2 # [ 86.973922] nix-grpc-daemon[786]: ts=2026-09-22T16:05:36Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:343085732worker2 # [ 86.977944] nix-grpc-daemon[786]: ts=2026-09-22T16:05:36Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34308 duration_s=0 paths=1 nar_bytes_out=1205733client # [ 87.089782] nix-daemon[1111]: reaped child process 1241, status = succeeded5734client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&2, in 1.84 seconds)5735(finished: subtest: an input only one worker has still reaches the builder, in 7.41 seconds)5736subtest: upload whose reference the worker lacks is completed from the cache5737client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache5738lb # [ 87.141168] envoy[925]: conn peer=[2001:db8:1::1]:53184 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3863 tx=2140 ms=17975739lb # [ 87.145243] envoy[925]: conn peer=[2001:db8:1::1]:53194 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=378 ms=545740lb # [ 87.149072] envoy[925]: conn peer=[2001:db8:1::1]:53208 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4311 tx=2364 ms=18405741lb # [ 87.153153] envoy[925]: conn peer=[2001:db8:1::1]:53210 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=585742lb # [ 87.156612] envoy[925]: conn peer=[2001:db8:1::1]:53212 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3851 tx=2068 ms=18535743lb # [ 87.160127] envoy[925]: conn peer=[2001:db8:1::1]:53228 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=565744lb # [ 87.163529] envoy[925]: conn peer=[2001:db8:1::1]:53232 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3711 tx=2101 ms=27245745lb # [ 87.167722] envoy[925]: conn peer=[2001:db8:1::1]:56228 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=329 ms=835746lb # [ 87.171191] envoy[925]: conn peer=[2001:db8:1::1]:56232 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3729 tx=2126 ms=15645747lb # [ 87.174671] envoy[925]: conn peer=[2001:db8:1::1]:56234 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=545748client # this derivation will be built:5749client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5750client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5751client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache, in 0.24 seconds)5752client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache5753client # this derivation will be built:5754client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5755client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5756client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache, in 0.23 seconds)5757client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5758client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.09 seconds)5759worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5760worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.12 seconds)5761worker2 # [ 87.818516] systemd[1]: Stopping Nix worker-protocol over gRPC...5762worker2 # [ 88.336288] nix-grpc-daemon[786]: ts=2026-09-22T16:05:37Z level=info event=signal_exit5763worker1 # [ 89.716809] nix-grpc-daemon[779]: ts=2026-09-22T16:05:38Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:358525764worker1 # [ 89.718699] nix-grpc-daemon[779]: ts=2026-09-22T16:05:38Z level=info event=worker_gone addr=192.168.1.4:500515765worker2 # [ 89.691545] nix-grpc-daemon[786]: ts=2026-09-22T16:05:38Z level=info event=scheduler_restarting5766worker2 # [ 89.711145] nix-grpc-daemon[786]: ts=2026-09-22T16:05:38Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5767worker2 # [ 89.731838] nix-daemon[854]: reaped child process 856, status = succeeded5768worker2 # [ 90.458538] nix-grpc-daemon[786]: E0000 00:00:1790093139.686839 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true5769worker2 # [ 90.463647] nix-grpc-daemon[786]: E0000 00:00:1790093139.687099 831 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5770worker2 # [ 91.360460] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5771worker2 # [ 91.362177] systemd[1]: Stopped Nix worker-protocol over gRPC.5772worker2 # [ 91.364188] systemd[1]: nix-grpc-daemon.service: Consumed 2.360s CPU time over 1min 14.187s wall clock time, 60.8M memory peak, 79.3K incoming IP traffic, 189.5K outgoing IP traffic.5773worker2 # [ 91.368790] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5774worker2 # [ 91.370599] systemd[1]: Closed Nix worker-protocol over gRPC.5775worker2 # [ 91.372195] systemd[1]: nix-grpc-daemon.socket: Consumed 321us CPU time over 1min 23.821s wall clock time, 256K memory peak, 86.1K incoming IP traffic, 73.1K outgoing IP traffic.5776lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5777lb: must succeed: curl -sf localhost:9901/clusters5778lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5779lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.03 seconds)5780client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25781client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5782worker1 # [ 91.565364] nix-grpc-daemon[779]: ts=2026-09-22T16:05:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:584325783worker1 # [ 91.569444] nix-grpc-daemon[779]: ts=2026-09-22T16:05:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:584325784client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.16 seconds)5785worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5786worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.01 seconds)5787worker2 # [ 91.592716] systemd[1]: Listening on Nix worker-protocol over gRPC.5788worker1 # [ 91.659772] systemd[1]: Stopping Nix worker-protocol over gRPC...5789worker1 # [ 92.298194] nix-grpc-daemon[779]: ts=2026-09-22T16:05:41Z level=info event=signal_exit5790worker2 # [ 92.358744] systemd[1]: Starting scheduler-token.service...5791worker2 # [ 92.389147] systemd[1]: scheduler-token.service: Deactivated successfully.5792worker2 # [ 92.390747] systemd[1]: Finished scheduler-token.service.5793worker2 # [ 92.391894] systemd[1]: scheduler-token.service: Consumed 18ms CPU time over 31ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5794worker2 # [ 92.396349] systemd[1]: Starting Nix worker-protocol over gRPC...5795worker2 # [ 92.477405] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:41Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5796worker2 # [ 92.479419] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:41Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5797worker2 # [ 92.482979] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:41Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15798worker2 # [ 92.486840] nix-grpc-daemon[1200]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5799worker2 # [ 92.489114] nix-grpc-daemon[1200]: E0000 00:00:1790093141.715530 1200 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5800worker2 # [ 92.494068] nix-grpc-daemon[1200]: E0000 00:00:1790093141.721297 1200 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5801worker2 # [ 92.496164] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:41Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5802worker2 # [ 92.501409] systemd[1]: Started Nix worker-protocol over gRPC.5803worker2 # [ 92.506459] nix-grpc-daemon[1200]: E0000 00:00:1790093141.735084 1242 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5804worker2 # [ 92.520065] nix-grpc-daemon[1200]: E0000 00:00:1790093141.748727 1242 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5805worker2 # [ 92.525054] nix-grpc-daemon[1200]: E0000 00:00:1790093141.751080 1242 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5806worker2 # [ 92.534387] nix-grpc-daemon[1200]: E0000 00:00:1790093141.763028 1242 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5807worker2 # [ 92.543814] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:41Z level=info event=scheduler_connected addr=lb:500515808worker2 # [ 92.545354] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5809lb # [ 92.732892] niks3-server[989]: 2026/09/22 16:05:42 INFO lead: released remote=192.168.1.3:518485810worker2 # [ 92.747080] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:41Z level=info event=scheduler_connected addr=lb:500515811worker2 # [ 92.751211] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5812worker2 # [ 92.952803] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:42Z level=info event=scheduler_connected addr=lb:500515813worker2 # [ 92.956885] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5814worker2 # [ 93.158297] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:42Z level=info event=scheduler_connected addr=lb:500515815worker2 # [ 93.162452] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5816worker2 # [ 93.363882] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:42Z level=info event=scheduler_connected addr=lb:500515817worker2 # [ 93.367871] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5818worker2 # [ 93.570440] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:42Z level=info event=scheduler_connected addr=lb:500515819worker2 # [ 93.573616] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5820worker1 # [ 93.716471] nix-grpc-daemon[779]: ts=2026-09-22T16:05:42Z level=info event=scheduler_restarting5821worker1 # [ 93.719983] nix-grpc-daemon[779]: ts=2026-09-22T16:05:42Z level=info event=scheduler_restarting addr=lb:500515822worker1 # [ 93.724183] nix-grpc-daemon[779]: ts=2026-09-22T16:05:42Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:358525823worker1 # [ 93.727910] nix-grpc-daemon[779]: ts=2026-09-22T16:05:42Z level=info event=worker_gone addr=192.168.1.3:500515824worker1 # [ 93.745634] nix-daemon[861]: reaped child process 863, status = succeeded5825worker2 # [ 93.771469] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:43Z level=info event=scheduler_connected addr=lb:500515826worker2 # [ 93.775302] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5827worker1 # [ 93.928382] nix-grpc-daemon[779]: ts=2026-09-22T16:05:43Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5828worker2 # [ 93.976468] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:43Z level=info event=scheduler_connected addr=lb:500515829worker2 # [ 93.980265] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5830worker2 # [ 94.181577] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:43Z level=info event=scheduler_connected addr=lb:500515831worker2 # [ 94.185388] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5832worker2 # [ 94.386692] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:43Z level=info event=scheduler_connected addr=lb:500515833worker2 # [ 94.390556] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5834worker1 # [ 94.421714] nix-grpc-daemon[779]: E0000 00:00:1790093143.630509 824 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true5835worker1 # [ 94.426732] nix-grpc-daemon[779]: E0000 00:00:1790093143.630665 824 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5836worker2 # [ 94.593223] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:43Z level=info event=scheduler_connected addr=lb:500515837worker2 # [ 94.596371] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5838worker2 # [ 94.794839] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:44Z level=info event=scheduler_connected addr=lb:500515839worker2 # [ 94.798758] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5840worker2 # [ 94.999783] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:44Z level=info event=scheduler_connected addr=lb:500515841worker2 # [ 95.003834] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5842worker2 # [ 95.205306] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:44Z level=info event=scheduler_connected addr=lb:500515843worker2 # [ 95.209248] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5844worker1 # [ 95.369191] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5845worker1 # [ 95.372467] systemd[1]: Stopped Nix worker-protocol over gRPC.5846worker1 # [ 95.376473] systemd[1]: nix-grpc-daemon.service: Consumed 3.027s CPU time over 1min 18.252s wall clock time, 71M memory peak, 113.2K incoming IP traffic, 245.8K outgoing IP traffic.5847worker1 # [ 95.388434] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5848worker1 # [ 95.392637] systemd[1]: Closed Nix worker-protocol over gRPC.5849worker1 # [ 95.396669] systemd[1]: nix-grpc-daemon.socket: Consumed 347us CPU time over 1min 27.796s wall clock time, 256K memory peak, 122.4K incoming IP traffic, 107.1K outgoing IP traffic.5850lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5851lb: must succeed: curl -sf localhost:9901/clusters5852worker2 # [ 95.410641] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:44Z level=info event=scheduler_connected addr=lb:500515853worker2 # [ 95.414794] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5854lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5855lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.05 seconds)5856client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25857worker2 # [ 95.567967] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544745858worker2 # [ 95.580255] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)5859worker2 # [ 95.597747] nix-grpc-daemon[1200]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 1 ms (attempt 1/5)5860worker2 # [ 95.605879] nix-grpc-daemon[1200]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 198 ms (attempt 2/5)5861worker2 # [ 95.615558] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:44Z level=info event=scheduler_connected addr=lb:500515862worker2 # [ 95.619430] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5863worker2 # [ 95.812525] nix-grpc-daemon[1200]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 329 ms (attempt 3/5)5864worker2 # [ 95.820436] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=info event=scheduler_connected addr=lb:500515865worker2 # [ 95.824369] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5866worker2 # [ 96.025589] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=info event=scheduler_connected addr=lb:500515867worker2 # [ 96.029473] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5868worker2 # [ 96.143368] nix-grpc-daemon[1200]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 487 ms (attempt 4/5)5869worker2 # [ 96.230847] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=info event=scheduler_connected addr=lb:500515870worker2 # [ 96.234869] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5871worker2 # [ 96.436382] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=info event=scheduler_connected addr=lb:500515872worker2 # [ 96.440224] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5873worker2 # [ 96.635094] nix-grpc-daemon[1200]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5874worker2 # [ 96.638163] nix-grpc-daemon[1200]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5875worker2 # [ 96.643992] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=info event=scheduler_connected addr=lb:500515876worker2 # [ 96.648211] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5877client # copying 2 paths...5878client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5879worker2 # [ 96.663527] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544745880client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5881worker2 # [ 96.677242] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)5882worker2 # [ 96.696352] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:45Z level=info event=niks3_push_started pid=12605883worker2 # [ 96.799029] nix-grpc-daemon[1260]: time=2026-09-22T16:05:46.027Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""5884worker2 # [ 96.845579] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:46Z level=info event=scheduler_connected addr=lb:500515885worker2 # [ 96.847737] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5886worker2 # [ 96.893088] nix-daemon[854]: accepted connection from pid 1273, user nix-grpc-daemon (trusted)5887worker2 # [ 96.907555] nix-daemon[854]: reaped child process 1276, status = succeeded5888lb # [ 96.956759] niks3-server[989]: 2026/09/22 16:05:46 INFO Received uploads request method=POST path=/api/pending_closures5889lb # [ 96.962372] niks3-server[989]: 2026/09/22 16:05:46 INFO Received uploads request method=POST path=/api/pending_closures5890worker2 # [ 96.922989] nix-grpc-daemon[1260]: time=2026-09-22T16:05:46.151Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5891worker2 # [ 96.924500] nix-grpc-daemon[1260]: time=2026-09-22T16:05:46.153Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5892worker2 # [ 96.926339] nix-grpc-daemon[1260]: time=2026-09-22T16:05:46.155Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5893lb # [ 96.987493] niks3-server[989]: 2026/09/22 16:05:46 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5894lb # [ 96.999584] niks3-server[989]: 2026/09/22 16:05:46 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5895lb # [ 97.005063] niks3-server[989]: 2026/09/22 16:05:46 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5896lb # [ 97.009817] niks3-server[989]: 2026/09/22 16:05:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5897lb # [ 97.011798] niks3-server[989]: 2026/09/22 16:05:46 INFO Signed narinfos id=35 count=15898lb # [ 97.014726] niks3-server[989]: 2026/09/22 16:05:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5899lb # [ 97.017053] niks3-server[989]: 2026/09/22 16:05:46 INFO Signed narinfos id=36 count=25900worker2 # [ 96.974923] nix-grpc-daemon[1260]: time=2026-09-22T16:05:46.202Z level=INFO msg="Uploading 3 narinfos"5901lb # [ 97.020197] niks3-server[989]: 2026/09/22 16:05:46 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5902lb # [ 97.033230] niks3-server[989]: 2026/09/22 16:05:46 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5903lb # [ 97.037532] niks3-server[989]: 2026/09/22 16:05:46 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5904lb # [ 97.041106] niks3-server[989]: 2026/09/22 16:05:46 INFO Completed upload id=355905lb # [ 97.042930] niks3-server[989]: 2026/09/22 16:05:46 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5906lb # [ 97.045264] niks3-server[989]: 2026/09/22 16:05:46 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5907lb # [ 97.047744] niks3-server[989]: 2026/09/22 16:05:46 INFO Completed upload id=365908worker2 # [ 97.005801] nix-grpc-daemon[1260]: time=2026-09-22T16:05:46.234Z level=INFO msg="Upload complete. (203ms)"5909lb # [ 97.049727] niks3-server[989]: 2026/09/22 16:05:46 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5910worker2 # [ 97.008888] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54474 duration_s=0 paths=2 nar_bytes_in=3125911worker2 # [ 97.017516] nix-daemon[854]: reaped child process 1258, status = succeeded5912client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.61 seconds)5913worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5914worker2 # [ 97.048496] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:46Z level=info event=scheduler_connected addr=lb:500515915worker2 # [ 97.050512] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5916worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.02 seconds)5917lb # [ 97.141700] envoy[925]: conn peer=[2001:db8:1::4]:51000 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=59438 tx=3097 ms=723495918lb # [ 97.145594] envoy[925]: conn peer=[2001:db8:1::1]:56248 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=519 ms=95919worker1 # [ 97.125873] systemd[1]: Listening on Nix worker-protocol over gRPC.5920lb # [ 97.149096] envoy[925]: conn peer=[2001:db8:1::3]:37874 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=16261 tx=3452 ms=765965921worker1 # [ 97.132458] systemd[1]: Starting Nix worker-protocol over gRPC...5922lb # [ 97.152746] envoy[925]: conn peer=[2001:db8:1::1]:52056 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1005 tx=305 ms=14545923worker1 # [ 97.216066] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5924worker1 # [ 97.218063] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5925worker1 # [ 97.221256] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15926worker1 # [ 97.225354] nix-grpc-daemon[1380]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5927worker1 # [ 97.228192] nix-grpc-daemon[1380]: E0000 00:00:1790093146.434416 1380 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5928worker1 # [ 97.233935] nix-grpc-daemon[1380]: E0000 00:00:1790093146.439767 1380 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5929worker1 # [ 97.236080] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5930worker1 # [ 97.237625] systemd[1]: Started Nix worker-protocol over gRPC.5931lb # [ 97.264261] niks3-server[989]: 2026/09/22 16:05:46 INFO lead: acquired remote=192.168.1.3:517585932worker1 # [ 97.244061] nix-grpc-daemon[1380]: E0000 00:00:1790093146.450580 1419 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5933worker1 # [ 97.249501] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=scheduler_take_over5934worker1 # [ 97.256987] nix-grpc-daemon[1380]: E0000 00:00:1790093146.465877 1419 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5935worker1 # [ 97.259640] nix-grpc-daemon[1380]: E0000 00:00:1790093146.466232 1419 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5936lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5937lb: must succeed: curl -sf localhost:9901/clusters5938worker1 # [ 97.271881] nix-grpc-daemon[1380]: E0000 00:00:1790093146.480901 1419 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5939worker2 # [ 97.252088] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:46Z level=info event=scheduler_connected addr=lb:500515940worker2 # [ 97.256230] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5941worker1 # [ 97.283955] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=scheduler_connected addr=lb:500515942worker1 # [ 97.285585] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5943lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5944worker2 # [ 97.457429] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:46Z level=info event=scheduler_connected addr=lb:500515945worker1 # [ 97.489186] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=scheduler_connected addr=lb:500515946worker1 # [ 97.494289] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:582945947worker1 # [ 97.498648] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05948worker1 # [ 97.504372] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=oidc_jwks provider=mock keys=15949worker1 # [ 97.507268] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:582945950worker1 # [ 97.511564] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:46Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05951lb: must succeed: curl -sf localhost:9901/clusters5952lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5953lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.08 seconds)5954(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.22 seconds)5955subtest: a niks3 restart does not cost the leader its role5956lb: must succeed: curl -sf localhost:9901/clusters5957lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5958lb: must succeed: curl -sf localhost:9901/clusters5959lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5960worker2 # [ 98.376702] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:47Z level=info event=scheduler_lock_lost status=2005961lb # [ 98.420091] niks3-server[989]: 2026/09/22 16:05:47 INFO Shutdown signal received, draining in-flight requests timeout=10s5962worker1 # [ 98.399322] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:47Z level=info event=scheduler_lock_lost status=2005963lb # [ 98.423974] niks3-server[989]: 2026/09/22 16:05:47 INFO lead: released remote=192.168.1.3:517585964lb # [ 98.425272] systemd[1]: Stopping niks3 server...5965lb # [ 98.435318] systemd[1]: niks3.service: Deactivated successfully.5966lb # [ 98.437061] systemd[1]: Stopped niks3 server.5967lb # [ 98.437769] systemd[1]: niks3.service: Consumed 685ms CPU time over 1min 17.937s wall clock time, 28.4M memory peak, 95.7K incoming IP traffic, 137.8K outgoing IP traffic.5968lb # [ 98.445250] systemd[1]: Starting niks3 server...5969lb # [ 98.494662] niks3-server[1128]: 2026/09/22 16:05:47 goose: no migrations to run. current version: 202609200000005970lb # [ 98.500815] niks3-server[1128]: 2026/09/22 16:05:47 OK 1_commit_pending_closure.sql (4.19ms)5971lb # [ 98.504408] niks3-server[1128]: 2026/09/22 16:05:47 OK 2_object_stats_trigger.sql (3.34ms)5972lb # [ 98.505458] niks3-server[1128]: 2026/09/22 16:05:47 goose: up to current file version: 25973lb # [ 98.507019] niks3-server[1128]: 2026/09/22 16:05:47 INFO Loaded signing key name=farm-test-1 path=/nix/store/hk0nhypl37jvkn3gr36v3y7qx593p39i-key5974lb # [ 98.518565] niks3-server[1128]: 2026/09/22 16:05:47 INFO Read proxy enabled — serving cache objects from S35975lb # [ 98.519894] niks3-server[1128]: 2026/09/22 16:05:47 INFO Using socket-activated listener address=0.0.0.0:57515976lb # [ 98.521928] systemd[1]: Started niks3 server.5977lb # [ 98.522669] niks3-server[1128]: 2026/09/22 16:05:47 INFO systemd watchdog enabled interval=15s5978lb # [ 98.525298] niks3-server[1128]: 2026/09/22 16:05:47 INFO Starting HTTP server address=0.0.0.0:57515979lb: waiting for unit niks3.service5980lb: (finished: waiting for unit niks3.service, in 0.02 seconds)5981lb: must succeed: curl -sf localhost:9901/clusters5982lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5983worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5984worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5985lb: must succeed: curl -sf localhost:9901/clusters5986lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5987lb: must succeed: curl -sf localhost:9901/clusters5988lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5989worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5990worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.05 seconds)5991client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&25992worker2 # [ 98.796261] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:48Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:544745993client # don't know how to build these paths:5994client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5995worker1 # [ 98.831825] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:583025996worker1 # [ 98.843176] nix-daemon[861]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)5997worker1 # [ 98.858700] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 41 ms (attempt 1/5)5998worker1 # [ 98.906733] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 149 ms (attempt 2/5)5999worker1 # [ 99.057925] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 329 ms (attempt 3/5)6000worker1 # [ 99.394643] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 208 ms (attempt 4/5)6001lb # [ 99.425977] niks3-server[1128]: 2026/09/22 16:05:48 INFO lead: acquired remote=192.168.1.3:517746002worker1 # [ 99.612748] nix-grpc-daemon[1380]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6003worker1 # [ 99.616406] nix-grpc-daemon[1380]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6004worker1 # [ 99.652834] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:48Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=06005worker1 # [ 99.658471] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:48Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.3:50051 clients=16006worker1 # [ 99.697128] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:583026007worker1 # [ 99.711995] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:583026008client # copying 1 paths to 'lb:50051'...6009client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...6010worker1 # [ 99.725747] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:583026011worker1 # [ 99.734474] nix-daemon[861]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6012worker1 # [ 99.756207] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:48Z level=info event=niks3_push_started pid=14526013worker1 # [ 99.857512] nix-grpc-daemon[1452]: time=2026-09-22T16:05:49.066Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6014worker1 # [ 99.944435] nix-daemon[861]: accepted connection from pid 1465, user nix-grpc-daemon (trusted)6015worker1 # [ 99.955941] nix-daemon[861]: reaped child process 1468, status = succeeded6016worker1 # [ 100.039944] nix-daemon[861]: accepted connection from pid 1470, user nix-grpc-daemon (trusted)6017worker1 # [ 100.052400] nix-daemon[861]: reaped child process 1473, status = succeeded6018lb # [ 100.082404] niks3-server[1128]: 2026/09/22 16:05:49 INFO Received uploads request method=POST path=/api/pending_closures6019worker1 # [ 100.077071] nix-grpc-daemon[1452]: time=2026-09-22T16:05:49.285Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6020worker1 # [ 100.078467] nix-grpc-daemon[1452]: time=2026-09-22T16:05:49.285Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"6021lb # [ 100.131161] niks3-server[1128]: 2026/09/22 16:05:49 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst6022lb # [ 100.137790] niks3-server[1128]: 2026/09/22 16:05:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6023lb # [ 100.156905] niks3-server[1128]: 2026/09/22 16:05:49 INFO Signed narinfos id=37 count=16024worker1 # [ 100.136476] nix-grpc-daemon[1452]: time=2026-09-22T16:05:49.345Z level=INFO msg="Uploading 1 narinfos"6025lb # [ 100.166503] niks3-server[1128]: 2026/09/22 16:05:49 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6026lb # [ 100.176284] niks3-server[1128]: 2026/09/22 16:05:49 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls6027lb # [ 100.181069] niks3-server[1128]: 2026/09/22 16:05:49 INFO Completed upload id=376028worker1 # [ 100.160446] nix-grpc-daemon[1452]: time=2026-09-22T16:05:49.369Z level=INFO msg="Upload complete. (299ms)"6029lb # [ 100.184290] niks3-server[1128]: 2026/09/22 16:05:49 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo6030worker1 # [ 100.165451] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58302 duration_s=0 paths=1 nar_bytes_in=5366031worker1 # [ 100.171801] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:583026032worker1 # [ 100.182194] nix-daemon[861]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6033worker1 # [ 100.193379] nix-daemon[861]: reaped child process 1450, status = succeeded6034worker1 # [ 100.197640] nix-daemon[861]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6035client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...6036client # farm-a> node-a: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv6037client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6038client # farm-a> LOG-farm-a-after-niks3-restart6039worker1 # [ 100.833148] nix-daemon[861]: reaped child process 1477, status = succeeded6040worker1 # [ 100.916168] nix-daemon[861]: accepted connection from pid 1493, user nix-grpc-daemon (trusted)6041worker1 # [ 100.927755] nix-daemon[861]: reaped child process 1496, status = succeeded6042lb # [ 100.956064] niks3-server[1128]: 2026/09/22 16:05:50 INFO Received uploads request method=POST path=/api/pending_closures6043worker1 # [ 100.943959] nix-grpc-daemon[1452]: time=2026-09-22T16:05:50.152Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6044worker1 # [ 100.946140] nix-grpc-daemon[1452]: time=2026-09-22T16:05:50.154Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"6045lb # [ 100.989343] niks3-server[1128]: 2026/09/22 16:05:50 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv6046lb # [ 101.006488] niks3-server[1128]: 2026/09/22 16:05:50 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst6047lb # [ 101.020425] niks3-server[1128]: 2026/09/22 16:05:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6048lb # [ 101.025905] niks3-server[1128]: 2026/09/22 16:05:50 INFO Signed narinfos id=38 count=16049worker1 # [ 101.005653] nix-grpc-daemon[1452]: time=2026-09-22T16:05:50.213Z level=INFO msg="Uploading 1 narinfos"6050lb # [ 101.036308] niks3-server[1128]: 2026/09/22 16:05:50 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls6051lb # [ 101.039527] niks3-server[1128]: 2026/09/22 16:05:50 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6052lb # [ 101.044495] niks3-server[1128]: 2026/09/22 16:05:50 INFO Completed upload id=386053worker1 # [ 101.023774] nix-grpc-daemon[1452]: time=2026-09-22T16:05:50.232Z level=INFO msg="Upload complete. (219ms)"6054lb # [ 101.049360] niks3-server[1128]: 2026/09/22 16:05:50 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo6055worker1 # [ 101.029722] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58302 duration_s=0 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=16056worker1 # [ 101.044490] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:50Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=06057worker1 # [ 101.047794] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:50Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.3:50051 clients=16058worker1 # [ 101.057434] nix-daemon[861]: reaped child process 1475, status = succeeded6059worker1 # [ 101.087662] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:583026060worker1 # [ 101.099690] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:583026061worker1 # [ 101.113734] nix-grpc-daemon[1452]: time=2026-09-22T16:05:50.322Z level=INFO msg="All 1 paths already cached"6062client # copying 1 paths to 'lb:50051'...6063client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...6064worker1 # [ 101.122199] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:583026065worker1 # [ 101.132202] nix-daemon[861]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6066worker1 # [ 101.226757] nix-daemon[861]: accepted connection from pid 1502, user nix-grpc-daemon (trusted)6067worker1 # [ 101.241271] nix-daemon[861]: reaped child process 1505, status = succeeded6068worker1 # [ 101.322068] nix-daemon[861]: accepted connection from pid 1507, user nix-grpc-daemon (trusted)6069worker1 # [ 101.335242] nix-daemon[861]: reaped child process 1510, status = succeeded6070lb # [ 101.363558] niks3-server[1128]: 2026/09/22 16:05:50 INFO Received uploads request method=POST path=/api/pending_closures6071worker1 # [ 101.353856] nix-grpc-daemon[1452]: time=2026-09-22T16:05:50.562Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6072worker1 # [ 101.355873] nix-grpc-daemon[1452]: time=2026-09-22T16:05:50.564Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"6073lb # [ 101.399099] niks3-server[1128]: 2026/09/22 16:05:50 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst6074lb # [ 101.410744] niks3-server[1128]: 2026/09/22 16:05:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6075lb # [ 101.415922] niks3-server[1128]: 2026/09/22 16:05:50 INFO Signed narinfos id=39 count=16076worker1 # [ 101.399155] nix-grpc-daemon[1452]: time=2026-09-22T16:05:50.608Z level=INFO msg="Uploading 1 narinfos"6077lb # [ 101.430743] niks3-server[1128]: 2026/09/22 16:05:50 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls6078lb # [ 101.438330] niks3-server[1128]: 2026/09/22 16:05:50 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6079lb # [ 101.447804] niks3-server[1128]: 2026/09/22 16:05:50 INFO Completed upload id=396080worker1 # [ 101.426982] nix-grpc-daemon[1452]: time=2026-09-22T16:05:50.635Z level=INFO msg="Upload complete. (284ms)"6081lb # [ 101.450210] niks3-server[1128]: 2026/09/22 16:05:50 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo6082worker1 # [ 101.431745] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58302 duration_s=0 paths=1 nar_bytes_in=6566083worker1 # [ 101.437518] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:583026084worker1 # [ 101.448106] nix-daemon[861]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6085worker1 # [ 101.455229] nix-daemon[861]: reaped child process 1500, status = succeeded6086worker1 # [ 101.460056] nix-daemon[861]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6087client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...6088client # farm-b> node-a: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6089client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6090client # farm-b> LOG-farm-b-after-niks3-restart6091worker1 # [ 102.274797] nix-daemon[861]: reaped child process 1514, status = succeeded6092worker1 # [ 102.348708] nix-daemon[861]: accepted connection from pid 1530, user nix-grpc-daemon (trusted)6093worker1 # [ 102.362129] nix-daemon[861]: reaped child process 1533, status = succeeded6094lb # [ 102.388708] niks3-server[1128]: 2026/09/22 16:05:51 INFO Received uploads request method=POST path=/api/pending_closures6095worker1 # [ 102.372584] nix-grpc-daemon[1452]: time=2026-09-22T16:05:51.581Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6096worker1 # [ 102.374102] nix-grpc-daemon[1452]: time=2026-09-22T16:05:51.583Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"6097lb # [ 102.411069] niks3-server[1128]: 2026/09/22 16:05:51 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6098lb # [ 102.414228] niks3-server[1128]: 2026/09/22 16:05:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6099lb # [ 102.419251] niks3-server[1128]: 2026/09/22 16:05:51 INFO Signed narinfos id=40 count=16100worker1 # [ 102.398515] nix-grpc-daemon[1452]: time=2026-09-22T16:05:51.607Z level=INFO msg="Uploading 1 narinfos"6101lb # [ 102.423358] niks3-server[1128]: 2026/09/22 16:05:51 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst6102lb # [ 102.425942] niks3-server[1128]: 2026/09/22 16:05:51 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls6103lb # [ 102.431716] niks3-server[1128]: 2026/09/22 16:05:51 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6104lb # [ 102.434939] niks3-server[1128]: 2026/09/22 16:05:51 INFO Completed upload id=406105worker1 # [ 102.414033] nix-grpc-daemon[1452]: time=2026-09-22T16:05:51.623Z level=INFO msg="Upload complete. (147ms)"6106lb # [ 102.438165] niks3-server[1128]: 2026/09/22 16:05:51 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo6107worker1 # [ 102.416445] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58302 duration_s=0 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=16108worker1 # [ 102.424642] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:51Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=06109worker1 # [ 102.426620] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:51Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.3:50051 clients=16110worker1 # [ 102.429851] nix-daemon[861]: reaped child process 1512, status = succeeded6111worker1 # [ 102.467938] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:583026112worker1 # [ 102.479291] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:583026113worker1 # [ 102.491507] nix-grpc-daemon[1452]: time=2026-09-22T16:05:51.700Z level=INFO msg="All 2 paths already cached"6114client # copying 1 paths to 'lb:50051'...6115client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...6116worker1 # [ 102.499504] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:583026117worker1 # [ 102.506459] nix-daemon[861]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6118worker1 # [ 102.601037] nix-daemon[861]: accepted connection from pid 1539, user nix-grpc-daemon (trusted)6119worker1 # [ 102.613554] nix-daemon[861]: reaped child process 1542, status = succeeded6120worker1 # [ 102.699247] nix-daemon[861]: accepted connection from pid 1544, user nix-grpc-daemon (trusted)6121worker1 # [ 102.711575] nix-daemon[861]: reaped child process 1547, status = succeeded6122lb # [ 102.740974] niks3-server[1128]: 2026/09/22 16:05:52 INFO Received uploads request method=POST path=/api/pending_closures6123worker1 # [ 102.728186] nix-grpc-daemon[1452]: time=2026-09-22T16:05:51.937Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6124worker1 # [ 102.730116] nix-grpc-daemon[1452]: time=2026-09-22T16:05:51.938Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"6125lb # [ 102.776538] niks3-server[1128]: 2026/09/22 16:05:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6126worker1 # [ 102.760799] nix-grpc-daemon[1452]: time=2026-09-22T16:05:51.969Z level=INFO msg="Uploading 1 narinfos"6127lb # [ 102.783139] niks3-server[1128]: 2026/09/22 16:05:52 INFO Signed narinfos id=41 count=16128lb # [ 102.790228] niks3-server[1128]: 2026/09/22 16:05:52 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst6129lb # [ 102.795601] niks3-server[1128]: 2026/09/22 16:05:52 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls6130lb # [ 102.808719] niks3-server[1128]: 2026/09/22 16:05:52 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6131lb # [ 102.816510] niks3-server[1128]: 2026/09/22 16:05:52 INFO Completed upload id=416132worker1 # [ 102.795469] nix-grpc-daemon[1452]: time=2026-09-22T16:05:52.004Z level=INFO msg="Upload complete. (277ms)"6133lb # [ 102.818974] niks3-server[1128]: 2026/09/22 16:05:52 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo6134worker1 # [ 102.799072] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58302 duration_s=0 paths=1 nar_bytes_in=7846135worker1 # [ 102.803912] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:583026136worker1 # [ 102.812179] nix-daemon[861]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6137worker1 # [ 102.820564] nix-daemon[861]: reaped child process 1537, status = succeeded6138worker1 # [ 102.825302] nix-daemon[861]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)6139client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...6140client # farm-top> node-a: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6141client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6142client # farm-top> LOG-farm-top-after-niks3-restart6143worker1 # [ 103.467293] nix-daemon[861]: reaped child process 1551, status = succeeded6144worker1 # [ 103.544867] nix-daemon[861]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)6145worker1 # [ 103.557515] nix-daemon[861]: reaped child process 1570, status = succeeded6146lb # [ 103.587071] niks3-server[1128]: 2026/09/22 16:05:53 INFO Received uploads request method=POST path=/api/pending_closures6147worker1 # [ 103.576416] nix-grpc-daemon[1452]: time=2026-09-22T16:05:52.785Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6148worker1 # [ 103.578331] nix-grpc-daemon[1452]: time=2026-09-22T16:05:52.786Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"6149lb # [ 103.624584] niks3-server[1128]: 2026/09/22 16:05:53 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst6150lb # [ 103.638207] niks3-server[1128]: 2026/09/22 16:05:53 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6151lb # [ 103.650084] niks3-server[1128]: 2026/09/22 16:05:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6152lb # [ 103.653828] niks3-server[1128]: 2026/09/22 16:05:53 INFO Signed narinfos id=42 count=16153worker1 # [ 103.633389] nix-grpc-daemon[1452]: time=2026-09-22T16:05:52.842Z level=INFO msg="Uploading 1 narinfos"6154lb # [ 103.662970] niks3-server[1128]: 2026/09/22 16:05:53 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6155lb # [ 103.666952] niks3-server[1128]: 2026/09/22 16:05:53 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls6156lb # [ 103.671095] niks3-server[1128]: 2026/09/22 16:05:53 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo6157lb # [ 103.673503] niks3-server[1128]: 2026/09/22 16:05:53 INFO Completed upload id=426158worker1 # [ 103.652804] nix-grpc-daemon[1452]: time=2026-09-22T16:05:52.861Z level=INFO msg="Upload complete. (221ms)"6159worker1 # [ 103.659186] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58302 duration_s=0 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=16160worker1 # [ 103.674120] nix-daemon[861]: reaped child process 1549, status = succeeded6161client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&2, in 5.02 seconds)6162client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath6163client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath, in 0.12 seconds)6164(finished: subtest: a niks3 restart does not cost the leader its role, in 5.48 seconds)6165subtest: scheduler down: the other node takes the lock and keeps it6166lb: must succeed: curl -sf localhost:9901/clusters6167lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6168lb: must succeed: curl -sf localhost:9901/clusters6169lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6170lb: must succeed: curl -sf localhost:9901/clusters6171lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6172lb: must succeed: curl -sf localhost:9901/clusters6173lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6174worker1 # [ 103.943189] systemd[1]: Stopping Nix worker-protocol over gRPC...6175worker1 # [ 104.235929] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:53Z level=info event=signal_exit6176lb # [ 104.428242] niks3-server[1128]: 2026/09/22 16:05:53 INFO lead: released remote=192.168.1.3:517746177worker2 # [ 105.386108] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:54Z level=info event=scheduler_restarting addr=lb:500516178worker1 # [ 105.406114] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:54Z level=info event=scheduler_restarting6179worker1 # [ 105.410267] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:54Z level=info event=scheduler_restarting addr=lb:500516180worker1 # [ 105.412989] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:54Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:582946181worker1 # [ 105.416766] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:54Z level=info event=worker_gone addr=192.168.1.4:500516182worker1 # [ 105.420193] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:54Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:582946183worker1 # [ 105.423793] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:54Z level=info event=worker_gone addr=192.168.1.3:500516184worker1 # [ 105.439639] nix-daemon[861]: reaped child process 1440, status = succeeded6185worker2 # [ 105.586144] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:54Z level=info event=scheduler_connected addr=lb:500516186worker2 # [ 105.588068] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6187worker1 # [ 105.613934] nix-grpc-daemon[1380]: ts=2026-09-22T16:05:54Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6188worker2 # [ 105.788769] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:55Z level=info event=scheduler_connected addr=lb:500516189worker2 # [ 105.790622] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6190worker2 # [ 105.991282] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:55Z level=info event=scheduler_connected addr=lb:500516191worker2 # [ 105.993168] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6192worker2 # [ 106.193957] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:55Z level=info event=scheduler_connected addr=lb:500516193worker2 # [ 106.195895] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6194worker1 # [ 106.393846] nix-grpc-daemon[1380]: E0000 00:00:1790093155.602566 1425 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6195worker1 # [ 106.398848] nix-grpc-daemon[1380]: E0000 00:00:1790093155.602810 1425 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6196worker2 # [ 106.396924] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:55Z level=info event=scheduler_connected addr=lb:500516197worker2 # [ 106.398727] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6198worker2 # [ 106.599341] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:55Z level=info event=scheduler_connected addr=lb:500516199worker2 # [ 106.601041] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6200worker1 # [ 106.623233] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6201worker1 # [ 106.627177] systemd[1]: Stopped Nix worker-protocol over gRPC.6202worker1 # [ 106.630727] systemd[1]: nix-grpc-daemon.service: Consumed 956ms CPU time over 9.495s wall clock time, 38.4M memory peak, 35.2K incoming IP traffic, 58.6K outgoing IP traffic.6203worker1 # [ 106.641519] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.6204worker1 # [ 106.645824] systemd[1]: Closed Nix worker-protocol over gRPC.6205worker1 # [ 106.649939] systemd[1]: nix-grpc-daemon.socket: Consumed 533us CPU time over 9.526s wall clock time, 256K memory peak, 33.3K incoming IP traffic, 24.1K outgoing IP traffic.6206worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over6207worker2 # [ 106.801788] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:56Z level=info event=scheduler_connected addr=lb:500516208worker2 # [ 106.803537] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6209worker2 # [ 107.004327] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:56Z level=info event=scheduler_connected addr=lb:500516210worker2 # [ 107.006041] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6211lb # [ 107.142512] envoy[925]: conn peer=[2001:db8:1::1]:52068 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9679 tx=4427 ms=48606212lb # [ 107.144216] envoy[925]: conn peer=[2001:db8:1::3]:34206 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1328 tx=513 ms=83326213worker2 # [ 107.206743] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:56Z level=info event=scheduler_connected addr=lb:500516214worker2 # [ 107.208572] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6215worker2 # [ 107.409612] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:56Z level=info event=scheduler_connected addr=lb:500516216worker2 # [ 107.411354] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6217worker2 # [ 107.612599] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:56Z level=info event=scheduler_connected addr=lb:500516218worker2 # [ 107.616582] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6219worker2 # [ 107.817743] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:57Z level=info event=scheduler_connected addr=lb:500516220worker2 # [ 107.821653] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6221worker2 # [ 108.022960] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:57Z level=info event=scheduler_connected addr=lb:500516222worker2 # [ 108.026848] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6223worker2 # [ 108.227417] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:57Z level=info event=scheduler_connected addr=lb:500516224worker2 # [ 108.229416] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6225worker2 # [ 108.430458] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:57Z level=info event=scheduler_connected addr=lb:500516226worker2 # [ 108.432286] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6227worker2 # [ 108.632905] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:57Z level=info event=scheduler_connected addr=lb:500516228worker2 # [ 108.634606] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6229worker2 # [ 108.835292] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=scheduler_connected addr=lb:500516230worker2 # [ 108.837088] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6231worker2 # [ 109.037777] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=scheduler_connected addr=lb:500516232worker2 # [ 109.039564] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6233worker2 # [ 109.240813] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=scheduler_connected addr=lb:500516234worker2 # [ 109.244609] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6235worker2 # [ 109.383333] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=scheduler_take_over6236lb # [ 109.426767] niks3-server[1128]: 2026/09/22 16:05:58 INFO lead: acquired remote=192.168.1.4:527026237worker2 # [ 109.446426] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=scheduler_connected addr=lb:500516238worker2 # [ 109.450094] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6239worker2 # [ 109.651473] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=scheduler_connected addr=lb:500516240worker2 # [ 109.666134] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=oidc_jwks provider=mock keys=16241worker2 # [ 109.670403] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:365506242worker2 # [ 109.674582] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:58Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06243worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 3.17 seconds)6244worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6245worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)6246client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&26247client # don't know how to build these paths:6248client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6249worker2 # [ 109.961955] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:544746250worker2 # [ 109.967165] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544746251worker2 # [ 109.985596] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:59Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=06252worker2 # [ 109.990367] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:59Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.4:50051 clients=16253worker2 # [ 110.034452] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544746254worker2 # [ 110.045661] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544746255client # copying 1 paths to 'lb:50051'...6256client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...6257worker2 # [ 110.056572] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544746258worker2 # [ 110.067334] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6259worker2 # [ 110.170554] nix-daemon[854]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6260worker2 # [ 110.184063] nix-daemon[854]: reaped child process 1317, status = succeeded6261worker2 # [ 110.265883] nix-daemon[854]: accepted connection from pid 1319, user nix-grpc-daemon (trusted)6262worker2 # [ 110.279685] nix-daemon[854]: reaped child process 1322, status = succeeded6263lb # [ 110.326753] niks3-server[1128]: 2026/09/22 16:05:59 INFO Received uploads request method=POST path=/api/pending_closures6264worker2 # [ 110.290527] nix-grpc-daemon[1260]: time=2026-09-22T16:05:59.518Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6265worker2 # [ 110.291969] nix-grpc-daemon[1260]: time=2026-09-22T16:05:59.518Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"6266lb # [ 110.347871] niks3-server[1128]: 2026/09/22 16:05:59 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst6267lb # [ 110.351362] niks3-server[1128]: 2026/09/22 16:05:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6268lb # [ 110.354595] niks3-server[1128]: 2026/09/22 16:05:59 INFO Signed narinfos id=43 count=16269worker2 # [ 110.311434] nix-grpc-daemon[1260]: time=2026-09-22T16:05:59.539Z level=INFO msg="Uploading 1 narinfos"6270lb # [ 110.363392] niks3-server[1128]: 2026/09/22 16:05:59 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6271lb # [ 110.369945] niks3-server[1128]: 2026/09/22 16:05:59 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls6272lb # [ 110.375479] niks3-server[1128]: 2026/09/22 16:05:59 INFO Completed upload id=436273worker2 # [ 110.332347] nix-grpc-daemon[1260]: time=2026-09-22T16:05:59.560Z level=INFO msg="Upload complete. (253ms)"6274worker2 # [ 110.334822] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54474 duration_s=0 paths=1 nar_bytes_in=5206275worker2 # [ 110.338793] nix-grpc-daemon[1200]: ts=2026-09-22T16:05:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544746276worker2 # [ 110.346538] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6277worker2 # [ 110.349205] nix-daemon[854]: reaped child process 1312, status = succeeded6278worker2 # [ 110.354800] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6279client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...6280client # farm-a> worker2: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6281client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6282client # farm-a> LOG-farm-a-on-standby6283worker2 # [ 111.170103] nix-daemon[854]: reaped child process 1326, status = succeeded6284worker2 # [ 111.244416] nix-daemon[854]: accepted connection from pid 1342, user nix-grpc-daemon (trusted)6285worker2 # [ 111.255792] nix-daemon[854]: reaped child process 1345, status = succeeded6286lb # [ 111.308376] niks3-server[1128]: 2026/09/22 16:06:00 INFO Received uploads request method=POST path=/api/pending_closures6287worker2 # [ 111.278866] nix-grpc-daemon[1260]: time=2026-09-22T16:06:00.507Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6288worker2 # [ 111.280501] nix-grpc-daemon[1260]: time=2026-09-22T16:06:00.509Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"6289lb # [ 111.347328] niks3-server[1128]: 2026/09/22 16:06:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6290lb # [ 111.350565] niks3-server[1128]: 2026/09/22 16:06:00 INFO Signed narinfos id=44 count=16291worker2 # [ 111.308879] nix-grpc-daemon[1260]: time=2026-09-22T16:06:00.537Z level=INFO msg="Uploading 1 narinfos"6292lb # [ 111.352198] niks3-server[1128]: 2026/09/22 16:06:00 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6293lb # [ 111.356245] niks3-server[1128]: 2026/09/22 16:06:00 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst6294lb # [ 111.360241] niks3-server[1128]: 2026/09/22 16:06:00 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls6295lb # [ 111.365222] niks3-server[1128]: 2026/09/22 16:06:00 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6296lb # [ 111.371631] niks3-server[1128]: 2026/09/22 16:06:00 INFO Completed upload id=446297worker2 # [ 111.328570] nix-grpc-daemon[1260]: time=2026-09-22T16:06:00.557Z level=INFO msg="Upload complete. (167ms)"6298lb # [ 111.373106] niks3-server[1128]: 2026/09/22 16:06:00 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo6299worker2 # [ 111.331290] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54474 duration_s=0 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=16300worker2 # [ 111.338352] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:00Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=06301worker2 # [ 111.340353] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:00Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.4:50051 clients=16302worker2 # [ 111.346566] nix-daemon[854]: reaped child process 1324, status = succeeded6303worker2 # [ 111.384297] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544746304worker2 # [ 111.394277] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544746305worker2 # [ 111.405131] nix-grpc-daemon[1260]: time=2026-09-22T16:06:00.633Z level=INFO msg="All 1 paths already cached"6306client # copying 1 paths to 'lb:50051'...6307client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...6308worker2 # [ 111.412130] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544746309worker2 # [ 111.421172] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6310worker2 # [ 111.507848] nix-daemon[854]: accepted connection from pid 1351, user nix-grpc-daemon (trusted)6311worker2 # [ 111.521979] nix-daemon[854]: reaped child process 1354, status = succeeded6312worker2 # [ 111.602462] nix-daemon[854]: accepted connection from pid 1356, user nix-grpc-daemon (trusted)6313worker2 # [ 111.615383] nix-daemon[854]: reaped child process 1359, status = succeeded6314lb # [ 111.664880] niks3-server[1128]: 2026/09/22 16:06:01 INFO Received uploads request method=POST path=/api/pending_closures6315worker2 # [ 111.626281] nix-grpc-daemon[1260]: time=2026-09-22T16:06:00.854Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6316worker2 # [ 111.627767] nix-grpc-daemon[1260]: time=2026-09-22T16:06:00.856Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"6317lb # [ 111.681834] niks3-server[1128]: 2026/09/22 16:06:01 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst6318lb # [ 111.684317] niks3-server[1128]: 2026/09/22 16:06:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6319lb # [ 111.686150] niks3-server[1128]: 2026/09/22 16:06:01 INFO Signed narinfos id=45 count=16320worker2 # [ 111.644445] nix-grpc-daemon[1260]: time=2026-09-22T16:06:00.872Z level=INFO msg="Uploading 1 narinfos"6321lb # [ 111.692877] niks3-server[1128]: 2026/09/22 16:06:01 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6322lb # [ 111.696069] niks3-server[1128]: 2026/09/22 16:06:01 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls6323lb # [ 111.698428] niks3-server[1128]: 2026/09/22 16:06:01 INFO Completed upload id=456324worker2 # [ 111.655356] nix-grpc-daemon[1260]: time=2026-09-22T16:06:00.884Z level=INFO msg="Upload complete. (225ms)"6325lb # [ 111.700661] niks3-server[1128]: 2026/09/22 16:06:01 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo6326worker2 # [ 111.657728] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54474 duration_s=0 paths=1 nar_bytes_in=6406327worker2 # [ 111.661885] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544746328worker2 # [ 111.667717] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6329worker2 # [ 111.669972] nix-daemon[854]: reaped child process 1349, status = succeeded6330worker2 # [ 111.675211] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6331client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...6332client # farm-b> worker2: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6333client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6334client # farm-b> LOG-farm-b-on-standby6335worker2 # [ 112.912764] nix-daemon[854]: reaped child process 1363, status = succeeded6336worker2 # [ 112.995101] nix-daemon[854]: accepted connection from pid 1379, user nix-grpc-daemon (trusted)6337worker2 # [ 113.008399] nix-daemon[854]: reaped child process 1382, status = succeeded6338lb # [ 113.059570] niks3-server[1128]: 2026/09/22 16:06:02 INFO Received uploads request method=POST path=/api/pending_closures6339worker2 # [ 113.021324] nix-grpc-daemon[1260]: time=2026-09-22T16:06:02.249Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6340worker2 # [ 113.023145] nix-grpc-daemon[1260]: time=2026-09-22T16:06:02.251Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"6341lb # [ 113.078066] niks3-server[1128]: 2026/09/22 16:06:02 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6342lb # [ 113.084201] niks3-server[1128]: 2026/09/22 16:06:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6343lb # [ 113.086266] niks3-server[1128]: 2026/09/22 16:06:02 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst6344lb # [ 113.089287] niks3-server[1128]: 2026/09/22 16:06:02 INFO Signed narinfos id=46 count=16345worker2 # [ 113.047769] nix-grpc-daemon[1260]: time=2026-09-22T16:06:02.276Z level=INFO msg="Uploading 1 narinfos"6346lb # [ 113.092752] niks3-server[1128]: 2026/09/22 16:06:02 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls6347lb # [ 113.098413] niks3-server[1128]: 2026/09/22 16:06:02 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6348lb # [ 113.102399] niks3-server[1128]: 2026/09/22 16:06:02 INFO Completed upload id=466349worker2 # [ 113.059345] nix-grpc-daemon[1260]: time=2026-09-22T16:06:02.288Z level=INFO msg="Upload complete. (176ms)"6350lb # [ 113.104199] niks3-server[1128]: 2026/09/22 16:06:02 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo6351worker2 # [ 113.062331] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54474 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=16352worker2 # [ 113.069341] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:02Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=06353worker2 # [ 113.071113] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:02Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.4:50051 clients=16354worker2 # [ 113.076682] nix-daemon[854]: reaped child process 1361, status = succeeded6355worker2 # [ 113.113678] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544746356worker2 # [ 113.122837] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544746357worker2 # [ 113.132435] nix-grpc-daemon[1260]: time=2026-09-22T16:06:02.361Z level=INFO msg="All 2 paths already cached"6358client # copying 1 paths to 'lb:50051'...6359client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...6360worker2 # [ 113.139751] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544746361worker2 # [ 113.148273] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6362worker2 # [ 113.243038] nix-daemon[854]: accepted connection from pid 1388, user nix-grpc-daemon (trusted)6363worker2 # [ 113.257539] nix-daemon[854]: reaped child process 1391, status = succeeded6364worker2 # [ 113.347937] nix-daemon[854]: accepted connection from pid 1393, user nix-grpc-daemon (trusted)6365worker2 # [ 113.364184] nix-daemon[854]: reaped child process 1396, status = succeeded6366lb # [ 113.412449] niks3-server[1128]: 2026/09/22 16:06:02 INFO Received uploads request method=POST path=/api/pending_closures6367worker2 # [ 113.374284] nix-grpc-daemon[1260]: time=2026-09-22T16:06:02.602Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6368worker2 # [ 113.375650] nix-grpc-daemon[1260]: time=2026-09-22T16:06:02.602Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6369lb # [ 113.427608] niks3-server[1128]: 2026/09/22 16:06:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6370lb # [ 113.429893] niks3-server[1128]: 2026/09/22 16:06:02 INFO Signed narinfos id=47 count=16371worker2 # [ 113.388098] nix-grpc-daemon[1260]: time=2026-09-22T16:06:02.616Z level=INFO msg="Uploading 1 narinfos"6372lb # [ 113.431417] niks3-server[1128]: 2026/09/22 16:06:02 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6373lb # [ 113.441533] niks3-server[1128]: 2026/09/22 16:06:02 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6374lb # [ 113.446193] niks3-server[1128]: 2026/09/22 16:06:02 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6375lb # [ 113.448548] niks3-server[1128]: 2026/09/22 16:06:02 INFO Completed upload id=476376worker2 # [ 113.406828] nix-grpc-daemon[1260]: time=2026-09-22T16:06:02.635Z level=INFO msg="Upload complete. (250ms)"6377lb # [ 113.451683] niks3-server[1128]: 2026/09/22 16:06:02 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6378worker2 # [ 113.409240] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54474 duration_s=0 paths=1 nar_bytes_in=7686379worker2 # [ 113.412835] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544746380worker2 # [ 113.420321] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6381worker2 # [ 113.422718] nix-daemon[854]: reaped child process 1386, status = succeeded6382worker2 # [ 113.428320] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6383client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6384client # farm-top> worker2: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6385client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6386client # farm-top> LOG-farm-top-on-standby6387worker2 # [ 114.645976] nix-daemon[854]: reaped child process 1400, status = succeeded6388worker2 # [ 114.718167] nix-daemon[854]: accepted connection from pid 1416, user nix-grpc-daemon (trusted)6389worker2 # [ 114.732107] nix-daemon[854]: reaped child process 1419, status = succeeded6390lb # [ 114.781223] niks3-server[1128]: 2026/09/22 16:06:04 INFO Received uploads request method=POST path=/api/pending_closures6391worker2 # [ 114.744410] nix-grpc-daemon[1260]: time=2026-09-22T16:06:03.972Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6392worker2 # [ 114.747059] nix-grpc-daemon[1260]: time=2026-09-22T16:06:03.974Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6393lb # [ 114.803814] niks3-server[1128]: 2026/09/22 16:06:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6394lb # [ 114.807051] niks3-server[1128]: 2026/09/22 16:06:04 INFO Signed narinfos id=48 count=16395worker2 # [ 114.765503] nix-grpc-daemon[1260]: time=2026-09-22T16:06:03.994Z level=INFO msg="Uploading 1 narinfos"6396lb # [ 114.812068] niks3-server[1128]: 2026/09/22 16:06:04 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6397lb # [ 114.815147] niks3-server[1128]: 2026/09/22 16:06:04 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6398lb # [ 114.817778] niks3-server[1128]: 2026/09/22 16:06:04 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6399lb # [ 114.824378] niks3-server[1128]: 2026/09/22 16:06:04 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6400lb # [ 114.828073] niks3-server[1128]: 2026/09/22 16:06:04 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6401lb # [ 114.832415] niks3-server[1128]: 2026/09/22 16:06:04 INFO Completed upload id=486402worker2 # [ 114.789365] nix-grpc-daemon[1260]: time=2026-09-22T16:06:04.018Z level=INFO msg="Upload complete. (153ms)"6403worker2 # [ 114.791968] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54474 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16404worker2 # [ 114.800062] nix-daemon[854]: reaped child process 1398, status = succeeded6405client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&2, in 5.00 seconds)6406client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath6407client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath, in 0.11 seconds)6408worker1 # [ 115.015969] systemd[1]: Listening on Nix worker-protocol over gRPC.6409worker1 # [ 115.025148] systemd[1]: Starting Nix worker-protocol over gRPC...6410worker1 # [ 115.116961] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:04Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6411worker1 # [ 115.126449] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:04Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6412worker1 # [ 115.128707] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:04Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16413worker1 # [ 115.135489] nix-grpc-daemon[1588]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6414worker1 # [ 115.137494] nix-grpc-daemon[1588]: E0000 00:00:1790093164.332560 1588 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6415worker1 # [ 115.143140] nix-grpc-daemon[1588]: E0000 00:00:1790093164.334528 1588 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6416worker1 # [ 115.145246] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:04Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6417worker1 # [ 115.146815] nix-grpc-daemon[1588]: E0000 00:00:1790093164.349871 1630 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6418worker1 # [ 115.153170] systemd[1]: Started Nix worker-protocol over gRPC.6419worker1 # [ 115.160072] nix-grpc-daemon[1588]: E0000 00:00:1790093164.367798 1630 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6420worker1 # [ 115.162158] nix-grpc-daemon[1588]: E0000 00:00:1790093164.367942 1630 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6421lb: must succeed: curl -sf localhost:9901/clusters6422worker1 # [ 115.174244] nix-grpc-daemon[1588]: E0000 00:00:1790093164.383240 1630 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6423worker1 # [ 115.187188] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:04Z level=info event=scheduler_connected addr=lb:500516424worker2 # [ 115.166624] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:04Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:365506425worker2 # [ 115.168225] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:04Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06426lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6427worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6428worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)6429lb: must succeed: curl -sf localhost:9901/clusters6430lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6431lb: must succeed: curl -sf localhost:9901/clusters6432lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6433worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6434worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6435(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 11.52 seconds)6436subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6437client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs6438client # [ 115.364600] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs.6439client # Running as unit: rs.service; invocation ID: bcafcfaafe9b47f08dca403e5c7c2b2d6440client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs, in 0.06 seconds)6441client # [ 115.500401] nix[1364]: don't know how to build these paths:6442client # [ 115.501339] nix[1364]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6443worker2 # [ 115.494383] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:04Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:544746444worker2 # [ 115.505136] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:04Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06445worker2 # [ 115.510643] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:04Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.4:50051 clients=16446worker2 # [ 115.549295] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544746447worker2 # [ 115.562651] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544746448client # [ 115.578161] nix[1364]: copying 1 paths to 'lb:50051'...6449worker2 # [ 115.574144] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544746450client # [ 115.580053] nix[1364]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6451worker2 # [ 115.583419] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6452worker2 # [ 115.685349] nix-daemon[854]: accepted connection from pid 1432, user nix-grpc-daemon (trusted)6453worker2 # [ 115.698272] nix-daemon[854]: reaped child process 1435, status = succeeded6454worker2 # [ 115.778964] nix-daemon[854]: accepted connection from pid 1437, user nix-grpc-daemon (trusted)6455worker2 # [ 115.791950] nix-daemon[854]: reaped child process 1440, status = succeeded6456lb # [ 115.841322] niks3-server[1128]: 2026/09/22 16:06:05 INFO Received uploads request method=POST path=/api/pending_closures6457worker2 # [ 115.803215] nix-grpc-daemon[1260]: time=2026-09-22T16:06:05.031Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6458worker2 # [ 115.804795] nix-grpc-daemon[1260]: time=2026-09-22T16:06:05.033Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6459lb # [ 115.857915] niks3-server[1128]: 2026/09/22 16:06:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6460lb # [ 115.860344] niks3-server[1128]: 2026/09/22 16:06:05 INFO Signed narinfos id=49 count=16461worker2 # [ 115.817540] nix-grpc-daemon[1260]: time=2026-09-22T16:06:05.046Z level=INFO msg="Uploading 1 narinfos"6462lb # [ 115.863883] niks3-server[1128]: 2026/09/22 16:06:05 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6463lb # [ 115.868072] niks3-server[1128]: 2026/09/22 16:06:05 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6464lb # [ 115.872703] niks3-server[1128]: 2026/09/22 16:06:05 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6465lb # [ 115.875482] niks3-server[1128]: 2026/09/22 16:06:05 INFO Completed upload id=496466worker2 # [ 115.833691] nix-grpc-daemon[1260]: time=2026-09-22T16:06:05.062Z level=INFO msg="Upload complete. (236ms)"6467lb # [ 115.877435] niks3-server[1128]: 2026/09/22 16:06:05 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6468worker2 # [ 115.837067] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54474 duration_s=0 paths=1 nar_bytes_in=4166469worker2 # [ 115.839405] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544746470worker2 # [ 115.845742] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6471worker2 # [ 115.847944] nix-daemon[854]: reaped child process 1430, status = succeeded6472worker2 # [ 115.852954] nix-daemon[854]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)6473client # [ 115.857586] nix[1364]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6474lb # [ 117.143227] envoy[925]: conn peer=[2001:db8:1::1]:60582 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9534 tx=4370 ms=48436475lb: must succeed: curl -sf localhost:9901/clusters6476lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6477lb: must succeed: curl -sf localhost:9901/clusters6478lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6479lb: must succeed: curl -sf localhost:9901/clusters6480lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6481lb: must succeed: curl -sf localhost:9901/clusters6482lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6483worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6484worker1 # [ 117.980316] systemd-journald[284]: Received client request to rotate journal, rotating.6485worker1 # [ 117.997045] systemd-journald[284]: Vacuuming done, freed 0B of archived journals from /var/log/journal/5d6ed502fabd4745ade0fae85cb7ca8c.6486worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.06 seconds)6487worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service6488worker2 # [ 117.880175] systemd[1]: Stopping Nix worker-protocol over gRPC...6489worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.03 seconds)6490client: must succeed: sleep 5.06491worker2 # [ 118.511340] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:07Z level=info event=draining6492client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6493worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6494worker2 # [ 123.075133] nix-daemon[854]: reaped child process 1444, status = succeeded6495worker2 # [ 123.147357] nix-daemon[854]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6496worker2 # [ 123.161135] nix-daemon[854]: reaped child process 1488, status = succeeded6497lb # [ 123.210578] niks3-server[1128]: 2026/09/22 16:06:12 INFO Received uploads request method=POST path=/api/pending_closures6498worker2 # [ 123.175880] nix-grpc-daemon[1260]: time=2026-09-22T16:06:12.404Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6499worker2 # [ 123.177645] nix-grpc-daemon[1260]: time=2026-09-22T16:06:12.406Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6500lb # [ 123.250204] niks3-server[1128]: 2026/09/22 16:06:12 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6501lb # [ 123.268882] niks3-server[1128]: 2026/09/22 16:06:12 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6502lb # [ 123.279832] niks3-server[1128]: 2026/09/22 16:06:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6503lb # [ 123.284837] niks3-server[1128]: 2026/09/22 16:06:12 INFO Signed narinfos id=50 count=16504worker2 # [ 123.244282] nix-grpc-daemon[1260]: time=2026-09-22T16:06:12.472Z level=INFO msg="Uploading 1 narinfos"6505lb # [ 123.294298] niks3-server[1128]: 2026/09/22 16:06:12 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6506lb # [ 123.299254] niks3-server[1128]: 2026/09/22 16:06:12 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6507lb # [ 123.303515] niks3-server[1128]: 2026/09/22 16:06:12 INFO Completed upload id=506508worker2 # [ 123.260678] nix-grpc-daemon[1260]: time=2026-09-22T16:06:12.489Z level=INFO msg="Upload complete. (195ms)"6509lb # [ 123.306258] niks3-server[1128]: 2026/09/22 16:06:12 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6510worker2 # [ 123.269133] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54474 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16511worker2 # [ 123.283521] nix-daemon[854]: reaped child process 1442, status = succeeded6512client # [ 123.301350] systemd[1]: rs.service: Deactivated successfully.6513client # [ 123.305302] systemd[1]: rs.service: Consumed 125ms CPU time over 7.940s wall clock time, 43.4M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.6514worker2 # [ 123.512453] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:12Z level=info event=signal_exit6515lb # [ 124.427920] niks3-server[1128]: 2026/09/22 16:06:13 INFO lead: released remote=192.168.1.4:527026516lb # [ 125.160807] niks3-server[1128]: 2026/09/22 16:06:14 INFO lead: acquired remote=192.168.1.3:354966517worker1 # [ 125.140845] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:14Z level=info event=scheduler_take_over6518worker1 # [ 125.407242] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:14Z level=info event=scheduler_restarting addr=lb:500516519worker2 # [ 125.390129] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:14Z level=info event=scheduler_restarting6520worker2 # [ 125.393426] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:14Z level=info event=scheduler_restarting addr=lb:500516521worker2 # [ 125.396910] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:14Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:365506522worker2 # [ 125.400426] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:14Z level=info event=worker_gone addr=192.168.1.3:500516523worker2 # [ 125.403575] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:14Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:365506524worker2 # [ 125.407355] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:14Z level=info event=worker_gone addr=192.168.1.4:500516525worker2 # [ 125.426339] nix-daemon[854]: reaped child process 1252, status = succeeded6526worker2 # [ 125.571956] nix-grpc-daemon[1200]: E0000 00:00:1790093174.800397 1245 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true6527worker2 # [ 125.576868] nix-grpc-daemon[1200]: E0000 00:00:1790093174.800539 1245 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6528worker1 # [ 125.607848] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:14Z level=info event=scheduler_connected addr=lb:500516529worker1 # [ 125.610240] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6530worker2 # [ 125.599548] nix-grpc-daemon[1200]: ts=2026-09-22T16:06:14Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6531worker1 # [ 125.810867] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:15Z level=info event=scheduler_connected addr=lb:500516532worker1 # [ 125.818108] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:15Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:438066533worker1 # [ 125.822333] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:15Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06534worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.14 seconds)6535client: waiting for success: ! systemctl is-active rs6536client: (finished: waiting for success: ! systemctl is-active rs, in 0.04 seconds)6537client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6538client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)6539client: must fail: journalctl -u rs | grep -q 'reconnecting'6540client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.01 seconds)6541lb: must succeed: curl -sf localhost:9901/clusters6542lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6543worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6544worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)6545lb # [ 127.143414] envoy[925]: conn peer=[2001:db8:1::1]:44318 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2973 tx=1541 ms=77856546lb # [ 127.147461] envoy[925]: conn peer=[2001:db8:1::4]:58860 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=39191 tx=2161 ms=330506547worker2 # [ 127.240348] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6548worker2 # [ 127.242055] systemd[1]: Stopped Nix worker-protocol over gRPC.6549worker2 # [ 127.242958] systemd[1]: nix-grpc-daemon.service: Consumed 1.413s CPU time over 34.845s wall clock time, 40M memory peak, 62.2K incoming IP traffic, 154.4K outgoing IP traffic.6550worker2 # [ 127.249654] systemd[1]: Starting scheduler-token.service...6551worker2 # [ 127.282988] systemd[1]: scheduler-token.service: Deactivated successfully.6552worker2 # [ 127.284615] systemd[1]: Finished scheduler-token.service.6553worker2 # [ 127.285917] systemd[1]: scheduler-token.service: Consumed 16ms CPU time over 34ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6554worker2 # [ 127.290114] systemd[1]: Starting Nix worker-protocol over gRPC...6555lb: must succeed: curl -sf localhost:9901/clusters6556worker2 # [ 127.361211] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:16Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6557worker2 # [ 127.367149] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:16Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6558worker2 # [ 127.370204] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:16Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16559worker2 # [ 127.375256] nix-grpc-daemon[1498]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6560worker2 # [ 127.378093] nix-grpc-daemon[1498]: E0000 00:00:1790093176.595484 1498 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6561worker2 # [ 127.380223] nix-grpc-daemon[1498]: E0000 00:00:1790093176.600920 1498 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6562lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6563worker2 # [ 127.385046] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:16Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6564worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6565worker2 # [ 127.386692] systemd[1]: Started Nix worker-protocol over gRPC.6566worker2 # [ 127.390132] nix-grpc-daemon[1498]: E0000 00:00:1790093176.611629 1537 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6567worker2 # [ 127.397677] nix-grpc-daemon[1498]: E0000 00:00:1790093176.626323 1537 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6568worker2 # [ 127.401116] nix-grpc-daemon[1498]: E0000 00:00:1790093176.628502 1537 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6569worker2 # [ 127.411646] nix-grpc-daemon[1498]: E0000 00:00:1790093176.640343 1537 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6570worker2 # [ 127.421187] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:16Z level=info event=scheduler_connected addr=lb:500516571worker1 # [ 127.453309] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:16Z level=info event=oidc_jwks provider=mock keys=16572worker1 # [ 127.457337] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:16Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:438066573worker1 # [ 127.461222] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:16Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06574worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6575(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.13 seconds)6576subtest: scheduler restart mid-build: build finishes, no double build6577client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid6578client # [ 127.497991] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid.6579client # Running as unit: mid.service; invocation ID: 71475837594c4160bece98875b2598476580client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid, in 0.06 seconds)6581worker1 # [ 127.654159] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:438106582worker1 # [ 127.667116] nix-daemon[861]: accepted connection from pid 1588, user nix-grpc-daemon (trusted)6583client # [ 127.661447] nix[1400]: don't know how to build these paths:6584client # [ 127.662334] nix[1400]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6585worker1 # [ 127.686542] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:16Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06586worker1 # [ 127.691201] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:16Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16587worker1 # [ 127.735572] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:438106588worker1 # [ 127.746339] nix-grpc-daemon[1588]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 89 ms (attempt 1/5)6589worker1 # [ 127.843774] nix-grpc-daemon[1588]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 33 ms (attempt 2/5)6590worker1 # [ 127.877972] nix-grpc-daemon[1588]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 24 ms (attempt 3/5)6591worker1 # [ 127.903610] nix-grpc-daemon[1588]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 753 ms (attempt 4/5)6592worker1 # [ 128.658846] nix-grpc-daemon[1588]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6593worker1 # [ 128.663725] nix-grpc-daemon[1588]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6594worker1 # [ 128.678109] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:438106595client # [ 128.670368] nix[1400]: copying 1 paths to 'lb:50051'...6596client # [ 128.674364] nix[1400]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6597worker1 # [ 128.692255] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:438106598worker1 # [ 128.701408] nix-daemon[861]: accepted connection from pid 1588, user nix-grpc-daemon (trusted)6599worker1 # [ 128.715484] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:17Z level=info event=niks3_push_started pid=17206600worker1 # [ 128.807893] nix-grpc-daemon[1720]: time=2026-09-22T16:06:18.016Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6601worker1 # [ 128.895753] nix-daemon[861]: accepted connection from pid 1733, user nix-grpc-daemon (trusted)6602worker1 # [ 128.907146] nix-daemon[861]: reaped child process 1736, status = succeeded6603worker1 # [ 128.992087] nix-daemon[861]: accepted connection from pid 1738, user nix-grpc-daemon (trusted)6604worker1 # [ 129.004725] nix-daemon[861]: reaped child process 1741, status = succeeded6605lb # [ 129.033422] niks3-server[1128]: 2026/09/22 16:06:18 INFO Received uploads request method=POST path=/api/pending_closures6606worker1 # [ 129.020201] nix-grpc-daemon[1720]: time=2026-09-22T16:06:18.229Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6607worker1 # [ 129.022602] nix-grpc-daemon[1720]: time=2026-09-22T16:06:18.230Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6608lb # [ 129.068874] niks3-server[1128]: 2026/09/22 16:06:18 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6609lb # [ 129.078082] niks3-server[1128]: 2026/09/22 16:06:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6610worker1 # [ 129.064166] nix-grpc-daemon[1720]: time=2026-09-22T16:06:18.272Z level=INFO msg="Uploading 1 narinfos"6611lb # [ 129.084689] niks3-server[1128]: 2026/09/22 16:06:18 INFO Signed narinfos id=51 count=16612lb # [ 129.100859] niks3-server[1128]: 2026/09/22 16:06:18 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6613lb # [ 129.105626] niks3-server[1128]: 2026/09/22 16:06:18 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6614lb # [ 129.113447] niks3-server[1128]: 2026/09/22 16:06:18 INFO Completed upload id=516615worker1 # [ 129.092344] nix-grpc-daemon[1720]: time=2026-09-22T16:06:18.301Z level=INFO msg="Upload complete. (281ms)"6616lb # [ 129.116531] niks3-server[1128]: 2026/09/22 16:06:18 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6617worker1 # [ 129.096549] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43810 duration_s=0 paths=1 nar_bytes_in=4246618worker1 # [ 129.101430] nix-grpc-daemon[1588]: ts=2026-09-22T16:06:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:438106619worker1 # [ 129.110780] nix-daemon[861]: accepted connection from pid 1588, user nix-grpc-daemon (trusted)6620worker1 # [ 129.114755] nix-daemon[861]: reaped child process 1718, status = succeeded6621worker1 # [ 129.119434] nix-daemon[861]: accepted connection from pid 1588, user nix-grpc-daemon (trusted)6622client # [ 129.101693] nix[1400]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6623lb: must succeed: curl -sf localhost:9901/clusters6624lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6625lb: must succeed: curl -sf localhost:9901/clusters6626lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6627worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6628worker1 # [ 130.951163] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1588 (nix-grpc-daemon) on client request.6629worker1 # [ 130.952777] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6630lb # [ 130.985708] niks3-server[1128]: 2026/09/22 16:06:20 INFO lead: released remote=192.168.1.3:354966631worker1 # [ 130.968692] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6632worker2 # [ 130.945744] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:20Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6633worker1 # [ 130.970816] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6634client # [ 130.951280] nix[1400]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6635worker1 # [ 130.973235] systemd[1]: nix-grpc-daemon.service: Consumed 404ms CPU time over 15.944s wall clock time, 35.7M memory peak, 14.9K incoming IP traffic, 26.9K outgoing IP traffic.6636client # [ 130.955995] nix[1400]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6637worker1 # [ 130.982034] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6638worker1 # [ 130.987258] systemd[1]: Starting Nix worker-protocol over gRPC...6639worker1 # [ 131.027722] nix-daemon[861]: reaped child process 1706, status = succeeded6640worker1 # [ 131.029971] nix-daemon[861]: reaped child process 1743, status = succeeded6641worker1 # [ 131.033841] nix-daemon[1745]: unexpected Nix daemon error: error: interrupted by the user6642worker1 # [ 131.040270] nix-daemon[861]: reaped child process 1745, status = failed with exit code 16643worker1 # [ 131.077862] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6644worker1 # [ 131.079877] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6645worker1 # [ 131.082925] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16646worker1 # [ 131.087052] nix-grpc-daemon[1782]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6647worker1 # [ 131.088652] nix-grpc-daemon[1782]: E0000 00:00:1790093180.296022 1782 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6648worker1 # [ 131.097060] nix-grpc-daemon[1782]: E0000 00:00:1790093180.301329 1782 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6649worker1 # [ 131.099945] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6650worker1 # [ 131.102599] systemd[1]: Started Nix worker-protocol over gRPC.6651lb # [ 131.124350] niks3-server[1128]: 2026/09/22 16:06:20 INFO lead: acquired remote=192.168.1.3:588186652worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.18 seconds)6653client: must succeed: sleep 5.06654worker1 # [ 131.105994] nix-grpc-daemon[1782]: E0000 00:00:1790093180.312826 1823 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6655worker1 # [ 131.110213] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=info event=scheduler_take_over6656worker1 # [ 131.117629] nix-grpc-daemon[1782]: E0000 00:00:1790093180.326629 1823 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6657worker1 # [ 131.122053] nix-grpc-daemon[1782]: E0000 00:00:1790093180.329023 1823 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6658worker1 # [ 131.131693] nix-grpc-daemon[1782]: E0000 00:00:1790093180.340698 1823 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6659worker1 # [ 131.139098] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=info event=scheduler_connected addr=lb:500516660worker1 # [ 131.143737] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:438206661worker1 # [ 131.145396] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06662worker2 # [ 131.146249] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:20Z level=info event=scheduler_connected addr=lb:500516663worker1 # [ 131.171941] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=info event=oidc_jwks provider=mock keys=16664worker1 # [ 131.173311] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:438206665worker1 # [ 131.174962] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06666worker1 # [ 131.481838] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06667worker1 # [ 131.487301] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16668worker1 # [ 131.534619] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:438326669worker1 # [ 131.544157] nix-daemon[861]: accepted connection from pid 1782, user nix-grpc-daemon (trusted)6670worker1 # [ 131.558142] nix-daemon[861]: accepted connection from pid 1782, user nix-grpc-daemon (trusted)6671client # [ 131.553860] nix[1400]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6672worker1 # [ 131.572613] nix-daemon[861]: accepted connection from pid 1782, user nix-grpc-daemon (trusted)6673client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6674client: waiting for success: ! systemctl is-active mid6675worker1 # [ 136.390672] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:25Z level=info event=niks3_push_started pid=18656676worker1 # [ 136.403951] nix-daemon[861]: reaped child process 1840, status = succeeded6677worker1 # [ 136.484980] nix-grpc-daemon[1865]: time=2026-09-22T16:06:25.693Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6678worker1 # [ 136.567744] nix-daemon[861]: accepted connection from pid 1877, user nix-grpc-daemon (trusted)6679worker1 # [ 136.580117] nix-daemon[861]: reaped child process 1880, status = succeeded6680lb # [ 136.608515] niks3-server[1128]: 2026/09/22 16:06:26 INFO Received uploads request method=POST path=/api/pending_closures6681worker1 # [ 136.592104] nix-grpc-daemon[1865]: time=2026-09-22T16:06:25.801Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6682worker1 # [ 136.593618] nix-grpc-daemon[1865]: time=2026-09-22T16:06:25.802Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6683lb # [ 136.638399] niks3-server[1128]: 2026/09/22 16:06:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6684lb # [ 136.640694] niks3-server[1128]: 2026/09/22 16:06:26 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6685worker1 # [ 136.622137] nix-grpc-daemon[1865]: time=2026-09-22T16:06:25.831Z level=INFO msg="Uploading 1 narinfos"6686lb # [ 136.643132] niks3-server[1128]: 2026/09/22 16:06:26 INFO Signed narinfos id=52 count=16687lb # [ 136.659899] niks3-server[1128]: 2026/09/22 16:06:26 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6688lb # [ 136.664530] niks3-server[1128]: 2026/09/22 16:06:26 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6689lb # [ 136.668982] niks3-server[1128]: 2026/09/22 16:06:26 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6690lb # [ 136.672789] niks3-server[1128]: 2026/09/22 16:06:26 INFO Completed upload id=526691worker1 # [ 136.653220] nix-grpc-daemon[1865]: time=2026-09-22T16:06:25.862Z level=INFO msg="Upload complete. (164ms)"6692lb # [ 136.674831] niks3-server[1128]: 2026/09/22 16:06:26 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6693worker1 # [ 136.655954] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43832 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16694worker1 # [ 136.664590] nix-daemon[861]: reaped child process 1838, status = succeeded6695client # [ 136.671523] systemd[1]: mid.service: Deactivated successfully.6696client # [ 136.675099] systemd[1]: mid.service: Consumed 142ms CPU time over 9.176s wall clock time, 43.5M memory peak, 8.8K incoming IP traffic, 10.4K outgoing IP traffic.6697lb # [ 137.143688] envoy[925]: conn peer=[2001:db8:1::3]:39806 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=993 tx=376 ms=157886698lb # [ 137.145281] envoy[925]: conn peer=[2001:db8:1::1]:56374 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3921 tx=1962 ms=90166699client: (finished: waiting for success: ! systemctl is-active mid, in 1.05 seconds)6700client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6701client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.02 seconds)6702(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 9.88 seconds)6703subtest: interrupting the client stops the build on the worker6704client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr6705client # [ 137.350466] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr.6706client # Running as unit: intr.service; invocation ID: 87865425693b41c3a14391407ff418136707client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr, in 0.03 seconds)6708worker2 # [ 137.470381] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:361786709worker2 # [ 137.475502] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6710client # [ 137.486710] nix[1436]: don't know how to build these paths:6711client # [ 137.487590] nix[1436]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6712worker1 # [ 137.509991] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:26Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06713worker1 # [ 137.511932] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:26Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16714worker2 # [ 137.531252] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361786715worker2 # [ 137.542346] nix-grpc-daemon[1498]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 71 ms (attempt 1/5)6716worker2 # [ 137.621133] nix-grpc-daemon[1498]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 145 ms (attempt 2/5)6717worker2 # [ 137.768459] nix-grpc-daemon[1498]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 277 ms (attempt 3/5)6718worker2 # [ 138.047581] nix-grpc-daemon[1498]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 517 ms (attempt 4/5)6719worker2 # [ 138.568275] nix-grpc-daemon[1498]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6720worker2 # [ 138.571427] nix-grpc-daemon[1498]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6721worker2 # [ 138.585539] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361786722client # [ 138.598982] nix[1436]: copying 1 paths to 'lb:50051'...6723client # [ 138.600652] nix[1436]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6724worker2 # [ 138.596709] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361786725worker2 # [ 138.608157] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6726worker2 # [ 138.627760] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:27Z level=info event=niks3_push_started pid=15946727worker2 # [ 138.724615] nix-grpc-daemon[1594]: time=2026-09-22T16:06:27.952Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""6728worker2 # [ 138.808612] nix-daemon[854]: accepted connection from pid 1607, user nix-grpc-daemon (trusted)6729worker2 # [ 138.820047] nix-daemon[854]: reaped child process 1610, status = succeeded6730worker2 # [ 138.903436] nix-daemon[854]: accepted connection from pid 1612, user nix-grpc-daemon (trusted)6731worker2 # [ 138.916346] nix-daemon[854]: reaped child process 1615, status = succeeded6732lb # [ 138.966054] niks3-server[1128]: 2026/09/22 16:06:28 INFO Received uploads request method=POST path=/api/pending_closures6733worker2 # [ 138.927379] nix-grpc-daemon[1594]: time=2026-09-22T16:06:28.156Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6734worker2 # [ 138.930375] nix-grpc-daemon[1594]: time=2026-09-22T16:06:28.157Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6735lb # [ 138.986565] niks3-server[1128]: 2026/09/22 16:06:28 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6736lb # [ 138.989104] niks3-server[1128]: 2026/09/22 16:06:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6737lb # [ 138.991296] niks3-server[1128]: 2026/09/22 16:06:28 INFO Signed narinfos id=53 count=16738worker2 # [ 138.949754] nix-grpc-daemon[1594]: time=2026-09-22T16:06:28.178Z level=INFO msg="Uploading 1 narinfos"6739lb # [ 138.997485] niks3-server[1128]: 2026/09/22 16:06:28 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6740lb # [ 139.001558] niks3-server[1128]: 2026/09/22 16:06:28 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6741lb # [ 139.007048] niks3-server[1128]: 2026/09/22 16:06:28 INFO Completed upload id=536742worker2 # [ 138.963979] nix-grpc-daemon[1594]: time=2026-09-22T16:06:28.192Z level=INFO msg="Upload complete. (236ms)"6743lb # [ 139.009940] niks3-server[1128]: 2026/09/22 16:06:28 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6744worker2 # [ 138.966306] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=0 paths=1 nar_bytes_in=4246745worker2 # [ 138.969410] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361786746worker2 # [ 138.976350] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6747worker2 # [ 138.978637] nix-daemon[854]: reaped child process 1592, status = succeeded6748client # [ 138.987991] nix[1436]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6749worker2 # [ 138.983791] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6750client: must succeed: systemctl kill -s INT intr6751client # [ 140.702335] systemd[1]: intr.service: Sent signal SIGINT to main process 1436 (nix) on client request.6752client # [ 140.705676] nix[1436]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6753client # [ 140.708608] nix[1436]: error: interrupted by the user6754worker2 # [ 140.703272] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:29Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6755client: (finished: must succeed: systemctl kill -s INT intr, in 0.02 seconds)6756client # [ 140.726663] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6757client # [ 140.728125] systemd[1]: intr.service: Failed with result 'exit-code'.6758client # [ 140.730057] systemd[1]: intr.service: Consumed 100ms CPU time over 3.379s wall clock time, 43.4M memory peak, 6.5K incoming IP traffic, 8.3K outgoing IP traffic.6759worker2 # [ 140.800784] nix-daemon[854]: reaped child process 1617, status = succeeded6760worker2 # [ 140.803247] nix-daemon[1619]: unexpected Nix daemon error: error: interrupted by the user6761worker2 # [ 140.806898] nix-daemon[854]: reaped child process 1619, status = failed with exit code 16762(finished: subtest: interrupting the client stops the build on the worker, in 4.55 seconds)6763subtest: a deploy mid-build drains: the build finishes, the new generation takes over6764client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw6765client # [ 141.921639] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw.6766client # Running as unit: sw.service; invocation ID: a30954c73d8d48d6ab454e1fcb04c26e6767client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw, in 0.06 seconds)6768worker1 # [ 142.075658] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:438326769client # [ 142.061864] nix[1457]: don't know how to build these paths:6770client # [ 142.063042] nix[1457]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6771worker1 # [ 142.083161] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:31Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06772worker1 # [ 142.085364] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:31Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16773worker1 # [ 142.127037] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:438326774worker1 # [ 142.131777] nix-grpc-daemon[1782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 81 ms (attempt 1/5)6775worker1 # [ 142.217063] nix-grpc-daemon[1782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 109 ms (attempt 2/5)6776worker1 # [ 142.333942] nix-grpc-daemon[1782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 186 ms (attempt 3/5)6777worker1 # [ 142.520904] nix-grpc-daemon[1782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 439 ms (attempt 4/5)6778worker1 # [ 142.961391] nix-grpc-daemon[1782]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6779worker1 # [ 142.962627] nix-grpc-daemon[1782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6780worker1 # [ 142.973466] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:438326781client # [ 142.964755] nix[1457]: copying 1 paths to 'lb:50051'...6782client # [ 142.968622] nix[1457]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6783worker1 # [ 142.986861] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:438326784worker1 # [ 142.998121] nix-daemon[861]: accepted connection from pid 1782, user nix-grpc-daemon (trusted)6785worker1 # [ 143.110072] nix-daemon[861]: accepted connection from pid 1921, user nix-grpc-daemon (trusted)6786worker1 # [ 143.125562] nix-daemon[861]: reaped child process 1928, status = succeeded6787worker1 # [ 143.201603] nix-daemon[861]: accepted connection from pid 1930, user nix-grpc-daemon (trusted)6788worker1 # [ 143.213468] nix-daemon[861]: reaped child process 1933, status = succeeded6789lb # [ 143.242955] niks3-server[1128]: 2026/09/22 16:06:32 INFO Received uploads request method=POST path=/api/pending_closures6790worker1 # [ 143.230325] nix-grpc-daemon[1865]: time=2026-09-22T16:06:32.438Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6791worker1 # [ 143.231689] nix-grpc-daemon[1865]: time=2026-09-22T16:06:32.438Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6792lb # [ 143.267518] niks3-server[1128]: 2026/09/22 16:06:32 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6793lb # [ 143.284925] niks3-server[1128]: 2026/09/22 16:06:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6794worker1 # [ 143.272633] nix-grpc-daemon[1865]: time=2026-09-22T16:06:32.481Z level=INFO msg="Uploading 1 narinfos"6795lb # [ 143.293478] niks3-server[1128]: 2026/09/22 16:06:32 INFO Signed narinfos id=54 count=16796lb # [ 143.305879] niks3-server[1128]: 2026/09/22 16:06:32 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6797lb # [ 143.315311] niks3-server[1128]: 2026/09/22 16:06:32 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6798lb # [ 143.319490] niks3-server[1128]: 2026/09/22 16:06:32 INFO Completed upload id=546799worker1 # [ 143.298626] nix-grpc-daemon[1865]: time=2026-09-22T16:06:32.507Z level=INFO msg="Upload complete. (288ms)"6800lb # [ 143.321514] niks3-server[1128]: 2026/09/22 16:06:32 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6801worker1 # [ 143.301313] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43832 duration_s=0 paths=1 nar_bytes_in=4166802worker1 # [ 143.304636] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:438326803worker1 # [ 143.311171] nix-daemon[861]: accepted connection from pid 1782, user nix-grpc-daemon (trusted)6804worker1 # [ 143.313552] nix-daemon[861]: reaped child process 1919, status = succeeded6805client # [ 143.300983] nix[1457]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6806worker1 # [ 143.318598] nix-daemon[861]: accepted connection from pid 1782, user nix-grpc-daemon (trusted)6807worker1: must succeed: systemctl reload nix-grpc-daemon.service6808worker1 # [ 145.439382] systemd[1]: Reloading Nix worker-protocol over gRPC...6809worker1 # [ 145.466579] systemd[1]: Reloaded Nix worker-protocol over gRPC.6810worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.06 seconds)6811worker1: must succeed: pgrep -f 'read -t [9]0 x'6812worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.03 seconds)6813client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&26814worker1 # [ 145.626574] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:438326815client # don't know how to build these paths:6816client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6817worker2 # [ 145.611898] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361786818worker1 # [ 145.654849] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:34Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06819worker1 # [ 145.657106] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:34Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16820worker2 # [ 145.680851] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361786821worker2 # [ 145.692285] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361786822client # copying 1 paths to 'lb:50051'...6823client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6824worker2 # [ 145.703538] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361786825worker2 # [ 145.713908] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6826worker2 # [ 145.812837] nix-daemon[854]: accepted connection from pid 1674, user nix-grpc-daemon (trusted)6827worker2 # [ 145.826671] nix-daemon[854]: reaped child process 1677, status = succeeded6828worker2 # [ 145.904932] nix-daemon[854]: accepted connection from pid 1679, user nix-grpc-daemon (trusted)6829worker2 # [ 145.918451] nix-daemon[854]: reaped child process 1682, status = succeeded6830lb # [ 145.967481] niks3-server[1128]: 2026/09/22 16:06:35 INFO Received uploads request method=POST path=/api/pending_closures6831worker2 # [ 145.928846] nix-grpc-daemon[1594]: time=2026-09-22T16:06:35.157Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6832worker2 # [ 145.930370] nix-grpc-daemon[1594]: time=2026-09-22T16:06:35.159Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6833lb # [ 145.983627] niks3-server[1128]: 2026/09/22 16:06:35 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6834lb # [ 145.988779] niks3-server[1128]: 2026/09/22 16:06:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6835lb # [ 145.991565] niks3-server[1128]: 2026/09/22 16:06:35 INFO Signed narinfos id=55 count=16836worker2 # [ 145.948288] nix-grpc-daemon[1594]: time=2026-09-22T16:06:35.176Z level=INFO msg="Uploading 1 narinfos"6837lb # [ 145.994463] niks3-server[1128]: 2026/09/22 16:06:35 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6838lb # [ 145.999084] niks3-server[1128]: 2026/09/22 16:06:35 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6839lb # [ 146.003021] niks3-server[1128]: 2026/09/22 16:06:35 INFO Completed upload id=556840worker2 # [ 145.960182] nix-grpc-daemon[1594]: time=2026-09-22T16:06:35.188Z level=INFO msg="Upload complete. (232ms)"6841lb # [ 146.004597] niks3-server[1128]: 2026/09/22 16:06:35 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6842worker2 # [ 145.962374] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=0 paths=1 nar_bytes_in=5286843worker2 # [ 145.966072] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361786844worker2 # [ 145.972423] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6845worker2 # [ 145.974677] nix-daemon[854]: reaped child process 1672, status = succeeded6846client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6847client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6848worker2 # [ 145.979830] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6849worker1 # [ 146.099438] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:35Z level=info event=draining6850client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6851client # farm-a> LOG-farm-a-during-drain6852lb # [ 147.145118] envoy[925]: conn peer=[2001:db8:1::1]:42068 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3006 tx=853 ms=32476853worker2 # [ 147.221956] nix-daemon[854]: reaped child process 1686, status = succeeded6854worker2 # [ 147.298342] nix-daemon[854]: accepted connection from pid 1702, user nix-grpc-daemon (trusted)6855worker2 # [ 147.312325] nix-daemon[854]: reaped child process 1705, status = succeeded6856lb # [ 147.362822] niks3-server[1128]: 2026/09/22 16:06:36 INFO Received uploads request method=POST path=/api/pending_closures6857worker2 # [ 147.325854] nix-grpc-daemon[1594]: time=2026-09-22T16:06:36.554Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6858worker2 # [ 147.327649] nix-grpc-daemon[1594]: time=2026-09-22T16:06:36.556Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6859lb # [ 147.401364] niks3-server[1128]: 2026/09/22 16:06:36 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6860lb # [ 147.413512] niks3-server[1128]: 2026/09/22 16:06:36 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6861lb # [ 147.417583] niks3-server[1128]: 2026/09/22 16:06:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6862worker2 # [ 147.380488] nix-grpc-daemon[1594]: time=2026-09-22T16:06:36.608Z level=INFO msg="Uploading 1 narinfos"6863lb # [ 147.423415] niks3-server[1128]: 2026/09/22 16:06:36 INFO Signed narinfos id=56 count=16864lb # [ 147.435055] niks3-server[1128]: 2026/09/22 16:06:36 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6865lb # [ 147.443254] niks3-server[1128]: 2026/09/22 16:06:36 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6866lb # [ 147.446744] niks3-server[1128]: 2026/09/22 16:06:36 INFO Completed upload id=566867worker2 # [ 147.403734] nix-grpc-daemon[1594]: time=2026-09-22T16:06:36.632Z level=INFO msg="Upload complete. (216ms)"6868lb # [ 147.449274] niks3-server[1128]: 2026/09/22 16:06:36 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6869worker2 # [ 147.409815] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16870worker1 # [ 147.435663] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:36Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06871worker1 # [ 147.438204] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:36Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16872worker2 # [ 147.423975] nix-daemon[854]: reaped child process 1684, status = succeeded6873worker2 # [ 147.456595] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361786874worker2 # [ 147.463843] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361786875worker2 # [ 147.478868] nix-grpc-daemon[1594]: time=2026-09-22T16:06:36.707Z level=INFO msg="All 1 paths already cached"6876client # copying 1 paths to 'lb:50051'...6877worker2 # [ 147.483123] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361786878client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6879worker2 # [ 147.490715] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6880worker2 # [ 147.575532] nix-daemon[854]: accepted connection from pid 1709, user nix-grpc-daemon (trusted)6881worker2 # [ 147.589187] nix-daemon[854]: reaped child process 1712, status = succeeded6882worker2 # [ 147.669855] nix-daemon[854]: accepted connection from pid 1714, user nix-grpc-daemon (trusted)6883worker2 # [ 147.683838] nix-daemon[854]: reaped child process 1717, status = succeeded6884lb # [ 147.731695] niks3-server[1128]: 2026/09/22 16:06:37 INFO Received uploads request method=POST path=/api/pending_closures6885worker2 # [ 147.698628] nix-grpc-daemon[1594]: time=2026-09-22T16:06:36.926Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6886worker2 # [ 147.700030] nix-grpc-daemon[1594]: time=2026-09-22T16:06:36.926Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6887lb # [ 147.763223] niks3-server[1128]: 2026/09/22 16:06:37 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6888lb # [ 147.770109] niks3-server[1128]: 2026/09/22 16:06:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6889worker2 # [ 147.733924] nix-grpc-daemon[1594]: time=2026-09-22T16:06:36.962Z level=INFO msg="Uploading 1 narinfos"6890lb # [ 147.776954] niks3-server[1128]: 2026/09/22 16:06:37 INFO Signed narinfos id=57 count=16891lb # [ 147.783603] niks3-server[1128]: 2026/09/22 16:06:37 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6892lb # [ 147.794815] niks3-server[1128]: 2026/09/22 16:06:37 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6893worker2 # [ 147.758518] nix-grpc-daemon[1594]: time=2026-09-22T16:06:36.987Z level=INFO msg="Upload complete. (261ms)"6894lb # [ 147.802113] niks3-server[1128]: 2026/09/22 16:06:37 INFO Completed upload id=576895worker2 # [ 147.760593] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=0 paths=1 nar_bytes_in=6406896lb # [ 147.805617] niks3-server[1128]: 2026/09/22 16:06:37 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6897worker2 # [ 147.764633] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361786898worker2 # [ 147.771147] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6899worker2 # [ 147.773485] nix-daemon[854]: reaped child process 1707, status = succeeded6900client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6901client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6902worker2 # [ 147.778730] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6903client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6904client # farm-b> LOG-farm-b-during-drain6905worker2 # [ 148.611626] nix-daemon[854]: reaped child process 1721, status = succeeded6906worker2 # [ 148.698365] nix-daemon[854]: accepted connection from pid 1737, user nix-grpc-daemon (trusted)6907worker2 # [ 148.709925] nix-daemon[854]: reaped child process 1740, status = succeeded6908lb # [ 148.762821] niks3-server[1128]: 2026/09/22 16:06:38 INFO Received uploads request method=POST path=/api/pending_closures6909worker2 # [ 148.726605] nix-grpc-daemon[1594]: time=2026-09-22T16:06:37.955Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6910worker2 # [ 148.728634] nix-grpc-daemon[1594]: time=2026-09-22T16:06:37.956Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6911lb # [ 148.797092] niks3-server[1128]: 2026/09/22 16:06:38 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6912lb # [ 148.808857] niks3-server[1128]: 2026/09/22 16:06:38 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6913lb # [ 148.817490] niks3-server[1128]: 2026/09/22 16:06:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6914worker2 # [ 148.778655] nix-grpc-daemon[1594]: time=2026-09-22T16:06:38.007Z level=INFO msg="Uploading 1 narinfos"6915lb # [ 148.821854] niks3-server[1128]: 2026/09/22 16:06:38 INFO Signed narinfos id=58 count=16916lb # [ 148.826465] niks3-server[1128]: 2026/09/22 16:06:38 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6917lb # [ 148.836441] niks3-server[1128]: 2026/09/22 16:06:38 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6918lb # [ 148.840292] niks3-server[1128]: 2026/09/22 16:06:38 INFO Completed upload id=586919worker2 # [ 148.797531] nix-grpc-daemon[1594]: time=2026-09-22T16:06:38.026Z level=INFO msg="Upload complete. (212ms)"6920lb # [ 148.843294] niks3-server[1128]: 2026/09/22 16:06:38 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6921worker1 # [ 148.829786] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:38Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06922worker2 # [ 148.803860] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16923worker1 # [ 148.831568] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:38Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16924worker2 # [ 148.819361] nix-daemon[854]: reaped child process 1719, status = succeeded6925worker2 # [ 148.851490] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361786926worker2 # [ 148.862872] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361786927worker2 # [ 148.876141] nix-grpc-daemon[1594]: time=2026-09-22T16:06:38.104Z level=INFO msg="All 2 paths already cached"6928client # copying 1 paths to 'lb:50051'...6929client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6930worker2 # [ 148.883653] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361786931worker2 # [ 148.892148] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6932worker2 # [ 148.990346] nix-daemon[854]: accepted connection from pid 1744, user nix-grpc-daemon (trusted)6933worker2 # [ 149.002830] nix-daemon[854]: reaped child process 1747, status = succeeded6934worker2 # [ 149.082489] nix-daemon[854]: accepted connection from pid 1749, user nix-grpc-daemon (trusted)6935worker2 # [ 149.094973] nix-daemon[854]: reaped child process 1752, status = succeeded6936lb # [ 149.144995] niks3-server[1128]: 2026/09/22 16:06:38 INFO Received uploads request method=POST path=/api/pending_closures6937worker2 # [ 149.110830] nix-grpc-daemon[1594]: time=2026-09-22T16:06:38.339Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6938worker2 # [ 149.112724] nix-grpc-daemon[1594]: time=2026-09-22T16:06:38.341Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6939lb # [ 149.174688] niks3-server[1128]: 2026/09/22 16:06:38 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6940lb # [ 149.183992] niks3-server[1128]: 2026/09/22 16:06:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6941worker2 # [ 149.147969] nix-grpc-daemon[1594]: time=2026-09-22T16:06:38.376Z level=INFO msg="Uploading 1 narinfos"6942lb # [ 149.190838] niks3-server[1128]: 2026/09/22 16:06:38 INFO Signed narinfos id=59 count=16943lb # [ 149.198592] niks3-server[1128]: 2026/09/22 16:06:38 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6944lb # [ 149.203973] niks3-server[1128]: 2026/09/22 16:06:38 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6945lb # [ 149.207772] niks3-server[1128]: 2026/09/22 16:06:38 INFO Completed upload id=596946worker2 # [ 149.164422] nix-grpc-daemon[1594]: time=2026-09-22T16:06:38.393Z level=INFO msg="Upload complete. (259ms)"6947worker2 # [ 149.166613] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=0 paths=1 nar_bytes_in=7686948lb # [ 149.210657] niks3-server[1128]: 2026/09/22 16:06:38 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6949worker2 # [ 149.169650] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361786950worker2 # [ 149.177191] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6951worker2 # [ 149.179213] nix-daemon[854]: reaped child process 1742, status = succeeded6952client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6953client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6954worker2 # [ 149.184525] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)6955client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6956client # farm-top> LOG-farm-top-during-drain6957worker2 # [ 149.809553] nix-daemon[854]: reaped child process 1756, status = succeeded6958worker2 # [ 149.886794] nix-daemon[854]: accepted connection from pid 1772, user nix-grpc-daemon (trusted)6959worker2 # [ 149.900446] nix-daemon[854]: reaped child process 1775, status = succeeded6960lb # [ 149.950227] niks3-server[1128]: 2026/09/22 16:06:39 INFO Received uploads request method=POST path=/api/pending_closures6961worker2 # [ 149.915763] nix-grpc-daemon[1594]: time=2026-09-22T16:06:39.144Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6962worker2 # [ 149.917650] nix-grpc-daemon[1594]: time=2026-09-22T16:06:39.145Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6963lb # [ 149.984691] niks3-server[1128]: 2026/09/22 16:06:39 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6964lb # [ 149.996703] niks3-server[1128]: 2026/09/22 16:06:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6965worker2 # [ 149.960262] nix-grpc-daemon[1594]: time=2026-09-22T16:06:39.188Z level=INFO msg="Uploading 1 narinfos"6966lb # [ 150.003277] niks3-server[1128]: 2026/09/22 16:06:39 INFO Signed narinfos id=60 count=16967lb # [ 150.009721] niks3-server[1128]: 2026/09/22 16:06:39 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6968lb # [ 150.011801] niks3-server[1128]: 2026/09/22 16:06:39 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6969lb # [ 150.016570] niks3-server[1128]: 2026/09/22 16:06:39 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6970lb # [ 150.019560] niks3-server[1128]: 2026/09/22 16:06:39 INFO Completed upload id=606971worker2 # [ 149.977839] nix-grpc-daemon[1594]: time=2026-09-22T16:06:39.206Z level=INFO msg="Upload complete. (186ms)"6972lb # [ 150.022904] niks3-server[1128]: 2026/09/22 16:06:39 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6973worker2 # [ 149.982689] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=0 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16974worker2 # [ 149.994896] nix-daemon[854]: reaped child process 1754, status = succeeded6975client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&2, in 4.52 seconds)6976client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath6977client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath, in 0.09 seconds)6978worker1: must succeed: pkill -f 'read -t [9]0 x'6979worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.04 seconds)6980client: waiting for success: ! systemctl is-active sw6981worker1 # [ 150.360875] nix-daemon[861]: reaped child process 1937, status = succeeded6982worker1 # [ 150.444788] nix-daemon[861]: accepted connection from pid 1985, user nix-grpc-daemon (trusted)6983worker1 # [ 150.456386] nix-daemon[861]: reaped child process 1988, status = succeeded6984lb # [ 150.484957] niks3-server[1128]: 2026/09/22 16:06:39 INFO Received uploads request method=POST path=/api/pending_closures6985worker1 # [ 150.472154] nix-grpc-daemon[1865]: time=2026-09-22T16:06:39.680Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6986worker1 # [ 150.473737] nix-grpc-daemon[1865]: time=2026-09-22T16:06:39.682Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6987lb # [ 150.513130] niks3-server[1128]: 2026/09/22 16:06:39 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6988lb # [ 150.528703] niks3-server[1128]: 2026/09/22 16:06:39 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6989lb # [ 150.538210] niks3-server[1128]: 2026/09/22 16:06:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6990lb # [ 150.543100] niks3-server[1128]: 2026/09/22 16:06:39 INFO Signed narinfos id=61 count=16991worker1 # [ 150.525403] nix-grpc-daemon[1865]: time=2026-09-22T16:06:39.733Z level=INFO msg="Uploading 1 narinfos"6992lb # [ 150.548996] niks3-server[1128]: 2026/09/22 16:06:40 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6993lb # [ 150.554235] niks3-server[1128]: 2026/09/22 16:06:40 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6994lb # [ 150.557651] niks3-server[1128]: 2026/09/22 16:06:40 INFO Completed upload id=616995lb # [ 150.559770] niks3-server[1128]: 2026/09/22 16:06:40 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6996worker1 # [ 150.538155] nix-grpc-daemon[1865]: time=2026-09-22T16:06:39.747Z level=INFO msg="Upload complete. (207ms)"6997worker1 # [ 150.543953] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43832 duration_s=7 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16998worker1 # [ 150.559502] nix-daemon[861]: reaped child process 1935, status = succeeded6999client # [ 150.562958] systemd[1]: sw.service: Deactivated successfully.7000client # [ 150.566636] systemd[1]: sw.service: Consumed 126ms CPU time over 8.641s wall clock time, 41.4M memory peak, 7.5K incoming IP traffic, 8.5K outgoing IP traffic.7001worker1 # [ 151.100940] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:40Z level=info event=signal_exit7002lb # [ 151.128514] niks3-server[1128]: 2026/09/22 16:06:40 INFO lead: released remote=192.168.1.3:588187003client: (finished: waiting for success: ! systemctl is-active sw, in 1.06 seconds)7004client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }7005client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.03 seconds)7006worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service7007worker2 # [ 152.085530] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:41Z level=info event=scheduler_restarting addr=lb:500517008worker1 # [ 152.112369] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:41Z level=info event=scheduler_restarting7009worker1 # [ 152.115876] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:41Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:438207010worker1 # [ 152.119858] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:41Z level=info event=worker_gone addr=192.168.1.4:500517011worker1 # [ 152.123313] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:41Z level=info event=scheduler_restarting addr=lb:500517012worker1 # [ 152.126738] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:41Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:438207013worker1 # [ 152.130482] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:41Z level=info event=worker_gone addr=192.168.1.3:500517014worker1 # [ 152.145478] nix-daemon[861]: reaped child process 1836, status = succeeded7015worker2 # [ 152.286044] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:41Z level=info event=scheduler_connected addr=lb:500517016worker2 # [ 152.287989] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7017worker1 # [ 152.321785] nix-grpc-daemon[1782]: ts=2026-09-22T16:06:41Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"7018worker2 # [ 152.384671] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:41Z level=info event=scheduler_take_over7019lb # [ 152.428270] niks3-server[1128]: 2026/09/22 16:06:41 INFO lead: acquired remote=192.168.1.4:551087020worker2 # [ 152.488966] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:41Z level=info event=scheduler_connected addr=lb:500517021worker2 # [ 152.490984] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7022worker1 # [ 152.705055] nix-grpc-daemon[1782]: E0000 00:00:1790093201.913827 1827 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 20, started true, shutdown true7023worker1 # [ 152.710578] nix-grpc-daemon[1782]: E0000 00:00:1790093201.919074 1827 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true7024worker2 # [ 152.691995] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:41Z level=info event=scheduler_connected addr=lb:500517025worker2 # [ 152.693888] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7026worker2 # [ 152.894818] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:42Z level=info event=scheduler_connected addr=lb:500517027worker2 # [ 152.896707] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7028worker2 # [ 153.098168] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:42Z level=info event=scheduler_connected addr=lb:500517029worker2 # [ 153.100049] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7030worker1 # [ 153.134717] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.7031worker1 # [ 153.137760] systemd[1]: nix-grpc-daemon.service: Consumed 617ms CPU time over 22.165s wall clock time, 36.5M memory peak, 24.8K incoming IP traffic, 32.2K outgoing IP traffic.7032worker1 # [ 153.152780] systemd[1]: Starting Nix worker-protocol over gRPC...7033worker1 # [ 153.239337] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:42Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"7034worker1 # [ 153.241422] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:42Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc7035worker1 # [ 153.244593] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:42Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=17036worker1 # [ 153.248544] nix-grpc-daemon[2002]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR7037worker1 # [ 153.251190] nix-grpc-daemon[2002]: E0000 00:00:1790093202.457597 2002 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7038worker1 # [ 153.256607] nix-grpc-daemon[2002]: E0000 00:00:1790093202.462941 2002 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7039worker1 # [ 153.260428] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:42Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket7040worker1 # [ 153.263622] systemd[1]: Started Nix worker-protocol over gRPC.7041worker1 # [ 153.266182] nix-grpc-daemon[2002]: E0000 00:00:1790093202.473800 2043 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7042worker1 # [ 153.279162] nix-grpc-daemon[2002]: E0000 00:00:1790093202.487258 2043 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7043worker1 # [ 153.281234] nix-grpc-daemon[2002]: E0000 00:00:1790093202.487409 2043 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7044worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.05 seconds)7045lb: waiting until exactly no endpoints of x86_64-linux are unhealthy7046lb: must succeed: curl -sf localhost:9901/clusters7047worker1 # [ 153.293552] nix-grpc-daemon[2002]: E0000 00:00:1790093202.502601 2043 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7048worker1 # [ 153.306884] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:42Z level=info event=scheduler_connected addr=lb:500517049worker1 # [ 153.308447] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7050worker2 # [ 153.301309] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:42Z level=info event=scheduler_connected addr=lb:500517051worker2 # [ 153.321148] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:42Z level=info event=oidc_jwks provider=mock keys=17052lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)7053worker2 # [ 153.325321] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:42Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:361307054worker2 # [ 153.329458] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:42Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=07055worker1 # [ 153.509650] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:42Z level=info event=scheduler_connected addr=lb:500517056worker2 # [ 153.490448] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:42Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:361307057worker2 # [ 153.494291] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:42Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=07058lb: must succeed: curl -sf localhost:9901/clusters7059lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7060lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.10 seconds)7061lb: must succeed: curl -sf localhost:9901/clusters7062lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7063worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7064worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7065(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 12.57 seconds)7066subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does7067client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&17068worker1 # [ 154.605919] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:602947069worker1 # [ 154.610871] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7070worker2 # [ 154.600219] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361787071worker2 # [ 154.641487] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:43Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=07072worker2 # [ 154.646655] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:43Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=17073worker2 # [ 154.651295] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:43Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=07074worker2 # [ 154.655265] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:43Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=07075worker1 # [ 154.713831] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947076worker1 # [ 154.725240] nix-grpc-daemon[2002]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 50 ms (attempt 1/5)7077worker1 # [ 154.782604] nix-grpc-daemon[2002]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 163 ms (attempt 2/5)7078worker1 # [ 154.947607] nix-grpc-daemon[2002]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 324 ms (attempt 3/5)7079worker1 # [ 155.272493] nix-grpc-daemon[2002]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 462 ms (attempt 4/5)7080worker1 # [ 155.737822] nix-grpc-daemon[2002]: disabling binary cache 'https://cache.nixos.org' for 60 seconds7081worker1 # [ 155.741679] nix-grpc-daemon[2002]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org7082worker1 # [ 155.756825] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602947083worker1 # [ 155.768557] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602947084worker1 # [ 155.779303] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7085worker1 # [ 155.798407] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:45Z level=info event=niks3_push_started pid=20627086worker1 # [ 155.893985] nix-grpc-daemon[2062]: time=2026-09-22T16:06:45.102Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""7087worker1 # [ 155.974460] nix-daemon[861]: accepted connection from pid 2075, user nix-grpc-daemon (trusted)7088worker1 # [ 155.986595] nix-daemon[861]: reaped child process 2078, status = succeeded7089worker1 # [ 156.064694] nix-daemon[861]: accepted connection from pid 2080, user nix-grpc-daemon (trusted)7090worker1 # [ 156.077192] nix-daemon[861]: reaped child process 2083, status = succeeded7091lb # [ 156.105339] niks3-server[1128]: 2026/09/22 16:06:45 INFO Received uploads request method=POST path=/api/pending_closures7092worker1 # [ 156.094290] nix-grpc-daemon[2062]: time=2026-09-22T16:06:45.303Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7093worker1 # [ 156.097054] nix-grpc-daemon[2062]: time=2026-09-22T16:06:45.304Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"7094lb # [ 156.143519] niks3-server[1128]: 2026/09/22 16:06:45 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst7095lb # [ 156.149762] niks3-server[1128]: 2026/09/22 16:06:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign7096lb # [ 156.155125] niks3-server[1128]: 2026/09/22 16:06:45 INFO Signed narinfos id=62 count=17097worker1 # [ 156.138065] nix-grpc-daemon[2062]: time=2026-09-22T16:06:45.347Z level=INFO msg="Uploading 1 narinfos"7098lb # [ 156.171869] niks3-server[1128]: 2026/09/22 16:06:45 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete7099lb # [ 156.184397] niks3-server[1128]: 2026/09/22 16:06:45 INFO Completed upload id=627100worker1 # [ 156.164733] nix-grpc-daemon[2062]: time=2026-09-22T16:06:45.373Z level=INFO msg="Upload complete. (267ms)"7101lb # [ 156.187747] niks3-server[1128]: 2026/09/22 16:06:45 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo7102worker1 # [ 156.167952] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=0 paths=1 nar_bytes_in=5047103worker1 # [ 156.172755] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947104lb # [ 156.196305] niks3-server[1128]: 2026/09/22 16:06:45 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls7105worker1 # [ 156.181715] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7106worker1 # [ 156.186685] nix-daemon[861]: reaped child process 2060, status = succeeded7107worker1 # [ 156.191252] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7108lb # [ 157.144797] envoy[925]: conn peer=[2001:db8:1::1]:46616 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9568 tx=4411 ms=43917109lb # [ 157.146322] envoy[925]: conn peer=[2001:db8:1::1]:42084 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2992 tx=1557 ms=84827110lb # [ 157.147768] envoy[925]: conn peer=[2001:db8:1::3]:42514 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1001 tx=425 ms=211767111worker1 # [ 157.205250] nix-daemon[861]: reaped child process 2087, status = succeeded7112worker1 # [ 157.277128] nix-daemon[861]: accepted connection from pid 2103, user nix-grpc-daemon (trusted)7113worker1 # [ 157.291968] nix-daemon[861]: reaped child process 2106, status = succeeded7114lb # [ 157.316602] niks3-server[1128]: 2026/09/22 16:06:46 INFO Received uploads request method=POST path=/api/pending_closures7115worker1 # [ 157.306957] nix-grpc-daemon[2062]: time=2026-09-22T16:06:46.515Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7116worker1 # [ 157.308463] nix-grpc-daemon[2062]: time=2026-09-22T16:06:46.517Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"7117lb # [ 157.359530] niks3-server[1128]: 2026/09/22 16:06:46 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv7118lb # [ 157.369522] niks3-server[1128]: 2026/09/22 16:06:46 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst7119lb # [ 157.380587] niks3-server[1128]: 2026/09/22 16:06:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign7120lb # [ 157.384257] niks3-server[1128]: 2026/09/22 16:06:46 INFO Signed narinfos id=63 count=17121worker1 # [ 157.364130] nix-grpc-daemon[2062]: time=2026-09-22T16:06:46.572Z level=INFO msg="Uploading 1 narinfos"7122lb # [ 157.390668] niks3-server[1128]: 2026/09/22 16:06:46 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls7123lb # [ 157.399096] niks3-server[1128]: 2026/09/22 16:06:46 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete7124lb # [ 157.403390] niks3-server[1128]: 2026/09/22 16:06:46 INFO Completed upload id=637125worker1 # [ 157.382544] nix-grpc-daemon[2062]: time=2026-09-22T16:06:46.591Z level=INFO msg="Upload complete. (184ms)"7126lb # [ 157.405620] niks3-server[1128]: 2026/09/22 16:06:46 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo7127worker1 # [ 157.388889] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=17128worker2 # [ 157.373941] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:46Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=17129worker1 # [ 157.403178] nix-daemon[861]: reaped child process 2085, status = succeeded7130worker2 # [ 157.380373] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:46Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=07131worker1 # [ 157.439326] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947132worker1 # [ 157.448915] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602947133worker1 # [ 157.457776] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602947134worker1 # [ 157.467157] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7135worker1 # [ 157.557361] nix-daemon[861]: accepted connection from pid 2110, user nix-grpc-daemon (trusted)7136worker1 # [ 157.569597] nix-daemon[861]: reaped child process 2113, status = succeeded7137worker1 # [ 157.646284] nix-daemon[861]: accepted connection from pid 2115, user nix-grpc-daemon (trusted)7138worker1 # [ 157.659731] nix-daemon[861]: reaped child process 2118, status = succeeded7139lb # [ 157.685380] niks3-server[1128]: 2026/09/22 16:06:47 INFO Received uploads request method=POST path=/api/pending_closures7140worker1 # [ 157.669569] nix-grpc-daemon[2062]: time=2026-09-22T16:06:46.878Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7141worker1 # [ 157.670924] nix-grpc-daemon[2062]: time=2026-09-22T16:06:46.878Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"7142lb # [ 157.699494] niks3-server[1128]: 2026/09/22 16:06:47 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst7143lb # [ 157.703756] niks3-server[1128]: 2026/09/22 16:06:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7144lb # [ 157.705724] niks3-server[1128]: 2026/09/22 16:06:47 INFO Signed narinfos id=64 count=17145worker1 # [ 157.684765] nix-grpc-daemon[2062]: time=2026-09-22T16:06:46.893Z level=INFO msg="Uploading 1 narinfos"7146lb # [ 157.708386] niks3-server[1128]: 2026/09/22 16:06:47 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls7147lb # [ 157.713058] niks3-server[1128]: 2026/09/22 16:06:47 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7148lb # [ 157.719199] niks3-server[1128]: 2026/09/22 16:06:47 INFO Completed upload id=647149worker1 # [ 157.698306] nix-grpc-daemon[2062]: time=2026-09-22T16:06:46.907Z level=INFO msg="Upload complete. (220ms)"7150lb # [ 157.721245] niks3-server[1128]: 2026/09/22 16:06:47 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo7151worker1 # [ 157.700696] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=0 paths=1 nar_bytes_in=5047152worker1 # [ 157.704276] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947153worker1 # [ 157.710094] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7154worker1 # [ 157.712471] nix-daemon[861]: reaped child process 2108, status = succeeded7155worker1 # [ 157.717281] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7156worker1 # [ 158.355960] nix-daemon[861]: reaped child process 2122, status = succeeded7157worker1 # [ 158.432358] nix-daemon[861]: accepted connection from pid 2138, user nix-grpc-daemon (trusted)7158worker1 # [ 158.444823] nix-daemon[861]: reaped child process 2141, status = succeeded7159lb # [ 158.472921] niks3-server[1128]: 2026/09/22 16:06:47 INFO Received uploads request method=POST path=/api/pending_closures7160worker1 # [ 158.460755] nix-grpc-daemon[2062]: time=2026-09-22T16:06:47.669Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7161worker1 # [ 158.462380] nix-grpc-daemon[2062]: time=2026-09-22T16:06:47.671Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"7162lb # [ 158.511204] niks3-server[1128]: 2026/09/22 16:06:47 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv7163lb # [ 158.530321] niks3-server[1128]: 2026/09/22 16:06:47 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst7164lb # [ 158.536730] niks3-server[1128]: 2026/09/22 16:06:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7165lb # [ 158.544817] niks3-server[1128]: 2026/09/22 16:06:47 INFO Signed narinfos id=65 count=17166worker1 # [ 158.526785] nix-grpc-daemon[2062]: time=2026-09-22T16:06:47.735Z level=INFO msg="Uploading 1 narinfos"7167lb # [ 158.553548] niks3-server[1128]: 2026/09/22 16:06:48 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls7168lb # [ 158.557845] niks3-server[1128]: 2026/09/22 16:06:48 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7169lb # [ 158.565048] niks3-server[1128]: 2026/09/22 16:06:48 INFO Completed upload id=657170lb # [ 158.566882] niks3-server[1128]: 2026/09/22 16:06:48 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo7171worker1 # [ 158.544284] nix-grpc-daemon[2062]: time=2026-09-22T16:06:47.753Z level=INFO msg="Upload complete. (219ms)"7172worker2 # [ 158.529059] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:47Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=17173worker1 # [ 158.550862] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=0 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=17174worker2 # [ 158.534357] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:47Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=07175worker1 # [ 158.566764] nix-daemon[861]: reaped child process 2120, status = succeeded7176worker1 # [ 158.599418] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947177worker1 # [ 158.611275] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602947178worker1 # [ 158.622930] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602947179worker1 # [ 158.633076] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7180worker1 # [ 158.727319] nix-daemon[861]: accepted connection from pid 2145, user nix-grpc-daemon (trusted)7181worker1 # [ 158.740404] nix-daemon[861]: reaped child process 2148, status = succeeded7182worker1 # [ 158.815950] nix-daemon[861]: accepted connection from pid 2150, user nix-grpc-daemon (trusted)7183worker1 # [ 158.828404] nix-daemon[861]: reaped child process 2153, status = succeeded7184lb # [ 158.855677] niks3-server[1128]: 2026/09/22 16:06:48 INFO Received uploads request method=POST path=/api/pending_closures7185worker1 # [ 158.843227] nix-grpc-daemon[2062]: time=2026-09-22T16:06:48.052Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7186worker1 # [ 158.844732] nix-grpc-daemon[2062]: time=2026-09-22T16:06:48.053Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"7187lb # [ 158.889051] niks3-server[1128]: 2026/09/22 16:06:48 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst7188lb # [ 158.898623] niks3-server[1128]: 2026/09/22 16:06:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7189lb # [ 158.903595] niks3-server[1128]: 2026/09/22 16:06:48 INFO Signed narinfos id=66 count=17190worker1 # [ 158.885800] nix-grpc-daemon[2062]: time=2026-09-22T16:06:48.094Z level=INFO msg="Uploading 1 narinfos"7191lb # [ 158.917245] niks3-server[1128]: 2026/09/22 16:06:48 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls7192lb # [ 158.926482] niks3-server[1128]: 2026/09/22 16:06:48 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7193lb # [ 158.934477] niks3-server[1128]: 2026/09/22 16:06:48 INFO Completed upload id=667194lb # [ 158.937182] niks3-server[1128]: 2026/09/22 16:06:48 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo7195worker1 # [ 158.914959] nix-grpc-daemon[2062]: time=2026-09-22T16:06:48.123Z level=INFO msg="Upload complete. (271ms)"7196worker1 # [ 158.919805] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=0 paths=1 nar_bytes_in=5047197worker1 # [ 158.926318] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947198worker1 # [ 158.939122] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7199worker1 # [ 158.946875] nix-daemon[861]: reaped child process 2143, status = succeeded7200worker1 # [ 158.954227] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7201worker1 # [ 159.769318] nix-daemon[861]: reaped child process 2157, status = succeeded7202worker1 # [ 159.843353] nix-daemon[861]: accepted connection from pid 2173, user nix-grpc-daemon (trusted)7203worker1 # [ 159.855923] nix-daemon[861]: reaped child process 2176, status = succeeded7204lb # [ 159.884192] niks3-server[1128]: 2026/09/22 16:06:49 INFO Received uploads request method=POST path=/api/pending_closures7205worker1 # [ 159.869132] nix-grpc-daemon[2062]: time=2026-09-22T16:06:49.077Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7206worker1 # [ 159.870723] nix-grpc-daemon[2062]: time=2026-09-22T16:06:49.079Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"7207lb # [ 159.914058] niks3-server[1128]: 2026/09/22 16:06:49 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv7208lb # [ 159.924474] niks3-server[1128]: 2026/09/22 16:06:49 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst7209lb # [ 159.932411] niks3-server[1128]: 2026/09/22 16:06:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7210lb # [ 159.936747] niks3-server[1128]: 2026/09/22 16:06:49 INFO Signed narinfos id=67 count=17211worker1 # [ 159.916302] nix-grpc-daemon[2062]: time=2026-09-22T16:06:49.125Z level=INFO msg="Uploading 1 narinfos"7212lb # [ 159.949535] niks3-server[1128]: 2026/09/22 16:06:49 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls7213lb # [ 159.953055] niks3-server[1128]: 2026/09/22 16:06:49 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7214lb # [ 159.957197] niks3-server[1128]: 2026/09/22 16:06:49 INFO Completed upload id=677215worker1 # [ 159.936271] nix-grpc-daemon[2062]: time=2026-09-22T16:06:49.145Z level=INFO msg="Upload complete. (175ms)"7216lb # [ 159.959069] niks3-server[1128]: 2026/09/22 16:06:49 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo7217worker1 # [ 159.941485] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=17218worker2 # [ 159.921381] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:49Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=17219worker1 # [ 159.954399] nix-daemon[861]: reaped child process 2155, status = succeeded7220worker2 # [ 159.930705] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:49Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=07221worker1 # [ 159.986271] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947222worker1 # [ 159.998351] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602947223worker1 # [ 160.011807] nix-grpc-daemon[2062]: time=2026-09-22T16:06:49.220Z level=INFO msg="All 1 paths already cached"7224worker1 # [ 160.019611] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602947225worker1 # [ 160.029168] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7226worker1 # [ 160.124867] nix-daemon[861]: accepted connection from pid 2180, user nix-grpc-daemon (trusted)7227worker1 # [ 160.138054] nix-daemon[861]: reaped child process 2183, status = succeeded7228worker1 # [ 160.222062] nix-daemon[861]: accepted connection from pid 2185, user nix-grpc-daemon (trusted)7229worker1 # [ 160.234512] nix-daemon[861]: reaped child process 2188, status = succeeded7230lb # [ 160.262321] niks3-server[1128]: 2026/09/22 16:06:49 INFO Received uploads request method=POST path=/api/pending_closures7231worker1 # [ 160.245836] nix-grpc-daemon[2062]: time=2026-09-22T16:06:49.454Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7232worker1 # [ 160.247313] nix-grpc-daemon[2062]: time=2026-09-22T16:06:49.454Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"7233lb # [ 160.275712] niks3-server[1128]: 2026/09/22 16:06:49 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst7234lb # [ 160.279026] niks3-server[1128]: 2026/09/22 16:06:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7235lb # [ 160.281474] niks3-server[1128]: 2026/09/22 16:06:49 INFO Signed narinfos id=68 count=17236worker1 # [ 160.261821] nix-grpc-daemon[2062]: time=2026-09-22T16:06:49.470Z level=INFO msg="Uploading 1 narinfos"7237lb # [ 160.284061] niks3-server[1128]: 2026/09/22 16:06:49 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls7238lb # [ 160.290844] niks3-server[1128]: 2026/09/22 16:06:49 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7239lb # [ 160.294529] niks3-server[1128]: 2026/09/22 16:06:49 INFO Completed upload id=687240worker1 # [ 160.273550] nix-grpc-daemon[2062]: time=2026-09-22T16:06:49.482Z level=INFO msg="Upload complete. (232ms)"7241lb # [ 160.296355] niks3-server[1128]: 2026/09/22 16:06:49 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo7242worker1 # [ 160.275606] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=0 paths=1 nar_bytes_in=6247243worker1 # [ 160.279742] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947244worker1 # [ 160.285091] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7245worker1 # [ 160.287307] nix-daemon[861]: reaped child process 2178, status = succeeded7246worker1 # [ 160.292456] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7247worker1 # [ 161.106586] nix-daemon[861]: reaped child process 2192, status = succeeded7248worker1 # [ 161.182928] nix-daemon[861]: accepted connection from pid 2208, user nix-grpc-daemon (trusted)7249worker1 # [ 161.195939] nix-daemon[861]: reaped child process 2211, status = succeeded7250lb # [ 161.224625] niks3-server[1128]: 2026/09/22 16:06:50 INFO Received uploads request method=POST path=/api/pending_closures7251worker1 # [ 161.214292] nix-grpc-daemon[2062]: time=2026-09-22T16:06:50.422Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7252worker1 # [ 161.215771] nix-grpc-daemon[2062]: time=2026-09-22T16:06:50.422Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"7253lb # [ 161.265283] niks3-server[1128]: 2026/09/22 16:06:50 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst7254lb # [ 161.276232] niks3-server[1128]: 2026/09/22 16:06:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7255worker1 # [ 161.262225] nix-grpc-daemon[2062]: time=2026-09-22T16:06:50.471Z level=INFO msg="Uploading 1 narinfos"7256lb # [ 161.283966] niks3-server[1128]: 2026/09/22 16:06:50 INFO Signed narinfos id=69 count=17257lb # [ 161.286589] niks3-server[1128]: 2026/09/22 16:06:50 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls7258lb # [ 161.291084] niks3-server[1128]: 2026/09/22 16:06:50 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv7259lb # [ 161.297026] niks3-server[1128]: 2026/09/22 16:06:50 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7260lb # [ 161.300789] niks3-server[1128]: 2026/09/22 16:06:50 INFO Completed upload id=697261worker1 # [ 161.280074] nix-grpc-daemon[2062]: time=2026-09-22T16:06:50.489Z level=INFO msg="Upload complete. (181ms)"7262lb # [ 161.302397] niks3-server[1128]: 2026/09/22 16:06:50 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo7263worker1 # [ 161.285455] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=17264worker2 # [ 161.264813] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:50Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=17265worker1 # [ 161.298131] nix-daemon[861]: reaped child process 2190, status = succeeded7266worker1 # [ 161.328517] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947267worker1 # [ 161.334540] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602947268worker2 # [ 161.308972] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:50Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=07269worker1 # [ 161.340316] nix-grpc-daemon[2062]: time=2026-09-22T16:06:50.549Z level=INFO msg="All 1 paths already cached"7270worker1 # [ 161.344030] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602947271worker1 # [ 161.348993] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7272worker1 # [ 161.432197] nix-daemon[861]: accepted connection from pid 2215, user nix-grpc-daemon (trusted)7273worker1 # [ 161.444469] nix-daemon[861]: reaped child process 2218, status = succeeded7274worker1 # [ 161.527827] nix-daemon[861]: accepted connection from pid 2220, user nix-grpc-daemon (trusted)7275worker1 # [ 161.539753] nix-daemon[861]: reaped child process 2223, status = succeeded7276lb # [ 161.568961] niks3-server[1128]: 2026/09/22 16:06:51 INFO Received uploads request method=POST path=/api/pending_closures7277worker1 # [ 161.557704] nix-grpc-daemon[2062]: time=2026-09-22T16:06:50.766Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7278worker1 # [ 161.559127] nix-grpc-daemon[2062]: time=2026-09-22T16:06:50.766Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"7279lb # [ 161.597137] niks3-server[1128]: 2026/09/22 16:06:51 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst7280lb # [ 161.608342] niks3-server[1128]: 2026/09/22 16:06:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7281worker1 # [ 161.592984] nix-grpc-daemon[2062]: time=2026-09-22T16:06:50.801Z level=INFO msg="Uploading 1 narinfos"7282lb # [ 161.613905] niks3-server[1128]: 2026/09/22 16:06:51 INFO Signed narinfos id=70 count=17283lb # [ 161.619098] niks3-server[1128]: 2026/09/22 16:06:51 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls7284lb # [ 161.628613] niks3-server[1128]: 2026/09/22 16:06:51 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7285lb # [ 161.632653] niks3-server[1128]: 2026/09/22 16:06:51 INFO Completed upload id=707286worker1 # [ 161.612866] nix-grpc-daemon[2062]: time=2026-09-22T16:06:50.821Z level=INFO msg="Upload complete. (259ms)"7287lb # [ 161.634570] niks3-server[1128]: 2026/09/22 16:06:51 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo7288worker1 # [ 161.615104] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=0 paths=1 nar_bytes_in=6247289worker1 # [ 161.617809] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947290worker1 # [ 161.625154] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7291worker1 # [ 161.627686] nix-daemon[861]: reaped child process 2213, status = succeeded7292worker1 # [ 161.632449] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7293worker1 # [ 162.848780] nix-daemon[861]: reaped child process 2227, status = succeeded7294worker1 # [ 162.917068] nix-daemon[861]: accepted connection from pid 2243, user nix-grpc-daemon (trusted)7295worker1 # [ 162.932467] nix-daemon[861]: reaped child process 2246, status = succeeded7296lb # [ 162.955547] niks3-server[1128]: 2026/09/22 16:06:52 INFO Received uploads request method=POST path=/api/pending_closures7297worker1 # [ 162.944867] nix-grpc-daemon[2062]: time=2026-09-22T16:06:52.153Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7298worker1 # [ 162.947061] nix-grpc-daemon[2062]: time=2026-09-22T16:06:52.155Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"7299lb # [ 162.995226] niks3-server[1128]: 2026/09/22 16:06:52 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv7300lb # [ 163.008249] niks3-server[1128]: 2026/09/22 16:06:52 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst7301lb # [ 163.018930] niks3-server[1128]: 2026/09/22 16:06:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7302lb # [ 163.025179] niks3-server[1128]: 2026/09/22 16:06:52 INFO Signed narinfos id=71 count=17303worker1 # [ 163.007546] nix-grpc-daemon[2062]: time=2026-09-22T16:06:52.216Z level=INFO msg="Uploading 1 narinfos"7304lb # [ 163.033115] niks3-server[1128]: 2026/09/22 16:06:52 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls7305lb # [ 163.041191] niks3-server[1128]: 2026/09/22 16:06:52 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7306worker1 # [ 163.023398] nix-grpc-daemon[2062]: time=2026-09-22T16:06:52.232Z level=INFO msg="Upload complete. (184ms)"7307lb # [ 163.044838] niks3-server[1128]: 2026/09/22 16:06:52 INFO Completed upload id=717308worker1 # [ 163.025983] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=17309lb # [ 163.048709] niks3-server[1128]: 2026/09/22 16:06:52 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo7310worker2 # [ 163.005503] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:52Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=17311worker1 # [ 163.033107] nix-daemon[861]: reaped child process 2225, status = succeeded7312worker2 # [ 163.014918] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:52Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=07313worker1 # [ 163.077105] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947314worker1 # [ 163.084819] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602947315worker1 # [ 163.093528] nix-grpc-daemon[2062]: time=2026-09-22T16:06:52.302Z level=INFO msg="All 1 paths already cached"7316worker1 # [ 163.098071] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602947317worker1 # [ 163.102545] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7318worker1 # [ 163.182542] nix-daemon[861]: accepted connection from pid 2250, user nix-grpc-daemon (trusted)7319worker1 # [ 163.195404] nix-daemon[861]: reaped child process 2253, status = succeeded7320worker1 # [ 163.273189] nix-daemon[861]: accepted connection from pid 2255, user nix-grpc-daemon (trusted)7321worker1 # [ 163.285926] nix-daemon[861]: reaped child process 2258, status = succeeded7322lb # [ 163.313263] niks3-server[1128]: 2026/09/22 16:06:52 INFO Received uploads request method=POST path=/api/pending_closures7323worker1 # [ 163.300429] nix-grpc-daemon[2062]: time=2026-09-22T16:06:52.509Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7324worker1 # [ 163.302285] nix-grpc-daemon[2062]: time=2026-09-22T16:06:52.511Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"7325lb # [ 163.340553] niks3-server[1128]: 2026/09/22 16:06:52 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst7326lb # [ 163.353402] niks3-server[1128]: 2026/09/22 16:06:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7327lb # [ 163.358488] niks3-server[1128]: 2026/09/22 16:06:52 INFO Signed narinfos id=72 count=17328worker1 # [ 163.341405] nix-grpc-daemon[2062]: time=2026-09-22T16:06:52.550Z level=INFO msg="Uploading 1 narinfos"7329lb # [ 163.374063] niks3-server[1128]: 2026/09/22 16:06:52 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7330worker1 # [ 163.361506] nix-grpc-daemon[2062]: time=2026-09-22T16:06:52.570Z level=INFO msg="Upload complete. (254ms)"7331lb # [ 163.382514] niks3-server[1128]: 2026/09/22 16:06:52 INFO Completed upload id=727332worker1 # [ 163.363721] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=0 paths=1 nar_bytes_in=6247333lb # [ 163.388133] niks3-server[1128]: 2026/09/22 16:06:52 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls7334worker1 # [ 163.367145] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947335lb # [ 163.389968] niks3-server[1128]: 2026/09/22 16:06:52 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo7336worker1 # [ 163.373934] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7337worker1 # [ 163.376249] nix-daemon[861]: reaped child process 2248, status = succeeded7338worker1 # [ 163.381392] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7339worker1 # [ 164.398882] nix-daemon[861]: reaped child process 2262, status = succeeded7340worker1 # [ 164.468367] nix-daemon[861]: accepted connection from pid 2278, user nix-grpc-daemon (trusted)7341worker1 # [ 164.481516] nix-daemon[861]: reaped child process 2281, status = succeeded7342lb # [ 164.509436] niks3-server[1128]: 2026/09/22 16:06:53 INFO Received uploads request method=POST path=/api/pending_closures7343worker1 # [ 164.496463] nix-grpc-daemon[2062]: time=2026-09-22T16:06:53.705Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7344worker1 # [ 164.497965] nix-grpc-daemon[2062]: time=2026-09-22T16:06:53.707Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"7345lb # [ 164.544161] niks3-server[1128]: 2026/09/22 16:06:53 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv7346lb # [ 164.552624] niks3-server[1128]: 2026/09/22 16:06:54 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst7347lb # [ 164.565116] niks3-server[1128]: 2026/09/22 16:06:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7348lb # [ 164.570586] niks3-server[1128]: 2026/09/22 16:06:54 INFO Signed narinfos id=73 count=17349worker1 # [ 164.551858] nix-grpc-daemon[2062]: time=2026-09-22T16:06:53.760Z level=INFO msg="Uploading 1 narinfos"7350lb # [ 164.575896] niks3-server[1128]: 2026/09/22 16:06:54 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls7351lb # [ 164.584221] niks3-server[1128]: 2026/09/22 16:06:54 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7352lb # [ 164.587717] niks3-server[1128]: 2026/09/22 16:06:54 INFO Completed upload id=737353worker1 # [ 164.566730] nix-grpc-daemon[2062]: time=2026-09-22T16:06:53.775Z level=INFO msg="Upload complete. (179ms)"7354lb # [ 164.589555] niks3-server[1128]: 2026/09/22 16:06:54 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo7355worker1 # [ 164.569437] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=17356worker2 # [ 164.549489] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:53Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=17357worker1 # [ 164.576711] nix-daemon[861]: reaped child process 2260, status = succeeded7358worker1 # [ 164.614041] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947359worker2 # [ 164.593383] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:53Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=07360worker1 # [ 164.621195] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602947361worker1 # [ 164.628302] nix-grpc-daemon[2062]: time=2026-09-22T16:06:53.837Z level=INFO msg="All 2 paths already cached"7362worker1 # [ 164.632167] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602947363worker1 # [ 164.637445] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7364worker1 # [ 164.716992] nix-daemon[861]: accepted connection from pid 2285, user nix-grpc-daemon (trusted)7365worker1 # [ 164.730077] nix-daemon[861]: reaped child process 2288, status = succeeded7366worker1 # [ 164.805329] nix-daemon[861]: accepted connection from pid 2290, user nix-grpc-daemon (trusted)7367worker1 # [ 164.817526] nix-daemon[861]: reaped child process 2293, status = succeeded7368lb # [ 164.844335] niks3-server[1128]: 2026/09/22 16:06:54 INFO Received uploads request method=POST path=/api/pending_closures7369worker1 # [ 164.830982] nix-grpc-daemon[2062]: time=2026-09-22T16:06:54.039Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7370worker1 # [ 164.832364] nix-grpc-daemon[2062]: time=2026-09-22T16:06:54.039Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7371lb # [ 164.872775] niks3-server[1128]: 2026/09/22 16:06:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7372worker1 # [ 164.858222] nix-grpc-daemon[2062]: time=2026-09-22T16:06:54.067Z level=INFO msg="Uploading 1 narinfos"7373lb # [ 164.880122] niks3-server[1128]: 2026/09/22 16:06:54 INFO Signed narinfos id=74 count=17374lb # [ 164.886573] niks3-server[1128]: 2026/09/22 16:06:54 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7375lb # [ 164.893501] niks3-server[1128]: 2026/09/22 16:06:54 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7376lb # [ 164.905981] niks3-server[1128]: 2026/09/22 16:06:54 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7377lb # [ 164.912775] niks3-server[1128]: 2026/09/22 16:06:54 INFO Completed upload id=747378worker1 # [ 164.894166] nix-grpc-daemon[2062]: time=2026-09-22T16:06:54.103Z level=INFO msg="Upload complete. (252ms)"7379lb # [ 164.916110] niks3-server[1128]: 2026/09/22 16:06:54 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7380worker1 # [ 164.896440] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=0 paths=1 nar_bytes_in=7527381worker1 # [ 164.901230] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947382worker1 # [ 164.907364] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7383worker1 # [ 164.908702] nix-daemon[861]: reaped child process 2283, status = succeeded7384worker1 # [ 164.913742] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7385worker1 # [ 166.354565] nix-daemon[861]: reaped child process 2297, status = succeeded7386worker1 # [ 166.425251] nix-daemon[861]: accepted connection from pid 2313, user nix-grpc-daemon (trusted)7387worker1 # [ 166.437896] nix-daemon[861]: reaped child process 2316, status = succeeded7388lb # [ 166.464863] niks3-server[1128]: 2026/09/22 16:06:55 INFO Received uploads request method=POST path=/api/pending_closures7389worker1 # [ 166.448375] nix-grpc-daemon[2062]: time=2026-09-22T16:06:55.657Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7390worker1 # [ 166.449919] nix-grpc-daemon[2062]: time=2026-09-22T16:06:55.658Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7391lb # [ 166.485225] niks3-server[1128]: 2026/09/22 16:06:55 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7392lb # [ 166.490410] niks3-server[1128]: 2026/09/22 16:06:55 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7393lb # [ 166.494542] niks3-server[1128]: 2026/09/22 16:06:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7394lb # [ 166.496520] niks3-server[1128]: 2026/09/22 16:06:55 INFO Signed narinfos id=75 count=17395worker1 # [ 166.476997] nix-grpc-daemon[2062]: time=2026-09-22T16:06:55.685Z level=INFO msg="Uploading 1 narinfos"7396lb # [ 166.501692] niks3-server[1128]: 2026/09/22 16:06:55 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7397lb # [ 166.505299] niks3-server[1128]: 2026/09/22 16:06:55 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7398lb # [ 166.509782] niks3-server[1128]: 2026/09/22 16:06:55 INFO Completed upload id=757399worker1 # [ 166.488817] nix-grpc-daemon[2062]: time=2026-09-22T16:06:55.697Z level=INFO msg="Upload complete. (166ms)"7400lb # [ 166.513334] niks3-server[1128]: 2026/09/22 16:06:55 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7401worker1 # [ 166.491398] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=17402worker2 # [ 166.471565] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:55Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17403worker1 # [ 166.498321] nix-daemon[861]: reaped child process 2295, status = succeeded7404worker1 # [ 166.500435] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947405worker1 # [ 166.506196] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602947406worker1 # [ 166.511772] nix-grpc-daemon[2062]: time=2026-09-22T16:06:55.720Z level=INFO msg="All 2 paths already cached"7407worker1 # [ 166.515828] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602947408worker1 # [ 166.519580] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7409worker1 # [ 166.596805] nix-daemon[861]: accepted connection from pid 2320, user nix-grpc-daemon (trusted)7410worker1 # [ 166.616039] nix-daemon[861]: reaped child process 2323, status = succeeded7411worker1 # [ 166.682056] nix-daemon[861]: accepted connection from pid 2325, user nix-grpc-daemon (trusted)7412worker1 # [ 166.694363] nix-daemon[861]: reaped child process 2328, status = succeeded7413lb # [ 166.722649] niks3-server[1128]: 2026/09/22 16:06:56 INFO Received uploads request method=POST path=/api/pending_closures7414worker1 # [ 166.709386] nix-grpc-daemon[2062]: time=2026-09-22T16:06:55.918Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7415worker1 # [ 166.710982] nix-grpc-daemon[2062]: time=2026-09-22T16:06:55.920Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7416lb # [ 166.751919] niks3-server[1128]: 2026/09/22 16:06:56 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7417lb # [ 166.762418] niks3-server[1128]: 2026/09/22 16:06:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign7418lb # [ 166.767262] niks3-server[1128]: 2026/09/22 16:06:56 INFO Signed narinfos id=76 count=17419worker1 # [ 166.749159] nix-grpc-daemon[2062]: time=2026-09-22T16:06:55.958Z level=INFO msg="Uploading 1 narinfos"7420lb # [ 166.780079] niks3-server[1128]: 2026/09/22 16:06:56 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7421lb # [ 166.787810] niks3-server[1128]: 2026/09/22 16:06:56 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete7422lb # [ 166.795092] niks3-server[1128]: 2026/09/22 16:06:56 INFO Completed upload id=767423worker1 # [ 166.773555] nix-grpc-daemon[2062]: time=2026-09-22T16:06:55.982Z level=INFO msg="Upload complete. (248ms)"7424lb # [ 166.798181] niks3-server[1128]: 2026/09/22 16:06:56 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7425worker1 # [ 166.777245] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=0 paths=1 nar_bytes_in=7527426worker1 # [ 166.783801] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947427worker1 # [ 166.794640] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7428worker1 # [ 166.796955] nix-daemon[861]: reaped child process 2318, status = succeeded7429worker1 # [ 166.802313] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7430worker1 # [ 167.837933] nix-daemon[861]: reaped child process 2332, status = succeeded7431worker1 # [ 167.920202] nix-daemon[861]: accepted connection from pid 2348, user nix-grpc-daemon (trusted)7432worker1 # [ 167.932761] nix-daemon[861]: reaped child process 2351, status = succeeded7433lb # [ 167.961621] niks3-server[1128]: 2026/09/22 16:06:57 INFO Received uploads request method=POST path=/api/pending_closures7434worker1 # [ 167.948902] nix-grpc-daemon[2062]: time=2026-09-22T16:06:57.157Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7435worker1 # [ 167.950393] nix-grpc-daemon[2062]: time=2026-09-22T16:06:57.159Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7436lb # [ 168.000694] niks3-server[1128]: 2026/09/22 16:06:57 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7437lb # [ 168.012115] niks3-server[1128]: 2026/09/22 16:06:57 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7438lb # [ 168.015931] niks3-server[1128]: 2026/09/22 16:06:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign7439lb # [ 168.021056] niks3-server[1128]: 2026/09/22 16:06:57 INFO Signed narinfos id=77 count=17440worker1 # [ 168.003581] nix-grpc-daemon[2062]: time=2026-09-22T16:06:57.212Z level=INFO msg="Uploading 1 narinfos"7441lb # [ 168.035841] niks3-server[1128]: 2026/09/22 16:06:57 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7442lb # [ 168.041582] niks3-server[1128]: 2026/09/22 16:06:57 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete7443lb # [ 168.048057] niks3-server[1128]: 2026/09/22 16:06:57 INFO Completed upload id=777444worker1 # [ 168.027196] nix-grpc-daemon[2062]: time=2026-09-22T16:06:57.236Z level=INFO msg="Upload complete. (217ms)"7445lb # [ 168.051754] niks3-server[1128]: 2026/09/22 16:06:57 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7446worker1 # [ 168.033791] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=17447worker2 # [ 168.013652] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:57Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17448worker1 # [ 168.050210] nix-daemon[861]: reaped child process 2330, status = succeeded7449worker1 # [ 168.078993] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947450worker1 # [ 168.092361] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602947451worker1 # [ 168.105500] nix-grpc-daemon[2062]: time=2026-09-22T16:06:57.314Z level=INFO msg="All 2 paths already cached"7452worker1 # [ 168.114356] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602947453worker1 # [ 168.123139] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7454worker1 # [ 168.216713] nix-daemon[861]: accepted connection from pid 2355, user nix-grpc-daemon (trusted)7455worker1 # [ 168.229555] nix-daemon[861]: reaped child process 2358, status = succeeded7456worker1 # [ 168.310917] nix-daemon[861]: accepted connection from pid 2360, user nix-grpc-daemon (trusted)7457worker1 # [ 168.324254] nix-daemon[861]: reaped child process 2363, status = succeeded7458lb # [ 168.350930] niks3-server[1128]: 2026/09/22 16:06:57 INFO Received uploads request method=POST path=/api/pending_closures7459worker1 # [ 168.337538] nix-grpc-daemon[2062]: time=2026-09-22T16:06:57.546Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7460worker1 # [ 168.339588] nix-grpc-daemon[2062]: time=2026-09-22T16:06:57.548Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7461lb # [ 168.381298] niks3-server[1128]: 2026/09/22 16:06:57 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7462lb # [ 168.391484] niks3-server[1128]: 2026/09/22 16:06:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign7463worker1 # [ 168.377304] nix-grpc-daemon[2062]: time=2026-09-22T16:06:57.586Z level=INFO msg="Uploading 1 narinfos"7464lb # [ 168.398605] niks3-server[1128]: 2026/09/22 16:06:57 INFO Signed narinfos id=78 count=17465lb # [ 168.406951] niks3-server[1128]: 2026/09/22 16:06:57 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7466lb # [ 168.423951] niks3-server[1128]: 2026/09/22 16:06:57 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete7467lb # [ 168.427479] niks3-server[1128]: 2026/09/22 16:06:57 INFO Completed upload id=787468worker1 # [ 168.406569] nix-grpc-daemon[2062]: time=2026-09-22T16:06:57.615Z level=INFO msg="Upload complete. (270ms)"7469lb # [ 168.429644] niks3-server[1128]: 2026/09/22 16:06:57 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7470worker1 # [ 168.409803] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=0 paths=1 nar_bytes_in=7527471worker1 # [ 168.415579] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602947472worker1 # [ 168.425704] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7473worker1 # [ 168.431428] nix-daemon[861]: reaped child process 2353, status = succeeded7474worker1 # [ 168.437047] nix-daemon[861]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7475worker1 # [ 169.474114] nix-daemon[861]: reaped child process 2367, status = succeeded7476worker1 # [ 169.532078] nix-daemon[861]: accepted connection from pid 2383, user nix-grpc-daemon (trusted)7477lb # [ 169.568397] niks3-server[1128]: 2026/09/22 16:06:59 INFO Received uploads request method=POST path=/api/pending_closures7478worker1 # [ 169.547940] nix-daemon[861]: reaped child process 2386, status = succeeded7479worker1 # [ 169.552119] nix-grpc-daemon[2062]: time=2026-09-22T16:06:58.760Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7480worker1 # [ 169.555147] nix-grpc-daemon[2062]: time=2026-09-22T16:06:58.762Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7481lb # [ 169.590032] niks3-server[1128]: 2026/09/22 16:06:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign7482lb # [ 169.594609] niks3-server[1128]: 2026/09/22 16:06:59 INFO Signed narinfos id=79 count=17483worker1 # [ 169.573810] nix-grpc-daemon[2062]: time=2026-09-22T16:06:58.782Z level=INFO msg="Uploading 1 narinfos"7484lb # [ 169.597456] niks3-server[1128]: 2026/09/22 16:06:59 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7485lb # [ 169.600983] niks3-server[1128]: 2026/09/22 16:06:59 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7486lb # [ 169.603948] niks3-server[1128]: 2026/09/22 16:06:59 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7487lb # [ 169.609415] niks3-server[1128]: 2026/09/22 16:06:59 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete7488lb # [ 169.612829] niks3-server[1128]: 2026/09/22 16:06:59 INFO Completed upload id=797489worker1 # [ 169.591856] nix-grpc-daemon[2062]: time=2026-09-22T16:06:58.800Z level=INFO msg="Upload complete. (147ms)"7490lb # [ 169.614824] niks3-server[1128]: 2026/09/22 16:06:59 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7491worker1 # [ 169.594396] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60294 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=17492worker1 # [ 169.601245] nix-daemon[861]: reaped child process 2365, status = succeeded7493client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 15.18 seconds)7494worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7495worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)7496client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]'7497client # [ 169.699872] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7498client # Running as unit: gpu.service; invocation ID: 987edad0a467409881595195ceb465987499client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.06 seconds)7500lb: must succeed: curl -sf localhost:9901/clusters7501lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7502lb: must succeed: curl -sf localhost:9901/clusters7503lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7504worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7505client # [ 169.828764] nix[1537]: don't know how to build these paths:7506worker2 # [ 169.822959] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:361787507client # [ 169.829795] nix[1537]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7508worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7509worker1 # [ 169.847244] nix-grpc-daemon[2002]: ts=2026-09-22T16:06:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602947510worker2 # [ 169.838677] nix-grpc-daemon[1498]: ts=2026-09-22T16:06:59Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07511client # [ 169.846087] nix[1537]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,kvm,nixos-test,vip}, waiting up to 30s for a worker7512lb: must succeed: curl -sf localhost:9901/clusters7513lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7514lb: must succeed: curl -sf localhost:9901/clusters7515lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7516worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7517worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)7518client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7519lb # [ 177.146966] envoy[925]: conn peer=[2001:db8:1::1]:58190 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27029 tx=12341 ms=150117520client # [ 199.847305] nix[1537]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,kvm,nixos-test,vip}7521client # [ 199.857485] nix[1537]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7522client # [ 199.889493] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7523client # [ 199.893418] systemd[1]: gpu.service: Failed with result 'exit-code'.7524client # [ 199.897420] systemd[1]: gpu.service: Consumed 142ms CPU time over 30.194s wall clock time, 35.5M memory peak, 5.2K incoming IP traffic, 5K outgoing IP traffic.7525client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.20 seconds)7526client: must succeed: journalctl -u gpu -o cat7527client: (finished: must succeed: journalctl -u gpu -o cat, in 0.01 seconds)7528lb: must succeed: curl -sf localhost:9901/clusters7529lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7530lb: must succeed: curl -sf localhost:9901/clusters7531lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7532worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7533worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7534(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 45.82 seconds)7535subtest: low disk drains a worker and builds go to the other7536worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7537worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.91 seconds)7538worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7539worker1 # [ 201.269879] nix-grpc-daemon[2002]: ts=2026-09-22T16:07:30Z level=info event=unhealthy reason=min_free7540worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.07 seconds)7541client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&27542client # don't know how to build these paths:7543worker2 # [ 202.376587] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:361787544client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7545worker1 # [ 202.404373] nix-grpc-daemon[2002]: ts=2026-09-22T16:07:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602947546worker2 # [ 202.400116] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:31Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07547worker2 # [ 202.402241] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:31Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17548worker2 # [ 202.444293] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361787549worker2 # [ 202.452549] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361787550client # copying 1 paths to 'lb:50051'...7551worker2 # [ 202.460849] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361787552client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7553worker2 # [ 202.465943] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)7554worker2 # [ 202.554381] nix-daemon[854]: accepted connection from pid 1809, user nix-grpc-daemon (trusted)7555worker2 # [ 202.567693] nix-daemon[854]: reaped child process 1812, status = succeeded7556worker2 # [ 202.655183] nix-daemon[854]: accepted connection from pid 1814, user nix-grpc-daemon (trusted)7557worker2 # [ 202.667691] nix-daemon[854]: reaped child process 1817, status = succeeded7558lb # [ 202.719832] niks3-server[1128]: 2026/09/22 16:07:32 INFO Received uploads request method=POST path=/api/pending_closures7559worker2 # [ 202.685796] nix-grpc-daemon[1594]: time=2026-09-22T16:07:31.913Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7560worker2 # [ 202.687248] nix-grpc-daemon[1594]: time=2026-09-22T16:07:31.914Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7561lb # [ 202.752147] niks3-server[1128]: 2026/09/22 16:07:32 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7562lb # [ 202.765128] niks3-server[1128]: 2026/09/22 16:07:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign7563lb # [ 202.771110] niks3-server[1128]: 2026/09/22 16:07:32 INFO Signed narinfos id=80 count=17564worker2 # [ 202.731263] nix-grpc-daemon[1594]: time=2026-09-22T16:07:31.959Z level=INFO msg="Uploading 1 narinfos"7565lb # [ 202.782516] niks3-server[1128]: 2026/09/22 16:07:32 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7566lb # [ 202.792292] niks3-server[1128]: 2026/09/22 16:07:32 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete7567worker2 # [ 202.753626] nix-grpc-daemon[1594]: time=2026-09-22T16:07:31.982Z level=INFO msg="Upload complete. (281ms)"7568lb # [ 202.796956] niks3-server[1128]: 2026/09/22 16:07:32 INFO Completed upload id=807569worker2 # [ 202.755700] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=0 paths=1 nar_bytes_in=5127570lb # [ 202.800426] niks3-server[1128]: 2026/09/22 16:07:32 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7571worker2 # [ 202.759575] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361787572worker2 # [ 202.766352] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)7573worker2 # [ 202.768770] nix-daemon[854]: reaped child process 1807, status = succeeded7574worker2 # [ 202.774035] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)7575client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7576client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7577client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7578client # farm-a> LOG-farm-a-drain7579worker2 # [ 204.190456] nix-daemon[854]: reaped child process 1821, status = succeeded7580worker2 # [ 204.267598] nix-daemon[854]: accepted connection from pid 1837, user nix-grpc-daemon (trusted)7581worker2 # [ 204.280885] nix-daemon[854]: reaped child process 1840, status = succeeded7582lb # [ 204.332314] niks3-server[1128]: 2026/09/22 16:07:33 INFO Received uploads request method=POST path=/api/pending_closures7583worker2 # [ 204.299148] nix-grpc-daemon[1594]: time=2026-09-22T16:07:33.527Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7584worker2 # [ 204.300760] nix-grpc-daemon[1594]: time=2026-09-22T16:07:33.529Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7585lb # [ 204.374987] niks3-server[1128]: 2026/09/22 16:07:33 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7586lb # [ 204.382118] niks3-server[1128]: 2026/09/22 16:07:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign7587worker2 # [ 204.344939] nix-grpc-daemon[1594]: time=2026-09-22T16:07:33.573Z level=INFO msg="Uploading 1 narinfos"7588lb # [ 204.388172] niks3-server[1128]: 2026/09/22 16:07:33 INFO Signed narinfos id=81 count=17589lb # [ 204.394149] niks3-server[1128]: 2026/09/22 16:07:33 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7590lb # [ 204.397787] niks3-server[1128]: 2026/09/22 16:07:33 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7591lb # [ 204.406849] niks3-server[1128]: 2026/09/22 16:07:33 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete7592lb # [ 204.409378] niks3-server[1128]: 2026/09/22 16:07:33 INFO Completed upload id=817593worker2 # [ 204.366304] nix-grpc-daemon[1594]: time=2026-09-22T16:07:33.594Z level=INFO msg="Upload complete. (185ms)"7594lb # [ 204.412820] niks3-server[1128]: 2026/09/22 16:07:33 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7595worker2 # [ 204.369351] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=17596worker2 # [ 204.375560] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:33Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07597worker2 # [ 204.377536] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:33Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17598worker2 # [ 204.382739] nix-daemon[854]: reaped child process 1819, status = succeeded7599worker2 # [ 204.418733] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361787600worker2 # [ 204.426659] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361787601worker2 # [ 204.433322] nix-grpc-daemon[1594]: time=2026-09-22T16:07:33.661Z level=INFO msg="All 1 paths already cached"7602client # copying 1 paths to 'lb:50051'...7603worker2 # [ 204.438281] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361787604client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7605worker2 # [ 204.444108] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)7606worker2 # [ 204.532178] nix-daemon[854]: accepted connection from pid 1846, user nix-grpc-daemon (trusted)7607worker2 # [ 204.546770] nix-daemon[854]: reaped child process 1849, status = succeeded7608worker2 # [ 204.633959] nix-daemon[854]: accepted connection from pid 1851, user nix-grpc-daemon (trusted)7609worker2 # [ 204.646914] nix-daemon[854]: reaped child process 1854, status = succeeded7610lb # [ 204.698325] niks3-server[1128]: 2026/09/22 16:07:34 INFO Received uploads request method=POST path=/api/pending_closures7611worker2 # [ 204.663289] nix-grpc-daemon[1594]: time=2026-09-22T16:07:33.891Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7612worker2 # [ 204.664699] nix-grpc-daemon[1594]: time=2026-09-22T16:07:33.891Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7613lb # [ 204.726929] niks3-server[1128]: 2026/09/22 16:07:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign7614lb # [ 204.735860] niks3-server[1128]: 2026/09/22 16:07:34 INFO Signed narinfos id=82 count=17615worker2 # [ 204.696172] nix-grpc-daemon[1594]: time=2026-09-22T16:07:33.924Z level=INFO msg="Uploading 1 narinfos"7616lb # [ 204.741647] niks3-server[1128]: 2026/09/22 16:07:34 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7617lb # [ 204.748110] niks3-server[1128]: 2026/09/22 16:07:34 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7618lb # [ 204.760446] niks3-server[1128]: 2026/09/22 16:07:34 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete7619lb # [ 204.766828] niks3-server[1128]: 2026/09/22 16:07:34 INFO Completed upload id=827620worker2 # [ 204.723586] nix-grpc-daemon[1594]: time=2026-09-22T16:07:33.952Z level=INFO msg="Upload complete. (274ms)"7621lb # [ 204.768939] niks3-server[1128]: 2026/09/22 16:07:34 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7622worker2 # [ 204.725936] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=0 paths=1 nar_bytes_in=6247623worker2 # [ 204.730467] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361787624worker2 # [ 204.737477] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)7625worker2 # [ 204.738818] nix-daemon[854]: reaped child process 1844, status = succeeded7626client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7627client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7628worker2 # [ 204.743943] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)7629client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7630client # farm-b> LOG-farm-b-drain7631worker2 # [ 205.385579] nix-daemon[854]: reaped child process 1858, status = succeeded7632worker2 # [ 205.468837] nix-daemon[854]: accepted connection from pid 1879, user nix-grpc-daemon (trusted)7633worker2 # [ 205.481339] nix-daemon[854]: reaped child process 1882, status = succeeded7634lb # [ 205.532915] niks3-server[1128]: 2026/09/22 16:07:34 INFO Received uploads request method=POST path=/api/pending_closures7635worker2 # [ 205.498638] nix-grpc-daemon[1594]: time=2026-09-22T16:07:34.727Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7636worker2 # [ 205.502059] nix-grpc-daemon[1594]: time=2026-09-22T16:07:34.728Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7637lb # [ 205.571253] niks3-server[1128]: 2026/09/22 16:07:35 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7638lb # [ 205.587217] niks3-server[1128]: 2026/09/22 16:07:35 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7639lb # [ 205.595115] niks3-server[1128]: 2026/09/22 16:07:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign7640lb # [ 205.600169] niks3-server[1128]: 2026/09/22 16:07:35 INFO Signed narinfos id=83 count=17641worker2 # [ 205.559929] nix-grpc-daemon[1594]: time=2026-09-22T16:07:34.788Z level=INFO msg="Uploading 1 narinfos"7642lb # [ 205.606294] niks3-server[1128]: 2026/09/22 16:07:35 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7643lb # [ 205.616072] niks3-server[1128]: 2026/09/22 16:07:35 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete7644lb # [ 205.619897] niks3-server[1128]: 2026/09/22 16:07:35 INFO Completed upload id=837645worker2 # [ 205.576807] nix-grpc-daemon[1594]: time=2026-09-22T16:07:34.805Z level=INFO msg="Upload complete. (226ms)"7646lb # [ 205.622363] niks3-server[1128]: 2026/09/22 16:07:35 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7647worker2 # [ 205.583463] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=0 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=17648worker2 # [ 205.595628] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:34Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07649worker2 # [ 205.599801] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:34Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17650worker2 # [ 205.603808] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361787651worker2 # [ 205.614338] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:361787652worker2 # [ 205.616794] nix-daemon[854]: reaped child process 1856, status = succeeded7653worker2 # [ 205.622233] nix-grpc-daemon[1594]: time=2026-09-22T16:07:34.850Z level=INFO msg="All 2 paths already cached"7654client # copying 1 paths to 'lb:50051'...7655worker2 # [ 205.627686] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:361787656client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7657worker2 # [ 205.633102] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)7658worker2 # [ 205.724798] nix-daemon[854]: accepted connection from pid 1888, user nix-grpc-daemon (trusted)7659worker2 # [ 205.739793] nix-daemon[854]: reaped child process 1891, status = succeeded7660worker2 # [ 205.824212] nix-daemon[854]: accepted connection from pid 1893, user nix-grpc-daemon (trusted)7661worker2 # [ 205.836726] nix-daemon[854]: reaped child process 1896, status = succeeded7662lb # [ 205.888778] niks3-server[1128]: 2026/09/22 16:07:35 INFO Received uploads request method=POST path=/api/pending_closures7663worker2 # [ 205.855344] nix-grpc-daemon[1594]: time=2026-09-22T16:07:35.084Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7664worker2 # [ 205.856897] nix-grpc-daemon[1594]: time=2026-09-22T16:07:35.085Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7665lb # [ 205.924621] niks3-server[1128]: 2026/09/22 16:07:35 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7666lb # [ 205.932385] niks3-server[1128]: 2026/09/22 16:07:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign7667worker2 # [ 205.894410] nix-grpc-daemon[1594]: time=2026-09-22T16:07:35.123Z level=INFO msg="Uploading 1 narinfos"7668lb # [ 205.937819] niks3-server[1128]: 2026/09/22 16:07:35 INFO Signed narinfos id=84 count=17669lb # [ 205.943221] niks3-server[1128]: 2026/09/22 16:07:35 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7670lb # [ 205.955484] niks3-server[1128]: 2026/09/22 16:07:35 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete7671worker2 # [ 205.920022] nix-grpc-daemon[1594]: time=2026-09-22T16:07:35.148Z level=INFO msg="Upload complete. (281ms)"7672lb # [ 205.963071] niks3-server[1128]: 2026/09/22 16:07:35 INFO Completed upload id=847673worker2 # [ 205.922115] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=0 paths=1 nar_bytes_in=7527674worker2 # [ 205.925169] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:361787675lb # [ 205.968676] niks3-server[1128]: 2026/09/22 16:07:35 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7676worker2 # [ 205.932724] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)7677worker2 # [ 205.935126] nix-daemon[854]: reaped child process 1886, status = succeeded7678worker2 # [ 205.940573] nix-daemon[854]: accepted connection from pid 1498, user nix-grpc-daemon (trusted)7679client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7680client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7681client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7682client # farm-top> LOG-farm-top-drain7683worker2 # [ 206.781268] nix-daemon[854]: reaped child process 1900, status = succeeded7684worker2 # [ 206.862362] nix-daemon[854]: accepted connection from pid 1916, user nix-grpc-daemon (trusted)7685worker2 # [ 206.875582] nix-daemon[854]: reaped child process 1919, status = succeeded7686lb # [ 206.927170] niks3-server[1128]: 2026/09/22 16:07:36 INFO Received uploads request method=POST path=/api/pending_closures7687worker2 # [ 206.890800] nix-grpc-daemon[1594]: time=2026-09-22T16:07:36.119Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7688worker2 # [ 206.892304] nix-grpc-daemon[1594]: time=2026-09-22T16:07:36.120Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7689lb # [ 206.960332] niks3-server[1128]: 2026/09/22 16:07:36 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7690lb # [ 206.969133] niks3-server[1128]: 2026/09/22 16:07:36 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7691lb # [ 206.976896] niks3-server[1128]: 2026/09/22 16:07:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign7692lb # [ 206.982714] niks3-server[1128]: 2026/09/22 16:07:36 INFO Signed narinfos id=85 count=17693worker2 # [ 206.942302] nix-grpc-daemon[1594]: time=2026-09-22T16:07:36.170Z level=INFO msg="Uploading 1 narinfos"7694lb # [ 206.987137] niks3-server[1128]: 2026/09/22 16:07:36 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7695lb # [ 206.993621] niks3-server[1128]: 2026/09/22 16:07:36 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete7696lb # [ 206.998077] niks3-server[1128]: 2026/09/22 16:07:36 INFO Completed upload id=857697lb # [ 206.999771] niks3-server[1128]: 2026/09/22 16:07:36 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7698worker2 # [ 206.956258] nix-grpc-daemon[1594]: time=2026-09-22T16:07:36.184Z level=INFO msg="Upload complete. (208ms)"7699worker2 # [ 206.962463] nix-grpc-daemon[1498]: ts=2026-09-22T16:07:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36178 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=17700worker2 # [ 206.975751] nix-daemon[854]: reaped child process 1898, status = succeeded7701client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&2, in 4.76 seconds)7702client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath7703lb # [ 207.149525] envoy[925]: conn peer=[2001:db8:1::1]:49174 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=562 ms=300397704lb # [ 207.153274] envoy[925]: conn peer=[2001:db8:1::1]:54210 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9499 tx=4381 ms=46027705client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath, in 0.11 seconds)7706worker1: must succeed: rm /nix/.rw-store/fill7707worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.12 seconds)7708worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7709worker1 # [ 208.275756] nix-grpc-daemon[2002]: ts=2026-09-22T16:07:37Z level=info event=healthy reason=min_free7710worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.07 seconds)7711(finished: subtest: low disk drains a worker and builds go to the other, in 8.09 seconds)7712(finished: run the VM test script, in 209.24 seconds)7713test script finished in 209.40s7714cleanup7715kill QemuMachine (pid 49)7716client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7717client # [2026-09-22T16:07:38Z INFO virtiofsd] Client disconnected, shutting down7718client # [2026-09-22T16:07:38Z INFO virtiofsd] Client disconnected, shutting down7719client # [2026-09-22T16:07:38Z INFO virtiofsd] Client disconnected, shutting down7720kill QemuMachine (pid 50)7721lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7722lb # [2026-09-22T16:07:38Z INFO virtiofsd] Client disconnected, shutting down7723lb # [2026-09-22T16:07:38Z INFO virtiofsd] Client disconnected, shutting down7724lb # [2026-09-22T16:07:38Z INFO virtiofsd] Client disconnected, shutting down7725kill QemuMachine (pid 51)7726worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7727worker1 # [2026-09-22T16:07:39Z INFO virtiofsd] Client disconnected, shutting down7728worker1 # [2026-09-22T16:07:39Z INFO virtiofsd] Client disconnected, shutting down7729worker1 # [2026-09-22T16:07:39Z INFO virtiofsd] Client disconnected, shutting down7730kill QemuMachine (pid 52)7731worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7732worker2 # [2026-09-22T16:07:39Z INFO virtiofsd] Client disconnected, shutting down7733worker2 # [2026-09-22T16:07:39Z INFO virtiofsd] Client disconnected, shutting down7734worker2 # [2026-09-22T16:07:39Z INFO virtiofsd] Client disconnected, shutting down7735(finished: cleanup, in 1.06 seconds)7736additionally exposed symbols:7737 client, lb, worker1, worker2,7738 vlan1,7739 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh